He might be just changing his own schedule to go to bed earlier or changing his feeding time. Take the cereal out of the bottle and mix it pretty thick and feed by spoon breakfast & supper (plus regular amount of milk). This worked for me. My pediatrician recommended 1-2 Tbsp 2x day by spoon at 3 1/2 months, but said never put cereal in his bottle. He sounds like it could be his ears or just really tired. Give it another couple of days before you go back to the doctor unless he really gets worse!