It could be the milk ... Any other symptoms of him not tolerating it or it upsetting him? - how are poops? Is he happy in himself? Eating well? Any eczema or refluxing?
If your not sure you could just ditch it, and try again in a month. If you get the same symptoms, you then know...
Hugs x