All 3 of mine have done this it does get worrying though as they're all slow weight gainers!!
I wouldn't worry too much, he will eventually get his appetite back!!
From being about 6 ds 1 snapped out of it and now we can't keep him full!!
DS 2 is a little bit better at the moment he's eating everything and ds3 is on hunger strike!! He'll eat breakfast, cereal and a little bit of toast then just a few yogurts

I'd say if he wants the milk definatly give it to him!!
You can also get vitamin drops for lo's who don't eat alot!