Hugs, I know it is hard not to worry, but try not to.
I would keep on offering the milk at his normal feed times. If he refuses, take it away, and offer it 15 mins later, if he refuses again, I would take it away. You don't want to force the issue. He will start taking the milk again.
Have you tried offering a DF to see if you can get a few calories in that way?