Well, I happen to have the American Academy of Pediatrics book on Nutrition in front of me. I'm going to copy down what it says about Toddler foods and I hope it helps:
"Each day a child between ages 1 and 3 needs about 40 calories for every inch of height, but the amount varies with each child's build and activity level.
Toddlers should have 2 to 3 servings of Dairy, totalling 300-450 calories: Examples are 1/2 cup Milk; 1 inch cube of Cheese; 1/3 cup Yogurt."
So, as long as your son is getting the proper number of servings of quality dairy a day, he should be perfectly fine. Some kids just don't like milk.