Milk is more important and so you offer it first and if he's not interested about an hour later, then you just give him what he's interested in (be it just a few teaspoons) as it shouldn't really matter how much he's having in terms of solids... he should be getting most of his calories from milk.
Cereal can be too harsh for some babies and instead of cereal you could try other foods (avacado and apple puree worked well for dd3).
The other reason (as I mentioned earlier) could be from too much A before bedtime
