hi I think you may need to give him more time to get used to eating properly after all those problems with milk allergy. it's not that easy to switch from 2-3 oz a feed to proper feeding.
what i also think is that 2hrs of activity may still be a bit too long for him. so that's why he may be showing tired signs only after some 30 minutes- he's not well rested after previous A time. try maybe limiting his activity to 1,45 hrs and see if it helps. and do try feeding him more or less every 4 hrs [but don't keep him hungry!

] to make him take more at each feed and not snack like he's used to. if you don't separate his feeds in time enough he will just keep on snacking.
and I understand you feel exhausted and hopeless sometimes- we all do. it's not that easy to be a good mom, is it. but keep your head up, you're doing a great job!

hth
Monika