It's completely normal that his dairy allergy would improve when you stopped eating dairy. I would ask if you stopped eating *all* dairy including hidden dairy? If you didn't, that might account for the improvement on formula. Also depends a bit on what's in the formula, that'll give you a hint on what you might consider excluding from your diet while BFing.
BM allergy I have never heard of, so as PP says, it'll most likely be something in the BM. Bear in mind that BM has many benefits that formula does not give, but it is of course your decision whether you stick with BF or not.
BTW, I'm sure you've been given excellent advice, but I would query the need to give you meds that mean you need to stop BFing, even for a few days. There usually are alternatives. If you have any questions about compatibility of drugs with BFing, contact the Breastfeeding Network Drugs in BM service where a qualified pharmacist who also knows a lot about BFing will help you. They even have a page on FB!