Coconut milk! I generally can't abide the taste of coconut but when I started having it unsweetened I realized I actually like it a great deal. I now have a morning smoothie which is half coconut milk right from the can. And some people do use it in coffee and tea, although frankly I prefer mine unsullied by additives. (coffee and tea, that is)
For more good calories, you can cook with coconut oil and even eat a teaspoon of it once (or even a couple times!) a day. It's a good fat, and can also be used topically, and it's antimicrobial (fight bacteria, fungi, and viruses) AND it is THE best source of lauric acid outside breast milk. We use so much we literally order it by the gallon online.
And WTH at the doc and HV? Allergies and intolerances aren't the same in the first place, and the baby has TWO parents, not just YOU, AND babies' systems aren't mature yet so allergies and intolerances can be more common in babies. Yeesh. Good on you for giving it a go!