If you have had digestive problems since college, then I would think it could be a problem.
So many people avoid dairy and gluten for many reasons. Some swear it just helps them feel better over all.
For me personally, I thought I was gluten sesitive because I had no digestive problems when on the carb diet. After just recently going to the allergist, I realized I am allergic to white pototoes. Since not eating them, I have had no digestive problems.
Heck, since your lo has to avoid dairy, I would try it myself and see how you feel.