Hugs.
Dairy intolerance is a hard one. I am not saying your Dr is right or wrong, but I have seen research, for example, that says most kids with dairy intolerance will get loose stools. Zara had dairy intolerance with hard stools - constipation. There is another set of research that indicates that kids can get either.
So, I think the best thing to do is try the elimination, there is just too wide and varied range of symptoms for Drs to ever really be sure.