Murray never had bad skin etc with dairy but would have very squashy poos, be windy and a snotty nose (what a lovely picture, eh!!) - never affected his sleep either. He had a test done for milk allergy or something but that came back fine (apparently it needs to be tested in lab an hour after the sample, which could not happen).
Murray would take Neocate which we used in cooking, then with pureed fruit and now can drink straight (and LOVES!!). I only give him water to drink, and he had Neocate once a day when on his dairy free diet. He has recently re-started dairy and seems okay with it
Btw as a child I would never drink milk and only took it on cereal - turns out I had a milk intolerence so that is probably why I rejected it.