I would start by double checking all the things you are eating to make sure there is no hidden dairy. How long have you been dairy free for? I might give in a couple of weeks to see before starting to eliminate other foods, and then I would def. start with the more common allergens like egg, wheat, etc.
I would take a guess that it falls under intolerance, as a true allergy generally has symptoms like hives/swelling/anaphylaxis whereas intolerance is mainly gastro issues (or at least it was for us, and ezcema).