IIWY I'd go to the Dr. if only for my peace of mind.
Alon has now a temporary milk intolerance and our Dr. said that the clearance take three weeks, with no dairy whatsoever, except for BF milk (which he is anyway not). He said that many parents see an improvement and go back to milk, but that it actually takes that long to clear the body.
But like I said, a Dr. would be my move.