Hi, I think it is worth giving a good week to see if any reactions build up. I guess you would be looking for unsettledness esp if it came with wind or gripy pains, changes in BM, mucous, rashes etc.
I totally understand why you want to add dairy back in and I hope he is OK with it. If not then at least you can go to the GP and say you have tried it and get some hypo-allergenic milk.