I don't think it would be an issue to be honest. It is not something I know much about though to be honest. My ds is allergic to both cows milk and goats milk so we don't really use either. We do have some cheese and yoghurt which is mostly used for my dd and dh's packed lunches. We do easy dairy occasionally in the house but have to be very careful.
Kelly x