Unfortunately, I think that if it is affecting him when You consume milk products, via your breastmilk, then it is not a good idea to introduce the directly to him. The amount of milk protein that comes through your breastmilk is quite small in comparison to when the milk products are consumed directly.
You are quite fortunate that he tolerates the goats milk, and that he likes those products, I would stick with those for now. Most babies grow out of these intolerances by 12 months, so maybe consider leaving this trial until that age. Both of my kids have multiple food intolerances, not diagnosed until a later age - and were diagnosed because they got quite serious, so if you can hold off and things are working for you, it isn't a bad idea.
With Zara, her tummy aches got so bad that she pretty much stopped eating - when she started to lose weight, it got serious. With Hugh, we caught it earlier, due to Zara's history, however his presented as behavioural issues - throwing 10-12 very huge tantrums a day, at less than 18 months old. Zara will be 4 in March, and still has not outgrown the intolerances. Hugh is also over 2, and has not outgrown them.