Usually pulling away at the breast means something different. "I'm not hungry, just sleepy" or "the flow is too fast/too slow" or "I have a burp/tummyache/reflux" etc.
The normal signs for a baby being upset by something *in* your milk are excessive gas, excema or diaper rash, blood or mucus in the bowel movement.
Good luck figuring it out!