Can you usually predict the time of day it's going to happen? You may want to try a few things. First, nursing in a quiet room (like his nursery) right when he wakes up may help. If you're finding that the afternoon is the tough time, check your own water intake to make sure you're drinking at least 3 - 32 oz glasses of water each day. (He may be upset there isn't enough milk) If it is in the morning, he could be mad that the let down is very strong. It could also be that he has gas or needs to pass a bowel movement? The eating will move his bowels, and that can be painful.
Another suggestion is to let him hold your hand when nursing to cut down on the flailing, or a small soft toy or blanket to cuddle with.