Honestly, my DS didn't do independent play for long periods of time until he was ~18 months old. At 4 months, I wouldn't expect 15-20 minutes to be the norm...for us, it was just a few minutes at a time. Less when he wanted to be mobile and wanted to be moved around. Now, he'll do about 15-20 minutes of one thing, 5 of another, 10 of another...flitting back and forth like a butterfly--a very energetic butterfly
At 4 months old, they're getting more and more interested in the things around them, more easily distracted. Also, just like you said, he's starting to track your movements. So, maybe try a change of scenery or a new toy if he's fussing. But don't feel bad or that he's never going to learn independent play if he doesn't want to/can't handle entertaining himself for long periods of time anymore. Some babies can, others can; it's really relative and normal
HTH a bit! Hoping others will share their experiences, too