At this age, he may well be tired and trying to get to sleep. I would try swaddling him +/- dummy (pacifier) and popping him down for a nap. He's tiny and 40-50mins is probably all he can do A time-wise.
If he were hungry, of course you would feed but it sounds like you're fairly certain he's not hungry, just looking to suck.