Yes, they can be done in 4-6 minutes, my dd was always finished in 5 minutes. But if you say he is slowing down and still swallowing I would let him stay for a while longer on your breast to get more. I think you can let him suck for the sole purpose of pacifying for a little while because this will help increase your supply and as long as he doesn't go to sleep that way you don't create a habbit.
Babies have a strong need to suck and some find their thumb early, some use a pacifier and some thrive on mom's breast. In early weeks, I wouldn't worry about it. Later, do what you think is the best for the two of you.