Oh those early mornings are hard (((hugs)))
Couple of thoughts. Perhaps you are rousing him too much? Could you just gently stroke his cheek, just enough that he sighs and moves, but not open his eyes?
If you can, I would definitely keep shh-patting until 6.30, then starting the day. If he is calm, the room is dark he may doze off and will start getting into the habit of sleeping later. When 6.30 comes around, turn on the lights, unswaddle to signify 'now it is the morning', then feed.
HTH
anna
x