the fact that he is starting to move around & waking up is a good indication it is time to remove the swaddle, so he starts to have more mobility to find his own "comfortable" sleeping position
Although I think part of the problem is that at 4mo they are getting really alert & because you are right next to him, he can sense you & in a light sleep phase may become more awake because he sees you, rather than "drifting off back to sleep"... it is a well documented fact tha co-sleeping babies DO wake more often & that goes for having the baby right next to you in a crib... can you move it further away/ hang a blanket over the side? so that he "can't see you" at night.
HTH