We tried unswaddling at 4 mos. for the same reason and it was a disaster. J was just not ready. We decided to keep her in the rock n play where she really couldn't roll over and I tried every few weeks (unsuccessfully) to unswaddle her. At 6 mos, we moved her into her crib, still swaddled. Recently, she figured out how to roll over in the swaddle, so we've pulled 1 arm out. She's adjusting fine and tomorrow we'll pull her other arm free. She's ready now.
Some LOs really take their time emerging from the swaddle. As long as they aren't rolling over at night in the swaddle, I do think there's no rush (of course, talk to your dr. also). Both of mine were happily swaddled til 7 mos. or so. (This is not the "norm", just food for thought. I felt much more relaxed about this after reading the Happiest baby on the block and dr. harvey karp's book about sleep as he's a big swaddle advocate)
I would continue every few weeks to pull an arm out. You'll know when he's ready.