Just our experience, we didn't wean the swaddle until about 5 months. I attempted to wean at 4 months as she had started rolling, but it was a disaster, she would wake herself up all night long. After 2 nights of neither of us sleeping I just decided to swaddle her again, and tried to wean again about 2 weeks later. That time wasn't any more successful, so again we swaddled and then tried to weeks later (about 5 months), and this time she was a champ. I started with 1 arm out, then both out, then removed altogether, and it only took about 3 days. I really think she just wasn't ready when I initially tried, and when she was ready the transition was so smooth and easy.
To me, by what you are describing and my own experience, I would maybe hold onto it for another week or 2. That short time can make all the difference in the world. And then start with 1 arm out and hold for a few days and see how it does.