I'd say you should try a few naps without them now. If he still moves around to the point where you don't feel comfortable, put them back and try again in a couple of weeks. Things change pretty fast, so just keep trying.
Just a note: you won't be able to stop him from rolling over when he finally does (trust me, I tried!!) so my advice is to make the crib as open as possible - remove blankets and plush toys and anything else that may block his air passages. Also, if you have a bumper and want something there to prevent his arms and legs from coming out, get a breatheable bumper. Then, just let him sleep however he sleeps. Keep an eye on him during his naps to get an idea of how he sleeps. This will not only allay your fears but will also give you an idea as to whether there is something else you should be aware of.
