no, not too late to start at all. And you can continue to swaddle until they're breaking out of it, or starting to roll in it. At this time of year I think I had him in just a normal cotton babygrow underneath, and could add a light cotton blanket over top if it was a chilly night.