Definitely a very gradual and slow thing. He needs to get used to the fact that even though you're not holding him, you're still there for him, which takes a long time for little brains to adjust to. And I definitely think that once he is able to get more involved in his toys, he won't need you there as much. In a few months when he starts moving around more, he'll be able to move away from you and come back to you too and that's a big step as well. Sounds like you're doing a great job with him!