Will you mind doing the same when he is older? If not, then I don't see any problem with it.
How does he sleep before the 4/5am waking? If he sleeps through till then, and then you take him to bed with you till 7am, I honestly do not think it's a problem. It's some cuddle time with your child, and babies grow so quickly, if you enjoy the time with him then continue.
BUT, if you don't enjoy it, then you may have to try PU/PD again. Have you tried wake to sleep?