OK well if you are continuing with shush-pat, you don't just keep going at it for hours, that will just exhaust you both. You do shush-pat for 45 mins (at the start of nap time) and after that if he's not settling, you abandon the nap. Take a break outside the nursery, both calm down, change his diaper if necessary, feed if it is near time to, then do your wind-down again and shush-pat again for 45 mins. He may skip all his naps BUT it won't be constant crying for hours on end because after 45 minutes you take a break.
Also, if you feel that shush-pat is making him more aggravated rather than being calming, experiment to find something that works. If he doesn't like the shush, use white noise or hum or sing or speak softly to him instead. If he doesn't like the pat on his back, try patting his butt, or rubbing his back, or stroking his head. If being in your arms doesn't work, try to settle him in his crib more than in your arms.
It's really important that you stay calm. earplugs or earbuds with some music playing can help. If you are freaking out, he's going to pick up on all that cortisol and be fearful that soemthing bad is happening. That's why I suggested the sling, so that YOU can calm down before attempting shush-pat again.
Is he swaddled? Does he take a dummy/paci/soother? Does he have a lovey?