I agree that it might be good for you to go with him for a while. Could you go and play there with him for a couple of hours?
Also, it doesn't sound like the caregiver is all that accommodating to his needs. Would he do better at a home daycare setting where there was only one or two more kids and therefore the caregiver could give him more individualized attention? My DD had to go to a babysitter part time when she was 18mo-2yo but the babysitter was a SAHM who only had my DD and her own DD so was able to spend lots of time with them giving the TLC she needed. She didn't have to conform to the structure of the day, rather the babysitter structured the day to the needs of the 2 girls and although she went through a bit of SA, she eventually loved the babysitter and looked forward to going. I hope this makes sense.
Good luck!