I totally, totally hear you on that. My DS is too touchy to sleep anywhere other than his own crib with a fan and the dark. When conditions are perfect he sleeps well, but the slightest thing means nap/bedtime refusal or EW/NW. Often it is things outside of my control (hello, dog barking! You know what I mean!), because I have structured our lives around naptime. I used to think that life would be easier when he was on one nap, and now I know that it won't be easier until he doesn't need the nap at all.
Remind me why we are both TTC #2??
Anyway, I say go for it. Bring the stroller, get him to sleep in there if you can, and enjoy every bite of lunch at a friend's house. You have earned it, Jo.