I'm in the early camp too. My almost 4-yr-old goes down around 7:30 (never naps), and my 1.5 yr-old goes down at the same time with a nap (2-3 hr naps), or before 7 if she missed her nap. I, too, love the evening down time. We also can just not eat with the kids, which I lament but hope will phase back in when they are older and staying up later.
All that said, I try to be flexible for fun stuff, esp. in the summer when it's light and warm. I think a day a week or so staying up late for social occasions doesn't harm the kids and keeps us involved in our social lives as well! Plus, sometimes we now get those fantastic late sleep-ins when we do cave and let the kids stay up late!