We haven't done the 'big' party yet. I went through the same as you - guilt that you're not putting on a big bash. But, at the time, ds was just not into the social scene like that yet - he's not a great mixer with other children, prefers the company of adults. He'll play one-on-one with a child, but tends to get overwhelmed with more than that.
Anyway, I finally decided to come down on the side of ignoring that nagging guilt, and really thinking what would make ds's day special for him. So, on his third birthday, we went to a local farm, had a nice day, and came back for a birthday tea with my parents, my grandmother, and our neighbours (who ds adores). He had a really good day, and I was happy that we did something for him, not what we were 'supposed' to do.