It's really not uncommon - humans are not meant to be able to digest dairy really, and I think those who don't have any issues with it are actually lucky. I also think a lot of dairy intolerance goes untreated for years, and is actually responsible for a lot of symptoms but people don't realize it. Like pp said, there are so many options these days. And maybe it won't have to be an all or nothing thing. Some dairy is more tolerated than others. Did you ever find a suitable ND? (I know we talked about this on another thread). It's just another thing that an ND can offer some advice on, or even a holistic nutritionist. Too bad you don't live in my town, I know a great one of each!
FWIW, my kids don't show any obvious signs of a dairy intolerance, but it's always in the back of my mind that eating a lot of dairy isn't necessarily good for them, so I do try to limit it anyway even though I technically don't *need* to.