Your LO isn't really missing out, if he/she doesn't get TV time. I think it all depends on why you're considering it, and it depends a lot on the type of programming. It's really a personal decision.
A lot of people don't want their LOs to watch ANY TV, and I respect that. But I also don't think there's anything wrong with them watching something like Mr Rogers or Sesame Street. I grew up on all the PBS shows, and I still fondly remember Big Bird, Cookie Monster, Grover, the Letter People (wish that was still on!). I'm convinced those show contributed to me reading at 2.5 yrs old.
Our DS is 8.5 mos, and he watches no more than 1 hr/day, no more than 30 mins at a time, and only PBS kids programs or his sing-along videos (we have one with live preschoolers singing & dancing, and an old muppet sing-along video). I'm a SAHM, so he gets PLENTY of mom time to balance it out.
As for your question about it being too stimulating... it depends on your LO, and what programming you choose.