Why Are Video Games Banded from Certain Households?

My kids love their Wii and I would hate to think how they would react if I threatened to remove it from our home. My kids are young so I don't let them play many games and I certainly don't let them play for long period of times but I have been criticized for letting them play video games at all. Recently I asked a mother who does not allow video games in her home why she was against video games. She simply told me that it was a personal choice of her and her husband. It makes me doubt my decision a little. Am I missing something? Are video games really rotting my children's minds?