Why does it bother you that I'm quiet?

Question for extroverts:

I've blogged before about how when I was a kid people would say to me "You're so quiet, you never talk."

Question: suppose we're in a randomly-assembled group (classmates working on a project, co-workers on the same shift, people who happen to live in the same neighbourhood waiting for a bus). Sufficient conversation is flowing among the group, but I personally am being quiet.

Why does it bother you that I'm being quiet?

This always happened in randomly-assembled groups with sufficient flow of conversation. Among friends, I'm better able to think of stuff to say (or babble mindlessly and boringly). When there is insufficient flow of conversation, people never seemed to tell me that I'm quiet. The vibe I got is that my quietness bothered them (rather than being a poorly-conceived attempt to draw me out), and googling around this idea I've found that extros are bothered/weirded out by quiet people.

So why, precisely, does it bother you? (Not that I can really do anything about it - I don't have a secret stash of witty conversation that I'm stingily withholding - but I cannot even begin to imagine why this would bother someone.)