Things They Should Study: is ESL harder when both parties are ESL?

I overheard a conversation today between two people, from two different countries, both of whom spoke English as a second language, speaking to each other in English because it's the lingua franca here in Toronto. They seemed to be having some difficulty, and I wondered if it's because both of them spoke English imperfectly in different ways, and they weren't accustomed to each other's imperfections. I didn't hear enough of the conversation to tell if this was the case, or if they would have had as much trouble with a native speaker of English.

However, it also occurred to me that it might be easier when both parties are ESL, because both their vocabularies evolved the same way, from textbook English. I was once told (by an expert in my field) that the typical speaker of English as a Second Language in their professional life has an English vocabulary numbering in the thousands of words, whereas a native speaker of English has an English vocabulary numbering in the hundreds of thousands of words. Most of the time we don't notice this. If someone speaking ESL knows words like "good", "great", "excellent", "fantastic", "wonderful", native speakers probably aren't going to notice that they don't know "groovy", "copacetic", "the bees knees", "gnarly", etc. But native speakers can sometimes come up with words like that and confuse ESL speakers, whereas other ESL speakers most likely wouldn't.

When I was in Germany, there were exchange students from all around Europe there, and how well I managed to converse with them varied based on the quality of their German (and, I'm sure, the quality of my German.) I can't identify any general trends. (My other languages were basically canceled out by the German immersion. After two weeks there, I couldn't even speak French, even though I could still understand it perfectly. When I reached for a French word, it came out in German.)

It would be really interesting to do research on this.