'non-English-speaker' is a compound formed from the prefix 'non-' (from Latin 'non' meaning 'not'), the adjective 'English' (from Old English 'Englisc', referring to England/its people), and the agent noun 'speaker' (from Old English roots 'speca'/'sprecan' meaning 'to speak').

'non-' has long been used as a negative prefix in English (via Latin/Old French usage); compounds using 'non-' plus adjectives or nouns (e.g. 'non-native', 'nonresident') became common in Early Modern to Modern English. The phrase 'non-English speaker' developed as a descriptive compound in the 19th–20th centuries and has appeared written as 'non-English speaker', 'non-English-speaker', or with variations in hyphenation.

The component parts originally meant 'not' + 'of or relating to England' + 'one who speaks'; together they came to be used simply to refer to a person who does not speak English. The basic negative-identity meaning has remained stable.