'square off' is formed from 'square' + 'off'. 'Square' ultimately comes from Latin via Old French, where Latin 'exquadrare' (from 'ex-' + 'quadrare') dealt with making something 'four-cornered' or 'even'; 'off' comes from Old English 'of'/'off' meaning 'away' or 'from'.

'square' passed into English from Old French (e.g. Old French 'esquier/escuare') and Latin roots related to 'four' ('quadra-'). The compound phrase 'square off' developed in English (notably in sporting/boxing contexts) to mean 'assume a squared stance' and then more generally 'prepare to confront' in the 18th–19th centuries.

Originally connected with the literal idea of making something square or taking a square stance, it evolved into a figurative sense of setting oneself 'away' from someone — i.e. opposing or confronting them; this figurative confrontational meaning is now the common usage.