What is an authentication protocol, and in which situations is one used?

A (usually cryptographic) communication protocol for transferring authentication data between two entities; it is the precondition for secure communication in networks and applications.

A protocol is an agreed sequence of messages. An authentication protocol is one whose job is to let one party convince another of an identity claim ("I am Alice", "I am the payroll service") without exposing secrets to an eavesdropper and without letting an attacker reuse old messages. Because the messages travel over networks the parties do not control, most such protocols are cryptographic: they rely on signatures, MACs, encryption or challenge-response.

Typical uses:

• System login: a subject authenticates when accessing a system or an application (signing in to a web app, a VPN or a workstation).
• Network authentication: subjects authenticate before they may use network resources (Wi-Fi with 802.1X, a file share, a domain).
• Transaction authentication: a specific action inside a system is authenticated on its own (confirming a payment, approving a high-risk change).

Without a sound authentication protocol, everything built on top (authorisation, auditing, confidentiality) rests on an unverified identity.

Go deeper:

• Authentication protocol (Wikipedia) — definition plus a catalogue of classic protocols (PAP, CHAP, EAP, Kerberos, RADIUS).