A (usually cryptographic) communication protocol for transferring authentication data between two entities; it is the precondition for secure communication in networks and applications.

A protocol is an agreed sequence of messages. An authentication protocol is one whose job is to let one party convince another of an identity claim ("I am Alice", "I am the payroll service") without exposing secrets to an eavesdropper and without letting an attacker reuse old messages. Because the messages travel over networks the parties do not control, most such protocols are cryptographic: they rely on signatures, MACs, encryption or challenge-response.

Typical uses:

• System login: a subject authenticates when accessing a system or an application (signing in to a web app, a VPN or a workstation).
• Network authentication: subjects authenticate before they may use network resources (Wi-Fi with 802.1X, a file share, a domain).
• Transaction authentication: a specific action inside a system is authenticated on its own (confirming a payment, approving a high-risk change).

Without a sound authentication protocol, everything built on top (authorisation, auditing, confidentiality) rests on an unverified identity.

Go deeper:

• Authentication protocol (Wikipedia) — definition plus a catalogue of classic protocols (PAP, CHAP, EAP, Kerberos, RADIUS).

• Two or more parties are involved, and each knows the protocol in advance.
• All parties follow the protocol.
• It is unambiguous: every step is exactly defined.
• It is complete: there is a defined action for every possible situation.

These four rules are what separate a protocol from "some messages that usually work":

1. Known in advance by all parties: you cannot negotiate the meaning of a message while receiving it. If the client and server disagree about what a field means, authentication silently fails or, worse, succeeds wrongly.
2. Followed by everyone: a protocol's security guarantees only hold if every participant executes it. A party that skips a check (say, a client that does not validate a signature) breaks the guarantee for everyone.
3. Unambiguous: each step must have one interpretation. Ambiguity is a classic source of attacks, where an attacker crafts a message that one side reads one way and the other side another way.
4. Complete: the protocol must say what to do for every case, including errors, timeouts and unexpected messages. Undefined situations are where implementations improvise, and improvised error handling is a common source of vulnerabilities.

Tip: OAuth and OIDC specifications are long precisely because of rule 4: much of their text describes error responses and validation steps, not the happy path.

Go deeper:

• Communication protocol (Wikipedia) — the general requirements a protocol must meet, with examples from networking.