SMTP fundamentals

Core Concepts of SMTP

Simple Mail Transfer Protocol (SMTP) is the standard protocol used to send email between servers and from mail clients to servers. For server administration, you mainly care about:

• How SMTP moves messages between systems
• Which ports and security modes are used
• How servers decide whether to accept, relay, or reject a message
• Basic commands and responses for troubleshooting

This chapter focuses on protocol fundamentals you’ll use when configuring and debugging MTAs like Postfix, Exim, or Sendmail (covered separately).

SMTP Roles: MUA, MTA, MSA, MDA

SMTP itself is primarily about server-to-server message transfer, but in practice you’ll encounter several roles:

• MUA (Mail User Agent): The client a user interacts with (e.g. Thunderbird, Outlook, webmail frontends).
• MTA (Mail Transfer Agent): Server software that routes and transfers messages between domains using SMTP (e.g. Postfix, Exim, Sendmail).
• MSA (Mail Submission Agent): The part of the server that receives outgoing mail from authenticated users (usually on port 587); typically the same software as the MTA, in a different role.
• MDA (Mail Delivery Agent): Delivers messages into user mailboxes (e.g. dovecot-lda, procmail, maildrop). This is not SMTP-focused.

SMTP fundamentals mainly concern MTA ↔ MTA and MUA/MSA ↔ MTA communication.

SMTP Ports and Security Modes

Common TCP ports used with SMTP:

• Port 25:
• Traditional SMTP for server-to-server transfer (MTA ↔ MTA).
• Also used for client-to-server in legacy setups, but should not require authentication for trusted local sources only.
• Port 587 (Submission):
• For authenticated message submission from clients (MUA ↔ MSA).
• Authentication and encryption are expected.
• Port 465 (SMTPS):
• Historically “smtps” (implicit TLS); deprecated, but widely used again today for submission with mandatory TLS from connection start.

Plain, STARTTLS, and Implicit TLS

Three common security patterns:

1. Plain SMTP
• Connection starts unencrypted.
• No TLS; traffic is visible in transit.
• Still common between MTAs on port 25, but opportunistic encryption (STARTTLS) is widely used.
2. STARTTLS (Opportunistic or Enforced)
• Connection starts plain, then upgraded to TLS using the STARTTLS command.
• Used on ports 25 and 587.
• Opportunistic: server offers STARTTLS; if not available, can fall back to plain (common between MTAs).
• Enforced: client refuses to send mail unless TLS is negotiated (used for submission and high-security links).
3. Implicit TLS (SMTPS)
• Connection is encrypted from the first byte; no plain-text EHLO phase.
• Commonly used on port 465.
• Client expects TLS immediately after TCP connection.

As an admin, you will:

• Accept inbound mail on port 25, typically with STARTTLS.
• Accept outbound mail submissions from users on port 587 and/or 465, always with authentication and encryption.

Basic SMTP Session Flow

A minimal SMTP session (without TLS, authentication, or extensions) looks like this:

1. TCP connection to port 25 or 587.
2. Server greeting:
   220 mail.example.com ESMTP
3. Client identifies itself:
• HELO client.example.org (old-style)
• or EHLO client.example.org (extended SMTP; almost always used today)
4. Envelope sender:
• MAIL FROM:<sender@example.org>
5. Envelope recipient(s):
• RCPT TO:<user@example.com>
• Repeat RCPT TO for multiple recipients.
6. Transfer message data:
• DATA
• Server replies 354 End data with <CR><LF>.<CR><LF>
• Client sends headers and body, then line with only a .
7. Close the session:
• QUIT
• Server replies 221 Bye and closes connection.

The envelope (MAIL FROM / RCPT TO) is what SMTP uses for routing and delivery decisions, not the From: or To: headers inside the message body.

Core SMTP Commands

Understanding the small set of fundamental commands is essential for testing with tools like telnet or openssl s_client.

Connection and Capability

• HELO <hostname>
  Legacy greeting. Useful in debugging, but modern servers expect EHLO.
• EHLO <hostname>
  Extended greeting. Server responds with a list of features, one per line:

  250-mail.example.com
  250-PIPELINING
  250-SIZE 52428800
  250-STARTTLS
  250-AUTH PLAIN LOGIN
  250-ENHANCEDSTATUSCODES
  250-8BITMIME
  250 DSN

Envelope Handling

• MAIL FROM:<address>
  Sets the envelope sender. May include parameters (extensions), e.g. MAIL FROM:<sender@example.org> SIZE=12345.
• RCPT TO:<address>
  Adds an envelope recipient. Server checks whether it can accept that recipient (local mailbox, relay, or forwarding allowed).
• RSET
  Resets the current transaction (clears envelope, but keeps the connection open).

Message Data

• DATA
  Begins the message content transfer:
• Everything after the DATA response until a line containing only . is the message.
• Includes headers and body. Example:

    DATA
    354 End data with <CR><LF>.<CR><LF>
    From: Sender <sender@example.org>
    To: User <user@example.com>
    Subject: Test message
    This is the message body.
    .
    250 2.0.0 OK: queued as 12345

• NOOP
  Does nothing, used to keep connection alive.
• VRFY, EXPN
  Used to verify/expand addresses; often disabled for privacy and anti-abuse reasons.

Session Termination

• QUIT
  Politely ends the SMTP session.

SMTP Response Codes

SMTP servers reply with 3-digit numeric codes and optional text. As an admin, you need to recognize common patterns.

Code Classes

The first digit indicates the category:

• 2xx — Success
  e.g. 250 OK, 220 Ready, 221 Bye
• 3xx — Intermediate
  e.g. 354 Start mail input
• 4xx — Transient failure
  e.g. 450 Mailbox unavailable (temporary), 451 Local error, 421 Service not available
  Clients are expected to retry later.
• 5xx — Permanent failure
  e.g. 550 Mailbox unavailable, 553 Invalid address, 554 Transaction failed
  Clients should not retry automatically.

Common Status Codes

Typical codes you’ll see in logs and diagnostics:

• 220 — Service ready (initial greeting).
• 250 — Requested action completed (e.g. command accepted, message queued).
• 354 — Start message input (response to DATA).
• 421 — Service not available (often due to overload, maintenance).
• 450/451/452 — Temporary failures; often used for graylisting, rate limiting, or resource issues.
• 550 — Requested action not taken. Could mean:
• Nonexistent user
• Rejected as spam
• Relay denied
• 553 — Invalid sender address.
• 554 — Transaction failed (generic permanent failure).

Some MTAs also support enhanced status codes in the form x.y.z, for example:
550 5.1.1 User unknown

Relaying, Open Relays, and Anti-Abuse Basics

One of the most critical SMTP concepts for an admin is relaying.

What Is Relaying?

Relaying is when your server accepts mail for a non-local recipient and forwards it to another server. Examples:

• Accepting mail from a local user and sending it to user@gmail.com.
• Accepting mail from another internal system and sending it out to Internet recipients.

Open Relay (What to Avoid)

An open relay is an SMTP server that will forward mail from any source to any destination without proper checks or authentication.

This is extremely dangerous because:

• Spammers will quickly find and abuse it.
• Your server will end up on spam blacklists.
• Legitimate mail from your domain will be blocked by other providers.

Correct Relay Policy

A typical sane relay policy:

• Accept and relay mail if:
• The sender is authenticated (via SMTP AUTH on submission ports), or
• The client IP is trusted (e.g. local network, VPN), or
• The recipient domain is local (your server is the final destination, not a relay).
• Reject relay attempts from unauthenticated/untrusted sources to arbitrary external recipients with codes like:
• 550 5.7.1 Relaying denied
• or similar.

Your MTA configuration (covered in the Postfix chapter) is where you enforce these rules, but understanding relay logic is part of SMTP fundamentals.

SMTP Envelope vs Message Headers

SMTP makes a critical distinction between:

• Envelope addresses:
• Provided via MAIL FROM and RCPT TO commands.
• Used for delivery and bounce handling.
• Not necessarily visible to the end user.
• Header addresses (inside the message section after DATA):
• From:, To:, Cc:, Reply-To:, etc.
• Displayed by MUAs.
• Can differ from envelope addresses.

Example:

• Envelope:
  MAIL FROM:<bounce-handler@example.org>
RCPT TO:<user@example.net>
• Headers:

  From: Marketing <noreply@example.org>
  To: Customer <user@example.net>

The bounce goes back to bounce-handler@example.org, not to noreply@example.org. This distinction matters when analyzing logs or configuring bounce handling.

Extended SMTP (ESMTP) and Common Extensions

EHLO introduces Extended SMTP, which allows new features. Key extensions you will see:

• SIZE
• Announces maximum acceptable message size.
• Example: 250-SIZE 52428800 (50 MB limit).
• STARTTLS
• Indicates support for upgrading to TLS.
• Clients then send STARTTLS to begin encryption.
• AUTH
• Lists supported authentication mechanisms for submission.
• Example: 250-AUTH PLAIN LOGIN
• Used mostly on ports 587/465.
• PIPELINING
• Allows clients to send multiple commands without waiting for each response, improving efficiency.
• 8BITMIME
• Indicates 8-bit clean transport, allowing messages with 8-bit characters without encoding them as 7-bit.
• DSN
• Delivery Status Notifications support (bounce/receipt formats).

As an admin, you’ll see these in EHLO responses and may enable/disable them in your MTA configuration.

TLS and Certificate Considerations (Protocol View)

From an SMTP perspective (not full certificate management details):

• Server certificate:
• Identifies your mail server during TLS negotiation.
• Should match the hostname clients use (e.g. mail.example.com).
• Must be trusted by major CAs for public Internet use.
• Opportunistic TLS on port 25:
• MTAs try STARTTLS if offered.
• If TLS fails or isn’t available, some will fall back to plain text (unless policy disallows).
• Strict TLS for submission:
• Clients should require TLS on ports 587/465.
• Authentication (AUTH) should only be allowed over a secure channel.

You will configure keys and certificates in your MTA or TLS terminator, but understanding how SMTP uses them is part of the fundamentals.

Basic SMTP Testing from the Command Line

Even with advanced tools available, direct testing at the SMTP level is invaluable.

Unencrypted Testing (Local or Lab Use Only)

Using telnet or nc:

telnet mail.example.com 25
# or
nc mail.example.com 25

Then type commands manually:

EHLO test.local
MAIL FROM:<you@example.org>
RCPT TO:<user@example.com>
DATA
From: You <you@example.org>
To: User <user@example.com>
Subject: Test
Hello from raw SMTP.
.
QUIT

Testing STARTTLS

Use openssl s_client:

openssl s_client -connect mail.example.com:587 -starttls smtp

After TLS is established, issue EHLO and other SMTP commands over the encrypted channel.

These tests help you verify:

• Connectivity (firewall, port openness)
• TLS handshake and certificate
• Authentication availability (AUTH in EHLO)
• Relay behavior and error messages

SMTP, Queues, and Delivery Attempts (High-Level View)

While actual queue management is part of MTA-specific chapters, SMTP fundamentals explain why queues exist:

• When delivering to a remote server, the SMTP client may see:
• DNS failures
• 4xx transient errors (e.g. graylisting, resource problems)
• Timeouts or connection refusals
• Instead of losing mail, MTAs usually:
• Place the message in a queue.
• Retry according to a backoff schedule (e.g. over hours or days).
• Eventually give up and send a bounce if delivery is impossible.

Understanding that SMTP clients differentiate between 4xx (retry later) and 5xx (fail now) is fundamental to reading and interpreting mail logs and behavior.

Summary of Key SMTP Fundamentals for Admins

• SMTP is a text-based protocol that moves mail between servers and from clients to servers.
• Port 25 is for server-to-server; 587/465 are for authenticated submission.
• Commands like EHLO, MAIL FROM, RCPT TO, DATA, and QUIT define the message transaction.
• Envelope addresses control routing and bounces; headers control what users see.
• Relaying must be restricted to authenticated or trusted sources to avoid becoming an open relay.
• Response codes (especially 2xx, 4xx, 5xx) drive delivery and retry decisions.
• TLS via STARTTLS or implicit TLS is essential for secure submission and increasingly important for server-to-server connections.
• Command-line testing of SMTP is a core skill for diagnosing mail delivery problems.

Subsequent chapters will build on these fundamentals to configure specific SMTP servers (like Postfix), handle IMAP/POP3 access, and implement spam and security controls.