5.3.1 SMTP fundamentals

Overview

Simple Mail Transfer Protocol, usually called SMTP, is the core protocol that moves email from one server to another across the internet. It is used primarily for sending, relaying, and routing messages between mail transfer agents. As a server administrator, understanding how SMTP works, what roles different servers play, and how the protocol flows at the command and response level is essential before you configure any production mail system.

This chapter focuses on the conceptual and technical foundations of SMTP. Practical configuration of a specific server implementation, such as Postfix, will be addressed in its own chapter.

SMTP roles and terminology

SMTP involves several distinct roles. Often, one physical machine can perform more than one of these, but conceptually it is useful to separate them.

A Mail User Agent, or MUA, is the email client that an end user interacts with. This might be a desktop client like Thunderbird, a mobile app, or a webmail interface. MUAs typically do not use raw SMTP directly over the internet. Instead, they submit outgoing mail to a nearby server using a special authenticated submission port, and receive mail using IMAP or POP which are covered elsewhere.

A Mail Submission Agent, or MSA, receives outgoing messages from MUAs. It enforces policies such as authentication, access control, rewriting headers, and applying submission-specific rules. By convention, MSAs listen on TCP port 587 or sometimes 465 when implicit TLS is used.

A Mail Transfer Agent, or MTA, is the SMTP server component that routes mail between domains. MTAs accept mail from MSAs or other MTAs, look up the destination domain using DNS, and deliver messages to the next hop server. Examples of MTAs include Postfix, Exim, and Microsoft Exchange.

A Mail Delivery Agent, or MDA, is responsible for delivering a message locally once the final MTA has accepted it. The MDA might store mail in a mailbox format such as Maildir or mbox, apply filtering rules, and interact with spam and virus scanners. Some software, such as Postfix combined with local delivery utilities, blurs the line between MTA and MDA.

In SMTP conversations, the sending MTA is usually described as the client because it initiates the connection, while the receiving MTA is described as the server.

SMTP ports and transport

Classic SMTP transport happens over TCP port 25. This is the well known port used for unauthenticated server to server communication on the public internet. It is where MTAs exchange mail and where receiving domains are expected to accept incoming mail.

For submission of mail from end users, port 587 is widely used as the mail submission port. On this port, servers usually require authentication, enforce TLS encryption, and apply submission specific policy such as requiring a valid sender address.

Some deployments use port 465 as a legacy implicit TLS submission port. On this port, the TLS handshake happens immediately after the TCP connection is established, as opposed to being negotiated later with the STARTTLS command.

It is important to distinguish between SMTP and message retrieval protocols. IMAP and POP are used by clients to fetch messages from their mailboxes. SMTP is concerned with the path that an email takes from the sender’s system, across intermediate MTAs, to the recipient’s domain.

How SMTP delivery works in practice

When a user sends an email, the MUA hands the message to an MSA which authenticates the user and then hands the message off to an MTA component. The MTA reads the envelope recipients from the SMTP session, not just from the visible headers of the message. For each destination domain, it performs a Domain Name System lookup of MX records.

An MX record identifies the mail exchanger hosts that accept mail for a specific domain. The MTA queries DNS for MX example.com. The answer might list several mail exchangers with different priorities. The MTA picks the lowest preference value first, and if that host is unreachable it tries the others in order.

Once the sending MTA has selected a destination host, it opens a TCP connection to port 25 on that host. If the connection and SMTP session succeed and the remote server issues a final acceptance code for the message, the sending MTA considers its job done. The receiving MTA then either delivers the mail locally using an MDA or acts as a relay that passes it further along the delivery chain.

If the destination cannot be reached or the message is temporarily rejected, the sending MTA queues the mail and retries over a period determined by local policy. If the error persists, the MTA returns a bounce message to the sender address describing the failure.

SMTP session structure and commands

SMTP is a text based protocol. Client and server exchange commands and responses encoded as ASCII over a persistent TCP connection. Understanding the sequence of events in a typical SMTP session helps with debugging problems and interpreting server logs.

A simple SMTP session usually follows this pattern. The client connects, the server sends a greeting banner, the client identifies itself, optional TLS negotiation and authentication may happen, then the client specifies the mail sender and recipients, sends the message content, and finally terminates the session.

Commands are one line, case insensitive, and can include arguments. Responses begin with a three digit numeric code followed by human readable text, sometimes continuing across multiple lines.

A minimal example in abstract form looks like this.

S: 220 mail.example.com ESMTP Ready
C: EHLO client.example.org
S: 250-mail.example.com Hello
S: 250-SIZE 52428800
S: 250-STARTTLS
S: 250 AUTH PLAIN LOGIN
C: STARTTLS
S: 220 Ready to start TLS
[... TLS handshake happens ...]
C: EHLO client.example.org
S: 250 mail.example.com Hello
C: AUTH LOGIN
[... authentication exchange ...]
C: MAIL FROM:<sender@example.org>
S: 250 OK
C: RCPT TO:<user@example.com>
S: 250 Accepted
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: Subject: Test
C:
C: This is a test message.
C: .
S: 250 Message accepted for delivery
C: QUIT
S: 221 Bye

The most important SMTP commands at the protocol level are the following.

HELO or EHLO is the first command a client sends after receiving the server greeting. HELO is the original command from the early SMTP specification. EHLO is the Extended SMTP version which tells the server that the client supports and wants to use modern extensions. The argument is usually the client’s host name or domain. Servers use this to decide how to respond and which capabilities to advertise.

MAIL FROM sets the envelope sender address for the message. This is not necessarily the same as the From header seen by the user. The envelope sender is used for bounce messages and for some anti spam checks. The address is wrapped in angle brackets. Servers may enforce policies on who is allowed to use certain sender addresses.

RCPT TO specifies a single envelope recipient. A message can contain multiple RCPT TO commands, one per recipient. The server can accept or reject each recipient individually, for example if a local address does not exist or is blocked.

DATA tells the server that the client is ready to send the message body. The server replies with code 354 inviting the client to send data. The client then transmits the full message including headers and body. The message is terminated by a line that contains only a single period followed by carriage return and line feed. This dot termination is a crucial detail for correct SMTP behavior.

RSET resets the current session state without closing the connection. It cancels the current message transaction so the client can start again with a new MAIL FROM.

VRFY and EXPN are commands to verify user identities and mailing lists. Many modern servers disable these commands or return generic responses because they can be abused for information gathering.

NOOP is a no operation command that the server simply acknowledges. It is often used to keep idle connections alive.

QUIT signals the end of the session. The server responds with a 221 code and closes the TCP connection.

After EHLO, the server may advertise support for extensions in the form of keywords, such as STARTTLS for opportunistic encryption or AUTH for authentication mechanisms. These capabilities significantly extended original SMTP functionality and are part of what is called Extended SMTP or ESMTP.

Response codes and status classes

SMTP server responses are organized by three digit status codes. The first digit indicates the basic class of the response. Understanding these classes helps you rapidly interpret SMTP logs and errors.

Codes starting with 2 represent success. For example, 220 indicates a service ready greeting and 250 indicates that a command completed successfully. When a 2xx code is returned after the final stage of the DATA command, it means the message has been accepted for delivery.

Codes starting with 3 indicate a positive intermediate response. The server is asking for more information or additional input. The main example is 354, used after the DATA command to request the message content.

Codes beginning with 4 signal transient failures. These represent conditions that might be resolved later. For instance, temporary network issues, overloaded servers, greylisting, or temporary DNS failures. MTAs generally queue mail when they receive 4xx responses and retry later. A common example is 451 which means requested action aborted, local error in processing.

Codes beginning with 5 represent permanent failures. These usually indicate invalid commands, policy rejections, or problems that are not expected to go away by retrying. For example, 550 can mean mailbox unavailable or user unknown. When an MTA receives a 5xx response after attempting to send a message, it should generate a non delivery report to the original sender.

Like many internet protocols, SMTP response codes are specified in standards documents, but individual servers may also include explanatory text. The numeric codes remain the authoritative indicator for automated systems.

Mail routing, relays, and open relays

SMTP itself does not choose arbitrary paths from sender to recipient. Instead, each MTA determines the next hop based on the envelope recipient domains and DNS. However, administrators have control over how their MTAs behave as relays.

A relay is an MTA that accepts mail from another MTA and forwards it onward rather than delivering locally. Many legitimate setups rely on relaying. For example, an internal application server might be configured to submit all outgoing notifications to a central outbound relay, which then delivers mail to external domains. Alternatively, enterprises may use a dedicated spam filtering appliance that acts as a relay in front of their real inbound server.

An open relay is an SMTP server that accepts mail from anyone on the internet and forwards it to arbitrary destination domains without authentication or suitable restrictions. Historically, many SMTP servers operated this way, but open relays are now universally considered a serious misconfiguration. Spammers actively seek out open relays to send large volumes of unsolicited mail. If your server is detected as an open relay, its IP address will almost certainly land on real time blacklists and other reputation systems.

Correct relay configuration is a central security responsibility. MTAs typically allow you to define networks that are permitted to relay, domain based rules, and authentication requirements for submission. Understanding the distinction between local delivery and relaying is fundamental when you begin configuring Postfix or other server software.

SMTP extensions and security concepts

Original SMTP transmitted everything in clear text without authentication. Modern usage relies heavily on a set of extensions to provide encryption, authentication, and other features. These are part of ESMTP and are negotiated using the EHLO command.

Transport security is usually provided through TLS. With the STARTTLS extension, a client begins by talking plain text SMTP, then issues STARTTLS after the initial EHLO. If the server supports it and replies positively, both sides perform a TLS handshake. From that point onward, the session is encrypted. In implicit TLS modes, used for example on port 465, the TLS handshake happens immediately after the TCP connection without an explicit STARTTLS command.

Authentication is advertised and negotiated with the AUTH extension. MUAs use this when submitting outgoing mail to an MSA. Common mechanisms include PLAIN and LOGIN, and more secure methods such as CRAM-MD5 or modern SASL based methods. The exact configuration of authentication mechanisms is handled in the MSA or MTA software and must be chosen with care to avoid exposing credentials.

In transit authentication and reputation systems such as SPF, DKIM, and DMARC are not part of the SMTP protocol itself, but they interact with SMTP sessions through DNS checks and header verification. These mechanisms allow receiving MTAs to evaluate whether incoming mail is authorized by the sending domain and whether headers have been tampered with. They influence acceptance policies, spam scoring, and reputation.

Many other extensions exist, such as 8BITMIME for 8 bit message bodies, SIZE for advertising maximum message size, PIPELINING for sending multiple commands without waiting for each response, and DSN for delivery status notifications. Each extension adds capabilities that can be discovered when you issue EHLO and read the server’s advertised features.

SMTP and message format

SMTP is the transport protocol for email. It does not define the internal structure of the message content beyond some basic requirements. The message transmitted during the DATA phase follows the internet message format defined separately, which uses textual headers followed by a blank line and then the body.

The visible From, To, Subject, Date, and other user friendly fields are part of the headers, and they are separate from the SMTP envelope addresses used by MAIL FROM and RCPT TO. This separation is important because servers may rewrite one without changing the other. For instance, bounces are sent to the envelope sender specified at the protocol level, not necessarily to the From header address seen by the recipient.

Modern email uses MIME to represent attachments, multiple body parts, and different character encodings. MIME also sits above SMTP. From the protocol’s perspective, the entire MIME encoded message is just opaque data within the DATA block.

Administrators must remember that when debugging delivery problems, the SMTP envelope often reveals more than the visible message headers, and server logs usually refer to these envelope level details.

Practical troubleshooting patterns

Knowing the basic SMTP flow and commands lets you perform manual tests and interpret logs when mail delivery fails. Even if you normally rely on configuration tools, it is helpful to be able to inspect an SMTP session directly.

On many systems you can use tools like telnet or openssl s_client to open a connection to port 25 or 587 and issue simple commands like EHLO, MAIL FROM, and RCPT TO interactively. This allows you to see exactly what responses a remote server returns to your MTA. For encrypted connections, you initiate TLS and then continue the conversation over the secured channel.

In log files, you will see references to key SMTP events, such as connection establishment, envelope sender and recipient addresses, receipt of the DATA command, and final acceptance or rejection codes. When delivery fails, look for 4xx or 5xx response codes and the point in the conversation where they occur. A 550 on RCPT TO is very different from a 550 after DATA has been sent, even if they share the same numeric code.

Common operational issues include DNS misconfiguration for MX records, blocked ports by firewalls or ISPs, misconfigured reverse DNS on outgoing IP addresses that causes reputation issues, rejection by remote servers due to missing or incorrect SPF and DKIM records, and policy based rejections that show up as 5xx codes combined with explanatory text.

By combining knowledge of SMTP fundamentals with careful reading of server logs and DNS data, you can systematically isolate where along the chain messages are failing, whether they are being rejected outright or merely delayed, and what configuration changes you need to make at the MTA, DNS, or network level.

Summary

SMTP provides the backbone transport for internet email. It is a simple text based protocol with a predictable sequence of commands and responses, extended over time with encryption, authentication, and other capabilities. Understanding the distinct roles of MUAs, MSAs, MTAs, and MDAs, the typical port usage, the core command set, and the meaning of response codes prepares you to configure and operate real mail servers securely. Later chapters that focus on specific server software and related protocols will build upon this foundation.