Docker Mailserver SMTP Connection Timeout: Test Ports 25, 465, and 587

A mail client reports “connection timed out.” Another server cannot deliver to your domain. Docker says the mail container is healthy, yet a port checker says SMTP is closed. These observations can all be true because they test different paths.

The useful fix is not to open every mail-related port. It is to identify which connection is failing, then follow that connection through DNS, the VPS network, host firewall, Docker port mapping, container listener, and TLS handshake. The examples use mail.example.com and a Compose service named mailserver. Replace them locally, and never paste an unredacted environment, real server address, mailbox password, or private key into a public diagnostic.

A timeout, a refusal, and a TLS error are different evidence. Preserve the exact error before changing the firewall.

First: know what each SMTP port is for

• 25/tcp — SMTP transport: mail servers use this to deliver mail to one another. Your server normally needs inbound port 25 to receive internet mail and outbound port 25 to deliver directly to other mail servers.
• 465/tcp — submissions with implicit TLS: a mail client connects through TLS from the first byte, authenticates, and submits outgoing mail.
• 587/tcp — message submission with STARTTLS: a client begins SMTP, upgrades the connection with STARTTLS, authenticates, and submits mail.

Ports 465 and 587 are not substitutes for public server-to-server delivery on port 25. Opening submission while inbound 25 is blocked may let your own client send into Docker Mailserver, but other domains still cannot deliver mail to you. Conversely, a working inbound port 25 does not prove that a phone or desktop client can authenticate on submission.

Docker Mailserver’s current port guide documents the protocol and encryption behaviour. Check the documentation matching your pinned image version rather than enabling every historical POP3 and IMAP port.

1. Classify the failure before testing

• Timed out: no TCP answer arrived. A firewall, provider filter, wrong address, routing problem, or silent packet drop is likely.
• Connection refused: the destination actively rejected TCP. The host was reachable, but nothing was listening on that address and port, or a firewall rejected it explicitly.
• Connected, then TLS failed: networking works. Investigate protocol choice, STARTTLS versus implicit TLS, certificate hostname, trust chain, or service configuration.
• Connected, then authentication failed: the listener and TLS path work. Investigate account credentials, SASL, bans, and client settings without widening network access.

2. Confirm DNS points at the intended mail host

Start from a machine outside the VPS. Confirm the mailbox domain’s MX target and the mail hostname’s address records:

dig +short MX example.com
dig +short A mail.example.com
dig +short AAAA mail.example.com

An old A record can send IPv4 clients to a previous server. A published AAAA record can make IPv6-capable senders try an unreachable or unconfigured IPv6 path even while IPv4 tests pass. Remove a wrong record or make that protocol work; do not assume clients will reliably fall back.

Cloudflare’s orange-cloud proxy does not proxy ordinary SMTP ports. Keep the mail hostname DNS-only unless you deliberately use a separate supported mail proxy. The MX record should name a hostname, not a copied address.

3. Test from outside with the correct TLS mode

Run these from another network, not only from the mail host:

# TCP reachability without sending credentials
nc -vz -w 5 mail.example.com 25
nc -vz -w 5 mail.example.com 465
nc -vz -w 5 mail.example.com 587

# Port 25 and 587: SMTP upgraded with STARTTLS
openssl s_client -starttls smtp -connect mail.example.com:25 \
  -servername mail.example.com -brief
openssl s_client -starttls smtp -connect mail.example.com:587 \
  -servername mail.example.com -brief

# Port 465: TLS starts immediately
openssl s_client -connect mail.example.com:465 \
  -servername mail.example.com -brief

Do not test port 465 with -starttls smtp, and do not treat a plain telnet session to 465 as proof that TLS is broken. Save whether TCP connected, whether a certificate was presented, and whether the SMTP banner belonged to the expected host.

4. Verify Docker published the ports

A process listening inside a container is not automatically reachable from the internet. Compose must publish the required host ports:

services:
  mailserver:
    image: mailserver/docker-mailserver:<pinned-version>
    hostname: mail.example.com
    ports:
      - "25:25"
      - "465:465"
      - "587:587"
      - "993:993"

Inspect the rendered configuration and the running mappings without printing environment values:

docker compose config --services
docker compose ps
docker port mailserver
ss -lnt | grep -E ':(25|465|587|993)[[:space:]]'

Docker’s port-publishing documentation warns that published ports are externally accessible by default when bound to all host addresses. Do not publish management ports or bind mail submission only to loopback by accident. If a Compose change added a port, recreate the service; a simple process restart does not apply a new container port mapping.

5. Check listeners and logs inside the container

If the host mapping exists but the connection is refused, verify that Postfix is actually listening:

docker compose exec mailserver ss -lnt
docker compose logs --since 15m mailserver
docker compose exec mailserver postconf myhostname

Compare the time of an external test with the logs. No log entry usually means the connection never reached Postfix. A logged TLS or authentication error proves that the network path reached the application, so stop changing perimeter rules and diagnose that later layer.

6. Audit every firewall layer, including Docker’s rules

There may be several independent filters: a provider firewall, the VPS host firewall, Docker-managed packet-filtering rules, and sometimes an upstream network block. Check policy without dumping unrelated addresses or rules into public notes.

sudo ufw status numbered
sudo nft list ruleset
# Or, on hosts using the iptables backend:
sudo iptables -S
sudo iptables -S DOCKER-USER

Do not assume that an “allow” in UFW fully describes traffic to published containers. Docker documents that published traffic can be diverted before it reaches UFW’s normal chains. Use Docker’s supported firewall model and the DOCKER-USER chain where appropriate; avoid flushing Docker-generated rules on a production host.

Restricting submission ports to known office addresses may break roaming phones and laptops. Prefer authenticated TLS, strong passwords, rate limits, and banning abusive clients. Port 25 must accept connections from arbitrary legitimate mail servers if you receive public email.

7. Separate inbound and outbound port 25 blocks

Many VPS and access providers apply anti-abuse controls to SMTP. Test the two directions independently:

• Inbound test: connect from an external host to mail.example.com:25.
• Outbound test: from the VPS, connect to port 25 on a mail server you are authorised to test.

If inbound reaches the container but outbound consistently times out before any Postfix remote response, ask the provider whether outbound SMTP is restricted and what legitimate-use approval process applies. Do not bypass a provider block with scans, rotating addresses, or unapproved tunnels. If direct delivery is unavailable, configure a reputable authenticated relay and update SPF and related mail policy to describe the real path.

8. Use packet capture only to answer one question

When rules look correct but an external connection still times out, a short, filtered capture can show whether packets arrive:

sudo tcpdump -ni any 'tcp port 25 or tcp port 465 or tcp port 587'

Start the capture, make one controlled external connection, then stop it. No inbound SYN suggests DNS, provider firewall, upstream routing, or the client path. Repeated SYN packets with no SYN-ACK points at the host’s filtering or routing. A completed handshake followed by application packets moves the investigation into Postfix, TLS, or authentication. Packet captures can contain addresses and payload metadata, so do not publish them unredacted.

A compact SMTP connectivity checklist

1. Record the exact timeout, refusal, TLS, or authentication error.
2. Identify whether the failing path is server delivery on 25 or client submission on 465/587.
3. Confirm MX, A, and AAAA records lead to the intended host.
4. Test TCP and the correct TLS mode from outside the VPS.
5. Confirm Compose publishes the port and the host is listening.
6. Confirm Postfix listens inside the container and compare logs with the test time.
7. Audit provider, host, and Docker firewall behaviour without flushing rules.
8. Test inbound and outbound port 25 separately.
9. Use one filtered packet capture if the dropping layer remains unclear.
10. After networking works, verify TLS, authentication, real send/receive, and logs.

A green container health check proves only what that health check exercises. Public SMTP works when an outside sender reaches the right address on port 25, Postfix answers, and a complete delivery succeeds. Client submission works when the chosen 465 or 587 TLS mode reaches the listener, presents the expected certificate, authenticates, and queues a message. Test those contracts separately and the vague “SMTP is down” report becomes a specific, fixable layer.