Docker Mailserver Private Key Does Not Match Certificate?

Docker Mailserver starts and reports that a private key does not match its certificate. Postfix may log a TLS library problem, Dovecot may refuse to load its SSL context, or one service may remain available with an older certificate while the other fails. The tempting response is to copy certificate files around until startup succeeds. That is risky: it can pair the wrong identity with the wrong key, hide a broken renewal handoff, or expose sensitive material in shell history.

A certificate contains a public key. Its private key must be the unique corresponding half of that pair. The safe diagnosis is therefore to derive and compare public-key fingerprints. You never need to print, paste, email, or commit a private key.

The certificate’s expiry date, filename, and domain label cannot prove that a private key matches it. Compare the derived public keys, then repair the narrowest wrong path, mount, or file pair.

First, preserve the failure evidence

Before restarting repeatedly, capture the service state and a small, private log window:

docker compose ps

docker compose logs --since 10m mailserver

Look for the first certificate, key, PEM, permission, or TLS error—not only the later health-check failure. Keep real hostnames, addresses, account names, full environment output, certificate automation credentials, and private-key paths out of public issues. Do not run docker compose down -v; TLS repair does not require deleting mail volumes.

1. Confirm the effective Docker Mailserver TLS mode

Docker Mailserver supports several certificate arrangements. Diagnose the one the running deployment actually uses. Inspect the relevant environment variable names and mounts without dumping secret values:

docker inspect mailserver \
  --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -E '^SSL_(TYPE|CERT_PATH|KEY_PATH)='

docker inspect mailserver \
  --format '{{range .Mounts}}{{println .Destination " read-only=" (not .RW)}}{{end}}'

With SSL_TYPE=letsencrypt, Docker Mailserver selects certificate material for its configured FQDN from the mounted Let’s Encrypt tree. With SSL_TYPE=manual, SSL_CERT_PATH and SSL_KEY_PATH must name readable files inside the container. Do not mix a certificate from one mode with a key path left over from another.

2. Validate both PEM inputs separately

Run these checks in the environment that can read the mounted files. The commands validate structure and metadata without displaying private-key contents:

# Certificate: parse it and show safe identifying fields
openssl x509 -in /path/to/public.crt \
  -noout -subject -issuer -serial -dates -ext subjectAltName

# Private key: parse and check its internal consistency
openssl pkey -in /path/to/private.key -check -noout

If the first command cannot parse the file, the path may contain the wrong format, an incomplete download, a certificate request, or a private key instead of a certificate. If the second prompts for a passphrase, the key is encrypted; Docker Mailserver’s documented manual setup expects a usable private key file rather than an interactive startup prompt. Do not remove encryption from the only copy: first establish a protected workflow and retain a secure backup.

The certificate input should contain the server leaf certificate followed by any required intermediate certificates. It should not contain a root certificate as a substitute for the leaf, and it should never contain a private key. Check only the PEM labels when classifying a file:

grep '^-----BEGIN ' /path/to/public.crt /path/to/private.key

3. Compare public-key fingerprints safely

Extract the public key from each input, encode it identically, and hash it:

openssl x509 -in /path/to/public.crt -pubkey -noout \
  | openssl pkey -pubin -outform DER \
  | openssl sha256

openssl pkey -in /path/to/private.key -pubout -outform DER \
  | openssl sha256

The two SHA-256 results must be identical. This method works across common RSA and elliptic-curve keys. Older troubleshooting snippets often compare RSA moduli, but those are algorithm-specific; comparing canonical public keys is clearer when the certificate might not use RSA.

Run the comparison on the host pair and again on the exact pair visible inside the container. A host match does not prove that Docker mounted those files:

docker compose exec mailserver sh -lc '
  openssl x509 -in "$SSL_CERT_PATH" -pubkey -noout |
    openssl pkey -pubin -outform DER | openssl sha256
  openssl pkey -in "$SSL_KEY_PATH" -pubout -outform DER |
    openssl sha256
'

Use that example only for a manual-mode installation where those variables are set. For Let’s Encrypt mode, substitute the documented in-container paths for your configured mail FQDN. The commands output fingerprints, not keys, but the paths themselves may still reveal infrastructure details; keep the transcript private.

4. If the host pair matches but the container pair does not

The fault is in the handoff to Docker. Check these boundaries in order:

1. Render docker compose config privately and confirm the intended source and destination paths.
2. Inspect the running container’s mounts; changing Compose does not alter an already-created container.
3. Confirm the files are readable inside the container without broadening the private key to world-readable.
4. For Let’s Encrypt, mount the certificate tree required by Docker Mailserver so live/ symlinks can resolve into archive/.
5. For manual mode, prefer a read-only directory mount containing the managed pair over unrelated one-off copies.
6. Recreate the service only after the rendered configuration points at the matching pair.

A common trap is a file-level bind mount combined with certificate replacement. The host automation writes a new inode, but the container remains attached to the old mounted file. A stable read-only directory mount makes atomic replacement and pair inspection easier. Match this advice to your certificate tool and Docker Mailserver version.

5. If the certificate and key genuinely do not match

Do not generate a random new private key and attach the existing certificate; a certificate authority signed the public key already embedded in that certificate. Choose the correct recovery based on provenance:

• Wrong file selected: locate the key created with the certificate request and verify it by public-key fingerprint.
• Renewal updated only half the pair: repair the renewal copy or mount step so the certificate and key move as one managed set.
• Duplicate certificate lineage: identify the certificate name your automation actually renews and point the service at that complete lineage.
• Matching private key is lost: obtain a new certificate for a newly generated protected key. The old certificate cannot be made to match.
• Key may be exposed: replace both key and certificate, protect the new key, and revoke the affected certificate when appropriate.

Never search for a key by printing every candidate to the terminal. Derive fingerprints locally, label the results, restrict access, and delete temporary diagnostic lists when finished.

6. Check identity and chain after the key pair matches

A matching pair can still be wrong for the service. Confirm that the leaf certificate covers the canonical mail hostname and that the chain is complete:

openssl x509 -in /path/to/public.crt \
  -noout -subject -issuer -dates -ext subjectAltName

The SAN list—not the filename—must include the hostname configured in mail clients. If several mailbox domains share mail.example.com, one certificate for that stable service name is usually sufficient. A matching certificate for an obsolete host is still the wrong identity.

7. Restart deliberately and test both mail services

After the in-container files parse, match, and cover the intended name, restart or recreate the service in a controlled window according to the change you made:

docker compose up -d --force-recreate mailserver
docker compose ps
docker compose logs --since 5m mailserver

Then test from outside the VPS. Postfix and Dovecot are separate TLS consumers, so verify both submission and IMAP:

# SMTP submission with STARTTLS
openssl s_client -connect mail.example.com:587 -starttls smtp \
  -servername mail.example.com -verify_hostname mail.example.com \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -serial -dates -fingerprint -sha256

# IMAPS with immediate TLS
openssl s_client -connect mail.example.com:993 \
  -servername mail.example.com -verify_hostname mail.example.com \
  </dev/null 2>/dev/null \
  | openssl x509 -noout -serial -dates -fingerprint -sha256

Finally, receive one harmless message and submit one authenticated message through the normal client settings. A healthy container alone does not prove that public DNS, port routing, Postfix, and Dovecot all serve the repaired certificate.

Compact acceptance checklist

1. The first startup error has been identified and preserved privately.
2. The effective SSL_TYPE and in-container paths match the intended setup.
3. The certificate and private key each parse successfully.
4. Their derived public-key SHA-256 fingerprints match on the host.
5. The same pair and fingerprints are visible inside the running container.
6. The leaf SAN covers the canonical mail hostname and the chain is complete.
7. The private key has not been printed, copied into Git, or made world-readable.
8. Postfix submission and Dovecot IMAPS serve the expected certificate externally.
9. One controlled receive-and-send test succeeds.
10. The renewal process updates the pair and reloads services together.

Authoritative references: Docker Mailserver’s official SSL/TLS guide documents its Let’s Encrypt and manual certificate modes, in-container paths, key requirements, and external checks. OpenSSL’s official x509 and pkey references document public-key extraction, output formats, and key checks. Docker’s bind-mount documentation explains read-only mounts and host/container path behavior. Use the documentation matching your pinned versions.