Coolify PostgreSQL “database files are incompatible with server”

A PostgreSQL resource in Coolify can enter a restart loop immediately after its Docker image changes across major versions. The useful log lines usually look like this:

FATAL: database files are incompatible with server
DETAIL: The data directory was initialized by PostgreSQL version OLD_MAJOR,
which is not compatible with this version NEW_MAJOR.

This is not an authentication problem and it does not prove that the data disappeared. PostgreSQL found an existing cluster, read its on-disk version, and refused to start a server whose major version cannot use that cluster directly. That refusal is a protection boundary.

Do not delete the Coolify volume, run initdb over it, empty the data directory, or keep changing image tags. Preserve the volume, prove both major versions, and choose recovery before migration.

What the error means

PostgreSQL guarantees that minor releases within one major version retain compatible internal storage. A patch update within major 16, for example, does not require a data-format migration. A move from major 16 to 17 is different: PostgreSQL requires a logical dump and restore, pg_upgrade, or a supported replication-based migration.

The error commonly follows an edit from postgres:16 to postgres:17, a service template update, or a floating tag that now resolves to a new major. It can also appear after the right image is attached to the wrong volume. Treat the configured image, running image, mount, and data-directory marker as four separate facts.

1. Stop restarts and preserve evidence

Pause the application, workers, schedulers, and any process that writes to this database. Stop the failing database resource through Coolify if repeated restarts are obscuring logs. Record the resource identifier, previous successful deployment time, configured image tag, current image identifier, exact PostgreSQL error, and attached persistent storage—without publishing credentials, connection URLs, host addresses, or private paths.

Before changing anything:

• take a storage snapshot if your authorised storage layer supports a crash-consistent copy;
• preserve the original volume and do not rename it ambiguously;
• preserve the last known-good logical backup outside that volume;
• confirm enough free space for a second cluster, dump files, indexes, and temporary upgrade work;
• note the last known image tag or digest from Coolify history and deployment logs.

A filesystem copy made while PostgreSQL was writing is not automatically a valid backup. If the old server can still be started safely, create a fresh logical backup after recovery. Coolify’s own backup guidance also says that a created file is not enough: restore a copy into a disposable target before relying on it.

2. Prove the data-directory major without starting it

A PostgreSQL cluster stores its major version in a plain-text PG_VERSION file at the effective PGDATA root. Inspect it read-only from an authorised maintenance context. First use container or Coolify configuration to identify the actual mount and PGDATA; do not assume a path copied from another PostgreSQL release.

# Generic shape only: substitute the verified stopped volume and path.
docker run --rm --read-only \
  -v DATABASE_VOLUME:/cluster:ro \
  alpine:PINNED_VERSION \
  sh -c 'cat /cluster/PG_VERSION'

If the marker is not at that path, stop rather than searching with a process that can modify files. PostgreSQL 17 and below commonly use /var/lib/postgresql/data in the official image. PostgreSQL 18 and later changed the default to a version-specific directory below /var/lib/postgresql. A mount-path change can therefore create a second empty cluster beside the first or make the expected marker appear missing.

Record only the resulting major number. Do not edit PG_VERSION. Changing that text cannot convert catalogs or data files and may remove the guard that is preventing damage.

3. Prove the server image major

Compare the effective image configured in Coolify with the image used by the failing container and the server binary inside that exact image:

docker run --rm postgres:PINNED_MAJOR postgres --version

Use a known major tag or immutable digest during incident work, not latest. Also check whether the resource came from a Compose service whose image value or variable overrides what the surrounding Coolify page suggests. The diagnosis is complete only when the data marker says one major and the effective server binary says another.

4. Restore service on the old major first

If this was an accidental image change and the new server refused before modifying the cluster, the narrow recovery is usually to configure the PostgreSQL resource with the matching old major image and its original effective PGDATA mount. Prefer the previously proven tag or digest. Do not create a new volume, change ownership broadly, or run initialisation scripts.

1. Keep applications and workers stopped.
2. Set the database image back to the cluster’s verified major.
3. Confirm the original volume targets the path expected by that image.
4. Start only PostgreSQL and wait for a normal ready-for-connections log.
5. Query the server version and database identity through a private authorised connection.
6. Run consistency-oriented application checks before resuming writes.
7. Create a new logical backup and restore-test it in isolation.

SHOW server_version;
SHOW server_version_num;
SELECT current_database(), current_user;

If the old major does not start, do not escalate to random tag changes. Recheck mount identity, PGDATA, ownership, disk capacity, and the first startup error. The separate Coolify PostgreSQL data-directory permission guide covers ownership failures without broad permission changes.

5. Choose a real major-upgrade method

Once the old cluster is healthy and backed up, choose an upgrade path based on database size, extensions, allowable downtime, and operational experience.

Logical dump and restore

For many small and medium Coolify databases, a parallel new PostgreSQL resource plus logical dump and restore is the clearest path. It leaves the source cluster intact during rehearsal, makes rollback boundaries understandable, and lets you validate the destination before changing the application connection.

1. Create a separate private PostgreSQL resource on the target major with a separate volume.
2. Install and verify target-compatible extensions.
3. Use the newer major’s pg_dump or pg_dumpall against the old server.
4. Restore into the new cluster and read every error, especially role, owner, extension, collation, and privilege failures.
5. Run schema, count, constraint, sequence, and application-level checks.
6. Practise the final write freeze and measure its duration.
7. During cutover, stop writers, take the final dump, restore it, verify recency, then switch the Coolify runtime connection and recreate consumers.

Coolify’s selected-database backups use custom-format pg_dump archives and its Import Backup workflow uses pg_restore. “Backup all databases” uses a different pg_dumpall flow. Match the archive to the restore tool and inspect version compatibility before the outage window.

pg_upgrade

pg_upgrade is faster for large clusters, but it is not equivalent to changing an image tag. It requires both old and new server binaries, old and newly initialised data directories, compatible build settings, matching extension binaries, stopped servers, and a deliberate copy, clone, link, or other transfer mode.

Run the new version’s pg_upgrade --check first in a tested environment that reproduces the actual mounts and extensions. PostgreSQL warns that link mode makes rollback unsafe after the new cluster starts because old and new clusters share files. Copy mode needs more time and space but keeps the old cluster unmodified. Do not improvise pg_upgrade inside the only Coolify volume during an outage.

Logical replication

For larger databases that need a shorter write freeze, logical replication can keep a new-major subscriber near current before cutover. It adds operational work: table eligibility, initial copy, sequences, schema changes, replication slots, extension compatibility, lag monitoring, and cutover ordering all need explicit handling. Use it only when the lower downtime justifies that complexity.

6. Account for the PostgreSQL 18 image layout change

The Docker Official Image changed its default PGDATA and declared volume for PostgreSQL 18 and later. Major 18 defaults to /var/lib/postgresql/18/docker below a volume mounted at /var/lib/postgresql. PostgreSQL 17 and earlier normally expect the persistent mount at /var/lib/postgresql/data.

This layout is designed to make future major upgrades easier, but it makes a 17-to-18 container edit especially unsuitable as a migration. Before building an upgrade procedure, document:

• the source image and effective PGDATA;
• the source volume and mount target;
• the target image’s default layout;
• whether an explicit PGDATA overrides either default;
• where old and new clusters will coexist during rehearsal.

Do not move database files merely to make the new entrypoint stop complaining. Directory placement and data-format migration are different problems.

7. Verify the destination before cutover

A successful server start proves only that PostgreSQL can open the target cluster. Before directing production traffic to it, verify:

• the expected target major, database names, roles, schemas, and extensions;
• representative table counts, latest-record timestamps, constraints, indexes, and sequence positions;
• application migrations against the intended database and schema;
• real read paths, one controlled write/read-back, workers, and scheduled jobs in an isolated test;
• encoding, collation, timezone assumptions, and extension behaviour;
• a new Coolify backup from the target and a disposable restore of that backup;
• the documented rollback point before old resources or volumes are removed.

At final cutover, freeze every writer—not just the web container. Queue consumers, scheduled jobs, admin scripts, and integrations can create a split dataset after the final dump. Change the application’s runtime connection through Coolify, recreate all long-running consumers, and assert unique database-backed content rather than accepting a generic HTTP 200.

Wrong fixes that turn recovery into loss

• Editing PG_VERSION: it changes a marker, not the on-disk format.
• Deleting the volume: it removes the cluster PostgreSQL was protecting.
• Running initdb on the old path: it can overwrite the evidence and data needed for recovery.
• Using latest again: the next pull can repeat the uncontrolled major jump.
• Mounting a new empty volume and declaring success: a healthy empty database is not the original database.
• Copying raw files into a new-major cluster: filesystem copy is not a PostgreSQL major upgrade.
• Restoring over the only source: build and verify a separate destination first.
• Deleting the old cluster immediately after cutover: retain it offline until verification and rollback windows close.

Compact recovery order

1. Stop writers and preserve the exact first error.
2. Snapshot or protect the original volume and last known-good backup.
3. Read the source cluster’s PG_VERSION without modifying it.
4. Identify the effective server image, binary major, mount, and PGDATA.
5. Recover the original cluster with its matching pinned major.
6. Create and restore-test a fresh logical backup.
7. Rehearse dump/restore, pg_upgrade, or replication into a separate target.
8. Freeze all writers, perform the final migration, and switch every consumer once.
9. Verify data, application behaviour, logs, and target-major backups.
10. Retain the stopped source until the rollback window closes.

The key distinction is recovery versus upgrade. Returning the matching old server to the untouched old cluster restores service; it does not complete an upgrade. A major upgrade creates or transforms a target cluster through a PostgreSQL-supported method and proves that the application and recovery process still work afterward.

Authoritative references: PostgreSQL documents major-version upgrade methods and minor-version storage compatibility and the requirements and rollback implications of pg_upgrade. The Docker Official Image documents its PGDATA and volume layouts, including the PostgreSQL 18 change. Coolify documents PostgreSQL backup formats and restore testing and its controlled database restore workflow.