A container starts, disappears, and comes back with the same error. The tempting fix is another restart. Before doing that, find out what made the process exit. A bad configuration, a full filesystem and a killed process can all produce the same Restarting status, but need different repairs.
This guide is for a Linux home server running standalone Docker Engine and Docker Compose v2. The shell examples use Bash and GNU tools, as found on Ubuntu and Debian. Docker Desktop, Swarm and Kubernetes have different storage or restart behaviour.
1. Capture the failure before changing anything
First, check which Docker daemon you are talking to and list stopped containers as well as running ones:
docker context show
docker ps -a --no-truncIf your Docker context points to another server, run the host checks later in this guide on that server. Use the same Docker account and connection throughout. Adding sudo to a rootless setup can point commands at a different daemon.
Choose the failing container from that list. Replace the example name below with its exact name or ID; this variable is reused in the following commands.
CONTAINER='replace-with-container-name'
docker logs --since 30m --tail 200 --timestamps "$CONTAINER"
docker inspect --format \
'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}} started={{.State.StartedAt}} finished={{.State.FinishedAt}} error={{json .State.Error}}' \
"$CONTAINER"Read the error immediately before the application exits. Repeated startup banners are usually less useful. Look for a missing setting, a database connection failure, a denied file operation or a failed migration.
The inspection output is a snapshot of the latest state, not a history of every crash. A rapidly restarting container may change state between commands. Save the relevant output and its timestamp before making changes. Redact tokens, passwords and private addresses before sharing it.
Empty Docker logs do not prove that the application has nothing to report. It may write to files, or its logging setup may not make logs available through this command. Check the application's documented log location.
Next, read recent events for the same container. The end time makes this command finish instead of waiting for future events:
docker events \
--since 30m \
--until "$(date --iso-8601=seconds)" \
--filter "container=$CONTAINER"Events can help line up an oom, kill or die with an application error. Docker only returns the last 256 retained events, so a busy host can have gaps even within that half-hour window. An absent event is not proof that something never happened.
Record the image and restart policy too:
docker inspect --format \
'image_ref={{.Config.Image}} image_id={{.Image}} restart_policy={{json .HostConfig.RestartPolicy}}' \
"$CONTAINER"This is especially useful if the problem began after an update. An image tag alone may not identify the exact build that ran.
2. Read the status and exit code together
| What you see | What to investigate |
|---|---|
Restarting | The main process has exited and Docker is attempting another start under its restart policy. Find the exit cause in logs and events. |
Up … (unhealthy) | The process is running, but a configured health check is failing. Inspect that check and the application response. |
Exit 0 | The process finished successfully. Was it a one-off task placed under an always policy, or a startup script that exited instead of keeping the service in the foreground? |
Exit 1 | A general failure. The preceding application message matters more than the number. |
Exit 137 | The process received SIGKILL. Check memory evidence, forced stops and other sources of a kill. |
Exit 143 | Usually termination by SIGTERM. Compare the time with a deployment, manual stop or shutdown. |
In standalone Docker Engine, a failed health check does not by itself restart a container. Restart policies act when the main process exits. An auto-heal container, systemd unit or another external manager can add its own behaviour, so check those if the sequence does not match.
For an unhealthy container, inspect the recent health-check output:
docker inspect --format '{{json .State.Health}}' "$CONTAINER"This is useful only when a health check is configured. A missing command inside the image, a wrong port or an endpoint requiring authentication can break the check while the service itself is still running. Health output can also contain sensitive information.
A restart policy is a recovery mechanism, not a repair. always and unless-stopped can restart a process after a successful exit; on-failure is tied to a non-zero exit. Choose a policy that matches the job after identifying why it stops.
3. Check memory before blaming exit code 137
137 often sends people straight to a RAM upgrade. It tells you the process was killed, but not who killed it. A stop operation that exceeds its grace period can also end with SIGKILL.
docker stats --no-stream "$CONTAINER"
docker inspect --format \
'memory_limit_bytes={{.HostConfig.Memory}} memory_swap_bytes={{.HostConfig.MemorySwap}}' \
"$CONTAINER"
free -h
sudo journalctl -k --since '30 minutes ago'The last two commands belong on the Docker host. The journal command needs permission to read the kernel log; some hosts use a different logging setup. Match timestamps and process details rather than assuming every host OOM message concerns this container.
OOMKilled=true, a matching event and a kernel message make a memory diagnosis much stronger. docker stats shows current usage, not the peak just before a crash. A memory-limit value of 0 means no container hard limit is set; the host still has finite memory.
Then check the workload. Did a scheduled import, thumbnail job or backup overlap with another heavy task? Did you give a database a limit smaller than its configured cache? The useful fix might be lower application concurrency, different scheduling, a leak fix or a carefully chosen limit change. Raising a limit without checking host capacity can move the failure to another service.
Do not disable the OOM killer as a general fix. Keep a record of the failing workload so you can repeat that workload after the change.
4. Check storage, configuration and permissions
No space left on device
An application may exit because it cannot create a lock file, write a database journal or append a log. Check both bytes and inodes on the host:
df -hT
df -i
docker inspect --format \
'{{range .Mounts}}{{println .Type .Source "->" .Destination "rw=" .RW}}{{end}}' \
"$CONTAINER"Use the mount list to identify the filesystem that actually holds the application data. The root filesystem can have room while a separate data disk is full. Quotas or a read-only filesystem can also prevent writes when df looks acceptable.
Follow the Docker disk-full troubleshooting guide for logs, build cache, volumes and inode problems. Inspect the data first. Deleting a volume because a database will not start can turn an outage into a lost database.
Missing settings or the wrong bind mount
If the log says a configuration file is missing, compare its path inside the container with the mount destination above. Then check the corresponding host source. A typo can leave the application looking at an empty directory instead of the existing data.
With Compose, validate the configuration from the directory and project used for this deployment:
docker compose config --quietThis checks that Compose can parse and resolve the configuration. It does not prove that credentials work, a database is reachable or an application accepts every setting. Preserve any original -f, --env-file and -p options when running Compose commands. Printing the full resolved configuration can expose secrets.
Permission denied
docker inspect --format 'configured_user={{json .Config.User}}' "$CONTAINER"Compare the image's documented UID/GID requirements with the specific mounted path and the failed operation. An empty configured user is not proof of which user the application ultimately runs as; entrypoint scripts can change identity. Rootless Docker and user namespaces also affect host ownership mapping.
Repair the narrow permission or ownership problem you can explain. Avoid recursive chmod 777 or changing every data file to your login user. The Docker security checklist covers why running everything as root or adding broad privileges creates other problems.
5. Check dependencies and networking
A web application can exit repeatedly because its database is still starting. Another common mistake is a database address of localhost: inside the application container, that means the application container itself.
Services on the same Compose network normally reach each other by service name and container port. If the database service is named db, the application might need db:5432, not the host's published port. Use the actual service name, database type and port from your deployment.
Basic depends_on ordering does not prove an application is ready to accept requests. A suitable health check with condition: service_healthy can gate initial startup; the application also needs retries for dependency outages after startup. Docker documents the distinction in its Compose startup-order guide.
For a small stack with a proxy and private backend services, the Docker Compose starter guide provides the surrounding deployment structure.
Check the dependency's own logs before changing network rules. An authentication failure, unfinished migration or wrong database name will not be solved by opening more ports.
If the application stays up but the public URL returns an error, follow the request path: proxy, application, database. A proxy error and a container restart are separate observations. The home-server reverse-proxy comparison helps identify which component owns that part of your setup.
6. Apply the fix without losing the deployment
Once the error points to a specific cause, change that one thing. Avoid combining an image upgrade, a volume move and a credentials change during the same repair. It makes the result harder to explain and a rollback harder to plan.
If repeated starts are causing more writes or failed migrations, collect the evidence and arrange a controlled stop using the application's documented shutdown grace period. Docker sends the configured stop signal, then forcibly kills the process if the timeout expires. A short timeout is a poor default for a busy database.
A manual stop suppresses the normal restart-policy cycle. With always, a daemon restart can start it again; unless-stopped keeps a manually stopped container stopped across that restart. An external supervisor may still intervene.
Editing Compose is different from restarting
docker compose restart restarts existing containers. It does not apply changes to their Compose-defined environment or other container configuration.
Before a repair that recreates a container, identify its persistent mounts and verify a usable backup. Files stored only in the container's writable layer are lost when that container is removed. Use the Docker backup guide to inventory what must survive, and the backup restore drill to check recoverability on a separate test system.
For a configuration-only repair, the following pattern applies a change to one existing Compose service. Run it from the original deployment directory, with its usual project, file and environment options. Replace app with the service key from your Compose file, not the generated container name.
SERVICE='app'
docker compose config --quiet && \
docker compose up -d --no-deps --no-build --pull never "$SERVICE"This can recreate the service and cause downtime. Use --no-deps only when its required services already exist and are healthy. The flags avoid a build or an image pull during this repair; they do not pin a mutable tag or guarantee that a locally retagged image is unchanged. Compare the image you recorded earlier. An image update needs its own planned backup and compatibility check.
Do not use docker compose down -v as a restart-loop fix: it can remove the project's data volumes. A fresh container starting successfully with empty data is not a successful recovery.
If a new application version migrated the database, switching back to an older image may be incompatible. Follow the application's rollback procedure and, where required, restore a matching data backup in isolation before changing the live service.
7. Prove the service stays useful
After Compose recreates a service, its container ID changes and its restart count starts again. Find the current container with docker compose ps -a and update CONTAINER before repeating the earlier checks.
Use the service as a user would. Log in, open an existing record and carry out a small normal write if the application allows it. Confirm that the expected data is present. A green container status alone does not establish this.
Repeat the workload that previously triggered the failure. Then compare logs, health status, resource usage and restart count over a suitable period. A container that survived two quiet minutes has not demonstrated that it can finish the nightly import.
Keep a short repair note:
- The first useful error and when it occurred.
- The relevant image, configuration and host conditions.
- The single change made and its rollback route.
- The user action or workload that passed afterward.
Finally, check that an alert reaches you when the service fails. Uptime Kuma monitoring can cover the application endpoint, while disk and memory monitoring cover host resources. A monitor running only on the failing host cannot report every host outage; use a check from another machine where that matters.
The next time the container stops, these notes give you something better than another blind restart: a known working configuration and evidence of what changed.
Comments
Post a Comment