[TOC]

记录一次docker磁盘分区容量不足导致无法启动docker的问题

收到tars管理后台发送的钉钉通知，10.8.0.30的tars应用节点挂掉了，ssh登录对应的服务器看看什么情况

1. 查看tars-node日志

docker logs tars-nodes

发现docker没启动，执行以下指令查看docker状态

systemctl status docker

输出如下：

May 17 09:09:43 yang systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
May 17 09:09:43 yang systemd[1]: docker.service: Failed with result 'exit-code'.
May 17 09:09:43 yang systemd[1]: Failed to start Docker Application Container Engine.

启动失败了，此处没有详细信息，再执行 journalctl -e 查看最新日志

跟docker相关的最新日志如下：

May 17 09:09:43 yang systemd[1]: Starting Docker Application Container Engine...
May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.197644854Z" level=info msg="Starting up"
May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.201422824Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.201459215Z" level=warning msg="failed to rename /var/lib/docker/tmp for background deletion: rename /var/lib/docker/tmp /var/lib/docker/tmp-old: file exists. Del>
May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.212087233Z" level=error msg="mkdir /var/lib/docker/overlay2/check-overlayfs-support2933421636/lower2: no space left on device" storage-driver=overlay2
May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.212110461Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"
May 17 09:09:43 yang dockerd[1261]: failed to start daemon: error initializing graphdriver: driver not supported
May 17 09:09:43 yang systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
May 17 09:09:43 yang systemd[1]: docker.service: Failed with result 'exit-code'.
May 17 09:09:43 yang systemd[1]: Failed to start Docker Application Container Engine.

发现两条错误日志

May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.212087233Z" level=error msg="mkdir /var/lib/docker/overlay2/check-overlayfs-support2933421636/lower2: no space left on device" storage-driver=overlay2
May 17 09:09:43 yang dockerd[1261]: time="2024-05-17T09:09:43.212110461Z" level=error msg="[graphdriver] prior storage driver overlay2 failed: driver not supported"

关键信息 no space left on device

报错可用空间不足

2. 查看磁盘容量情况

执行df -h

发现docker挂载的分区都满了

先cd进去各容器分区目录看看

cd /var/lib/docker/containers/e51d3bc1456bedb430d824a8cff00c76884f4c151329e98d5b4532c29fd50ef0

3. 删除容器日志

先删除1-100行较早期的日志

sudo sed -i '1,100d' e51d3bc1456bedb430d824a8cff00c76884f4c151329e98d5b4532c29fd50ef0-json.log

然后尝试启动docker

systemctl restart docker

成功运行

但是删除这100行日志仅仅是解决了启动的问题，很快空间又会占满了

所以还要再进一步清理docker空间，清楚无用的容器、卷、网络、空镜像

docker system prune -a #该指令不加 -a 参数则只会清除悬空镜像，未被使用的镜像不会被删除。

再查看空间容量

df -h

发现已经腾空了1.7G的空间