Server Fault Asked by Shailesh Sutar on March 4, 2021
My server’s inode usage has gone up to 100% due to which I am not able to uninstall extra linux-kernel-headers and linux-kernel-images
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 523970 318 100% /
none 481856 2 481854 1% /sys/fs/cgroup
udev 480561 396 480165 1% /dev
tmpfs 481856 317 481539 1% /run
none 481856 1 481855 1% /run/lock
none 481856 1 481855 1% /run/shm
none 481856 3 481853 1% /run/user
/dev/xvdb 262144 11 262133 1% /mnt
Also HDD usage are up to only 78%
$ sudo df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 5.7G 1.7G 78% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 1.9G 12K 1.9G 1% /dev
tmpfs 377M 368K 377M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 0 1.9G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/xvdb 3.9G 8.1M 3.7G 1% /mnt
When I try to remove headers and images using sudo apt-get autoremove command, It gives me below error.
$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
linux-headers-3.13.0-83-generic : Depends: linux-headers-3.13.0-83 but it is not installed
E: Unmet dependencies. Try using -f.
When I do sudo apt-get -f install, It gives me below error.
dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb (--unpack):
unable to create `/usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h.dpkg-new' (while processing `./usr/src/linux-headers-3.13.0-83/arch/sh/include/asm/sparsemem.h'): No space left on device
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-headers-3.13.0-83_3.13.0-83.127_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
When I do $ sudo dpkg --configure -a, It gives me below error.
libpostfix-dns.so.1 -> libpostfix-dns.so.1.0.1
/sbin/ldconfig.real: Can't create temporary cache file /etc/ld.so.cache~: No space left on device
dpkg: error processing package libc-bin (--configure):
subprocess installed post-installation script returned error exit status 1
dpkg: error: unable to create new file '/var/lib/dpkg/status-new': No space left on device
Now I am not sure how to tackle this issue. Any experts advice would be a great input. I am not sure from where should I start deleting files. Here is description of filesystem.
/$ sudo du -sh *
9.6M bin
418M boot
12K dev
6.9M etc
216K home
0 initrd.img
0 initrd.img.old
789M lib
4.0K lib64
16K lost+found
4.0K media
28K mnt
684M opt
du: cannot access ‘proc/21817/task/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/task/21817/fdinfo/4’: No such file or directory
du: cannot access ‘proc/21817/fd/4’: No such file or directory
du: cannot access ‘proc/21817/fdinfo/4’: No such file or directory
0 proc
23M root
372K run
9.4M sbin
4.0K srv
0 sys
19M tmp
2.9G usr
669M var
0 vmlinuz
0 vmlinuz.old
Note: I have already deleted log files older than 20 days. It didn’t help to minimise inode usage and I am stuck now.
In my case the / partition was showing as 100% inode usage due to which I was not able to uninstall any extra linux-headers and extra kernel linux-images.
1. First I deleted all the logs which were older than 20 days
2. I moved some files from /var/cache/ folder to /mnt/test/ folder which caused 3% free inodes on my system.
$ sudo apt-get clean
$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
linux-headers-3.13.0-67 linux-headers-3.13.0-67-generic
linux-headers-3.13.0-68 linux-headers-3.13.0-68-generic
linux-headers-3.13.0-70 linux-headers-3.13.0-70-generic
linux-headers-3.13.0-71 linux-headers-3.13.0-71-generic
linux-headers-3.13.0-73 linux-headers-3.13.0-73-generic
linux-headers-3.13.0-74 linux-headers-3.13.0-74-generic
linux-headers-3.13.0-76 linux-headers-3.13.0-76-generic
linux-headers-3.13.0-77 linux-headers-3.13.0-77-generic
linux-image-3.13.0-67-generic linux-image-3.13.0-68-generic
linux-image-3.13.0-70-generic linux-image-3.13.0-71-generic
linux-image-3.13.0-73-generic linux-image-3.13.0-74-generic
linux-image-3.13.0-76-generic linux-image-3.13.0-77-generic
0 upgraded, 0 newly installed, 24 to remove and 144 not upgraded.
After this operation, 955 MB disk space will be freed.
Do you want to continue? [Y/n] Y
After uninstalling those headers and images, inode usages are dropped to 64% now
$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 331671 192617 64% /
none 481856 2 481854 1% /sys/fs/cgroup
udev 480561 396 480165 1% /dev
tmpfs 481856 317 481539 1% /run
none 481856 1 481855 1% /run/lock
none 481856 1 481855 1% /run/shm
none 481856 3 481853 1% /run/user
/dev/xvdb 262144 1933 260211 1% /mnt
of course this is not the only way to deal with 100% inode usage problem. But In my case it worked since my /mnt/ partition had free inodes.
Answered by Shailesh Sutar on March 4, 2021
Given your df -i output and seeing that you are not using any LVM - the only solution you have is to start deleting files.
Every file present on the system / partition will consume one inode.
Directories also consume inode.
So you need to figure out what's driving this number of files up.
Perhaps it's some process creating many temp files.
Perhaps it's log files.
You can use below command to see which directory is the biggest offender
find / -printf "%hn"|grep -v "^/proc"|grep -v "^/sys"|cut -d/ -f1-3|sort|uniq -c|sort -rn
Answered by Dmitry Zayats on March 4, 2021
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP