AnswerBun.com

Can't run fsck or format device - device is in use

Unix & Linux Asked by jkroepke on October 30, 2020

today I crashed my boot partition. While is normally no problem due backups, I can’t write to the partition anymore.

[email protected] ~ # touch /boot/a
touch: cannot touch '/boot/a': Structure needs cleaning

dmesg shows me:

[317873.920810] EXT4-fs error (device md2): ext4_find_dest_de:1932: inode #2: block 518: comm touch: bad entry in directory: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0, size=1024

Even after successfully umount the device, I can’t fsck

[email protected] ~ # umount /boot
[email protected] ~ # e2fsck /dev/md2
e2fsck 1.45.5 (07-Jan-2020)
/dev/md2 is in use.
e2fsck: Cannot continue, aborting.


[email protected] ~ # fsck.ext4 /dev/md2
e2fsck 1.45.5 (07-Jan-2020)
/dev/md2 is in use.
e2fsck: Cannot continue, aborting.

I’m unable to reformat or destroy the partition:

[email protected] ~ # wipefs /dev/md2
DEVICE OFFSET TYPE UUID                                 LABEL
md2    0x438  ext4 2112045a-2e84-4c69-8c67-d76e8ad7a13a boot
[email protected] ~ # wipefs -a /dev/md2
wipefs: error: /dev/md2: probing initialization failed: Device or resource busy
[email protected] ~ # mkfs.ext4 -L boot /dev/md2
mke2fs 1.45.5 (07-Jan-2020)
/dev/md2 contains a ext4 file system labelled 'boot'
    last mounted on /boot on Fri Sep 18 21:33:34 2020
Proceed anyway? (y,N) y
/dev/md2 is apparently in use by the system; will not make a filesystem here!

I also can’t stop the mdam array to set the while raid array to fail:

[email protected] ~ # mdadm --stop /dev/md2 --force
mdadm: Cannot get exclusive access to /dev/md2:Perhaps a running process, mounted filesystem or active volume group?


[email protected] ~ # mdadm /dev/md2 --fail /dev/nvme0n1p2 --remove /dev/nvme0n1p2
mdadm: set /dev/nvme0n1p2 faulty in /dev/md2
mdadm: hot removed /dev/nvme0n1p2 from /dev/md2
[email protected] ~ # mdadm /dev/md2 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
mdadm: set device faulty failed for /dev/nvme1n1p2:  Device or resource busy

mdadm --add /dev/md2 /dev/nvme0n1p2

[email protected] ~ # mdadm /dev/md2 --fail /dev/nvme1n1p2 --remove /dev/nvme1n1p2
mdadm: set /dev/nvme1n1p2 faulty in /dev/md2
mdadm: hot removed /dev/nvme1n1p2 from /dev/md2
[email protected] ~ # mdadm /dev/md2 --fail /dev/nvme0n1p2 --remove /dev/nvme0n1p2
mdadm: set device faulty failed for /dev/nvme0n1p2:  Device or resource busy

fuser or lsof does now shoe any process except kernel threads:

[email protected] ~ # lsof | grep md2
md2_raid1     262                              root  cwd       DIR                9,4     4096          2 /
md2_raid1     262                              root  rtd       DIR                9,4     4096          2 /
md2_raid1     262                              root  txt   unknown                                        /proc/262/exe
jbd2/md2-     599                              root  cwd       DIR                9,4     4096          2 /
jbd2/md2-     599                              root  rtd       DIR                9,4     4096          2 /
jbd2/md2-     599                              root  txt   unknown                                        /proc/599/exe
[email protected] ~ # fuser -v /dev/md2
[email protected] ~ # fuser -v /dev
                     USER        PID ACCESS COMMAND
/dev:                root     kernel mount /dev
                     root         57 .rc.. kdevtmpfs

I’m not using LVM, no userspace is installed:

[email protected] ~ # vgs

Command 'vgs' not found, but can be installed with:

apt install lvm2

The mdadm RAID looks clean:

[email protected] ~ # mdadm -D /dev/md2
/dev/md2:
           Version : 1.2
     Creation Time : Wed Jul 17 22:22:07 2019
        Raid Level : raid1
        Array Size : 523712 (511.44 MiB 536.28 MB)
     Used Dev Size : 523712 (511.44 MiB 536.28 MB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

       Update Time : Tue Sep 22 14:30:52 2020
             State : clean
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : resync

              Name : rescue:boot
              UUID : b8cce71f:11cb0913:e75f8623:0846cd86
            Events : 176

    Number   Major   Minor   RaidDevice State
       3     259        2        0      active sync   /dev/nvme0n1p2
       2     259        9        1      active sync   /dev/nvme1n1p2

I’m running Ubuntu 20.04.

Linux hades 5.4.0-47-generic #51-Ubuntu SMP Fri Sep 4 19:50:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linu

I can’t restart the server, because the system isn’t bootable now. Currently I do not have physical access to the machine.

I already shutdown all services except systemd, ssh, polkit, rsyslog and mdadm monitor.

How I could identify the source which is using the device?

Add your own answers!

Related Questions

Get active sink in PulseAudio

1  Asked on January 4, 2021 by inspectormustache

     

How do I run 32-bit programs on a 64-bit Debian/Ubuntu?

3  Asked on January 3, 2021 by gilles-so-stop-being-evil

         

Grub’s default kernel priority

0  Asked on January 3, 2021 by abhishek-deshpande

         

Kali Linux 2020.2 Stuck on dragon logo

1  Asked on January 2, 2021 by robotix

     

How to fix Ubuntu 18.04 LTS random freezes again

0  Asked on January 2, 2021 by manmax

 

X11 windows created by root

2  Asked on December 31, 2020 by saga

   

Kubuntu 20.04 Display Errror

0  Asked on December 31, 2020 by emil-dworniczak

       

How to exclude folders in sshfs mounting?

1  Asked on December 30, 2020

 

Ubuntu upgrade leads unmet dependencies

1  Asked on December 29, 2020 by jonkobu

     

How to know on which LAN a machine is connected?

0  Asked on December 29, 2020 by jpc

       

Ask a Question

Get help from others!

© 2022 AnswerBun.com. All rights reserved. Sites we Love: PCI Database, MenuIva, UKBizDB, Menu Kuliner, Sharing RPP