Making bit identical ext2 filesystems

The answer to your original question is a tool called genext2fs. If you supply the -f switch, then it will create bit-by-bit identical output, given the same input. This is either proven by its own test-suite which compares the created images with a pre-computed md5sum of the correct output or by this test (executed inside the source dir):

$ ./genext2fs -f -B 1024 -b 40 -d m4 rootfs.img
$ md5sum rootfs.img
322053a8962acc599eaabb2dfde28783  rootfs.img
$ rm rootfs.img
$ ./genext2fs -f -B 1024 -b 40 -d m4 rootfs.img
$ md5sum rootfs.img
322053a8962acc599eaabb2dfde28783  rootfs.img

You can mount the resulting image and check that its content is indeed the same as the packed directory:

$ sudo mount rootfs.img /mnt
$ ls -lha /mnt
total 27K
drwxr-xr-x  3 root  root  1.0K Jan  1  1970 .
drwxr-xr-x 25 root  root  4.0K Mar 22 14:56 ..
-rw-r--r--  1 josch josch 1.5K Mar 22 23:05 ac_func_scanf_can_malloc.m4
-rw-r--r--  1 josch josch 2.4K Mar 22 14:24 ac_func_snprintf.m4
drwx------  2 root  root   16K Jan  1  1970 lost+found
$ rmdir /mnt/lost+found
$ diff -rq m4 /mnt
$ echo $?
0

Have fun!

You could also try e2image:

The e2image program will save critical ext2, ext3, or ext4 filesystem metadata located on device to a file.

Despite that by default e2image saves metadata only, it's able to save data too, by the use of -a option. (See section "Including data" by the link)

The -a option can be specified to include all data. This will give an image that is suitable to use to clone the entire FS or for backup purposes. Note that this option only works with the raw or QCOW2 formats.

QCOW2 format is more preferable for backups than raw format. QCOW2 image is a normal file with size that is close to used space of partition to back up. Raw image is a sparse file with all the implications - tools that don't handle sparse files specifically will process not only used space, but free space too.

So, usage examples are:

• Backup sda1 partition, including files & metadata, to the file boot-part.qcow2:

sudo mount -o remount,ro /dev/sda1
sudo e2image -a -Q /dev/sda1 boot-part.qcow2

Note that remounting as read-only is required to guarantee that no one will write to the partition, while backup is in progress. After all, you are able to remount as read-write by sudo mount -o remount,rw /dev/sda1 if you need to.

• Restore partition /dev/sda1 from QCOW2 image file boot-part.qcow2:

sudo umount /dev/sda1
sudo e2image -r boot-part.qcow2 /dev/sda1

Note that umount is a must since you can't rewrite superblocks of a mounted partition. If /dev/sda1 is not mounted that time, you can skip this step. :)

It's also worth noting that in restoring scenario size of /dev/sda1 should be equal or greater than partition inside image file, otherwise you will get error: e2image: Invalid argument while trying to convert qcow2 image (boot-part.qcow2) into raw image (/dev/sda1).

Note: This will not be a full answer; just a partial one, or, at least, a hint

---

I need to be able to run my script that creates the image and have the output be bit-for-bit identical each time.

The first problem you have to achieve that, are the disk signatures in your msdos partition tables (offset of 440 in the MBR, 4 bytes long). If your MBRs are different, your are failing in your goal just at the first sector. Each time you execute mklabel inside parted, you are generating a new disk signature. You can overcome that, overwriting those four bytes with the same random signature, like this:

printf RAMDOM_SIGNATURE | xxd -p -r | dd bs=1 count=4 seek=440 of=YOUR_DOT_BIN conv=notrunc 2> /dev/null

RANDOM_SIGNATURE could be something like '73396992'

I've made a little mod to your script, with this fix:

dd if=/dev/zero bs=1024 count=46112 of="$1"
parted "$1" <<EOF
unit
s
mklabel
msdos
mkpart
p
ext2
63s
45119s
set
1
boot
on
q
EOF

printf "$2" | xxd -p -r | dd bs=1 count=4 seek=440 of="$1" conv=notrunc 2> /dev/null

losetup -o $(expr 63 * 512) /dev/loop0 "$1"

mke2fs -b 1024 -t ext2 /dev/loop0 22528

#clear some parameters
tune2fs -i 0 /dev/loop0 # interval between check
tune2fs -L LABEL /dev/loop0
tune2fs -U 00000000-0000-0000-0000-000000000000 /dev/loop0 #uuid
tune2fs -c 0 /dev/loop0 #mount count

#mount /dev/loop0 mnt
## make a dummy file
#echo HELLO > mnt/a.txt
#umount mnt

losetup -d /dev/loop0

Now, you can call the script like this

./script_name BIN_FILE_NAME RANDOM_SIGNATURE

Now, if you do this:

./test.sh cf00.bin '73396992'
./test.sh cf01.bin '73396992'
./test.sh cf02.bin '73396992'
./test.sh cf03.bin '73396992'

and then this:

dd if=cf00.bin count=63 2>/dev/null | sha1sum
dd if=cf01.bin count=63 2>/dev/null | sha1sum
dd if=cf02.bin count=63 2>/dev/null | sha1sum
dd if=cf03.bin count=63 2>/dev/null | sha1sum

You'll see that all of those files are identical till just before the filesystem in the first partition (try the same with your original script, and the sums will differ from one another).

You probably notice that in my version of the script, I commented out the lines that wrote the a.txt file. I did this, cause there is no point in trying to fix that, when you can't make the filesystems identical, even with no files on them. And this is the case: the filesystems differ, even with no files, so, first, we need to fix that.

If you run dumpe2fs against the filesystem partition on each image, dump that to a file, and then use diff against any pair of dumps, you'll see something like this:

25c25
< Filesystem created:       Sat Jun 15 07:37:32 2019
---
> Filesystem created:       Sat Jun 15 07:37:40 2019
27c27
< Last write time:          Sat Jun 15 07:37:33 2019
---
> Last write time:          Sat Jun 15 07:37:40 2019
30c30
< Last checked:             Sat Jun 15 07:37:32 2019
---
> Last checked:             Sat Jun 15 07:37:40 2019
37c37
< Directory Hash Seed:      603130ae-82de-4530-9772-f68ae3d6df5f
---
> Directory Hash Seed:      1d9c5af8-a48e-4221-9e70-8fa2ccc6936f

So, at very least, at a very high level (after this, you need to go deeper, at the lowest level, i.e.: the actual byte by byte comparison ) the filesystems differ in the details showed just above. Get around that first.

Even if you change the date in the machine, you'll no be successful in tampering the timestamps and making them equal, cause there are gaps of time you can control in the program execution. In that case, you'll need to freeze your clock, at least, from the program that creates the filesystem perspective. You can dig on that, but I think that this is not the way to go, cause you said that they need to execute your script in their machines: you don't want to mess with their clock. So, IMHO, the way to go, is probably tampering the correct bytes on the filesystem, like I did with the disk signature. Search around that.

Also, don't forget the Superblock backups... track them down. If they contain different data on each filesystem, they will propagate differences in the byte range they reside.

Lastly, bare in mind that when you copy a file, you don't have direct control over the 'distribution' of the file bytes inside the filesystem... If you can't clone, you need to find a way to control that too.

IMHO this all seems to be made overly complicated. When tar alone seems like the obvious solution. tar can create just about any file system, including cdfs (--options cd9660:*). It will also allow you to time stamp the output file to any of that of the most recent -m || --modification-time, --gid id || --gname name, --acls || --no-acls, --same-owner || --no-same-owner, ...

Or you could create your filesystem. Perform a chown -Rh someone:somegroup . within your file tree, and chmod it to your liking and use either tar, or rsync to place the file tree into your prepared filesystem. Then everything would be consistent -- same date, same owner/group && perms.

Well that's the way I'd approach something like this. :)

HTH