device-mapper "delay" devices

Look at the "delay" target for device-mapper devices. This is exactly why it exists.

Example

Here's an example of how to get that going:

Create a place to read/write from

[root@centos6 ~]# dd if=/dev/zero of=/tmp/100M-of-zeroes bs=1024k count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.141834 s, 739 MB/s

Make it into a block device

Dev-mapper only maps from one block device to another, not between files and block devices. That's the job of the loopback device.

[root@centos6 ~]# losetup --show --find /tmp/100M-of-zeroes
/dev/loop0

Get the device size in blocks

Since this is what dev-mapper will need here in a moment...

[root@centos6 ~]# blockdev --getsize /dev/loop0
204800

Set up a "slow" device

# echo "0 204800 delay /dev/loop0 0 200" | dmsetup create dm-slow
(about a 30 second pause here with no output)

The fields in the device mapper setup table in the "echo" command above are:

1. starting sector of this section of the device mapper device (0)
2. number of sectors of this section of the device mapper device (204800)
3. the type of device mapper device for this section (delay)
4. the first argument to "delay", which is the device to use for real reads/writes after the delay (/dev/loop/0)
5. the second argument to "delay" which is the offset in the source device to use (0)
6. the third argument to "delay" which is the ms of time to delay reads (or reads and writes if no further parameters are specified.) (200)

We only have one line since we're treating the entire device mapper device the same, but this lets you have different sectors with different backing devices, only have some of them slow, only having some of them give errors, etc.

See https://linux.die.net/man/8/dmsetup for more info, including the possibly-also-useful "flakey" mapper type. Authoritative documentation on device-mapper's delay feature is at https://www.kernel.org/doc/Documentation/device-mapper/delay.txt

Is it slow?

[root@centos6 ~]# dd if=/dev/mapper/dm-slow of=/dev/null count=25000
25000+0 records in
25000+0 records out
12800000 bytes (13 MB) copied, 10.2028 s, 1.3 MB/s

Yeah, that's pretty slow, especially compared to the original:

[root@centos6 ~]# dd if=/dev/loop0 of=/dev/null count=25000
25000+0 records in
25000+0 records out
12800000 bytes (13 MB) copied, 0.0361308 s, 354 MB/s

So the mapped device is definitely introducing a delay.

Combine the above

I intentionally broke things apart so the process was easy to follow. However, you could easily combine steps above into fewer commands.

You can try a simple command like that:

while true
do
find / -type f -exec cat {} ; >/dev/null 2>/dev/null
done

Note: it will not make any modification, just cat

You can try using dm_delay to simulate delayed IO for a device. Here is some document you might need: http://www.mjmwired.net/kernel/Documentation/device-mapper/delay.txt and here are some useful links for iostats: iostat - How does %util affect %idle and %iowait https://stackoverflow.com/questions/4458183/how-the-util-of-iostat-is-computed

With fio and blktrace, you can replay an I/O trace. What type of disruption are you trying to simulate?

If the delays you wish to induce are from an existing production system, you can use that as the basis for your trace.