30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890671: Fri Jan 7 13:33:20 2022 read: IOPS=2379, BW=9517KiB/s (9745kB/s)(279MiB/30009msec) clat (usec): min=25, max=25069, avg=58.77, stdev=399.63 lat (usec): min=25, max=25069, avg=58.91, stdev=399.63 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 95], 99.90th=[ 375], 99.95th=[12256], | 99.99th=[16909] bw ( KiB/s): min= 7640, max=13176, per=100.00%, avg=9519.47, stdev=958.15, samples=60 iops : min= 1910, max= 3294, avg=2379.87, stdev=239.54, samples=60 write: IOPS=595, BW=2381KiB/s (2438kB/s)(69.8MiB/30009msec) clat (usec): min=191, max=43060, avg=1436.02, stdev=1685.91 lat (usec): min=191, max=43061, avg=1436.30, stdev=1685.91 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 586], 10.00th=[ 807], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 2032, max= 3392, per=100.00%, avg=2381.20, stdev=207.83, samples=60 iops : min= 508, max= 848, avg=595.30, stdev=51.96, samples=60 lat (usec) : 50=54.22%, 100=25.44%, 250=0.35%, 500=0.71%, 750=0.87% lat (usec) : 1000=2.11% lat (msec) : 2=15.15%, 4=0.72%, 10=0.31%, 20=0.09%, 50=0.03% cpu : usr=1.40%, sys=3.80%, ctx=89263, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71396,17860,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9517KiB/s (9745kB/s), 9517KiB/s-9517KiB/s (9745kB/s-9745kB/s), io=279MiB (292MB), run=30009-30009msec WRITE: bw=2381KiB/s (2438kB/s), 2381KiB/s-2381KiB/s (2438kB/s-2438kB/s), io=69.8MiB (73.2MB), run=30009-30009msec Disk stats (read/write): vdk: ios=71359/17849, merge=0/0, ticks=770/18001, in_queue=28352, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891551: Fri Jan 7 13:33:57 2022 read: IOPS=2373, BW=9496KiB/s (9724kB/s)(278MiB/30005msec) slat (usec): min=4, max=434, avg=10.03, stdev= 7.25 clat (nsec): min=656, max=22787k, avg=46050.11, stdev=405823.30 lat (usec): min=21, max=22794, avg=56.24, stdev=405.86 clat percentiles (usec): | 1.00th=[ 11], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 36], | 70.00th=[ 37], 80.00th=[ 40], 90.00th=[ 44], 95.00th=[ 48], | 99.00th=[ 62], 99.50th=[ 73], 99.90th=[ 273], 99.95th=[13566], | 99.99th=[16909] bw ( KiB/s): min= 7728, max=14536, per=100.00%, avg=9497.50, stdev=1048.75, samples=60 iops : min= 1932, max= 3634, avg=2374.37, stdev=262.19, samples=60 write: IOPS=594, BW=2377KiB/s (2434kB/s)(69.6MiB/30005msec) slat (usec): min=6, max=257, avg=11.09, stdev= 6.52 clat (nsec): min=1662, max=43555k, avg=1438270.07, stdev=1680494.61 lat (usec): min=193, max=43563, avg=1449.54, stdev=1680.46 clat percentiles (usec): | 1.00th=[ 262], 5.00th=[ 603], 10.00th=[ 832], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 1904, max= 3760, per=100.00%, avg=2376.88, stdev=263.02, samples=60 iops : min= 476, max= 940, avg=594.22, stdev=65.75, samples=60 lat (nsec) : 750=0.06%, 1000=0.24% lat (usec) : 2=0.38%, 4=0.03%, 10=0.10%, 20=0.57%, 50=75.89% lat (usec) : 100=2.58%, 250=0.21%, 500=0.65%, 750=0.77%, 1000=2.08% lat (msec) : 2=15.32%, 4=0.70%, 10=0.31%, 20=0.10%, 50=0.02% cpu : usr=1.34%, sys=6.01%, ctx=88234, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71229,17827,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9496KiB/s (9724kB/s), 9496KiB/s-9496KiB/s (9724kB/s-9724kB/s), io=278MiB (292MB), run=30005-30005msec WRITE: bw=2377KiB/s (2434kB/s), 2377KiB/s-2377KiB/s (2434kB/s-2434kB/s), io=69.6MiB (73.0MB), run=30005-30005msec Disk stats (read/write): vdk: ios=70885/17724, merge=0/0, ticks=768/18198, in_queue=28137, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892541: Fri Jan 7 13:34:34 2022 read: IOPS=2388, BW=9553KiB/s (9782kB/s)(280MiB/30001msec) clat (usec): min=24, max=23089, avg=57.94, stdev=383.87 lat (usec): min=24, max=23089, avg=58.08, stdev=383.87 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 64], | 99.00th=[ 80], 99.50th=[ 94], 99.90th=[ 412], 99.95th=[12125], | 99.99th=[16909] bw ( KiB/s): min= 8016, max=14853, per=100.00%, avg=9565.93, stdev=1066.84, samples=59 iops : min= 2004, max= 3713, avg=2391.47, stdev=266.69, samples=59 write: IOPS=597, BW=2392KiB/s (2449kB/s)(70.1MiB/30001msec) clat (usec): min=201, max=43097, avg=1431.62, stdev=1651.65 lat (usec): min=201, max=43097, avg=1431.90, stdev=1651.65 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 594], 10.00th=[ 807], 20.00th=[ 1012], | 30.00th=[ 1123], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2024], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[38536], | 99.99th=[42206] bw ( KiB/s): min= 2016, max= 3831, per=100.00%, avg=2393.47, stdev=254.66, samples=59 iops : min= 504, max= 957, avg=598.36, stdev=63.59, samples=59 lat (usec) : 50=55.55%, 100=24.13%, 250=0.30%, 500=0.69%, 750=0.92% lat (usec) : 1000=2.15% lat (msec) : 2=15.16%, 4=0.67%, 10=0.31%, 20=0.09%, 50=0.02% cpu : usr=1.43%, sys=3.75%, ctx=89592, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71648,17939,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9553KiB/s (9782kB/s), 9553KiB/s-9553KiB/s (9782kB/s-9782kB/s), io=280MiB (293MB), run=30001-30001msec WRITE: bw=2392KiB/s (2449kB/s), 2392KiB/s-2392KiB/s (2449kB/s-2449kB/s), io=70.1MiB (73.5MB), run=30001-30001msec Disk stats (read/write): vdk: ios=71110/17793, merge=0/0, ticks=736/17810, in_queue=28166, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892992: Fri Jan 7 13:35:11 2022 read: IOPS=2413, BW=9653KiB/s (9885kB/s)(283MiB/30026msec) slat (usec): min=2, max=386, avg=10.19, stdev= 6.65 clat (nsec): min=637, max=2495.4M, avg=834355.20, stdev=35637942.41 lat (usec): min=17, max=2495.5k, avg=844.72, stdev=35637.95 clat percentiles (usec): | 1.00th=[ 20], 5.00th=[ 27], 10.00th=[ 29], | 20.00th=[ 31], 30.00th=[ 33], 40.00th=[ 34], | 50.00th=[ 36], 60.00th=[ 38], 70.00th=[ 40], | 80.00th=[ 43], 90.00th=[ 49], 95.00th=[ 58], | 99.00th=[ 95], 99.50th=[ 118], 99.90th=[ 510], | 99.95th=[ 591397], 99.99th=[2365588] bw ( KiB/s): min= 7384, max=15360, per=100.00%, avg=9661.67, stdev=1240.84, samples=60 iops : min= 1846, max= 3840, avg=2415.42, stdev=310.21, samples=60 write: IOPS=603, BW=2416KiB/s (2474kB/s)(70.8MiB/30026msec) slat (usec): min=2, max=240, avg=11.47, stdev= 6.90 clat (usec): min=341, max=77359, avg=13153.72, stdev=6420.71 lat (usec): min=345, max=77374, avg=13165.37, stdev=6420.91 clat percentiles (usec): | 1.00th=[ 1287], 5.00th=[ 5276], 10.00th=[ 8094], 20.00th=[10028], | 30.00th=[11076], 40.00th=[11863], 50.00th=[12649], 60.00th=[13173], | 70.00th=[13829], 80.00th=[14877], 90.00th=[19530], 95.00th=[21890], | 99.00th=[31851], 99.50th=[61604], 99.90th=[69731], 99.95th=[71828], | 99.99th=[77071] bw ( KiB/s): min= 2032, max= 3880, per=100.00%, avg=2416.85, stdev=254.39, samples=60 iops : min= 508, max= 970, avg=604.20, stdev=63.59, samples=60 lat (nsec) : 750=0.02%, 1000=0.14% lat (usec) : 2=0.19%, 4=0.02%, 10=0.10%, 20=0.39%, 50=72.05% lat (usec) : 100=6.37%, 250=0.58%, 500=0.03%, 750=0.03%, 1000=0.05% lat (msec) : 2=0.40%, 4=0.27%, 10=3.23%, 20=14.19%, 50=1.72% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.56%, sys=6.01%, ctx=88252, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72461,18135,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9653KiB/s (9885kB/s), 9653KiB/s-9653KiB/s (9885kB/s-9885kB/s), io=283MiB (297MB), run=30026-30026msec WRITE: bw=2416KiB/s (2474kB/s), 2416KiB/s-2416KiB/s (2474kB/s-2474kB/s), io=70.8MiB (74.3MB), run=30026-30026msec Disk stats (read/write): vdk: ios=72460/18126, merge=0/0, ticks=57554/229072, in_queue=298050, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894403: Fri Jan 7 13:35:48 2022 read: IOPS=2384, BW=9536KiB/s (9765kB/s)(279MiB/30001msec) clat (usec): min=24, max=19602, avg=56.93, stdev=375.86 lat (usec): min=25, max=19602, avg=57.06, stdev=375.86 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 36], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 77], 99.50th=[ 88], 99.90th=[ 379], 99.95th=[11600], | 99.99th=[16319] bw ( KiB/s): min= 6512, max=13280, per=100.00%, avg=9537.03, stdev=1135.88, samples=60 iops : min= 1628, max= 3320, avg=2384.25, stdev=283.97, samples=60 write: IOPS=596, BW=2388KiB/s (2445kB/s)(69.9MiB/30001msec) clat (usec): min=193, max=49365, avg=1438.72, stdev=1699.20 lat (usec): min=194, max=49365, avg=1438.99, stdev=1699.19 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 603], 10.00th=[ 816], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[47973] bw ( KiB/s): min= 1792, max= 3456, per=100.00%, avg=2387.62, stdev=240.13, samples=60 iops : min= 448, max= 864, avg=596.90, stdev=60.03, samples=60 lat (usec) : 50=57.93%, 100=21.82%, 250=0.25%, 500=0.68%, 750=0.84% lat (usec) : 1000=2.08% lat (msec) : 2=15.26%, 4=0.71%, 10=0.31%, 20=0.09%, 50=0.02% cpu : usr=1.46%, sys=3.60%, ctx=89435, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71523,17907,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9536KiB/s (9765kB/s), 9536KiB/s-9536KiB/s (9765kB/s-9765kB/s), io=279MiB (293MB), run=30001-30001msec WRITE: bw=2388KiB/s (2445kB/s), 2388KiB/s-2388KiB/s (2445kB/s-2445kB/s), io=69.9MiB (73.3MB), run=30001-30001msec Disk stats (read/write): vdk: ios=71072/17779, merge=0/0, ticks=721/18023, in_queue=28182, util=94.14% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894977: Fri Jan 7 13:36:25 2022 read: IOPS=2428, BW=9715KiB/s (9948kB/s)(285MiB/30092msec) slat (usec): min=2, max=216, avg= 9.68, stdev= 4.81 clat (nsec): min=645, max=2516.7M, avg=927068.42, stdev=39487511.04 lat (usec): min=18, max=2516.7k, avg=936.90, stdev=39487.51 clat percentiles (usec): | 1.00th=[ 23], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 33], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 41], | 80.00th=[ 45], 90.00th=[ 55], 95.00th=[ 71], | 99.00th=[ 174], 99.50th=[ 338], 99.90th=[ 1123], | 99.95th=[ 505414], 99.99th=[2499806] bw ( KiB/s): min= 6848, max=17872, per=100.00%, avg=9743.05, stdev=1392.17, samples=60 iops : min= 1712, max= 4468, avg=2435.73, stdev=348.05, samples=60 write: IOPS=607, BW=2428KiB/s (2487kB/s)(71.4MiB/30092msec) slat (usec): min=2, max=197, avg=11.02, stdev= 5.51 clat (msec): min=4, max=2489, avg=78.55, stdev=65.69 lat (msec): min=4, max=2489, avg=78.56, stdev=65.69 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 55], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 82], 90.00th=[ 93], 95.00th=[ 120], | 99.00th=[ 245], 99.50th=[ 472], 99.90th=[ 986], 99.95th=[ 1167], | 99.99th=[ 2500] bw ( KiB/s): min= 1896, max= 4216, per=100.00%, avg=2429.47, stdev=299.36, samples=60 iops : min= 474, max= 1054, avg=607.33, stdev=74.86, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.12%, 4=0.02%, 10=0.06%, 20=0.31%, 50=69.13% lat (usec) : 100=8.63%, 250=1.11%, 500=0.28%, 750=0.10%, 1000=0.07% lat (msec) : 2=0.03%, 10=0.16%, 20=0.14%, 50=0.23%, 100=17.94% lat (msec) : 250=1.35%, 500=0.12%, 750=0.06%, 1000=0.03%, 2000=0.03% lat (msec) : >=2000=0.02% cpu : usr=1.31%, sys=5.93%, ctx=88281, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=73084,18269,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9715KiB/s (9948kB/s), 9715KiB/s-9715KiB/s (9948kB/s-9948kB/s), io=285MiB (299MB), run=30092-30092msec WRITE: bw=2428KiB/s (2487kB/s), 2428KiB/s-2428KiB/s (2487kB/s-2487kB/s), io=71.4MiB (74.8MB), run=30092-30092msec Disk stats (read/write): vdk: ios=73080/18223, merge=0/0, ticks=62816/1420001, in_queue=1497796, util=99.47% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895819: Fri Jan 7 13:37:02 2022 read: IOPS=2381, BW=9527KiB/s (9756kB/s)(279MiB/30001msec) clat (usec): min=25, max=24142, avg=57.27, stdev=382.06 lat (usec): min=25, max=24142, avg=57.41, stdev=382.06 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 80], 99.50th=[ 90], 99.90th=[ 441], 99.95th=[11469], | 99.99th=[16057] bw ( KiB/s): min= 7808, max=14360, per=100.00%, avg=9527.17, stdev=1042.54, samples=60 iops : min= 1952, max= 3590, avg=2381.78, stdev=260.68, samples=60 write: IOPS=596, BW=2385KiB/s (2442kB/s)(69.9MiB/30001msec) clat (usec): min=190, max=43847, avg=1438.93, stdev=1679.00 lat (usec): min=190, max=43847, avg=1439.21, stdev=1679.00 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 603], 10.00th=[ 824], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2008], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min= 1928, max= 3704, per=100.00%, avg=2384.72, stdev=252.07, samples=60 iops : min= 482, max= 926, avg=596.17, stdev=63.02, samples=60 lat (usec) : 50=57.43%, 100=22.27%, 250=0.28%, 500=0.69%, 750=0.85% lat (usec) : 1000=2.04% lat (msec) : 2=15.33%, 4=0.68%, 10=0.30%, 20=0.09%, 50=0.03% cpu : usr=1.26%, sys=3.83%, ctx=89347, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71456,17887,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9527KiB/s (9756kB/s), 9527KiB/s-9527KiB/s (9756kB/s-9756kB/s), io=279MiB (293MB), run=30001-30001msec WRITE: bw=2385KiB/s (2442kB/s), 2385KiB/s-2385KiB/s (2442kB/s-2442kB/s), io=69.9MiB (73.3MB), run=30001-30001msec Disk stats (read/write): vdk: ios=71445/17878, merge=0/0, ticks=727/18101, in_queue=28355, util=94.08% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896944: Fri Jan 7 13:37:39 2022 read: IOPS=2447, BW=9789KiB/s (10.0MB/s)(288MiB/30173msec) slat (usec): min=2, max=381, avg= 9.39, stdev= 5.14 clat (nsec): min=776, max=2558.0M, avg=999600.67, stdev=38958969.80 lat (usec): min=22, max=2558.0k, avg=1009.14, stdev=38958.97 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 42], | 80.00th=[ 47], 90.00th=[ 61], 95.00th=[ 113], | 99.00th=[ 693], 99.50th=[ 1106], 99.90th=[ 2343], | 99.95th=[ 801113], 99.99th=[2197816] bw ( KiB/s): min= 7168, max=20376, per=100.00%, avg=9844.90, stdev=1682.82, samples=60 iops : min= 1792, max= 5094, avg=2461.22, stdev=420.71, samples=60 write: IOPS=611, BW=2445KiB/s (2504kB/s)(72.1MiB/30173msec) slat (usec): min=2, max=364, avg=10.63, stdev= 5.45 clat (msec): min=5, max=3422, avg=159.44, stdev=151.04 lat (msec): min=5, max=3422, avg=159.45, stdev=151.04 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 105], 10.00th=[ 113], 20.00th=[ 123], | 30.00th=[ 133], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 209], | 99.00th=[ 468], 99.50th=[ 969], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 3104] bw ( KiB/s): min= 2064, max= 4400, per=100.00%, avg=2447.18, stdev=308.64, samples=60 iops : min= 516, max= 1100, avg=611.77, stdev=77.17, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.12%, 50=67.43% lat (usec) : 100=7.79%, 250=2.76%, 500=0.70%, 750=0.35%, 1000=0.26% lat (msec) : 2=0.36%, 4=0.05%, 10=0.01%, 20=0.25%, 50=0.11% lat (msec) : 100=0.33%, 250=18.70%, 500=0.41%, 750=0.07%, 1000=0.05% lat (msec) : 2000=0.06%, >=2000=0.06% cpu : usr=1.36%, sys=5.70%, ctx=86380, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73844,18446,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9789KiB/s (10.0MB/s), 9789KiB/s-9789KiB/s (10.0MB/s-10.0MB/s), io=288MiB (302MB), run=30173-30173msec WRITE: bw=2445KiB/s (2504kB/s), 2445KiB/s-2445KiB/s (2504kB/s-2504kB/s), io=72.1MiB (75.6MB), run=30173-30173msec Disk stats (read/write): vdk: ios=73836/18354, merge=0/0, ticks=65333/2908788, in_queue=2997191, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897505: Fri Jan 7 13:38:16 2022 read: IOPS=2347, BW=9392KiB/s (9617kB/s)(275MiB/30005msec) clat (usec): min=25, max=22592, avg=60.42, stdev=404.74 lat (usec): min=25, max=22592, avg=60.56, stdev=404.74 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 86], 99.50th=[ 101], 99.90th=[ 420], 99.95th=[12911], | 99.99th=[16581] bw ( KiB/s): min= 7336, max=13931, per=100.00%, avg=9394.22, stdev=1025.81, samples=60 iops : min= 1834, max= 3482, avg=2348.53, stdev=256.40, samples=60 write: IOPS=587, BW=2350KiB/s (2407kB/s)(68.9MiB/30005msec) clat (usec): min=211, max=48998, avg=1451.05, stdev=1728.25 lat (usec): min=211, max=48998, avg=1451.35, stdev=1728.26 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 578], 10.00th=[ 799], 20.00th=[ 1012], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2024], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[39060], | 99.99th=[44827] bw ( KiB/s): min= 1912, max= 3567, per=100.00%, avg=2350.72, stdev=253.34, samples=60 iops : min= 478, max= 891, avg=587.67, stdev=63.27, samples=60 lat (usec) : 50=49.66%, 100=29.90%, 250=0.39%, 500=0.77%, 750=0.94% lat (usec) : 1000=2.08% lat (msec) : 2=15.14%, 4=0.65%, 10=0.34%, 20=0.11%, 50=0.03% cpu : usr=1.33%, sys=3.97%, ctx=88083, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70451,17630,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9392KiB/s (9617kB/s), 9392KiB/s-9392KiB/s (9617kB/s-9617kB/s), io=275MiB (289MB), run=30005-30005msec WRITE: bw=2350KiB/s (2407kB/s), 2350KiB/s-2350KiB/s (2407kB/s-2407kB/s), io=68.9MiB (72.2MB), run=30005-30005msec Disk stats (read/write): vdk: ios=70451/17629, merge=0/0, ticks=766/18027, in_queue=28241, util=93.64% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898367: Fri Jan 7 13:38:53 2022 read: IOPS=2448, BW=9795KiB/s (10.0MB/s)(289MiB/30212msec) slat (usec): min=2, max=276048, avg=327.66, stdev=2229.13 clat (usec): min=525, max=2681.1k, avg=26270.47, stdev=41316.96 lat (usec): min=528, max=2681.1k, avg=26598.26, stdev=41384.64 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 8455], 10.00th=[ 13960], | 20.00th=[ 17695], 30.00th=[ 19792], 40.00th=[ 21627], | 50.00th=[ 23462], 60.00th=[ 25560], 70.00th=[ 27919], | 80.00th=[ 30802], 90.00th=[ 35914], 95.00th=[ 41681], | 99.00th=[ 80217], 99.50th=[ 181404], 99.90th=[ 299893], | 99.95th=[ 534774], 99.99th=[2432697] bw ( KiB/s): min= 6976, max=18256, per=100.00%, avg=9854.50, stdev=1526.32, samples=60 iops : min= 1744, max= 4564, avg=2463.62, stdev=381.58, samples=60 write: IOPS=611, BW=2446KiB/s (2504kB/s)(72.2MiB/30212msec) slat (usec): min=2, max=55906, avg=301.40, stdev=1448.35 clat (msec): min=10, max=2893, avg=220.06, stdev=166.38 lat (msec): min=10, max=2893, avg=220.37, stdev=166.37 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 136], 10.00th=[ 155], 20.00th=[ 178], | 30.00th=[ 188], 40.00th=[ 197], 50.00th=[ 203], 60.00th=[ 211], | 70.00th=[ 220], 80.00th=[ 232], 90.00th=[ 257], 95.00th=[ 292], | 99.00th=[ 802], 99.50th=[ 1519], 99.90th=[ 2735], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 1920, max= 3752, per=100.00%, avg=2446.40, stdev=267.14, samples=60 iops : min= 480, max= 938, avg=611.58, stdev=66.79, samples=60 lat (usec) : 750=0.22%, 1000=0.37% lat (msec) : 2=1.40%, 4=1.34%, 10=1.13%, 20=20.16%, 50=53.34% lat (msec) : 100=2.01%, 250=17.48%, 500=2.20%, 750=0.11%, 1000=0.06% lat (msec) : 2000=0.12%, >=2000=0.06% cpu : usr=0.69%, sys=1.95%, ctx=10099, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73985,18472,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9795KiB/s (10.0MB/s), 9795KiB/s-9795KiB/s (10.0MB/s-10.0MB/s), io=289MiB (303MB), run=30212-30212msec WRITE: bw=2446KiB/s (2504kB/s), 2446KiB/s-2446KiB/s (2504kB/s-2504kB/s), io=72.2MiB (75.7MB), run=30212-30212msec Disk stats (read/write): vdk: ios=73983/18472, merge=0/0, ticks=67520/3588215, in_queue=3672672, util=99.28% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898986: Fri Jan 7 13:39:30 2022 read: IOPS=2327, BW=36.4MiB/s (38.1MB/s)(1091MiB/30001msec) clat (usec): min=28, max=84814, avg=64.46, stdev=582.01 lat (usec): min=28, max=84814, avg=64.61, stdev=582.01 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 89], 99.50th=[ 102], 99.90th=[ 441], 99.95th=[12780], | 99.99th=[22152] bw ( KiB/s): min=29536, max=57440, per=100.00%, avg=37242.63, stdev=4694.10, samples=60 iops : min= 1846, max= 3590, avg=2327.65, stdev=293.38, samples=60 write: IOPS=582, BW=9315KiB/s (9539kB/s)(273MiB/30001msec) clat (usec): min=232, max=45573, avg=1449.75, stdev=1606.69 lat (usec): min=234, max=45575, avg=1451.19, stdev=1606.66 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 594], 10.00th=[ 832], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2057], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[19006], 99.95th=[38536], | 99.99th=[44303] bw ( KiB/s): min= 7168, max=14784, per=100.00%, avg=9315.82, stdev=1037.92, samples=60 iops : min= 448, max= 924, avg=582.23, stdev=64.87, samples=60 lat (usec) : 50=43.18%, 100=36.38%, 250=0.34%, 500=0.74%, 750=0.90% lat (usec) : 1000=1.84% lat (msec) : 2=15.41%, 4=0.86%, 10=0.24%, 20=0.10%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.31%, sys=3.92%, ctx=87300, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69825,17467,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.4MiB/s (38.1MB/s), 36.4MiB/s-36.4MiB/s (38.1MB/s-38.1MB/s), io=1091MiB (1144MB), run=30001-30001msec WRITE: bw=9315KiB/s (9539kB/s), 9315KiB/s-9315KiB/s (9539kB/s-9539kB/s), io=273MiB (286MB), run=30001-30001msec Disk stats (read/write): vdk: ios=69408/17350, merge=0/0, ticks=890/17538, in_queue=28138, util=94.12% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899803: Fri Jan 7 13:40:07 2022 read: IOPS=2313, BW=36.2MiB/s (37.9MB/s)(1085MiB/30002msec) slat (usec): min=5, max=217, avg=10.48, stdev= 4.69 clat (nsec): min=720, max=24076k, avg=49581.92, stdev=412675.27 lat (usec): min=24, max=24088, avg=60.21, stdev=412.73 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 34], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 39], | 70.00th=[ 41], 80.00th=[ 43], 90.00th=[ 48], 95.00th=[ 52], | 99.00th=[ 67], 99.50th=[ 78], 99.90th=[ 445], 99.95th=[12911], | 99.99th=[17171] bw ( KiB/s): min=29760, max=63520, per=100.00%, avg=37024.97, stdev=4923.91, samples=60 iops : min= 1860, max= 3970, avg=2314.03, stdev=307.73, samples=60 write: IOPS=578, BW=9255KiB/s (9477kB/s)(271MiB/30002msec) slat (usec): min=8, max=104, avg=12.70, stdev= 4.71 clat (usec): min=220, max=42559, avg=1467.31, stdev=1667.84 lat (usec): min=232, max=42569, avg=1480.17, stdev=1667.88 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 603], 10.00th=[ 840], 20.00th=[ 1057], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min= 7488, max=16320, per=100.00%, avg=9255.57, stdev=1191.18, samples=60 iops : min= 468, max= 1020, avg=578.47, stdev=74.45, samples=60 lat (nsec) : 750=0.01%, 1000=0.09% lat (usec) : 2=0.12%, 4=0.01%, 10=0.03%, 20=0.12%, 50=74.40% lat (usec) : 100=5.04%, 250=0.12%, 500=0.72%, 750=0.81%, 1000=1.82% lat (msec) : 2=15.43%, 4=0.85%, 10=0.30%, 20=0.10%, 50=0.03% cpu : usr=1.31%, sys=5.96%, ctx=86522, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69417,17354,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.2MiB/s (37.9MB/s), 36.2MiB/s-36.2MiB/s (37.9MB/s-37.9MB/s), io=1085MiB (1137MB), run=30002-30002msec WRITE: bw=9255KiB/s (9477kB/s), 9255KiB/s-9255KiB/s (9477kB/s-9477kB/s), io=271MiB (284MB), run=30002-30002msec Disk stats (read/write): vdk: ios=69030/17249, merge=0/0, ticks=775/18022, in_queue=28150, util=94.09% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901191: Fri Jan 7 13:40:44 2022 read: IOPS=2323, BW=36.3MiB/s (38.1MB/s)(1089MiB/30001msec) clat (usec): min=27, max=20253, avg=60.80, stdev=384.35 lat (usec): min=28, max=20253, avg=60.93, stdev=384.35 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 441], 99.95th=[11469], | 99.99th=[17433] bw ( KiB/s): min=28096, max=51456, per=100.00%, avg=37184.62, stdev=4039.28, samples=60 iops : min= 1756, max= 3216, avg=2324.03, stdev=252.46, samples=60 write: IOPS=581, BW=9297KiB/s (9520kB/s)(272MiB/30001msec) clat (usec): min=235, max=45089, avg=1467.90, stdev=1665.52 lat (usec): min=236, max=45090, avg=1469.33, stdev=1665.51 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 619], 10.00th=[ 840], 20.00th=[ 1057], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2040], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min= 7328, max=13344, per=100.00%, avg=9296.82, stdev=838.69, samples=60 iops : min= 458, max= 834, avg=581.05, stdev=52.42, samples=60 lat (usec) : 50=47.14%, 100=32.51%, 250=0.26%, 500=0.70%, 750=0.83% lat (usec) : 1000=1.79% lat (msec) : 2=15.59%, 4=0.78%, 10=0.27%, 20=0.10%, 50=0.02% cpu : usr=1.47%, sys=3.62%, ctx=87156, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69719,17432,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=1089MiB (1142MB), run=30001-30001msec WRITE: bw=9297KiB/s (9520kB/s), 9297KiB/s-9297KiB/s (9520kB/s-9520kB/s), io=272MiB (286MB), run=30001-30001msec Disk stats (read/write): vdk: ios=69230/17311, merge=0/0, ticks=725/17855, in_queue=28220, util=94.32% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901855: Fri Jan 7 13:41:21 2022 read: IOPS=2372, BW=37.1MiB/s (38.9MB/s)(1113MiB/30027msec) slat (usec): min=2, max=222, avg=11.01, stdev= 4.44 clat (nsec): min=655, max=2491.3M, avg=713732.42, stdev=31654778.79 lat (usec): min=20, max=2491.3k, avg=724.91, stdev=31654.78 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 38], 60.00th=[ 40], 70.00th=[ 43], | 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 59], | 99.00th=[ 92], 99.50th=[ 131], 99.90th=[ 914], | 99.95th=[ 429917], 99.99th=[1971323] bw ( KiB/s): min=28256, max=63744, per=100.00%, avg=37987.77, stdev=4965.67, samples=60 iops : min= 1766, max= 3984, avg=2374.23, stdev=310.35, samples=60 write: IOPS=593, BW=9498KiB/s (9726kB/s)(279MiB/30027msec) slat (usec): min=3, max=148, avg=13.57, stdev= 4.72 clat (usec): min=341, max=76803, avg=13923.73, stdev=6468.51 lat (usec): min=349, max=76817, avg=13937.47, stdev=6468.68 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 6390], 10.00th=[ 8586], 20.00th=[10552], | 30.00th=[11731], 40.00th=[12649], 50.00th=[13304], 60.00th=[13960], | 70.00th=[14615], 80.00th=[15533], 90.00th=[20317], 95.00th=[22676], | 99.00th=[34341], 99.50th=[63177], 99.90th=[68682], 99.95th=[72877], | 99.99th=[77071] bw ( KiB/s): min= 7840, max=16064, per=100.00%, avg=9503.57, stdev=1073.38, samples=60 iops : min= 490, max= 1004, avg=593.97, stdev=67.08, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.03%, 20=0.08%, 50=70.62% lat (usec) : 100=8.52%, 250=0.49%, 500=0.07%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.24%, 4=0.10%, 10=3.00%, 20=14.59%, 50=1.92% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.36%, sys=6.43%, ctx=87333, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71225,17825,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1113MiB (1167MB), run=30027-30027msec WRITE: bw=9498KiB/s (9726kB/s), 9498KiB/s-9498KiB/s (9726kB/s-9726kB/s), io=279MiB (292MB), run=30027-30027msec Disk stats (read/write): vdk: ios=70882/17718, merge=0/0, ticks=44539/237745, in_queue=296396, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902749: Fri Jan 7 13:41:58 2022 read: IOPS=2328, BW=36.4MiB/s (38.1MB/s)(1092MiB/30002msec) clat (usec): min=25, max=22942, avg=60.54, stdev=393.36 lat (usec): min=25, max=22942, avg=60.68, stdev=393.36 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 98], 99.90th=[ 310], 99.95th=[12649], | 99.99th=[16909] bw ( KiB/s): min=30208, max=63200, per=100.00%, avg=37256.53, stdev=4754.94, samples=60 iops : min= 1888, max= 3950, avg=2328.53, stdev=297.18, samples=60 write: IOPS=582, BW=9324KiB/s (9548kB/s)(273MiB/30002msec) clat (usec): min=238, max=43414, avg=1463.73, stdev=1642.74 lat (usec): min=240, max=43415, avg=1465.16, stdev=1642.74 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 603], 10.00th=[ 840], 20.00th=[ 1057], | 30.00th=[ 1172], 40.00th=[ 1221], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2057], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[34866], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min= 7456, max=16256, per=100.00%, avg=9324.27, stdev=1194.86, samples=60 iops : min= 466, max= 1016, avg=582.77, stdev=74.68, samples=60 lat (usec) : 50=47.71%, 100=31.90%, 250=0.30%, 500=0.70%, 750=0.84% lat (usec) : 1000=1.76% lat (msec) : 2=15.54%, 4=0.86%, 10=0.27%, 20=0.10%, 50=0.02% cpu : usr=1.44%, sys=3.79%, ctx=87347, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69856,17484,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=36.4MiB/s (38.1MB/s), 36.4MiB/s-36.4MiB/s (38.1MB/s-38.1MB/s), io=1092MiB (1145MB), run=30002-30002msec WRITE: bw=9324KiB/s (9548kB/s), 9324KiB/s-9324KiB/s (9548kB/s-9548kB/s), io=273MiB (286MB), run=30002-30002msec Disk stats (read/write): vdk: ios=69500/17380, merge=0/0, ticks=737/17936, in_queue=28197, util=94.10% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903781: Fri Jan 7 13:42:35 2022 read: IOPS=2375, BW=37.1MiB/s (38.9MB/s)(1117MiB/30091msec) slat (usec): min=2, max=280, avg= 9.82, stdev= 4.28 clat (nsec): min=728, max=2512.1M, avg=773125.86, stdev=32376464.21 lat (usec): min=24, max=2512.1k, avg=783.09, stdev=32376.47 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 40], 70.00th=[ 42], | 80.00th=[ 46], 90.00th=[ 56], 95.00th=[ 72], | 99.00th=[ 198], 99.50th=[ 457], 99.90th=[ 1287], | 99.95th=[ 541066], 99.99th=[1786774] bw ( KiB/s): min=27872, max=69867, per=100.00%, avg=38131.00, stdev=5600.50, samples=60 iops : min= 1742, max= 4366, avg=2383.12, stdev=349.98, samples=60 write: IOPS=594, BW=9515KiB/s (9744kB/s)(280MiB/30091msec) slat (usec): min=3, max=148, avg=12.92, stdev= 4.60 clat (msec): min=7, max=2468, avg=80.91, stdev=75.79 lat (msec): min=7, max=2468, avg=80.93, stdev=75.79 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 55], 10.00th=[ 60], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 70], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 84], 90.00th=[ 93], 95.00th=[ 123], | 99.00th=[ 264], 99.50th=[ 567], 99.90th=[ 1217], 99.95th=[ 1603], | 99.99th=[ 2039] bw ( KiB/s): min= 7552, max=16256, per=100.00%, avg=9521.20, stdev=1108.09, samples=60 iops : min= 472, max= 1016, avg=595.07, stdev=69.25, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.04%, 20=0.12%, 50=68.21% lat (usec) : 100=9.90%, 250=0.95%, 500=0.30%, 750=0.17%, 1000=0.10% lat (msec) : 2=0.03%, 10=0.04%, 20=0.26%, 50=0.24%, 100=17.92% lat (msec) : 250=1.37%, 500=0.10%, 750=0.06%, 1000=0.04%, 2000=0.04% lat (msec) : >=2000=0.01% cpu : usr=1.42%, sys=5.67%, ctx=86641, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=71486,17895,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1117MiB (1171MB), run=30091-30091msec WRITE: bw=9515KiB/s (9744kB/s), 9515KiB/s-9515KiB/s (9744kB/s-9744kB/s), io=280MiB (293MB), run=30091-30091msec Disk stats (read/write): vdk: ios=71482/17857, merge=0/0, ticks=50137/1433876, in_queue=1500736, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904245: Fri Jan 7 13:43:12 2022 read: IOPS=2315, BW=36.2MiB/s (37.9MB/s)(1086MiB/30002msec) clat (usec): min=27, max=22761, avg=60.52, stdev=386.30 lat (usec): min=27, max=22761, avg=60.66, stdev=386.30 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 343], 99.95th=[11863], | 99.99th=[16712] bw ( KiB/s): min=28160, max=59680, per=100.00%, avg=37058.38, stdev=4777.27, samples=60 iops : min= 1760, max= 3730, avg=2316.13, stdev=298.57, samples=60 write: IOPS=579, BW=9269KiB/s (9491kB/s)(272MiB/30002msec) clat (usec): min=237, max=49502, avg=1474.03, stdev=1706.63 lat (usec): min=238, max=49504, avg=1475.52, stdev=1706.62 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 611], 10.00th=[ 857], 20.00th=[ 1057], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1778], 95.00th=[ 2073], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 7392, max=15328, per=100.00%, avg=9269.12, stdev=1118.89, samples=60 iops : min= 462, max= 958, avg=579.32, stdev=69.93, samples=60 lat (usec) : 50=47.77%, 100=31.85%, 250=0.30%, 500=0.70%, 750=0.78% lat (usec) : 1000=1.76% lat (msec) : 2=15.58%, 4=0.86%, 10=0.29%, 20=0.10%, 50=0.02% cpu : usr=1.48%, sys=3.73%, ctx=86874, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69482,17380,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=36.2MiB/s (37.9MB/s), 36.2MiB/s-36.2MiB/s (37.9MB/s-37.9MB/s), io=1086MiB (1138MB), run=30002-30002msec WRITE: bw=9269KiB/s (9491kB/s), 9269KiB/s-9269KiB/s (9491kB/s-9491kB/s), io=272MiB (285MB), run=30002-30002msec Disk stats (read/write): vdk: ios=69036/17251, merge=0/0, ticks=719/17958, in_queue=28140, util=94.12% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905217: Fri Jan 7 13:43:49 2022 read: IOPS=2374, BW=37.1MiB/s (38.9MB/s)(1120MiB/30175msec) slat (usec): min=2, max=499, avg= 9.75, stdev= 5.98 clat (nsec): min=687, max=2652.0M, avg=800631.68, stdev=31761067.93 lat (usec): min=23, max=2652.0k, avg=810.53, stdev=31761.06 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 49], 90.00th=[ 69], 95.00th=[ 147], | 99.00th=[ 1352], 99.50th=[ 2089], 99.90th=[ 5276], | 99.95th=[ 505414], 99.99th=[1719665] bw ( KiB/s): min=30592, max=75040, per=100.00%, avg=38218.25, stdev=5871.34, samples=60 iops : min= 1912, max= 4690, avg=2388.60, stdev=366.93, samples=60 write: IOPS=594, BW=9513KiB/s (9741kB/s)(280MiB/30175msec) slat (usec): min=3, max=262, avg=13.11, stdev= 7.10 clat (msec): min=4, max=2409, avg=164.91, stdev=109.33 lat (msec): min=4, max=2409, avg=164.92, stdev=109.33 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 117], 10.00th=[ 129], 20.00th=[ 136], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 201], 95.00th=[ 218], | 99.00th=[ 523], 99.50th=[ 877], 99.90th=[ 1854], 99.95th=[ 2198], | 99.99th=[ 2366] bw ( KiB/s): min= 8224, max=16192, per=100.00%, avg=9518.08, stdev=1088.49, samples=60 iops : min= 514, max= 1012, avg=594.87, stdev=68.03, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.07%, 4=0.01%, 10=0.02%, 20=0.11%, 50=64.92% lat (usec) : 100=8.99%, 250=2.82%, 500=0.89%, 750=0.57%, 1000=0.37% lat (msec) : 2=0.72%, 4=0.28%, 10=0.12%, 20=0.14%, 50=0.26% lat (msec) : 100=0.25%, 250=18.79%, 500=0.38%, 750=0.08%, 1000=0.06% lat (msec) : 2000=0.08%, >=2000=0.02% cpu : usr=1.36%, sys=5.66%, ctx=83197, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=71653,17940,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1120MiB (1174MB), run=30175-30175msec WRITE: bw=9513KiB/s (9741kB/s), 9513KiB/s-9513KiB/s (9741kB/s-9741kB/s), io=280MiB (294MB), run=30175-30175msec Disk stats (read/write): vdk: ios=71649/17864, merge=0/0, ticks=49973/2926791, in_queue=3000693, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905795: Fri Jan 7 13:44:26 2022 read: IOPS=2302, BW=35.0MiB/s (37.7MB/s)(1079MiB/30001msec) clat (usec): min=28, max=24963, avg=60.02, stdev=393.46 lat (usec): min=28, max=24963, avg=60.16, stdev=393.46 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 66], | 99.00th=[ 81], 99.50th=[ 92], 99.90th=[ 445], 99.95th=[11863], | 99.99th=[16450] bw ( KiB/s): min=25952, max=61824, per=100.00%, avg=36835.75, stdev=5129.78, samples=60 iops : min= 1622, max= 3864, avg=2302.22, stdev=320.62, samples=60 write: IOPS=575, BW=9209KiB/s (9430kB/s)(270MiB/30001msec) clat (usec): min=223, max=89461, avg=1487.51, stdev=1848.58 lat (usec): min=224, max=89462, avg=1488.95, stdev=1848.59 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 611], 10.00th=[ 865], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[42730], | 99.99th=[46400] bw ( KiB/s): min= 6720, max=15840, per=100.00%, avg=9209.32, stdev=1152.52, samples=60 iops : min= 420, max= 990, avg=575.57, stdev=72.03, samples=60 lat (usec) : 50=50.14%, 100=29.58%, 250=0.20%, 500=0.69%, 750=0.78% lat (usec) : 1000=1.66% lat (msec) : 2=15.68%, 4=0.83%, 10=0.31%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.50%, sys=3.60%, ctx=86341, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69065,17268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=35.0MiB/s (37.7MB/s), 35.0MiB/s-35.0MiB/s (37.7MB/s-37.7MB/s), io=1079MiB (1132MB), run=30001-30001msec WRITE: bw=9209KiB/s (9430kB/s), 9209KiB/s-9209KiB/s (9430kB/s-9430kB/s), io=270MiB (283MB), run=30001-30001msec Disk stats (read/write): vdk: ios=68613/17170, merge=0/0, ticks=715/18135, in_queue=28195, util=94.23% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906615: Fri Jan 7 13:45:03 2022 read: IOPS=2377, BW=37.1MiB/s (38.9MB/s)(1122MiB/30199msec) slat (usec): min=2, max=255154, avg=333.00, stdev=2292.12 clat (usec): min=654, max=2534.0k, avg=26951.25, stdev=38773.46 lat (usec): min=657, max=2534.0k, avg=27284.40, stdev=38855.73 clat percentiles (usec): | 1.00th=[ 1270], 5.00th=[ 6980], 10.00th=[ 14353], | 20.00th=[ 18220], 30.00th=[ 20579], 40.00th=[ 22414], | 50.00th=[ 24511], 60.00th=[ 26608], 70.00th=[ 28705], | 80.00th=[ 31589], 90.00th=[ 36439], 95.00th=[ 42206], | 99.00th=[ 84411], 99.50th=[ 198181], 99.90th=[ 263193], | 99.95th=[ 616563], 99.99th=[2264925] bw ( KiB/s): min=30688, max=75200, per=100.00%, avg=38247.63, stdev=5829.87, samples=60 iops : min= 1918, max= 4700, avg=2390.47, stdev=364.37, samples=60 write: IOPS=595, BW=9520KiB/s (9749kB/s)(281MiB/30199msec) slat (usec): min=3, max=64391, avg=329.66, stdev=1427.52 clat (msec): min=5, max=2977, avg=226.73, stdev=154.00 lat (msec): min=5, max=2977, avg=227.06, stdev=154.00 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 163], 10.00th=[ 174], 20.00th=[ 184], | 30.00th=[ 192], 40.00th=[ 201], 50.00th=[ 209], 60.00th=[ 215], | 70.00th=[ 224], 80.00th=[ 236], 90.00th=[ 264], 95.00th=[ 305], | 99.00th=[ 818], 99.50th=[ 1167], 99.90th=[ 2601], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 7680, max=15648, per=99.95%, avg=9515.50, stdev=1109.10, samples=60 iops : min= 480, max= 978, avg=594.72, stdev=69.32, samples=60 lat (usec) : 750=0.02%, 1000=0.28% lat (msec) : 2=1.67%, 4=1.49%, 10=1.05%, 20=17.43%, 50=56.14% lat (msec) : 100=1.85%, 250=17.07%, 500=2.57%, 750=0.15%, 1000=0.10% lat (msec) : 2000=0.11%, >=2000=0.05% cpu : usr=0.78%, sys=2.01%, ctx=9837, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=71784,17969,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1122MiB (1176MB), run=30199-30199msec WRITE: bw=9520KiB/s (9749kB/s), 9520KiB/s-9520KiB/s (9749kB/s-9749kB/s), io=281MiB (294MB), run=30199-30199msec Disk stats (read/write): vdk: ios=71781/17886, merge=0/0, ticks=60820/3575651, in_queue=3669696, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908077: Fri Jan 7 13:45:40 2022 read: IOPS=2257, BW=70.6MiB/s (73.0MB/s)(2117MiB/30001msec) clat (usec): min=31, max=21520, avg=65.18, stdev=395.56 lat (usec): min=31, max=21521, avg=65.32, stdev=395.56 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 89], 99.50th=[ 100], 99.90th=[ 441], 99.95th=[11994], | 99.99th=[16909] bw ( KiB/s): min=54848, max=112256, per=100.00%, avg=72250.67, stdev=8511.43, samples=60 iops : min= 1714, max= 3508, avg=2257.83, stdev=265.98, samples=60 write: IOPS=564, BW=17.6MiB/s (18.5MB/s)(529MiB/30001msec) clat (usec): min=274, max=87736, avg=1499.46, stdev=1751.64 lat (usec): min=277, max=87739, avg=1502.43, stdev=1751.63 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 635], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1401], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min=14464, max=28608, per=100.00%, avg=18065.07, stdev=1957.56, samples=60 iops : min= 452, max= 894, avg=564.53, stdev=61.17, samples=60 lat (usec) : 50=35.49%, 100=44.11%, 250=0.29%, 500=0.61%, 750=0.90% lat (usec) : 1000=1.55% lat (msec) : 2=15.68%, 4=0.98%, 10=0.25%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.50%, sys=3.69%, ctx=84678, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67735,16937,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=70.6MiB/s (73.0MB/s), 70.6MiB/s-70.6MiB/s (73.0MB/s-73.0MB/s), io=2117MiB (2220MB), run=30001-30001msec WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=529MiB (555MB), run=30001-30001msec Disk stats (read/write): vdk: ios=67371/16837, merge=0/0, ticks=745/17667, in_queue=28106, util=93.89% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908661: Fri Jan 7 13:46:17 2022 read: IOPS=2234, BW=69.8MiB/s (73.2MB/s)(2095MiB/30001msec) slat (usec): min=5, max=289, avg=11.42, stdev= 4.77 clat (nsec): min=696, max=25735k, avg=53391.23, stdev=417550.57 lat (usec): min=27, max=25747, avg=64.96, stdev=417.57 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 39], 50.00th=[ 40], 60.00th=[ 42], | 70.00th=[ 44], 80.00th=[ 47], 90.00th=[ 52], 95.00th=[ 56], | 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 375], 99.95th=[12780], | 99.99th=[16909] bw ( KiB/s): min=52672, max=106496, per=99.82%, avg=71376.27, stdev=8543.52, samples=59 iops : min= 1646, max= 3328, avg=2230.51, stdev=266.99, samples=59 write: IOPS=559, BW=17.5MiB/s (18.3MB/s)(524MiB/30001msec) slat (usec): min=9, max=366, avg=15.14, stdev= 5.66 clat (usec): min=260, max=43515, avg=1505.58, stdev=1689.89 lat (usec): min=273, max=43528, avg=1520.89, stdev=1689.94 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 611], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[34866], 99.95th=[40633], | 99.99th=[42730] bw ( KiB/s): min=15296, max=27840, per=99.90%, avg=17877.69, stdev=1822.70, samples=59 iops : min= 478, max= 870, avg=558.68, stdev=56.96, samples=59 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.10%, 4=0.01%, 10=0.03%, 20=0.08%, 50=69.73% lat (usec) : 100=9.80%, 250=0.08%, 500=0.66%, 750=0.85%, 1000=1.54% lat (msec) : 2=15.72%, 4=0.91%, 10=0.28%, 20=0.13%, 50=0.03% cpu : usr=1.32%, sys=6.12%, ctx=83640, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67038,16778,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=69.8MiB/s (73.2MB/s), 69.8MiB/s-69.8MiB/s (73.2MB/s-73.2MB/s), io=2095MiB (2197MB), run=30001-30001msec WRITE: bw=17.5MiB/s (18.3MB/s), 17.5MiB/s-17.5MiB/s (18.3MB/s-18.3MB/s), io=524MiB (550MB), run=30001-30001msec Disk stats (read/write): vdk: ios=66655/16685, merge=0/0, ticks=777/17850, in_queue=28100, util=93.81% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909511: Fri Jan 7 13:46:54 2022 read: IOPS=2212, BW=69.2MiB/s (72.5MB/s)(2075MiB/30001msec) clat (usec): min=29, max=34930, avg=65.18, stdev=430.45 lat (usec): min=29, max=34930, avg=65.32, stdev=430.45 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 86], 99.50th=[ 96], 99.90th=[ 457], 99.95th=[12649], | 99.99th=[17171] bw ( KiB/s): min=55424, max=111872, per=100.00%, avg=70822.82, stdev=8954.67, samples=60 iops : min= 1732, max= 3496, avg=2213.20, stdev=279.83, samples=60 write: IOPS=553, BW=17.3MiB/s (18.1MB/s)(519MiB/30001msec) clat (usec): min=274, max=43660, avg=1533.95, stdev=1683.77 lat (usec): min=276, max=43663, avg=1536.86, stdev=1683.75 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 881], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1287], 50.00th=[ 1336], 60.00th=[ 1418], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min=14016, max=28608, per=100.00%, avg=17723.87, stdev=1956.35, samples=60 iops : min= 438, max= 894, avg=553.87, stdev=61.13, samples=60 lat (usec) : 50=37.99%, 100=41.67%, 250=0.23%, 500=0.63%, 750=0.79% lat (usec) : 1000=1.47% lat (msec) : 2=15.74%, 4=1.03%, 10=0.30%, 20=0.13%, 50=0.02% cpu : usr=1.31%, sys=3.68%, ctx=83015, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66392,16616,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=69.2MiB/s (72.5MB/s), 69.2MiB/s-69.2MiB/s (72.5MB/s-72.5MB/s), io=2075MiB (2176MB), run=30001-30001msec WRITE: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=519MiB (544MB), run=30001-30001msec Disk stats (read/write): vdk: ios=65805/16483, merge=0/0, ticks=778/17851, in_queue=28277, util=94.61% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910042: Fri Jan 7 13:47:31 2022 read: IOPS=2291, BW=71.6MiB/s (75.1MB/s)(2150MiB/30022msec) slat (usec): min=2, max=405, avg=11.21, stdev= 4.36 clat (nsec): min=713, max=2512.9M, avg=701808.52, stdev=30184539.26 lat (usec): min=30, max=2512.9k, avg=713.17, stdev=30184.52 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 37], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 48], 90.00th=[ 53], 95.00th=[ 60], | 99.00th=[ 96], 99.50th=[ 145], 99.90th=[ 734], | 99.95th=[ 329253], 99.99th=[1769997] bw ( KiB/s): min=56320, max=126080, per=100.00%, avg=73392.00, stdev=9539.17, samples=60 iops : min= 1760, max= 3940, avg=2293.50, stdev=298.10, samples=60 write: IOPS=573, BW=17.9MiB/s (18.8MB/s)(538MiB/30022msec) slat (usec): min=5, max=565, avg=15.45, stdev= 6.45 clat (usec): min=410, max=74300, avg=14571.55, stdev=6097.39 lat (usec): min=442, max=74315, avg=14587.16, stdev=6097.58 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 8291], 10.00th=[10028], 20.00th=[11600], | 30.00th=[12518], 40.00th=[13304], 50.00th=[13960], 60.00th=[14615], | 70.00th=[15270], 80.00th=[16319], 90.00th=[19792], 95.00th=[22938], | 99.00th=[32637], 99.50th=[63701], 99.90th=[69731], 99.95th=[72877], | 99.99th=[73925] bw ( KiB/s): min=15168, max=31680, per=100.00%, avg=18344.53, stdev=2133.21, samples=60 iops : min= 474, max= 990, avg=573.27, stdev=66.66, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=67.94% lat (usec) : 100=11.25%, 250=0.58%, 500=0.06%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.19%, 4=0.10%, 10=1.68%, 20=16.09%, 50=1.79% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.45%, sys=5.96%, ctx=84907, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68807,17206,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=71.6MiB/s (75.1MB/s), 71.6MiB/s-71.6MiB/s (75.1MB/s-75.1MB/s), io=2150MiB (2255MB), run=30022-30022msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=538MiB (564MB), run=30022-30022msec Disk stats (read/write): vdk: ios=68411/17114, merge=0/0, ticks=43436/240773, in_queue=296780, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910982: Fri Jan 7 13:48:08 2022 read: IOPS=2214, BW=69.2MiB/s (72.6MB/s)(2076MiB/30002msec) clat (usec): min=28, max=67037, avg=65.37, stdev=535.98 lat (usec): min=29, max=67037, avg=65.51, stdev=535.98 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 94], 99.50th=[ 110], 99.90th=[ 644], 99.95th=[13042], | 99.99th=[16909] bw ( KiB/s): min=55040, max=111808, per=100.00%, avg=70864.42, stdev=9147.08, samples=60 iops : min= 1720, max= 3494, avg=2214.48, stdev=285.86, samples=60 write: IOPS=554, BW=17.3MiB/s (18.2MB/s)(520MiB/30002msec) clat (usec): min=279, max=131525, avg=1531.58, stdev=1933.13 lat (usec): min=282, max=131528, avg=1534.65, stdev=1933.12 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 889], 20.00th=[ 1106], | 30.00th=[ 1221], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1418], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[25560], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=12480, max=28544, per=100.00%, avg=17737.22, stdev=2165.66, samples=60 iops : min= 390, max= 892, avg=554.28, stdev=67.67, samples=60 lat (usec) : 50=41.69%, 100=37.72%, 250=0.45%, 500=0.63%, 750=0.82% lat (usec) : 1000=1.45% lat (msec) : 2=15.79%, 4=1.03%, 10=0.27%, 20=0.13%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.36%, sys=3.89%, ctx=83065, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66429,16628,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=69.2MiB/s (72.6MB/s), 69.2MiB/s-69.2MiB/s (72.6MB/s-72.6MB/s), io=2076MiB (2177MB), run=30002-30002msec WRITE: bw=17.3MiB/s (18.2MB/s), 17.3MiB/s-17.3MiB/s (18.2MB/s-18.2MB/s), io=520MiB (545MB), run=30002-30002msec Disk stats (read/write): vdk: ios=66039/16537, merge=0/0, ticks=859/17844, in_queue=28229, util=94.28% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911955: Fri Jan 7 13:48:45 2022 read: IOPS=2258, BW=70.6MiB/s (74.0MB/s)(2126MiB/30118msec) slat (usec): min=2, max=216, avg=10.97, stdev= 4.30 clat (nsec): min=753, max=2513.8M, avg=834351.65, stdev=34583007.56 lat (usec): min=26, max=2513.8k, avg=845.48, stdev=34583.01 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 57], 95.00th=[ 68], | 99.00th=[ 215], 99.50th=[ 498], 99.90th=[ 1221], | 99.95th=[ 476054], 99.99th=[2139096] bw ( KiB/s): min=56192, max=136000, per=100.00%, avg=72566.52, stdev=10321.75, samples=60 iops : min= 1756, max= 4250, avg=2267.70, stdev=322.56, samples=60 write: IOPS=565, BW=17.7MiB/s (18.5MB/s)(532MiB/30118msec) slat (usec): min=4, max=148, avg=16.39, stdev= 5.67 clat (msec): min=7, max=2047, avg=85.06, stdev=68.90 lat (msec): min=7, max=2048, avg=85.07, stdev=68.90 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 60], 10.00th=[ 64], 20.00th=[ 69], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 101], 95.00th=[ 124], | 99.00th=[ 226], 99.50th=[ 535], 99.90th=[ 1099], 99.95th=[ 1318], | 99.99th=[ 1670] bw ( KiB/s): min=15488, max=31680, per=100.00%, avg=18106.12, stdev=2131.86, samples=60 iops : min= 484, max= 990, avg=565.80, stdev=66.64, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.06%, 50=65.07% lat (usec) : 100=13.08%, 250=0.96%, 500=0.32%, 750=0.16%, 1000=0.10% lat (msec) : 2=0.07%, 10=0.03%, 20=0.26%, 50=0.17%, 100=17.61% lat (msec) : 250=1.77%, 500=0.10%, 750=0.05%, 1000=0.04%, 2000=0.05% lat (msec) : >=2000=0.01% cpu : usr=1.46%, sys=5.82%, ctx=82512, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=68032,17021,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=70.6MiB/s (74.0MB/s), 70.6MiB/s-70.6MiB/s (74.0MB/s-74.0MB/s), io=2126MiB (2229MB), run=30118-30118msec WRITE: bw=17.7MiB/s (18.5MB/s), 17.7MiB/s-17.7MiB/s (18.5MB/s-18.5MB/s), io=532MiB (558MB), run=30118-30118msec Disk stats (read/write): vdk: ios=68029/17009, merge=0/0, ticks=49399/1434454, in_queue=1500477, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912587: Fri Jan 7 13:49:22 2022 read: IOPS=2231, BW=69.7MiB/s (73.1MB/s)(2092MiB/30001msec) clat (usec): min=28, max=67865, avg=65.02, stdev=528.20 lat (usec): min=28, max=67865, avg=65.18, stdev=528.20 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 90], 99.50th=[ 105], 99.90th=[ 660], 99.95th=[13173], | 99.99th=[21627] bw ( KiB/s): min=50048, max=103424, per=99.97%, avg=71395.07, stdev=8032.02, samples=59 iops : min= 1564, max= 3232, avg=2231.08, stdev=251.00, samples=59 write: IOPS=558, BW=17.5MiB/s (18.3MB/s)(524MiB/30001msec) clat (usec): min=271, max=52060, avg=1519.10, stdev=1662.30 lat (usec): min=273, max=52062, avg=1522.18, stdev=1662.22 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 644], 10.00th=[ 889], 20.00th=[ 1106], | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1336], 60.00th=[ 1418], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min=13632, max=26752, per=100.00%, avg=17884.76, stdev=1754.72, samples=59 iops : min= 426, max= 836, avg=558.90, stdev=54.84, samples=59 lat (usec) : 50=45.02%, 100=34.46%, 250=0.37%, 500=0.64%, 750=0.77% lat (usec) : 1000=1.40% lat (msec) : 2=15.91%, 4=1.03%, 10=0.25%, 20=0.12%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.57%, sys=3.58%, ctx=83716, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66956,16755,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=69.7MiB/s (73.1MB/s), 69.7MiB/s-69.7MiB/s (73.1MB/s-73.1MB/s), io=2092MiB (2194MB), run=30001-30001msec WRITE: bw=17.5MiB/s (18.3MB/s), 17.5MiB/s-17.5MiB/s (18.3MB/s-18.3MB/s), io=524MiB (549MB), run=30001-30001msec Disk stats (read/write): vdk: ios=66542/16657, merge=0/0, ticks=902/17797, in_queue=28143, util=94.07% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913422: Fri Jan 7 13:49:59 2022 read: IOPS=2291, BW=71.6MiB/s (75.1MB/s)(2162MiB/30186msec) slat (usec): min=2, max=313, avg=11.06, stdev= 6.06 clat (nsec): min=675, max=2512.3M, avg=941903.43, stdev=36432374.09 lat (usec): min=24, max=2512.3k, avg=953.12, stdev=36432.38 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 33], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 44], 70.00th=[ 48], | 80.00th=[ 52], 90.00th=[ 64], 95.00th=[ 100], | 99.00th=[ 1057], 99.50th=[ 1844], 99.90th=[ 3195], | 99.95th=[ 616563], 99.99th=[1920992] bw ( KiB/s): min=60032, max=144576, per=100.00%, avg=73794.13, stdev=11238.67, samples=60 iops : min= 1876, max= 4518, avg=2306.05, stdev=351.20, samples=60 write: IOPS=573, BW=17.9MiB/s (18.8MB/s)(541MiB/30186msec) slat (usec): min=4, max=329, avg=17.66, stdev= 8.11 clat (msec): min=12, max=2086, avg=170.57, stdev=100.37 lat (msec): min=12, max=2086, avg=170.59, stdev=100.37 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 125], 10.00th=[ 134], 20.00th=[ 144], | 30.00th=[ 150], 40.00th=[ 157], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 205], 95.00th=[ 222], | 99.00th=[ 397], 99.50th=[ 927], 99.90th=[ 1670], 99.95th=[ 1670], | 99.99th=[ 2056] bw ( KiB/s): min=14976, max=30976, per=100.00%, avg=18359.65, stdev=2019.06, samples=60 iops : min= 468, max= 968, avg=573.73, stdev=63.09, samples=60 lat (nsec) : 750=0.01%, 1000=0.08% lat (usec) : 2=0.07%, 4=0.01%, 10=0.04%, 20=0.19%, 50=60.57% lat (usec) : 100=15.03%, 250=1.63%, 500=0.80%, 750=0.45%, 1000=0.28% lat (msec) : 2=0.52%, 4=0.25%, 20=0.19%, 50=0.20%, 100=0.22% lat (msec) : 250=18.90%, 500=0.35%, 750=0.06%, 1000=0.05%, 2000=0.10% lat (msec) : >=2000=0.01% cpu : usr=1.51%, sys=5.91%, ctx=81050, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=69182,17305,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=71.6MiB/s (75.1MB/s), 71.6MiB/s-71.6MiB/s (75.1MB/s-75.1MB/s), io=2162MiB (2267MB), run=30186-30186msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=541MiB (567MB), run=30186-30186msec Disk stats (read/write): vdk: ios=69177/17215, merge=0/0, ticks=54247/2918276, in_queue=3005593, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914733: Fri Jan 7 13:50:36 2022 read: IOPS=2220, BW=69.4MiB/s (72.8MB/s)(2083MiB/30015msec) clat (usec): min=27, max=24174, avg=62.32, stdev=416.03 lat (usec): min=28, max=24174, avg=62.45, stdev=416.03 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 510], 99.95th=[12780], | 99.99th=[17957] bw ( KiB/s): min=55488, max=105408, per=100.00%, avg=71108.32, stdev=8118.92, samples=60 iops : min= 1734, max= 3294, avg=2222.10, stdev=253.69, samples=60 write: IOPS=555, BW=17.4MiB/s (18.2MB/s)(521MiB/30015msec) clat (usec): min=274, max=88012, avg=1538.57, stdev=1864.97 lat (usec): min=277, max=88015, avg=1541.54, stdev=1864.97 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 644], 10.00th=[ 898], 20.00th=[ 1123], | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1336], 60.00th=[ 1418], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[40633], | 99.99th=[51643] bw ( KiB/s): min=13376, max=27136, per=100.00%, avg=17799.62, stdev=1762.88, samples=60 iops : min= 418, max= 848, avg=556.23, stdev=55.09, samples=60 lat (usec) : 50=46.53%, 100=33.12%, 250=0.23%, 500=0.63%, 750=0.74% lat (usec) : 1000=1.38% lat (msec) : 2=15.95%, 4=0.99%, 10=0.28%, 20=0.13%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.46%, sys=3.62%, ctx=83347, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66655,16686,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=69.4MiB/s (72.8MB/s), 69.4MiB/s-69.4MiB/s (72.8MB/s-72.8MB/s), io=2083MiB (2184MB), run=30015-30015msec WRITE: bw=17.4MiB/s (18.2MB/s), 17.4MiB/s-17.4MiB/s (18.2MB/s-18.2MB/s), io=521MiB (547MB), run=30015-30015msec Disk stats (read/write): vdk: ios=66297/16591, merge=0/0, ticks=753/18050, in_queue=28201, util=94.37% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915349: Fri Jan 7 13:51:13 2022 read: IOPS=2302, BW=71.0MiB/s (75.5MB/s)(2172MiB/30188msec) slat (usec): min=2, max=200176, avg=338.12, stdev=2061.70 clat (usec): min=810, max=2552.2k, avg=27937.55, stdev=42865.22 lat (usec): min=814, max=2552.2k, avg=28275.81, stdev=42919.32 clat percentiles (usec): | 1.00th=[ 1319], 5.00th=[ 4817], 10.00th=[ 15270], | 20.00th=[ 19006], 30.00th=[ 21365], 40.00th=[ 23200], | 50.00th=[ 25297], 60.00th=[ 27132], 70.00th=[ 29492], | 80.00th=[ 32900], 90.00th=[ 38011], 95.00th=[ 43779], | 99.00th=[ 87557], 99.50th=[ 177210], 99.90th=[ 225444], | 99.95th=[ 734004], 99.99th=[2533360] bw ( KiB/s): min=58240, max=148032, per=100.00%, avg=74080.47, stdev=11237.12, samples=60 iops : min= 1820, max= 4626, avg=2315.00, stdev=351.16, samples=60 write: IOPS=575, BW=17.0MiB/s (18.9MB/s)(543MiB/30188msec) slat (usec): min=4, max=171743, avg=363.87, stdev=2120.98 clat (msec): min=6, max=2698, avg=233.83, stdev=131.89 lat (msec): min=6, max=2698, avg=234.20, stdev=131.87 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 167], 10.00th=[ 182], 20.00th=[ 194], | 30.00th=[ 207], 40.00th=[ 215], 50.00th=[ 222], 60.00th=[ 226], | 70.00th=[ 234], 80.00th=[ 245], 90.00th=[ 271], 95.00th=[ 313], | 99.00th=[ 718], 99.50th=[ 1267], 99.90th=[ 1703], 99.95th=[ 2198], | 99.99th=[ 2702] bw ( KiB/s): min=15198, max=30400, per=99.85%, avg=18397.95, stdev=2057.51, samples=60 iops : min= 474, max= 950, avg=574.92, stdev=64.32, samples=60 lat (usec) : 1000=0.15% lat (msec) : 2=1.86%, 4=1.64%, 10=1.01%, 20=14.25%, 50=58.87% lat (msec) : 100=2.12%, 250=16.66%, 500=3.08%, 750=0.13%, 1000=0.06% lat (msec) : 2000=0.14%, >=2000=0.03% cpu : usr=1.12%, sys=2.06%, ctx=9697, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=69510,17383,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=71.0MiB/s (75.5MB/s), 71.0MiB/s-71.0MiB/s (75.5MB/s-75.5MB/s), io=2172MiB (2278MB), run=30188-30188msec WRITE: bw=17.0MiB/s (18.9MB/s), 17.0MiB/s-17.0MiB/s (18.9MB/s-18.9MB/s), io=543MiB (570MB), run=30188-30188msec Disk stats (read/write): vdk: ios=69509/17297, merge=0/0, ticks=67890/3561707, in_queue=3672754, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916224: Fri Jan 7 13:51:50 2022 read: IOPS=2067, BW=129MiB/s (135MB/s)(3876MiB/30001msec) clat (usec): min=36, max=25606, avg=76.76, stdev=444.76 lat (usec): min=36, max=25606, avg=76.91, stdev=444.77 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 117], 99.50th=[ 139], 99.90th=[ 709], 99.95th=[12911], | 99.99th=[19268] bw ( KiB/s): min=99968, max=211712, per=100.00%, avg=132366.24, stdev=17436.08, samples=59 iops : min= 1562, max= 3308, avg=2068.20, stdev=272.46, samples=59 write: IOPS=519, BW=32.5MiB/s (34.0MB/s)(974MiB/30001msec) clat (usec): min=358, max=95496, avg=1605.48, stdev=1842.68 lat (usec): min=364, max=95502, avg=1611.69, stdev=1842.64 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 660], 10.00th=[ 873], 20.00th=[ 1156], | 30.00th=[ 1287], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2245], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=26496, max=54912, per=100.00%, avg=33258.22, stdev=3869.38, samples=59 iops : min= 414, max= 858, avg=519.64, stdev=60.48, samples=59 lat (usec) : 50=5.34%, 100=72.88%, 250=1.51%, 500=0.53%, 750=0.99% lat (usec) : 1000=1.32% lat (msec) : 2=15.52%, 4=1.49%, 10=0.26%, 20=0.13%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.65%, sys=3.72%, ctx=77605, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62023,15577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3876MiB (4065MB), run=30001-30001msec WRITE: bw=32.5MiB/s (34.0MB/s), 32.5MiB/s-32.5MiB/s (34.0MB/s-34.0MB/s), io=974MiB (1021MB), run=30001-30001msec Disk stats (read/write): vdk: ios=61770/15497, merge=0/0, ticks=730/17186, in_queue=28144, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916842: Fri Jan 7 13:52:27 2022 read: IOPS=2072, BW=130MiB/s (136MB/s)(3887MiB/30001msec) slat (usec): min=6, max=191, avg=13.57, stdev= 4.49 clat (nsec): min=786, max=26887k, avg=61826.26, stdev=421557.85 lat (usec): min=36, max=26905, avg=75.56, stdev=421.61 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 98], 99.90th=[ 469], 99.95th=[13042], | 99.99th=[17433] bw ( KiB/s): min=102144, max=199936, per=100.00%, avg=132931.53, stdev=15268.17, samples=59 iops : min= 1596, max= 3124, avg=2077.05, stdev=238.56, samples=59 write: IOPS=520, BW=32.6MiB/s (34.1MB/s)(977MiB/30001msec) slat (usec): min=11, max=186, avg=20.63, stdev= 6.20 clat (usec): min=344, max=43226, avg=1590.55, stdev=1665.78 lat (usec): min=361, max=43245, avg=1611.37, stdev=1665.93 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 652], 10.00th=[ 881], 20.00th=[ 1156], | 30.00th=[ 1270], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1582], 80.00th=[ 1713], 90.00th=[ 1958], 95.00th=[ 2245], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35390], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min=26112, max=52352, per=100.00%, avg=33385.27, stdev=3411.39, samples=59 iops : min= 408, max= 818, avg=521.64, stdev=53.30, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=48.40%, 100=31.10% lat (usec) : 250=0.26%, 500=0.50%, 750=0.96%, 1000=1.31% lat (msec) : 2=15.55%, 4=1.44%, 10=0.27%, 20=0.14%, 50=0.02% cpu : usr=1.55%, sys=6.25%, ctx=77792, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62188,15627,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3887MiB (4076MB), run=30001-30001msec WRITE: bw=32.6MiB/s (34.1MB/s), 32.6MiB/s-32.6MiB/s (34.1MB/s-34.1MB/s), io=977MiB (1024MB), run=30001-30001msec Disk stats (read/write): vdk: ios=61909/15540, merge=0/0, ticks=728/17199, in_queue=27916, util=93.32% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917790: Fri Jan 7 13:53:04 2022 read: IOPS=2075, BW=130MiB/s (136MB/s)(3891MiB/30001msec) clat (usec): min=32, max=32572, avg=69.62, stdev=454.91 lat (usec): min=32, max=32572, avg=69.77, stdev=454.92 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 102], 99.50th=[ 120], 99.90th=[ 906], 99.95th=[12911], | 99.99th=[18220] bw ( KiB/s): min=98560, max=227456, per=100.00%, avg=132815.30, stdev=18834.42, samples=60 iops : min= 1540, max= 3554, avg=2075.23, stdev=294.29, samples=60 write: IOPS=521, BW=32.6MiB/s (34.2MB/s)(977MiB/30001msec) clat (usec): min=353, max=45356, avg=1626.30, stdev=1676.80 lat (usec): min=359, max=45362, avg=1632.74, stdev=1676.75 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 685], 10.00th=[ 955], 20.00th=[ 1205], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=28032, max=58112, per=100.00%, avg=33359.03, stdev=4181.81, samples=60 iops : min= 438, max= 908, avg=521.22, stdev=65.35, samples=60 lat (usec) : 50=27.00%, 100=52.06%, 250=0.71%, 500=0.51%, 750=0.80% lat (usec) : 1000=1.06% lat (msec) : 2=15.86%, 4=1.59%, 10=0.25%, 20=0.13%, 50=0.03% cpu : usr=1.60%, sys=3.76%, ctx=77899, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62253,15637,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3891MiB (4080MB), run=30001-30001msec WRITE: bw=32.6MiB/s (34.2MB/s), 32.6MiB/s-32.6MiB/s (34.2MB/s-34.2MB/s), io=977MiB (1025MB), run=30001-30001msec Disk stats (read/write): vdk: ios=61974/15558, merge=0/0, ticks=770/17509, in_queue=27995, util=93.58% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918732: Fri Jan 7 13:53:41 2022 read: IOPS=2108, BW=132MiB/s (138MB/s)(3958MiB/30036msec) slat (usec): min=3, max=549, avg=12.90, stdev= 6.32 clat (nsec): min=691, max=2490.7M, avg=928410.23, stdev=38690908.71 lat (usec): min=30, max=2490.7k, avg=941.46, stdev=38690.91 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 38], | 20.00th=[ 40], 30.00th=[ 42], 40.00th=[ 43], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 51], | 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 70], | 99.00th=[ 103], 99.50th=[ 137], 99.90th=[ 1004], | 99.95th=[ 717226], 99.99th=[2499806] bw ( KiB/s): min=105216, max=239583, per=100.00%, avg=135099.80, stdev=17854.13, samples=60 iops : min= 1644, max= 3743, avg=2110.90, stdev=278.92, samples=60 write: IOPS=529, BW=33.1MiB/s (34.7MB/s)(994MiB/30036msec) slat (usec): min=8, max=346, avg=21.73, stdev=10.80 clat (usec): min=723, max=83611, avg=15102.92, stdev=6629.14 lat (usec): min=747, max=83630, avg=15124.84, stdev=6628.80 clat percentiles (usec): | 1.00th=[ 2311], 5.00th=[ 6390], 10.00th=[ 9372], 20.00th=[12125], | 30.00th=[13173], 40.00th=[13960], 50.00th=[14746], 60.00th=[15401], | 70.00th=[16188], 80.00th=[16909], 90.00th=[20841], 95.00th=[23725], | 99.00th=[33817], 99.50th=[65274], 99.90th=[73925], 99.95th=[82314], | 99.99th=[83362] bw ( KiB/s): min=28672, max=60537, per=100.00%, avg=33927.52, stdev=4112.74, samples=60 iops : min= 448, max= 945, avg=530.08, stdev=64.18, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.05%, 50=53.78% lat (usec) : 100=25.07%, 250=0.74%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.09%, 4=0.48%, 10=1.66%, 20=15.64%, 50=2.04% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.76%, sys=5.90%, ctx=78155, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63323,15906,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3958MiB (4150MB), run=30036-30036msec WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=994MiB (1042MB), run=30036-30036msec Disk stats (read/write): vdk: ios=62935/15821, merge=0/0, ticks=53285/231295, in_queue=296659, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919316: Fri Jan 7 13:54:18 2022 read: IOPS=2050, BW=128MiB/s (134MB/s)(3844MiB/30001msec) clat (usec): min=30, max=25802, avg=66.89, stdev=410.02 lat (usec): min=30, max=25802, avg=67.03, stdev=410.02 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 74], | 99.00th=[ 92], 99.50th=[ 103], 99.90th=[ 404], 99.95th=[12911], | 99.99th=[17171] bw ( KiB/s): min=88704, max=179328, per=100.00%, avg=131230.65, stdev=15671.40, samples=60 iops : min= 1386, max= 2802, avg=2050.47, stdev=244.85, samples=60 write: IOPS=515, BW=32.2MiB/s (33.8MB/s)(966MiB/30001msec) clat (usec): min=350, max=107624, avg=1660.94, stdev=1946.39 lat (usec): min=356, max=107630, avg=1667.12, stdev=1946.37 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 701], 10.00th=[ 996], 20.00th=[ 1221], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 2008], 95.00th=[ 2278], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[38011], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=25088, max=46464, per=100.00%, avg=32960.05, stdev=3167.15, samples=60 iops : min= 392, max= 726, avg=515.03, stdev=49.51, samples=60 lat (usec) : 50=30.38%, 100=49.07%, 250=0.38%, 500=0.48%, 750=0.67% lat (usec) : 1000=0.91% lat (msec) : 2=15.95%, 4=1.71%, 10=0.26%, 20=0.15%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.55%, sys=3.56%, ctx=76970, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61510,15451,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=3844MiB (4031MB), run=30001-30001msec WRITE: bw=32.2MiB/s (33.8MB/s), 32.2MiB/s-32.2MiB/s (33.8MB/s-33.8MB/s), io=966MiB (1013MB), run=30001-30001msec Disk stats (read/write): vdk: ios=61142/15361, merge=0/0, ticks=680/17940, in_queue=28188, util=94.26% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920162: Fri Jan 7 13:54:55 2022 read: IOPS=2090, BW=131MiB/s (137MB/s)(3938MiB/30136msec) slat (usec): min=3, max=936, avg=13.56, stdev=16.72 clat (nsec): min=689, max=2520.5M, avg=949040.29, stdev=37628648.81 lat (usec): min=27, max=2520.5k, avg=962.81, stdev=37628.77 clat percentiles (usec): | 1.00th=[ 22], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 37], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 43], 60.00th=[ 47], 70.00th=[ 51], | 80.00th=[ 57], 90.00th=[ 67], 95.00th=[ 84], | 99.00th=[ 947], 99.50th=[ 2704], 99.90th=[ 17695], | 99.95th=[ 509608], 99.99th=[2499806] bw ( KiB/s): min=99456, max=242688, per=100.00%, avg=134402.13, stdev=18788.69, samples=60 iops : min= 1554, max= 3792, avg=2100.03, stdev=293.57, samples=60 write: IOPS=525, BW=32.9MiB/s (34.5MB/s)(990MiB/30136msec) slat (usec): min=7, max=1383, avg=25.78, stdev=30.03 clat (msec): min=4, max=2271, avg=91.23, stdev=72.67 lat (msec): min=6, max=2271, avg=91.26, stdev=72.67 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 63], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 80], 40.00th=[ 82], 50.00th=[ 85], 60.00th=[ 88], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 108], 95.00th=[ 130], | 99.00th=[ 218], 99.50th=[ 502], 99.90th=[ 1217], 99.95th=[ 1737], | 99.99th=[ 2056] bw ( KiB/s): min=26880, max=56448, per=100.00%, avg=33693.87, stdev=3679.30, samples=60 iops : min= 420, max= 882, avg=526.47, stdev=57.49, samples=60 lat (nsec) : 750=0.01%, 1000=0.21% lat (usec) : 2=0.25%, 4=0.02%, 10=0.06%, 20=0.21%, 50=53.47% lat (usec) : 100=22.53%, 250=1.35%, 500=0.68%, 750=0.23%, 1000=0.14% lat (msec) : 2=0.31%, 4=0.14%, 10=0.11%, 20=0.47%, 50=0.15% lat (msec) : 100=16.92%, 250=2.52%, 500=0.09%, 750=0.05%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.02% cpu : usr=1.98%, sys=6.08%, ctx=75136, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=63002,15843,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3938MiB (4129MB), run=30136-30136msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=990MiB (1038MB), run=30136-30136msec Disk stats (read/write): vdk: ios=63001/15832, merge=0/0, ticks=53585/1431572, in_queue=1497012, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921315: Fri Jan 7 13:55:32 2022 read: IOPS=2081, BW=130MiB/s (136MB/s)(3903MiB/30001msec) clat (usec): min=37, max=64337, avg=76.39, stdev=482.99 lat (usec): min=37, max=64337, avg=76.53, stdev=482.99 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 85], | 99.00th=[ 105], 99.50th=[ 120], 99.90th=[ 709], 99.95th=[12518], | 99.99th=[17171] bw ( KiB/s): min=95872, max=212992, per=100.00%, avg=133357.03, stdev=17048.21, samples=59 iops : min= 1498, max= 3328, avg=2083.69, stdev=266.37, samples=59 write: IOPS=523, BW=32.7MiB/s (34.3MB/s)(981MiB/30001msec) clat (usec): min=367, max=43440, avg=1591.80, stdev=1640.58 lat (usec): min=374, max=43447, avg=1598.20, stdev=1640.53 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 660], 10.00th=[ 881], 20.00th=[ 1156], | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[19268], 99.95th=[40633], | 99.99th=[42206] bw ( KiB/s): min=26112, max=55552, per=100.00%, avg=33502.42, stdev=3889.13, samples=59 iops : min= 408, max= 868, avg=523.47, stdev=60.77, samples=59 lat (usec) : 50=4.64%, 100=74.23%, 250=0.93%, 500=0.48%, 750=0.96% lat (usec) : 1000=1.28% lat (msec) : 2=15.57%, 4=1.51%, 10=0.23%, 20=0.16%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.58%, sys=3.80%, ctx=78156, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62447,15700,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3903MiB (4093MB), run=30001-30001msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=981MiB (1029MB), run=30001-30001msec Disk stats (read/write): vdk: ios=62073/15591, merge=0/0, ticks=771/17131, in_queue=28033, util=93.73% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922075: Fri Jan 7 13:56:10 2022 read: IOPS=2123, BW=133MiB/s (139MB/s)(4010MiB/30206msec) slat (usec): min=2, max=369, avg=10.91, stdev= 5.50 clat (nsec): min=740, max=2526.5M, avg=1040995.73, stdev=40673598.45 lat (usec): min=27, max=2526.5k, avg=1052.06, stdev=40673.57 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 47], 70.00th=[ 52], | 80.00th=[ 58], 90.00th=[ 71], 95.00th=[ 137], | 99.00th=[ 1352], 99.50th=[ 2442], 99.90th=[ 3818], | 99.95th=[ 549454], 99.99th=[2499806] bw ( KiB/s): min=98304, max=293248, per=100.00%, avg=136865.35, stdev=24791.66, samples=60 iops : min= 1536, max= 4582, avg=2138.50, stdev=387.39, samples=60 write: IOPS=533, BW=33.3MiB/s (34.9MB/s)(1007MiB/30206msec) slat (usec): min=8, max=306, avg=21.32, stdev= 9.47 clat (msec): min=11, max=3124, avg=183.27, stdev=122.94 lat (msec): min=11, max=3124, avg=183.29, stdev=122.94 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 121], 10.00th=[ 134], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 167], 50.00th=[ 171], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 218], 95.00th=[ 243], | 99.00th=[ 477], 99.50th=[ 1036], 99.90th=[ 1804], 99.95th=[ 2039], | 99.99th=[ 3004] bw ( KiB/s): min=24704, max=63232, per=100.00%, avg=34158.87, stdev=4527.46, samples=60 iops : min= 386, max= 988, avg=533.72, stdev=70.76, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.04%, 50=52.90% lat (usec) : 100=22.11%, 250=1.85%, 500=0.90%, 750=0.57%, 1000=0.36% lat (msec) : 2=0.53%, 4=0.49%, 20=0.09%, 50=0.33%, 100=0.26% lat (msec) : 250=18.53%, 500=0.69%, 750=0.05%, 1000=0.05%, 2000=0.11% lat (msec) : >=2000=0.03% cpu : usr=1.80%, sys=5.23%, ctx=74895, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64153,16109,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=4010MiB (4204MB), run=30206-30206msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1007MiB (1056MB), run=30206-30206msec Disk stats (read/write): vdk: ios=64152/16015, merge=0/0, ticks=61989/2914741, in_queue=3004688, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923043: Fri Jan 7 13:56:46 2022 read: IOPS=2058, BW=129MiB/s (135MB/s)(3861MiB/30007msec) clat (usec): min=32, max=26021, avg=66.06, stdev=418.93 lat (usec): min=32, max=26021, avg=66.19, stdev=418.93 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 87], 99.50th=[ 97], 99.90th=[ 445], 99.95th=[12780], | 99.99th=[16581] bw ( KiB/s): min=98944, max=191872, per=100.00%, avg=131786.63, stdev=16299.30, samples=60 iops : min= 1546, max= 2998, avg=2059.15, stdev=254.68, samples=60 write: IOPS=516, BW=32.3MiB/s (33.9MB/s)(969MiB/30007msec) clat (usec): min=344, max=107611, avg=1658.25, stdev=1918.65 lat (usec): min=350, max=107616, avg=1664.31, stdev=1918.58 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 717], 10.00th=[ 1004], 20.00th=[ 1221], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2311], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36439], 99.95th=[39584], | 99.99th=[46924] bw ( KiB/s): min=22400, max=49792, per=100.00%, avg=33067.72, stdev=3533.61, samples=60 iops : min= 350, max= 778, avg=516.67, stdev=55.22, samples=60 lat (usec) : 50=34.66%, 100=44.94%, 250=0.25%, 500=0.49%, 750=0.63% lat (usec) : 1000=0.90% lat (msec) : 2=16.08%, 4=1.60%, 10=0.27%, 20=0.15%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.58%, sys=3.42%, ctx=77277, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61774,15500,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3861MiB (4048MB), run=30007-30007msec WRITE: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=969MiB (1016MB), run=30007-30007msec Disk stats (read/write): vdk: ios=61371/15420, merge=0/0, ticks=689/18005, in_queue=28254, util=94.48% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923714: Fri Jan 7 13:57:24 2022 read: IOPS=2132, BW=133MiB/s (140MB/s)(4029MiB/30235msec) slat (usec): min=2, max=271636, avg=367.37, stdev=2369.96 clat (usec): min=1051, max=2603.4k, avg=30186.24, stdev=45413.86 lat (usec): min=1055, max=2605.9k, avg=30553.76, stdev=45486.94 clat percentiles (usec): | 1.00th=[ 1893], 5.00th=[ 5997], 10.00th=[ 16450], | 20.00th=[ 20579], 30.00th=[ 23200], 40.00th=[ 25297], | 50.00th=[ 27132], 60.00th=[ 29492], 70.00th=[ 32113], | 80.00th=[ 35390], 90.00th=[ 40633], 95.00th=[ 46924], | 99.00th=[ 87557], 99.50th=[ 223347], 99.90th=[ 299893], | 99.95th=[ 792724], 99.99th=[2432697] bw ( KiB/s): min=108288, max=276480, per=100.00%, avg=137405.88, stdev=22266.19, samples=60 iops : min= 1692, max= 4320, avg=2146.97, stdev=347.89, samples=60 write: IOPS=535, BW=33.5MiB/s (35.1MB/s)(1012MiB/30235msec) slat (usec): min=7, max=201725, avg=380.21, stdev=2296.35 clat (msec): min=7, max=4844, avg=251.43, stdev=286.25 lat (msec): min=7, max=4844, avg=251.81, stdev=286.23 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 197], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 236], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 292], 95.00th=[ 338], | 99.00th=[ 1099], 99.50th=[ 2005], 99.90th=[ 571], 99.95th=[ 571], | 99.99th=[ 571] bw ( KiB/s): min=28032, max=55168, per=99.96%, avg=34252.07, stdev=3689.26, samples=60 iops : min= 438, max= 862, avg=535.17, stdev=57.65, samples=60 lat (msec) : 2=0.94%, 4=2.01%, 10=1.66%, 20=9.78%, 50=62.81% lat (msec) : 100=2.62%, 250=14.40%, 500=5.38%, 750=0.08%, 1000=0.08% lat (msec) : 2000=0.13%, >=2000=0.12% cpu : usr=1.65%, sys=2.03%, ctx=9019, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64464,16188,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4029MiB (4225MB), run=30235-30235msec WRITE: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=1012MiB (1061MB), run=30235-30235msec Disk stats (read/write): vdk: ios=64459/16110, merge=0/0, ticks=67202/3572913, in_queue=3675790, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924582: Fri Jan 7 13:58:00 2022 read: IOPS=1979, BW=155MiB/s (162MB/s)(4640MiB/30001msec) clat (usec): min=40, max=27418, avg=85.48, stdev=512.12 lat (usec): min=40, max=27418, avg=85.63, stdev=512.12 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 54], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 78], 90.00th=[ 86], 95.00th=[ 94], | 99.00th=[ 127], 99.50th=[ 159], 99.90th=[ 9503], 99.95th=[15008], | 99.99th=[23462] bw ( KiB/s): min=122880, max=261440, per=100.00%, avg=158370.25, stdev=21442.97, samples=60 iops : min= 1536, max= 3268, avg=1979.62, stdev=268.04, samples=60 write: IOPS=497, BW=38.9MiB/s (40.8MB/s)(1167MiB/30001msec) clat (usec): min=408, max=94722, avg=1651.71, stdev=1846.73 lat (usec): min=415, max=94730, avg=1659.89, stdev=1846.70 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 685], 10.00th=[ 873], 20.00th=[ 1172], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2057], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[37487], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min=28480, max=67360, per=100.00%, avg=39833.37, stdev=4895.52, samples=60 iops : min= 356, max= 842, avg=497.92, stdev=61.13, samples=60 lat (usec) : 50=0.79%, 100=76.44%, 250=2.46%, 500=0.40%, 750=1.11% lat (usec) : 1000=1.31% lat (msec) : 2=15.09%, 4=1.95%, 10=0.24%, 20=0.18%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.69%, sys=3.81%, ctx=74328, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59387,14938,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4640MiB (4865MB), run=30001-30001msec WRITE: bw=38.9MiB/s (40.8MB/s), 38.9MiB/s-38.9MiB/s (40.8MB/s-40.8MB/s), io=1167MiB (1224MB), run=30001-30001msec Disk stats (read/write): vdk: ios=59089/14857, merge=0/0, ticks=922/16818, in_queue=28062, util=93.75% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925580: Fri Jan 7 13:58:37 2022 read: IOPS=1993, BW=156MiB/s (163MB/s)(4673MiB/30007msec) slat (usec): min=7, max=364, avg=14.14, stdev= 4.63 clat (nsec): min=763, max=22485k, avg=66261.58, stdev=444339.51 lat (usec): min=37, max=22512, avg=80.56, stdev=444.39 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 90], 99.50th=[ 105], 99.90th=[ 6915], 99.95th=[13698], | 99.99th=[18744] bw ( KiB/s): min=126720, max=254080, per=100.00%, avg=159506.92, stdev=18788.95, samples=60 iops : min= 1584, max= 3176, avg=1993.82, stdev=234.86, samples=60 write: IOPS=500, BW=39.1MiB/s (41.0MB/s)(1174MiB/30007msec) slat (usec): min=12, max=141, avg=22.65, stdev= 6.52 clat (usec): min=379, max=43597, avg=1646.36, stdev=1719.14 lat (usec): min=405, max=43618, avg=1669.18, stdev=1719.09 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 676], 10.00th=[ 889], 20.00th=[ 1172], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[36439], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min=33760, max=65760, per=100.00%, avg=40053.37, stdev=4592.07, samples=60 iops : min= 422, max= 822, avg=500.65, stdev=57.39, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=40.10%, 100=39.32% lat (usec) : 250=0.33%, 500=0.42%, 750=1.03%, 1000=1.15% lat (msec) : 2=15.36%, 4=1.79%, 10=0.26%, 20=0.17%, 50=0.02% cpu : usr=1.59%, sys=6.12%, ctx=74829, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59815,15021,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4673MiB (4900MB), run=30007-30007msec WRITE: bw=39.1MiB/s (41.0MB/s), 39.1MiB/s-39.1MiB/s (41.0MB/s-41.0MB/s), io=1174MiB (1231MB), run=30007-30007msec Disk stats (read/write): vdk: ios=59447/14954, merge=0/0, ticks=791/17105, in_queue=27885, util=93.17% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926122: Fri Jan 7 13:59:14 2022 read: IOPS=1998, BW=156MiB/s (164MB/s)(4685MiB/30013msec) clat (usec): min=32, max=25656, avg=72.63, stdev=458.44 lat (usec): min=32, max=25656, avg=72.77, stdev=458.44 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 99], 99.50th=[ 115], 99.90th=[10028], 99.95th=[13698], | 99.99th=[18482] bw ( KiB/s): min=125920, max=254720, per=100.00%, avg=159915.33, stdev=20042.45, samples=60 iops : min= 1574, max= 3184, avg=1998.93, stdev=250.52, samples=60 write: IOPS=501, BW=39.2MiB/s (41.1MB/s)(1177MiB/30013msec) clat (usec): min=399, max=46574, avg=1687.77, stdev=1682.72 lat (usec): min=404, max=46581, avg=1695.28, stdev=1682.66 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 709], 10.00th=[ 988], 20.00th=[ 1254], | 30.00th=[ 1369], 40.00th=[ 1434], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1827], 90.00th=[ 2089], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[18744], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=33760, max=66080, per=100.00%, avg=40164.03, stdev=4430.89, samples=60 iops : min= 422, max= 826, avg=502.05, stdev=55.39, samples=60 lat (usec) : 50=18.05%, 100=61.14%, 250=0.62%, 500=0.37%, 750=0.83% lat (usec) : 1000=0.90% lat (msec) : 2=15.54%, 4=2.10%, 10=0.26%, 20=0.17%, 50=0.02% cpu : usr=1.46%, sys=3.80%, ctx=75034, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59967,15061,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4685MiB (4912MB), run=30013-30013msec WRITE: bw=39.2MiB/s (41.1MB/s), 39.2MiB/s-39.2MiB/s (41.1MB/s-41.1MB/s), io=1177MiB (1234MB), run=30013-30013msec Disk stats (read/write): vdk: ios=59574/14972, merge=0/0, ticks=814/17520, in_queue=28213, util=94.30% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926960: Fri Jan 7 13:59:51 2022 read: IOPS=2057, BW=161MiB/s (169MB/s)(4826MiB/30027msec) slat (usec): min=4, max=256, avg=14.07, stdev= 6.31 clat (nsec): min=766, max=2492.3M, avg=1178585.91, stdev=45277650.31 lat (usec): min=33, max=2492.3k, avg=1192.84, stdev=45277.65 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 49], 60.00th=[ 52], 70.00th=[ 56], | 80.00th=[ 60], 90.00th=[ 68], 95.00th=[ 77], | 99.00th=[ 113], 99.50th=[ 149], 99.90th=[ 39584], | 99.95th=[ 968885], 99.99th=[2499806] bw ( KiB/s): min=124480, max=313760, per=100.00%, avg=164736.75, stdev=25463.51, samples=60 iops : min= 1556, max= 3922, avg=2059.20, stdev=318.29, samples=60 write: IOPS=516, BW=40.3MiB/s (42.3MB/s)(1211MiB/30027msec) slat (usec): min=8, max=353, avg=25.06, stdev=12.06 clat (usec): min=491, max=75389, avg=14582.28, stdev=6517.16 lat (usec): min=509, max=75408, avg=14607.56, stdev=6518.41 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 5407], 10.00th=[ 7898], 20.00th=[11207], | 30.00th=[12780], 40.00th=[13698], 50.00th=[14484], 60.00th=[15270], | 70.00th=[16057], 80.00th=[16909], 90.00th=[19792], 95.00th=[23725], | 99.00th=[32900], 99.50th=[64226], 99.90th=[71828], 99.95th=[73925], | 99.99th=[74974] bw ( KiB/s): min=36000, max=78880, per=100.00%, avg=41312.17, stdev=5529.57, samples=60 iops : min= 450, max= 986, avg=516.40, stdev=69.12, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.10%, 4=0.01%, 10=0.02%, 20=0.05%, 50=43.38% lat (usec) : 100=35.11%, 250=1.07%, 500=0.06%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.20%, 4=0.51%, 10=2.40%, 20=14.98%, 50=1.83% lat (msec) : 100=0.14%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=1.99%, sys=6.08%, ctx=76209, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61773,15500,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4826MiB (5060MB), run=30027-30027msec WRITE: bw=40.3MiB/s (42.3MB/s), 40.3MiB/s-40.3MiB/s (42.3MB/s-42.3MB/s), io=1211MiB (1270MB), run=30027-30027msec Disk stats (read/write): vdk: ios=61362/15411, merge=0/0, ticks=68552/216673, in_queue=296391, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927537: Fri Jan 7 14:00:28 2022 read: IOPS=1995, BW=156MiB/s (163MB/s)(4678MiB/30002msec) clat (usec): min=34, max=25487, avg=71.72, stdev=460.99 lat (usec): min=34, max=25487, avg=71.86, stdev=461.00 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 94], 99.50th=[ 106], 99.90th=[ 9372], 99.95th=[14222], | 99.99th=[17957] bw ( KiB/s): min=120160, max=245120, per=100.00%, avg=159700.55, stdev=18578.64, samples=60 iops : min= 1502, max= 3064, avg=1996.20, stdev=232.24, samples=60 write: IOPS=501, BW=39.2MiB/s (41.1MB/s)(1176MiB/30002msec) clat (usec): min=399, max=50572, avg=1692.61, stdev=1681.85 lat (usec): min=407, max=50580, avg=1700.19, stdev=1681.81 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 725], 10.00th=[ 1020], 20.00th=[ 1270], | 30.00th=[ 1369], 40.00th=[ 1434], 50.00th=[ 1516], 60.00th=[ 1598], | 70.00th=[ 1696], 80.00th=[ 1827], 90.00th=[ 2073], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[18744], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min=32800, max=64000, per=100.00%, avg=40132.03, stdev=4346.51, samples=60 iops : min= 410, max= 800, avg=501.60, stdev=54.35, samples=60 lat (usec) : 50=21.33%, 100=58.05%, 250=0.43%, 500=0.34%, 750=0.78% lat (usec) : 1000=0.84% lat (msec) : 2=15.67%, 4=2.13%, 10=0.23%, 20=0.17%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.44%, sys=3.70%, ctx=74932, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59874,15047,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4678MiB (4905MB), run=30002-30002msec WRITE: bw=39.2MiB/s (41.1MB/s), 39.2MiB/s-39.2MiB/s (41.1MB/s-41.1MB/s), io=1176MiB (1233MB), run=30002-30002msec Disk stats (read/write): vdk: ios=59411/14946, merge=0/0, ticks=826/17564, in_queue=28180, util=94.44% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928859: Fri Jan 7 14:01:05 2022 read: IOPS=2035, BW=159MiB/s (167MB/s)(4788MiB/30112msec) slat (usec): min=2, max=164, avg=11.15, stdev= 4.59 clat (nsec): min=776, max=2516.0M, avg=1177122.38, stdev=44537019.96 lat (usec): min=29, max=2516.0k, avg=1188.42, stdev=44537.01 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 46], 70.00th=[ 52], | 80.00th=[ 58], 90.00th=[ 66], 95.00th=[ 76], | 99.00th=[ 289], 99.50th=[ 603], 99.90th=[ 1631], | 99.95th=[ 977273], 99.99th=[2332034] bw ( KiB/s): min=128800, max=300160, per=100.00%, avg=163446.32, stdev=24299.24, samples=60 iops : min= 1610, max= 3752, avg=2043.05, stdev=303.73, samples=60 write: IOPS=511, BW=39.9MiB/s (41.9MB/s)(1203MiB/30112msec) slat (usec): min=8, max=223, avg=23.27, stdev= 9.76 clat (msec): min=4, max=2644, avg=93.03, stdev=81.70 lat (msec): min=4, max=2644, avg=93.06, stdev=81.70 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 83], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 92], 80.00th=[ 96], 90.00th=[ 112], 95.00th=[ 132], | 99.00th=[ 257], 99.50th=[ 575], 99.90th=[ 1284], 99.95th=[ 1770], | 99.99th=[ 2265] bw ( KiB/s): min=35040, max=69280, per=100.00%, avg=40932.02, stdev=4574.41, samples=60 iops : min= 438, max= 866, avg=511.62, stdev=57.20, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.03%, 20=0.04%, 50=54.23% lat (usec) : 100=23.66%, 250=0.98%, 500=0.45%, 750=0.15%, 1000=0.08% lat (msec) : 2=0.16%, 10=0.03%, 20=0.31%, 50=0.15%, 100=16.27% lat (msec) : 250=3.13%, 500=0.09%, 750=0.05%, 1000=0.03%, 2000=0.06% lat (msec) : >=2000=0.02% cpu : usr=1.78%, sys=5.15%, ctx=74420, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=61284,15394,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4788MiB (5020MB), run=30112-30112msec WRITE: bw=39.9MiB/s (41.9MB/s), 39.9MiB/s-39.9MiB/s (41.9MB/s-41.9MB/s), io=1203MiB (1261MB), run=30112-30112msec Disk stats (read/write): vdk: ios=61282/15382, merge=0/0, ticks=66833/1420993, in_queue=1501326, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929906: Fri Jan 7 14:01:42 2022 read: IOPS=1993, BW=156MiB/s (163MB/s)(4673MiB/30001msec) clat (usec): min=33, max=63144, avg=74.70, stdev=534.70 lat (usec): min=33, max=63144, avg=74.84, stdev=534.70 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 51], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 98], 99.50th=[ 113], 99.90th=[ 7898], 99.95th=[14746], | 99.99th=[19268] bw ( KiB/s): min=126400, max=254720, per=100.00%, avg=159506.67, stdev=19548.44, samples=60 iops : min= 1580, max= 3184, avg=1993.83, stdev=244.36, samples=60 write: IOPS=500, BW=39.1MiB/s (41.0MB/s)(1174MiB/30001msec) clat (usec): min=399, max=44891, avg=1683.36, stdev=1684.04 lat (usec): min=407, max=44898, avg=1691.21, stdev=1683.97 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 701], 10.00th=[ 988], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[18220], 99.95th=[42206], | 99.99th=[43779] bw ( KiB/s): min=31840, max=66080, per=100.00%, avg=40053.33, stdev=4490.22, samples=60 iops : min= 398, max= 826, avg=500.67, stdev=56.13, samples=60 lat (usec) : 50=14.74%, 100=64.46%, 250=0.60%, 500=0.36%, 750=0.88% lat (usec) : 1000=0.87% lat (msec) : 2=15.65%, 4=1.97%, 10=0.26%, 20=0.17%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.52%, sys=3.88%, ctx=74842, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59815,15021,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4673MiB (4900MB), run=30001-30001msec WRITE: bw=39.1MiB/s (41.0MB/s), 39.1MiB/s-39.1MiB/s (41.0MB/s-41.0MB/s), io=1174MiB (1231MB), run=30001-30001msec Disk stats (read/write): vdk: ios=59297/14910, merge=0/0, ticks=893/17384, in_queue=28109, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930580: Fri Jan 7 14:02:20 2022 read: IOPS=2032, BW=159MiB/s (166MB/s)(4797MiB/30213msec) slat (usec): min=2, max=863, avg=11.27, stdev= 8.86 clat (nsec): min=711, max=2545.1M, avg=1044693.07, stdev=38904373.12 lat (usec): min=26, max=2545.1k, avg=1056.11, stdev=38904.37 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 44], 60.00th=[ 48], 70.00th=[ 55], | 80.00th=[ 62], 90.00th=[ 76], 95.00th=[ 169], | 99.00th=[ 1958], 99.50th=[ 2835], 99.90th=[ 10552], | 99.95th=[ 683672], 99.99th=[2264925] bw ( KiB/s): min=117920, max=349440, per=100.00%, avg=163746.83, stdev=28689.29, samples=60 iops : min= 1474, max= 4368, avg=2046.82, stdev=358.62, samples=60 write: IOPS=510, BW=39.9MiB/s (41.8MB/s)(1205MiB/30213msec) slat (usec): min=7, max=744, avg=25.12, stdev=16.88 clat (msec): min=12, max=3205, avg=191.55, stdev=146.95 lat (msec): min=12, max=3205, avg=191.57, stdev=146.95 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 130], 10.00th=[ 142], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 176], 50.00th=[ 182], 60.00th=[ 188], | 70.00th=[ 192], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 253], | 99.00th=[ 489], 99.50th=[ 835], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min=33440, max=74240, per=100.00%, avg=40892.05, stdev=5065.28, samples=60 iops : min= 418, max= 928, avg=511.13, stdev=63.32, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.06%, 50=50.48% lat (usec) : 100=23.94%, 250=1.94%, 500=0.86%, 750=0.61%, 1000=0.40% lat (msec) : 2=0.70%, 4=0.69%, 10=0.02%, 20=0.16%, 50=0.28% lat (msec) : 100=0.21%, 250=18.35%, 500=0.91%, 750=0.10%, 1000=0.04% lat (msec) : 2000=0.07%, >=2000=0.04% cpu : usr=1.87%, sys=5.15%, ctx=71348, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=61402,15430,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=4797MiB (5030MB), run=30213-30213msec WRITE: bw=39.9MiB/s (41.8MB/s), 39.9MiB/s-39.9MiB/s (41.8MB/s-41.8MB/s), io=1205MiB (1264MB), run=30213-30213msec Disk stats (read/write): vdk: ios=61399/15366, merge=0/0, ticks=56517/2921947, in_queue=3003363, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931407: Fri Jan 7 14:02:56 2022 read: IOPS=2008, BW=157MiB/s (165MB/s)(4708MiB/30001msec) clat (usec): min=32, max=26614, avg=70.84, stdev=486.17 lat (usec): min=32, max=26614, avg=70.98, stdev=486.17 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 93], 99.50th=[ 108], 99.90th=[ 8094], 99.95th=[14615], | 99.99th=[18220] bw ( KiB/s): min=115520, max=244160, per=100.00%, avg=160706.72, stdev=20102.69, samples=60 iops : min= 1444, max= 3052, avg=2008.82, stdev=251.28, samples=60 write: IOPS=504, BW=39.4MiB/s (41.3MB/s)(1183MiB/30001msec) clat (usec): min=397, max=52803, avg=1683.74, stdev=1714.84 lat (usec): min=405, max=52811, avg=1691.19, stdev=1714.78 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 725], 10.00th=[ 1029], 20.00th=[ 1270], | 30.00th=[ 1352], 40.00th=[ 1434], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[36439], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min=32800, max=63840, per=100.00%, avg=40378.85, stdev=4294.20, samples=60 iops : min= 410, max= 798, avg=504.73, stdev=53.68, samples=60 lat (usec) : 50=29.86%, 100=49.51%, 250=0.43%, 500=0.38%, 750=0.74% lat (usec) : 1000=0.79% lat (msec) : 2=15.88%, 4=1.98%, 10=0.24%, 20=0.16%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.52%, sys=3.51%, ctx=75410, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60261,15142,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=157MiB/s (165MB/s), 157MiB/s-157MiB/s (165MB/s-165MB/s), io=4708MiB (4937MB), run=30001-30001msec WRITE: bw=39.4MiB/s (41.3MB/s), 39.4MiB/s-39.4MiB/s (41.3MB/s-41.3MB/s), io=1183MiB (1240MB), run=30001-30001msec Disk stats (read/write): vdk: ios=59856/15039, merge=0/0, ticks=871/17650, in_queue=28280, util=94.54% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932349: Fri Jan 7 14:03:34 2022 read: IOPS=2049, BW=160MiB/s (168MB/s)(4844MiB/30262msec) slat (usec): min=2, max=258275, avg=370.07, stdev=2128.34 clat (usec): min=1141, max=2569.7k, avg=31386.37, stdev=46746.36 lat (usec): min=1144, max=2569.7k, avg=31756.58, stdev=46814.84 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 32], | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 42], 95.00th=[ 48], | 99.00th=[ 88], 99.50th=[ 243], 99.90th=[ 292], 99.95th=[ 810], | 99.99th=[ 2534] bw ( KiB/s): min=121760, max=367360, per=100.00%, avg=165186.67, stdev=30300.23, samples=60 iops : min= 1522, max= 4592, avg=2064.83, stdev=378.75, samples=60 write: IOPS=514, BW=40.2MiB/s (42.1MB/s)(1216MiB/30262msec) slat (usec): min=7, max=261338, avg=442.84, stdev=3641.32 clat (msec): min=7, max=3442, avg=261.74, stdev=153.01 lat (msec): min=10, max=3442, avg=262.18, stdev=152.97 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 182], 10.00th=[ 199], 20.00th=[ 222], | 30.00th=[ 232], 40.00th=[ 243], 50.00th=[ 249], 60.00th=[ 257], | 70.00th=[ 264], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 347], | 99.00th=[ 718], 99.50th=[ 1250], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 3440] bw ( KiB/s): min=34240, max=75680, per=99.99%, avg=41154.67, stdev=5352.28, samples=60 iops : min= 428, max= 946, avg=514.43, stdev=66.90, samples=60 lat (msec) : 2=0.40%, 4=2.59%, 10=1.94%, 20=8.16%, 50=64.23% lat (msec) : 100=2.57%, 250=10.10%, 500=9.61%, 750=0.16%, 1000=0.07% lat (msec) : 2000=0.10%, >=2000=0.06% cpu : usr=1.78%, sys=2.00%, ctx=8604, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62009,15569,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=4844MiB (5080MB), run=30262-30262msec WRITE: bw=40.2MiB/s (42.1MB/s), 40.2MiB/s-40.2MiB/s (42.1MB/s-42.1MB/s), io=1216MiB (1275MB), run=30262-30262msec Disk stats (read/write): vdk: ios=62006/15487, merge=0/0, ticks=71720/3569585, in_queue=3675913, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932839: Fri Jan 7 14:04:10 2022 read: IOPS=1789, BW=224MiB/s (235MB/s)(6710MiB/30001msec) clat (usec): min=46, max=24627, avg=93.24, stdev=491.40 lat (usec): min=46, max=24627, avg=93.39, stdev=491.40 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 62], 20.00th=[ 65], | 30.00th=[ 67], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 97], 95.00th=[ 106], | 99.00th=[ 135], 99.50th=[ 159], 99.90th=[ 9896], 99.95th=[15139], | 99.99th=[18744] bw ( KiB/s): min=128000, max=370176, per=100.00%, avg=229095.97, stdev=30570.21, samples=59 iops : min= 1000, max= 2892, avg=1789.80, stdev=238.83, samples=59 write: IOPS=449, BW=56.2MiB/s (58.9MB/s)(1686MiB/30001msec) clat (usec): min=523, max=156275, avg=1830.96, stdev=2171.18 lat (usec): min=535, max=156288, avg=1843.63, stdev=2171.21 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 914], 10.00th=[ 1074], 20.00th=[ 1254], | 30.00th=[ 1401], 40.00th=[ 1516], 50.00th=[ 1614], 60.00th=[ 1713], | 70.00th=[ 1827], 80.00th=[ 1958], 90.00th=[ 2212], 95.00th=[ 2573], | 99.00th=[ 9634], 99.50th=[10290], 99.90th=[36439], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=34048, max=96000, per=100.00%, avg=57608.05, stdev=6969.71, samples=59 iops : min= 266, max= 750, avg=450.03, stdev=54.46, samples=59 lat (usec) : 50=0.03%, 100=73.49%, 250=6.21%, 500=0.06%, 750=0.52% lat (usec) : 1000=0.92% lat (msec) : 2=15.12%, 4=3.03%, 10=0.39%, 20=0.20%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=1.74%, sys=3.67%, ctx=67180, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53682,13491,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=224MiB/s (235MB/s), 224MiB/s-224MiB/s (235MB/s-235MB/s), io=6710MiB (7036MB), run=30001-30001msec WRITE: bw=56.2MiB/s (58.9MB/s), 56.2MiB/s-56.2MiB/s (58.9MB/s-58.9MB/s), io=1686MiB (1768MB), run=30001-30001msec Disk stats (read/write): vdk: ios=53478/13428, merge=0/0, ticks=816/17333, in_queue=28156, util=94.00% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933834: Fri Jan 7 14:04:47 2022 read: IOPS=1788, BW=224MiB/s (234MB/s)(6706MiB/30002msec) slat (usec): min=7, max=119, avg=14.03, stdev= 4.54 clat (nsec): min=798, max=26793k, avg=65507.62, stdev=525069.03 lat (usec): min=36, max=26802, avg=79.69, stdev=525.12 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 82], 99.50th=[ 91], 99.90th=[10814], 99.95th=[15664], | 99.99th=[20841] bw ( KiB/s): min=65280, max=354816, per=100.00%, avg=228885.33, stdev=34367.00, samples=60 iops : min= 510, max= 2772, avg=1788.17, stdev=268.49, samples=60 write: IOPS=449, BW=56.2MiB/s (58.9MB/s)(1685MiB/30002msec) slat (usec): min=14, max=174, avg=28.52, stdev=10.09 clat (usec): min=509, max=259600, avg=1872.17, stdev=2882.91 lat (usec): min=532, max=259626, avg=1900.87, stdev=2882.95 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 783], 10.00th=[ 1029], 20.00th=[ 1369], | 30.00th=[ 1483], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2638], | 99.00th=[10028], 99.50th=[10421], 99.90th=[37487], 99.95th=[42730], | 99.99th=[69731] bw ( KiB/s): min=13824, max=91392, per=100.00%, avg=57514.67, stdev=8295.84, samples=60 iops : min= 108, max= 714, avg=449.33, stdev=64.81, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.04%, 50=51.01%, 100=28.58% lat (usec) : 250=0.14%, 500=0.01%, 750=0.86%, 1000=1.08% lat (msec) : 2=13.90%, 4=3.83%, 10=0.23%, 20=0.24%, 50=0.04% lat (msec) : 100=0.01%, 500=0.01% cpu : usr=1.77%, sys=5.14%, ctx=67111, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53645,13481,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=224MiB/s (234MB/s), 224MiB/s-224MiB/s (234MB/s-234MB/s), io=6706MiB (7031MB), run=30002-30002msec WRITE: bw=56.2MiB/s (58.9MB/s), 56.2MiB/s-56.2MiB/s (58.9MB/s-58.9MB/s), io=1685MiB (1767MB), run=30002-30002msec Disk stats (read/write): vdk: ios=53406/13411, merge=0/0, ticks=871/17669, in_queue=28118, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934417: Fri Jan 7 14:05:24 2022 read: IOPS=1801, BW=225MiB/s (236MB/s)(6757MiB/30001msec) clat (usec): min=38, max=27011, avg=80.34, stdev=537.52 lat (usec): min=38, max=27011, avg=80.48, stdev=537.52 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 105], 99.50th=[ 121], 99.90th=[11469], 99.95th=[15270], | 99.99th=[20317] bw ( KiB/s): min=164864, max=367616, per=100.00%, avg=230629.07, stdev=29531.38, samples=60 iops : min= 1288, max= 2872, avg=1801.78, stdev=230.72, samples=60 write: IOPS=452, BW=56.6MiB/s (59.3MB/s)(1697MiB/30001msec) clat (usec): min=522, max=51795, avg=1869.04, stdev=1723.64 lat (usec): min=540, max=51808, avg=1881.29, stdev=1723.48 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 816], 10.00th=[ 1090], 20.00th=[ 1401], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1680], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2040], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[10028], 99.50th=[10421], 99.90th=[21365], 99.95th=[41157], | 99.99th=[44303] bw ( KiB/s): min=47360, max=94976, per=100.00%, avg=57934.60, stdev=6491.58, samples=60 iops : min= 370, max= 742, avg=452.60, stdev=50.72, samples=60 lat (usec) : 50=3.11%, 100=75.76%, 250=0.92%, 500=0.02%, 750=0.75% lat (usec) : 1000=0.95% lat (msec) : 2=14.02%, 4=3.96%, 10=0.21%, 20=0.27%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.66%, sys=3.34%, ctx=67639, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54052,13579,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6757MiB (7085MB), run=30001-30001msec WRITE: bw=56.6MiB/s (59.3MB/s), 56.6MiB/s-56.6MiB/s (59.3MB/s-59.3MB/s), io=1697MiB (1780MB), run=30001-30001msec Disk stats (read/write): vdk: ios=53801/13508, merge=0/0, ticks=892/17544, in_queue=28269, util=94.50% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935720: Fri Jan 7 14:06:01 2022 read: IOPS=1832, BW=229MiB/s (240MB/s)(6880MiB/30043msec) slat (usec): min=5, max=230, avg=13.69, stdev= 4.47 clat (nsec): min=788, max=2531.3M, avg=1549395.59, stdev=54822080.89 lat (usec): min=36, max=2531.3k, avg=1563.24, stdev=54822.08 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 41], 30.00th=[ 43], 40.00th=[ 44], | 50.00th=[ 46], 60.00th=[ 49], 70.00th=[ 53], | 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 76], | 99.00th=[ 98], 99.50th=[ 125], 99.90th=[ 77071], | 99.95th=[1535116], 99.99th=[2499806] bw ( KiB/s): min=184832, max=462848, per=100.00%, avg=234831.27, stdev=36595.44, samples=60 iops : min= 1444, max= 3616, avg=1834.60, stdev=285.90, samples=60 write: IOPS=460, BW=57.5MiB/s (60.3MB/s)(1729MiB/30043msec) slat (usec): min=13, max=208, avg=29.05, stdev= 9.33 clat (usec): min=1422, max=157668, avg=15462.81, stdev=7191.98 lat (usec): min=1456, max=157695, avg=15492.03, stdev=7192.64 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 6], 10.00th=[ 11], 20.00th=[ 13], | 30.00th=[ 14], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 16], | 70.00th=[ 17], 80.00th=[ 18], 90.00th=[ 22], 95.00th=[ 25], | 99.00th=[ 37], 99.50th=[ 67], 99.90th=[ 79], 99.95th=[ 96], | 99.99th=[ 157] bw ( KiB/s): min=45824, max=116480, per=100.00%, avg=58986.32, stdev=8703.55, samples=60 iops : min= 358, max= 910, avg=460.80, stdev=68.00, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=51.60% lat (usec) : 100=27.51%, 250=0.54%, 500=0.08%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.76%, 10=1.21%, 20=15.79%, 50=2.17% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.03% cpu : usr=1.85%, sys=5.18%, ctx=68261, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55039,13829,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6880MiB (7214MB), run=30043-30043msec WRITE: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=1729MiB (1813MB), run=30043-30043msec Disk stats (read/write): vdk: ios=54726/13749, merge=0/0, ticks=79568/205721, in_queue=296777, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936663: Fri Jan 7 14:06:38 2022 read: IOPS=1790, BW=224MiB/s (235MB/s)(6715MiB/30001msec) clat (usec): min=39, max=27983, avg=80.61, stdev=495.69 lat (usec): min=39, max=27983, avg=80.75, stdev=495.69 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 56], | 30.00th=[ 58], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 86], | 99.00th=[ 103], 99.50th=[ 117], 99.90th=[11076], 99.95th=[15139], | 99.99th=[18220] bw ( KiB/s): min=129024, max=356096, per=100.00%, avg=229208.83, stdev=29251.55, samples=60 iops : min= 1008, max= 2782, avg=1790.68, stdev=228.53, samples=60 write: IOPS=449, BW=56.2MiB/s (58.0MB/s)(1687MiB/30001msec) clat (usec): min=516, max=144501, avg=1880.20, stdev=2134.74 lat (usec): min=528, max=144534, avg=1893.30, stdev=2134.79 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 807], 10.00th=[ 1106], 20.00th=[ 1401], | 30.00th=[ 1516], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9896], 99.50th=[10421], 99.90th=[36439], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min=31232, max=92160, per=100.00%, avg=57589.28, stdev=7078.23, samples=60 iops : min= 244, max= 720, avg=449.90, stdev=55.29, samples=60 lat (usec) : 50=2.22%, 100=76.72%, 250=0.87%, 500=0.02%, 750=0.75% lat (usec) : 1000=0.93% lat (msec) : 2=14.11%, 4=3.86%, 10=0.26%, 20=0.24%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.73%, sys=3.48%, ctx=67229, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53719,13498,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=224MiB/s (235MB/s), 224MiB/s-224MiB/s (235MB/s-235MB/s), io=6715MiB (7041MB), run=30001-30001msec WRITE: bw=56.2MiB/s (58.0MB/s), 56.2MiB/s-56.2MiB/s (58.0MB/s-58.0MB/s), io=1687MiB (1769MB), run=30001-30001msec Disk stats (read/write): vdk: ios=53431/13415, merge=0/0, ticks=832/17624, in_queue=28169, util=94.11% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937335: Fri Jan 7 14:07:16 2022 read: IOPS=1838, BW=230MiB/s (241MB/s)(6923MiB/30122msec) slat (usec): min=3, max=148, avg=12.80, stdev= 4.92 clat (nsec): min=838, max=2538.8M, avg=1492043.07, stdev=52974059.56 lat (usec): min=35, max=2538.8k, avg=1505.01, stdev=52974.07 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 49], 60.00th=[ 53], 70.00th=[ 61], | 80.00th=[ 74], 90.00th=[ 84], 95.00th=[ 97], | 99.00th=[ 445], 99.50th=[ 1029], 99.90th=[ 141558], | 99.95th=[1249903], 99.99th=[2533360] bw ( KiB/s): min=168192, max=465152, per=100.00%, avg=236308.47, stdev=36845.59, samples=60 iops : min= 1314, max= 3634, avg=1846.13, stdev=287.87, samples=60 write: IOPS=462, BW=57.8MiB/s (60.6MB/s)(1740MiB/30122msec) slat (usec): min=11, max=245, avg=33.68, stdev=14.44 clat (msec): min=11, max=2352, avg=102.14, stdev=61.23 lat (msec): min=11, max=2352, avg=102.17, stdev=61.23 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 79], 10.00th=[ 84], 20.00th=[ 89], | 30.00th=[ 91], 40.00th=[ 94], 50.00th=[ 96], 60.00th=[ 99], | 70.00th=[ 103], 80.00th=[ 107], 90.00th=[ 118], 95.00th=[ 140], | 99.00th=[ 230], 99.50th=[ 464], 99.90th=[ 969], 99.95th=[ 1183], | 99.99th=[ 1921] bw ( KiB/s): min=49152, max=108032, per=100.00%, avg=59211.72, stdev=7291.87, samples=60 iops : min= 384, max= 844, avg=462.53, stdev=56.99, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=42.59% lat (usec) : 100=33.74%, 250=2.30%, 500=0.50%, 750=0.23%, 1000=0.09% lat (msec) : 2=0.25%, 4=0.07%, 20=0.32%, 50=0.18%, 100=12.42% lat (msec) : 250=7.01%, 500=0.10%, 750=0.06%, 1000=0.03%, 2000=0.03% lat (msec) : >=2000=0.03% cpu : usr=2.07%, sys=5.08%, ctx=67100, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=55381,13922,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6923MiB (7259MB), run=30122-30122msec WRITE: bw=57.8MiB/s (60.6MB/s), 57.8MiB/s-57.8MiB/s (60.6MB/s-60.6MB/s), io=1740MiB (1825MB), run=30122-30122msec Disk stats (read/write): vdk: ios=55378/13922, merge=0/0, ticks=74905/1412943, in_queue=1501992, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938223: Fri Jan 7 14:07:52 2022 read: IOPS=1789, BW=224MiB/s (235MB/s)(6710MiB/30001msec) clat (usec): min=38, max=65073, avg=79.55, stdev=604.60 lat (usec): min=38, max=65074, avg=79.69, stdev=604.60 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 81], | 99.00th=[ 96], 99.50th=[ 105], 99.90th=[11731], 99.95th=[16581], | 99.99th=[20317] bw ( KiB/s): min=160768, max=363008, per=100.00%, avg=229055.77, stdev=29863.56, samples=60 iops : min= 1256, max= 2836, avg=1789.48, stdev=233.30, samples=60 write: IOPS=449, BW=56.2MiB/s (58.9MB/s)(1687MiB/30001msec) clat (usec): min=511, max=118748, avg=1886.76, stdev=1964.25 lat (usec): min=523, max=118757, avg=1898.85, stdev=1964.13 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 824], 10.00th=[ 1156], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1680], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2040], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[36963], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=47360, max=93696, per=100.00%, avg=57563.47, stdev=6441.05, samples=60 iops : min= 370, max= 732, avg=449.70, stdev=50.32, samples=60 lat (usec) : 50=4.66%, 100=74.69%, 250=0.47%, 500=0.01%, 750=0.74% lat (usec) : 1000=0.74% lat (msec) : 2=14.22%, 4=3.98%, 10=0.23%, 20=0.24%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.62%, sys=3.26%, ctx=67183, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53683,13492,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=224MiB/s (235MB/s), 224MiB/s-224MiB/s (235MB/s-235MB/s), io=6710MiB (7036MB), run=30001-30001msec WRITE: bw=56.2MiB/s (58.9MB/s), 56.2MiB/s-56.2MiB/s (58.9MB/s-58.9MB/s), io=1687MiB (1768MB), run=30001-30001msec Disk stats (read/write): vdk: ios=53397/13409, merge=0/0, ticks=960/17659, in_queue=28286, util=94.54% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938825: Fri Jan 7 14:08:30 2022 read: IOPS=1840, BW=230MiB/s (241MB/s)(6958MiB/30237msec) slat (usec): min=3, max=268, avg=12.00, stdev= 5.69 clat (nsec): min=794, max=2534.8M, avg=1643297.74, stdev=55597859.93 lat (usec): min=34, max=2534.8k, avg=1655.46, stdev=55597.83 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 44], 40.00th=[ 46], | 50.00th=[ 49], 60.00th=[ 53], 70.00th=[ 63], | 80.00th=[ 76], 90.00th=[ 89], 95.00th=[ 174], | 99.00th=[ 2212], 99.50th=[ 3884], 99.90th=[ 162530], | 99.95th=[1484784], 99.99th=[2533360] bw ( KiB/s): min=172032, max=544000, per=100.00%, avg=237498.55, stdev=45588.31, samples=60 iops : min= 1344, max= 4250, avg=1855.43, stdev=356.18, samples=60 write: IOPS=462, BW=57.8MiB/s (60.6MB/s)(1749MiB/30237msec) slat (usec): min=10, max=349, avg=33.62, stdev=15.46 clat (msec): min=4, max=2509, avg=209.50, stdev=88.66 lat (msec): min=4, max=2509, avg=209.53, stdev=88.66 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 194], 40.00th=[ 199], 50.00th=[ 203], 60.00th=[ 207], | 70.00th=[ 213], 80.00th=[ 224], 90.00th=[ 245], 95.00th=[ 268], | 99.00th=[ 451], 99.50th=[ 743], 99.90th=[ 1351], 99.95th=[ 1804], | 99.99th=[ 2265] bw ( KiB/s): min=47104, max=113664, per=100.00%, avg=59282.28, stdev=7848.91, samples=60 iops : min= 368, max= 888, avg=463.10, stdev=61.34, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.03%, 50=43.20% lat (usec) : 100=30.73%, 250=2.46%, 500=0.93%, 750=0.58%, 1000=0.40% lat (msec) : 2=0.64%, 4=0.49%, 10=0.29%, 20=0.07%, 50=0.35% lat (msec) : 100=0.25%, 250=17.74%, 500=1.52%, 750=0.08%, 1000=0.04% lat (msec) : 2000=0.08%, >=2000=0.04% cpu : usr=2.25%, sys=4.70%, ctx=65071, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=55663,13989,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6958MiB (7296MB), run=30237-30237msec WRITE: bw=57.8MiB/s (60.6MB/s), 57.8MiB/s-57.8MiB/s (60.6MB/s-60.6MB/s), io=1749MiB (1834MB), run=30237-30237msec Disk stats (read/write): vdk: ios=55659/13916, merge=0/0, ticks=81668/2893787, in_queue=3002163, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939643: Fri Jan 7 14:09:06 2022 read: IOPS=1797, BW=225MiB/s (236MB/s)(6739MiB/30002msec) clat (usec): min=38, max=34787, avg=78.01, stdev=502.25 lat (usec): min=38, max=34787, avg=78.15, stdev=502.25 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 99], 99.50th=[ 112], 99.90th=[11338], 99.95th=[14615], | 99.99th=[18220] bw ( KiB/s): min=171264, max=388096, per=100.00%, avg=230041.75, stdev=33113.35, samples=60 iops : min= 1338, max= 3032, avg=1797.20, stdev=258.70, samples=60 write: IOPS=451, BW=56.4MiB/s (59.2MB/s)(1693MiB/30002msec) clat (usec): min=522, max=59063, avg=1883.85, stdev=1828.77 lat (usec): min=534, max=59075, avg=1895.88, stdev=1828.66 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 816], 10.00th=[ 1139], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1680], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[10028], 99.50th=[10290], 99.90th=[36963], 99.95th=[41157], | 99.99th=[52167] bw ( KiB/s): min=42240, max=101120, per=100.00%, avg=57794.00, stdev=7150.09, samples=60 iops : min= 330, max= 790, avg=451.50, stdev=55.86, samples=60 lat (usec) : 50=3.98%, 100=75.18%, 250=0.64%, 500=0.02%, 750=0.76% lat (usec) : 1000=0.80% lat (msec) : 2=14.15%, 4=3.95%, 10=0.24%, 20=0.26%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.68%, sys=3.27%, ctx=67464, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53914,13546,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6739MiB (7067MB), run=30002-30002msec WRITE: bw=56.4MiB/s (59.2MB/s), 56.4MiB/s-56.4MiB/s (59.2MB/s-59.2MB/s), io=1693MiB (1776MB), run=30002-30002msec Disk stats (read/write): vdk: ios=53628/13474, merge=0/0, ticks=841/17719, in_queue=28261, util=94.34% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940611: Fri Jan 7 14:09:44 2022 read: IOPS=1822, BW=228MiB/s (239MB/s)(6902MiB/30301msec) slat (usec): min=3, max=273864, avg=422.16, stdev=2681.20 clat (usec): min=1432, max=2829.3k, avg=35783.34, stdev=58983.93 lat (usec): min=1436, max=2829.3k, avg=36205.66, stdev=59054.60 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 35], | 70.00th=[ 38], 80.00th=[ 42], 90.00th=[ 48], 95.00th=[ 55], | 99.00th=[ 108], 99.50th=[ 279], 99.90th=[ 321], 99.95th=[ 1385], | 99.99th=[ 2534] bw ( KiB/s): min=153600, max=504304, per=100.00%, avg=235313.35, stdev=44460.53, samples=60 iops : min= 1200, max= 3939, avg=1838.35, stdev=347.25, samples=60 write: IOPS=457, BW=57.2MiB/s (60.0MB/s)(1734MiB/30301msec) slat (usec): min=14, max=269655, avg=472.15, stdev=3641.86 clat (msec): min=31, max=2580, avg=292.23, stdev=132.05 lat (msec): min=31, max=2580, avg=292.70, stdev=132.03 clat percentiles (msec): | 1.00th=[ 42], 5.00th=[ 163], 10.00th=[ 215], 20.00th=[ 259], | 30.00th=[ 268], 40.00th=[ 275], 50.00th=[ 284], 60.00th=[ 288], | 70.00th=[ 296], 80.00th=[ 313], 90.00th=[ 342], 95.00th=[ 405], | 99.00th=[ 852], 99.50th=[ 1368], 99.90th=[ 1720], 99.95th=[ 1787], | 99.99th=[ 2265] bw ( KiB/s): min=39680, max=98244, per=100.00%, avg=58626.87, stdev=7091.66, samples=60 iops : min= 310, max= 767, avg=457.98, stdev=55.36, samples=60 lat (msec) : 2=0.12%, 4=1.31%, 10=3.70%, 20=4.64%, 50=64.15% lat (msec) : 100=5.57%, 250=3.02%, 500=16.83%, 750=0.37%, 1000=0.09% lat (msec) : 2000=0.15%, >=2000=0.03% cpu : usr=2.38%, sys=1.87%, ctx=7700, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=55212,13871,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6902MiB (7237MB), run=30301-30301msec WRITE: bw=57.2MiB/s (60.0MB/s), 57.2MiB/s-57.2MiB/s (60.0MB/s-60.0MB/s), io=1734MiB (1818MB), run=30301-30301msec Disk stats (read/write): vdk: ios=55210/13871, merge=0/0, ticks=92027/3573763, in_queue=3691892, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941229: Fri Jan 7 14:10:20 2022 read: IOPS=1180, BW=295MiB/s (309MB/s)(8851MiB/30003msec) clat (usec): min=69, max=22555, avg=117.64, stdev=348.67 lat (usec): min=69, max=22556, avg=117.80, stdev=348.68 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 83], 10.00th=[ 85], 20.00th=[ 89], | 30.00th=[ 92], 40.00th=[ 96], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 112], 80.00th=[ 120], 90.00th=[ 133], 95.00th=[ 145], | 99.00th=[ 188], 99.50th=[ 223], 99.90th=[ 5997], 99.95th=[ 9896], | 99.99th=[12518] bw ( KiB/s): min=217600, max=548864, per=100.00%, avg=302112.95, stdev=48057.12, samples=60 iops : min= 850, max= 2144, avg=1180.08, stdev=187.71, samples=60 write: IOPS=296, BW=74.2MiB/s (77.8MB/s)(2226MiB/30003msec) clat (usec): min=869, max=55496, avg=2864.59, stdev=2763.42 lat (usec): min=891, max=55519, avg=2891.94, stdev=2763.21 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1106], 10.00th=[ 1287], 20.00th=[ 1614], | 30.00th=[ 1876], 40.00th=[ 2024], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2540], 80.00th=[ 2966], 90.00th=[ 4817], 95.00th=[ 9241], | 99.00th=[12649], 99.50th=[13173], 99.90th=[46924], 99.95th=[54264], | 99.99th=[55313] bw ( KiB/s): min=55296, max=145920, per=100.00%, avg=75969.58, stdev=12540.12, samples=60 iops : min= 216, max= 570, avg=296.75, stdev=48.98, samples=60 lat (usec) : 100=39.24%, 250=40.37%, 500=0.12%, 750=0.04%, 1000=0.48% lat (msec) : 2=7.35%, 4=9.00%, 10=2.55%, 20=0.82%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.66%, sys=2.74%, ctx=44316, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=35405,8904,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=8851MiB (9281MB), run=30003-30003msec WRITE: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=2226MiB (2334MB), run=30003-30003msec Disk stats (read/write): vdk: ios=35262/8862, merge=0/0, ticks=400/20829, in_queue=28402, util=94.90% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942442: Fri Jan 7 14:10:57 2022 read: IOPS=1350, BW=338MiB/s (354MB/s)(9.89GiB/30001msec) slat (usec): min=12, max=161, avg=19.29, stdev= 5.19 clat (nsec): min=1371, max=67554k, avg=90027.98, stdev=610501.14 lat (usec): min=58, max=67570, avg=109.48, stdev=610.55 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 58], 10.00th=[ 59], 20.00th=[ 61], | 30.00th=[ 63], 40.00th=[ 65], 50.00th=[ 68], 60.00th=[ 72], | 70.00th=[ 76], 80.00th=[ 81], 90.00th=[ 88], 95.00th=[ 95], | 99.00th=[ 114], 99.50th=[ 128], 99.90th=[ 8979], 99.95th=[15401], | 99.99th=[20579] bw ( KiB/s): min=245248, max=559104, per=100.00%, avg=345748.28, stdev=49355.09, samples=60 iops : min= 958, max= 2184, avg=1350.57, stdev=192.79, samples=60 write: IOPS=340, BW=85.2MiB/s (89.4MB/s)(2557MiB/30001msec) slat (usec): min=23, max=118, avg=43.19, stdev= 7.76 clat (usec): min=774, max=122499, avg=2448.06, stdev=2628.20 lat (usec): min=813, max=122538, avg=2491.44, stdev=2628.00 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1614], | 30.00th=[ 1860], 40.00th=[ 1975], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2376], 80.00th=[ 2606], 90.00th=[ 3032], 95.00th=[ 4424], | 99.00th=[10683], 99.50th=[11469], 99.90th=[37487], 99.95th=[54264], | 99.99th=[78119] bw ( KiB/s): min=60928, max=149504, per=100.00%, avg=87281.68, stdev=11003.60, samples=60 iops : min= 238, max= 584, avg=340.93, stdev=42.99, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 50=0.30%, 100=77.24%, 250=2.16% lat (usec) : 500=0.02%, 750=0.01%, 1000=0.64% lat (msec) : 2=7.78%, 4=10.40%, 10=0.94%, 20=0.48%, 50=0.02% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.67%, sys=4.83%, ctx=50770, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40517,10228,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=338MiB/s (354MB/s), 338MiB/s-338MiB/s (354MB/s-354MB/s), io=9.89GiB (10.6GB), run=30001-30001msec WRITE: bw=85.2MiB/s (89.4MB/s), 85.2MiB/s-85.2MiB/s (89.4MB/s-89.4MB/s), io=2557MiB (2681MB), run=30001-30001msec Disk stats (read/write): vdk: ios=40265/10166, merge=0/0, ticks=756/20024, in_queue=28177, util=94.20% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943465: Fri Jan 7 14:11:34 2022 read: IOPS=1353, BW=338MiB/s (355MB/s)(9.91GiB/30001msec) clat (usec): min=52, max=51514, avg=95.41, stdev=529.62 lat (usec): min=52, max=51514, avg=95.55, stdev=529.63 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 95], 95.00th=[ 102], | 99.00th=[ 124], 99.50th=[ 141], 99.90th=[ 8094], 99.95th=[13698], | 99.99th=[19530] bw ( KiB/s): min=259072, max=582656, per=100.00%, avg=346553.22, stdev=47433.91, samples=60 iops : min= 1012, max= 2276, avg=1353.72, stdev=185.29, samples=60 write: IOPS=341, BW=85.4MiB/s (89.6MB/s)(2562MiB/30001msec) clat (usec): min=807, max=92932, avg=2516.50, stdev=2493.13 lat (usec): min=830, max=92966, avg=2540.96, stdev=2493.03 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1074], 10.00th=[ 1237], 20.00th=[ 1811], | 30.00th=[ 1942], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2638], 90.00th=[ 3097], 95.00th=[ 4555], | 99.00th=[10683], 99.50th=[11207], 99.90th=[40109], 99.95th=[53216], | 99.99th=[70779] bw ( KiB/s): min=61952, max=154624, per=100.00%, avg=87455.33, stdev=11097.73, samples=60 iops : min= 242, max= 604, avg=341.60, stdev=43.35, samples=60 lat (usec) : 100=75.06%, 250=4.62%, 500=0.03%, 750=0.01%, 1000=0.62% lat (msec) : 2=6.66%, 4=11.49%, 10=0.92%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.78%, sys=2.84%, ctx=50869, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40609,10249,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=338MiB/s (355MB/s), 338MiB/s-338MiB/s (355MB/s-355MB/s), io=9.91GiB (10.6GB), run=30001-30001msec WRITE: bw=85.4MiB/s (89.6MB/s), 85.4MiB/s-85.4MiB/s (89.6MB/s-89.6MB/s), io=2562MiB (2687MB), run=30001-30001msec Disk stats (read/write): vdk: ios=40396/10203, merge=0/0, ticks=678/20567, in_queue=28371, util=94.81% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943993: Fri Jan 7 14:12:11 2022 read: IOPS=1414, BW=354MiB/s (371MB/s)(10.4GiB/30048msec) slat (usec): min=7, max=258, avg=17.32, stdev= 5.75 clat (nsec): min=814, max=2292.7M, avg=410855.00, stdev=19584558.19 lat (usec): min=51, max=2292.7k, avg=428.33, stdev=19584.61 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 55], 40.00th=[ 58], | 50.00th=[ 62], 60.00th=[ 67], 70.00th=[ 76], | 80.00th=[ 94], 90.00th=[ 115], 95.00th=[ 131], | 99.00th=[ 198], 99.50th=[ 297], 99.90th=[ 30802], | 99.95th=[ 42206], 99.99th=[1098908] bw ( KiB/s): min=294912, max=725504, per=100.00%, avg=362683.73, stdev=56141.50, samples=60 iops : min= 1152, max= 2834, avg=1416.73, stdev=219.30, samples=60 write: IOPS=356, BW=89.2MiB/s (93.6MB/s)(2681MiB/30048msec) slat (usec): min=20, max=433, avg=49.90, stdev=20.09 clat (msec): min=2, max=101, avg=26.26, stdev=10.73 lat (msec): min=2, max=101, avg=26.31, stdev=10.73 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 18], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 31], 80.00th=[ 34], 90.00th=[ 39], 95.00th=[ 43], | 99.00th=[ 59], 99.50th=[ 81], 99.90th=[ 91], 99.95th=[ 101], | 99.99th=[ 102] bw ( KiB/s): min=75776, max=183296, per=100.00%, avg=91477.33, stdev=13079.90, samples=60 iops : min= 296, max= 716, avg=357.33, stdev=51.09, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.01%, 50=5.81%, 100=60.31% lat (usec) : 250=13.19%, 500=0.29%, 750=0.07%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.02%, 10=1.14%, 20=4.02%, 50=14.69% lat (msec) : 100=0.36%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=2.12%, sys=4.53%, ctx=50025, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=42507,10725,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=354MiB/s (371MB/s), 354MiB/s-354MiB/s (371MB/s-371MB/s), io=10.4GiB (11.1GB), run=30048-30048msec WRITE: bw=89.2MiB/s (93.6MB/s), 89.2MiB/s-89.2MiB/s (93.6MB/s-93.6MB/s), io=2681MiB (2811MB), run=30048-30048msec Disk stats (read/write): vdk: ios=42232/10653, merge=0/0, ticks=8651/275180, in_queue=296485, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944970: Fri Jan 7 14:12:48 2022 read: IOPS=1356, BW=339MiB/s (356MB/s)(9.93GiB/30002msec) clat (usec): min=51, max=22565, avg=96.94, stdev=475.78 lat (usec): min=51, max=22565, avg=97.08, stdev=475.78 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 71], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 98], 95.00th=[ 105], | 99.00th=[ 127], 99.50th=[ 145], 99.90th=[ 9372], 99.95th=[13566], | 99.99th=[19792] bw ( KiB/s): min=206848, max=570880, per=100.00%, avg=347218.93, stdev=49150.28, samples=60 iops : min= 808, max= 2230, avg=1356.32, stdev=192.00, samples=60 write: IOPS=342, BW=85.5MiB/s (89.7MB/s)(2566MiB/30002msec) clat (usec): min=798, max=99416, avg=2504.47, stdev=2566.75 lat (usec): min=822, max=99441, avg=2529.80, stdev=2566.61 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1074], 10.00th=[ 1221], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2638], 90.00th=[ 3064], 95.00th=[ 4424], | 99.00th=[10683], 99.50th=[11207], 99.90th=[52691], 99.95th=[53740], | 99.99th=[69731] bw ( KiB/s): min=46592, max=152064, per=99.99%, avg=87583.33, stdev=11386.89, samples=60 iops : min= 182, max= 594, avg=342.12, stdev=44.48, samples=60 lat (usec) : 100=73.57%, 250=6.14%, 500=0.02%, 750=0.01%, 1000=0.60% lat (msec) : 2=6.75%, 4=11.43%, 10=0.92%, 20=0.53%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.69%, sys=3.20%, ctx=50965, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40691,10265,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=339MiB/s (356MB/s), 339MiB/s-339MiB/s (356MB/s-356MB/s), io=9.93GiB (10.7GB), run=30002-30002msec WRITE: bw=85.5MiB/s (89.7MB/s), 85.5MiB/s-85.5MiB/s (89.7MB/s-89.7MB/s), io=2566MiB (2691MB), run=30002-30002msec Disk stats (read/write): vdk: ios=40340/10189, merge=0/0, ticks=650/20464, in_queue=28223, util=94.40% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945598: Fri Jan 7 14:13:26 2022 read: IOPS=1402, BW=351MiB/s (368MB/s)(10.3GiB/30199msec) slat (usec): min=4, max=348, avg=13.13, stdev= 6.89 clat (nsec): min=856, max=2541.5M, avg=1983208.60, stdev=59465255.47 lat (usec): min=46, max=2541.5k, avg=1996.50, stdev=59465.23 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 59], | 50.00th=[ 63], 60.00th=[ 70], 70.00th=[ 91], | 80.00th=[ 114], 90.00th=[ 133], 95.00th=[ 169], | 99.00th=[ 1156], 99.50th=[ 3032], 99.90th=[ 692061], | 99.95th=[1803551], 99.99th=[2533360] bw ( KiB/s): min=231424, max=785408, per=100.00%, avg=361401.23, stdev=65255.70, samples=60 iops : min= 904, max= 3068, avg=1411.70, stdev=254.91, samples=60 write: IOPS=354, BW=88.5MiB/s (92.8MB/s)(2674MiB/30199msec) slat (usec): min=22, max=691, avg=54.60, stdev=28.03 clat (msec): min=15, max=2305, avg=133.19, stdev=79.20 lat (msec): min=15, max=2305, avg=133.25, stdev=79.20 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 111], | 30.00th=[ 116], 40.00th=[ 122], 50.00th=[ 127], 60.00th=[ 132], | 70.00th=[ 138], 80.00th=[ 144], 90.00th=[ 159], 95.00th=[ 182], | 99.00th=[ 313], 99.50th=[ 600], 99.90th=[ 1334], 99.95th=[ 1620], | 99.99th=[ 2123] bw ( KiB/s): min=74388, max=180224, per=100.00%, avg=90879.72, stdev=13108.06, samples=60 iops : min= 290, max= 704, avg=354.97, stdev=51.22, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=6.52% lat (usec) : 100=51.14%, 250=19.72%, 500=0.78%, 750=0.43%, 1000=0.30% lat (msec) : 2=0.33%, 4=0.41%, 10=0.05%, 20=0.05%, 50=0.50% lat (msec) : 100=1.23%, 250=18.12%, 500=0.17%, 750=0.08%, 1000=0.05% lat (msec) : 2000=0.05%, >=2000=0.04% cpu : usr=2.52%, sys=3.74%, ctx=47971, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=42352,10695,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=10.3GiB (11.1GB), run=30199-30199msec WRITE: bw=88.5MiB/s (92.8MB/s), 88.5MiB/s-88.5MiB/s (92.8MB/s-92.8MB/s), io=2674MiB (2804MB), run=30199-30199msec Disk stats (read/write): vdk: ios=42348/10663, merge=0/0, ticks=75951/1408854, in_queue=1501546, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946466: Fri Jan 7 14:14:02 2022 read: IOPS=1340, BW=335MiB/s (351MB/s)(9.82GiB/30001msec) clat (usec): min=52, max=21109, avg=92.88, stdev=414.47 lat (usec): min=52, max=21109, avg=93.01, stdev=414.47 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 95], 95.00th=[ 102], | 99.00th=[ 123], 99.50th=[ 139], 99.90th=[ 7767], 99.95th=[11469], | 99.99th=[17433] bw ( KiB/s): min=271360, max=568320, per=100.00%, avg=343546.73, stdev=45124.76, samples=59 iops : min= 1060, max= 2220, avg=1341.97, stdev=176.28, samples=59 write: IOPS=338, BW=84.6MiB/s (88.7MB/s)(2539MiB/30001msec) clat (usec): min=798, max=61723, avg=2554.35, stdev=2339.93 lat (usec): min=822, max=61749, avg=2578.25, stdev=2339.72 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1811], | 30.00th=[ 1958], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2474], 80.00th=[ 2704], 90.00th=[ 3261], 95.00th=[ 4752], | 99.00th=[10814], 99.50th=[11994], 99.90th=[15008], 99.95th=[53740], | 99.99th=[55313] bw ( KiB/s): min=69771, max=151552, per=100.00%, avg=86741.42, stdev=10713.44, samples=59 iops : min= 272, max= 592, avg=338.81, stdev=41.87, samples=59 lat (usec) : 100=75.14%, 250=4.55%, 500=0.04%, 750=0.01%, 1000=0.56% lat (msec) : 2=6.32%, 4=11.66%, 10=1.16%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.75%, sys=2.82%, ctx=50388, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40225,10154,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=335MiB/s (351MB/s), 335MiB/s-335MiB/s (351MB/s-351MB/s), io=9.82GiB (10.5GB), run=30001-30001msec WRITE: bw=84.6MiB/s (88.7MB/s), 84.6MiB/s-84.6MiB/s (88.7MB/s-88.7MB/s), io=2539MiB (2662MB), run=30001-30001msec Disk stats (read/write): vdk: ios=40005/10106, merge=0/0, ticks=560/20891, in_queue=28347, util=94.64% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947440: Fri Jan 7 14:14:40 2022 read: IOPS=1406, BW=352MiB/s (369MB/s)(10.4GiB/30310msec) slat (usec): min=4, max=293, avg=12.91, stdev= 7.08 clat (nsec): min=926, max=2734.2M, avg=2065627.21, stdev=59304670.03 lat (usec): min=44, max=2734.2k, avg=2078.69, stdev=59304.65 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 59], | 50.00th=[ 64], 60.00th=[ 72], 70.00th=[ 104], | 80.00th=[ 119], 90.00th=[ 159], 95.00th=[ 586], | 99.00th=[ 4948], 99.50th=[ 6718], 99.90th=[ 809501], | 99.95th=[1602225], 99.99th=[2533360] bw ( KiB/s): min=289280, max=912384, per=100.00%, avg=363673.60, stdev=81686.14, samples=60 iops : min= 1130, max= 3564, avg=1420.60, stdev=319.09, samples=60 write: IOPS=354, BW=88.7MiB/s (92.0MB/s)(2688MiB/30310msec) slat (usec): min=17, max=390, avg=52.67, stdev=25.29 clat (msec): min=16, max=3838, avg=273.56, stdev=168.65 lat (msec): min=16, max=3839, avg=273.61, stdev=168.65 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 165], 10.00th=[ 207], 20.00th=[ 236], | 30.00th=[ 249], 40.00th=[ 257], 50.00th=[ 266], 60.00th=[ 271], | 70.00th=[ 279], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 355], | 99.00th=[ 542], 99.50th=[ 1234], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3608] bw ( KiB/s): min=75776, max=183296, per=100.00%, avg=90948.27, stdev=13587.17, samples=60 iops : min= 296, max= 716, avg=355.27, stdev=53.07, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=6.26%, 100=48.86% lat (usec) : 250=18.83%, 500=1.56%, 750=0.79%, 1000=0.47% lat (msec) : 2=1.27%, 4=0.82%, 10=0.84%, 20=0.01%, 50=0.11% lat (msec) : 100=0.58%, 250=5.58%, 500=13.59%, 750=0.18%, 1000=0.04% lat (msec) : 2000=0.10%, >=2000=0.08% cpu : usr=2.48%, sys=3.51%, ctx=45966, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42624,10752,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=10.4GiB (11.2GB), run=30310-30310msec WRITE: bw=88.7MiB/s (92.0MB/s), 88.7MiB/s-88.7MiB/s (92.0MB/s-92.0MB/s), io=2688MiB (2819MB), run=30310-30310msec Disk stats (read/write): vdk: ios=42622/10752, merge=0/0, ticks=80395/2920836, in_queue=3012743, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948014: Fri Jan 7 14:15:17 2022 read: IOPS=1334, BW=334MiB/s (350MB/s)(9.78GiB/30001msec) clat (usec): min=52, max=51054, avg=96.18, stdev=483.59 lat (usec): min=52, max=51054, avg=96.32, stdev=483.59 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 98], 95.00th=[ 105], | 99.00th=[ 133], 99.50th=[ 155], 99.90th=[ 8848], 99.95th=[11469], | 99.99th=[16188] bw ( KiB/s): min=252416, max=568320, per=100.00%, avg=342137.49, stdev=44130.06, samples=59 iops : min= 986, max= 2220, avg=1336.47, stdev=172.38, samples=59 write: IOPS=337, BW=84.3MiB/s (88.3MB/s)(2528MiB/30001msec) clat (usec): min=833, max=61319, avg=2553.10, stdev=2312.31 lat (usec): min=852, max=61346, avg=2577.72, stdev=2312.07 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1811], | 30.00th=[ 1942], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2474], 80.00th=[ 2671], 90.00th=[ 3228], 95.00th=[ 4752], | 99.00th=[10814], 99.50th=[12256], 99.90th=[14484], 99.95th=[53216], | 99.99th=[54789] bw ( KiB/s): min=72192, max=151552, per=100.00%, avg=86311.05, stdev=10498.99, samples=59 iops : min= 282, max= 592, avg=337.15, stdev=41.01, samples=59 lat (usec) : 100=73.56%, 250=6.09%, 500=0.04%, 750=0.02%, 1000=0.57% lat (msec) : 2=6.36%, 4=11.70%, 10=1.04%, 20=0.61%, 100=0.02% cpu : usr=1.73%, sys=2.96%, ctx=50164, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40046,10111,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=9.78GiB (10.5GB), run=30001-30001msec WRITE: bw=84.3MiB/s (88.3MB/s), 84.3MiB/s-84.3MiB/s (88.3MB/s-88.3MB/s), io=2528MiB (2651MB), run=30001-30001msec Disk stats (read/write): vdk: ios=39769/10045, merge=0/0, ticks=622/20785, in_queue=28304, util=94.63% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949367: Fri Jan 7 14:15:54 2022 read: IOPS=1407, BW=352MiB/s (369MB/s)(10.4GiB/30373msec) slat (usec): min=4, max=331834, avg=546.01, stdev=4389.50 clat (msec): min=2, max=2614, avg=47.00, stdev=72.81 lat (msec): min=2, max=2614, avg=47.55, stdev=72.91 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 38], 50.00th=[ 42], 60.00th=[ 46], | 70.00th=[ 50], 80.00th=[ 56], 90.00th=[ 67], 95.00th=[ 78], | 99.00th=[ 279], 99.50th=[ 347], 99.90th=[ 919], 99.95th=[ 2106], | 99.99th=[ 2601] bw ( KiB/s): min=269312, max=924672, per=100.00%, avg=364143.42, stdev=83474.93, samples=60 iops : min= 1052, max= 3612, avg=1422.42, stdev=326.07, samples=60 write: IOPS=354, BW=88.7MiB/s (93.0MB/s)(2694MiB/30373msec) slat (usec): min=32, max=318271, avg=607.59, stdev=4414.77 clat (msec): min=33, max=3780, avg=374.32, stdev=179.97 lat (msec): min=46, max=3807, avg=374.93, stdev=179.97 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 224], 10.00th=[ 296], 20.00th=[ 326], | 30.00th=[ 342], 40.00th=[ 355], 50.00th=[ 368], 60.00th=[ 376], | 70.00th=[ 388], 80.00th=[ 405], 90.00th=[ 435], 95.00th=[ 514], | 99.00th=[ 726], 99.50th=[ 1250], 99.90th=[ 3171], 99.95th=[ 3171], | 99.99th=[ 3608] bw ( KiB/s): min=74752, max=175616, per=100.00%, avg=90885.90, stdev=12992.26, samples=60 iops : min= 292, max= 686, avg=355.00, stdev=50.75, samples=60 lat (msec) : 4=0.10%, 10=5.10%, 20=5.72%, 50=45.73%, 100=22.00% lat (msec) : 250=1.57%, 500=18.58%, 750=0.92%, 1000=0.07%, 2000=0.10% lat (msec) : >=2000=0.10% cpu : usr=3.16%, sys=1.93%, ctx=5691, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42743,10777,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=10.4GiB (11.2GB), run=30373-30373msec WRITE: bw=88.7MiB/s (93.0MB/s), 88.7MiB/s-88.7MiB/s (93.0MB/s-93.0MB/s), io=2694MiB (2825MB), run=30373-30373msec Disk stats (read/write): vdk: ios=42738/10748, merge=0/0, ticks=106455/3548229, in_queue=3701501, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950049: Fri Jan 7 14:16:30 2022 read: IOPS=925, BW=463MiB/s (485MB/s)(13.6GiB/30002msec) clat (usec): min=89, max=22222, avg=149.16, stdev=574.82 lat (usec): min=89, max=22223, avg=149.32, stdev=574.82 clat percentiles (usec): | 1.00th=[ 97], 5.00th=[ 102], 10.00th=[ 104], 20.00th=[ 106], | 30.00th=[ 110], 40.00th=[ 114], 50.00th=[ 118], 60.00th=[ 124], | 70.00th=[ 131], 80.00th=[ 139], 90.00th=[ 153], 95.00th=[ 167], | 99.00th=[ 223], 99.50th=[ 255], 99.90th=[11994], 99.95th=[15008], | 99.99th=[19006] bw ( KiB/s): min=339968, max=612352, per=100.00%, avg=474073.30, stdev=57919.20, samples=60 iops : min= 664, max= 1196, avg=925.90, stdev=113.13, samples=60 write: IOPS=233, BW=117MiB/s (122MB/s)(3501MiB/30002msec) clat (usec): min=1421, max=62903, avg=3634.63, stdev=2622.63 lat (usec): min=1470, max=62961, avg=3684.36, stdev=2622.34 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1696], 10.00th=[ 1893], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3523], 80.00th=[ 3884], 90.00th=[ 6063], 95.00th=[ 7373], | 99.00th=[11994], 99.50th=[15008], 99.90th=[20317], 99.95th=[54264], | 99.99th=[62653] bw ( KiB/s): min=88064, max=177152, per=100.00%, avg=119474.35, stdev=12675.58, samples=60 iops : min= 172, max= 346, avg=233.33, stdev=24.76, samples=60 lat (usec) : 100=2.17%, 250=77.27%, 500=0.24%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.30%, 4=14.22%, 10=3.04%, 20=0.70%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.73%, sys=2.69%, ctx=34782, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27776,7001,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=463MiB/s (485MB/s), 463MiB/s-463MiB/s (485MB/s-485MB/s), io=13.6GiB (14.6GB), run=30002-30002msec WRITE: bw=117MiB/s (122MB/s), 117MiB/s-117MiB/s (122MB/s-122MB/s), io=3501MiB (3671MB), run=30002-30002msec Disk stats (read/write): vdk: ios=27662/6966, merge=0/0, ticks=655/21642, in_queue=28394, util=94.92% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950866: Fri Jan 7 14:17:08 2022 read: IOPS=945, BW=473MiB/s (496MB/s)(13.9GiB/30001msec) slat (usec): min=18, max=125, avg=28.47, stdev= 6.52 clat (usec): min=34, max=23045, avg=139.11, stdev=606.67 lat (usec): min=99, max=23073, avg=167.76, stdev=606.79 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 114], | 70.00th=[ 120], 80.00th=[ 128], 90.00th=[ 139], 95.00th=[ 151], | 99.00th=[ 188], 99.50th=[ 217], 99.90th=[12387], 99.95th=[14877], | 99.99th=[22676] bw ( KiB/s): min=357376, max=642048, per=100.00%, avg=485514.85, stdev=57235.27, samples=59 iops : min= 698, max= 1254, avg=948.27, stdev=111.79, samples=59 write: IOPS=238, BW=119MiB/s (125MB/s)(3584MiB/30001msec) slat (usec): min=40, max=256, avg=75.22, stdev=15.86 clat (usec): min=1429, max=57754, avg=3438.04, stdev=2537.59 lat (usec): min=1502, max=57829, avg=3513.49, stdev=2537.51 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1631], 10.00th=[ 1696], 20.00th=[ 2114], | 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3064], 60.00th=[ 3228], | 70.00th=[ 3458], 80.00th=[ 3785], 90.00th=[ 5538], 95.00th=[ 6980], | 99.00th=[11863], 99.50th=[14484], 99.90th=[32637], 99.95th=[54264], | 99.99th=[57934] bw ( KiB/s): min=97280, max=181248, per=100.00%, avg=122446.10, stdev=11222.52, samples=59 iops : min= 190, max= 354, avg=239.15, stdev=21.92, samples=59 lat (usec) : 50=0.01%, 100=25.79%, 250=53.78%, 500=0.08%, 750=0.01% lat (msec) : 2=3.80%, 4=13.14%, 10=2.75%, 20=0.61%, 50=0.03% lat (msec) : 100=0.02% cpu : usr=1.68%, sys=4.51%, ctx=35549, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28378,7167,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=473MiB/s (496MB/s), 473MiB/s-473MiB/s (496MB/s-496MB/s), io=13.9GiB (14.9GB), run=30001-30001msec WRITE: bw=119MiB/s (125MB/s), 119MiB/s-119MiB/s (125MB/s-125MB/s), io=3584MiB (3758MB), run=30001-30001msec Disk stats (read/write): vdk: ios=28240/7129, merge=0/0, ticks=718/20745, in_queue=28372, util=94.81% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951815: Fri Jan 7 14:17:44 2022 read: IOPS=972, BW=486MiB/s (510MB/s)(14.2GiB/30002msec) clat (usec): min=72, max=17419, avg=127.19, stdev=519.94 lat (usec): min=72, max=17419, avg=127.34, stdev=519.94 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 86], 10.00th=[ 88], 20.00th=[ 91], | 30.00th=[ 93], 40.00th=[ 96], 50.00th=[ 99], 60.00th=[ 103], | 70.00th=[ 111], 80.00th=[ 119], 90.00th=[ 129], 95.00th=[ 139], | 99.00th=[ 169], 99.50th=[ 196], 99.90th=[11863], 99.95th=[13042], | 99.99th=[16712] bw ( KiB/s): min=374784, max=640000, per=100.00%, avg=497936.45, stdev=52636.71, samples=60 iops : min= 732, max= 1250, avg=972.52, stdev=102.81, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3685MiB/30002msec) clat (usec): min=1368, max=70770, avg=3508.76, stdev=2557.36 lat (usec): min=1421, max=70818, avg=3558.90, stdev=2556.71 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2442], | 30.00th=[ 2802], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3785], 90.00th=[ 4686], 95.00th=[ 6783], | 99.00th=[11863], 99.50th=[13173], 99.90th=[47449], 99.95th=[57410], | 99.99th=[70779] bw ( KiB/s): min=100352, max=181248, per=100.00%, avg=125751.57, stdev=10858.56, samples=60 iops : min= 196, max= 354, avg=245.60, stdev=21.20, samples=60 lat (usec) : 100=41.76%, 250=37.84%, 500=0.06%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.79%, 4=14.20%, 10=2.75%, 20=0.56%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=2.06%, sys=2.19%, ctx=36548, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29175,7369,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=486MiB/s (510MB/s), 486MiB/s-486MiB/s (510MB/s-510MB/s), io=14.2GiB (15.3GB), run=30002-30002msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3685MiB (3863MB), run=30002-30002msec Disk stats (read/write): vdk: ios=28997/7327, merge=0/0, ticks=631/21777, in_queue=28421, util=94.97% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952398: Fri Jan 7 14:18:22 2022 read: IOPS=989, BW=495MiB/s (519MB/s)(14.5GiB/30037msec) slat (usec): min=6, max=984, avg=20.90, stdev= 9.59 clat (nsec): min=1361, max=91036k, avg=181404.21, stdev=1612195.49 lat (usec): min=69, max=91049, avg=202.46, stdev=1612.22 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 70], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 83], 60.00th=[ 93], | 70.00th=[ 110], 80.00th=[ 176], 90.00th=[ 196], 95.00th=[ 221], | 99.00th=[ 334], 99.50th=[ 529], 99.90th=[31589], 99.95th=[39584], | 99.99th=[57934] bw ( KiB/s): min=370688, max=756736, per=100.00%, avg=507431.65, stdev=62790.46, samples=60 iops : min= 724, max= 1478, avg=991.07, stdev=122.62, samples=60 write: IOPS=249, BW=125MiB/s (131MB/s)(3751MiB/30037msec) slat (usec): min=35, max=402, avg=91.43, stdev=42.28 clat (msec): min=2, max=126, avg=39.13, stdev=15.72 lat (msec): min=2, max=126, avg=39.22, stdev=15.72 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 35], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 47], 80.00th=[ 52], 90.00th=[ 58], 95.00th=[ 65], | 99.00th=[ 84], 99.50th=[ 96], 99.90th=[ 116], 99.95th=[ 118], | 99.99th=[ 127] bw ( KiB/s): min=110592, max=205824, per=99.98%, avg=127855.10, stdev=12050.31, samples=60 iops : min= 216, max= 402, avg=249.73, stdev=23.55, samples=60 lat (usec) : 2=0.01%, 50=0.05%, 100=51.63%, 250=26.17%, 500=1.57% lat (usec) : 750=0.14%, 1000=0.06% lat (msec) : 2=0.08%, 4=0.01%, 10=0.59%, 20=1.78%, 50=13.50% lat (msec) : 100=4.36%, 250=0.07% cpu : usr=2.69%, sys=3.37%, ctx=35605, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29730,7502,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=495MiB/s (519MB/s), 495MiB/s-495MiB/s (519MB/s-519MB/s), io=14.5GiB (15.6GB), run=30037-30037msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3751MiB (3933MB), run=30037-30037msec Disk stats (read/write): vdk: ios=29553/7450, merge=0/0, ticks=2006/287515, in_queue=296653, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953226: Fri Jan 7 14:18:59 2022 read: IOPS=966, BW=483MiB/s (507MB/s)(14.2GiB/30001msec) clat (usec): min=75, max=17402, avg=132.34, stdev=535.02 lat (usec): min=75, max=17402, avg=132.50, stdev=535.02 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 141], | 99.00th=[ 163], 99.50th=[ 184], 99.90th=[11338], 99.95th=[13698], | 99.99th=[16712] bw ( KiB/s): min=390144, max=648192, per=100.00%, avg=494978.73, stdev=53762.85, samples=60 iops : min= 762, max= 1266, avg=966.73, stdev=105.03, samples=60 write: IOPS=244, BW=122MiB/s (128MB/s)(3665MiB/30001msec) clat (usec): min=1367, max=62697, avg=3512.26, stdev=2475.84 lat (usec): min=1404, max=62743, avg=3560.31, stdev=2475.24 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1631], 10.00th=[ 1713], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3818], 90.00th=[ 4686], 95.00th=[ 6980], | 99.00th=[11863], 99.50th=[13960], 99.90th=[19792], 99.95th=[54789], | 99.99th=[62653] bw ( KiB/s): min=103424, max=185344, per=100.00%, avg=125089.45, stdev=10527.18, samples=60 iops : min= 202, max= 362, avg=244.30, stdev=20.57, samples=60 lat (usec) : 100=30.97%, 250=48.65%, 500=0.03%, 750=0.01% lat (msec) : 2=2.97%, 4=13.96%, 10=2.78%, 20=0.62%, 100=0.02% cpu : usr=1.89%, sys=2.65%, ctx=36339, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29001,7330,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=483MiB/s (507MB/s), 483MiB/s-483MiB/s (507MB/s-507MB/s), io=14.2GiB (15.2GB), run=30001-30001msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3665MiB (3843MB), run=30001-30001msec Disk stats (read/write): vdk: ios=28860/7290, merge=0/0, ticks=657/21747, in_queue=28440, util=94.98% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954255: Fri Jan 7 14:19:36 2022 read: IOPS=991, BW=496MiB/s (520MB/s)(14.6GiB/30202msec) slat (usec): min=6, max=789, avg=15.74, stdev= 8.23 clat (usec): min=27, max=315113, avg=394.01, stdev=5159.01 lat (usec): min=68, max=315148, avg=409.91, stdev=5159.01 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 87], | 70.00th=[ 104], 80.00th=[ 184], 90.00th=[ 202], 95.00th=[ 285], | 99.00th=[ 3621], 99.50th=[ 5866], 99.90th=[ 94897], 99.95th=[124257], | 99.99th=[193987] bw ( KiB/s): min=376832, max=957440, per=100.00%, avg=511258.35, stdev=79390.81, samples=60 iops : min= 736, max= 1870, avg=998.52, stdev=155.07, samples=60 write: IOPS=250, BW=125MiB/s (131MB/s)(3782MiB/30202msec) slat (usec): min=35, max=372, avg=91.74, stdev=45.52 clat (msec): min=12, max=2664, avg=197.91, stdev=113.09 lat (msec): min=12, max=2664, avg=198.00, stdev=113.09 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 72], 10.00th=[ 83], 20.00th=[ 102], | 30.00th=[ 127], 40.00th=[ 159], 50.00th=[ 194], 60.00th=[ 224], | 70.00th=[ 249], 80.00th=[ 275], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 460], 99.50th=[ 567], 99.90th=[ 1318], 99.95th=[ 1854], | 99.99th=[ 2668] bw ( KiB/s): min=108544, max=215040, per=100.00%, avg=128243.77, stdev=13584.60, samples=60 iops : min= 212, max= 420, avg=250.45, stdev=26.53, samples=60 lat (usec) : 50=0.07%, 100=54.77%, 250=20.49%, 500=1.64%, 750=0.53% lat (usec) : 1000=0.29% lat (msec) : 2=0.75%, 4=0.56%, 10=0.59%, 20=0.01%, 50=0.24% lat (msec) : 100=3.76%, 250=10.33%, 500=5.83%, 750=0.08%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.73%, sys=2.77%, ctx=34949, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=29956,7563,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=496MiB/s (520MB/s), 496MiB/s-496MiB/s (520MB/s-520MB/s), io=14.6GiB (15.7GB), run=30202-30202msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3782MiB (3965MB), run=30202-30202msec Disk stats (read/write): vdk: ios=29956/7520, merge=0/0, ticks=7806/1476146, in_queue=1500148, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954774: Fri Jan 7 14:20:13 2022 read: IOPS=950, BW=475MiB/s (498MB/s)(13.9GiB/30001msec) clat (usec): min=100, max=51083, avg=161.91, stdev=610.50 lat (usec): min=100, max=51083, avg=162.06, stdev=610.50 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 114], 10.00th=[ 116], 20.00th=[ 119], | 30.00th=[ 122], 40.00th=[ 126], 50.00th=[ 131], 60.00th=[ 137], | 70.00th=[ 145], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 182], | 99.00th=[ 225], 99.50th=[ 251], 99.90th=[11994], 99.95th=[13566], | 99.99th=[17171] bw ( KiB/s): min=352256, max=645120, per=100.00%, avg=486673.95, stdev=55868.19, samples=60 iops : min= 688, max= 1260, avg=950.53, stdev=109.12, samples=60 write: IOPS=239, BW=120MiB/s (126MB/s)(3598MiB/30001msec) clat (usec): min=1411, max=62226, avg=3463.95, stdev=2416.14 lat (usec): min=1437, max=62278, avg=3518.91, stdev=2415.17 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 1680], 10.00th=[ 1729], 20.00th=[ 2147], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3490], 80.00th=[ 3818], 90.00th=[ 5669], 95.00th=[ 6980], | 99.00th=[11863], 99.50th=[13829], 99.90th=[21365], 99.95th=[51643], | 99.99th=[62129] bw ( KiB/s): min=100352, max=184320, per=99.99%, avg=122781.70, stdev=11108.70, samples=60 iops : min= 196, max= 360, avg=239.80, stdev=21.70, samples=60 lat (usec) : 250=79.45%, 500=0.22%, 1000=0.01% lat (msec) : 2=3.74%, 4=13.07%, 10=2.85%, 20=0.64%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=2.07%, sys=2.38%, ctx=35717, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28515,7195,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=475MiB/s (498MB/s), 475MiB/s-475MiB/s (498MB/s-498MB/s), io=13.9GiB (14.9GB), run=30001-30001msec WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=3598MiB (3772MB), run=30001-30001msec Disk stats (read/write): vdk: ios=28345/7153, merge=0/0, ticks=668/20806, in_queue=28397, util=94.95% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956003: Fri Jan 7 14:20:50 2022 read: IOPS=995, BW=498MiB/s (522MB/s)(14.8GiB/30417msec) slat (usec): min=5, max=498, avg=14.57, stdev=10.64 clat (nsec): min=1432, max=502707k, avg=993022.68, stdev=7361619.23 lat (usec): min=70, max=502733, avg=1007.74, stdev=7361.61 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 94], | 70.00th=[ 182], 80.00th=[ 198], 90.00th=[ 758], 95.00th=[ 5145], | 99.00th=[ 14091], 99.50th=[ 16450], 99.90th=[107480], 99.95th=[132645], | 99.99th=[214959] bw ( KiB/s): min=108544, max=1111040, per=100.00%, avg=516891.02, stdev=142125.70, samples=60 iops : min= 212, max= 2170, avg=1009.53, stdev=277.59, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3819MiB/30417msec) slat (usec): min=35, max=531, avg=97.40, stdev=49.84 clat (msec): min=14, max=3019, avg=394.07, stdev=156.20 lat (msec): min=14, max=3019, avg=394.17, stdev=156.19 clat percentiles (msec): | 1.00th=[ 106], 5.00th=[ 205], 10.00th=[ 259], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 347], 50.00th=[ 384], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 477], 90.00th=[ 527], 95.00th=[ 609], | 99.00th=[ 844], 99.50th=[ 978], 99.90th=[ 2232], 99.95th=[ 2567], | 99.99th=[ 3004] bw ( KiB/s): min=100352, max=195584, per=100.00%, avg=128661.97, stdev=12537.96, samples=60 iops : min= 196, max= 382, avg=251.27, stdev=24.48, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=0.07%, 100=49.60%, 250=18.55% lat (usec) : 500=2.47%, 750=1.14%, 1000=0.64% lat (msec) : 2=1.33%, 4=1.40%, 10=2.67%, 20=1.81%, 50=0.13% lat (msec) : 100=0.13%, 250=1.54%, 500=15.65%, 750=2.49%, 1000=0.27% lat (msec) : 2000=0.07%, >=2000=0.02% cpu : usr=2.81%, sys=2.58%, ctx=31929, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30284,7637,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=14.8GiB (15.9GB), run=30417-30417msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3819MiB (4004MB), run=30417-30417msec Disk stats (read/write): vdk: ios=30284/7570, merge=0/0, ticks=22316/2956503, in_queue=3017565, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956742: Fri Jan 7 14:21:27 2022 read: IOPS=973, BW=487MiB/s (510MB/s)(14.3GiB/30003msec) clat (usec): min=79, max=23761, avg=132.57, stdev=557.44 lat (usec): min=79, max=23762, avg=132.72, stdev=557.44 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 92], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 97], 40.00th=[ 99], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 143], | 99.00th=[ 174], 99.50th=[ 200], 99.90th=[11076], 99.95th=[14615], | 99.99th=[17171] bw ( KiB/s): min=337920, max=629760, per=100.00%, avg=498483.77, stdev=57249.95, samples=60 iops : min= 660, max= 1230, avg=973.60, stdev=111.82, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3689MiB/30003msec) clat (usec): min=1441, max=62514, avg=3484.61, stdev=2444.81 lat (usec): min=1494, max=62541, avg=3532.94, stdev=2444.16 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3490], 80.00th=[ 3785], 90.00th=[ 4621], 95.00th=[ 6652], | 99.00th=[11731], 99.50th=[12387], 99.90th=[47449], 99.95th=[54264], | 99.99th=[62653] bw ( KiB/s): min=104448, max=173056, per=100.00%, avg=125905.23, stdev=9678.39, samples=60 iops : min= 204, max= 338, avg=245.90, stdev=18.90, samples=60 lat (usec) : 100=34.89%, 250=44.71%, 500=0.05%, 750=0.02% lat (msec) : 2=2.94%, 4=14.06%, 10=2.74%, 20=0.56%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.87%, sys=2.69%, ctx=36591, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29207,7378,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=487MiB/s (510MB/s), 487MiB/s-487MiB/s (510MB/s-510MB/s), io=14.3GiB (15.3GB), run=30003-30003msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3689MiB (3868MB), run=30003-30003msec Disk stats (read/write): vdk: ios=28973/7319, merge=0/0, ticks=673/21623, in_queue=28432, util=94.98% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957724: Fri Jan 7 14:22:04 2022 read: IOPS=984, BW=492MiB/s (516MB/s)(14.6GiB/30441msec) slat (usec): min=6, max=506472, avg=776.33, stdev=6610.81 clat (msec): min=9, max=594, avg=65.03, stdev=55.57 lat (msec): min=9, max=594, avg=65.81, stdev=55.96 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 17], 10.00th=[ 24], 20.00th=[ 34], | 30.00th=[ 44], 40.00th=[ 53], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 73], 80.00th=[ 83], 90.00th=[ 97], 95.00th=[ 116], | 99.00th=[ 456], 99.50th=[ 498], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 592] bw ( KiB/s): min=21504, max=1186816, per=100.00%, avg=510664.07, stdev=187937.94, samples=60 iops : min= 42, max= 2318, avg=997.38, stdev=367.07, samples=60 write: IOPS=248, BW=124MiB/s (130MB/s)(3785MiB/30441msec) slat (usec): min=137, max=420799, avg=877.53, stdev=6039.52 clat (msec): min=18, max=2721, avg=542.38, stdev=191.21 lat (msec): min=33, max=2722, avg=543.26, stdev=191.08 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 259], 10.00th=[ 372], 20.00th=[ 422], | 30.00th=[ 468], 40.00th=[ 502], 50.00th=[ 542], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 625], 90.00th=[ 701], 95.00th=[ 877], | 99.00th=[ 1150], 99.50th=[ 1267], 99.90th=[ 2400], 99.95th=[ 2567], | 99.99th=[ 2735] bw ( KiB/s): min=11264, max=211968, per=99.64%, avg=126860.28, stdev=34467.68, samples=60 iops : min= 22, max= 414, avg=247.77, stdev=67.32, samples=60 lat (msec) : 10=0.01%, 20=5.98%, 50=24.19%, 100=42.84%, 250=6.79% lat (msec) : 500=7.63%, 750=11.03%, 1000=1.13%, 2000=0.36%, >=2000=0.03% cpu : usr=4.60%, sys=1.62%, ctx=3813, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=29984,7570,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=492MiB/s (516MB/s), 492MiB/s-492MiB/s (516MB/s-516MB/s), io=14.6GiB (15.7GB), run=30441-30441msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3785MiB (3969MB), run=30441-30441msec Disk stats (read/write): vdk: ios=29984/7516, merge=0/0, ticks=67242/3588384, in_queue=3710032, util=99.56% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958709: Fri Jan 7 14:22:41 2022 read: IOPS=607, BW=607MiB/s (637MB/s)(17.8GiB/30001msec) clat (usec): min=144, max=45367, avg=240.18, stdev=928.03 lat (usec): min=144, max=45367, avg=240.34, stdev=928.03 clat percentiles (usec): | 1.00th=[ 153], 5.00th=[ 157], 10.00th=[ 161], 20.00th=[ 163], | 30.00th=[ 167], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 194], 80.00th=[ 208], 90.00th=[ 225], 95.00th=[ 243], | 99.00th=[ 334], 99.50th=[ 453], 99.90th=[16450], 99.95th=[17957], | 99.99th=[25297] bw ( KiB/s): min=481280, max=845824, per=100.00%, avg=621735.85, stdev=79396.73, samples=60 iops : min= 470, max= 826, avg=607.15, stdev=77.52, samples=60 write: IOPS=153, BW=154MiB/s (161MB/s)(4607MiB/30001msec) clat (usec): min=2566, max=64102, avg=5454.82, stdev=3135.08 lat (usec): min=2650, max=64201, avg=5553.21, stdev=3133.77 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2704], 10.00th=[ 2737], 20.00th=[ 3458], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8979], 95.00th=[11338], | 99.00th=[15664], 99.50th=[17695], 99.90th=[51119], 99.95th=[56361], | 99.99th=[64226] bw ( KiB/s): min=131072, max=202752, per=99.99%, avg=157234.53, stdev=10921.76, samples=60 iops : min= 128, max= 198, avg=153.53, stdev=10.66, samples=60 lat (usec) : 250=76.56%, 500=2.90%, 750=0.03% lat (msec) : 2=0.01%, 4=5.21%, 10=13.49%, 20=1.75%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=2.07%, sys=2.34%, ctx=22826, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18213,4607,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=607MiB/s (637MB/s), 607MiB/s-607MiB/s (637MB/s-637MB/s), io=17.8GiB (19.1GB), run=30001-30001msec WRITE: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4607MiB (4831MB), run=30001-30001msec Disk stats (read/write): vdk: ios=18098/4570, merge=0/0, ticks=964/22325, in_queue=28385, util=94.99% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959288: Fri Jan 7 14:23:18 2022 read: IOPS=603, BW=604MiB/s (633MB/s)(17.7GiB/30002msec) slat (usec): min=28, max=176, avg=41.29, stdev= 8.63 clat (nsec): min=1313, max=28038k, avg=209905.64, stdev=928668.84 lat (usec): min=148, max=28072, avg=251.39, stdev=928.77 clat percentiles (usec): | 1.00th=[ 124], 5.00th=[ 127], 10.00th=[ 129], 20.00th=[ 133], | 30.00th=[ 135], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 151], | 70.00th=[ 163], 80.00th=[ 176], 90.00th=[ 192], 95.00th=[ 212], | 99.00th=[ 322], 99.50th=[ 453], 99.90th=[16909], 99.95th=[18482], | 99.99th=[23987] bw ( KiB/s): min=411648, max=862208, per=100.00%, avg=618530.13, stdev=86537.98, samples=60 iops : min= 402, max= 842, avg=604.03, stdev=84.51, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4577MiB/30002msec) slat (usec): min=67, max=378, avg=132.68, stdev=31.56 clat (usec): min=2528, max=58937, avg=5417.70, stdev=3099.45 lat (usec): min=2636, max=59063, avg=5550.63, stdev=3098.38 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3359], | 30.00th=[ 4178], 40.00th=[ 4686], 50.00th=[ 5014], 60.00th=[ 5211], | 70.00th=[ 5473], 80.00th=[ 5866], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[16057], 99.50th=[18220], 99.90th=[26346], 99.95th=[55837], | 99.99th=[58983] bw ( KiB/s): min=129024, max=212992, per=99.99%, avg=156194.13, stdev=12379.45, samples=60 iops : min= 126, max= 208, avg=152.53, stdev=12.09, samples=60 lat (usec) : 2=0.01%, 100=0.01%, 250=78.03%, 500=1.41%, 750=0.04% lat (usec) : 1000=0.02% lat (msec) : 2=0.01%, 4=5.59%, 10=13.06%, 20=1.78%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=3.65%, ctx=22708, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18121,4577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=17.7GiB (19.0GB), run=30002-30002msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4577MiB (4799MB), run=30002-30002msec Disk stats (read/write): vdk: ios=18013/4549, merge=0/0, ticks=979/22251, in_queue=28535, util=95.34% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960090: Fri Jan 7 14:23:55 2022 read: IOPS=610, BW=611MiB/s (640MB/s)(17.9GiB/30005msec) clat (usec): min=132, max=46920, avg=228.82, stdev=996.07 lat (usec): min=133, max=46920, avg=228.98, stdev=996.08 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 208], 95.00th=[ 227], | 99.00th=[ 314], 99.50th=[ 457], 99.90th=[17171], 99.95th=[19006], | 99.99th=[30540] bw ( KiB/s): min=432993, max=833536, per=100.00%, avg=625463.22, stdev=90647.25, samples=60 iops : min= 422, max= 814, avg=610.78, stdev=88.55, samples=60 write: IOPS=154, BW=155MiB/s (162MB/s)(4642MiB/30005msec) clat (usec): min=2536, max=56379, avg=5453.00, stdev=2845.49 lat (usec): min=2587, max=56491, avg=5551.08, stdev=2844.23 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3687], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5538], 80.00th=[ 5866], 90.00th=[ 8586], 95.00th=[11338], | 99.00th=[15008], 99.50th=[17695], 99.90th=[23462], 99.95th=[50594], | 99.99th=[56361] bw ( KiB/s): min=135168, max=208896, per=100.00%, avg=158423.38, stdev=11311.52, samples=60 iops : min= 132, max= 204, avg=154.70, stdev=11.05, samples=60 lat (usec) : 250=77.65%, 500=1.77%, 750=0.04% lat (msec) : 2=0.01%, 4=4.75%, 10=14.03%, 20=1.70%, 50=0.04% lat (msec) : 100=0.01% cpu : usr=1.97%, sys=2.48%, ctx=22973, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18323,4642,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=611MiB/s (640MB/s), 611MiB/s-611MiB/s (640MB/s-640MB/s), io=17.9GiB (19.2GB), run=30005-30005msec WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4642MiB (4867MB), run=30005-30005msec Disk stats (read/write): vdk: ios=18230/4612, merge=0/0, ticks=1038/22562, in_queue=28408, util=94.97% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961055: Fri Jan 7 14:24:32 2022 read: IOPS=574, BW=575MiB/s (603MB/s)(16.9GiB/30066msec) slat (usec): min=9, max=129, avg=22.82, stdev= 9.37 clat (usec): min=13, max=109052, avg=345.32, stdev=2947.51 lat (usec): min=114, max=109075, avg=368.31, stdev=2947.68 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 110], 20.00th=[ 112], | 30.00th=[ 115], 40.00th=[ 117], 50.00th=[ 121], 60.00th=[ 128], | 70.00th=[ 141], 80.00th=[ 326], 90.00th=[ 347], 95.00th=[ 367], | 99.00th=[ 553], 99.50th=[ 2180], 99.90th=[ 49546], 99.95th=[ 62129], | 99.99th=[106431] bw ( KiB/s): min=428032, max=948224, per=100.00%, avg=589802.53, stdev=93362.22, samples=60 iops : min= 418, max= 926, avg=575.97, stdev=91.17, samples=60 write: IOPS=145, BW=146MiB/s (153MB/s)(4388MiB/30066msec) slat (usec): min=62, max=557, avg=167.64, stdev=85.68 clat (msec): min=3, max=210, avg=66.87, stdev=26.97 lat (msec): min=3, max=210, avg=67.04, stdev=26.97 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 19], 10.00th=[ 28], 20.00th=[ 46], | 30.00th=[ 57], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 110], | 99.00th=[ 138], 99.50th=[ 153], 99.90th=[ 174], 99.95th=[ 186], | 99.99th=[ 211] bw ( KiB/s): min=122880, max=239616, per=99.99%, avg=149430.88, stdev=14477.89, samples=60 iops : min= 120, max= 234, avg=145.92, stdev=14.14, samples=60 lat (usec) : 20=0.01%, 100=0.14%, 250=62.16%, 500=16.50%, 750=0.29% lat (usec) : 1000=0.11% lat (msec) : 2=0.10%, 4=0.13%, 10=0.08%, 20=1.14%, 50=3.72% lat (msec) : 100=13.91%, 250=1.70% cpu : usr=2.81%, sys=1.94%, ctx=21268, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17280,4388,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=575MiB/s (603MB/s), 575MiB/s-575MiB/s (603MB/s-603MB/s), io=16.9GiB (18.1GB), run=30066-30066msec WRITE: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=4388MiB (4601MB), run=30066-30066msec Disk stats (read/write): vdk: ios=17221/4356, merge=0/0, ticks=2950/288682, in_queue=297145, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961578: Fri Jan 7 14:25:09 2022 read: IOPS=610, BW=610MiB/s (640MB/s)(17.9GiB/30005msec) clat (usec): min=129, max=46003, avg=224.08, stdev=1020.49 lat (usec): min=129, max=46004, avg=224.24, stdev=1020.49 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 141], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 151], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 186], 90.00th=[ 198], 95.00th=[ 210], | 99.00th=[ 262], 99.50th=[ 392], 99.90th=[17957], 99.95th=[19792], | 99.99th=[29492] bw ( KiB/s): min=489472, max=835584, per=100.00%, avg=624837.67, stdev=86531.03, samples=60 iops : min= 478, max= 816, avg=610.18, stdev=84.50, samples=60 write: IOPS=154, BW=155MiB/s (162MB/s)(4637MiB/30005msec) clat (usec): min=2532, max=65227, avg=5480.44, stdev=3067.06 lat (usec): min=2631, max=65287, avg=5577.27, stdev=3064.95 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3523], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5932], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[15926], 99.50th=[17171], 99.90th=[23987], 99.95th=[56361], | 99.99th=[65274] bw ( KiB/s): min=135168, max=219136, per=100.00%, avg=158257.23, stdev=12238.44, samples=60 iops : min= 132, max= 214, avg=154.53, stdev=11.96, samples=60 lat (usec) : 250=78.78%, 500=0.67% lat (msec) : 2=0.02%, 4=5.02%, 10=13.67%, 20=1.76%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=2.31%, ctx=22954, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18304,4637,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=610MiB/s (640MB/s), 610MiB/s-610MiB/s (640MB/s-640MB/s), io=17.9GiB (19.2GB), run=30005-30005msec WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4637MiB (4862MB), run=30005-30005msec Disk stats (read/write): vdk: ios=18199/4601, merge=0/0, ticks=1068/22636, in_queue=28452, util=95.14% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962981: Fri Jan 7 14:25:46 2022 read: IOPS=596, BW=597MiB/s (626MB/s)(17.7GiB/30315msec) slat (usec): min=9, max=349, avg=20.47, stdev=10.55 clat (nsec): min=1065, max=433779k, avg=1084851.40, stdev=9499768.91 lat (usec): min=111, max=433795, avg=1105.51, stdev=9499.61 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 124], 60.00th=[ 135], | 70.00th=[ 206], 80.00th=[ 338], 90.00th=[ 416], 95.00th=[ 2507], | 99.00th=[ 13435], 99.50th=[ 16450], 99.90th=[154141], 99.95th=[221250], | 99.99th=[329253] bw ( KiB/s): min=413696, max=1189888, per=100.00%, avg=617682.58, stdev=122186.95, samples=60 iops : min= 404, max= 1162, avg=603.20, stdev=119.31, samples=60 write: IOPS=150, BW=151MiB/s (158MB/s)(4570MiB/30315msec) slat (usec): min=62, max=665, avg=179.65, stdev=91.39 clat (msec): min=9, max=1272, avg=327.02, stdev=170.66 lat (msec): min=9, max=1272, avg=327.20, stdev=170.66 clat percentiles (msec): | 1.00th=[ 58], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 171], | 30.00th=[ 203], 40.00th=[ 241], 50.00th=[ 292], 60.00th=[ 351], | 70.00th=[ 418], 80.00th=[ 477], 90.00th=[ 558], 95.00th=[ 634], | 99.00th=[ 802], 99.50th=[ 885], 99.90th=[ 1083], 99.95th=[ 1116], | 99.99th=[ 1267] bw ( KiB/s): min=118784, max=237568, per=99.94%, avg=154277.28, stdev=17961.63, samples=60 iops : min= 116, max= 232, avg=150.65, stdev=17.54, samples=60 lat (usec) : 2=0.01%, 100=0.13%, 250=56.89%, 500=15.50%, 750=1.10% lat (usec) : 1000=0.54% lat (msec) : 2=1.22%, 4=1.31%, 10=1.69%, 20=1.14%, 50=0.16% lat (msec) : 100=0.40%, 250=8.27%, 500=8.23%, 750=3.12%, 1000=0.24% lat (msec) : 2000=0.04% cpu : usr=2.88%, sys=2.04%, ctx=20356, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=18097,4570,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=597MiB/s (626MB/s), 597MiB/s-597MiB/s (626MB/s-626MB/s), io=17.7GiB (18.0GB), run=30315-30315msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4570MiB (4792MB), run=30315-30315msec Disk stats (read/write): vdk: ios=18097/4564, merge=0/0, ticks=14878/1482087, in_queue=1504953, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963558: Fri Jan 7 14:26:23 2022 read: IOPS=615, BW=615MiB/s (645MB/s)(18.0GiB/30004msec) clat (usec): min=130, max=27522, avg=222.30, stdev=917.03 lat (usec): min=131, max=27522, avg=222.46, stdev=917.04 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 190], 90.00th=[ 202], 95.00th=[ 217], | 99.00th=[ 289], 99.50th=[ 420], 99.90th=[16581], 99.95th=[19006], | 99.99th=[24511] bw ( KiB/s): min=468992, max=886784, per=100.00%, avg=629986.68, stdev=91125.84, samples=60 iops : min= 458, max= 866, avg=615.22, stdev=88.99, samples=60 write: IOPS=155, BW=156MiB/s (163MB/s)(4670MiB/30004msec) clat (usec): min=2551, max=59373, avg=5440.01, stdev=3055.91 lat (usec): min=2641, max=59482, avg=5536.88, stdev=3054.64 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3654], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5538], 80.00th=[ 5866], 90.00th=[ 8225], 95.00th=[11076], | 99.00th=[15795], 99.50th=[17695], 99.90th=[55837], 99.95th=[55837], | 99.99th=[59507] bw ( KiB/s): min=133120, max=206848, per=100.00%, avg=159373.58, stdev=10724.49, samples=60 iops : min= 130, max= 202, avg=155.63, stdev=10.48, samples=60 lat (usec) : 250=78.33%, 500=1.12%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=4.84%, 10=13.94%, 20=1.66%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.86%, sys=2.56%, ctx=23135, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18456,4670,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=615MiB/s (645MB/s), 615MiB/s-615MiB/s (645MB/s-645MB/s), io=18.0GiB (19.4GB), run=30004-30004msec WRITE: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4670MiB (4897MB), run=30004-30004msec Disk stats (read/write): vdk: ios=18312/4639, merge=0/0, ticks=983/22621, in_queue=28392, util=94.85% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964378: Fri Jan 7 14:27:00 2022 read: IOPS=594, BW=595MiB/s (624MB/s)(17.8GiB/30641msec) slat (usec): min=9, max=870, avg=19.73, stdev=18.66 clat (usec): min=71, max=922928, avg=3145.32, stdev=16901.43 lat (usec): min=113, max=922953, avg=3165.21, stdev=16902.34 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 131], 60.00th=[ 227], | 70.00th=[ 343], 80.00th=[ 562], 90.00th=[ 7308], 95.00th=[ 20841], | 99.00th=[ 36963], 99.50th=[ 43779], 99.90th=[196084], 99.95th=[252707], | 99.99th=[876610] bw ( KiB/s): min=38989, max=1558528, per=100.00%, avg=622291.82, stdev=232210.98, samples=60 iops : min= 38, max= 1522, avg=607.70, stdev=226.77, samples=60 write: IOPS=150, BW=151MiB/s (158MB/s)(4613MiB/30641msec) slat (usec): min=63, max=1575, avg=206.31, stdev=121.24 clat (msec): min=31, max=2449, avg=651.22, stdev=259.49 lat (msec): min=31, max=2449, avg=651.42, stdev=259.47 clat percentiles (msec): | 1.00th=[ 127], 5.00th=[ 296], 10.00th=[ 393], 20.00th=[ 468], | 30.00th=[ 506], 40.00th=[ 542], 50.00th=[ 600], 60.00th=[ 676], | 70.00th=[ 751], 80.00th=[ 818], 90.00th=[ 953], 95.00th=[ 1116], | 99.00th=[ 1536], 99.50th=[ 1653], 99.90th=[ 1921], 99.95th=[ 1921], | 99.99th=[ 2467] bw ( KiB/s): min= 2052, max=243712, per=98.31%, avg=151561.67, stdev=51764.28, samples=61 iops : min= 2, max= 238, avg=148.00, stdev=50.55, samples=61 lat (usec) : 100=0.12%, 250=48.41%, 500=14.73%, 750=1.89%, 1000=1.07% lat (msec) : 2=2.29%, 4=1.62%, 10=2.55%, 20=2.89%, 50=3.99% lat (msec) : 100=0.13%, 250=0.69%, 500=5.19%, 750=8.45%, 1000=4.34% lat (msec) : 2000=1.63%, >=2000=0.01% cpu : usr=3.22%, sys=1.92%, ctx=17650, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18230,4613,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=595MiB/s (624MB/s), 595MiB/s-595MiB/s (624MB/s-624MB/s), io=17.8GiB (19.1GB), run=30641-30641msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4613MiB (4837MB), run=30641-30641msec Disk stats (read/write): vdk: ios=18230/4594, merge=0/0, ticks=47019/2951942, in_queue=3019882, util=99.54% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965443: Fri Jan 7 14:27:37 2022 read: IOPS=615, BW=615MiB/s (645MB/s)(18.0GiB/30002msec) clat (usec): min=115, max=25332, avg=208.79, stdev=957.88 lat (usec): min=115, max=25332, avg=208.94, stdev=957.88 clat percentiles (usec): | 1.00th=[ 124], 5.00th=[ 129], 10.00th=[ 130], 20.00th=[ 133], | 30.00th=[ 135], 40.00th=[ 139], 50.00th=[ 141], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 172], 90.00th=[ 186], 95.00th=[ 198], | 99.00th=[ 265], 99.50th=[ 375], 99.90th=[17171], 99.95th=[19268], | 99.99th=[24249] bw ( KiB/s): min=462848, max=868352, per=100.00%, avg=630070.35, stdev=90217.05, samples=60 iops : min= 452, max= 848, avg=615.28, stdev=88.09, samples=60 write: IOPS=155, BW=156MiB/s (163MB/s)(4672MiB/30002msec) clat (usec): min=2526, max=57773, avg=5492.11, stdev=2966.86 lat (usec): min=2595, max=57885, avg=5587.85, stdev=2966.08 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3884], | 30.00th=[ 4555], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5866], 90.00th=[ 7898], 95.00th=[11076], | 99.00th=[15664], 99.50th=[17433], 99.90th=[54789], 99.95th=[55837], | 99.99th=[57934] bw ( KiB/s): min=131072, max=212992, per=99.99%, avg=159448.00, stdev=12387.39, samples=60 iops : min= 128, max= 208, avg=155.68, stdev=12.10, samples=60 lat (usec) : 250=78.84%, 500=0.65% lat (msec) : 2=0.01%, 4=4.25%, 10=14.56%, 20=1.63%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=2.00%, sys=1.47%, ctx=23136, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18458,4672,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=615MiB/s (645MB/s), 615MiB/s-615MiB/s (645MB/s-645MB/s), io=18.0GiB (19.4GB), run=30002-30002msec WRITE: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4672MiB (4899MB), run=30002-30002msec Disk stats (read/write): vdk: ios=18438/4665, merge=0/0, ticks=1043/23028, in_queue=28895, util=95.79% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966063: Fri Jan 7 14:28:15 2022 read: IOPS=592, BW=592MiB/s (621MB/s)(17.8GiB/30745msec) slat (usec): min=11, max=803614, avg=1155.29, stdev=9897.32 clat (msec): min=16, max=922, avg=111.17, stdev=113.37 lat (msec): min=17, max=932, avg=112.32, stdev=113.90 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 36], 20.00th=[ 54], | 30.00th=[ 67], 40.00th=[ 81], 50.00th=[ 94], 60.00th=[ 107], | 70.00th=[ 120], 80.00th=[ 136], 90.00th=[ 163], 95.00th=[ 209], | 99.00th=[ 802], 99.50th=[ 852], 99.90th=[ 902], 99.95th=[ 902], | 99.99th=[ 919] bw ( KiB/s): min=38912, max=1386496, per=100.00%, avg=619447.67, stdev=244318.74, samples=60 iops : min= 38, max= 1354, avg=604.92, stdev=238.59, samples=60 write: IOPS=149, BW=150MiB/s (157MB/s)(4604MiB/30745msec) slat (usec): min=272, max=778515, avg=1938.48, stdev=19734.78 clat (msec): min=42, max=2454, avg=888.46, stdev=317.78 lat (msec): min=44, max=2454, avg=890.40, stdev=317.04 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 439], 10.00th=[ 550], 20.00th=[ 667], | 30.00th=[ 726], 40.00th=[ 785], 50.00th=[ 844], 60.00th=[ 919], | 70.00th=[ 995], 80.00th=[ 1099], 90.00th=[ 1284], 95.00th=[ 1469], | 99.00th=[ 1921], 99.50th=[ 2056], 99.90th=[ 2366], 99.95th=[ 2467], | 99.99th=[ 2467] bw ( KiB/s): min= 2048, max=286720, per=97.75%, avg=149889.56, stdev=64159.00, samples=61 iops : min= 2, max= 280, avg=146.36, stdev=62.65, samples=61 lat (msec) : 20=0.02%, 50=14.63%, 100=29.74%, 250=33.29%, 500=1.92% lat (msec) : 750=5.84%, 1000=8.64%, 2000=5.73%, >=2000=0.18% cpu : usr=5.30%, sys=1.38%, ctx=2195, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18205,4604,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=592MiB/s (621MB/s), 592MiB/s-592MiB/s (621MB/s-621MB/s), io=17.8GiB (19.1GB), run=30745-30745msec WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=4604MiB (4828MB), run=30745-30745msec Disk stats (read/write): vdk: ios=18205/4601, merge=0/0, ticks=114381/3615323, in_queue=3743547, util=99.40% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966914: Fri Jan 7 14:28:51 2022 read: IOPS=174, BW=350MiB/s (367MB/s)(10.3GiB/30002msec) clat (usec): min=288, max=16677, avg=505.63, stdev=867.48 lat (usec): min=288, max=16677, avg=505.82, stdev=867.49 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 302], 10.00th=[ 306], 20.00th=[ 314], | 30.00th=[ 322], 40.00th=[ 330], 50.00th=[ 343], 60.00th=[ 359], | 70.00th=[ 400], 80.00th=[ 685], 90.00th=[ 791], 95.00th=[ 832], | 99.00th=[ 1029], 99.50th=[ 8586], 99.90th=[13173], 99.95th=[13829], | 99.99th=[16712] bw ( KiB/s): min=184320, max=593920, per=100.00%, avg=358426.38, stdev=85787.21, samples=60 iops : min= 90, max= 290, avg=175.00, stdev=41.88, samples=60 write: IOPS=45, BW=90.0MiB/s (95.4MB/s)(2730MiB/30002msec) clat (msec): min=5, max=126, avg=19.74, stdev= 9.66 lat (msec): min=5, max=126, avg=20.02, stdev= 9.66 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 10], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 26], 90.00th=[ 34], 95.00th=[ 37], | 99.00th=[ 43], 99.50th=[ 46], 99.90th=[ 65], 99.95th=[ 127], | 99.99th=[ 127] bw ( KiB/s): min=77824, max=110592, per=99.94%, avg=93122.02, stdev=6740.86, samples=60 iops : min= 38, max= 54, avg=45.47, stdev= 3.29, samples=60 lat (usec) : 500=61.33%, 750=4.35%, 1000=12.83% lat (msec) : 2=0.30%, 4=0.02%, 10=4.61%, 20=7.01%, 50=9.46% lat (msec) : 100=0.06%, 250=0.02% cpu : usr=1.33%, sys=1.49%, ctx=11858, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5250,1365,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=350MiB/s (367MB/s), 350MiB/s-350MiB/s (367MB/s-367MB/s), io=10.3GiB (11.0GB), run=30002-30002msec WRITE: bw=90.0MiB/s (95.4MB/s), 90.0MiB/s-90.0MiB/s (95.4MB/s-95.4MB/s), io=2730MiB (2863MB), run=30002-30002msec Disk stats (read/write): vdk: ios=15681/4065, merge=0/0, ticks=490/77857, in_queue=85949, util=97.03% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967510: Fri Jan 7 14:29:28 2022 read: IOPS=265, BW=531MiB/s (557MB/s)(15.6GiB/30001msec) slat (usec): min=52, max=215, avg=71.30, stdev=13.08 clat (usec): min=119, max=22120, avg=355.09, stdev=995.34 lat (usec): min=245, max=22220, avg=426.62, stdev=995.91 clat percentiles (usec): | 1.00th=[ 196], 5.00th=[ 198], 10.00th=[ 200], 20.00th=[ 206], | 30.00th=[ 210], 40.00th=[ 215], 50.00th=[ 219], 60.00th=[ 229], | 70.00th=[ 251], 80.00th=[ 318], 90.00th=[ 529], 95.00th=[ 611], | 99.00th=[ 766], 99.50th=[ 9503], 99.90th=[15795], 99.95th=[16188], | 99.99th=[22152] bw ( KiB/s): min=258048, max=839680, per=100.00%, avg=544027.42, stdev=118846.24, samples=59 iops : min= 126, max= 410, avg=265.63, stdev=58.02, samples=59 write: IOPS=67, BW=135MiB/s (141MB/s)(4046MiB/30001msec) slat (usec): min=130, max=802, avg=298.85, stdev=109.46 clat (usec): min=4653, max=46742, avg=12840.69, stdev=3613.45 lat (usec): min=4895, max=46992, avg=13139.86, stdev=3628.70 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7373], 10.00th=[ 8586], 20.00th=[ 9765], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12911], 60.00th=[14091], | 70.00th=[14877], 80.00th=[15664], 90.00th=[16581], 95.00th=[17433], | 99.00th=[23200], 99.50th=[25560], 99.90th=[39584], 99.95th=[42206], | 99.99th=[46924] bw ( KiB/s): min=110592, max=151552, per=99.99%, avg=138088.24, stdev=7603.93, samples=59 iops : min= 54, max= 74, avg=67.42, stdev= 3.71, samples=59 lat (usec) : 250=55.56%, 500=15.18%, 750=8.10%, 1000=0.35% lat (msec) : 2=0.03%, 4=0.02%, 10=4.60%, 20=15.70%, 50=0.46% cpu : usr=1.79%, sys=2.63%, ctx=9996, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7964,2023,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=531MiB/s (557MB/s), 531MiB/s-531MiB/s (557MB/s-557MB/s), io=15.6GiB (16.7GB), run=30001-30001msec WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4046MiB (4243MB), run=30001-30001msec Disk stats (read/write): vdk: ios=15876/4024, merge=0/0, ticks=652/49533, in_queue=56760, util=96.21% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968309: Fri Jan 7 14:30:05 2022 read: IOPS=269, BW=539MiB/s (566MB/s)(15.8GiB/30003msec) clat (usec): min=226, max=19222, avg=386.42, stdev=983.67 lat (usec): min=226, max=19222, avg=386.60, stdev=983.67 clat percentiles (usec): | 1.00th=[ 235], 5.00th=[ 241], 10.00th=[ 245], 20.00th=[ 249], | 30.00th=[ 253], 40.00th=[ 258], 50.00th=[ 265], 60.00th=[ 269], | 70.00th=[ 289], 80.00th=[ 343], 90.00th=[ 490], 95.00th=[ 635], | 99.00th=[ 799], 99.50th=[ 9765], 99.90th=[15664], 99.95th=[16188], | 99.99th=[19268] bw ( KiB/s): min=290816, max=897024, per=100.00%, avg=552361.47, stdev=128336.96, samples=60 iops : min= 142, max= 438, avg=269.70, stdev=62.67, samples=60 write: IOPS=68, BW=137MiB/s (144MB/s)(4108MiB/30003msec) clat (usec): min=4662, max=49104, avg=12844.78, stdev=3530.28 lat (usec): min=4862, max=49296, avg=13073.99, stdev=3538.26 clat percentiles (usec): | 1.00th=[ 4948], 5.00th=[ 7373], 10.00th=[ 8455], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12911], 60.00th=[14222], | 70.00th=[15008], 80.00th=[15795], 90.00th=[16581], 95.00th=[17171], | 99.00th=[22676], 99.50th=[24249], 99.90th=[25822], 99.95th=[44303], | 99.99th=[49021] bw ( KiB/s): min=126976, max=159744, per=99.97%, avg=140156.52, stdev=6965.05, samples=60 iops : min= 62, max= 78, avg=68.43, stdev= 3.40, samples=60 lat (usec) : 250=18.31%, 500=53.71%, 750=6.63%, 1000=0.52% lat (msec) : 2=0.07%, 4=0.02%, 10=4.37%, 20=16.01%, 50=0.35% cpu : usr=1.83%, sys=1.46%, ctx=10152, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8091,2054,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=539MiB/s (566MB/s), 539MiB/s-539MiB/s (566MB/s-566MB/s), io=15.8GiB (16.0GB), run=30003-30003msec WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=4108MiB (4308MB), run=30003-30003msec Disk stats (read/write): vdk: ios=16100/4083, merge=0/0, ticks=655/41838, in_queue=48756, util=96.45% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969568: Fri Jan 7 14:30:42 2022 read: IOPS=351, BW=702MiB/s (737MB/s)(20.7GiB/30120msec) slat (usec): min=20, max=314, avg=38.24, stdev=13.73 clat (usec): min=56, max=869127, avg=889.82, stdev=16891.83 lat (usec): min=205, max=869442, avg=928.29, stdev=16893.25 clat percentiles (usec): | 1.00th=[ 184], 5.00th=[ 188], 10.00th=[ 190], 20.00th=[ 194], | 30.00th=[ 198], 40.00th=[ 202], 50.00th=[ 208], 60.00th=[ 215], | 70.00th=[ 237], 80.00th=[ 619], 90.00th=[ 652], 95.00th=[ 676], | 99.00th=[ 898], 99.50th=[ 1926], 99.90th=[164627], 99.95th=[480248], | 99.99th=[692061] bw ( KiB/s): min=512000, max=1105920, per=100.00%, avg=722222.58, stdev=123865.56, samples=60 iops : min= 250, max= 540, avg=352.60, stdev=60.47, samples=60 write: IOPS=89, BW=179MiB/s (187MB/s)(5378MiB/30120msec) slat (usec): min=119, max=956, avg=347.14, stdev=179.85 clat (msec): min=7, max=2495, avg=107.96, stdev=140.29 lat (msec): min=8, max=2495, avg=108.30, stdev=140.30 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 19], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 39], 40.00th=[ 46], 50.00th=[ 59], 60.00th=[ 80], | 70.00th=[ 106], 80.00th=[ 150], 90.00th=[ 255], 95.00th=[ 372], | 99.00th=[ 659], 99.50th=[ 743], 99.90th=[ 1267], 99.95th=[ 1770], | 99.99th=[ 2500] bw ( KiB/s): min=163840, max=200704, per=100.00%, avg=182910.87, stdev=8693.22, samples=60 iops : min= 80, max= 98, avg=89.30, stdev= 4.24, samples=60 lat (usec) : 100=0.01%, 250=57.44%, 500=5.15%, 750=15.71%, 1000=0.81% lat (msec) : 2=0.22%, 4=0.10%, 10=0.14%, 20=1.91%, 50=6.87% lat (msec) : 100=5.13%, 250=4.39%, 500=1.63%, 750=0.40%, 1000=0.06% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=3.08%, sys=2.04%, ctx=13249, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10578,2689,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=702MiB/s (737MB/s), 702MiB/s-702MiB/s (737MB/s-737MB/s), io=20.7GiB (22.2GB), run=30120-30120msec WRITE: bw=179MiB/s (187MB/s), 179MiB/s-179MiB/s (187MB/s-187MB/s), io=5378MiB (5639MB), run=30120-30120msec Disk stats (read/write): vdk: ios=21156/5368, merge=0/0, ticks=6273/573248, in_queue=591580, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970154: Fri Jan 7 14:31:19 2022 read: IOPS=270, BW=540MiB/s (566MB/s)(15.8GiB/30011msec) clat (usec): min=217, max=19193, avg=360.05, stdev=1003.23 lat (usec): min=217, max=19194, avg=360.22, stdev=1003.23 clat percentiles (usec): | 1.00th=[ 223], 5.00th=[ 229], 10.00th=[ 233], 20.00th=[ 237], | 30.00th=[ 241], 40.00th=[ 245], 50.00th=[ 249], 60.00th=[ 258], | 70.00th=[ 273], 80.00th=[ 326], 90.00th=[ 383], 95.00th=[ 457], | 99.00th=[ 734], 99.50th=[ 9896], 99.90th=[15795], 99.95th=[16450], | 99.99th=[19268] bw ( KiB/s): min=266240, max=827392, per=100.00%, avg=553418.28, stdev=120686.42, samples=60 iops : min= 130, max= 404, avg=270.20, stdev=58.94, samples=60 write: IOPS=68, BW=137MiB/s (144MB/s)(4116MiB/30011msec) clat (usec): min=4660, max=46145, avg=12942.86, stdev=3472.33 lat (usec): min=4833, max=46363, avg=13154.36, stdev=3476.56 clat percentiles (usec): | 1.00th=[ 5080], 5.00th=[ 7439], 10.00th=[ 8586], 20.00th=[10028], | 30.00th=[11076], 40.00th=[11731], 50.00th=[13042], 60.00th=[14353], | 70.00th=[15139], 80.00th=[15795], 90.00th=[16712], 95.00th=[17433], | 99.00th=[22152], 99.50th=[24511], 99.90th=[31589], 99.95th=[40109], | 99.99th=[46400] bw ( KiB/s): min=118784, max=159744, per=100.00%, avg=140438.18, stdev=7500.12, samples=60 iops : min= 58, max= 78, avg=68.57, stdev= 3.67, samples=60 lat (usec) : 250=40.23%, 500=36.69%, 750=2.15%, 1000=0.18% lat (msec) : 4=0.01%, 10=4.15%, 20=16.29%, 50=0.30% cpu : usr=1.59%, sys=1.22%, ctx=10171, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8106,2058,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=540MiB/s (566MB/s), 540MiB/s-540MiB/s (566MB/s-566MB/s), io=15.8GiB (16.0GB), run=30011-30011msec WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=4116MiB (4316MB), run=30011-30011msec Disk stats (read/write): vdk: ios=16126/4088, merge=0/0, ticks=662/41412, in_queue=48067, util=96.78% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971112: Fri Jan 7 14:31:56 2022 read: IOPS=357, BW=714MiB/s (749MB/s)(21.4GiB/30620msec) slat (usec): min=19, max=581, avg=33.77, stdev=18.01 clat (usec): min=95, max=2839.3k, avg=3880.43, stdev=62758.62 lat (usec): min=205, max=2839.3k, avg=3914.40, stdev=62758.72 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 190], 10.00th=[ 192], | 20.00th=[ 196], 30.00th=[ 202], 40.00th=[ 206], | 50.00th=[ 215], 60.00th=[ 243], 70.00th=[ 594], | 80.00th=[ 660], 90.00th=[ 1188], 95.00th=[ 8586], | 99.00th=[ 28967], 99.50th=[ 45351], 99.90th=[1098908], | 99.95th=[1535116], 99.99th=[2533360] bw ( KiB/s): min=307200, max=1306624, per=100.00%, avg=746169.13, stdev=173411.74, samples=60 iops : min= 150, max= 638, avg=364.30, stdev=84.65, samples=60 write: IOPS=90, BW=182MiB/s (190MB/s)(5560MiB/30620msec) slat (usec): min=119, max=1160, avg=384.29, stdev=203.51 clat (msec): min=31, max=3731, avg=534.62, stdev=310.57 lat (msec): min=32, max=3732, avg=535.00, stdev=310.59 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 271], 10.00th=[ 326], 20.00th=[ 372], | 30.00th=[ 393], 40.00th=[ 409], 50.00th=[ 439], 60.00th=[ 481], | 70.00th=[ 542], 80.00th=[ 642], 90.00th=[ 835], 95.00th=[ 1099], | 99.00th=[ 1955], 99.50th=[ 2265], 99.90th=[ 2869], 99.95th=[ 3004], | 99.99th=[ 3742] bw ( KiB/s): min= 4096, max=225280, per=98.74%, avg=183601.39, stdev=27921.31, samples=61 iops : min= 2, max= 110, avg=89.64, stdev=13.63, samples=61 lat (usec) : 100=0.01%, 250=48.67%, 500=6.69%, 750=14.62%, 1000=1.41% lat (msec) : 2=1.33%, 4=1.40%, 10=2.01%, 20=2.07%, 50=1.14% lat (msec) : 100=0.23%, 250=0.79%, 500=12.21%, 750=4.63%, 1000=1.39% lat (msec) : 2000=1.17%, >=2000=0.21% cpu : usr=3.67%, sys=1.64%, ctx=12131, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=10932,2780,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=714MiB/s (749MB/s), 714MiB/s-714MiB/s (749MB/s-749MB/s), io=21.4GiB (22.9GB), run=30620-30620msec WRITE: bw=182MiB/s (190MB/s), 182MiB/s-182MiB/s (190MB/s-190MB/s), io=5560MiB (5830MB), run=30620-30620msec Disk stats (read/write): vdk: ios=21864/5550, merge=0/0, ticks=46400/2935258, in_queue=3002113, util=99.60% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972158: Fri Jan 7 14:32:33 2022 read: IOPS=269, BW=539MiB/s (565MB/s)(15.8GiB/30004msec) clat (usec): min=212, max=24570, avg=350.98, stdev=1017.32 lat (usec): min=212, max=24570, avg=351.15, stdev=1017.33 clat percentiles (usec): | 1.00th=[ 219], 5.00th=[ 225], 10.00th=[ 229], 20.00th=[ 233], | 30.00th=[ 237], 40.00th=[ 241], 50.00th=[ 247], 60.00th=[ 255], | 70.00th=[ 273], 80.00th=[ 314], 90.00th=[ 355], 95.00th=[ 412], | 99.00th=[ 725], 99.50th=[ 9372], 99.90th=[15926], 99.95th=[16450], | 99.99th=[24511] bw ( KiB/s): min=290816, max=860160, per=100.00%, avg=551850.55, stdev=120890.16, samples=60 iops : min= 142, max= 420, avg=269.45, stdev=59.03, samples=60 write: IOPS=68, BW=137MiB/s (143MB/s)(4098MiB/30004msec) clat (usec): min=4655, max=42033, avg=13041.37, stdev=3529.47 lat (usec): min=4841, max=42234, avg=13248.20, stdev=3526.99 clat percentiles (usec): | 1.00th=[ 5342], 5.00th=[ 7504], 10.00th=[ 8717], 20.00th=[10028], | 30.00th=[11076], 40.00th=[11863], 50.00th=[13304], 60.00th=[14353], | 70.00th=[15139], 80.00th=[15926], 90.00th=[16712], 95.00th=[17433], | 99.00th=[23987], 99.50th=[25297], 99.90th=[41157], 99.95th=[41157], | 99.99th=[42206] bw ( KiB/s): min=122880, max=155648, per=99.96%, avg=139805.35, stdev=7442.98, samples=60 iops : min= 60, max= 76, avg=68.25, stdev= 3.63, samples=60 lat (usec) : 250=43.54%, 500=34.07%, 750=1.52%, 1000=0.14% lat (msec) : 2=0.01%, 4=0.02%, 10=4.00%, 20=16.34%, 50=0.37% cpu : usr=1.50%, sys=1.17%, ctx=10141, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8084,2049,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=15.8GiB (16.0GB), run=30004-30004msec WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4098MiB (4297MB), run=30004-30004msec Disk stats (read/write): vdk: ios=16146/4093, merge=0/0, ticks=662/42039, in_queue=48585, util=96.64% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972682: Fri Jan 7 14:33:11 2022 read: IOPS=356, BW=712MiB/s (747MB/s)(21.4GiB/30722msec) slat (usec): min=21, max=503061, avg=2156.82, stdev=12400.57 clat (msec): min=19, max=2873, avg=88.30, stdev=95.14 lat (msec): min=19, max=2873, avg=90.46, stdev=96.20 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 35], 20.00th=[ 48], | 30.00th=[ 57], 40.00th=[ 65], 50.00th=[ 73], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 107], 90.00th=[ 127], 95.00th=[ 150], | 99.00th=[ 542], 99.50th=[ 592], 99.90th=[ 1011], 99.95th=[ 1385], | 99.99th=[ 2702] bw ( KiB/s): min= 4096, max=1482752, per=100.00%, avg=732781.11, stdev=274264.55, samples=61 iops : min= 2, max= 724, avg=357.80, stdev=133.92, samples=61 write: IOPS=90, BW=181MiB/s (190MB/s)(5562MiB/30722msec) slat (usec): min=546, max=83226, avg=2288.95, stdev=5278.77 clat (msec): min=122, max=3092, avg=745.71, stdev=362.16 lat (msec): min=122, max=3092, avg=748.00, stdev=362.10 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 388], 10.00th=[ 443], 20.00th=[ 527], | 30.00th=[ 575], 40.00th=[ 609], 50.00th=[ 642], 60.00th=[ 693], | 70.00th=[ 776], 80.00th=[ 927], 90.00th=[ 1150], 95.00th=[ 1385], | 99.00th=[ 2400], 99.50th=[ 2668], 99.90th=[ 3037], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 4096, max=307200, per=99.72%, avg=184866.13, stdev=61217.29, samples=60 iops : min= 2, max= 150, avg=90.27, stdev=29.89, samples=60 lat (msec) : 20=0.01%, 50=18.19%, 100=42.05%, 250=17.50%, 500=4.08% lat (msec) : 750=11.39%, 1000=3.46%, 2000=2.94%, >=2000=0.38% cpu : usr=6.17%, sys=1.72%, ctx=2743, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10939,2781,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=712MiB/s (747MB/s), 712MiB/s-712MiB/s (747MB/s-747MB/s), io=21.4GiB (22.9GB), run=30722-30722msec WRITE: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=5562MiB (5832MB), run=30722-30722msec Disk stats (read/write): vdk: ios=21878/5528, merge=0/0, ticks=117805/3587910, in_queue=3754420, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973644: Fri Jan 7 14:33:47 2022 read: IOPS=269, BW=539MiB/s (565MB/s)(15.8GiB/30007msec) clat (usec): min=219, max=19024, avg=365.26, stdev=965.09 lat (usec): min=219, max=19024, avg=365.44, stdev=965.10 clat percentiles (usec): | 1.00th=[ 227], 5.00th=[ 235], 10.00th=[ 239], 20.00th=[ 243], | 30.00th=[ 249], 40.00th=[ 253], 50.00th=[ 262], 60.00th=[ 273], | 70.00th=[ 289], 80.00th=[ 334], 90.00th=[ 383], 95.00th=[ 461], | 99.00th=[ 717], 99.50th=[ 9765], 99.90th=[15664], 99.95th=[16188], | 99.99th=[19006] bw ( KiB/s): min=290816, max=864256, per=100.00%, avg=551912.32, stdev=124590.58, samples=60 iops : min= 142, max= 422, avg=269.47, stdev=60.81, samples=60 write: IOPS=68, BW=137MiB/s (143MB/s)(4098MiB/30007msec) clat (usec): min=4646, max=49960, avg=12987.16, stdev=3526.08 lat (usec): min=4851, max=50131, avg=13192.97, stdev=3527.44 clat percentiles (usec): | 1.00th=[ 5014], 5.00th=[ 7570], 10.00th=[ 8586], 20.00th=[10028], | 30.00th=[11076], 40.00th=[11863], 50.00th=[13173], 60.00th=[14353], | 70.00th=[15139], 80.00th=[15795], 90.00th=[16581], 95.00th=[17433], | 99.00th=[23462], 99.50th=[25822], 99.90th=[36963], 99.95th=[40633], | 99.99th=[50070] bw ( KiB/s): min=126976, max=155648, per=99.98%, avg=139819.42, stdev=6374.88, samples=60 iops : min= 62, max= 76, avg=68.27, stdev= 3.11, samples=60 lat (usec) : 250=26.59%, 500=50.19%, 750=2.28%, 1000=0.16% lat (msec) : 2=0.06%, 4=0.01%, 10=4.17%, 20=16.22%, 50=0.32% cpu : usr=1.57%, sys=1.36%, ctx=10138, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8084,2049,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=15.8GiB (16.0GB), run=30007-30007msec WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4098MiB (4297MB), run=30007-30007msec Disk stats (read/write): vdk: ios=16052/4074, merge=0/0, ticks=642/41730, in_queue=48321, util=96.03% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974267: Fri Jan 7 14:34:25 2022 read: IOPS=355, BW=711MiB/s (746MB/s)(21.3GiB/30725msec) slat (usec): min=22, max=502052, avg=1976.46, stdev=9253.88 clat (msec): min=86, max=2903, avg=316.18, stdev=150.46 lat (msec): min=86, max=2903, avg=318.16, stdev=150.81 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 130], 10.00th=[ 184], 20.00th=[ 232], | 30.00th=[ 262], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 313], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 439], 95.00th=[ 634], | 99.00th=[ 844], 99.50th=[ 885], 99.90th=[ 1603], 99.95th=[ 2165], | 99.99th=[ 2903] bw ( KiB/s): min=73728, max=1245184, per=100.00%, avg=737904.48, stdev=240403.65, samples=60 iops : min= 36, max= 608, avg=360.25, stdev=117.36, samples=60 write: IOPS=90, BW=181MiB/s (189MB/s)(5546MiB/30725msec) slat (usec): min=506, max=510662, avg=3014.87, stdev=17271.53 clat (msec): min=126, max=3504, avg=954.90, stdev=336.73 lat (msec): min=278, max=3505, avg=957.92, stdev=335.96 clat percentiles (msec): | 1.00th=[ 464], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1133], 90.00th=[ 1351], 95.00th=[ 1552], | 99.00th=[ 2333], 99.50th=[ 2668], 99.90th=[ 3239], 99.95th=[ 3272], | 99.99th=[ 3507] bw ( KiB/s): min= 4096, max=385024, per=99.52%, avg=183944.97, stdev=78579.40, samples=60 iops : min= 2, max= 188, avg=89.80, stdev=38.36, samples=60 lat (msec) : 100=1.03%, 250=19.26%, 500=54.22%, 750=7.94%, 1000=11.33% lat (msec) : 2000=5.81%, >=2000=0.42% cpu : usr=6.31%, sys=1.74%, ctx=2724, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10927,2773,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=711MiB/s (746MB/s), 711MiB/s-711MiB/s (746MB/s-746MB/s), io=21.3GiB (22.9GB), run=30725-30725msec WRITE: bw=181MiB/s (189MB/s), 181MiB/s-181MiB/s (189MB/s-189MB/s), io=5546MiB (5815MB), run=30725-30725msec Disk stats (read/write): vdk: ios=21853/5528, merge=0/0, ticks=128042/3583900, in_queue=3751168, util=98.97% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975109: Fri Jan 7 14:35:01 2022 read: IOPS=160, BW=640MiB/s (672MB/s)(18.8GiB/30009msec) clat (usec): min=420, max=23297, avg=836.37, stdev=1376.26 lat (usec): min=421, max=23298, avg=836.57, stdev=1376.27 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 445], 10.00th=[ 449], 20.00th=[ 457], | 30.00th=[ 469], 40.00th=[ 478], 50.00th=[ 490], 60.00th=[ 515], | 70.00th=[ 611], 80.00th=[ 1336], 90.00th=[ 1385], 95.00th=[ 1450], | 99.00th=[ 8848], 99.50th=[12518], 99.90th=[17433], 99.95th=[20317], | 99.99th=[23200] bw ( KiB/s): min=393216, max=1089536, per=100.00%, avg=656080.40, stdev=163167.03, samples=60 iops : min= 96, max= 266, avg=160.17, stdev=39.84, samples=60 write: IOPS=41, BW=165MiB/s (173MB/s)(4956MiB/30009msec) clat (usec): min=10818, max=72217, avg=20382.21, stdev=4798.60 lat (usec): min=11280, max=72609, avg=20963.66, stdev=4839.47 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14091], 10.00th=[14877], 20.00th=[16057], | 30.00th=[17171], 40.00th=[18744], 50.00th=[20317], 60.00th=[21890], | 70.00th=[22938], 80.00th=[23987], 90.00th=[25035], 95.00th=[26346], | 99.00th=[33424], 99.50th=[41157], 99.90th=[50070], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min=147456, max=188416, per=99.96%, avg=169040.30, stdev=9785.05, samples=60 iops : min= 36, max= 46, avg=41.27, stdev= 2.39, samples=60 lat (usec) : 500=43.33%, 750=17.12%, 1000=0.78% lat (msec) : 2=17.22%, 4=0.10%, 10=0.26%, 20=10.59%, 50=10.56% lat (msec) : 100=0.03% cpu : usr=2.58%, sys=2.04%, ctx=6055, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4805,1239,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=640MiB/s (672MB/s), 640MiB/s-640MiB/s (672MB/s-672MB/s), io=18.8GiB (20.2GB), run=30009-30009msec WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=4956MiB (5197MB), run=30009-30009msec Disk stats (read/write): vdk: ios=19020/4922, merge=0/0, ticks=4917/81438, in_queue=96418, util=96.42% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976337: Fri Jan 7 14:35:38 2022 read: IOPS=161, BW=648MiB/s (679MB/s)(18.0GiB/30007msec) slat (usec): min=73, max=258, avg=96.34, stdev=18.18 clat (usec): min=289, max=23282, avg=698.52, stdev=1430.42 lat (usec): min=403, max=23371, avg=795.15, stdev=1431.40 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 334], 10.00th=[ 338], 20.00th=[ 347], | 30.00th=[ 355], 40.00th=[ 363], 50.00th=[ 367], 60.00th=[ 379], | 70.00th=[ 424], 80.00th=[ 1037], 90.00th=[ 1221], 95.00th=[ 1270], | 99.00th=[ 8979], 99.50th=[13173], 99.90th=[17957], 99.95th=[20055], | 99.99th=[23200] bw ( KiB/s): min=278528, max=1073152, per=100.00%, avg=663585.92, stdev=155415.53, samples=60 iops : min= 68, max= 262, avg=162.00, stdev=37.93, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5012MiB/30007msec) slat (usec): min=265, max=1745, avg=641.56, stdev=316.85 clat (msec): min=10, max=121, avg=20.21, stdev= 5.99 lat (msec): min=11, max=122, avg=20.85, stdev= 6.03 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 28], | 99.00th=[ 36], 99.50th=[ 41], 99.90th=[ 86], 99.95th=[ 123], | 99.99th=[ 123] bw ( KiB/s): min=139264, max=188416, per=99.95%, avg=170945.47, stdev=10608.74, samples=60 iops : min= 34, max= 46, avg=41.73, stdev= 2.59, samples=60 lat (usec) : 500=59.19%, 750=2.27%, 1000=1.72% lat (msec) : 2=15.31%, 4=0.02%, 10=0.31%, 20=11.27%, 50=9.86% lat (msec) : 100=0.03%, 250=0.02% cpu : usr=2.55%, sys=2.00%, ctx=6126, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4860,1253,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=648MiB/s (679MB/s), 648MiB/s-648MiB/s (679MB/s-679MB/s), io=18.0GiB (20.4GB), run=30007-30007msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5012MiB (5255MB), run=30007-30007msec Disk stats (read/write): vdk: ios=19340/4985, merge=0/0, ticks=4644/81438, in_queue=95758, util=96.58% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977096: Fri Jan 7 14:36:15 2022 read: IOPS=163, BW=654MiB/s (685MB/s)(19.2GiB/30010msec) clat (usec): min=372, max=23501, avg=740.17, stdev=1440.36 lat (usec): min=372, max=23501, avg=740.37, stdev=1440.37 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 400], | 30.00th=[ 404], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 474], 80.00th=[ 979], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[ 9110], 99.50th=[13173], 99.90th=[18220], 99.95th=[20317], | 99.99th=[23462] bw ( KiB/s): min=335872, max=1081344, per=100.00%, avg=669443.17, stdev=150811.55, samples=60 iops : min= 82, max= 264, avg=163.43, stdev=36.82, samples=60 write: IOPS=42, BW=168MiB/s (177MB/s)(5052MiB/30010msec) clat (usec): min=10831, max=81947, avg=20312.49, stdev=5459.01 lat (usec): min=11229, max=82381, avg=20875.10, stdev=5484.48 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13829], 10.00th=[14615], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20055], 60.00th=[21890], | 70.00th=[22938], 80.00th=[23987], 90.00th=[25035], 95.00th=[26084], | 99.00th=[39584], 99.50th=[50070], 99.90th=[66847], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min=147456, max=196608, per=99.96%, avg=172310.53, stdev=10657.19, samples=60 iops : min= 36, max= 48, avg=42.07, stdev= 2.60, samples=60 lat (usec) : 500=57.07%, 750=4.51%, 1000=2.32% lat (msec) : 2=14.60%, 4=0.03%, 10=0.28%, 20=10.98%, 50=10.09% lat (msec) : 100=0.13% cpu : usr=2.54%, sys=1.35%, ctx=6197, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4903,1263,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=654MiB/s (685MB/s), 654MiB/s-654MiB/s (685MB/s-685MB/s), io=19.2GiB (20.6GB), run=30010-30010msec WRITE: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=5052MiB (5297MB), run=30010-30010msec Disk stats (read/write): vdk: ios=19495/5016, merge=0/0, ticks=4256/82109, in_queue=95953, util=96.73% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977962: Fri Jan 7 14:36:52 2022 read: IOPS=186, BW=745MiB/s (782MB/s)(21.0GiB/30187msec) slat (usec): min=40, max=327, avg=61.24, stdev=18.45 clat (usec): min=256, max=2534.4k, avg=5722.82, stdev=74920.72 lat (usec): min=370, max=2534.4k, avg=5784.33, stdev=74920.44 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 338], 10.00th=[ 343], | 20.00th=[ 351], 30.00th=[ 359], 40.00th=[ 367], | 50.00th=[ 375], 60.00th=[ 392], 70.00th=[ 494], | 80.00th=[ 1221], 90.00th=[ 1270], 95.00th=[ 1352], | 99.00th=[ 35390], 99.50th=[ 299893], 99.90th=[1400898], | 99.95th=[1904215], 99.99th=[2533360] bw ( KiB/s): min=368640, max=1433600, per=100.00%, avg=768104.77, stdev=204818.66, samples=60 iops : min= 90, max= 350, avg=187.50, stdev=49.96, samples=60 write: IOPS=48, BW=196MiB/s (205MB/s)(5908MiB/30187msec) slat (usec): min=239, max=1869, avg=792.66, stdev=394.07 clat (msec): min=18, max=2536, avg=181.41, stdev=222.69 lat (msec): min=18, max=2537, avg=182.20, stdev=222.70 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 55], | 30.00th=[ 69], 40.00th=[ 83], 50.00th=[ 109], 60.00th=[ 142], | 70.00th=[ 190], 80.00th=[ 257], 90.00th=[ 388], 95.00th=[ 550], | 99.00th=[ 1150], 99.50th=[ 1301], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=163840, max=221184, per=100.00%, avg=200450.90, stdev=10389.76, samples=60 iops : min= 40, max= 54, avg=48.93, stdev= 2.54, samples=60 lat (usec) : 500=55.74%, 750=2.21%, 1000=0.94% lat (msec) : 2=18.46%, 4=0.35%, 10=0.28%, 20=0.23%, 50=3.49% lat (msec) : 100=6.62%, 250=6.90%, 500=3.31%, 750=0.82%, 1000=0.24% lat (msec) : 2000=0.34%, >=2000=0.08% cpu : usr=3.80%, sys=1.59%, ctx=7012, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5626,1477,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=745MiB/s (782MB/s), 745MiB/s-745MiB/s (782MB/s-782MB/s), io=21.0GiB (23.6GB), run=30187-30187msec WRITE: bw=196MiB/s (205MB/s), 196MiB/s-196MiB/s (205MB/s-205MB/s), io=5908MiB (6195MB), run=30187-30187msec Disk stats (read/write): vdk: ios=22504/5892, merge=0/0, ticks=40021/1041942, in_queue=1097460, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978558: Fri Jan 7 14:37:29 2022 read: IOPS=163, BW=654MiB/s (686MB/s)(19.2GiB/30001msec) clat (usec): min=377, max=29060, avg=755.06, stdev=1485.37 lat (usec): min=377, max=29060, avg=755.27, stdev=1485.38 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 482], 80.00th=[ 988], 90.00th=[ 1270], 95.00th=[ 1336], | 99.00th=[ 9372], 99.50th=[13566], 99.90th=[18482], 99.95th=[21627], | 99.99th=[28967] bw ( KiB/s): min=376832, max=1007616, per=100.00%, avg=670312.27, stdev=143593.67, samples=59 iops : min= 92, max= 246, avg=163.63, stdev=35.06, samples=59 write: IOPS=42, BW=168MiB/s (177MB/s)(5052MiB/30001msec) clat (usec): min=10711, max=74729, avg=20242.95, stdev=4946.76 lat (usec): min=11143, max=75153, avg=20808.91, stdev=4990.97 clat percentiles (usec): | 1.00th=[12518], 5.00th=[13960], 10.00th=[14615], 20.00th=[15664], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20055], 60.00th=[21890], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[26346], | 99.00th=[32900], 99.50th=[34341], 99.90th=[56886], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=155648, max=188416, per=100.00%, avg=172436.61, stdev=10360.58, samples=59 iops : min= 38, max= 46, avg=42.07, stdev= 2.52, samples=59 lat (usec) : 500=56.76%, 750=4.75%, 1000=2.20% lat (msec) : 2=14.75%, 4=0.06%, 10=0.24%, 20=10.96%, 50=10.23% lat (msec) : 100=0.03% cpu : usr=2.63%, sys=1.75%, ctx=6179, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4905,1263,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=654MiB/s (686MB/s), 654MiB/s-654MiB/s (686MB/s-686MB/s), io=19.2GiB (20.6GB), run=30001-30001msec WRITE: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=5052MiB (5297MB), run=30001-30001msec Disk stats (read/write): vdk: ios=19612/5045, merge=0/0, ticks=4340/82026, in_queue=95975, util=96.44% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979524: Fri Jan 7 14:38:07 2022 read: IOPS=187, BW=752MiB/s (788MB/s)(22.5GiB/30627msec) slat (usec): min=45, max=589472, avg=3978.43, stdev=14514.73 clat (msec): min=16, max=2882, avg=81.08, stdev=112.55 lat (msec): min=16, max=2882, avg=85.06, stdev=113.67 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 30], 20.00th=[ 38], | 30.00th=[ 46], 40.00th=[ 54], 50.00th=[ 63], 60.00th=[ 72], | 70.00th=[ 85], 80.00th=[ 101], 90.00th=[ 126], 95.00th=[ 150], | 99.00th=[ 514], 99.50th=[ 651], 99.90th=[ 2005], 99.95th=[ 2467], | 99.99th=[ 2869] bw ( KiB/s): min= 8192, max=1712128, per=100.00%, avg=771222.44, stdev=338527.26, samples=61 iops : min= 2, max= 418, avg=188.26, stdev=82.64, samples=61 write: IOPS=48, BW=196MiB/s (205MB/s)(6000MiB/30627msec) slat (usec): min=1119, max=444426, avg=4710.31, stdev=17794.39 clat (msec): min=81, max=3113, avg=688.69, stdev=264.24 lat (msec): min=82, max=3162, avg=693.40, stdev=264.35 clat percentiles (msec): | 1.00th=[ 222], 5.00th=[ 384], 10.00th=[ 451], 20.00th=[ 527], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 625], 60.00th=[ 667], | 70.00th=[ 726], 80.00th=[ 827], 90.00th=[ 1011], 95.00th=[ 1217], | 99.00th=[ 1519], 99.50th=[ 1754], 99.90th=[ 3004], 99.95th=[ 3104], | 99.99th=[ 3104] bw ( KiB/s): min= 8192, max=385024, per=100.00%, avg=203576.51, stdev=65422.59, samples=59 iops : min= 2, max= 94, avg=49.69, stdev=15.97, samples=59 lat (msec) : 20=0.43%, 50=28.36%, 100=34.79%, 250=14.41%, 500=3.42% lat (msec) : 750=12.72%, 1000=3.56%, 2000=2.20%, >=2000=0.11% cpu : usr=6.68%, sys=1.62%, ctx=2681, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5757,1500,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=752MiB/s (788MB/s), 752MiB/s-752MiB/s (788MB/s-788MB/s), io=22.5GiB (24.1GB), run=30627-30627msec WRITE: bw=196MiB/s (205MB/s), 196MiB/s-196MiB/s (205MB/s-205MB/s), io=6000MiB (6291MB), run=30627-30627msec Disk stats (read/write): vdk: ios=23028/5929, merge=0/0, ticks=140534/3568905, in_queue=3770459, util=99.35% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980490: Fri Jan 7 14:38:43 2022 read: IOPS=163, BW=654MiB/s (686MB/s)(19.2GiB/30022msec) clat (usec): min=374, max=23476, avg=750.86, stdev=1491.71 lat (usec): min=374, max=23476, avg=751.08, stdev=1491.71 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 469], 80.00th=[ 988], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[ 9896], 99.50th=[14091], 99.90th=[18482], 99.95th=[21103], | 99.99th=[23462] bw ( KiB/s): min=294912, max=1196032, per=100.00%, avg=670264.57, stdev=162330.26, samples=60 iops : min= 72, max= 292, avg=163.63, stdev=39.63, samples=60 write: IOPS=42, BW=169MiB/s (177MB/s)(5060MiB/30022msec) clat (usec): min=10655, max=80823, avg=20229.94, stdev=5008.95 lat (usec): min=11171, max=81226, avg=20806.64, stdev=5054.64 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13960], 10.00th=[14615], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18220], 50.00th=[20055], 60.00th=[21890], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[26346], | 99.00th=[33162], 99.50th=[35914], 99.90th=[57410], 99.95th=[81265], | 99.99th=[81265] bw ( KiB/s): min=147456, max=188416, per=100.00%, avg=172583.87, stdev=9402.79, samples=60 iops : min= 36, max= 46, avg=42.13, stdev= 2.30, samples=60 lat (usec) : 500=57.81%, 750=3.61%, 1000=2.32% lat (msec) : 2=14.74%, 4=0.05%, 10=0.19%, 20=10.87%, 50=10.37% lat (msec) : 100=0.05% cpu : usr=2.57%, sys=1.63%, ctx=6178, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4909,1265,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=654MiB/s (686MB/s), 654MiB/s-654MiB/s (686MB/s-686MB/s), io=19.2GiB (20.6GB), run=30022-30022msec WRITE: bw=169MiB/s (177MB/s), 169MiB/s-169MiB/s (177MB/s-177MB/s), io=5060MiB (5306MB), run=30022-30022msec Disk stats (read/write): vdk: ios=19564/5029, merge=0/0, ticks=4308/81659, in_queue=95434, util=95.99% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981073: Fri Jan 7 14:39:21 2022 read: IOPS=183, BW=736MiB/s (772MB/s)(22.0GiB/30642msec) slat (usec): min=45, max=462765, avg=3958.79, stdev=13540.36 clat (msec): min=80, max=2398, avg=299.68, stdev=144.55 lat (msec): min=80, max=2398, avg=303.64, stdev=145.21 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 110], 10.00th=[ 153], 20.00th=[ 205], | 30.00th=[ 234], 40.00th=[ 259], 50.00th=[ 284], 60.00th=[ 305], | 70.00th=[ 334], 80.00th=[ 368], 90.00th=[ 426], 95.00th=[ 575], | 99.00th=[ 751], 99.50th=[ 860], 99.90th=[ 1552], 99.95th=[ 1854], | 99.99th=[ 2400] bw ( KiB/s): min= 8192, max=1613824, per=99.53%, avg=749951.08, stdev=303657.06, samples=61 iops : min= 2, max= 394, avg=183.08, stdev=74.15, samples=61 write: IOPS=48, BW=193MiB/s (203MB/s)(5924MiB/30642msec) slat (usec): min=1119, max=622612, avg=5181.84, stdev=21742.05 clat (msec): min=133, max=4357, avg=904.35, stdev=346.24 lat (msec): min=135, max=4359, avg=909.54, stdev=346.17 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 718], | 30.00th=[ 760], 40.00th=[ 793], 50.00th=[ 844], 60.00th=[ 885], | 70.00th=[ 953], 80.00th=[ 1036], 90.00th=[ 1200], 95.00th=[ 1385], | 99.00th=[ 2165], 99.50th=[ 3239], 99.90th=[ 34], 99.95th=[ 34], | 99.99th=[ 34] bw ( KiB/s): min=24576, max=368640, per=100.00%, avg=199258.32, stdev=70654.59, samples=59 iops : min= 6, max= 90, avg=48.64, stdev=17.25, samples=59 lat (msec) : 100=2.37%, 250=26.74%, 500=45.08%, 750=9.85%, 1000=10.73% lat (msec) : 2000=4.92%, >=2000=0.31% cpu : usr=6.48%, sys=1.73%, ctx=2631, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5637,1481,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=736MiB/s (772MB/s), 736MiB/s-736MiB/s (772MB/s-772MB/s), io=22.0GiB (23.6GB), run=30642-30642msec WRITE: bw=193MiB/s (203MB/s), 193MiB/s-193MiB/s (203MB/s-203MB/s), io=5924MiB (6212MB), run=30642-30642msec Disk stats (read/write): vdk: ios=22548/5918, merge=0/0, ticks=134718/3612286, in_queue=3765932, util=99.09% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981893: Fri Jan 7 14:39:57 2022 read: IOPS=164, BW=658MiB/s (690MB/s)(19.3GiB/30001msec) clat (usec): min=381, max=23389, avg=744.82, stdev=1436.09 lat (usec): min=381, max=23389, avg=745.02, stdev=1436.09 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 441], | 70.00th=[ 474], 80.00th=[ 955], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[ 9241], 99.50th=[13435], 99.90th=[18482], 99.95th=[20317], | 99.99th=[23462] bw ( KiB/s): min=286720, max=1097728, per=100.00%, avg=673943.93, stdev=157814.43, samples=60 iops : min= 70, max= 268, avg=164.52, stdev=38.51, samples=60 write: IOPS=42, BW=170MiB/s (178MB/s)(5088MiB/30001msec) clat (usec): min=11114, max=75657, avg=20120.78, stdev=4700.14 lat (usec): min=11501, max=76067, avg=20683.58, stdev=4745.73 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13960], 10.00th=[14615], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20055], 60.00th=[21890], | 70.00th=[22938], 80.00th=[23725], 90.00th=[24773], 95.00th=[26084], | 99.00th=[33424], 99.50th=[34341], 99.90th=[46924], 99.95th=[76022], | 99.99th=[76022] bw ( KiB/s): min=155648, max=188416, per=99.92%, avg=173533.60, stdev=9106.46, samples=60 iops : min= 38, max= 46, avg=42.35, stdev= 2.23, samples=60 lat (usec) : 500=56.97%, 750=4.77%, 1000=2.43% lat (msec) : 2=14.30%, 4=0.05%, 10=0.24%, 20=10.97%, 50=10.24% lat (msec) : 100=0.02% cpu : usr=2.53%, sys=1.79%, ctx=6216, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4936,1272,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=658MiB/s (690MB/s), 658MiB/s-658MiB/s (690MB/s-690MB/s), io=19.3GiB (20.7GB), run=30001-30001msec WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=5088MiB (5335MB), run=30001-30001msec Disk stats (read/write): vdk: ios=19720/5072, merge=0/0, ticks=4173/81980, in_queue=95734, util=95.33% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983242: Fri Jan 7 14:40:35 2022 read: IOPS=184, BW=739MiB/s (775MB/s)(22.1GiB/30637msec) slat (usec): min=45, max=474737, avg=3956.97, stdev=13703.46 clat (msec): min=261, max=3382, avg=725.27, stdev=185.14 lat (msec): min=261, max=3382, avg=729.22, stdev=185.80 clat percentiles (msec): | 1.00th=[ 380], 5.00th=[ 456], 10.00th=[ 531], 20.00th=[ 592], | 30.00th=[ 642], 40.00th=[ 684], 50.00th=[ 718], 60.00th=[ 751], | 70.00th=[ 793], 80.00th=[ 835], 90.00th=[ 927], 95.00th=[ 1036], | 99.00th=[ 1234], 99.50th=[ 1301], 99.90th=[ 2299], 99.95th=[ 2735], | 99.99th=[ 3373] bw ( KiB/s): min= 8192, max=1368064, per=99.66%, avg=753843.75, stdev=296195.21, samples=60 iops : min= 2, max= 334, avg=184.03, stdev=72.32, samples=60 write: IOPS=48, BW=194MiB/s (203MB/s)(5940MiB/30637msec) slat (usec): min=1128, max=459011, avg=5110.12, stdev=19551.46 clat (msec): min=570, max=3490, avg=1313.43, stdev=345.90 lat (msec): min=591, max=3491, avg=1318.54, stdev=344.71 clat percentiles (msec): | 1.00th=[ 718], 5.00th=[ 902], 10.00th=[ 1003], 20.00th=[ 1099], | 30.00th=[ 1167], 40.00th=[ 1217], 50.00th=[ 1267], 60.00th=[ 1318], | 70.00th=[ 1385], 80.00th=[ 1469], 90.00th=[ 1636], 95.00th=[ 1787], | 99.00th=[ 2903], 99.50th=[ 3406], 99.90th=[ 3473], 99.95th=[ 3507], | 99.99th=[ 3507] bw ( KiB/s): min=24576, max=565248, per=99.52%, avg=197589.64, stdev=110637.84, samples=59 iops : min= 6, max= 138, avg=48.24, stdev=27.01, samples=59 lat (msec) : 500=6.17%, 750=40.47%, 1000=29.88%, 2000=22.74%, >=2000=0.74% cpu : usr=6.78%, sys=1.62%, ctx=2664, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5658,1485,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=739MiB/s (775MB/s), 739MiB/s-739MiB/s (775MB/s-775MB/s), io=22.1GiB (23.7GB), run=30637-30637msec WRITE: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=5940MiB (6229MB), run=30637-30637msec Disk stats (read/write): vdk: ios=22632/5896, merge=0/0, ticks=147640/3575397, in_queue=3761416, util=98.30%