30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890674: Fri Jan 7 13:33:20 2022 read: IOPS=2371, BW=9487KiB/s (9715kB/s)(278MiB/30001msec) clat (usec): min=25, max=65646, avg=59.51, stdev=456.33 lat (usec): min=25, max=65646, avg=59.65, stdev=456.33 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 510], 99.95th=[11994], | 99.99th=[16909] bw ( KiB/s): min= 7240, max=13600, per=100.00%, avg=9512.97, stdev=1001.42, samples=59 iops : min= 1810, max= 3400, avg=2378.20, stdev=250.37, samples=59 write: IOPS=593, BW=2374KiB/s (2431kB/s)(69.6MiB/30001msec) clat (usec): min=187, max=50163, avg=1437.94, stdev=1640.53 lat (usec): min=187, max=50163, avg=1438.22, stdev=1640.53 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 603], 10.00th=[ 832], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1778], 95.00th=[ 2073], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[20055], 99.95th=[41157], | 99.99th=[45351] bw ( KiB/s): min= 1904, max= 3520, per=100.00%, avg=2381.98, stdev=236.27, samples=59 iops : min= 476, max= 880, avg=595.49, stdev=59.06, samples=59 lat (usec) : 50=53.22%, 100=26.46%, 250=0.33%, 500=0.65%, 750=0.84% lat (usec) : 1000=1.91% lat (msec) : 2=15.34%, 4=0.87%, 10=0.26%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.33%, sys=3.80%, ctx=88971, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71156,17805,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9487KiB/s (9715kB/s), 9487KiB/s-9487KiB/s (9715kB/s-9715kB/s), io=278MiB (291MB), run=30001-30001msec WRITE: bw=2374KiB/s (2431kB/s), 2374KiB/s-2374KiB/s (2431kB/s-2431kB/s), io=69.6MiB (72.9MB), run=30001-30001msec Disk stats (read/write): vdi: ios=70716/17682, merge=0/0, ticks=772/17847, in_queue=28124, util=94.01% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891543: Fri Jan 7 13:33:57 2022 read: IOPS=2362, BW=9449KiB/s (9676kB/s)(277MiB/30001msec) slat (usec): min=5, max=287, avg= 9.78, stdev= 5.24 clat (nsec): min=665, max=26133k, avg=45562.22, stdev=389167.99 lat (usec): min=22, max=26145, avg=55.50, stdev=389.21 clat percentiles (usec): | 1.00th=[ 20], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 36], | 70.00th=[ 38], 80.00th=[ 40], 90.00th=[ 45], 95.00th=[ 49], | 99.00th=[ 61], 99.50th=[ 70], 99.90th=[ 169], 99.95th=[12125], | 99.99th=[16581] bw ( KiB/s): min= 7400, max=14276, per=100.00%, avg=9450.12, stdev=1208.51, samples=60 iops : min= 1850, max= 3569, avg=2362.52, stdev=302.13, samples=60 write: IOPS=590, BW=2362KiB/s (2419kB/s)(69.2MiB/30001msec) slat (usec): min=6, max=316, avg=10.84, stdev= 5.43 clat (usec): min=181, max=45017, avg=1452.16, stdev=1693.15 lat (usec): min=192, max=45026, avg=1463.16, stdev=1693.22 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 611], 10.00th=[ 857], 20.00th=[ 1057], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1795], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min= 1880, max= 3647, per=100.00%, avg=2362.32, stdev=256.48, samples=60 iops : min= 470, max= 911, avg=590.57, stdev=64.06, samples=60 lat (nsec) : 750=0.02%, 1000=0.13% lat (usec) : 2=0.22%, 4=0.02%, 10=0.08%, 20=0.40%, 50=76.07% lat (usec) : 100=2.94%, 250=0.20%, 500=0.62%, 750=0.71%, 1000=1.81% lat (msec) : 2=15.46%, 4=0.93%, 10=0.25%, 20=0.11%, 50=0.03% cpu : usr=1.24%, sys=5.86%, ctx=88102, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70870,17717,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9449KiB/s (9676kB/s), 9449KiB/s-9449KiB/s (9676kB/s-9676kB/s), io=277MiB (290MB), run=30001-30001msec WRITE: bw=2362KiB/s (2419kB/s), 2362KiB/s-2362KiB/s (2419kB/s-2419kB/s), io=69.2MiB (72.6MB), run=30001-30001msec Disk stats (read/write): vdi: ios=70348/17603, merge=0/0, ticks=721/18299, in_queue=28127, util=93.94% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892542: Fri Jan 7 13:34:34 2022 read: IOPS=2373, BW=9493KiB/s (9721kB/s)(278MiB/30001msec) clat (usec): min=26, max=19446, avg=58.35, stdev=379.57 lat (usec): min=26, max=19446, avg=58.49, stdev=379.57 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 80], 99.50th=[ 92], 99.90th=[ 461], 99.95th=[12256], | 99.99th=[16319] bw ( KiB/s): min= 7248, max=14256, per=100.00%, avg=9496.83, stdev=1050.63, samples=59 iops : min= 1812, max= 3564, avg=2374.20, stdev=262.66, samples=59 write: IOPS=593, BW=2376KiB/s (2433kB/s)(69.6MiB/30001msec) clat (usec): min=199, max=46915, avg=1441.09, stdev=1660.48 lat (usec): min=199, max=46915, avg=1441.37, stdev=1660.48 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 594], 10.00th=[ 832], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1778], 95.00th=[ 2073], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min= 1816, max= 3664, per=100.00%, avg=2377.02, stdev=243.92, samples=59 iops : min= 454, max= 916, avg=594.25, stdev=60.98, samples=59 lat (usec) : 50=53.77%, 100=25.95%, 250=0.25%, 500=0.71%, 750=0.81% lat (usec) : 1000=1.96% lat (msec) : 2=15.30%, 4=0.87%, 10=0.27%, 20=0.09%, 50=0.02% cpu : usr=1.43%, sys=3.69%, ctx=89029, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71202,17819,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9493KiB/s (9721kB/s), 9493KiB/s-9493KiB/s (9721kB/s-9721kB/s), io=278MiB (292MB), run=30001-30001msec WRITE: bw=2376KiB/s (2433kB/s), 2376KiB/s-2376KiB/s (2433kB/s-2433kB/s), io=69.6MiB (72.0MB), run=30001-30001msec Disk stats (read/write): vdi: ios=71167/17807, merge=0/0, ticks=727/18031, in_queue=28388, util=94.12% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892993: Fri Jan 7 13:35:11 2022 read: IOPS=2441, BW=9764KiB/s (9998kB/s)(286MiB/30036msec) slat (usec): min=2, max=358, avg= 9.67, stdev= 4.70 clat (nsec): min=621, max=2494.1M, avg=742312.27, stdev=31272855.66 lat (usec): min=22, max=2494.1k, avg=752.13, stdev=31272.87 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 38], 70.00th=[ 41], | 80.00th=[ 44], 90.00th=[ 49], 95.00th=[ 56], | 99.00th=[ 84], 99.50th=[ 103], 99.90th=[ 424], | 99.95th=[ 497026], 99.99th=[1568670] bw ( KiB/s): min= 7920, max=17002, per=100.00%, avg=9898.08, stdev=1272.88, samples=60 iops : min= 1980, max= 4250, avg=2474.12, stdev=318.21, samples=60 write: IOPS=609, BW=2439KiB/s (2498kB/s)(71.6MiB/30036msec) slat (usec): min=2, max=320, avg=10.91, stdev= 5.16 clat (usec): min=444, max=76260, avg=13358.38, stdev=6075.79 lat (usec): min=451, max=76274, avg=13369.45, stdev=6075.99 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 5997], 10.00th=[ 7898], 20.00th=[10290], | 30.00th=[11469], 40.00th=[12256], 50.00th=[12911], 60.00th=[13566], | 70.00th=[14222], 80.00th=[15008], 90.00th=[18482], 95.00th=[21890], | 99.00th=[31589], 99.50th=[61604], 99.90th=[65799], 99.95th=[69731], | 99.99th=[74974] bw ( KiB/s): min= 2060, max= 4274, per=100.00%, avg=2472.23, stdev=284.65, samples=60 iops : min= 515, max= 1068, avg=617.73, stdev=71.10, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.05%, 4=0.01%, 10=0.03%, 20=0.11%, 50=72.95% lat (usec) : 100=6.38%, 250=0.34%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.23%, 4=0.13%, 10=3.24%, 20=14.73%, 50=1.51% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.43%, sys=5.95%, ctx=89607, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73318,18318,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9764KiB/s (9998kB/s), 9764KiB/s-9764KiB/s (9998kB/s-9998kB/s), io=286MiB (300MB), run=30036-30036msec WRITE: bw=2439KiB/s (2498kB/s), 2439KiB/s-2439KiB/s (2498kB/s-2498kB/s), io=71.6MiB (75.0MB), run=30036-30036msec Disk stats (read/write): vdi: ios=73289/18308, merge=0/0, ticks=50484/235242, in_queue=298119, util=99.49% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894408: Fri Jan 7 13:35:48 2022 read: IOPS=2366, BW=9468KiB/s (9695kB/s)(277MiB/30001msec) clat (usec): min=25, max=26282, avg=57.87, stdev=392.18 lat (usec): min=25, max=26282, avg=58.01, stdev=392.18 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 64], | 99.00th=[ 81], 99.50th=[ 92], 99.90th=[ 347], 99.95th=[12911], | 99.99th=[16712] bw ( KiB/s): min= 7640, max=14824, per=100.00%, avg=9468.27, stdev=1104.72, samples=60 iops : min= 1910, max= 3706, avg=2367.07, stdev=276.18, samples=60 write: IOPS=592, BW=2369KiB/s (2426kB/s)(69.4MiB/30001msec) clat (usec): min=195, max=56804, avg=1447.79, stdev=1700.84 lat (usec): min=196, max=56805, avg=1448.06, stdev=1700.84 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 611], 10.00th=[ 840], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1778], 95.00th=[ 2073], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[36963], 99.95th=[41681], | 99.99th=[44303] bw ( KiB/s): min= 1912, max= 3824, per=100.00%, avg=2368.80, stdev=260.56, samples=60 iops : min= 478, max= 956, avg=592.20, stdev=65.14, samples=60 lat (usec) : 50=56.21%, 100=23.50%, 250=0.27%, 500=0.68%, 750=0.79% lat (usec) : 1000=1.95% lat (msec) : 2=15.36%, 4=0.84%, 10=0.27%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.46%, sys=3.59%, ctx=88785, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71012,17767,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9468KiB/s (9695kB/s), 9468KiB/s-9468KiB/s (9695kB/s-9695kB/s), io=277MiB (291MB), run=30001-30001msec WRITE: bw=2369KiB/s (2426kB/s), 2369KiB/s-2369KiB/s (2426kB/s-2426kB/s), io=69.4MiB (72.8MB), run=30001-30001msec Disk stats (read/write): vdi: ios=70408/17618, merge=0/0, ticks=726/18002, in_queue=28214, util=94.28% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894978: Fri Jan 7 13:36:25 2022 read: IOPS=2426, BW=9705KiB/s (9938kB/s)(285MiB/30110msec) slat (usec): min=2, max=204, avg= 9.58, stdev= 4.36 clat (nsec): min=611, max=2520.3M, avg=731479.82, stdev=32345162.53 lat (usec): min=21, max=2520.3k, avg=741.21, stdev=32345.16 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 37], 60.00th=[ 40], 70.00th=[ 42], | 80.00th=[ 47], 90.00th=[ 58], 95.00th=[ 75], | 99.00th=[ 188], 99.50th=[ 355], 99.90th=[ 873], | 99.95th=[ 408945], 99.99th=[2197816] bw ( KiB/s): min= 8072, max=17448, per=100.00%, avg=9739.33, stdev=1342.02, samples=60 iops : min= 2018, max= 4362, avg=2434.83, stdev=335.51, samples=60 write: IOPS=606, BW=2426KiB/s (2484kB/s)(71.3MiB/30110msec) slat (usec): min=2, max=165, avg=10.89, stdev= 4.76 clat (msec): min=5, max=2383, avg=79.41, stdev=62.67 lat (msec): min=5, max=2383, avg=79.42, stdev=62.67 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 57], 10.00th=[ 61], 20.00th=[ 65], | 30.00th=[ 67], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 93], 95.00th=[ 121], | 99.00th=[ 230], 99.50th=[ 464], 99.90th=[ 1003], 99.95th=[ 1167], | 99.99th=[ 1905] bw ( KiB/s): min= 2080, max= 4096, per=100.00%, avg=2429.47, stdev=274.30, samples=60 iops : min= 520, max= 1024, avg=607.37, stdev=68.57, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.15%, 50=67.47% lat (usec) : 100=10.41%, 250=1.25%, 500=0.37%, 750=0.13%, 1000=0.02% lat (msec) : 2=0.01%, 10=0.18%, 20=0.11%, 50=0.18%, 100=17.98% lat (msec) : 250=1.38%, 500=0.11%, 750=0.05%, 1000=0.03%, 2000=0.03% lat (msec) : >=2000=0.01% cpu : usr=1.37%, sys=5.73%, ctx=87171, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=73052,18263,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9705KiB/s (9938kB/s), 9705KiB/s-9705KiB/s (9938kB/s-9938kB/s), io=285MiB (299MB), run=30110-30110msec WRITE: bw=2426KiB/s (2484kB/s), 2426KiB/s-2426KiB/s (2484kB/s-2484kB/s), io=71.3MiB (74.8MB), run=30110-30110msec Disk stats (read/write): vdi: ios=73009/18211, merge=0/0, ticks=45137/1434089, in_queue=1496858, util=99.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895820: Fri Jan 7 13:37:02 2022 read: IOPS=2369, BW=9478KiB/s (9706kB/s)(278MiB/30002msec) clat (usec): min=25, max=26857, avg=58.57, stdev=410.06 lat (usec): min=25, max=26857, avg=58.72, stdev=410.06 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 59], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 92], 99.90th=[ 453], 99.95th=[12649], | 99.99th=[16450] bw ( KiB/s): min= 7504, max=13192, per=100.00%, avg=9478.32, stdev=1059.76, samples=60 iops : min= 1876, max= 3298, avg=2369.57, stdev=264.93, samples=60 write: IOPS=592, BW=2371KiB/s (2428kB/s)(69.5MiB/30002msec) clat (usec): min=206, max=43993, avg=1442.97, stdev=1672.46 lat (usec): min=206, max=43993, avg=1443.25, stdev=1672.46 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 603], 10.00th=[ 840], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1762], 95.00th=[ 2073], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min= 1968, max= 3408, per=100.00%, avg=2371.10, stdev=235.87, samples=60 iops : min= 492, max= 852, avg=592.77, stdev=58.96, samples=60 lat (usec) : 50=55.32%, 100=24.37%, 250=0.28%, 500=0.73%, 750=0.77% lat (usec) : 1000=1.89% lat (msec) : 2=15.44%, 4=0.84%, 10=0.25%, 20=0.09%, 50=0.03% cpu : usr=1.52%, sys=3.57%, ctx=88883, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71090,17785,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9478KiB/s (9706kB/s), 9478KiB/s-9478KiB/s (9706kB/s-9706kB/s), io=278MiB (291MB), run=30002-30002msec WRITE: bw=2371KiB/s (2428kB/s), 2371KiB/s-2371KiB/s (2428kB/s-2428kB/s), io=69.5MiB (72.8MB), run=30002-30002msec Disk stats (read/write): vdi: ios=71011/17765, merge=0/0, ticks=767/18076, in_queue=28392, util=94.23% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896945: Fri Jan 7 13:37:39 2022 read: IOPS=2403, BW=9616KiB/s (9846kB/s)(283MiB/30158msec) slat (usec): min=2, max=573, avg= 9.73, stdev= 6.92 clat (nsec): min=675, max=2515.1M, avg=740634.92, stdev=31797220.88 lat (usec): min=19, max=2515.1k, avg=750.52, stdev=31797.21 clat percentiles (usec): | 1.00th=[ 22], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 40], 70.00th=[ 43], | 80.00th=[ 49], 90.00th=[ 65], 95.00th=[ 122], | 99.00th=[ 824], 99.50th=[ 1287], 99.90th=[ 2606], | 99.95th=[ 425722], 99.99th=[1769997] bw ( KiB/s): min= 6854, max=18156, per=100.00%, avg=9735.43, stdev=1389.17, samples=60 iops : min= 1713, max= 4539, avg=2433.50, stdev=347.34, samples=60 write: IOPS=601, BW=2406KiB/s (2464kB/s)(70.9MiB/30158msec) slat (usec): min=2, max=338, avg=11.15, stdev= 7.41 clat (msec): min=4, max=2953, avg=163.14, stdev=149.18 lat (msec): min=4, max=2953, avg=163.16, stdev=149.18 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 104], 10.00th=[ 117], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 169], 90.00th=[ 192], 95.00th=[ 213], | 99.00th=[ 642], 99.50th=[ 1053], 99.90th=[ 2265], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min= 1943, max= 3895, per=100.00%, avg=2423.03, stdev=269.79, samples=60 iops : min= 485, max= 973, avg=605.38, stdev=67.40, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.17%, 4=0.01%, 10=0.08%, 20=0.36%, 50=65.12% lat (usec) : 100=9.28%, 250=2.83%, 500=0.72%, 750=0.44%, 1000=0.27% lat (msec) : 2=0.42%, 4=0.14%, 10=0.04%, 20=0.23%, 50=0.06% lat (msec) : 100=0.46%, 250=18.59%, 500=0.42%, 750=0.08%, 1000=0.07% lat (msec) : 2000=0.07%, >=2000=0.06% cpu : usr=1.42%, sys=5.70%, ctx=84492, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=72497,18141,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9616KiB/s (9846kB/s), 9616KiB/s-9616KiB/s (9846kB/s-9846kB/s), io=283MiB (297MB), run=30158-30158msec WRITE: bw=2406KiB/s (2464kB/s), 2406KiB/s-2406KiB/s (2464kB/s-2464kB/s), io=70.9MiB (74.3MB), run=30158-30158msec Disk stats (read/write): vdi: ios=72460/18037, merge=0/0, ticks=49214/2922385, in_queue=2996300, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897508: Fri Jan 7 13:38:16 2022 read: IOPS=2364, BW=9459KiB/s (9686kB/s)(277MiB/30009msec) clat (usec): min=24, max=22929, avg=59.25, stdev=358.09 lat (usec): min=24, max=22929, avg=59.39, stdev=358.10 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 67], | 99.00th=[ 86], 99.50th=[ 100], 99.90th=[ 322], 99.95th=[11076], | 99.99th=[15270] bw ( KiB/s): min= 6976, max=15072, per=100.00%, avg=9462.05, stdev=1208.62, samples=60 iops : min= 1744, max= 3768, avg=2365.50, stdev=302.15, samples=60 write: IOPS=591, BW=2366KiB/s (2423kB/s)(69.3MiB/30009msec) clat (usec): min=205, max=49760, avg=1444.26, stdev=1683.50 lat (usec): min=205, max=49760, avg=1444.55, stdev=1683.50 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 586], 10.00th=[ 816], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1795], 95.00th=[ 2089], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[36439], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min= 1864, max= 3896, per=100.00%, avg=2366.75, stdev=272.05, samples=60 iops : min= 466, max= 974, avg=591.68, stdev=68.01, samples=60 lat (usec) : 50=48.55%, 100=31.04%, 250=0.38%, 500=0.76%, 750=0.85% lat (usec) : 1000=2.05% lat (msec) : 2=15.12%, 4=0.87%, 10=0.27%, 20=0.09%, 50=0.02% cpu : usr=1.38%, sys=3.90%, ctx=88719, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70963,17751,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9459KiB/s (9686kB/s), 9459KiB/s-9459KiB/s (9686kB/s-9686kB/s), io=277MiB (291MB), run=30009-30009msec WRITE: bw=2366KiB/s (2423kB/s), 2366KiB/s-2366KiB/s (2423kB/s-2423kB/s), io=69.3MiB (72.7MB), run=30009-30009msec Disk stats (read/write): vdi: ios=70892/17727, merge=0/0, ticks=674/17889, in_queue=28283, util=93.90% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898369: Fri Jan 7 13:38:53 2022 read: IOPS=2398, BW=9593KiB/s (9823kB/s)(283MiB/30199msec) slat (usec): min=2, max=182903, avg=328.22, stdev=1863.35 clat (usec): min=539, max=2529.5k, avg=26559.73, stdev=32786.85 lat (usec): min=541, max=2529.5k, avg=26888.08, stdev=32858.96 clat percentiles (usec): | 1.00th=[ 1270], 5.00th=[ 7308], 10.00th=[ 14484], | 20.00th=[ 17957], 30.00th=[ 20317], 40.00th=[ 22414], | 50.00th=[ 24249], 60.00th=[ 26084], 70.00th=[ 28443], | 80.00th=[ 31589], 90.00th=[ 36439], 95.00th=[ 42730], | 99.00th=[ 82314], 99.50th=[ 175113], 99.90th=[ 225444], | 99.95th=[ 400557], 99.99th=[1551893] bw ( KiB/s): min= 7320, max=17764, per=100.00%, avg=9703.60, stdev=1465.02, samples=60 iops : min= 1830, max= 4441, avg=2425.78, stdev=366.25, samples=60 write: IOPS=600, BW=2401KiB/s (2458kB/s)(70.8MiB/30199msec) slat (usec): min=2, max=201368, avg=333.55, stdev=2455.68 clat (msec): min=12, max=2947, avg=225.25, stdev=174.22 lat (msec): min=12, max=2948, avg=225.58, stdev=174.21 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 144], 10.00th=[ 165], 20.00th=[ 178], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 203], 60.00th=[ 211], | 70.00th=[ 224], 80.00th=[ 239], 90.00th=[ 275], 95.00th=[ 305], | 99.00th=[ 1011], 99.50th=[ 1452], 99.90th=[ 2601], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 1864, max= 3601, per=100.00%, avg=2412.80, stdev=274.46, samples=60 iops : min= 466, max= 900, avg=603.07, stdev=68.53, samples=60 lat (usec) : 750=0.10%, 1000=0.28% lat (msec) : 2=1.48%, 4=1.49%, 10=1.13%, 20=18.32%, 50=54.90% lat (msec) : 100=2.18%, 250=17.05%, 500=2.68%, 750=0.09%, 1000=0.07% lat (msec) : 2000=0.14%, >=2000=0.08% cpu : usr=0.69%, sys=1.87%, ctx=9980, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=72421,18126,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9593KiB/s (9823kB/s), 9593KiB/s-9593KiB/s (9823kB/s-9823kB/s), io=283MiB (297MB), run=30199-30199msec WRITE: bw=2401KiB/s (2458kB/s), 2401KiB/s-2401KiB/s (2458kB/s-2458kB/s), io=70.8MiB (74.2MB), run=30199-30199msec Disk stats (read/write): vdi: ios=72405/18001, merge=0/0, ticks=47904/3582292, in_queue=3659276, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898987: Fri Jan 7 13:39:30 2022 read: IOPS=2307, BW=36.1MiB/s (37.8MB/s)(1082MiB/30002msec) clat (usec): min=26, max=23457, avg=62.80, stdev=398.17 lat (usec): min=27, max=23457, avg=62.95, stdev=398.18 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 92], 99.50th=[ 105], 99.90th=[ 510], 99.95th=[11338], | 99.99th=[18744] bw ( KiB/s): min=29082, max=54848, per=100.00%, avg=36934.25, stdev=4313.22, samples=60 iops : min= 1817, max= 3428, avg=2308.37, stdev=269.59, samples=60 write: IOPS=577, BW=9236KiB/s (9458kB/s)(271MiB/30002msec) clat (usec): min=226, max=45028, avg=1470.87, stdev=1669.99 lat (usec): min=228, max=45029, avg=1472.39, stdev=1669.97 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 594], 10.00th=[ 848], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1369], | 70.00th=[ 1467], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min= 7599, max=14112, per=100.00%, avg=9237.40, stdev=979.19, samples=60 iops : min= 474, max= 882, avg=577.30, stdev=61.24, samples=60 lat (usec) : 50=41.55%, 100=37.94%, 250=0.40%, 500=0.74%, 750=0.81% lat (usec) : 1000=1.58% lat (msec) : 2=15.63%, 4=0.98%, 10=0.24%, 20=0.10%, 50=0.03% cpu : usr=1.53%, sys=3.67%, ctx=86564, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69243,17319,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.1MiB/s (37.8MB/s), 36.1MiB/s-36.1MiB/s (37.8MB/s-37.8MB/s), io=1082MiB (1134MB), run=30002-30002msec WRITE: bw=9236KiB/s (9458kB/s), 9236KiB/s-9236KiB/s (9458kB/s-9458kB/s), io=271MiB (284MB), run=30002-30002msec Disk stats (read/write): vdi: ios=68711/17186, merge=0/0, ticks=717/17860, in_queue=28147, util=94.22% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899804: Fri Jan 7 13:40:07 2022 read: IOPS=2329, BW=36.4MiB/s (38.2MB/s)(1093MiB/30032msec) slat (usec): min=5, max=211, avg=10.72, stdev= 5.55 clat (nsec): min=675, max=21102k, avg=49067.32, stdev=379587.42 lat (usec): min=23, max=21120, avg=59.96, stdev=379.67 clat percentiles (usec): | 1.00th=[ 21], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 33], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 39], | 70.00th=[ 41], 80.00th=[ 44], 90.00th=[ 49], 95.00th=[ 54], | 99.00th=[ 69], 99.50th=[ 80], 99.90th=[ 408], 99.95th=[11731], | 99.99th=[17171] bw ( KiB/s): min=29152, max=52768, per=100.00%, avg=37306.07, stdev=4188.50, samples=60 iops : min= 1822, max= 3298, avg=2331.62, stdev=261.80, samples=60 write: IOPS=583, BW=9330KiB/s (9554kB/s)(274MiB/30032msec) slat (usec): min=8, max=201, avg=13.07, stdev= 6.05 clat (usec): min=219, max=43735, avg=1454.15, stdev=1606.43 lat (usec): min=229, max=43749, avg=1467.39, stdev=1606.44 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 611], 10.00th=[ 873], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1221], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1811], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[38011], | 99.99th=[43254] bw ( KiB/s): min= 7808, max=13600, per=100.00%, avg=9339.48, stdev=875.20, samples=60 iops : min= 488, max= 850, avg=583.72, stdev=54.70, samples=60 lat (nsec) : 750=0.01%, 1000=0.15% lat (usec) : 2=0.21%, 4=0.02%, 10=0.05%, 20=0.25%, 50=72.80% lat (usec) : 100=6.30%, 250=0.12%, 500=0.71%, 750=0.71%, 1000=1.62% lat (msec) : 2=15.71%, 4=1.02%, 10=0.21%, 20=0.10%, 50=0.02% cpu : usr=1.33%, sys=6.14%, ctx=86998, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69947,17512,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.4MiB/s (38.2MB/s), 36.4MiB/s-36.4MiB/s (38.2MB/s-38.2MB/s), io=1093MiB (1146MB), run=30032-30032msec WRITE: bw=9330KiB/s (9554kB/s), 9330KiB/s-9330KiB/s (9554kB/s-9554kB/s), io=274MiB (287MB), run=30032-30032msec Disk stats (read/write): vdi: ios=69529/17390, merge=0/0, ticks=704/18019, in_queue=28045, util=93.80% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901192: Fri Jan 7 13:40:44 2022 read: IOPS=2305, BW=36.0MiB/s (37.8MB/s)(1081MiB/30001msec) clat (usec): min=26, max=63069, avg=62.42, stdev=459.94 lat (usec): min=26, max=63069, avg=62.55, stdev=459.94 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 87], 99.50th=[ 101], 99.90th=[ 326], 99.95th=[12387], | 99.99th=[17695] bw ( KiB/s): min=29024, max=59680, per=100.00%, avg=36963.80, stdev=4292.07, samples=59 iops : min= 1814, max= 3730, avg=2310.24, stdev=268.25, samples=59 write: IOPS=576, BW=9227KiB/s (9449kB/s)(270MiB/30001msec) clat (usec): min=233, max=44989, avg=1474.34, stdev=1636.35 lat (usec): min=234, max=44991, avg=1475.79, stdev=1636.34 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 611], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[17957], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min= 7584, max=15328, per=100.00%, avg=9251.80, stdev=1033.47, samples=59 iops : min= 474, max= 958, avg=578.24, stdev=64.59, samples=59 lat (usec) : 50=44.88%, 100=34.71%, 250=0.32%, 500=0.72%, 750=0.75% lat (usec) : 1000=1.60% lat (msec) : 2=15.66%, 4=0.99%, 10=0.25%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.31%, sys=3.77%, ctx=86486, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69180,17302,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s (37.8MB/s-37.8MB/s), io=1081MiB (1133MB), run=30001-30001msec WRITE: bw=9227KiB/s (9449kB/s), 9227KiB/s-9227KiB/s (9449kB/s-9449kB/s), io=270MiB (283MB), run=30001-30001msec Disk stats (read/write): vdi: ios=68833/17214, merge=0/0, ticks=716/17910, in_queue=28197, util=94.30% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901856: Fri Jan 7 13:41:21 2022 read: IOPS=2353, BW=36.8MiB/s (38.6MB/s)(1104MiB/30033msec) slat (usec): min=2, max=169, avg=10.93, stdev= 4.14 clat (nsec): min=755, max=2494.3M, avg=698602.79, stdev=31611615.60 lat (usec): min=26, max=2494.4k, avg=709.69, stdev=31611.61 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 37], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 48], 90.00th=[ 54], 95.00th=[ 61], | 99.00th=[ 94], 99.50th=[ 130], 99.90th=[ 652], | 99.95th=[ 341836], 99.99th=[1786774] bw ( KiB/s): min=30752, max=65442, per=100.00%, avg=37701.97, stdev=4928.40, samples=60 iops : min= 1922, max= 4090, avg=2356.37, stdev=308.02, samples=60 write: IOPS=588, BW=9417KiB/s (9643kB/s)(276MiB/30033msec) slat (usec): min=4, max=113, avg=13.76, stdev= 4.43 clat (usec): min=457, max=82529, avg=14126.56, stdev=6314.24 lat (usec): min=467, max=82543, avg=14140.50, stdev=6314.46 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 8029], 10.00th=[ 9372], 20.00th=[10814], | 30.00th=[11994], 40.00th=[12911], 50.00th=[13566], 60.00th=[14222], | 70.00th=[14746], 80.00th=[15664], 90.00th=[18482], 95.00th=[22414], | 99.00th=[37487], 99.50th=[63177], 99.90th=[70779], 99.95th=[73925], | 99.99th=[82314] bw ( KiB/s): min= 7648, max=16448, per=100.00%, avg=9426.18, stdev=1109.40, samples=60 iops : min= 478, max= 1028, avg=589.13, stdev=69.34, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.05%, 50=67.69% lat (usec) : 100=11.46%, 250=0.55%, 500=0.06%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.24%, 4=0.05%, 10=2.50%, 20=15.51%, 50=1.54% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.39%, sys=6.30%, ctx=86889, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70687,17677,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=36.8MiB/s (38.6MB/s), 36.8MiB/s-36.8MiB/s (38.6MB/s-38.6MB/s), io=1104MiB (1158MB), run=30033-30033msec WRITE: bw=9417KiB/s (9643kB/s), 9417KiB/s-9417KiB/s (9643kB/s-9643kB/s), io=276MiB (290MB), run=30033-30033msec Disk stats (read/write): vdi: ios=70179/17562, merge=0/0, ticks=44061/239427, in_queue=296116, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902762: Fri Jan 7 13:41:58 2022 read: IOPS=2313, BW=36.2MiB/s (37.9MB/s)(1085MiB/30007msec) clat (usec): min=26, max=24794, avg=61.07, stdev=391.66 lat (usec): min=26, max=24794, avg=61.22, stdev=391.66 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 330], 99.95th=[12125], | 99.99th=[16712] bw ( KiB/s): min=26368, max=56352, per=100.00%, avg=37029.78, stdev=4509.40, samples=60 iops : min= 1648, max= 3522, avg=2314.33, stdev=281.86, samples=60 write: IOPS=578, BW=9255KiB/s (9478kB/s)(271MiB/30007msec) clat (usec): min=228, max=44332, avg=1474.27, stdev=1621.12 lat (usec): min=230, max=44333, avg=1475.70, stdev=1621.14 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 619], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1467], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min= 6848, max=14368, per=100.00%, avg=9257.68, stdev=975.99, samples=60 iops : min= 428, max= 898, avg=578.58, stdev=61.00, samples=60 lat (usec) : 50=45.57%, 100=34.05%, 250=0.29%, 500=0.72%, 750=0.72% lat (usec) : 1000=1.67% lat (msec) : 2=15.59%, 4=1.02%, 10=0.24%, 20=0.10%, 50=0.02% cpu : usr=1.55%, sys=3.62%, ctx=86795, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69426,17358,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=36.2MiB/s (37.9MB/s), 36.2MiB/s-36.2MiB/s (37.9MB/s-37.9MB/s), io=1085MiB (1137MB), run=30007-30007msec WRITE: bw=9255KiB/s (9478kB/s), 9255KiB/s-9255KiB/s (9478kB/s-9478kB/s), io=271MiB (284MB), run=30007-30007msec Disk stats (read/write): vdi: ios=69056/17260, merge=0/0, ticks=719/17931, in_queue=28186, util=94.19% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903782: Fri Jan 7 13:42:35 2022 read: IOPS=2359, BW=36.9MiB/s (38.7MB/s)(1109MiB/30085msec) slat (usec): min=2, max=221, avg= 9.94, stdev= 4.98 clat (nsec): min=713, max=2517.0M, avg=746943.65, stdev=32861054.49 lat (usec): min=23, max=2518.0k, avg=757.03, stdev=32861.05 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 43], | 80.00th=[ 48], 90.00th=[ 57], 95.00th=[ 73], | 99.00th=[ 196], 99.50th=[ 420], 99.90th=[ 1287], | 99.95th=[ 392168], 99.99th=[1988101] bw ( KiB/s): min=28928, max=63904, per=100.00%, avg=37859.08, stdev=5202.34, samples=60 iops : min= 1808, max= 3994, avg=2366.17, stdev=325.15, samples=60 write: IOPS=590, BW=9442KiB/s (9669kB/s)(277MiB/30085msec) slat (usec): min=3, max=195, avg=13.04, stdev= 5.98 clat (msec): min=8, max=2476, avg=81.66, stdev=66.00 lat (msec): min=8, max=2476, avg=81.68, stdev=66.00 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 74], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 96], 95.00th=[ 123], | 99.00th=[ 249], 99.50th=[ 575], 99.90th=[ 986], 99.95th=[ 1133], | 99.99th=[ 2022] bw ( KiB/s): min= 7584, max=14816, per=100.00%, avg=9444.37, stdev=994.83, samples=60 iops : min= 474, max= 926, avg=590.27, stdev=62.17, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.14%, 50=67.08% lat (usec) : 100=10.74%, 250=1.26%, 500=0.28%, 750=0.13%, 1000=0.09% lat (msec) : 2=0.05%, 10=0.03%, 20=0.27%, 50=0.15%, 100=17.76% lat (msec) : 250=1.61%, 500=0.10%, 750=0.07%, 1000=0.04%, 2000=0.03% lat (msec) : >=2000=0.01% cpu : usr=1.21%, sys=5.88%, ctx=86067, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=70983,17754,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=1109MiB (1163MB), run=30085-30085msec WRITE: bw=9442KiB/s (9669kB/s), 9442KiB/s-9442KiB/s (9669kB/s-9669kB/s), io=277MiB (291MB), run=30085-30085msec Disk stats (read/write): vdi: ios=70981/17716, merge=0/0, ticks=49059/1434514, in_queue=1499967, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904244: Fri Jan 7 13:43:12 2022 read: IOPS=2306, BW=36.0MiB/s (37.8MB/s)(1081MiB/30001msec) clat (usec): min=27, max=73011, avg=62.98, stdev=532.85 lat (usec): min=27, max=73011, avg=63.12, stdev=532.85 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 87], 99.50th=[ 99], 99.90th=[ 355], 99.95th=[12911], | 99.99th=[16712] bw ( KiB/s): min=23168, max=59872, per=100.00%, avg=36905.30, stdev=4821.40, samples=60 iops : min= 1448, max= 3742, avg=2306.58, stdev=301.37, samples=60 write: IOPS=576, BW=9231KiB/s (9452kB/s)(270MiB/30001msec) clat (usec): min=226, max=107975, avg=1471.35, stdev=1793.74 lat (usec): min=228, max=107978, avg=1472.81, stdev=1793.73 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 619], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1188], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1811], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[33424], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min= 5952, max=15424, per=100.00%, avg=9231.23, stdev=1129.45, samples=60 iops : min= 372, max= 964, avg=576.93, stdev=70.61, samples=60 lat (usec) : 50=45.23%, 100=34.38%, 250=0.30%, 500=0.71%, 750=0.76% lat (usec) : 1000=1.57% lat (msec) : 2=15.71%, 4=0.98%, 10=0.23%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.42%, sys=3.77%, ctx=86504, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69192,17308,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s (37.8MB/s-37.8MB/s), io=1081MiB (1134MB), run=30001-30001msec WRITE: bw=9231KiB/s (9452kB/s), 9231KiB/s-9231KiB/s (9452kB/s-9452kB/s), io=270MiB (284MB), run=30001-30001msec Disk stats (read/write): vdi: ios=68735/17190, merge=0/0, ticks=821/17873, in_queue=28215, util=94.38% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905218: Fri Jan 7 13:43:49 2022 read: IOPS=2336, BW=36.5MiB/s (38.3MB/s)(1102MiB/30186msec) slat (usec): min=2, max=821, avg= 9.83, stdev= 7.38 clat (nsec): min=760, max=2508.6M, avg=859324.92, stdev=35821584.16 lat (usec): min=23, max=2508.6k, avg=869.30, stdev=35821.57 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 45], | 80.00th=[ 50], 90.00th=[ 69], 95.00th=[ 153], | 99.00th=[ 1123], 99.50th=[ 1795], 99.90th=[ 6652], | 99.95th=[ 417334], 99.99th=[2499806] bw ( KiB/s): min=28192, max=60288, per=100.00%, avg=37620.97, stdev=4875.90, samples=60 iops : min= 1762, max= 3768, avg=2351.28, stdev=304.71, samples=60 write: IOPS=584, BW=9355KiB/s (9579kB/s)(276MiB/30186msec) slat (usec): min=3, max=492, avg=13.12, stdev= 9.21 clat (msec): min=3, max=3707, avg=167.51, stdev=119.52 lat (msec): min=3, max=3707, avg=167.52, stdev=119.52 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 110], 10.00th=[ 129], 20.00th=[ 138], | 30.00th=[ 146], 40.00th=[ 150], 50.00th=[ 155], 60.00th=[ 159], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 203], 95.00th=[ 224], | 99.00th=[ 575], 99.50th=[ 911], 99.90th=[ 1754], 99.95th=[ 2567], | 99.99th=[ 3708] bw ( KiB/s): min= 6784, max=12416, per=100.00%, avg=9362.50, stdev=1129.99, samples=60 iops : min= 424, max= 776, avg=585.15, stdev=70.62, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.07%, 50=64.30% lat (usec) : 100=9.69%, 250=2.80%, 500=1.17%, 750=0.55%, 1000=0.35% lat (msec) : 2=0.62%, 4=0.17%, 10=0.08%, 20=0.16%, 50=0.21% lat (msec) : 100=0.24%, 250=18.67%, 500=0.53%, 750=0.11%, 1000=0.06% lat (msec) : 2000=0.08%, >=2000=0.03% cpu : usr=1.19%, sys=5.68%, ctx=81966, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70537,17649,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=36.5MiB/s (38.3MB/s), 36.5MiB/s-36.5MiB/s (38.3MB/s-38.3MB/s), io=1102MiB (1156MB), run=30186-30186msec WRITE: bw=9355KiB/s (9579kB/s), 9355KiB/s-9355KiB/s (9579kB/s-9579kB/s), io=276MiB (289MB), run=30186-30186msec Disk stats (read/write): vdi: ios=70533/17573, merge=0/0, ticks=53369/2923869, in_queue=3000275, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905798: Fri Jan 7 13:44:26 2022 read: IOPS=2299, BW=35.9MiB/s (37.7MB/s)(1078MiB/30002msec) clat (usec): min=25, max=23878, avg=60.74, stdev=398.95 lat (usec): min=25, max=23879, avg=60.88, stdev=398.95 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 67], | 99.00th=[ 82], 99.50th=[ 93], 99.90th=[ 359], 99.95th=[12911], | 99.99th=[16450] bw ( KiB/s): min=29472, max=56352, per=100.00%, avg=36792.17, stdev=4140.67, samples=60 iops : min= 1842, max= 3522, avg=2299.50, stdev=258.80, samples=60 write: IOPS=574, BW=9195KiB/s (9415kB/s)(269MiB/30002msec) clat (usec): min=234, max=50760, avg=1487.08, stdev=1704.36 lat (usec): min=235, max=50761, avg=1488.56, stdev=1704.35 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 627], 10.00th=[ 873], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[17695], 99.95th=[42206], | 99.99th=[46924] bw ( KiB/s): min= 7584, max=14368, per=100.00%, avg=9195.27, stdev=947.96, samples=60 iops : min= 474, max= 898, avg=574.70, stdev=59.25, samples=60 lat (usec) : 50=47.66%, 100=32.05%, 250=0.21%, 500=0.70%, 750=0.77% lat (usec) : 1000=1.61% lat (msec) : 2=15.64%, 4=0.95%, 10=0.26%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.47%, sys=3.62%, ctx=86226, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68981,17241,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=35.9MiB/s (37.7MB/s), 35.9MiB/s-35.9MiB/s (37.7MB/s-37.7MB/s), io=1078MiB (1130MB), run=30002-30002msec WRITE: bw=9195KiB/s (9415kB/s), 9195KiB/s-9195KiB/s (9415kB/s-9415kB/s), io=269MiB (282MB), run=30002-30002msec Disk stats (read/write): vdi: ios=68455/17129, merge=0/0, ticks=724/18085, in_queue=28202, util=94.26% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906616: Fri Jan 7 13:45:03 2022 read: IOPS=2361, BW=36.9MiB/s (38.7MB/s)(1115MiB/30218msec) slat (usec): min=2, max=201070, avg=332.91, stdev=2044.75 clat (usec): min=554, max=2565.1k, avg=27171.10, stdev=38417.03 lat (usec): min=557, max=2565.1k, avg=27504.15, stdev=38478.48 clat percentiles (usec): | 1.00th=[ 1188], 5.00th=[ 6587], 10.00th=[ 14877], | 20.00th=[ 18482], 30.00th=[ 20841], 40.00th=[ 22938], | 50.00th=[ 24773], 60.00th=[ 26608], 70.00th=[ 28705], | 80.00th=[ 31589], 90.00th=[ 36439], 95.00th=[ 42206], | 99.00th=[ 85459], 99.50th=[ 181404], 99.90th=[ 238027], | 99.95th=[ 650118], 99.99th=[2004878] bw ( KiB/s): min=28864, max=77120, per=100.00%, avg=38034.75, stdev=6405.01, samples=60 iops : min= 1804, max= 4820, avg=2377.13, stdev=400.32, samples=60 write: IOPS=590, BW=9452KiB/s (9679kB/s)(279MiB/30218msec) slat (usec): min=3, max=179149, avg=339.75, stdev=1967.51 clat (msec): min=14, max=2891, avg=228.16, stdev=168.31 lat (msec): min=14, max=2891, avg=228.50, stdev=168.29 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 146], 10.00th=[ 167], 20.00th=[ 180], | 30.00th=[ 190], 40.00th=[ 199], 50.00th=[ 207], 60.00th=[ 215], | 70.00th=[ 224], 80.00th=[ 243], 90.00th=[ 268], 95.00th=[ 300], | 99.00th=[ 1167], 99.50th=[ 1401], 99.90th=[ 2635], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 7808, max=16032, per=100.00%, avg=9451.87, stdev=1122.83, samples=60 iops : min= 488, max= 1002, avg=590.73, stdev=70.18, samples=60 lat (usec) : 750=0.06%, 1000=0.33% lat (msec) : 2=1.63%, 4=1.53%, 10=0.95%, 20=16.17%, 50=56.97% lat (msec) : 100=2.26%, 250=16.90%, 500=2.76%, 750=0.09%, 1000=0.10% lat (msec) : 2000=0.21%, >=2000=0.04% cpu : usr=0.85%, sys=1.87%, ctx=9930, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=71374,17852,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=1115MiB (1169MB), run=30218-30218msec WRITE: bw=9452KiB/s (9679kB/s), 9452KiB/s-9452KiB/s (9679kB/s-9679kB/s), io=279MiB (292MB), run=30218-30218msec Disk stats (read/write): vdi: ios=71369/17756, merge=0/0, ticks=57720/3576950, in_queue=3666103, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908076: Fri Jan 7 13:45:40 2022 read: IOPS=2207, BW=68.0MiB/s (72.4MB/s)(2070MiB/30003msec) clat (usec): min=31, max=23213, avg=66.13, stdev=399.84 lat (usec): min=31, max=23213, avg=66.27, stdev=399.84 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 90], 99.50th=[ 102], 99.90th=[ 449], 99.95th=[11731], | 99.99th=[18220] bw ( KiB/s): min=55936, max=116288, per=100.00%, avg=70664.48, stdev=8689.93, samples=60 iops : min= 1748, max= 3634, avg=2208.25, stdev=271.57, samples=60 write: IOPS=552, BW=17.3MiB/s (18.1MB/s)(518MiB/30003msec) clat (usec): min=277, max=117220, avg=1534.03, stdev=1895.19 lat (usec): min=280, max=117223, avg=1537.01, stdev=1895.16 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 627], 10.00th=[ 873], 20.00th=[ 1106], | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1352], 60.00th=[ 1418], | 70.00th=[ 1516], 80.00th=[ 1663], 90.00th=[ 1909], 95.00th=[ 2180], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[39584], | 99.99th=[46400] bw ( KiB/s): min=14016, max=30080, per=100.00%, avg=17685.88, stdev=2068.59, samples=60 iops : min= 438, max= 940, avg=552.68, stdev=64.64, samples=60 lat (usec) : 50=31.72%, 100=47.84%, 250=0.33%, 500=0.63%, 750=0.81% lat (usec) : 1000=1.41% lat (msec) : 2=15.64%, 4=1.24%, 10=0.24%, 20=0.12%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.32%, sys=3.77%, ctx=82831, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66246,16581,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=68.0MiB/s (72.4MB/s), 68.0MiB/s-68.0MiB/s (72.4MB/s-72.4MB/s), io=2070MiB (2171MB), run=30003-30003msec WRITE: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=518MiB (543MB), run=30003-30003msec Disk stats (read/write): vdi: ios=65800/16481, merge=0/0, ticks=722/17782, in_queue=28181, util=94.15% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908675: Fri Jan 7 13:46:17 2022 read: IOPS=2206, BW=68.0MiB/s (72.3MB/s)(2069MiB/30001msec) slat (usec): min=5, max=195, avg=11.38, stdev= 3.80 clat (nsec): min=679, max=24998k, avg=54143.44, stdev=420032.75 lat (usec): min=28, max=25007, avg=65.68, stdev=420.07 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 45], 80.00th=[ 48], 90.00th=[ 53], 95.00th=[ 58], | 99.00th=[ 71], 99.50th=[ 81], 99.90th=[ 416], 99.95th=[12518], | 99.99th=[18220] bw ( KiB/s): min=49024, max=116096, per=100.00%, avg=70620.80, stdev=10287.28, samples=60 iops : min= 1532, max= 3628, avg=2206.90, stdev=321.48, samples=60 write: IOPS=552, BW=17.3MiB/s (18.1MB/s)(518MiB/30001msec) slat (usec): min=9, max=136, avg=15.04, stdev= 4.04 clat (usec): min=259, max=154520, avg=1524.75, stdev=2058.55 lat (usec): min=277, max=154536, avg=1539.96, stdev=2058.61 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 603], 10.00th=[ 873], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1401], | 70.00th=[ 1500], 80.00th=[ 1614], 90.00th=[ 1876], 95.00th=[ 2180], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[40633], | 99.99th=[46400] bw ( KiB/s): min=11840, max=29888, per=100.00%, avg=17681.07, stdev=2351.48, samples=60 iops : min= 370, max= 934, avg=552.53, stdev=73.48, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.04%, 50=68.34% lat (usec) : 100=11.34%, 250=0.09%, 500=0.69%, 750=0.81%, 1000=1.37% lat (msec) : 2=15.72%, 4=1.12%, 10=0.26%, 20=0.12%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=1.53%, sys=5.76%, ctx=82725, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66207,16577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=68.0MiB/s (72.3MB/s), 68.0MiB/s-68.0MiB/s (72.3MB/s-72.3MB/s), io=2069MiB (2169MB), run=30001-30001msec WRITE: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=518MiB (543MB), run=30001-30001msec Disk stats (read/write): vdi: ios=65795/16478, merge=0/0, ticks=766/17932, in_queue=28052, util=93.75% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909512: Fri Jan 7 13:46:54 2022 read: IOPS=2216, BW=69.3MiB/s (72.6MB/s)(2078MiB/30001msec) clat (usec): min=27, max=34825, avg=62.98, stdev=424.75 lat (usec): min=27, max=34825, avg=63.11, stdev=424.76 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 88], 99.50th=[ 103], 99.90th=[ 676], 99.95th=[12649], | 99.99th=[16909] bw ( KiB/s): min=59008, max=116992, per=99.85%, avg=70823.27, stdev=8473.85, samples=59 iops : min= 1844, max= 3656, avg=2213.22, stdev=264.81, samples=59 write: IOPS=554, BW=17.3MiB/s (18.2MB/s)(520MiB/30001msec) clat (usec): min=262, max=49200, avg=1539.50, stdev=1670.11 lat (usec): min=265, max=49203, avg=1542.43, stdev=1670.09 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 660], 10.00th=[ 930], 20.00th=[ 1139], | 30.00th=[ 1237], 40.00th=[ 1303], 50.00th=[ 1369], 60.00th=[ 1434], | 70.00th=[ 1532], 80.00th=[ 1663], 90.00th=[ 1893], 95.00th=[ 2180], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[37487], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=13696, max=30400, per=100.00%, avg=17760.07, stdev=2160.74, samples=59 iops : min= 428, max= 950, avg=555.00, stdev=67.52, samples=59 lat (usec) : 50=44.77%, 100=34.78%, 250=0.33%, 500=0.63%, 750=0.64% lat (usec) : 1000=1.28% lat (msec) : 2=15.97%, 4=1.23%, 10=0.25%, 20=0.11%, 50=0.02% cpu : usr=1.38%, sys=3.61%, ctx=83146, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66497,16645,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=69.3MiB/s (72.6MB/s), 69.3MiB/s-69.3MiB/s (72.6MB/s-72.6MB/s), io=2078MiB (2179MB), run=30001-30001msec WRITE: bw=17.3MiB/s (18.2MB/s), 17.3MiB/s-17.3MiB/s (18.2MB/s-18.2MB/s), io=520MiB (545MB), run=30001-30001msec Disk stats (read/write): vdi: ios=65954/16520, merge=0/0, ticks=760/17920, in_queue=28166, util=94.28% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910043: Fri Jan 7 13:47:31 2022 read: IOPS=2256, BW=70.5MiB/s (73.9MB/s)(2118MiB/30035msec) slat (usec): min=2, max=193, avg=11.13, stdev= 4.24 clat (nsec): min=766, max=2490.1M, avg=852519.19, stdev=36115622.96 lat (usec): min=26, max=2490.1k, avg=863.79, stdev=36115.62 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 39], 60.00th=[ 42], 70.00th=[ 44], | 80.00th=[ 47], 90.00th=[ 53], 95.00th=[ 59], | 99.00th=[ 89], 99.50th=[ 119], 99.90th=[ 1074], | 99.95th=[ 501220], 99.99th=[2298479] bw ( KiB/s): min=53632, max=112256, per=100.00%, avg=72295.60, stdev=8680.43, samples=60 iops : min= 1676, max= 3508, avg=2259.22, stdev=271.26, samples=60 write: IOPS=564, BW=17.6MiB/s (18.5MB/s)(530MiB/30035msec) slat (nsec): min=6127, max=96205, avg=15306.24, stdev=4474.60 clat (usec): min=556, max=96143, avg=14239.15, stdev=6678.10 lat (usec): min=568, max=96167, avg=14254.62, stdev=6678.33 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 7504], 10.00th=[ 9372], 20.00th=[11338], | 30.00th=[12387], 40.00th=[13042], 50.00th=[13566], 60.00th=[14222], | 70.00th=[14877], 80.00th=[15664], 90.00th=[18482], 95.00th=[22414], | 99.00th=[40109], 99.50th=[64750], 99.90th=[72877], 99.95th=[81265], | 99.99th=[94897] bw ( KiB/s): min=13824, max=28032, per=100.00%, avg=18075.52, stdev=1968.48, samples=60 iops : min= 432, max= 876, avg=564.85, stdev=61.51, samples=60 lat (nsec) : 1000=0.04% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.07%, 50=69.06% lat (usec) : 100=10.18%, 250=0.44%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.18%, 4=0.24%, 10=2.26%, 20=15.61%, 50=1.54% lat (msec) : 100=0.19%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.29%, sys=6.04%, ctx=83646, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67773,16951,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.9MB/s), 70.5MiB/s-70.5MiB/s (73.9MB/s-73.9MB/s), io=2118MiB (2221MB), run=30035-30035msec WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=530MiB (555MB), run=30035-30035msec Disk stats (read/write): vdi: ios=67464/16859, merge=0/0, ticks=51464/231521, in_queue=296603, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910968: Fri Jan 7 13:48:08 2022 read: IOPS=2190, BW=68.4MiB/s (71.8MB/s)(2053MiB/30001msec) clat (usec): min=28, max=61743, avg=65.65, stdev=489.41 lat (usec): min=28, max=61743, avg=65.79, stdev=489.41 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 93], 99.50th=[ 112], 99.90th=[ 701], 99.95th=[13960], | 99.99th=[16909] bw ( KiB/s): min=47104, max=97664, per=100.00%, avg=70085.33, stdev=8108.43, samples=60 iops : min= 1472, max= 3052, avg=2190.17, stdev=253.39, samples=60 write: IOPS=548, BW=17.1MiB/s (17.0MB/s)(514MiB/30001msec) clat (usec): min=267, max=173052, avg=1549.48, stdev=2137.05 lat (usec): min=270, max=173055, avg=1552.53, stdev=2137.03 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 898], 20.00th=[ 1123], | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1352], 60.00th=[ 1418], | 70.00th=[ 1516], 80.00th=[ 1647], 90.00th=[ 1909], 95.00th=[ 2180], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=12032, max=25536, per=100.00%, avg=17550.93, stdev=1923.84, samples=60 iops : min= 376, max= 798, avg=548.47, stdev=60.12, samples=60 lat (usec) : 50=40.11%, 100=39.28%, 250=0.45%, 500=0.67%, 750=0.73% lat (usec) : 1000=1.36% lat (msec) : 2=15.76%, 4=1.22%, 10=0.28%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.46%, sys=3.71%, ctx=82164, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=65705,16455,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=68.4MiB/s (71.8MB/s), 68.4MiB/s-68.4MiB/s (71.8MB/s-71.8MB/s), io=2053MiB (2153MB), run=30001-30001msec WRITE: bw=17.1MiB/s (17.0MB/s), 17.1MiB/s-17.1MiB/s (17.0MB/s-17.0MB/s), io=514MiB (539MB), run=30001-30001msec Disk stats (read/write): vdi: ios=65221/16359, merge=0/0, ticks=830/17958, in_queue=28247, util=94.37% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911957: Fri Jan 7 13:48:45 2022 read: IOPS=2273, BW=71.0MiB/s (74.5MB/s)(2137MiB/30087msec) slat (usec): min=2, max=136, avg=10.80, stdev= 4.47 clat (nsec): min=736, max=2522.1M, avg=824961.42, stdev=35237313.23 lat (usec): min=25, max=2522.1k, avg=835.92, stdev=35237.32 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 71], | 99.00th=[ 302], 99.50th=[ 725], 99.90th=[ 2671], | 99.95th=[ 534774], 99.99th=[2399142] bw ( KiB/s): min=56896, max=136000, per=100.00%, avg=72956.97, stdev=10645.47, samples=60 iops : min= 1778, max= 4250, avg=2279.88, stdev=332.69, samples=60 write: IOPS=568, BW=17.8MiB/s (18.6MB/s)(535MiB/30087msec) slat (usec): min=5, max=221, avg=16.53, stdev= 6.15 clat (msec): min=8, max=2085, avg=84.52, stdev=71.35 lat (msec): min=8, max=2085, avg=84.54, stdev=71.35 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 61], 10.00th=[ 65], 20.00th=[ 69], | 30.00th=[ 72], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 128], | 99.00th=[ 211], 99.50th=[ 558], 99.90th=[ 1183], 99.95th=[ 1452], | 99.99th=[ 1737] bw ( KiB/s): min=14912, max=31680, per=100.00%, avg=18207.02, stdev=2163.22, samples=60 iops : min= 466, max= 990, avg=568.97, stdev=67.60, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.08%, 50=64.97% lat (usec) : 100=12.83%, 250=1.05%, 500=0.38%, 750=0.16%, 1000=0.11% lat (msec) : 2=0.18%, 4=0.02%, 10=0.05%, 20=0.25%, 50=0.18% lat (msec) : 100=17.74%, 250=1.63%, 500=0.09%, 750=0.06%, 1000=0.03% lat (msec) : 2000=0.05%, >=2000=0.01% cpu : usr=1.46%, sys=5.84%, ctx=82803, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=68395,17115,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=71.0MiB/s (74.5MB/s), 71.0MiB/s-71.0MiB/s (74.5MB/s-74.5MB/s), io=2137MiB (2241MB), run=30087-30087msec WRITE: bw=17.8MiB/s (18.6MB/s), 17.8MiB/s-17.8MiB/s (18.6MB/s-18.6MB/s), io=535MiB (561MB), run=30087-30087msec Disk stats (read/write): vdi: ios=68393/17111, merge=0/0, ticks=51387/1435335, in_queue=1500118, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912588: Fri Jan 7 13:49:22 2022 read: IOPS=2214, BW=69.2MiB/s (72.6MB/s)(2076MiB/30002msec) clat (usec): min=28, max=77637, avg=64.60, stdev=556.33 lat (usec): min=28, max=77638, avg=64.74, stdev=556.33 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 71], | 99.00th=[ 93], 99.50th=[ 109], 99.90th=[ 562], 99.95th=[13173], | 99.99th=[17171] bw ( KiB/s): min=46848, max=141632, per=100.00%, avg=70869.80, stdev=12038.97, samples=60 iops : min= 1464, max= 4426, avg=2214.67, stdev=376.21, samples=60 write: IOPS=554, BW=17.3MiB/s (18.2MB/s)(520MiB/30002msec) clat (usec): min=275, max=100670, avg=1534.33, stdev=1819.03 lat (usec): min=278, max=100672, avg=1537.30, stdev=1818.98 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 545], 10.00th=[ 865], 20.00th=[ 1123], | 30.00th=[ 1221], 40.00th=[ 1287], 50.00th=[ 1352], 60.00th=[ 1418], | 70.00th=[ 1516], 80.00th=[ 1663], 90.00th=[ 1909], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[29230], 99.95th=[41681], | 99.99th=[43779] bw ( KiB/s): min=12224, max=36736, per=100.00%, avg=17738.33, stdev=2928.10, samples=60 iops : min= 382, max= 1148, avg=554.32, stdev=91.50, samples=60 lat (usec) : 50=44.44%, 100=34.98%, 250=0.43%, 500=0.92%, 750=0.71% lat (usec) : 1000=1.23% lat (msec) : 2=15.66%, 4=1.23%, 10=0.25%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.46%, sys=3.65%, ctx=83071, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66434,16629,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=69.2MiB/s (72.6MB/s), 69.2MiB/s-69.2MiB/s (72.6MB/s-72.6MB/s), io=2076MiB (2177MB), run=30002-30002msec WRITE: bw=17.3MiB/s (18.2MB/s), 17.3MiB/s-17.3MiB/s (18.2MB/s-18.2MB/s), io=520MiB (545MB), run=30002-30002msec Disk stats (read/write): vdi: ios=65937/16514, merge=0/0, ticks=853/17904, in_queue=28174, util=94.24% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913421: Fri Jan 7 13:49:59 2022 read: IOPS=2269, BW=70.9MiB/s (74.4MB/s)(2140MiB/30172msec) slat (usec): min=2, max=412, avg=10.99, stdev= 5.49 clat (nsec): min=663, max=2525.1M, avg=1087614.64, stdev=44834390.88 lat (usec): min=25, max=2525.1k, avg=1098.75, stdev=44834.38 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 46], 70.00th=[ 49], | 80.00th=[ 54], 90.00th=[ 66], 95.00th=[ 98], | 99.00th=[ 955], 99.50th=[ 1598], 99.90th=[ 2606], | 99.95th=[ 526386], 99.99th=[2499806] bw ( KiB/s): min=53482, max=144960, per=100.00%, avg=73063.35, stdev=11715.00, samples=60 iops : min= 1671, max= 4530, avg=2283.18, stdev=366.09, samples=60 write: IOPS=567, BW=17.7MiB/s (18.6MB/s)(536MiB/30172msec) slat (usec): min=4, max=203, avg=17.41, stdev= 7.05 clat (msec): min=11, max=2778, avg=171.62, stdev=95.95 lat (msec): min=11, max=2778, avg=171.64, stdev=95.95 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 116], 10.00th=[ 131], 20.00th=[ 148], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 161], 60.00th=[ 165], | 70.00th=[ 171], 80.00th=[ 182], 90.00th=[ 205], 95.00th=[ 228], | 99.00th=[ 435], 99.50th=[ 894], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 2433] bw ( KiB/s): min=13659, max=31104, per=100.00%, avg=18175.12, stdev=2210.70, samples=60 iops : min= 426, max= 972, avg=567.95, stdev=69.11, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.06%, 50=58.07% lat (usec) : 100=17.74%, 250=1.69%, 500=0.80%, 750=0.43%, 1000=0.25% lat (msec) : 2=0.51%, 4=0.17%, 20=0.17%, 50=0.19%, 100=0.29% lat (msec) : 250=18.74%, 500=0.47%, 750=0.07%, 1000=0.06%, 2000=0.07% lat (msec) : >=2000=0.03% cpu : usr=1.53%, sys=5.95%, ctx=80570, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=68488,17137,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=70.9MiB/s (74.4MB/s), 70.9MiB/s-70.9MiB/s (74.4MB/s-74.4MB/s), io=2140MiB (2244MB), run=30172-30172msec WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=536MiB (562MB), run=30172-30172msec Disk stats (read/write): vdi: ios=68488/17077, merge=0/0, ticks=70156/2909962, in_queue=3003466, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914734: Fri Jan 7 13:50:36 2022 read: IOPS=2218, BW=69.3MiB/s (72.7MB/s)(2080MiB/30001msec) clat (usec): min=27, max=64097, avg=63.64, stdev=485.14 lat (usec): min=27, max=64097, avg=63.78, stdev=485.14 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 449], 99.95th=[12649], | 99.99th=[18744] bw ( KiB/s): min=60736, max=101632, per=100.00%, avg=71005.70, stdev=7509.04, samples=60 iops : min= 1898, max= 3176, avg=2218.90, stdev=234.65, samples=60 write: IOPS=555, BW=17.4MiB/s (18.2MB/s)(521MiB/30001msec) clat (usec): min=275, max=71410, avg=1534.80, stdev=1707.86 lat (usec): min=278, max=71413, avg=1537.81, stdev=1707.85 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 660], 10.00th=[ 922], 20.00th=[ 1139], | 30.00th=[ 1237], 40.00th=[ 1287], 50.00th=[ 1352], 60.00th=[ 1434], | 70.00th=[ 1532], 80.00th=[ 1647], 90.00th=[ 1893], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[40109], | 99.99th=[45351] bw ( KiB/s): min=15583, max=26304, per=100.00%, avg=17777.18, stdev=1649.81, samples=60 iops : min= 486, max= 822, avg=555.52, stdev=51.58, samples=60 lat (usec) : 50=44.42%, 100=35.22%, 250=0.24%, 500=0.63%, 750=0.66% lat (usec) : 1000=1.23% lat (msec) : 2=16.03%, 4=1.20%, 10=0.22%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.29%, sys=3.77%, ctx=83233, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66559,16665,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=69.3MiB/s (72.7MB/s), 69.3MiB/s-69.3MiB/s (72.7MB/s-72.7MB/s), io=2080MiB (2181MB), run=30001-30001msec WRITE: bw=17.4MiB/s (18.2MB/s), 17.4MiB/s-17.4MiB/s (18.2MB/s-18.2MB/s), io=521MiB (546MB), run=30001-30001msec Disk stats (read/write): vdi: ios=66053/16543, merge=0/0, ticks=794/17928, in_queue=28216, util=94.48% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915357: Fri Jan 7 13:51:13 2022 read: IOPS=2252, BW=70.4MiB/s (73.8MB/s)(2130MiB/30264msec) slat (usec): min=2, max=228111, avg=348.72, stdev=2225.86 clat (usec): min=757, max=2723.9k, avg=28484.55, stdev=41962.11 lat (usec): min=770, max=2723.9k, avg=28833.41, stdev=42031.42 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 4948], 10.00th=[ 15270], | 20.00th=[ 19268], 30.00th=[ 21627], 40.00th=[ 23987], | 50.00th=[ 25822], 60.00th=[ 27919], 70.00th=[ 30278], | 80.00th=[ 33162], 90.00th=[ 38536], 95.00th=[ 44827], | 99.00th=[ 89654], 99.50th=[ 206570], 99.90th=[ 299893], | 99.95th=[ 450888], 99.99th=[2533360] bw ( KiB/s): min=56320, max=150464, per=100.00%, avg=72649.15, stdev=11907.62, samples=60 iops : min= 1760, max= 4702, avg=2270.27, stdev=372.11, samples=60 write: IOPS=563, BW=17.6MiB/s (18.5MB/s)(533MiB/30264msec) slat (usec): min=4, max=193077, avg=355.31, stdev=2147.53 clat (msec): min=5, max=3065, avg=239.12, stdev=216.02 lat (msec): min=21, max=3065, avg=239.48, stdev=215.99 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 140], 10.00th=[ 157], 20.00th=[ 176], | 30.00th=[ 194], 40.00th=[ 205], 50.00th=[ 213], 60.00th=[ 222], | 70.00th=[ 232], 80.00th=[ 247], 90.00th=[ 275], 95.00th=[ 330], | 99.00th=[ 1452], 99.50th=[ 1905], 99.90th=[ 3004], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min=14720, max=31488, per=100.00%, avg=18057.75, stdev=2208.05, samples=60 iops : min= 460, max= 984, avg=564.30, stdev=69.00, samples=60 lat (usec) : 1000=0.12% lat (msec) : 2=1.28%, 4=2.02%, 10=1.36%, 20=13.35%, 50=59.53% lat (msec) : 100=2.15%, 250=16.47%, 500=3.25%, 750=0.10%, 1000=0.08% lat (msec) : 2000=0.19%, >=2000=0.11% cpu : usr=1.04%, sys=1.98%, ctx=9555, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=68168,17063,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=70.4MiB/s (73.8MB/s), 70.4MiB/s-70.4MiB/s (73.8MB/s-73.8MB/s), io=2130MiB (2234MB), run=30264-30264msec WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=533MiB (559MB), run=30264-30264msec Disk stats (read/write): vdi: ios=68166/16983, merge=0/0, ticks=60433/3572833, in_queue=3672430, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916225: Fri Jan 7 13:51:50 2022 read: IOPS=2048, BW=128MiB/s (134MB/s)(3842MiB/30001msec) clat (usec): min=35, max=59569, avg=76.31, stdev=492.48 lat (usec): min=35, max=59569, avg=76.46, stdev=492.48 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 87], | 99.00th=[ 117], 99.50th=[ 139], 99.90th=[ 725], 99.95th=[13304], | 99.99th=[19530] bw ( KiB/s): min=104576, max=203776, per=100.00%, avg=131129.44, stdev=16297.78, samples=59 iops : min= 1634, max= 3184, avg=2048.86, stdev=254.64, samples=59 write: IOPS=514, BW=32.2MiB/s (33.7MB/s)(965MiB/30001msec) clat (usec): min=366, max=45607, avg=1624.39, stdev=1619.27 lat (usec): min=372, max=45613, avg=1630.62, stdev=1619.20 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 676], 10.00th=[ 922], 20.00th=[ 1205], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[17695], 99.95th=[38011], | 99.99th=[44303] bw ( KiB/s): min=26496, max=52864, per=99.99%, avg=32936.17, stdev=3610.60, samples=59 iops : min= 414, max= 826, avg=514.59, stdev=56.43, samples=59 lat (usec) : 50=7.13%, 100=71.10%, 250=1.52%, 500=0.52%, 750=0.89% lat (usec) : 1000=1.05% lat (msec) : 2=15.74%, 4=1.64%, 10=0.24%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.57%, sys=3.73%, ctx=76913, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61466,15441,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=3842MiB (4028MB), run=30001-30001msec WRITE: bw=32.2MiB/s (33.7MB/s), 32.2MiB/s-32.2MiB/s (33.7MB/s-33.7MB/s), io=965MiB (1012MB), run=30001-30001msec Disk stats (read/write): vdi: ios=61108/15351, merge=0/0, ticks=758/17238, in_queue=28164, util=94.17% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916844: Fri Jan 7 13:52:27 2022 read: IOPS=2064, BW=129MiB/s (135MB/s)(3872MiB/30001msec) slat (usec): min=6, max=153, avg=13.48, stdev= 4.47 clat (nsec): min=811, max=24767k, avg=62608.39, stdev=416836.83 lat (usec): min=34, max=24783, avg=76.25, stdev=416.97 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 100], 99.90th=[ 1205], 99.95th=[12649], | 99.99th=[17957] bw ( KiB/s): min=111488, max=212736, per=100.00%, avg=132795.20, stdev=15462.53, samples=59 iops : min= 1742, max= 3324, avg=2074.92, stdev=241.59, samples=59 write: IOPS=518, BW=32.4MiB/s (33.0MB/s)(972MiB/30001msec) slat (usec): min=11, max=257, avg=20.53, stdev= 6.43 clat (usec): min=334, max=43568, avg=1597.05, stdev=1643.43 lat (usec): min=351, max=43586, avg=1617.76, stdev=1643.54 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 652], 10.00th=[ 906], 20.00th=[ 1172], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36439], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min=25728, max=55424, per=100.00%, avg=33356.03, stdev=3909.70, samples=59 iops : min= 402, max= 866, avg=521.19, stdev=61.09, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=45.81%, 100=33.66% lat (usec) : 250=0.28%, 500=0.50%, 750=0.91%, 1000=1.14% lat (msec) : 2=15.75%, 4=1.49%, 10=0.23%, 20=0.15%, 50=0.02% cpu : usr=1.53%, sys=6.21%, ctx=77486, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61946,15550,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3872MiB (4060MB), run=30001-30001msec WRITE: bw=32.4MiB/s (33.0MB/s), 32.4MiB/s-32.4MiB/s (33.0MB/s-33.0MB/s), io=972MiB (1019MB), run=30001-30001msec Disk stats (read/write): vdi: ios=61640/15474, merge=0/0, ticks=700/17115, in_queue=27954, util=93.48% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917794: Fri Jan 7 13:53:04 2022 read: IOPS=2067, BW=129MiB/s (135MB/s)(3877MiB/30001msec) clat (usec): min=32, max=23293, avg=70.20, stdev=428.28 lat (usec): min=32, max=23293, avg=70.34, stdev=428.28 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 104], 99.50th=[ 122], 99.90th=[ 635], 99.95th=[13173], | 99.99th=[17695] bw ( KiB/s): min=91520, max=211072, per=100.00%, avg=132371.97, stdev=16348.42, samples=59 iops : min= 1430, max= 3298, avg=2068.29, stdev=255.44, samples=59 write: IOPS=519, BW=32.5MiB/s (34.0MB/s)(974MiB/30001msec) clat (usec): min=350, max=45543, avg=1631.35, stdev=1654.91 lat (usec): min=356, max=45548, avg=1637.59, stdev=1654.90 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 693], 10.00th=[ 988], 20.00th=[ 1237], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=25344, max=54400, per=100.00%, avg=33257.29, stdev=3741.61, samples=59 iops : min= 396, max= 850, avg=519.63, stdev=58.48, samples=59 lat (usec) : 50=22.96%, 100=56.04%, 250=0.79%, 500=0.53%, 750=0.70% lat (usec) : 1000=0.91% lat (msec) : 2=16.01%, 4=1.67%, 10=0.20%, 20=0.16%, 50=0.02% cpu : usr=1.62%, sys=3.72%, ctx=77616, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62028,15579,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3877MiB (4065MB), run=30001-30001msec WRITE: bw=32.5MiB/s (34.0MB/s), 32.5MiB/s-32.5MiB/s (34.0MB/s-34.0MB/s), io=974MiB (1021MB), run=30001-30001msec Disk stats (read/write): vdi: ios=61697/15487, merge=0/0, ticks=722/17501, in_queue=28055, util=93.86% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918736: Fri Jan 7 13:53:41 2022 read: IOPS=2082, BW=130MiB/s (136MB/s)(3909MiB/30034msec) slat (usec): min=4, max=504, avg=12.71, stdev= 4.68 clat (nsec): min=781, max=2525.6M, avg=996542.01, stdev=42577801.79 lat (usec): min=32, max=2525.6k, avg=1009.41, stdev=42577.82 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 37], 10.00th=[ 39], | 20.00th=[ 41], 30.00th=[ 42], 40.00th=[ 44], | 50.00th=[ 46], 60.00th=[ 49], 70.00th=[ 52], | 80.00th=[ 56], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 96], 99.50th=[ 127], 99.90th=[ 570], | 99.95th=[ 484443], 99.99th=[2499806] bw ( KiB/s): min=107008, max=222464, per=100.00%, avg=133446.65, stdev=16806.54, samples=60 iops : min= 1672, max= 3476, avg=2085.07, stdev=262.62, samples=60 write: IOPS=523, BW=32.7MiB/s (34.3MB/s)(983MiB/30034msec) slat (usec): min=9, max=227, avg=21.20, stdev= 7.06 clat (usec): min=661, max=75170, avg=15056.58, stdev=6655.71 lat (usec): min=675, max=75196, avg=15077.94, stdev=6656.23 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 8160], 10.00th=[ 9765], 20.00th=[11731], | 30.00th=[12911], 40.00th=[13566], 50.00th=[14353], 60.00th=[15139], | 70.00th=[15926], 80.00th=[16909], 90.00th=[20841], 95.00th=[23725], | 99.00th=[40109], 99.50th=[64750], 99.90th=[72877], 99.95th=[73925], | 99.99th=[74974] bw ( KiB/s): min=26880, max=55808, per=100.00%, avg=33529.72, stdev=3859.01, samples=60 iops : min= 420, max= 872, avg=523.87, stdev=60.32, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=52.10% lat (usec) : 100=27.03%, 250=0.58%, 500=0.04%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.06%, 4=0.38%, 10=1.79%, 20=15.64%, 50=2.04% lat (msec) : 100=0.18%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.60%, sys=5.86%, ctx=77412, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62548,15721,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3909MiB (4099MB), run=30034-30034msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=983MiB (1030MB), run=30034-30034msec Disk stats (read/write): vdi: ios=62139/15604, merge=0/0, ticks=56648/227110, in_queue=296258, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919315: Fri Jan 7 13:54:18 2022 read: IOPS=2044, BW=128MiB/s (134MB/s)(3835MiB/30010msec) clat (usec): min=32, max=27910, avg=68.80, stdev=435.91 lat (usec): min=32, max=27910, avg=68.94, stdev=435.92 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 93], 99.50th=[ 106], 99.90th=[ 506], 99.95th=[13960], | 99.99th=[17433] bw ( KiB/s): min=89984, max=212736, per=100.00%, avg=130922.65, stdev=17343.72, samples=60 iops : min= 1406, max= 3324, avg=2045.63, stdev=271.00, samples=60 write: IOPS=513, BW=32.1MiB/s (33.7MB/s)(964MiB/30010msec) clat (usec): min=354, max=92796, avg=1657.93, stdev=1875.58 lat (usec): min=359, max=92802, avg=1664.22, stdev=1875.66 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 685], 10.00th=[ 1004], 20.00th=[ 1237], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2024], 95.00th=[ 2311], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[38011], 99.95th=[42730], | 99.99th=[44827] bw ( KiB/s): min=22528, max=55296, per=100.00%, avg=32894.83, stdev=3989.84, samples=60 iops : min= 352, max= 864, avg=513.95, stdev=62.37, samples=60 lat (usec) : 50=26.37%, 100=53.03%, 250=0.42%, 500=0.48%, 750=0.71% lat (usec) : 1000=0.84% lat (msec) : 2=15.94%, 4=1.81%, 10=0.23%, 20=0.15%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.56%, sys=3.55%, ctx=76789, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61364,15419,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=3835MiB (4022MB), run=30010-30010msec WRITE: bw=32.1MiB/s (33.7MB/s), 32.1MiB/s-32.1MiB/s (33.7MB/s-33.7MB/s), io=964MiB (1010MB), run=30010-30010msec Disk stats (read/write): vdi: ios=61066/15340, merge=0/0, ticks=722/17718, in_queue=28150, util=94.20% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920165: Fri Jan 7 13:54:55 2022 read: IOPS=2092, BW=131MiB/s (137MB/s)(3937MiB/30104msec) slat (usec): min=2, max=325, avg=11.25, stdev= 7.13 clat (nsec): min=758, max=2513.3M, avg=936826.63, stdev=39444916.72 lat (usec): min=27, max=2513.4k, avg=948.25, stdev=39444.90 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 34], | 20.00th=[ 37], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 46], 70.00th=[ 51], | 80.00th=[ 56], 90.00th=[ 65], 95.00th=[ 77], | 99.00th=[ 355], 99.50th=[ 685], 99.90th=[ 1434], | 99.95th=[ 480248], 99.99th=[2499806] bw ( KiB/s): min=102144, max=240512, per=100.00%, avg=134383.32, stdev=18253.25, samples=60 iops : min= 1596, max= 3758, avg=2099.73, stdev=285.21, samples=60 write: IOPS=526, BW=32.9MiB/s (34.5MB/s)(990MiB/30104msec) slat (usec): min=7, max=352, avg=22.05, stdev=12.16 clat (msec): min=4, max=2271, avg=91.21, stdev=74.96 lat (msec): min=4, max=2271, avg=91.23, stdev=74.96 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 65], 10.00th=[ 71], 20.00th=[ 75], | 30.00th=[ 79], 40.00th=[ 81], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 107], 95.00th=[ 132], | 99.00th=[ 220], 99.50th=[ 584], 99.90th=[ 1200], 99.95th=[ 1737], | 99.99th=[ 2039] bw ( KiB/s): min=29056, max=55552, per=100.00%, avg=33700.45, stdev=3597.85, samples=60 iops : min= 454, max= 868, avg=526.57, stdev=56.22, samples=60 lat (nsec) : 1000=0.06% lat (usec) : 2=0.15%, 4=0.04%, 10=0.04%, 20=0.12%, 50=54.93% lat (usec) : 100=22.30%, 250=1.19%, 500=0.53%, 750=0.17%, 1000=0.10% lat (msec) : 2=0.20%, 4=0.01%, 10=0.03%, 20=0.29%, 50=0.14% lat (msec) : 100=17.06%, 250=2.39%, 500=0.09%, 750=0.05%, 1000=0.05% lat (msec) : 2000=0.04%, >=2000=0.02% cpu : usr=1.78%, sys=5.41%, ctx=76030, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=62993,15841,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3937MiB (4128MB), run=30104-30104msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=990MiB (1038MB), run=30104-30104msec Disk stats (read/write): vdi: ios=62988/15802, merge=0/0, ticks=51335/1431105, in_queue=1501860, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921316: Fri Jan 7 13:55:32 2022 read: IOPS=2060, BW=129MiB/s (135MB/s)(3865MiB/30007msec) clat (usec): min=32, max=50481, avg=70.62, stdev=469.76 lat (usec): min=32, max=50481, avg=70.77, stdev=469.76 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 97], 99.50th=[ 112], 99.90th=[ 537], 99.95th=[13698], | 99.99th=[17695] bw ( KiB/s): min=97920, max=212480, per=100.00%, avg=131933.87, stdev=15457.12, samples=60 iops : min= 1530, max= 3320, avg=2061.47, stdev=241.52, samples=60 write: IOPS=517, BW=32.3MiB/s (33.9MB/s)(970MiB/30007msec) clat (usec): min=344, max=93449, avg=1637.25, stdev=1813.33 lat (usec): min=351, max=93454, avg=1643.65, stdev=1813.28 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 693], 10.00th=[ 996], 20.00th=[ 1237], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 2008], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36963], 99.95th=[40633], | 99.99th=[47449] bw ( KiB/s): min=23808, max=55168, per=100.00%, avg=33100.80, stdev=3983.69, samples=60 iops : min= 372, max= 862, avg=517.20, stdev=62.25, samples=60 lat (usec) : 50=22.13%, 100=57.15%, 250=0.56%, 500=0.50%, 750=0.68% lat (usec) : 1000=0.89% lat (msec) : 2=15.95%, 4=1.76%, 10=0.21%, 20=0.15%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.46%, sys=3.85%, ctx=77371, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61844,15517,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3865MiB (4053MB), run=30007-30007msec WRITE: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=970MiB (1017MB), run=30007-30007msec Disk stats (read/write): vdi: ios=61371/15419, merge=0/0, ticks=764/17577, in_queue=28174, util=94.27% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922076: Fri Jan 7 13:56:10 2022 read: IOPS=2102, BW=131MiB/s (138MB/s)(3970MiB/30205msec) slat (usec): min=2, max=384, avg=10.36, stdev= 5.51 clat (nsec): min=695, max=2522.0M, avg=1070927.41, stdev=42964713.01 lat (usec): min=28, max=2522.0k, avg=1081.44, stdev=42964.69 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 48], 70.00th=[ 53], | 80.00th=[ 59], 90.00th=[ 71], 95.00th=[ 120], | 99.00th=[ 1254], 99.50th=[ 2212], 99.90th=[ 3392], | 99.95th=[ 650118], 99.99th=[2499806] bw ( KiB/s): min=100681, max=295040, per=100.00%, avg=135510.67, stdev=24882.27, samples=60 iops : min= 1573, max= 4610, avg=2117.33, stdev=388.80, samples=60 write: IOPS=528, BW=33.0MiB/s (34.6MB/s)(998MiB/30205msec) slat (usec): min=6, max=365, avg=21.98, stdev=10.47 clat (msec): min=9, max=3922, avg=184.85, stdev=155.50 lat (msec): min=9, max=3922, avg=184.87, stdev=155.50 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 121], 10.00th=[ 138], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 167], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 215], 95.00th=[ 239], | 99.00th=[ 435], 99.50th=[ 1301], 99.90th=[ 2534], 99.95th=[ 3440], | 99.99th=[ 3842] bw ( KiB/s): min=26752, max=63360, per=100.00%, avg=33848.52, stdev=4495.63, samples=60 iops : min= 418, max= 990, avg=528.87, stdev=70.26, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.06%, 50=52.37% lat (usec) : 100=22.89%, 250=1.74%, 500=0.92%, 750=0.57%, 1000=0.27% lat (msec) : 2=0.50%, 4=0.42%, 10=0.01%, 20=0.08%, 50=0.31% lat (msec) : 100=0.18%, 250=18.80%, 500=0.54%, 750=0.06%, 1000=0.02% lat (msec) : 2000=0.10%, >=2000=0.06% cpu : usr=1.76%, sys=5.13%, ctx=74359, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=63516,15964,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3970MiB (4163MB), run=30205-30205msec WRITE: bw=33.0MiB/s (34.6MB/s), 33.0MiB/s-33.0MiB/s (34.6MB/s-34.6MB/s), io=998MiB (1046MB), run=30205-30205msec Disk stats (read/write): vdi: ios=63515/15866, merge=0/0, ticks=63251/2909944, in_queue=3004400, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923044: Fri Jan 7 13:56:46 2022 read: IOPS=2063, BW=129MiB/s (135MB/s)(3869MiB/30002msec) clat (usec): min=32, max=24902, avg=66.31, stdev=417.90 lat (usec): min=32, max=24902, avg=66.44, stdev=417.91 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 56], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 88], 99.50th=[ 98], 99.90th=[ 255], 99.95th=[12518], | 99.99th=[17433] bw ( KiB/s): min=111872, max=212352, per=100.00%, avg=132081.20, stdev=15078.45, samples=60 iops : min= 1748, max= 3318, avg=2063.75, stdev=235.61, samples=60 write: IOPS=517, BW=32.4MiB/s (33.9MB/s)(971MiB/30002msec) clat (usec): min=360, max=94914, avg=1652.24, stdev=1819.04 lat (usec): min=365, max=94920, avg=1658.30, stdev=1819.01 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 717], 10.00th=[ 1029], 20.00th=[ 1254], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2008], 95.00th=[ 2311], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[37487], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min=24320, max=55168, per=100.00%, avg=33154.30, stdev=3933.94, samples=60 iops : min= 380, max= 862, avg=518.03, stdev=61.47, samples=60 lat (usec) : 50=33.32%, 100=46.25%, 250=0.29%, 500=0.47%, 750=0.61% lat (usec) : 1000=0.81% lat (msec) : 2=16.11%, 4=1.75%, 10=0.21%, 20=0.15%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.44%, sys=3.52%, ctx=77460, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61909,15541,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3869MiB (4057MB), run=30002-30002msec WRITE: bw=32.4MiB/s (33.9MB/s), 32.4MiB/s-32.4MiB/s (33.9MB/s-33.9MB/s), io=971MiB (1018MB), run=30002-30002msec Disk stats (read/write): vdi: ios=61428/15436, merge=0/0, ticks=694/17841, in_queue=28226, util=94.39% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923715: Fri Jan 7 13:57:24 2022 read: IOPS=2091, BW=131MiB/s (137MB/s)(3954MiB/30251msec) slat (usec): min=2, max=267215, avg=378.84, stdev=2586.99 clat (usec): min=898, max=2685.0k, avg=30768.27, stdev=49004.46 lat (usec): min=901, max=2685.0k, avg=31147.26, stdev=49075.77 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 30], | 70.00th=[ 33], 80.00th=[ 36], 90.00th=[ 42], 95.00th=[ 48], | 99.00th=[ 94], 99.50th=[ 222], 99.90th=[ 284], 99.95th=[ 609], | 99.99th=[ 2567] bw ( KiB/s): min=104960, max=264704, per=100.00%, avg=134819.93, stdev=21456.69, samples=60 iops : min= 1640, max= 4136, avg=2106.55, stdev=335.27, samples=60 write: IOPS=525, BW=32.8MiB/s (34.4MB/s)(993MiB/30251msec) slat (usec): min=6, max=104560, avg=369.38, stdev=1699.26 clat (msec): min=15, max=3386, avg=256.22, stdev=131.90 lat (msec): min=17, max=3389, avg=256.59, stdev=131.90 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 176], 10.00th=[ 203], 20.00th=[ 222], | 30.00th=[ 230], 40.00th=[ 236], 50.00th=[ 243], 60.00th=[ 251], | 70.00th=[ 259], 80.00th=[ 275], 90.00th=[ 305], 95.00th=[ 338], | 99.00th=[ 625], 99.50th=[ 961], 99.90th=[ 2467], 99.95th=[ 2467], | 99.99th=[ 3272] bw ( KiB/s): min=25728, max=52352, per=100.00%, avg=33642.78, stdev=3549.93, samples=60 iops : min= 402, max= 818, avg=525.65, stdev=55.47, samples=60 lat (usec) : 1000=0.01% lat (msec) : 2=0.67%, 4=2.28%, 10=1.84%, 20=9.61%, 50=62.50% lat (msec) : 100=2.72%, 250=12.12%, 500=7.93%, 750=0.16%, 1000=0.04% lat (msec) : 2000=0.08%, >=2000=0.05% cpu : usr=1.64%, sys=1.92%, ctx=8881, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=63267,15895,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3954MiB (4146MB), run=30251-30251msec WRITE: bw=32.8MiB/s (34.4MB/s), 32.8MiB/s-32.8MiB/s (34.4MB/s-34.4MB/s), io=993MiB (1042MB), run=30251-30251msec Disk stats (read/write): vdi: ios=63261/15815, merge=0/0, ticks=67764/3569715, in_queue=3674272, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924580: Fri Jan 7 13:58:00 2022 read: IOPS=1965, BW=154MiB/s (161MB/s)(4607MiB/30001msec) clat (usec): min=40, max=59347, avg=85.64, stdev=560.24 lat (usec): min=40, max=59347, avg=85.79, stdev=560.24 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 55], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 67], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 79], 90.00th=[ 86], 95.00th=[ 94], | 99.00th=[ 124], 99.50th=[ 149], 99.90th=[ 1778], 99.95th=[14222], | 99.99th=[20317] bw ( KiB/s): min=125280, max=247520, per=100.00%, avg=157244.92, stdev=18514.18, samples=60 iops : min= 1566, max= 3094, avg=1965.55, stdev=231.43, samples=60 write: IOPS=494, BW=38.6MiB/s (40.5MB/s)(1158MiB/30001msec) clat (usec): min=408, max=49397, avg=1665.38, stdev=1716.95 lat (usec): min=415, max=49405, avg=1674.26, stdev=1716.86 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 685], 10.00th=[ 889], 20.00th=[ 1205], | 30.00th=[ 1336], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1811], 90.00th=[ 2073], 95.00th=[ 2409], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[17695], 99.95th=[42730], | 99.99th=[45876] bw ( KiB/s): min=28800, max=64480, per=100.00%, avg=39534.58, stdev=4472.31, samples=60 iops : min= 360, max= 806, avg=494.17, stdev=55.91, samples=60 lat (usec) : 50=0.83%, 100=76.48%, 250=2.40%, 500=0.35%, 750=1.10% lat (usec) : 1000=1.19% lat (msec) : 2=15.10%, 4=2.10%, 10=0.23%, 20=0.18%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.70%, sys=3.74%, ctx=73796, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=58965,14826,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4607MiB (4830MB), run=30001-30001msec WRITE: bw=38.6MiB/s (40.5MB/s), 38.6MiB/s-38.6MiB/s (40.5MB/s-40.5MB/s), io=1158MiB (1215MB), run=30001-30001msec Disk stats (read/write): vdi: ios=58720/14752, merge=0/0, ticks=909/16891, in_queue=28122, util=93.91% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925581: Fri Jan 7 13:58:37 2022 read: IOPS=1984, BW=155MiB/s (163MB/s)(4650MiB/30001msec) slat (usec): min=6, max=285, avg=14.13, stdev= 4.90 clat (nsec): min=757, max=27994k, avg=67048.41, stdev=475791.56 lat (usec): min=39, max=28004, avg=81.33, stdev=475.84 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 72], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 1237], 99.95th=[14091], | 99.99th=[20055] bw ( KiB/s): min=125440, max=231200, per=100.00%, avg=158734.95, stdev=17884.16, samples=60 iops : min= 1568, max= 2890, avg=1984.18, stdev=223.56, samples=60 write: IOPS=498, BW=38.0MiB/s (40.9MB/s)(1169MiB/30001msec) slat (usec): min=12, max=257, avg=22.86, stdev= 7.19 clat (usec): min=393, max=44401, avg=1650.61, stdev=1654.41 lat (usec): min=412, max=44420, avg=1673.65, stdev=1654.30 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 685], 10.00th=[ 922], 20.00th=[ 1221], | 30.00th=[ 1336], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36439], 99.95th=[41681], | 99.99th=[44303] bw ( KiB/s): min=33440, max=59840, per=100.00%, avg=39902.52, stdev=3715.67, samples=60 iops : min= 418, max= 748, avg=498.77, stdev=46.46, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.02%, 50=38.81% lat (usec) : 100=40.64%, 250=0.29%, 500=0.40%, 750=0.96%, 1000=1.07% lat (msec) : 2=15.36%, 4=2.03%, 10=0.20%, 20=0.17%, 50=0.03% cpu : usr=1.74%, sys=5.92%, ctx=74475, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59524,14964,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=4650MiB (4876MB), run=30001-30001msec WRITE: bw=38.0MiB/s (40.9MB/s), 38.0MiB/s-38.0MiB/s (40.9MB/s-40.9MB/s), io=1169MiB (1226MB), run=30001-30001msec Disk stats (read/write): vdi: ios=59093/14858, merge=0/0, ticks=831/16982, in_queue=27898, util=93.31% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926123: Fri Jan 7 13:59:14 2022 read: IOPS=1985, BW=155MiB/s (163MB/s)(4654MiB/30002msec) clat (usec): min=34, max=25803, avg=73.27, stdev=452.44 lat (usec): min=34, max=25803, avg=73.42, stdev=452.45 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 100], 99.50th=[ 116], 99.90th=[ 2278], 99.95th=[13566], | 99.99th=[18744] bw ( KiB/s): min=118240, max=255040, per=100.00%, avg=158861.33, stdev=18601.55, samples=60 iops : min= 1478, max= 3188, avg=1985.77, stdev=232.52, samples=60 write: IOPS=499, BW=38.0MiB/s (40.9MB/s)(1170MiB/30002msec) clat (usec): min=390, max=47997, avg=1696.09, stdev=1695.90 lat (usec): min=398, max=48004, avg=1703.84, stdev=1695.85 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 709], 10.00th=[ 1029], 20.00th=[ 1287], | 30.00th=[ 1369], 40.00th=[ 1450], 50.00th=[ 1516], 60.00th=[ 1614], | 70.00th=[ 1713], 80.00th=[ 1860], 90.00th=[ 2114], 95.00th=[ 2409], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[39584], 99.95th=[41681], | 99.99th=[44303] bw ( KiB/s): min=33280, max=66240, per=100.00%, avg=39922.67, stdev=4311.25, samples=60 iops : min= 416, max= 828, avg=499.03, stdev=53.89, samples=60 lat (usec) : 50=15.37%, 100=63.76%, 250=0.65%, 500=0.39%, 750=0.78% lat (usec) : 1000=0.76% lat (msec) : 2=15.57%, 4=2.34%, 10=0.18%, 20=0.18%, 50=0.03% cpu : usr=1.78%, sys=3.51%, ctx=74554, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59573,14972,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=4654MiB (4880MB), run=30002-30002msec WRITE: bw=38.0MiB/s (40.9MB/s), 38.0MiB/s-38.0MiB/s (40.9MB/s-40.9MB/s), io=1170MiB (1227MB), run=30002-30002msec Disk stats (read/write): vdi: ios=59212/14887, merge=0/0, ticks=798/17444, in_queue=28068, util=93.87% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926958: Fri Jan 7 13:59:51 2022 read: IOPS=2042, BW=160MiB/s (167MB/s)(4793MiB/30044msec) slat (usec): min=3, max=232, avg=13.18, stdev= 4.77 clat (nsec): min=724, max=2502.0M, avg=991885.09, stdev=39680184.70 lat (usec): min=31, max=2502.0k, avg=1005.23, stdev=39680.18 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 51], | 80.00th=[ 56], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 98], 99.50th=[ 121], 99.90th=[ 1123], | 99.95th=[ 734004], 99.99th=[2499806] bw ( KiB/s): min=127840, max=303360, per=100.00%, avg=163597.47, stdev=23311.32, samples=60 iops : min= 1598, max= 3792, avg=2044.97, stdev=291.39, samples=60 write: IOPS=513, BW=40.1MiB/s (42.0MB/s)(1204MiB/30044msec) slat (usec): min=9, max=197, avg=22.56, stdev= 7.21 clat (usec): min=1789, max=70888, avg=15454.84, stdev=6218.81 lat (usec): min=1803, max=70914, avg=15477.57, stdev=6218.81 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 9896], 10.00th=[10814], 20.00th=[11994], | 30.00th=[13042], 40.00th=[13829], 50.00th=[14615], 60.00th=[15664], | 70.00th=[16450], 80.00th=[17433], 90.00th=[20055], 95.00th=[24511], | 99.00th=[36439], 99.50th=[64750], 99.90th=[68682], 99.95th=[69731], | 99.99th=[70779] bw ( KiB/s): min=32480, max=76960, per=100.00%, avg=41098.62, stdev=5411.21, samples=60 iops : min= 406, max= 962, avg=513.72, stdev=67.65, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.04%, 50=54.10% lat (usec) : 100=24.96%, 250=0.58%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.32%, 10=0.78%, 20=16.98%, 50=1.85% lat (msec) : 100=0.17%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.56%, sys=6.02%, ctx=75888, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61350,15416,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=160MiB/s (167MB/s), 160MiB/s-160MiB/s (167MB/s-167MB/s), io=4793MiB (5026MB), run=30044-30044msec WRITE: bw=40.1MiB/s (42.0MB/s), 40.1MiB/s-40.1MiB/s (42.0MB/s-42.0MB/s), io=1204MiB (1263MB), run=30044-30044msec Disk stats (read/write): vdi: ios=61053/15334, merge=0/0, ticks=52077/229389, in_queue=296745, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927606: Fri Jan 7 14:00:28 2022 read: IOPS=1977, BW=154MiB/s (162MB/s)(4635MiB/30001msec) clat (usec): min=33, max=25258, avg=71.45, stdev=466.10 lat (usec): min=33, max=25258, avg=71.59, stdev=466.10 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 94], 99.50th=[ 106], 99.90th=[ 1303], 99.95th=[13960], | 99.99th=[18482] bw ( KiB/s): min=127775, max=245440, per=100.00%, avg=158230.35, stdev=18189.58, samples=60 iops : min= 1597, max= 3068, avg=1977.83, stdev=227.39, samples=60 write: IOPS=497, BW=38.8MiB/s (40.7MB/s)(1166MiB/30001msec) clat (usec): min=385, max=45429, avg=1710.86, stdev=1692.18 lat (usec): min=392, max=45437, avg=1718.61, stdev=1692.11 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 734], 10.00th=[ 1057], 20.00th=[ 1287], | 30.00th=[ 1385], 40.00th=[ 1450], 50.00th=[ 1532], 60.00th=[ 1614], | 70.00th=[ 1713], 80.00th=[ 1844], 90.00th=[ 2114], 95.00th=[ 2409], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[37487], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min=33667, max=64160, per=100.00%, avg=39790.37, stdev=4154.45, samples=60 iops : min= 420, max= 802, avg=497.32, stdev=51.97, samples=60 lat (usec) : 50=22.30%, 100=57.06%, 250=0.44%, 500=0.35%, 750=0.74% lat (usec) : 1000=0.74% lat (msec) : 2=15.58%, 4=2.37%, 10=0.22%, 20=0.17%, 50=0.03% cpu : usr=1.42%, sys=3.63%, ctx=74253, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59327,14919,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=4635MiB (4860MB), run=30001-30001msec WRITE: bw=38.8MiB/s (40.7MB/s), 38.8MiB/s-38.8MiB/s (40.7MB/s-40.7MB/s), io=1166MiB (1222MB), run=30001-30001msec Disk stats (read/write): vdi: ios=58985/14835, merge=0/0, ticks=806/17692, in_queue=28217, util=94.55% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928862: Fri Jan 7 14:01:05 2022 read: IOPS=2031, BW=159MiB/s (166MB/s)(4777MiB/30096msec) slat (usec): min=2, max=218, avg=11.22, stdev= 4.46 clat (nsec): min=896, max=2512.5M, avg=942762.04, stdev=37954259.19 lat (usec): min=29, max=2512.5k, avg=954.13, stdev=37954.25 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 52], | 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 77], | 99.00th=[ 351], 99.50th=[ 758], 99.90th=[ 1876], | 99.95th=[ 574620], 99.99th=[2499806] bw ( KiB/s): min=123680, max=293600, per=100.00%, avg=163088.63, stdev=22459.53, samples=60 iops : min= 1546, max= 3670, avg=2038.57, stdev=280.75, samples=60 write: IOPS=510, BW=39.9MiB/s (41.8MB/s)(1200MiB/30096msec) slat (usec): min=7, max=196, avg=24.06, stdev= 9.87 clat (msec): min=4, max=2610, avg=94.10, stdev=86.16 lat (msec): min=5, max=2610, avg=94.13, stdev=86.16 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 84], 50.00th=[ 86], 60.00th=[ 89], | 70.00th=[ 92], 80.00th=[ 97], 90.00th=[ 110], 95.00th=[ 133], | 99.00th=[ 264], 99.50th=[ 617], 99.90th=[ 1368], 99.95th=[ 1770], | 99.99th=[ 2400] bw ( KiB/s): min=34240, max=68480, per=100.00%, avg=40848.03, stdev=4423.95, samples=60 iops : min= 428, max= 856, avg=510.55, stdev=55.32, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.03%, 50=52.99% lat (usec) : 100=24.88%, 250=0.90%, 500=0.45%, 750=0.19%, 1000=0.08% lat (msec) : 2=0.25%, 10=0.01%, 20=0.34%, 50=0.12%, 100=16.39% lat (msec) : 250=3.03%, 500=0.10%, 750=0.05%, 1000=0.03%, 2000=0.07% lat (msec) : >=2000=0.01% cpu : usr=1.95%, sys=5.03%, ctx=74176, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=61147,15364,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=4777MiB (5009MB), run=30096-30096msec WRITE: bw=39.9MiB/s (41.8MB/s), 39.9MiB/s-39.9MiB/s (41.8MB/s-41.8MB/s), io=1200MiB (1259MB), run=30096-30096msec Disk stats (read/write): vdi: ios=61146/15347, merge=0/0, ticks=52764/1433713, in_queue=1500890, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929907: Fri Jan 7 14:01:42 2022 read: IOPS=2000, BW=156MiB/s (164MB/s)(4689MiB/30001msec) clat (usec): min=33, max=65081, avg=75.31, stdev=581.46 lat (usec): min=33, max=65081, avg=75.45, stdev=581.46 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 99], 99.50th=[ 116], 99.90th=[ 898], 99.95th=[14222], | 99.99th=[18482] bw ( KiB/s): min=125600, max=265440, per=100.00%, avg=160056.28, stdev=20332.43, samples=60 iops : min= 1570, max= 3318, avg=2000.70, stdev=254.15, samples=60 write: IOPS=502, BW=39.3MiB/s (41.2MB/s)(1178MiB/30001msec) clat (usec): min=393, max=44734, avg=1673.54, stdev=1556.88 lat (usec): min=400, max=44742, avg=1681.30, stdev=1556.81 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 709], 10.00th=[ 1020], 20.00th=[ 1270], | 30.00th=[ 1369], 40.00th=[ 1434], 50.00th=[ 1516], 60.00th=[ 1598], | 70.00th=[ 1713], 80.00th=[ 1844], 90.00th=[ 2089], 95.00th=[ 2376], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[16581], 99.95th=[40109], | 99.99th=[44303] bw ( KiB/s): min=31360, max=68800, per=100.00%, avg=40209.35, stdev=4744.02, samples=60 iops : min= 392, max= 860, avg=502.62, stdev=59.30, samples=60 lat (usec) : 50=12.88%, 100=66.27%, 250=0.65%, 500=0.36%, 750=0.82% lat (usec) : 1000=0.78% lat (msec) : 2=15.60%, 4=2.26%, 10=0.18%, 20=0.17%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.58%, sys=3.81%, ctx=75104, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60020,15078,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4689MiB (4917MB), run=30001-30001msec WRITE: bw=39.3MiB/s (41.2MB/s), 39.3MiB/s-39.3MiB/s (41.2MB/s-41.2MB/s), io=1178MiB (1235MB), run=30001-30001msec Disk stats (read/write): vdi: ios=59566/14970, merge=0/0, ticks=888/17247, in_queue=28018, util=93.82% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930582: Fri Jan 7 14:02:20 2022 read: IOPS=2039, BW=159MiB/s (167MB/s)(4813MiB/30204msec) slat (usec): min=2, max=223, avg=11.12, stdev= 5.20 clat (nsec): min=740, max=2526.4M, avg=1157225.61, stdev=43643616.27 lat (usec): min=30, max=2526.4k, avg=1168.51, stdev=43643.61 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 35], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 46], 60.00th=[ 50], 70.00th=[ 57], | 80.00th=[ 64], 90.00th=[ 77], 95.00th=[ 137], | 99.00th=[ 1647], 99.50th=[ 2868], 99.90th=[ 4424], | 99.95th=[ 960496], 99.99th=[2499806] bw ( KiB/s): min=125440, max=339520, per=100.00%, avg=164285.65, stdev=27721.25, samples=60 iops : min= 1568, max= 4244, avg=2053.57, stdev=346.51, samples=60 write: IOPS=512, BW=40.0MiB/s (41.9MB/s)(1208MiB/30204msec) slat (usec): min=10, max=178, avg=25.46, stdev=10.70 clat (msec): min=4, max=3125, avg=190.60, stdev=146.44 lat (msec): min=5, max=3125, avg=190.62, stdev=146.44 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 132], 10.00th=[ 140], 20.00th=[ 157], | 30.00th=[ 169], 40.00th=[ 174], 50.00th=[ 180], 60.00th=[ 184], | 70.00th=[ 190], 80.00th=[ 201], 90.00th=[ 226], 95.00th=[ 249], | 99.00th=[ 439], 99.50th=[ 1020], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min=31520, max=70880, per=100.00%, avg=40977.42, stdev=5128.77, samples=60 iops : min= 394, max= 886, avg=512.22, stdev=64.11, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.04%, 50=48.54% lat (usec) : 100=26.38%, 250=1.90%, 500=0.89%, 750=0.50%, 1000=0.39% lat (msec) : 2=0.55%, 4=0.57%, 10=0.03%, 20=0.09%, 50=0.30% lat (msec) : 100=0.26%, 250=18.49%, 500=0.77%, 750=0.06%, 1000=0.04% lat (msec) : 2000=0.08%, >=2000=0.05% cpu : usr=1.96%, sys=5.09%, ctx=72103, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=61606,15465,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4813MiB (5047MB), run=30204-30204msec WRITE: bw=40.0MiB/s (41.9MB/s), 40.0MiB/s-40.0MiB/s (41.9MB/s-41.9MB/s), io=1208MiB (1267MB), run=30204-30204msec Disk stats (read/write): vdi: ios=61605/15396, merge=0/0, ticks=66332/2911952, in_queue=3002926, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931408: Fri Jan 7 14:02:56 2022 read: IOPS=1960, BW=153MiB/s (161MB/s)(4595MiB/30002msec) clat (usec): min=32, max=28889, avg=71.43, stdev=481.50 lat (usec): min=33, max=28889, avg=71.57, stdev=481.50 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 95], 99.50th=[ 109], 99.90th=[ 1401], 99.95th=[14222], | 99.99th=[19530] bw ( KiB/s): min=113440, max=224160, per=100.00%, avg=156859.98, stdev=19803.76, samples=60 iops : min= 1418, max= 2802, avg=1960.73, stdev=247.56, samples=60 write: IOPS=493, BW=38.5MiB/s (40.4MB/s)(1156MiB/30002msec) clat (usec): min=404, max=43765, avg=1728.24, stdev=1717.67 lat (usec): min=411, max=43769, avg=1735.67, stdev=1717.61 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 750], 10.00th=[ 1074], 20.00th=[ 1303], | 30.00th=[ 1385], 40.00th=[ 1467], 50.00th=[ 1532], 60.00th=[ 1614], | 70.00th=[ 1713], 80.00th=[ 1860], 90.00th=[ 2114], 95.00th=[ 2409], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[36963], 99.95th=[38536], | 99.99th=[43254] bw ( KiB/s): min=29920, max=58240, per=100.00%, avg=39447.78, stdev=3854.56, samples=60 iops : min= 374, max= 728, avg=493.07, stdev=48.20, samples=60 lat (usec) : 50=26.44%, 100=52.87%, 250=0.47%, 500=0.37%, 750=0.67% lat (usec) : 1000=0.69% lat (msec) : 2=15.69%, 4=2.34%, 10=0.21%, 20=0.21%, 50=0.03% cpu : usr=1.57%, sys=3.33%, ctx=73622, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=58819,14793,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=4595MiB (4818MB), run=30002-30002msec WRITE: bw=38.5MiB/s (40.4MB/s), 38.5MiB/s-38.5MiB/s (40.4MB/s-40.4MB/s), io=1156MiB (1212MB), run=30002-30002msec Disk stats (read/write): vdi: ios=58518/14694, merge=0/0, ticks=819/17752, in_queue=28274, util=94.59% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932350: Fri Jan 7 14:03:34 2022 read: IOPS=2017, BW=158MiB/s (165MB/s)(4763MiB/30224msec) slat (usec): min=2, max=264256, avg=389.47, stdev=2697.74 clat (usec): min=1001, max=2559.1k, avg=31776.87, stdev=42809.05 lat (usec): min=1006, max=2559.2k, avg=32166.48, stdev=42901.68 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 25], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 32], | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 44], 95.00th=[ 51], | 99.00th=[ 89], 99.50th=[ 230], 99.90th=[ 300], 99.95th=[ 600], | 99.99th=[ 2232] bw ( KiB/s): min=128320, max=365280, per=100.00%, avg=162410.52, stdev=31144.93, samples=60 iops : min= 1604, max= 4566, avg=2030.12, stdev=389.32, samples=60 write: IOPS=506, BW=39.6MiB/s (41.5MB/s)(1197MiB/30224msec) slat (usec): min=8, max=45000, avg=398.47, stdev=1604.31 clat (msec): min=15, max=3661, avg=266.08, stdev=163.98 lat (msec): min=18, max=3661, avg=266.48, stdev=163.97 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 174], 10.00th=[ 197], 20.00th=[ 226], | 30.00th=[ 236], 40.00th=[ 245], 50.00th=[ 253], 60.00th=[ 259], | 70.00th=[ 268], 80.00th=[ 284], 90.00th=[ 305], 95.00th=[ 342], | 99.00th=[ 684], 99.50th=[ 1435], 99.90th=[ 2802], 99.95th=[ 2802], | 99.99th=[ 3675] bw ( KiB/s): min=33120, max=75360, per=99.85%, avg=40492.02, stdev=5353.20, samples=60 iops : min= 414, max= 942, avg=506.15, stdev=66.92, samples=60 lat (msec) : 2=0.66%, 4=2.13%, 10=1.88%, 20=7.42%, 50=64.02% lat (msec) : 100=3.80%, 250=9.27%, 500=10.39%, 750=0.21%, 1000=0.04% lat (msec) : 2000=0.11%, >=2000=0.07% cpu : usr=1.78%, sys=1.84%, ctx=8569, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=60965,15321,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=158MiB/s (165MB/s), 158MiB/s-158MiB/s (165MB/s-165MB/s), io=4763MiB (4994MB), run=30224-30224msec WRITE: bw=39.6MiB/s (41.5MB/s), 39.6MiB/s-39.6MiB/s (41.5MB/s-41.5MB/s), io=1197MiB (1255MB), run=30224-30224msec Disk stats (read/write): vdi: ios=60962/15225, merge=0/0, ticks=57210/3568006, in_queue=3673398, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932850: Fri Jan 7 14:04:10 2022 read: IOPS=1751, BW=219MiB/s (230MB/s)(6570MiB/30001msec) clat (usec): min=47, max=28795, avg=94.05, stdev=513.10 lat (usec): min=48, max=28795, avg=94.20, stdev=513.10 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 61], 10.00th=[ 62], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 98], 95.00th=[ 108], | 99.00th=[ 135], 99.50th=[ 155], 99.90th=[ 8225], 99.95th=[14877], | 99.99th=[20317] bw ( KiB/s): min=119296, max=375808, per=100.00%, avg=224367.31, stdev=33421.56, samples=59 iops : min= 932, max= 2936, avg=1752.86, stdev=261.11, samples=59 write: IOPS=440, BW=55.1MiB/s (57.8MB/s)(1653MiB/30001msec) clat (usec): min=518, max=209379, avg=1873.03, stdev=2633.91 lat (usec): min=530, max=209391, avg=1885.41, stdev=2633.96 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 914], 10.00th=[ 1090], 20.00th=[ 1270], | 30.00th=[ 1434], 40.00th=[ 1549], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2278], 95.00th=[ 2638], | 99.00th=[ 9634], 99.50th=[ 10290], 99.90th=[ 37487], 99.95th=[ 42206], | 99.99th=[100140] bw ( KiB/s): min=29696, max=97792, per=99.97%, avg=56417.36, stdev=7608.14, samples=59 iops : min= 232, max= 764, avg=440.75, stdev=59.44, samples=59 lat (usec) : 50=0.01%, 100=73.15%, 250=6.56%, 500=0.05%, 750=0.55% lat (usec) : 1000=0.87% lat (msec) : 2=14.71%, 4=3.45%, 10=0.41%, 20=0.20%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.74%, sys=3.54%, ctx=65796, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=52561,13227,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=6570MiB (6889MB), run=30001-30001msec WRITE: bw=55.1MiB/s (57.8MB/s), 55.1MiB/s-55.1MiB/s (57.8MB/s-57.8MB/s), io=1653MiB (1734MB), run=30001-30001msec Disk stats (read/write): vdi: ios=52364/13175, merge=0/0, ticks=788/17444, in_queue=28150, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933835: Fri Jan 7 14:04:47 2022 read: IOPS=1797, BW=225MiB/s (236MB/s)(6742MiB/30002msec) slat (usec): min=8, max=173, avg=15.21, stdev= 4.62 clat (nsec): min=960, max=25998k, avg=73243.54, stdev=498596.22 lat (usec): min=45, max=26011, avg=88.61, stdev=498.65 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 77], | 99.00th=[ 92], 99.50th=[ 103], 99.90th=[10290], 99.95th=[14877], | 99.99th=[19530] bw ( KiB/s): min=169472, max=367616, per=100.00%, avg=230134.85, stdev=28070.54, samples=60 iops : min= 1324, max= 2872, avg=1797.92, stdev=219.30, samples=60 write: IOPS=451, BW=56.5MiB/s (59.2MB/s)(1694MiB/30002msec) slat (usec): min=16, max=150, avg=27.70, stdev= 6.11 clat (usec): min=514, max=45020, avg=1825.97, stdev=1682.18 lat (usec): min=540, max=45052, avg=1853.86, stdev=1682.18 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 791], 10.00th=[ 1012], 20.00th=[ 1352], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2040], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[ 9634], 99.50th=[10421], 99.90th=[39060], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min=48384, max=94976, per=100.00%, avg=57815.23, stdev=6074.55, samples=60 iops : min= 378, max= 742, avg=451.67, stdev=47.46, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=20.60% lat (usec) : 100=58.82%, 250=0.36%, 500=0.01%, 750=0.79%, 1000=1.21% lat (msec) : 2=13.70%, 4=4.06%, 10=0.18%, 20=0.22%, 50=0.03% cpu : usr=1.55%, sys=5.66%, ctx=67496, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53936,13551,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6742MiB (7069MB), run=30002-30002msec WRITE: bw=56.5MiB/s (59.2MB/s), 56.5MiB/s-56.5MiB/s (59.2MB/s-59.2MB/s), io=1694MiB (1776MB), run=30002-30002msec Disk stats (read/write): vdi: ios=53597/13460, merge=0/0, ticks=824/17098, in_queue=28044, util=93.74% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934418: Fri Jan 7 14:05:24 2022 read: IOPS=1759, BW=220MiB/s (231MB/s)(6597MiB/30002msec) clat (usec): min=38, max=28357, avg=80.30, stdev=534.89 lat (usec): min=38, max=28358, avg=80.44, stdev=534.89 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 78], 95.00th=[ 85], | 99.00th=[ 104], 99.50th=[ 119], 99.90th=[10552], 99.95th=[16450], | 99.99th=[21103] bw ( KiB/s): min=162816, max=383744, per=100.00%, avg=225184.27, stdev=30041.94, samples=60 iops : min= 1272, max= 2998, avg=1759.25, stdev=234.70, samples=60 write: IOPS=442, BW=55.3MiB/s (57.0MB/s)(1659MiB/30002msec) clat (usec): min=524, max=141661, avg=1920.67, stdev=2152.40 lat (usec): min=537, max=141674, avg=1932.78, stdev=2152.35 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 807], 10.00th=[ 1139], 20.00th=[ 1434], | 30.00th=[ 1532], 40.00th=[ 1631], 50.00th=[ 1713], 60.00th=[ 1827], | 70.00th=[ 1926], 80.00th=[ 2089], 90.00th=[ 2376], 95.00th=[ 2671], | 99.00th=[10028], 99.50th=[10421], 99.90th=[39584], 99.95th=[42206], | 99.99th=[45876] bw ( KiB/s): min=36864, max=99584, per=100.00%, avg=56629.05, stdev=7116.69, samples=60 iops : min= 288, max= 778, avg=442.40, stdev=55.60, samples=60 lat (usec) : 50=2.09%, 100=76.79%, 250=0.90%, 500=0.03%, 750=0.74% lat (usec) : 1000=0.86% lat (msec) : 2=13.42%, 4=4.66%, 10=0.23%, 20=0.25%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.65%, sys=3.24%, ctx=66060, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=52776,13273,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=6597MiB (6917MB), run=30002-30002msec WRITE: bw=55.3MiB/s (57.0MB/s), 55.3MiB/s-55.3MiB/s (57.0MB/s-57.0MB/s), io=1659MiB (1740MB), run=30002-30002msec Disk stats (read/write): vdi: ios=52498/13211, merge=0/0, ticks=851/17768, in_queue=28244, util=94.48% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935724: Fri Jan 7 14:06:01 2022 read: IOPS=1820, BW=228MiB/s (239MB/s)(6839MiB/30053msec) slat (usec): min=5, max=439, avg=14.86, stdev= 8.00 clat (nsec): min=866, max=2497.7M, avg=1383196.83, stdev=50262765.60 lat (usec): min=34, max=2497.7k, avg=1398.24, stdev=50262.74 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 48], | 50.00th=[ 50], 60.00th=[ 54], 70.00th=[ 59], | 80.00th=[ 66], 90.00th=[ 76], 95.00th=[ 86], | 99.00th=[ 157], 99.50th=[ 253], 99.90th=[ 79168], | 99.95th=[1266680], 99.99th=[2499806] bw ( KiB/s): min=180480, max=377344, per=100.00%, avg=233423.60, stdev=27704.19, samples=60 iops : min= 1410, max= 2948, avg=1823.62, stdev=216.44, samples=60 write: IOPS=457, BW=57.2MiB/s (59.0MB/s)(1719MiB/30053msec) slat (usec): min=14, max=821, avg=31.50, stdev=16.69 clat (usec): min=730, max=75819, avg=16249.84, stdev=6751.73 lat (usec): min=749, max=75847, avg=16281.53, stdev=6750.25 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 9503], 10.00th=[10683], 20.00th=[12387], | 30.00th=[13435], 40.00th=[14615], 50.00th=[15533], 60.00th=[16450], | 70.00th=[17171], 80.00th=[18744], 90.00th=[21365], 95.00th=[25560], | 99.00th=[43254], 99.50th=[64750], 99.90th=[71828], 99.95th=[74974], | 99.99th=[76022] bw ( KiB/s): min=44032, max=96256, per=100.00%, avg=58638.75, stdev=6282.19, samples=60 iops : min= 344, max= 752, avg=458.10, stdev=49.08, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.10%, 4=0.01%, 10=0.02%, 20=0.05%, 50=39.33% lat (usec) : 100=38.40%, 250=1.58%, 500=0.26%, 750=0.04%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.34%, 10=1.06%, 20=16.03%, 50=2.46% lat (msec) : 100=0.19%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=2.04%, sys=5.34%, ctx=67172, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54711,13749,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6839MiB (7171MB), run=30053-30053msec WRITE: bw=57.2MiB/s (59.0MB/s), 57.2MiB/s-57.2MiB/s (59.0MB/s-59.0MB/s), io=1719MiB (1802MB), run=30053-30053msec Disk stats (read/write): vdi: ios=54359/13658, merge=0/0, ticks=70374/215220, in_queue=296456, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936742: Fri Jan 7 14:06:38 2022 read: IOPS=1780, BW=223MiB/s (233MB/s)(6679MiB/30002msec) clat (usec): min=46, max=33612, avg=92.06, stdev=528.89 lat (usec): min=46, max=33612, avg=92.21, stdev=528.89 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 65], | 30.00th=[ 68], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 93], 95.00th=[ 100], | 99.00th=[ 121], 99.50th=[ 135], 99.90th=[10028], 99.95th=[15533], | 99.99th=[20317] bw ( KiB/s): min=163328, max=370176, per=100.00%, avg=227984.00, stdev=28666.31, samples=60 iops : min= 1276, max= 2892, avg=1781.12, stdev=223.96, samples=60 write: IOPS=447, BW=55.9MiB/s (58.6MB/s)(1677MiB/30002msec) clat (usec): min=528, max=129253, avg=1847.39, stdev=2030.97 lat (usec): min=542, max=129264, avg=1860.71, stdev=2030.83 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 791], 10.00th=[ 1020], 20.00th=[ 1369], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1680], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9896], 99.50th=[10421], 99.90th=[36439], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min=41216, max=96000, per=100.00%, avg=57247.60, stdev=6700.87, samples=60 iops : min= 322, max= 750, avg=447.23, stdev=52.36, samples=60 lat (usec) : 50=0.05%, 100=76.04%, 250=3.72%, 500=0.02%, 750=0.77% lat (usec) : 1000=1.16% lat (msec) : 2=13.80%, 4=3.98%, 10=0.19%, 20=0.23%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.69%, sys=3.54%, ctx=66853, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53432,13418,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=223MiB/s (233MB/s), 223MiB/s-223MiB/s (233MB/s-233MB/s), io=6679MiB (7003MB), run=30002-30002msec WRITE: bw=55.9MiB/s (58.6MB/s), 55.9MiB/s-55.9MiB/s (58.6MB/s-58.6MB/s), io=1677MiB (1759MB), run=30002-30002msec Disk stats (read/write): vdi: ios=53225/13356, merge=0/0, ticks=852/17075, in_queue=28200, util=94.12% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937330: Fri Jan 7 14:07:16 2022 read: IOPS=1820, BW=228MiB/s (239MB/s)(6851MiB/30106msec) slat (usec): min=3, max=178, avg=12.75, stdev= 5.35 clat (nsec): min=836, max=2522.9M, avg=1173248.11, stdev=43059558.04 lat (usec): min=35, max=2522.9k, avg=1186.18, stdev=43059.55 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 44], 30.00th=[ 45], 40.00th=[ 48], | 50.00th=[ 50], 60.00th=[ 55], 70.00th=[ 62], | 80.00th=[ 74], 90.00th=[ 85], 95.00th=[ 100], | 99.00th=[ 474], 99.50th=[ 1012], 99.90th=[ 110625], | 99.95th=[ 817890], 99.99th=[2499806] bw ( KiB/s): min=193792, max=474368, per=100.00%, avg=233857.32, stdev=37849.75, samples=60 iops : min= 1514, max= 3706, avg=1826.98, stdev=295.70, samples=60 write: IOPS=457, BW=57.2MiB/s (59.0MB/s)(1722MiB/30106msec) slat (usec): min=10, max=265, avg=36.68, stdev=17.03 clat (msec): min=2, max=2338, avg=104.52, stdev=75.65 lat (msec): min=2, max=2338, avg=104.56, stdev=75.65 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 81], 10.00th=[ 85], 20.00th=[ 90], | 30.00th=[ 92], 40.00th=[ 95], 50.00th=[ 97], 60.00th=[ 100], | 70.00th=[ 104], 80.00th=[ 109], 90.00th=[ 118], 95.00th=[ 146], | 99.00th=[ 222], 99.50th=[ 502], 99.90th=[ 1334], 99.95th=[ 1754], | 99.99th=[ 2265] bw ( KiB/s): min=48128, max=109824, per=100.00%, avg=58553.10, stdev=7596.51, samples=60 iops : min= 376, max= 858, avg=457.40, stdev=59.34, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=39.93% lat (usec) : 100=36.00%, 250=2.60%, 500=0.56%, 750=0.21%, 1000=0.14% lat (msec) : 2=0.27%, 4=0.05%, 10=0.01%, 20=0.21%, 50=0.34% lat (msec) : 100=11.78%, 250=7.60%, 500=0.09%, 750=0.06%, 1000=0.02% lat (msec) : 2000=0.06%, >=2000=0.02% cpu : usr=2.36%, sys=4.86%, ctx=66310, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=54805,13772,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6851MiB (7183MB), run=30106-30106msec WRITE: bw=57.2MiB/s (59.0MB/s), 57.2MiB/s-57.2MiB/s (59.0MB/s-59.0MB/s), io=1722MiB (1805MB), run=30106-30106msec Disk stats (read/write): vdi: ios=54805/13746, merge=0/0, ticks=60715/1427498, in_queue=1500386, util=99.72% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938225: Fri Jan 7 14:07:52 2022 read: IOPS=1793, BW=224MiB/s (235MB/s)(6725MiB/30001msec) clat (usec): min=37, max=28247, avg=78.39, stdev=514.62 lat (usec): min=37, max=28247, avg=78.53, stdev=514.62 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 82], | 99.00th=[ 97], 99.50th=[ 108], 99.90th=[10683], 99.95th=[14746], | 99.99th=[20841] bw ( KiB/s): min=175616, max=382717, per=99.94%, avg=229385.00, stdev=30986.82, samples=59 iops : min= 1372, max= 2989, avg=1792.05, stdev=242.00, samples=59 write: IOPS=450, BW=56.3MiB/s (59.0MB/s)(1688MiB/30001msec) clat (usec): min=511, max=71607, avg=1888.27, stdev=1768.76 lat (usec): min=523, max=71621, avg=1900.71, stdev=1768.64 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 832], 10.00th=[ 1188], 20.00th=[ 1450], | 30.00th=[ 1532], 40.00th=[ 1614], 50.00th=[ 1713], 60.00th=[ 1811], | 70.00th=[ 1909], 80.00th=[ 2057], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[ 9896], 99.50th=[10421], 99.90th=[36439], 99.95th=[41157], | 99.99th=[46924] bw ( KiB/s): min=43520, max=99270, per=100.00%, avg=57658.92, stdev=7108.59, samples=59 iops : min= 340, max= 775, avg=450.42, stdev=55.50, samples=59 lat (usec) : 50=3.20%, 100=76.10%, 250=0.54%, 500=0.01%, 750=0.73% lat (usec) : 1000=0.67% lat (msec) : 2=13.95%, 4=4.36%, 10=0.18%, 20=0.23%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.54%, sys=3.35%, ctx=67309, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53796,13507,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=224MiB/s (235MB/s), 224MiB/s-224MiB/s (235MB/s-235MB/s), io=6725MiB (7051MB), run=30001-30001msec WRITE: bw=56.3MiB/s (59.0MB/s), 56.3MiB/s-56.3MiB/s (59.0MB/s-59.0MB/s), io=1688MiB (1770MB), run=30001-30001msec Disk stats (read/write): vdi: ios=53397/13408, merge=0/0, ticks=845/17683, in_queue=28216, util=94.35% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938826: Fri Jan 7 14:08:30 2022 read: IOPS=1838, BW=230MiB/s (241MB/s)(6948MiB/30233msec) slat (usec): min=3, max=270, avg=11.98, stdev= 5.54 clat (nsec): min=889, max=2524.1M, avg=1315863.13, stdev=45428272.02 lat (usec): min=34, max=2524.1k, avg=1328.00, stdev=45428.31 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 56], 70.00th=[ 66], | 80.00th=[ 77], 90.00th=[ 93], 95.00th=[ 202], | 99.00th=[ 2057], 99.50th=[ 3884], 99.90th=[ 145753], | 99.95th=[1115685], 99.99th=[2499806] bw ( KiB/s): min=172800, max=527104, per=100.00%, avg=237166.82, stdev=43914.67, samples=60 iops : min= 1350, max= 4118, avg=1852.85, stdev=343.08, samples=60 write: IOPS=461, BW=57.7MiB/s (60.6MB/s)(1746MiB/30233msec) slat (usec): min=10, max=393, avg=33.51, stdev=15.39 clat (msec): min=5, max=2683, avg=211.11, stdev=98.48 lat (msec): min=5, max=2683, avg=211.14, stdev=98.48 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 192], 40.00th=[ 197], 50.00th=[ 201], 60.00th=[ 205], | 70.00th=[ 213], 80.00th=[ 226], 90.00th=[ 251], 95.00th=[ 271], | 99.00th=[ 584], 99.50th=[ 860], 99.90th=[ 1385], 99.95th=[ 1703], | 99.99th=[ 2400] bw ( KiB/s): min=50176, max=109568, per=100.00%, avg=59182.62, stdev=7380.50, samples=60 iops : min= 392, max= 856, avg=462.33, stdev=57.66, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=40.12% lat (usec) : 100=33.07%, 250=3.00%, 500=1.06%, 750=0.55%, 1000=0.42% lat (msec) : 2=0.79%, 4=0.45%, 10=0.29%, 20=0.05%, 50=0.37% lat (msec) : 100=0.22%, 250=17.46%, 500=1.79%, 750=0.09%, 1000=0.08% lat (msec) : 2000=0.09%, >=2000=0.02% cpu : usr=2.21%, sys=4.74%, ctx=64604, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=55585,13967,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6948MiB (7286MB), run=30233-30233msec WRITE: bw=57.7MiB/s (60.6MB/s), 57.7MiB/s-57.7MiB/s (60.6MB/s-60.6MB/s), io=1746MiB (1831MB), run=30233-30233msec Disk stats (read/write): vdi: ios=55582/13877, merge=0/0, ticks=64584/2910316, in_queue=3001673, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939645: Fri Jan 7 14:09:06 2022 read: IOPS=1778, BW=222MiB/s (233MB/s)(6669MiB/30002msec) clat (usec): min=37, max=63917, avg=80.29, stdev=596.33 lat (usec): min=38, max=63917, avg=80.43, stdev=596.33 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 84], | 99.00th=[ 104], 99.50th=[ 118], 99.90th=[10290], 99.95th=[15926], | 99.99th=[22152] bw ( KiB/s): min=133632, max=388096, per=100.00%, avg=227632.45, stdev=31169.87, samples=60 iops : min= 1044, max= 3032, avg=1778.37, stdev=243.52, samples=60 write: IOPS=446, BW=55.8MiB/s (58.5MB/s)(1675MiB/30002msec) clat (usec): min=516, max=136481, avg=1898.86, stdev=2066.52 lat (usec): min=529, max=136492, avg=1911.09, stdev=2066.45 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 824], 10.00th=[ 1172], 20.00th=[ 1434], | 30.00th=[ 1532], 40.00th=[ 1614], 50.00th=[ 1713], 60.00th=[ 1795], | 70.00th=[ 1909], 80.00th=[ 2073], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[ 9896], 99.50th=[10421], 99.90th=[38011], 99.95th=[42206], | 99.99th=[43254] bw ( KiB/s): min=36864, max=101120, per=100.00%, avg=57164.15, stdev=7418.14, samples=60 iops : min= 288, max= 790, avg=446.57, stdev=57.96, samples=60 lat (usec) : 50=3.44%, 100=75.47%, 250=0.90%, 500=0.02%, 750=0.76% lat (usec) : 1000=0.73% lat (msec) : 2=13.84%, 4=4.39%, 10=0.21%, 20=0.23%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.62%, sys=3.31%, ctx=66757, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53348,13398,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=6669MiB (6992MB), run=30002-30002msec WRITE: bw=55.8MiB/s (58.5MB/s), 55.8MiB/s-55.8MiB/s (58.5MB/s-58.5MB/s), io=1675MiB (1756MB), run=30002-30002msec Disk stats (read/write): vdi: ios=53050/13317, merge=0/0, ticks=919/17722, in_queue=28255, util=94.43% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940612: Fri Jan 7 14:09:44 2022 read: IOPS=1828, BW=229MiB/s (240MB/s)(6919MiB/30280msec) slat (usec): min=3, max=259226, avg=427.20, stdev=2788.54 clat (usec): min=1567, max=2583.2k, avg=35132.47, stdev=47122.82 lat (usec): min=1574, max=2583.2k, avg=35559.82, stdev=47224.50 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 20], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 30], 50.00th=[ 32], 60.00th=[ 35], | 70.00th=[ 38], 80.00th=[ 42], 90.00th=[ 47], 95.00th=[ 55], | 99.00th=[ 107], 99.50th=[ 253], 99.90th=[ 326], 99.95th=[ 852], | 99.99th=[ 2534] bw ( KiB/s): min=164352, max=513280, per=100.00%, avg=235914.93, stdev=44230.56, samples=60 iops : min= 1284, max= 4010, avg=1843.07, stdev=345.55, samples=60 write: IOPS=459, BW=57.5MiB/s (60.2MB/s)(1740MiB/30280msec) slat (usec): min=11, max=226206, avg=446.21, stdev=2581.63 clat (msec): min=10, max=3340, avg=293.13, stdev=200.96 lat (msec): min=10, max=3340, avg=293.58, stdev=200.94 clat percentiles (msec): | 1.00th=[ 46], 5.00th=[ 144], 10.00th=[ 197], 20.00th=[ 247], | 30.00th=[ 264], 40.00th=[ 271], 50.00th=[ 279], 60.00th=[ 288], | 70.00th=[ 296], 80.00th=[ 309], 90.00th=[ 347], 95.00th=[ 393], | 99.00th=[ 944], 99.50th=[ 1770], 99.90th=[ 3339], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min=47872, max=100864, per=99.98%, avg=58824.17, stdev=7107.05, samples=60 iops : min= 374, max= 788, avg=459.53, stdev=55.52, samples=60 lat (msec) : 2=0.06%, 4=1.10%, 10=3.86%, 20=3.95%, 50=65.45% lat (msec) : 100=5.13%, 250=4.14%, 500=15.67%, 750=0.31%, 1000=0.09% lat (msec) : 2000=0.11%, >=2000=0.11% cpu : usr=2.29%, sys=1.96%, ctx=7728, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=55355,13919,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6919MiB (7255MB), run=30280-30280msec WRITE: bw=57.5MiB/s (60.2MB/s), 57.5MiB/s-57.5MiB/s (60.2MB/s-60.2MB/s), io=1740MiB (1824MB), run=30280-30280msec Disk stats (read/write): vdi: ios=55352/13815, merge=0/0, ticks=65009/3566667, in_queue=3676490, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941231: Fri Jan 7 14:10:20 2022 read: IOPS=1180, BW=295MiB/s (309MB/s)(8854MiB/30006msec) clat (usec): min=60, max=22867, avg=109.18, stdev=396.88 lat (usec): min=60, max=22867, avg=109.35, stdev=396.88 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 75], 10.00th=[ 78], 20.00th=[ 81], | 30.00th=[ 83], 40.00th=[ 87], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 110], 90.00th=[ 121], 95.00th=[ 133], | 99.00th=[ 167], 99.50th=[ 192], 99.90th=[ 6718], 99.95th=[11469], | 99.99th=[16188] bw ( KiB/s): min=214528, max=555520, per=100.00%, avg=302235.68, stdev=52090.94, samples=60 iops : min= 838, max= 2170, avg=1180.60, stdev=203.48, samples=60 write: IOPS=296, BW=74.2MiB/s (77.8MB/s)(2227MiB/30006msec) clat (usec): min=865, max=61967, avg=2897.06, stdev=2849.35 lat (usec): min=889, max=61991, avg=2924.19, stdev=2848.98 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1090], 10.00th=[ 1352], 20.00th=[ 1663], | 30.00th=[ 1909], 40.00th=[ 2089], 50.00th=[ 2245], 60.00th=[ 2409], | 70.00th=[ 2573], 80.00th=[ 2933], 90.00th=[ 4752], 95.00th=[ 9372], | 99.00th=[12911], 99.50th=[13304], 99.90th=[52167], 99.95th=[54789], | 99.99th=[62129] bw ( KiB/s): min=56832, max=148480, per=100.00%, avg=76017.57, stdev=12958.94, samples=60 iops : min= 222, max= 580, avg=296.93, stdev=50.62, samples=60 lat (usec) : 100=53.85%, 250=25.85%, 500=0.07%, 750=0.01%, 1000=0.51% lat (msec) : 2=6.58%, 4=10.01%, 10=2.21%, 20=0.89%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.73%, sys=2.61%, ctx=44332, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=35417,8909,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=8854MiB (9284MB), run=30006-30006msec WRITE: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=2227MiB (2335MB), run=30006-30006msec Disk stats (read/write): vdi: ios=35260/8859, merge=0/0, ticks=437/21115, in_queue=28435, util=95.09% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942443: Fri Jan 7 14:10:57 2022 read: IOPS=1268, BW=317MiB/s (333MB/s)(9513MiB/30001msec) slat (usec): min=12, max=758, avg=20.42, stdev= 7.01 clat (nsec): min=1063, max=54266k, avg=95282.24, stdev=459720.12 lat (usec): min=67, max=54284, avg=115.88, stdev=459.84 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 110], | 99.00th=[ 131], 99.50th=[ 143], 99.90th=[ 7570], 99.95th=[11469], | 99.99th=[14091] bw ( KiB/s): min=223744, max=515584, per=100.00%, avg=325834.66, stdev=43426.09, samples=59 iops : min= 874, max= 2014, avg=1272.78, stdev=169.64, samples=59 write: IOPS=320, BW=80.1MiB/s (84.0MB/s)(2405MiB/30001msec) slat (usec): min=24, max=409, avg=44.66, stdev=11.06 clat (usec): min=843, max=54482, avg=2607.58, stdev=2468.66 lat (usec): min=884, max=54523, avg=2652.46, stdev=2468.58 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1565], | 30.00th=[ 1893], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2474], 80.00th=[ 2737], 90.00th=[ 4080], 95.00th=[ 5211], | 99.00th=[11469], 99.50th=[12780], 99.90th=[17695], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=66048, max=138240, per=100.00%, avg=82186.24, stdev=9713.24, samples=59 iops : min= 258, max= 540, avg=321.02, stdev=37.93, samples=59 lat (usec) : 2=0.01%, 20=0.01%, 50=0.08%, 100=71.21%, 250=8.37% lat (usec) : 500=0.03%, 750=0.01%, 1000=0.59% lat (msec) : 2=7.20%, 4=10.34%, 10=1.48%, 20=0.66%, 100=0.02% cpu : usr=1.55%, sys=4.74%, ctx=47683, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=38053,9618,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=317MiB/s (333MB/s), 317MiB/s-317MiB/s (333MB/s-333MB/s), io=9513MiB (9975MB), run=30001-30001msec WRITE: bw=80.1MiB/s (84.0MB/s), 80.1MiB/s-80.1MiB/s (84.0MB/s-84.0MB/s), io=2405MiB (2521MB), run=30001-30001msec Disk stats (read/write): vdi: ios=37842/9562, merge=0/0, ticks=506/20486, in_queue=28213, util=94.38% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943467: Fri Jan 7 14:11:35 2022 read: IOPS=1331, BW=333MiB/s (349MB/s)(9988MiB/30009msec) clat (usec): min=55, max=58595, avg=101.22, stdev=562.81 lat (usec): min=55, max=58595, avg=101.37, stdev=562.81 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 70], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 84], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 102], 95.00th=[ 110], | 99.00th=[ 131], 99.50th=[ 149], 99.90th=[ 8586], 99.95th=[12518], | 99.99th=[22152] bw ( KiB/s): min=264192, max=555520, per=100.00%, avg=340906.67, stdev=40874.12, samples=60 iops : min= 1032, max= 2170, avg=1331.67, stdev=159.66, samples=60 write: IOPS=336, BW=84.1MiB/s (88.2MB/s)(2524MiB/30009msec) clat (usec): min=794, max=82240, avg=2539.00, stdev=2423.70 lat (usec): min=818, max=82273, avg=2563.56, stdev=2423.85 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1074], 10.00th=[ 1221], 20.00th=[ 1795], | 30.00th=[ 1958], 40.00th=[ 2073], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2474], 80.00th=[ 2704], 90.00th=[ 3195], 95.00th=[ 4555], | 99.00th=[10683], 99.50th=[12518], 99.90th=[40109], 99.95th=[53216], | 99.99th=[61604] bw ( KiB/s): min=57856, max=148480, per=100.00%, avg=86126.93, stdev=10330.33, samples=60 iops : min= 226, max= 580, avg=336.43, stdev=40.35, samples=60 lat (usec) : 100=70.60%, 250=9.07%, 500=0.03%, 750=0.01%, 1000=0.61% lat (msec) : 2=6.22%, 4=11.89%, 10=1.04%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.64%, sys=2.74%, ctx=50053, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39950,10094,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=333MiB/s (349MB/s), 333MiB/s-333MiB/s (349MB/s-349MB/s), io=9988MiB (10.5GB), run=30009-30009msec WRITE: bw=84.1MiB/s (88.2MB/s), 84.1MiB/s-84.1MiB/s (88.2MB/s-88.2MB/s), io=2524MiB (2646MB), run=30009-30009msec Disk stats (read/write): vdi: ios=39719/10032, merge=0/0, ticks=673/20535, in_queue=28420, util=95.04% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943994: Fri Jan 7 14:12:11 2022 read: IOPS=1388, BW=347MiB/s (364MB/s)(10.2GiB/30031msec) slat (usec): min=8, max=312, avg=17.30, stdev= 5.90 clat (nsec): min=1171, max=2491.6M, avg=573177.89, stdev=23690174.95 lat (usec): min=51, max=2491.6k, avg=590.63, stdev=23690.22 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 52], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 58], | 50.00th=[ 62], 60.00th=[ 68], 70.00th=[ 77], | 80.00th=[ 97], 90.00th=[ 117], 95.00th=[ 133], | 99.00th=[ 196], 99.50th=[ 310], 99.90th=[ 37487], | 99.95th=[ 283116], 99.99th=[1300235] bw ( KiB/s): min=240128, max=700928, per=100.00%, avg=355761.33, stdev=58993.59, samples=60 iops : min= 938, max= 2738, avg=1389.67, stdev=230.45, samples=60 write: IOPS=350, BW=87.7MiB/s (91.9MB/s)(2633MiB/30031msec) slat (usec): min=22, max=233, avg=50.41, stdev=20.27 clat (msec): min=2, max=117, avg=26.11, stdev=10.49 lat (msec): min=2, max=117, avg=26.16, stdev=10.48 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 10], 10.00th=[ 15], 20.00th=[ 20], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 28], | 70.00th=[ 30], 80.00th=[ 33], 90.00th=[ 37], 95.00th=[ 42], | 99.00th=[ 67], 99.50th=[ 79], 99.90th=[ 101], 99.95th=[ 109], | 99.99th=[ 115] bw ( KiB/s): min=61440, max=175104, per=100.00%, avg=89819.23, stdev=12700.42, samples=60 iops : min= 240, max= 684, avg=350.83, stdev=49.61, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.62% lat (usec) : 100=60.38%, 250=14.30%, 500=0.27%, 750=0.10%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.02%, 10=1.02%, 20=3.76%, 50=15.13% lat (msec) : 100=0.29%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=2.13%, sys=4.42%, ctx=49493, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41690,10533,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=347MiB/s (364MB/s), 347MiB/s-347MiB/s (364MB/s-364MB/s), io=10.2GiB (10.9GB), run=30031-30031msec WRITE: bw=87.7MiB/s (91.9MB/s), 87.7MiB/s-87.7MiB/s (91.9MB/s-91.9MB/s), io=2633MiB (2761MB), run=30031-30031msec Disk stats (read/write): vdi: ios=41468/10468, merge=0/0, ticks=19133/268030, in_queue=296393, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944971: Fri Jan 7 14:12:48 2022 read: IOPS=1336, BW=334MiB/s (350MB/s)(9.79GiB/30001msec) clat (usec): min=54, max=37641, avg=99.30, stdev=540.45 lat (usec): min=54, max=37641, avg=99.44, stdev=540.45 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 68], 20.00th=[ 71], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 91], 90.00th=[ 98], 95.00th=[ 106], | 99.00th=[ 131], 99.50th=[ 151], 99.90th=[ 8979], 99.95th=[13960], | 99.99th=[21627] bw ( KiB/s): min=198144, max=553472, per=100.00%, avg=342263.13, stdev=46385.66, samples=60 iops : min= 774, max= 2162, avg=1336.95, stdev=181.19, samples=60 write: IOPS=337, BW=84.4MiB/s (88.5MB/s)(2533MiB/30001msec) clat (usec): min=806, max=104663, avg=2533.19, stdev=2551.95 lat (usec): min=830, max=104686, avg=2559.49, stdev=2551.83 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1057], 10.00th=[ 1221], 20.00th=[ 1827], | 30.00th=[ 1958], 40.00th=[ 2089], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2474], 80.00th=[ 2704], 90.00th=[ 3097], 95.00th=[ 4359], | 99.00th=[10683], 99.50th=[12387], 99.90th=[42730], 99.95th=[54789], | 99.99th=[70779] bw ( KiB/s): min=54784, max=146944, per=99.99%, avg=86442.65, stdev=10526.24, samples=60 iops : min= 214, max= 574, avg=337.65, stdev=41.11, samples=60 lat (usec) : 100=72.99%, 250=6.67%, 500=0.04%, 750=0.01%, 1000=0.62% lat (msec) : 2=5.96%, 4=12.29%, 10=0.87%, 20=0.50%, 50=0.03% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.80%, sys=3.03%, ctx=50246, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40109,10131,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=9.79GiB (10.5GB), run=30001-30001msec WRITE: bw=84.4MiB/s (88.5MB/s), 84.4MiB/s-84.4MiB/s (88.5MB/s-88.5MB/s), io=2533MiB (2656MB), run=30001-30001msec Disk stats (read/write): vdi: ios=40109/10130, merge=0/0, ticks=704/20712, in_queue=28451, util=94.52% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945599: Fri Jan 7 14:13:26 2022 read: IOPS=1371, BW=343MiB/s (359MB/s)(10.1GiB/30181msec) slat (usec): min=4, max=156, avg=13.99, stdev= 5.80 clat (nsec): min=1039, max=2535.9M, avg=1656983.58, stdev=52769949.16 lat (usec): min=48, max=2535.0k, avg=1671.12, stdev=52769.93 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], | 20.00th=[ 55], 30.00th=[ 57], 40.00th=[ 60], | 50.00th=[ 64], 60.00th=[ 72], 70.00th=[ 93], | 80.00th=[ 115], 90.00th=[ 135], 95.00th=[ 174], | 99.00th=[ 1172], 99.50th=[ 2606], 99.90th=[ 408945], | 99.95th=[1283458], 99.99th=[2533360] bw ( KiB/s): min=257024, max=792625, per=100.00%, avg=353210.97, stdev=66523.10, samples=60 iops : min= 1004, max= 3096, avg=1379.72, stdev=259.84, samples=60 write: IOPS=346, BW=86.6MiB/s (90.9MB/s)(2615MiB/30181msec) slat (usec): min=18, max=255, avg=54.46, stdev=24.11 clat (msec): min=14, max=3510, avg=137.57, stdev=78.56 lat (msec): min=14, max=3510, avg=137.62, stdev=78.56 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 95], 10.00th=[ 107], 20.00th=[ 115], | 30.00th=[ 122], 40.00th=[ 127], 50.00th=[ 132], 60.00th=[ 138], | 70.00th=[ 142], 80.00th=[ 150], 90.00th=[ 165], 95.00th=[ 190], | 99.00th=[ 284], 99.50th=[ 502], 99.90th=[ 1183], 99.95th=[ 1469], | 99.99th=[ 2903] bw ( KiB/s): min=70144, max=181098, per=100.00%, avg=88861.05, stdev=13682.80, samples=60 iops : min= 274, max= 707, avg=347.08, stdev=53.40, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=4.09%, 100=53.17% lat (usec) : 250=20.07%, 500=0.82%, 750=0.43%, 1000=0.33% lat (msec) : 2=0.37%, 4=0.37%, 10=0.05%, 20=0.04%, 50=0.44% lat (msec) : 100=0.82%, 250=18.63%, 500=0.17%, 750=0.06%, 1000=0.03% lat (msec) : 2000=0.05%, >=2000=0.03% cpu : usr=2.45%, sys=3.73%, ctx=46830, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=41388,10460,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=343MiB/s (359MB/s), 343MiB/s-343MiB/s (359MB/s-359MB/s), io=10.1GiB (10.8GB), run=30181-30181msec WRITE: bw=86.6MiB/s (90.9MB/s), 86.6MiB/s-86.6MiB/s (90.9MB/s-90.9MB/s), io=2615MiB (2742MB), run=30181-30181msec Disk stats (read/write): vdi: ios=41386/10433, merge=0/0, ticks=63402/1424961, in_queue=1500774, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946468: Fri Jan 7 14:14:02 2022 read: IOPS=1336, BW=334MiB/s (350MB/s)(9.79GiB/30001msec) clat (usec): min=50, max=25897, avg=93.51, stdev=411.92 lat (usec): min=50, max=25897, avg=93.66, stdev=411.92 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 124], 99.50th=[ 141], 99.90th=[ 7963], 99.95th=[11469], | 99.99th=[13566] bw ( KiB/s): min=244224, max=592384, per=100.00%, avg=342636.08, stdev=47357.50, samples=59 iops : min= 954, max= 2314, avg=1338.41, stdev=185.00, samples=59 write: IOPS=337, BW=84.4MiB/s (88.5MB/s)(2532MiB/30001msec) clat (usec): min=808, max=61038, avg=2558.79, stdev=2300.44 lat (usec): min=833, max=61063, avg=2583.10, stdev=2300.28 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1074], 10.00th=[ 1205], 20.00th=[ 1827], | 30.00th=[ 1975], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2507], 80.00th=[ 2704], 90.00th=[ 3261], 95.00th=[ 4621], | 99.00th=[10814], 99.50th=[12256], 99.90th=[18482], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min=71168, max=155648, per=100.00%, avg=86464.03, stdev=10583.41, samples=59 iops : min= 278, max= 608, avg=337.73, stdev=41.35, samples=59 lat (usec) : 100=74.53%, 250=5.17%, 500=0.02%, 750=0.01%, 1000=0.56% lat (msec) : 2=5.91%, 4=12.20%, 10=1.05%, 20=0.53%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.67%, sys=2.90%, ctx=50244, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40104,10129,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=9.79GiB (10.5GB), run=30001-30001msec WRITE: bw=84.4MiB/s (88.5MB/s), 84.4MiB/s-84.4MiB/s (88.5MB/s-88.5MB/s), io=2532MiB (2655MB), run=30001-30001msec Disk stats (read/write): vdi: ios=39818/10060, merge=0/0, ticks=554/20872, in_queue=28367, util=94.86% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947441: Fri Jan 7 14:14:40 2022 read: IOPS=1367, BW=342MiB/s (359MB/s)(10.1GiB/30342msec) slat (usec): min=4, max=276, avg=12.84, stdev= 6.41 clat (nsec): min=1145, max=2541.7M, avg=1764293.86, stdev=53961148.09 lat (usec): min=47, max=2541.7k, avg=1777.30, stdev=53961.12 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 60], | 50.00th=[ 64], 60.00th=[ 73], 70.00th=[ 104], | 80.00th=[ 119], 90.00th=[ 155], 95.00th=[ 594], | 99.00th=[ 5866], 99.50th=[ 7767], 99.90th=[ 295699], | 99.95th=[1098908], 99.99th=[2533360] bw ( KiB/s): min=243200, max=849920, per=100.00%, avg=354136.13, stdev=76196.71, samples=60 iops : min= 950, max= 3320, avg=1383.33, stdev=297.65, samples=60 write: IOPS=345, BW=86.4MiB/s (90.6MB/s)(2621MiB/30342msec) slat (usec): min=19, max=715, avg=52.56, stdev=25.46 clat (msec): min=17, max=3354, avg=282.29, stdev=118.42 lat (msec): min=17, max=3354, avg=282.35, stdev=118.42 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 165], 10.00th=[ 215], 20.00th=[ 247], | 30.00th=[ 259], 40.00th=[ 268], 50.00th=[ 275], 60.00th=[ 284], | 70.00th=[ 292], 80.00th=[ 305], 90.00th=[ 334], 95.00th=[ 380], | 99.00th=[ 768], 99.50th=[ 1099], 99.90th=[ 1318], 99.95th=[ 1502], | 99.99th=[ 2802] bw ( KiB/s): min=74752, max=169984, per=100.00%, avg=88624.20, stdev=12430.06, samples=60 iops : min= 292, max= 664, avg=346.17, stdev=48.56, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=4.93%, 100=50.12% lat (usec) : 250=19.11%, 500=1.32%, 750=0.70%, 1000=0.50% lat (msec) : 2=1.19%, 4=0.74%, 10=0.96%, 20=0.12%, 50=0.12% lat (msec) : 100=0.61%, 250=3.70%, 500=15.31%, 750=0.27%, 1000=0.09% lat (msec) : 2000=0.14%, >=2000=0.04% cpu : usr=2.37%, sys=3.44%, ctx=44823, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=41500,10483,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=342MiB/s (359MB/s), 342MiB/s-342MiB/s (359MB/s-359MB/s), io=10.1GiB (10.9GB), run=30342-30342msec WRITE: bw=86.4MiB/s (90.6MB/s), 86.4MiB/s-86.4MiB/s (90.6MB/s-90.6MB/s), io=2621MiB (2748MB), run=30342-30342msec Disk stats (read/write): vdi: ios=41498/10473, merge=0/0, ticks=66375/2932526, in_queue=3012631, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948015: Fri Jan 7 14:15:17 2022 read: IOPS=1335, BW=334MiB/s (350MB/s)(9.79GiB/30001msec) clat (usec): min=52, max=31214, avg=93.77, stdev=402.11 lat (usec): min=52, max=31214, avg=93.92, stdev=402.11 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 133], 99.50th=[ 151], 99.90th=[ 8029], 99.95th=[10683], | 99.99th=[13829] bw ( KiB/s): min=253440, max=527872, per=100.00%, avg=342593.22, stdev=42081.17, samples=59 iops : min= 990, max= 2062, avg=1338.24, stdev=164.39, samples=59 write: IOPS=337, BW=84.4MiB/s (88.5MB/s)(2531MiB/30001msec) clat (usec): min=795, max=61441, avg=2558.94, stdev=2386.81 lat (usec): min=821, max=61470, avg=2583.19, stdev=2386.58 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1074], 10.00th=[ 1205], 20.00th=[ 1827], | 30.00th=[ 1958], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2507], 80.00th=[ 2704], 90.00th=[ 3195], 95.00th=[ 4621], | 99.00th=[10814], 99.50th=[12518], 99.90th=[52691], 99.95th=[53740], | 99.99th=[54789] bw ( KiB/s): min=74240, max=141312, per=100.00%, avg=86455.59, stdev=9021.06, samples=59 iops : min= 290, max= 552, avg=337.69, stdev=35.24, samples=59 lat (usec) : 100=73.68%, 250=5.98%, 500=0.04%, 750=0.01%, 1000=0.59% lat (msec) : 2=6.03%, 4=12.05%, 10=1.06%, 20=0.54%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.73%, sys=2.95%, ctx=50215, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40080,10124,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=9.79GiB (10.5GB), run=30001-30001msec WRITE: bw=84.4MiB/s (88.5MB/s), 84.4MiB/s-84.4MiB/s (88.5MB/s-88.5MB/s), io=2531MiB (2654MB), run=30001-30001msec Disk stats (read/write): vdi: ios=39781/10048, merge=0/0, ticks=540/20785, in_queue=28403, util=95.02% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949368: Fri Jan 7 14:15:54 2022 read: IOPS=1378, BW=345MiB/s (361MB/s)(10.2GiB/30371msec) slat (usec): min=4, max=338477, avg=560.58, stdev=4856.31 clat (msec): min=3, max=2629, avg=47.58, stdev=66.59 lat (msec): min=3, max=2629, avg=48.14, stdev=66.72 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 13], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 39], 50.00th=[ 43], 60.00th=[ 46], | 70.00th=[ 51], 80.00th=[ 57], 90.00th=[ 70], 95.00th=[ 82], | 99.00th=[ 288], 99.50th=[ 347], 99.90th=[ 542], 99.95th=[ 1418], | 99.99th=[ 2601] bw ( KiB/s): min=252416, max=901120, per=100.00%, avg=356559.62, stdev=82126.38, samples=60 iops : min= 986, max= 3520, avg=1392.80, stdev=320.81, samples=60 write: IOPS=348, BW=87.1MiB/s (91.3MB/s)(2646MiB/30371msec) slat (usec): min=32, max=70633, avg=606.63, stdev=3257.65 clat (msec): min=35, max=3528, avg=382.80, stdev=212.62 lat (msec): min=45, max=3528, avg=383.40, stdev=212.62 clat percentiles (msec): | 1.00th=[ 69], 5.00th=[ 218], 10.00th=[ 271], 20.00th=[ 313], | 30.00th=[ 338], 40.00th=[ 355], 50.00th=[ 372], 60.00th=[ 380], | 70.00th=[ 397], 80.00th=[ 414], 90.00th=[ 451], 95.00th=[ 550], | 99.00th=[ 969], 99.50th=[ 1217], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 3540] bw ( KiB/s): min=66560, max=169472, per=100.00%, avg=89201.90, stdev=13080.04, samples=60 iops : min= 260, max= 662, avg=348.43, stdev=51.09, samples=60 lat (msec) : 4=0.01%, 10=5.01%, 20=6.09%, 50=43.47%, 100=23.74% lat (msec) : 250=2.14%, 500=18.17%, 750=0.94%, 1000=0.19%, 2000=0.14% lat (msec) : >=2000=0.10% cpu : usr=3.37%, sys=1.74%, ctx=5564, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=41853,10583,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=345MiB/s (361MB/s), 345MiB/s-345MiB/s (361MB/s-361MB/s), io=10.2GiB (10.0GB), run=30371-30371msec WRITE: bw=87.1MiB/s (91.3MB/s), 87.1MiB/s-87.1MiB/s (91.3MB/s-91.3MB/s), io=2646MiB (2774MB), run=30371-30371msec Disk stats (read/write): vdi: ios=41849/10534, merge=0/0, ticks=89422/3560695, in_queue=3699986, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950050: Fri Jan 7 14:16:31 2022 read: IOPS=917, BW=459MiB/s (481MB/s)(13.4GiB/30004msec) clat (usec): min=97, max=56484, avg=158.51, stdev=594.71 lat (usec): min=97, max=56484, avg=158.66, stdev=594.71 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 112], 10.00th=[ 114], 20.00th=[ 117], | 30.00th=[ 120], 40.00th=[ 124], 50.00th=[ 129], 60.00th=[ 137], | 70.00th=[ 143], 80.00th=[ 153], 90.00th=[ 167], 95.00th=[ 184], | 99.00th=[ 235], 99.50th=[ 269], 99.90th=[10552], 99.95th=[13173], | 99.99th=[17433] bw ( KiB/s): min=352256, max=616448, per=100.00%, avg=469572.27, stdev=53376.80, samples=60 iops : min= 688, max= 1204, avg=917.13, stdev=104.25, samples=60 write: IOPS=230, BW=115MiB/s (121MB/s)(3464MiB/30004msec) clat (usec): min=1387, max=63290, avg=3642.14, stdev=2609.31 lat (usec): min=1418, max=63316, avg=3691.88, stdev=2608.91 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1713], 10.00th=[ 1893], 20.00th=[ 2442], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3163], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3851], 90.00th=[ 5997], 95.00th=[ 7504], | 99.00th=[12256], 99.50th=[14615], 99.90th=[23462], 99.95th=[54264], | 99.99th=[63177] bw ( KiB/s): min=89088, max=178176, per=100.00%, avg=118220.80, stdev=12435.26, samples=60 iops : min= 174, max= 348, avg=230.90, stdev=24.29, samples=60 lat (usec) : 100=0.02%, 250=79.28%, 500=0.42%, 750=0.01% lat (msec) : 2=2.22%, 4=14.44%, 10=2.87%, 20=0.71%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.75%, sys=2.63%, ctx=34449, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27514,6928,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=459MiB/s (481MB/s), 459MiB/s-459MiB/s (481MB/s-481MB/s), io=13.4GiB (14.4GB), run=30004-30004msec WRITE: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=3464MiB (3632MB), run=30004-30004msec Disk stats (read/write): vdi: ios=27313/6883, merge=0/0, ticks=601/21434, in_queue=28432, util=95.10% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950869: Fri Jan 7 14:17:08 2022 read: IOPS=947, BW=474MiB/s (497MB/s)(13.9GiB/30006msec) slat (usec): min=17, max=128, avg=27.95, stdev= 7.21 clat (nsec): min=1276, max=21050k, avg=135391.40, stdev=542546.69 lat (usec): min=98, max=21074, avg=163.52, stdev=542.69 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 90], 10.00th=[ 92], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 113], | 70.00th=[ 119], 80.00th=[ 127], 90.00th=[ 139], 95.00th=[ 155], | 99.00th=[ 200], 99.50th=[ 229], 99.90th=[12387], 99.95th=[13304], | 99.99th=[16712] bw ( KiB/s): min=375808, max=630784, per=100.00%, avg=485100.15, stdev=55521.29, samples=60 iops : min= 734, max= 1232, avg=947.43, stdev=108.45, samples=60 write: IOPS=239, BW=120MiB/s (125MB/s)(3588MiB/30006msec) slat (usec): min=38, max=402, avg=82.82, stdev=31.92 clat (usec): min=1369, max=62413, avg=3442.81, stdev=2532.63 lat (usec): min=1415, max=62487, avg=3525.84, stdev=2532.10 clat percentiles (usec): | 1.00th=[ 1434], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3785], 90.00th=[ 4686], 95.00th=[ 6587], | 99.00th=[11863], 99.50th=[14091], 99.90th=[52691], 99.95th=[55837], | 99.99th=[62653] bw ( KiB/s): min=94208, max=183296, per=100.00%, avg=122444.55, stdev=10819.82, samples=60 iops : min= 184, max= 358, avg=239.13, stdev=21.13, samples=60 lat (usec) : 2=0.01%, 50=0.01%, 100=29.39%, 250=50.18%, 500=0.10% lat (usec) : 750=0.01% lat (msec) : 2=3.31%, 4=13.80%, 10=2.62%, 20=0.56%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=2.04%, sys=4.28%, ctx=35612, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28422,7175,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=13.9GiB (14.9GB), run=30006-30006msec WRITE: bw=120MiB/s (125MB/s), 120MiB/s-120MiB/s (125MB/s-125MB/s), io=3588MiB (3762MB), run=30006-30006msec Disk stats (read/write): vdi: ios=28326/7146, merge=0/0, ticks=637/20813, in_queue=28260, util=94.48% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951817: Fri Jan 7 14:17:44 2022 read: IOPS=962, BW=481MiB/s (505MB/s)(14.1GiB/30003msec) clat (usec): min=75, max=21310, avg=131.22, stdev=558.27 lat (usec): min=75, max=21310, avg=131.37, stdev=558.27 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 88], 10.00th=[ 90], 20.00th=[ 93], | 30.00th=[ 95], 40.00th=[ 98], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 133], 95.00th=[ 143], | 99.00th=[ 178], 99.50th=[ 196], 99.90th=[11994], 99.95th=[14353], | 99.99th=[17695] bw ( KiB/s): min=393216, max=627712, per=100.00%, avg=492766.55, stdev=56490.19, samples=60 iops : min= 768, max= 1226, avg=962.43, stdev=110.33, samples=60 write: IOPS=243, BW=122MiB/s (128MB/s)(3649MiB/30003msec) clat (usec): min=1460, max=65746, avg=3532.55, stdev=2415.37 lat (usec): min=1505, max=65804, avg=3583.51, stdev=2414.63 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1647], 10.00th=[ 1762], 20.00th=[ 2540], | 30.00th=[ 2868], 40.00th=[ 3064], 50.00th=[ 3228], 60.00th=[ 3392], | 70.00th=[ 3589], 80.00th=[ 3851], 90.00th=[ 4490], 95.00th=[ 6718], | 99.00th=[11731], 99.50th=[12387], 99.90th=[45351], 99.95th=[54789], | 99.99th=[65799] bw ( KiB/s): min=105472, max=175104, per=100.00%, avg=124522.02, stdev=10147.45, samples=60 iops : min= 206, max= 342, avg=243.20, stdev=19.83, samples=60 lat (usec) : 100=37.36%, 250=42.25%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.46%, 4=14.49%, 10=2.83%, 20=0.53%, 50=0.02% lat (msec) : 100=0.02% cpu : usr=1.86%, sys=2.36%, ctx=36177, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28872,7297,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=14.1GiB (15.1GB), run=30003-30003msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3649MiB (3826MB), run=30003-30003msec Disk stats (read/write): vdi: ios=28723/7253, merge=0/0, ticks=675/21784, in_queue=28442, util=95.12% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952399: Fri Jan 7 14:18:22 2022 read: IOPS=976, BW=488MiB/s (512MB/s)(14.3GiB/30034msec) slat (usec): min=6, max=126, avg=17.21, stdev= 6.73 clat (nsec): min=1695, max=90289k, avg=177151.39, stdev=1609681.55 lat (usec): min=70, max=90302, avg=194.52, stdev=1609.69 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 74], | 30.00th=[ 76], 40.00th=[ 79], 50.00th=[ 82], 60.00th=[ 88], | 70.00th=[ 101], 80.00th=[ 172], 90.00th=[ 194], 95.00th=[ 212], | 99.00th=[ 314], 99.50th=[ 486], 99.90th=[31851], 99.95th=[39060], | 99.99th=[55837] bw ( KiB/s): min=343040, max=686428, per=100.00%, avg=500687.83, stdev=62793.51, samples=60 iops : min= 670, max= 1340, avg=977.88, stdev=122.62, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3704MiB/30034msec) slat (usec): min=35, max=456, avg=91.21, stdev=44.36 clat (msec): min=2, max=124, avg=39.67, stdev=15.48 lat (msec): min=2, max=124, avg=39.76, stdev=15.48 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 14], 10.00th=[ 21], 20.00th=[ 29], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 47], 80.00th=[ 52], 90.00th=[ 58], 95.00th=[ 65], | 99.00th=[ 88], 99.50th=[ 97], 99.90th=[ 113], 99.95th=[ 116], | 99.99th=[ 125] bw ( KiB/s): min=99328, max=193923, per=99.99%, avg=126256.57, stdev=12145.11, samples=60 iops : min= 194, max= 378, avg=246.57, stdev=23.66, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.05%, 100=55.61% lat (usec) : 250=22.59%, 500=1.20%, 750=0.10%, 1000=0.03% lat (msec) : 2=0.09%, 4=0.01%, 10=0.50%, 20=1.47%, 50=13.67% lat (msec) : 100=4.59%, 250=0.07% cpu : usr=2.63%, sys=3.00%, ctx=35352, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29334,7407,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30034-30034msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3704MiB (3883MB), run=30034-30034msec Disk stats (read/write): vdi: ios=29153/7355, merge=0/0, ticks=1968/287935, in_queue=296739, util=99.72% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953228: Fri Jan 7 14:18:59 2022 read: IOPS=966, BW=483MiB/s (507MB/s)(14.2GiB/30002msec) clat (usec): min=78, max=18866, avg=133.54, stdev=552.21 lat (usec): min=78, max=18866, avg=133.70, stdev=552.21 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 96], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 133], 95.00th=[ 143], | 99.00th=[ 174], 99.50th=[ 192], 99.90th=[12649], 99.95th=[13566], | 99.99th=[17171] bw ( KiB/s): min=398336, max=631808, per=100.00%, avg=494914.88, stdev=53307.05, samples=60 iops : min= 778, max= 1234, avg=966.60, stdev=104.12, samples=60 write: IOPS=244, BW=122MiB/s (128MB/s)(3665MiB/30002msec) clat (usec): min=1367, max=55401, avg=3506.35, stdev=2324.84 lat (usec): min=1425, max=55457, avg=3556.25, stdev=2324.25 clat percentiles (usec): | 1.00th=[ 1450], 5.00th=[ 1663], 10.00th=[ 1745], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3064], 50.00th=[ 3228], 60.00th=[ 3392], | 70.00th=[ 3589], 80.00th=[ 3851], 90.00th=[ 4490], 95.00th=[ 6587], | 99.00th=[11600], 99.50th=[12387], 99.90th=[21627], 99.95th=[54264], | 99.99th=[55313] bw ( KiB/s): min=108544, max=171008, per=100.00%, avg=125073.68, stdev=9232.25, samples=60 iops : min= 212, max= 334, avg=244.27, stdev=18.01, samples=60 lat (usec) : 100=30.46%, 250=49.12%, 500=0.06%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.65%, 4=14.19%, 10=2.98%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=2.68%, ctx=36334, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28997,7329,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=483MiB/s (507MB/s), 483MiB/s-483MiB/s (507MB/s-507MB/s), io=14.2GiB (15.2GB), run=30002-30002msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3665MiB (3843MB), run=30002-30002msec Disk stats (read/write): vdi: ios=28843/7283, merge=0/0, ticks=675/21681, in_queue=28367, util=94.84% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954257: Fri Jan 7 14:19:36 2022 read: IOPS=973, BW=487MiB/s (510MB/s)(14.3GiB/30193msec) slat (usec): min=5, max=178, avg=15.81, stdev= 6.81 clat (nsec): min=1728, max=271240k, avg=395046.97, stdev=4946475.38 lat (usec): min=70, max=271251, avg=411.01, stdev=4946.40 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 88], | 70.00th=[ 105], 80.00th=[ 186], 90.00th=[ 204], 95.00th=[ 293], | 99.00th=[ 4359], 99.50th=[ 7242], 99.90th=[ 91751], 99.95th=[112722], | 99.99th=[198181] bw ( KiB/s): min=373760, max=908288, per=100.00%, avg=501587.00, stdev=72695.59, samples=60 iops : min= 730, max= 1774, avg=979.65, stdev=141.99, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3711MiB/30193msec) slat (usec): min=35, max=370, avg=94.68, stdev=46.92 clat (msec): min=11, max=2653, avg=201.63, stdev=104.77 lat (msec): min=11, max=2653, avg=201.73, stdev=104.77 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 73], 10.00th=[ 86], 20.00th=[ 109], | 30.00th=[ 138], 40.00th=[ 171], 50.00th=[ 211], 60.00th=[ 232], | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 305], 95.00th=[ 330], | 99.00th=[ 426], 99.50th=[ 518], 99.90th=[ 1200], 99.95th=[ 1620], | 99.99th=[ 2668] bw ( KiB/s): min=104448, max=204800, per=99.97%, avg=125819.60, stdev=12374.54, samples=60 iops : min= 204, max= 400, avg=245.73, stdev=24.17, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 50=0.04%, 100=54.63%, 250=20.58% lat (usec) : 500=1.50%, 750=0.62%, 1000=0.35% lat (msec) : 2=0.70%, 4=0.59%, 10=0.65%, 20=0.03%, 50=0.20% lat (msec) : 100=3.19%, 250=10.70%, 500=6.10%, 750=0.06%, 1000=0.02% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.68%, sys=2.81%, ctx=34325, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=29389,7422,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=487MiB/s (510MB/s), 487MiB/s-487MiB/s (510MB/s-510MB/s), io=14.3GiB (15.4GB), run=30193-30193msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3711MiB (3891MB), run=30193-30193msec Disk stats (read/write): vdi: ios=29389/7378, merge=0/0, ticks=7403/1476314, in_queue=1498882, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954775: Fri Jan 7 14:20:13 2022 read: IOPS=957, BW=479MiB/s (502MB/s)(14.0GiB/30005msec) clat (usec): min=82, max=19809, avg=138.62, stdev=528.63 lat (usec): min=82, max=19809, avg=138.77, stdev=528.63 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 96], 10.00th=[ 98], 20.00th=[ 101], | 30.00th=[ 104], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 116], | 70.00th=[ 123], 80.00th=[ 130], 90.00th=[ 141], 95.00th=[ 151], | 99.00th=[ 186], 99.50th=[ 208], 99.90th=[11076], 99.95th=[14091], | 99.99th=[19268] bw ( KiB/s): min=333466, max=612352, per=100.00%, avg=490572.17, stdev=53987.58, samples=60 iops : min= 651, max= 1196, avg=958.03, stdev=105.47, samples=60 write: IOPS=241, BW=121MiB/s (127MB/s)(3627MiB/30005msec) clat (usec): min=1425, max=55653, avg=3526.83, stdev=2482.42 lat (usec): min=1451, max=55818, avg=3578.45, stdev=2481.65 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2802], 40.00th=[ 3032], 50.00th=[ 3195], 60.00th=[ 3392], | 70.00th=[ 3556], 80.00th=[ 3851], 90.00th=[ 4686], 95.00th=[ 6849], | 99.00th=[11731], 99.50th=[12649], 99.90th=[52167], 99.95th=[54264], | 99.99th=[55837] bw ( KiB/s): min=103839, max=176128, per=100.00%, avg=123860.28, stdev=10342.49, samples=60 iops : min= 202, max= 344, avg=241.73, stdev=20.24, samples=60 lat (usec) : 100=13.58%, 250=66.01%, 500=0.07%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.93%, 4=13.98%, 10=2.82%, 20=0.58%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=2.04%, sys=2.65%, ctx=35991, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28723,7254,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=479MiB/s (502MB/s), 479MiB/s-479MiB/s (502MB/s-502MB/s), io=14.0GiB (15.1GB), run=30005-30005msec WRITE: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=3627MiB (3803MB), run=30005-30005msec Disk stats (read/write): vdi: ios=28723/7252, merge=0/0, ticks=639/21667, in_queue=28509, util=94.68% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956007: Fri Jan 7 14:20:50 2022 read: IOPS=977, BW=489MiB/s (512MB/s)(14.5GiB/30447msec) slat (usec): min=5, max=644, avg=14.58, stdev=10.30 clat (nsec): min=1404, max=158510k, avg=771641.76, stdev=4810160.74 lat (usec): min=69, max=158519, avg=786.37, stdev=4810.15 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 83], 60.00th=[ 92], | 70.00th=[ 178], 80.00th=[ 196], 90.00th=[ 482], 95.00th=[ 2900], | 99.00th=[ 12911], 99.50th=[ 14877], 99.90th=[ 90702], 99.95th=[121111], | 99.99th=[141558] bw ( KiB/s): min=370688, max=1057792, per=100.00%, avg=507952.57, stdev=92109.97, samples=60 iops : min= 724, max= 2066, avg=992.05, stdev=179.91, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3756MiB/30447msec) slat (usec): min=31, max=1357, avg=99.16, stdev=58.26 clat (msec): min=15, max=3049, avg=402.05, stdev=149.06 lat (msec): min=15, max=3049, avg=402.15, stdev=149.05 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 213], 10.00th=[ 271], 20.00th=[ 296], | 30.00th=[ 330], 40.00th=[ 359], 50.00th=[ 401], 60.00th=[ 430], | 70.00th=[ 456], 80.00th=[ 485], 90.00th=[ 527], 95.00th=[ 617], | 99.00th=[ 835], 99.50th=[ 944], 99.90th=[ 2089], 99.95th=[ 2366], | 99.99th=[ 3037] bw ( KiB/s): min=102400, max=184320, per=100.00%, avg=126493.05, stdev=11392.55, samples=60 iops : min= 200, max= 360, avg=247.03, stdev=22.26, samples=60 lat (usec) : 2=0.01%, 50=0.03%, 100=50.58%, 250=19.08%, 500=2.25% lat (usec) : 750=0.98%, 1000=0.64% lat (msec) : 2=1.59%, 4=1.11%, 10=2.00%, 20=1.41%, 50=0.02% lat (msec) : 100=0.28%, 250=1.31%, 500=15.59%, 750=2.65%, 1000=0.34% lat (msec) : 2000=0.07%, >=2000=0.02% cpu : usr=2.79%, sys=2.58%, ctx=32135, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=29760,7511,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=489MiB/s (512MB/s), 489MiB/s-489MiB/s (512MB/s-512MB/s), io=14.5GiB (15.6GB), run=30447-30447msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3756MiB (3938MB), run=30447-30447msec Disk stats (read/write): vdi: ios=29760/7427, merge=0/0, ticks=15945/2955804, in_queue=3020245, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956765: Fri Jan 7 14:21:27 2022 read: IOPS=968, BW=484MiB/s (508MB/s)(14.2GiB/30002msec) clat (usec): min=77, max=27625, avg=132.70, stdev=587.42 lat (usec): min=77, max=27626, avg=132.85, stdev=587.42 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 90], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 97], 40.00th=[ 99], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 122], 90.00th=[ 131], 95.00th=[ 141], | 99.00th=[ 178], 99.50th=[ 204], 99.90th=[11338], 99.95th=[14877], | 99.99th=[21890] bw ( KiB/s): min=358092, max=642048, per=100.00%, avg=496174.10, stdev=54445.11, samples=60 iops : min= 699, max= 1254, avg=969.08, stdev=106.36, samples=60 write: IOPS=244, BW=122MiB/s (128MB/s)(3674MiB/30002msec) clat (usec): min=1445, max=62685, avg=3498.58, stdev=2377.86 lat (usec): min=1491, max=62709, avg=3549.46, stdev=2377.06 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1663], 10.00th=[ 1762], 20.00th=[ 2540], | 30.00th=[ 2835], 40.00th=[ 3064], 50.00th=[ 3228], 60.00th=[ 3392], | 70.00th=[ 3556], 80.00th=[ 3818], 90.00th=[ 4424], 95.00th=[ 6521], | 99.00th=[11863], 99.50th=[13698], 99.90th=[16581], 99.95th=[54789], | 99.99th=[62653] bw ( KiB/s): min=97280, max=183296, per=100.00%, avg=125379.33, stdev=10847.74, samples=60 iops : min= 190, max= 358, avg=244.87, stdev=21.20, samples=60 lat (usec) : 100=36.33%, 250=43.26%, 500=0.06%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.54%, 4=14.48%, 10=2.80%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.86%, sys=2.72%, ctx=36422, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29071,7347,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=484MiB/s (508MB/s), 484MiB/s-484MiB/s (508MB/s-508MB/s), io=14.2GiB (15.2GB), run=30002-30002msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3674MiB (3852MB), run=30002-30002msec Disk stats (read/write): vdi: ios=28868/7292, merge=0/0, ticks=693/21681, in_queue=28349, util=94.72% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957725: Fri Jan 7 14:22:04 2022 read: IOPS=979, BW=490MiB/s (514MB/s)(14.6GiB/30449msec) slat (usec): min=6, max=465909, avg=788.95, stdev=6564.74 clat (msec): min=9, max=601, avg=65.41, stdev=54.76 lat (msec): min=10, max=610, avg=66.20, stdev=55.16 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 20], 10.00th=[ 27], 20.00th=[ 38], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 60], 60.00th=[ 66], | 70.00th=[ 73], 80.00th=[ 81], 90.00th=[ 94], 95.00th=[ 110], | 99.00th=[ 464], 99.50th=[ 498], 99.90th=[ 535], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=35840, max=1154048, per=100.00%, avg=508004.63, stdev=195521.29, samples=60 iops : min= 70, max= 2254, avg=992.17, stdev=381.90, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3762MiB/30449msec) slat (usec): min=136, max=468169, avg=846.49, stdev=6367.86 clat (msec): min=22, max=2795, avg=545.72, stdev=193.08 lat (msec): min=40, max=2795, avg=546.56, stdev=192.96 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 292], 10.00th=[ 380], 20.00th=[ 435], | 30.00th=[ 477], 40.00th=[ 514], 50.00th=[ 542], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 617], 90.00th=[ 684], 95.00th=[ 844], | 99.00th=[ 1167], 99.50th=[ 1485], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2802] bw ( KiB/s): min= 7168, max=230400, per=99.69%, avg=126128.33, stdev=38047.11, samples=60 iops : min= 14, max= 450, avg=246.32, stdev=74.33, samples=60 lat (msec) : 10=0.01%, 20=4.33%, 50=23.68%, 100=45.77%, 250=5.86% lat (msec) : 500=7.10%, 750=11.82%, 1000=1.04%, 2000=0.35%, >=2000=0.05% cpu : usr=4.44%, sys=1.68%, ctx=3799, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=29834,7524,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=490MiB/s (514MB/s), 490MiB/s-490MiB/s (514MB/s-514MB/s), io=14.6GiB (15.6GB), run=30449-30449msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3762MiB (3945MB), run=30449-30449msec Disk stats (read/write): vdi: ios=29834/7463, merge=0/0, ticks=69709/3589107, in_queue=3711367, util=99.55% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958710: Fri Jan 7 14:22:41 2022 read: IOPS=591, BW=591MiB/s (620MB/s)(17.3GiB/30004msec) clat (usec): min=155, max=25381, avg=253.49, stdev=886.86 lat (usec): min=155, max=25381, avg=253.65, stdev=886.86 clat percentiles (usec): | 1.00th=[ 163], 5.00th=[ 169], 10.00th=[ 172], 20.00th=[ 176], | 30.00th=[ 180], 40.00th=[ 184], 50.00th=[ 188], 60.00th=[ 198], | 70.00th=[ 210], 80.00th=[ 223], 90.00th=[ 241], 95.00th=[ 265], | 99.00th=[ 355], 99.50th=[ 449], 99.90th=[16909], 99.95th=[17957], | 99.99th=[21365] bw ( KiB/s): min=442368, max=854016, per=100.00%, avg=605748.17, stdev=85524.53, samples=60 iops : min= 432, max= 834, avg=591.53, stdev=83.54, samples=60 write: IOPS=149, BW=150MiB/s (157MB/s)(4499MiB/30004msec) clat (usec): min=2585, max=56772, avg=5560.31, stdev=3165.36 lat (usec): min=2678, max=56883, avg=5659.61, stdev=3164.06 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2704], 10.00th=[ 2769], 20.00th=[ 3458], | 30.00th=[ 4293], 40.00th=[ 4817], 50.00th=[ 5145], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 9110], 95.00th=[11600], | 99.00th=[16188], 99.50th=[18744], 99.90th=[52167], 99.95th=[55837], | 99.99th=[56886] bw ( KiB/s): min=122880, max=210944, per=100.00%, avg=153546.95, stdev=12070.47, samples=60 iops : min= 120, max= 206, avg=149.93, stdev=11.79, samples=60 lat (usec) : 250=73.88%, 500=5.54%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.12%, 10=13.52%, 20=1.85%, 50=0.03% lat (msec) : 100=0.02% cpu : usr=1.85%, sys=2.48%, ctx=22265, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17745,4499,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=591MiB/s (620MB/s), 591MiB/s-591MiB/s (620MB/s-620MB/s), io=17.3GiB (18.6GB), run=30004-30004msec WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=4499MiB (4718MB), run=30004-30004msec Disk stats (read/write): vdi: ios=17621/4464, merge=0/0, ticks=928/22268, in_queue=28436, util=95.24% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959286: Fri Jan 7 14:23:18 2022 read: IOPS=602, BW=603MiB/s (632MB/s)(17.7GiB/30002msec) slat (usec): min=29, max=185, avg=40.17, stdev= 8.62 clat (usec): min=85, max=48187, avg=210.58, stdev=1006.20 lat (usec): min=144, max=48222, avg=250.95, stdev=1006.30 clat percentiles (usec): | 1.00th=[ 121], 5.00th=[ 125], 10.00th=[ 126], 20.00th=[ 130], | 30.00th=[ 133], 40.00th=[ 137], 50.00th=[ 141], 60.00th=[ 147], | 70.00th=[ 161], 80.00th=[ 172], 90.00th=[ 188], 95.00th=[ 208], | 99.00th=[ 318], 99.50th=[ 506], 99.90th=[17171], 99.95th=[19530], | 99.99th=[28967] bw ( KiB/s): min=466944, max=827392, per=100.00%, avg=617130.67, stdev=82909.96, samples=60 iops : min= 456, max= 808, avg=602.67, stdev=80.97, samples=60 write: IOPS=152, BW=152MiB/s (160MB/s)(4565MiB/30002msec) slat (usec): min=70, max=371, avg=131.01, stdev=26.25 clat (usec): min=2534, max=67646, avg=5437.82, stdev=3049.46 lat (usec): min=2627, max=67770, avg=5569.07, stdev=3047.73 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3392], | 30.00th=[ 4293], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5669], 80.00th=[ 5997], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[15533], 99.50th=[18482], 99.90th=[26084], 99.95th=[55837], | 99.99th=[67634] bw ( KiB/s): min=124928, max=210944, per=99.98%, avg=155784.53, stdev=11510.54, samples=60 iops : min= 122, max= 206, avg=152.13, stdev=11.24, samples=60 lat (usec) : 100=0.02%, 250=78.26%, 500=1.15%, 750=0.07%, 1000=0.02% lat (msec) : 2=0.01%, 4=5.43%, 10=13.31%, 20=1.63%, 50=0.09% lat (msec) : 100=0.01% cpu : usr=2.15%, sys=3.39%, ctx=22655, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18080,4565,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=603MiB/s (632MB/s), 603MiB/s-603MiB/s (632MB/s-632MB/s), io=17.7GiB (18.0GB), run=30002-30002msec WRITE: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=4565MiB (4787MB), run=30002-30002msec Disk stats (read/write): vdi: ios=17947/4536, merge=0/0, ticks=1038/22242, in_queue=28484, util=95.18% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960091: Fri Jan 7 14:23:55 2022 read: IOPS=606, BW=606MiB/s (636MB/s)(17.8GiB/30004msec) clat (usec): min=132, max=26428, avg=229.60, stdev=981.48 lat (usec): min=132, max=26428, avg=229.76, stdev=981.48 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 208], 95.00th=[ 227], | 99.00th=[ 310], 99.50th=[ 424], 99.90th=[17957], 99.95th=[19530], | 99.99th=[26084] bw ( KiB/s): min=481280, max=786432, per=100.00%, avg=620750.97, stdev=79511.01, samples=60 iops : min= 470, max= 768, avg=606.18, stdev=77.67, samples=60 write: IOPS=153, BW=153MiB/s (161MB/s)(4600MiB/30004msec) clat (usec): min=2547, max=59357, avg=5507.68, stdev=2927.43 lat (usec): min=2635, max=59461, avg=5605.18, stdev=2926.37 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3687], | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5211], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6063], 90.00th=[ 7767], 95.00th=[11076], | 99.00th=[15533], 99.50th=[16909], 99.90th=[22414], 99.95th=[55313], | 99.99th=[59507] bw ( KiB/s): min=131072, max=206848, per=100.00%, avg=156989.32, stdev=12049.21, samples=60 iops : min= 128, max= 202, avg=153.30, stdev=11.77, samples=60 lat (usec) : 250=77.59%, 500=1.85%, 750=0.04%, 1000=0.01% lat (msec) : 4=4.61%, 10=14.23%, 20=1.58%, 50=0.06%, 100=0.02% cpu : usr=1.89%, sys=2.53%, ctx=22796, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18185,4600,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=606MiB/s (636MB/s), 606MiB/s-606MiB/s (636MB/s-636MB/s), io=17.8GiB (19.1GB), run=30004-30004msec WRITE: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=4600MiB (4823MB), run=30004-30004msec Disk stats (read/write): vdi: ios=18047/4562, merge=0/0, ticks=1039/22614, in_queue=28440, util=95.17% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961058: Fri Jan 7 14:24:32 2022 read: IOPS=570, BW=571MiB/s (599MB/s)(16.8GiB/30067msec) slat (usec): min=9, max=127, avg=23.22, stdev= 9.23 clat (usec): min=54, max=85063, avg=325.11, stdev=2475.87 lat (usec): min=112, max=85090, avg=348.49, stdev=2475.84 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 110], 10.00th=[ 112], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 145], 80.00th=[ 326], 90.00th=[ 347], 95.00th=[ 367], | 99.00th=[ 562], 99.50th=[ 2343], 99.90th=[46400], 99.95th=[52691], | 99.99th=[82314] bw ( KiB/s): min=393216, max=892928, per=100.00%, avg=586001.07, stdev=90459.61, samples=60 iops : min= 384, max= 872, avg=572.27, stdev=88.34, samples=60 write: IOPS=144, BW=145MiB/s (152MB/s)(4353MiB/30067msec) slat (usec): min=62, max=709, avg=167.11, stdev=84.92 clat (msec): min=3, max=180, avg=67.50, stdev=26.53 lat (msec): min=3, max=180, avg=67.67, stdev=26.53 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 20], 10.00th=[ 27], 20.00th=[ 47], | 30.00th=[ 59], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 75], | 70.00th=[ 81], 80.00th=[ 88], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 134], 99.50th=[ 144], 99.90th=[ 161], 99.95th=[ 165], | 99.99th=[ 182] bw ( KiB/s): min=124928, max=223232, per=99.99%, avg=148241.07, stdev=13165.49, samples=60 iops : min= 122, max= 218, avg=144.77, stdev=12.86, samples=60 lat (usec) : 100=0.16%, 250=62.04%, 500=16.61%, 750=0.29%, 1000=0.10% lat (msec) : 2=0.11%, 4=0.15%, 10=0.05%, 20=1.12%, 50=3.58% lat (msec) : 100=14.00%, 250=1.80% cpu : usr=2.61%, sys=2.12%, ctx=21073, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17168,4353,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=571MiB/s (599MB/s), 571MiB/s-571MiB/s (599MB/s-599MB/s), io=16.8GiB (18.0GB), run=30067-30067msec WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=4353MiB (4564MB), run=30067-30067msec Disk stats (read/write): vdi: ios=17168/4351, merge=0/0, ticks=2545/291284, in_queue=299107, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961577: Fri Jan 7 14:25:09 2022 read: IOPS=607, BW=608MiB/s (637MB/s)(17.8GiB/30002msec) clat (usec): min=121, max=26666, avg=212.81, stdev=919.48 lat (usec): min=121, max=26666, avg=212.96, stdev=919.48 clat percentiles (usec): | 1.00th=[ 130], 5.00th=[ 135], 10.00th=[ 137], 20.00th=[ 141], | 30.00th=[ 143], 40.00th=[ 145], 50.00th=[ 149], 60.00th=[ 153], | 70.00th=[ 161], 80.00th=[ 178], 90.00th=[ 192], 95.00th=[ 204], | 99.00th=[ 269], 99.50th=[ 388], 99.90th=[16712], 99.95th=[17695], | 99.99th=[24773] bw ( KiB/s): min=466944, max=806912, per=100.00%, avg=622266.80, stdev=76130.47, samples=60 iops : min= 456, max= 788, avg=607.67, stdev=74.38, samples=60 write: IOPS=153, BW=154MiB/s (161MB/s)(4614MiB/30002msec) clat (usec): min=2544, max=64878, avg=5554.58, stdev=3070.58 lat (usec): min=2648, max=64987, avg=5652.39, stdev=3069.28 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3884], | 30.00th=[ 4621], 40.00th=[ 5014], 50.00th=[ 5211], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 5997], 90.00th=[ 8094], 95.00th=[11207], | 99.00th=[15270], 99.50th=[16909], 99.90th=[54789], 99.95th=[61080], | 99.99th=[64750] bw ( KiB/s): min=126976, max=200704, per=99.99%, avg=157462.47, stdev=10918.29, samples=60 iops : min= 124, max= 196, avg=153.77, stdev=10.66, samples=60 lat (usec) : 250=78.83%, 500=0.63%, 750=0.01% lat (msec) : 2=0.01%, 4=4.27%, 10=14.50%, 20=1.69%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=1.78%, ctx=22852, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18230,4614,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=608MiB/s (637MB/s), 608MiB/s-608MiB/s (637MB/s-637MB/s), io=17.8GiB (19.1GB), run=30002-30002msec WRITE: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4614MiB (4838MB), run=30002-30002msec Disk stats (read/write): vdi: ios=18121/4576, merge=0/0, ticks=983/22899, in_queue=28597, util=95.69% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962982: Fri Jan 7 14:25:46 2022 read: IOPS=597, BW=597MiB/s (626MB/s)(17.7GiB/30297msec) slat (usec): min=9, max=317, avg=20.27, stdev= 9.20 clat (usec): min=52, max=640443, avg=1194.66, stdev=13485.58 lat (usec): min=116, max=640458, avg=1215.11, stdev=13485.63 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 110], 10.00th=[ 112], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 125], 60.00th=[ 135], | 70.00th=[ 190], 80.00th=[ 338], 90.00th=[ 388], 95.00th=[ 1942], | 99.00th=[ 12649], 99.50th=[ 16057], 99.90th=[193987], 99.95th=[274727], | 99.99th=[624952] bw ( KiB/s): min=409600, max=1151217, per=100.00%, avg=619842.43, stdev=124125.35, samples=60 iops : min= 400, max= 1124, avg=605.23, stdev=121.22, samples=60 write: IOPS=150, BW=151MiB/s (158MB/s)(4569MiB/30297msec) slat (usec): min=60, max=545, avg=184.35, stdev=95.54 clat (msec): min=9, max=2029, avg=326.44, stdev=185.39 lat (msec): min=10, max=2029, avg=326.62, stdev=185.38 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 124], 10.00th=[ 138], 20.00th=[ 165], | 30.00th=[ 197], 40.00th=[ 234], 50.00th=[ 284], 60.00th=[ 347], | 70.00th=[ 409], 80.00th=[ 472], 90.00th=[ 567], 95.00th=[ 667], | 99.00th=[ 911], 99.50th=[ 969], 99.90th=[ 1318], 99.95th=[ 1401], | 99.99th=[ 2022] bw ( KiB/s): min=114688, max=229409, per=100.00%, avg=154789.05, stdev=14602.01, samples=60 iops : min= 112, max= 224, avg=151.05, stdev=14.25, samples=60 lat (usec) : 100=0.09%, 250=57.48%, 500=15.61%, 750=0.98%, 1000=0.54% lat (msec) : 2=1.22%, 4=1.43%, 10=1.18%, 20=1.02%, 50=0.17% lat (msec) : 100=0.39%, 250=8.42%, 500=8.20%, 750=2.74%, 1000=0.43% lat (msec) : 2000=0.08%, >=2000=0.01% cpu : usr=3.08%, sys=1.90%, ctx=20502, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=18096,4569,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=597MiB/s (626MB/s), 597MiB/s-597MiB/s (626MB/s-626MB/s), io=17.7GiB (18.0GB), run=30297-30297msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4569MiB (4791MB), run=30297-30297msec Disk stats (read/write): vdi: ios=18096/4558, merge=0/0, ticks=16733/1476142, in_queue=1504073, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963565: Fri Jan 7 14:26:23 2022 read: IOPS=606, BW=607MiB/s (636MB/s)(17.8GiB/30002msec) clat (usec): min=131, max=26567, avg=224.63, stdev=907.95 lat (usec): min=131, max=26567, avg=224.79, stdev=907.95 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 206], 95.00th=[ 223], | 99.00th=[ 289], 99.50th=[ 457], 99.90th=[16909], 99.95th=[18220], | 99.99th=[22938] bw ( KiB/s): min=444416, max=774144, per=100.00%, avg=621179.32, stdev=72480.49, samples=60 iops : min= 434, max= 756, avg=606.62, stdev=70.78, samples=60 write: IOPS=153, BW=153MiB/s (161MB/s)(4601MiB/30002msec) clat (usec): min=2536, max=62372, avg=5525.12, stdev=3111.27 lat (usec): min=2647, max=62434, avg=5622.59, stdev=3110.07 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3687], | 30.00th=[ 4490], 40.00th=[ 4948], 50.00th=[ 5211], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 5997], 90.00th=[ 8094], 95.00th=[11338], | 99.00th=[15795], 99.50th=[18744], 99.90th=[53740], 99.95th=[56361], | 99.99th=[62129] bw ( KiB/s): min=126976, max=206848, per=99.99%, avg=157018.87, stdev=11914.27, samples=60 iops : min= 124, max= 202, avg=153.33, stdev=11.63, samples=60 lat (usec) : 250=78.23%, 500=1.24%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.65%, 10=14.07%, 20=1.71%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.86%, sys=2.55%, ctx=22805, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18198,4601,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=607MiB/s (636MB/s), 607MiB/s-607MiB/s (636MB/s-636MB/s), io=17.8GiB (19.1GB), run=30002-30002msec WRITE: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=4601MiB (4824MB), run=30002-30002msec Disk stats (read/write): vdi: ios=18069/4564, merge=0/0, ticks=974/22695, in_queue=28387, util=94.85% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964380: Fri Jan 7 14:27:00 2022 read: IOPS=594, BW=595MiB/s (624MB/s)(17.8GiB/30648msec) slat (usec): min=9, max=1012, avg=19.43, stdev=21.03 clat (usec): min=61, max=728330, avg=2789.13, stdev=13776.19 lat (usec): min=116, max=728346, avg=2808.73, stdev=13776.66 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 111], 10.00th=[ 112], 20.00th=[ 116], | 30.00th=[ 118], 40.00th=[ 123], 50.00th=[ 133], 60.00th=[ 219], | 70.00th=[ 343], 80.00th=[ 545], 90.00th=[ 6390], 95.00th=[ 17957], | 99.00th=[ 33162], 99.50th=[ 44303], 99.90th=[173016], 99.95th=[221250], | 99.99th=[650118] bw ( KiB/s): min=53569, max=1338602, per=100.00%, avg=628285.58, stdev=216238.84, samples=60 iops : min= 52, max= 1307, avg=613.03, stdev=211.18, samples=60 write: IOPS=150, BW=151MiB/s (158MB/s)(4614MiB/30648msec) slat (usec): min=62, max=1046, avg=197.99, stdev=103.84 clat (msec): min=26, max=2442, avg=652.68, stdev=274.94 lat (msec): min=27, max=2443, avg=652.88, stdev=274.93 clat percentiles (msec): | 1.00th=[ 130], 5.00th=[ 284], 10.00th=[ 380], 20.00th=[ 460], | 30.00th=[ 498], 40.00th=[ 542], 50.00th=[ 592], 60.00th=[ 667], | 70.00th=[ 751], 80.00th=[ 852], 90.00th=[ 995], 95.00th=[ 1133], | 99.00th=[ 1502], 99.50th=[ 1687], 99.90th=[ 2433], 99.95th=[ 2433], | 99.99th=[ 2433] bw ( KiB/s): min= 2081, max=253935, per=99.29%, avg=153072.62, stdev=50243.94, samples=61 iops : min= 2, max= 247, avg=148.98, stdev=48.97, samples=61 lat (usec) : 100=0.08%, 250=48.85%, 500=14.63%, 750=1.64%, 1000=1.12% lat (msec) : 2=2.14%, 4=1.72%, 10=3.06%, 20=3.16%, 50=3.09% lat (msec) : 100=0.13%, 250=0.90%, 500=5.48%, 750=8.00%, 1000=4.03% lat (msec) : 2000=1.92%, >=2000=0.07% cpu : usr=3.23%, sys=1.77%, ctx=17689, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18230,4614,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=595MiB/s (624MB/s), 595MiB/s-595MiB/s (624MB/s-624MB/s), io=17.8GiB (19.1GB), run=30648-30648msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4614MiB (4838MB), run=30648-30648msec Disk stats (read/write): vdi: ios=18230/4590, merge=0/0, ticks=41699/2957231, in_queue=3021614, util=99.51% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965446: Fri Jan 7 14:27:37 2022 read: IOPS=607, BW=608MiB/s (637MB/s)(17.8GiB/30006msec) clat (usec): min=130, max=26081, avg=223.22, stdev=935.92 lat (usec): min=130, max=26081, avg=223.38, stdev=935.92 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 190], 90.00th=[ 202], 95.00th=[ 217], | 99.00th=[ 289], 99.50th=[ 416], 99.90th=[17433], 99.95th=[19792], | 99.99th=[25035] bw ( KiB/s): min=471040, max=829440, per=100.00%, avg=622250.67, stdev=83442.86, samples=60 iops : min= 460, max= 810, avg=607.67, stdev=81.49, samples=60 write: IOPS=153, BW=154MiB/s (161MB/s)(4613MiB/30006msec) clat (usec): min=2545, max=66066, avg=5515.34, stdev=3107.40 lat (usec): min=2624, max=66120, avg=5613.25, stdev=3105.51 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3654], | 30.00th=[ 4490], 40.00th=[ 4883], 50.00th=[ 5211], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 5997], 90.00th=[ 8029], 95.00th=[11207], | 99.00th=[15270], 99.50th=[17957], 99.90th=[52691], 99.95th=[56886], | 99.99th=[66323] bw ( KiB/s): min=135168, max=210944, per=100.00%, avg=157422.93, stdev=11477.14, samples=60 iops : min= 132, max= 206, avg=153.73, stdev=11.21, samples=60 lat (usec) : 250=78.37%, 500=1.09%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.69%, 10=14.07%, 20=1.65%, 50=0.07% lat (msec) : 100=0.02% cpu : usr=1.90%, sys=2.41%, ctx=22851, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18230,4613,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=608MiB/s (637MB/s), 608MiB/s-608MiB/s (637MB/s-637MB/s), io=17.8GiB (19.1GB), run=30006-30006msec WRITE: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4613MiB (4837MB), run=30006-30006msec Disk stats (read/write): vdi: ios=18216/4607, merge=0/0, ticks=984/22820, in_queue=28630, util=94.94% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966062: Fri Jan 7 14:28:15 2022 read: IOPS=592, BW=592MiB/s (621MB/s)(17.8GiB/30786msec) slat (usec): min=11, max=795357, avg=1241.84, stdev=12512.54 clat (msec): min=18, max=1023, avg=111.35, stdev=113.94 lat (msec): min=18, max=1023, avg=112.59, stdev=114.68 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 37], 20.00th=[ 55], | 30.00th=[ 68], 40.00th=[ 80], 50.00th=[ 91], 60.00th=[ 104], | 70.00th=[ 117], 80.00th=[ 136], 90.00th=[ 161], 95.00th=[ 209], | 99.00th=[ 802], 99.50th=[ 835], 99.90th=[ 885], 99.95th=[ 894], | 99.99th=[ 953] bw ( KiB/s): min= 2048, max=1273856, per=100.00%, avg=610190.49, stdev=252179.67, samples=61 iops : min= 2, max= 1244, avg=595.89, stdev=246.27, samples=61 write: IOPS=149, BW=150MiB/s (157MB/s)(4615MiB/30786msec) slat (usec): min=273, max=697242, avg=1584.57, stdev=12492.57 clat (msec): min=58, max=2894, avg=886.79, stdev=341.05 lat (msec): min=59, max=2895, avg=888.37, stdev=340.61 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 414], 10.00th=[ 535], 20.00th=[ 634], | 30.00th=[ 709], 40.00th=[ 776], 50.00th=[ 835], 60.00th=[ 902], | 70.00th=[ 1003], 80.00th=[ 1116], 90.00th=[ 1334], 95.00th=[ 1536], | 99.00th=[ 1787], 99.50th=[ 2123], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2903] bw ( KiB/s): min=12288, max=288768, per=99.51%, avg=152750.90, stdev=62288.90, samples=60 iops : min= 12, max= 282, avg=149.17, stdev=60.83, samples=60 lat (msec) : 20=0.05%, 50=13.50%, 100=32.13%, 250=31.59%, 500=2.59% lat (msec) : 750=6.05%, 1000=8.01%, 2000=5.95%, >=2000=0.12% cpu : usr=5.24%, sys=1.45%, ctx=2185, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18233,4615,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=592MiB/s (621MB/s), 592MiB/s-592MiB/s (621MB/s-621MB/s), io=17.8GiB (19.1GB), run=30786-30786msec WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=4615MiB (4839MB), run=30786-30786msec Disk stats (read/write): vdi: ios=18233/4603, merge=0/0, ticks=117463/3608564, in_queue=3746822, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966915: Fri Jan 7 14:28:51 2022 read: IOPS=264, BW=530MiB/s (556MB/s)(15.5GiB/30011msec) clat (usec): min=247, max=17390, avg=452.15, stdev=960.88 lat (usec): min=247, max=17390, avg=452.34, stdev=960.88 clat percentiles (usec): | 1.00th=[ 253], 5.00th=[ 262], 10.00th=[ 269], 20.00th=[ 273], | 30.00th=[ 281], 40.00th=[ 285], 50.00th=[ 297], 60.00th=[ 310], | 70.00th=[ 334], 80.00th=[ 570], 90.00th=[ 701], 95.00th=[ 742], | 99.00th=[ 848], 99.50th=[ 9241], 99.90th=[15270], 99.95th=[15795], | 99.99th=[17433] bw ( KiB/s): min=319488, max=847872, per=100.00%, avg=542835.97, stdev=112151.26, samples=60 iops : min= 156, max= 414, avg=265.00, stdev=54.68, samples=60 write: IOPS=67, BW=134MiB/s (141MB/s)(4036MiB/30011msec) clat (usec): min=5666, max=61073, avg=12801.91, stdev=3747.38 lat (usec): min=6215, max=61266, avg=13078.09, stdev=3768.05 clat percentiles (usec): | 1.00th=[ 6718], 5.00th=[ 8455], 10.00th=[ 8979], 20.00th=[ 9896], | 30.00th=[10683], 40.00th=[11338], 50.00th=[12125], 60.00th=[13435], | 70.00th=[14746], 80.00th=[15795], 90.00th=[16581], 95.00th=[17433], | 99.00th=[23987], 99.50th=[25560], 99.90th=[49021], 99.95th=[58459], | 99.99th=[61080] bw ( KiB/s): min=114688, max=155648, per=100.00%, avg=137715.58, stdev=8349.21, samples=60 iops : min= 56, max= 76, avg=67.23, stdev= 4.09, samples=60 lat (usec) : 250=0.11%, 500=62.80%, 750=13.62%, 1000=2.66% lat (msec) : 2=0.04%, 10=4.38%, 20=15.99%, 50=0.37%, 100=0.02% cpu : usr=2.08%, sys=1.96%, ctx=9984, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7950,2018,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=530MiB/s (556MB/s), 530MiB/s-530MiB/s (556MB/s-556MB/s), io=15.5GiB (16.7GB), run=30011-30011msec WRITE: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=4036MiB (4232MB), run=30011-30011msec Disk stats (read/write): vdi: ios=15846/4015, merge=0/0, ticks=632/44143, in_queue=51702, util=95.63% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967493: Fri Jan 7 14:29:28 2022 read: IOPS=268, BW=537MiB/s (563MB/s)(15.7GiB/30011msec) slat (usec): min=53, max=202, avg=73.05, stdev=13.83 clat (usec): min=151, max=25793, avg=366.53, stdev=1038.60 lat (usec): min=257, max=25860, avg=439.81, stdev=1039.24 clat percentiles (usec): | 1.00th=[ 206], 5.00th=[ 210], 10.00th=[ 212], 20.00th=[ 217], | 30.00th=[ 221], 40.00th=[ 225], 50.00th=[ 231], 60.00th=[ 241], | 70.00th=[ 269], 80.00th=[ 326], 90.00th=[ 519], 95.00th=[ 611], | 99.00th=[ 783], 99.50th=[ 9110], 99.90th=[15533], 99.95th=[16712], | 99.99th=[25822] bw ( KiB/s): min=352256, max=819200, per=100.00%, avg=549771.43, stdev=114959.37, samples=60 iops : min= 172, max= 400, avg=268.43, stdev=56.13, samples=60 write: IOPS=68, BW=136MiB/s (143MB/s)(4086MiB/30011msec) slat (usec): min=133, max=885, avg=291.32, stdev=102.99 clat (usec): min=4647, max=55688, avg=12653.71, stdev=3845.57 lat (usec): min=4915, max=55950, avg=12945.35, stdev=3860.06 clat percentiles (usec): | 1.00th=[ 5473], 5.00th=[ 7111], 10.00th=[ 8291], 20.00th=[ 9634], | 30.00th=[10552], 40.00th=[11207], 50.00th=[11994], 60.00th=[13435], | 70.00th=[14877], 80.00th=[15926], 90.00th=[16712], 95.00th=[17433], | 99.00th=[23200], 99.50th=[25560], 99.90th=[47449], 99.95th=[48497], | 99.99th=[55837] bw ( KiB/s): min=122880, max=151552, per=99.99%, avg=139404.90, stdev=7903.63, samples=60 iops : min= 60, max= 74, avg=68.07, stdev= 3.86, samples=60 lat (usec) : 250=52.16%, 500=19.18%, 750=7.49%, 1000=0.37% lat (msec) : 2=0.06%, 4=0.02%, 10=4.72%, 20=15.56%, 50=0.44% lat (msec) : 100=0.01% cpu : usr=1.75%, sys=2.71%, ctx=10108, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8053,2043,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=537MiB/s (563MB/s), 537MiB/s-537MiB/s (563MB/s-563MB/s), io=15.7GiB (16.9GB), run=30011-30011msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4086MiB (4284MB), run=30011-30011msec Disk stats (read/write): vdi: ios=15992/4062, merge=0/0, ticks=698/49270, in_queue=56708, util=96.21% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968310: Fri Jan 7 14:30:05 2022 read: IOPS=270, BW=542MiB/s (568MB/s)(15.9GiB/30011msec) clat (usec): min=221, max=19799, avg=375.36, stdev=975.18 lat (usec): min=221, max=19799, avg=375.54, stdev=975.19 clat percentiles (usec): | 1.00th=[ 231], 5.00th=[ 237], 10.00th=[ 239], 20.00th=[ 243], | 30.00th=[ 247], 40.00th=[ 251], 50.00th=[ 258], 60.00th=[ 265], | 70.00th=[ 285], 80.00th=[ 334], 90.00th=[ 453], 95.00th=[ 586], | 99.00th=[ 775], 99.50th=[ 9241], 99.90th=[15664], 99.95th=[16319], | 99.99th=[19792] bw ( KiB/s): min=299008, max=1003520, per=100.00%, avg=554791.53, stdev=122038.26, samples=60 iops : min= 146, max= 490, avg=270.87, stdev=59.59, samples=60 write: IOPS=68, BW=137MiB/s (144MB/s)(4126MiB/30011msec) clat (usec): min=4655, max=52648, avg=12835.32, stdev=3682.99 lat (usec): min=4858, max=52874, avg=13057.74, stdev=3689.65 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 7570], 10.00th=[ 8717], 20.00th=[10028], | 30.00th=[10814], 40.00th=[11469], 50.00th=[12125], 60.00th=[13698], | 70.00th=[15139], 80.00th=[16057], 90.00th=[16909], 95.00th=[17695], | 99.00th=[21627], 99.50th=[25035], 99.90th=[44827], 99.95th=[50594], | 99.99th=[52691] bw ( KiB/s): min=122880, max=159744, per=100.00%, avg=140779.95, stdev=8663.29, samples=60 iops : min= 60, max= 78, avg=68.73, stdev= 4.23, samples=60 lat (usec) : 250=28.72%, 500=44.69%, 750=5.43%, 1000=0.41% lat (msec) : 2=0.03%, 4=0.01%, 10=4.07%, 20=16.33%, 50=0.29% lat (msec) : 100=0.02% cpu : usr=1.68%, sys=1.33%, ctx=10197, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8126,2063,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=542MiB/s (568MB/s), 542MiB/s-542MiB/s (568MB/s-568MB/s), io=15.9GiB (17.0GB), run=30011-30011msec WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=4126MiB (4326MB), run=30011-30011msec Disk stats (read/write): vdi: ios=16168/4096, merge=0/0, ticks=646/41838, in_queue=48671, util=96.58% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969569: Fri Jan 7 14:30:42 2022 read: IOPS=347, BW=695MiB/s (729MB/s)(20.5GiB/30120msec) slat (usec): min=20, max=324, avg=37.51, stdev=14.78 clat (usec): min=156, max=1106.2k, avg=1064.44, stdev=19297.72 lat (usec): min=208, max=1106.2k, avg=1102.17, stdev=19297.64 clat percentiles (usec): | 1.00th=[ 188], 5.00th=[ 190], 10.00th=[ 194], 20.00th=[ 198], | 30.00th=[ 200], 40.00th=[ 204], 50.00th=[ 210], 60.00th=[ 217], | 70.00th=[ 241], 80.00th=[ 619], 90.00th=[ 652], 95.00th=[ 676], | 99.00th=[ 898], 99.50th=[ 2089], 99.90th=[283116], 99.95th=[522191], | 99.99th=[650118] bw ( KiB/s): min=352256, max=1138688, per=100.00%, avg=714976.22, stdev=134663.65, samples=60 iops : min= 172, max= 556, avg=349.10, stdev=65.76, samples=60 write: IOPS=88, BW=177MiB/s (186MB/s)(5330MiB/30120msec) slat (usec): min=116, max=1160, avg=344.60, stdev=180.38 clat (msec): min=6, max=1434, avg=108.29, stdev=120.52 lat (msec): min=6, max=1434, avg=108.63, stdev=120.52 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 31], | 30.00th=[ 39], 40.00th=[ 50], 50.00th=[ 65], 60.00th=[ 86], | 70.00th=[ 115], 80.00th=[ 159], 90.00th=[ 257], 95.00th=[ 359], | 99.00th=[ 584], 99.50th=[ 684], 99.90th=[ 961], 99.95th=[ 1062], | 99.99th=[ 1435] bw ( KiB/s): min=155648, max=192512, per=100.00%, avg=181254.15, stdev=8216.14, samples=60 iops : min= 76, max= 94, avg=88.50, stdev= 4.01, samples=60 lat (usec) : 250=57.12%, 500=5.90%, 750=15.25%, 1000=0.84% lat (msec) : 2=0.20%, 4=0.13%, 10=0.06%, 20=1.28%, 50=6.94% lat (msec) : 100=5.15%, 250=4.92%, 500=1.83%, 750=0.30%, 1000=0.05% lat (msec) : 2000=0.02% cpu : usr=3.13%, sys=1.88%, ctx=13113, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10473,2665,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=695MiB/s (729MB/s), 695MiB/s-695MiB/s (729MB/s-729MB/s), io=20.5GiB (21.0GB), run=30120-30120msec WRITE: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=5330MiB (5589MB), run=30120-30120msec Disk stats (read/write): vdi: ios=20946/5314, merge=0/0, ticks=8464/568952, in_queue=589929, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970098: Fri Jan 7 14:31:19 2022 read: IOPS=257, BW=515MiB/s (540MB/s)(15.1GiB/30011msec) clat (usec): min=286, max=27478, avg=454.44, stdev=1239.43 lat (usec): min=286, max=27478, avg=454.61, stdev=1239.43 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 302], 10.00th=[ 306], 20.00th=[ 310], | 30.00th=[ 314], 40.00th=[ 322], 50.00th=[ 326], 60.00th=[ 334], | 70.00th=[ 355], 80.00th=[ 408], 90.00th=[ 465], 95.00th=[ 545], | 99.00th=[ 832], 99.50th=[10421], 99.90th=[21627], 99.95th=[23725], | 99.99th=[27395] bw ( KiB/s): min=258565, max=913408, per=100.00%, avg=527421.82, stdev=119009.71, samples=60 iops : min= 126, max= 446, avg=257.50, stdev=58.11, samples=60 write: IOPS=65, BW=131MiB/s (137MB/s)(3928MiB/30011msec) clat (usec): min=4731, max=72110, avg=13271.28, stdev=6669.93 lat (usec): min=4935, max=72303, avg=13482.25, stdev=6662.92 clat percentiles (usec): | 1.00th=[ 5473], 5.00th=[ 5932], 10.00th=[ 6456], 20.00th=[ 7373], | 30.00th=[ 8094], 40.00th=[ 8717], 50.00th=[10421], 60.00th=[15401], | 70.00th=[17433], 80.00th=[19530], 90.00th=[21890], 95.00th=[23725], | 99.00th=[30016], 99.50th=[32375], 99.90th=[62129], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min=110592, max=184320, per=100.00%, avg=134024.67, stdev=10027.51, samples=60 iops : min= 54, max= 90, avg=65.43, stdev= 4.90, samples=60 lat (usec) : 500=74.25%, 750=4.31%, 1000=0.61% lat (msec) : 2=0.04%, 10=10.06%, 20=7.04%, 50=3.65%, 100=0.03% cpu : usr=1.47%, sys=1.84%, ctx=9699, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7725,1964,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=515MiB/s (540MB/s), 515MiB/s-515MiB/s (540MB/s-540MB/s), io=15.1GiB (16.2GB), run=30011-30011msec WRITE: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3928MiB (4119MB), run=30011-30011msec Disk stats (read/write): vdi: ios=15374/3904, merge=0/0, ticks=857/49529, in_queue=56919, util=96.11% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971113: Fri Jan 7 14:31:56 2022 read: IOPS=349, BW=700MiB/s (734MB/s)(20.9GiB/30570msec) slat (usec): min=19, max=906, avg=33.99, stdev=19.71 clat (usec): min=67, max=2596.4k, avg=6740.81, stdev=89517.57 lat (usec): min=209, max=2596.4k, avg=6775.00, stdev=89517.64 clat percentiles (usec): | 1.00th=[ 190], 5.00th=[ 194], 10.00th=[ 198], | 20.00th=[ 202], 30.00th=[ 206], 40.00th=[ 212], | 50.00th=[ 219], 60.00th=[ 243], 70.00th=[ 619], | 80.00th=[ 660], 90.00th=[ 1188], 95.00th=[ 11207], | 99.00th=[ 34866], 99.50th=[ 263193], 99.90th=[1736442], | 99.95th=[2533360], 99.99th=[2566915] bw ( KiB/s): min=259084, max=1190220, per=100.00%, avg=731117.18, stdev=180659.13, samples=60 iops : min= 126, max= 581, avg=356.70, stdev=88.16, samples=60 write: IOPS=88, BW=178MiB/s (187MB/s)(5440MiB/30570msec) slat (usec): min=120, max=1470, avg=394.54, stdev=206.26 clat (msec): min=49, max=3389, avg=534.61, stdev=263.33 lat (msec): min=49, max=3390, avg=535.01, stdev=263.33 clat percentiles (msec): | 1.00th=[ 220], 5.00th=[ 296], 10.00th=[ 347], 20.00th=[ 380], | 30.00th=[ 401], 40.00th=[ 426], 50.00th=[ 456], 60.00th=[ 498], | 70.00th=[ 558], 80.00th=[ 651], 90.00th=[ 844], 95.00th=[ 1003], | 99.00th=[ 1536], 99.50th=[ 1821], 99.90th=[ 2869], 99.95th=[ 3004], | 99.99th=[ 3406] bw ( KiB/s): min= 4120, max=254460, per=98.66%, avg=179779.15, stdev=32250.91, samples=61 iops : min= 2, max= 124, avg=87.61, stdev=15.72, samples=61 lat (usec) : 100=0.01%, 250=49.06%, 500=6.06%, 750=14.57%, 1000=1.50% lat (msec) : 2=1.58%, 4=0.99%, 10=1.68%, 20=2.50%, 50=1.24% lat (msec) : 100=0.07%, 250=0.48%, 500=11.98%, 750=5.28%, 1000=1.83% lat (msec) : 2000=1.02%, >=2000=0.13% cpu : usr=3.60%, sys=1.70%, ctx=11921, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=10694,2720,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=700MiB/s (734MB/s), 700MiB/s-700MiB/s (734MB/s-734MB/s), io=20.9GiB (22.4GB), run=30570-30570msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=5440MiB (5704MB), run=30570-30570msec Disk stats (read/write): vdi: ios=21387/5430, merge=0/0, ticks=80243/2878290, in_queue=2978020, util=99.56% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972159: Fri Jan 7 14:32:33 2022 read: IOPS=270, BW=541MiB/s (567MB/s)(15.8GiB/30001msec) clat (usec): min=226, max=17556, avg=367.84, stdev=975.49 lat (usec): min=226, max=17556, avg=368.02, stdev=975.49 clat percentiles (usec): | 1.00th=[ 233], 5.00th=[ 241], 10.00th=[ 243], 20.00th=[ 247], | 30.00th=[ 253], 40.00th=[ 258], 50.00th=[ 265], 60.00th=[ 273], | 70.00th=[ 289], 80.00th=[ 334], 90.00th=[ 379], 95.00th=[ 453], | 99.00th=[ 734], 99.50th=[ 9372], 99.90th=[15533], 99.95th=[16319], | 99.99th=[17433] bw ( KiB/s): min=299008, max=856064, per=99.81%, avg=552838.54, stdev=113227.19, samples=59 iops : min= 146, max= 418, avg=269.93, stdev=55.29, samples=59 write: IOPS=68, BW=137MiB/s (144MB/s)(4118MiB/30001msec) clat (usec): min=4655, max=49374, avg=12901.76, stdev=3693.85 lat (usec): min=4797, max=49580, avg=13110.88, stdev=3694.10 clat percentiles (usec): | 1.00th=[ 5604], 5.00th=[ 7504], 10.00th=[ 8717], 20.00th=[10028], | 30.00th=[10814], 40.00th=[11469], 50.00th=[12125], 60.00th=[13566], | 70.00th=[15139], 80.00th=[16188], 90.00th=[16909], 95.00th=[17695], | 99.00th=[23987], 99.50th=[25297], 99.90th=[35914], 99.95th=[47973], | 99.99th=[49546] bw ( KiB/s): min=126976, max=155648, per=99.87%, avg=140379.22, stdev=7401.28, samples=59 iops : min= 62, max= 76, avg=68.54, stdev= 3.62, samples=59 lat (usec) : 250=19.70%, 500=57.14%, 750=2.16%, 1000=0.24% lat (msec) : 2=0.03%, 4=0.01%, 10=4.08%, 20=16.24%, 50=0.40% cpu : usr=1.62%, sys=1.41%, ctx=10180, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8114,2059,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s), io=15.8GiB (17.0GB), run=30001-30001msec WRITE: bw=137MiB/s (144MB/s), 137MiB/s-137MiB/s (144MB/s-144MB/s), io=4118MiB (4318MB), run=30001-30001msec Disk stats (read/write): vdi: ios=16182/4106, merge=0/0, ticks=648/42431, in_queue=49069, util=96.31% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972684: Fri Jan 7 14:33:11 2022 read: IOPS=350, BW=700MiB/s (734MB/s)(20.0GiB/30684msec) slat (usec): min=22, max=566153, avg=2189.25, stdev=12608.05 clat (msec): min=19, max=2674, avg=91.16, stdev=104.05 lat (msec): min=19, max=2674, avg=93.35, stdev=104.93 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 36], 20.00th=[ 50], | 30.00th=[ 59], 40.00th=[ 67], 50.00th=[ 75], 60.00th=[ 86], | 70.00th=[ 96], 80.00th=[ 110], 90.00th=[ 131], 95.00th=[ 159], | 99.00th=[ 575], 99.50th=[ 693], 99.90th=[ 1552], 99.95th=[ 2072], | 99.99th=[ 2601] bw ( KiB/s): min= 4096, max=1286144, per=100.00%, avg=718903.75, stdev=274619.27, samples=61 iops : min= 2, max= 628, avg=351.02, stdev=134.10, samples=61 write: IOPS=89, BW=178MiB/s (187MB/s)(5462MiB/30684msec) slat (usec): min=556, max=105032, avg=2358.45, stdev=5625.79 clat (msec): min=102, max=3268, avg=753.10, stdev=335.84 lat (msec): min=103, max=3269, avg=755.46, stdev=335.71 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 409], 10.00th=[ 477], 20.00th=[ 550], | 30.00th=[ 584], 40.00th=[ 625], 50.00th=[ 667], 60.00th=[ 718], | 70.00th=[ 793], 80.00th=[ 919], 90.00th=[ 1133], 95.00th=[ 1351], | 99.00th=[ 2232], 99.50th=[ 2702], 99.90th=[ 3171], 99.95th=[ 3205], | 99.99th=[ 3272] bw ( KiB/s): min= 4096, max=307200, per=99.89%, avg=182074.03, stdev=66623.03, samples=60 iops : min= 2, max= 150, avg=88.90, stdev=32.53, samples=60 lat (msec) : 20=0.01%, 50=16.17%, 100=42.15%, 250=19.85%, 500=2.84% lat (msec) : 750=11.76%, 1000=4.04%, 2000=2.89%, >=2000=0.30% cpu : usr=6.07%, sys=1.62%, ctx=2683, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10741,2731,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=700MiB/s (734MB/s), 700MiB/s-700MiB/s (734MB/s-734MB/s), io=20.0GiB (22.5GB), run=30684-30684msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=5462MiB (5727MB), run=30684-30684msec Disk stats (read/write): vdi: ios=21482/5457, merge=0/0, ticks=127667/3604579, in_queue=3750315, util=99.27% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973643: Fri Jan 7 14:33:47 2022 read: IOPS=269, BW=539MiB/s (565MB/s)(15.8GiB/30008msec) clat (usec): min=225, max=25920, avg=378.95, stdev=1012.56 lat (usec): min=225, max=25920, avg=379.14, stdev=1012.56 clat percentiles (usec): | 1.00th=[ 235], 5.00th=[ 241], 10.00th=[ 243], 20.00th=[ 249], | 30.00th=[ 253], 40.00th=[ 260], 50.00th=[ 269], 60.00th=[ 277], | 70.00th=[ 297], 80.00th=[ 343], 90.00th=[ 400], 95.00th=[ 494], | 99.00th=[ 742], 99.50th=[10028], 99.90th=[15139], 99.95th=[16450], | 99.99th=[25822] bw ( KiB/s): min=339968, max=802816, per=100.00%, avg=551951.58, stdev=113114.97, samples=60 iops : min= 166, max= 392, avg=269.50, stdev=55.24, samples=60 write: IOPS=68, BW=137MiB/s (143MB/s)(4100MiB/30008msec) clat (usec): min=4674, max=82418, avg=12921.52, stdev=4243.31 lat (usec): min=4856, max=82645, avg=13132.47, stdev=4245.87 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 7504], 10.00th=[ 8717], 20.00th=[10028], | 30.00th=[10814], 40.00th=[11469], 50.00th=[12125], 60.00th=[13566], | 70.00th=[15139], 80.00th=[16188], 90.00th=[16909], 95.00th=[17695], | 99.00th=[24249], 99.50th=[27395], 99.90th=[50594], 99.95th=[66323], | 99.99th=[82314] bw ( KiB/s): min=110592, max=155648, per=99.98%, avg=139883.05, stdev=8917.21, samples=60 iops : min= 54, max= 76, avg=68.30, stdev= 4.35, samples=60 lat (usec) : 250=18.29%, 500=57.68%, 750=3.04%, 1000=0.20% lat (msec) : 2=0.05%, 10=4.07%, 20=16.32%, 50=0.33%, 100=0.03% cpu : usr=1.55%, sys=1.52%, ctx=10152, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8085,2050,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=15.8GiB (16.0GB), run=30008-30008msec WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4100MiB (4299MB), run=30008-30008msec Disk stats (read/write): vdi: ios=16036/4070, merge=0/0, ticks=678/42176, in_queue=48932, util=95.92% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974268: Fri Jan 7 14:34:25 2022 read: IOPS=344, BW=690MiB/s (723MB/s)(20.7GiB/30671msec) slat (usec): min=23, max=491110, avg=2159.20, stdev=10377.50 clat (msec): min=82, max=2948, avg=326.53, stdev=147.01 lat (msec): min=82, max=2953, avg=328.69, stdev=147.34 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 140], 10.00th=[ 197], 20.00th=[ 247], | 30.00th=[ 275], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 380], 90.00th=[ 439], 95.00th=[ 575], | 99.00th=[ 818], 99.50th=[ 894], 99.90th=[ 2022], 99.95th=[ 2366], | 99.99th=[ 2869] bw ( KiB/s): min= 4096, max=1329763, per=99.51%, avg=703033.39, stdev=246300.02, samples=61 iops : min= 2, max= 649, avg=343.26, stdev=120.25, samples=61 write: IOPS=87, BW=176MiB/s (184MB/s)(5384MiB/30671msec) slat (usec): min=532, max=468832, avg=2657.95, stdev=11290.44 clat (msec): min=169, max=3700, avg=979.99, stdev=317.66 lat (msec): min=170, max=3701, avg=982.65, stdev=317.83 clat percentiles (msec): | 1.00th=[ 456], 5.00th=[ 634], 10.00th=[ 701], 20.00th=[ 768], | 30.00th=[ 818], 40.00th=[ 869], 50.00th=[ 911], 60.00th=[ 961], | 70.00th=[ 1036], 80.00th=[ 1150], 90.00th=[ 1334], 95.00th=[ 1519], | 99.00th=[ 2089], 99.50th=[ 2400], 99.90th=[ 3339], 99.95th=[ 3373], | 99.99th=[ 3708] bw ( KiB/s): min=12288, max=360448, per=100.00%, avg=180791.61, stdev=63979.93, samples=59 iops : min= 6, max= 176, avg=88.27, stdev=31.24, samples=59 lat (msec) : 100=0.54%, 250=16.39%, 500=57.61%, 750=6.78%, 1000=11.55% lat (msec) : 2000=6.72%, >=2000=0.41% cpu : usr=6.01%, sys=1.58%, ctx=2705, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10580,2692,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=690MiB/s (723MB/s), 690MiB/s-690MiB/s (723MB/s-723MB/s), io=20.7GiB (22.2GB), run=30671-30671msec WRITE: bw=176MiB/s (184MB/s), 176MiB/s-176MiB/s (184MB/s-184MB/s), io=5384MiB (5646MB), run=30671-30671msec Disk stats (read/write): vdi: ios=21160/5368, merge=0/0, ticks=129700/3595155, in_queue=3751903, util=98.97% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975111: Fri Jan 7 14:35:01 2022 read: IOPS=162, BW=650MiB/s (682MB/s)(19.1GiB/30016msec) clat (usec): min=406, max=21640, avg=836.82, stdev=1529.25 lat (usec): min=406, max=21640, avg=837.04, stdev=1529.26 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 433], 10.00th=[ 437], 20.00th=[ 445], | 30.00th=[ 457], 40.00th=[ 465], 50.00th=[ 478], 60.00th=[ 498], | 70.00th=[ 578], 80.00th=[ 1287], 90.00th=[ 1352], 95.00th=[ 1418], | 99.00th=[ 9896], 99.50th=[14222], 99.90th=[19268], 99.95th=[21103], | 99.99th=[21627] bw ( KiB/s): min=385024, max=1032192, per=100.00%, avg=666443.00, stdev=152042.42, samples=60 iops : min= 94, max= 252, avg=162.70, stdev=37.12, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5020MiB/30016msec) clat (msec): min=11, max=127, avg=20.05, stdev= 6.33 lat (msec): min=11, max=127, avg=20.65, stdev= 6.36 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 27], | 99.00th=[ 34], 99.50th=[ 48], 99.90th=[ 91], 99.95th=[ 128], | 99.99th=[ 128] bw ( KiB/s): min=114688, max=196608, per=99.98%, avg=171218.82, stdev=13720.23, samples=60 iops : min= 28, max= 48, avg=41.80, stdev= 3.35, samples=60 lat (usec) : 500=48.37%, 750=12.16%, 1000=0.98% lat (msec) : 2=16.93%, 4=0.08%, 10=0.26%, 20=12.48%, 50=8.64% lat (msec) : 100=0.08%, 250=0.02% cpu : usr=2.76%, sys=1.94%, ctx=6145, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4881,1255,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=650MiB/s (682MB/s), 650MiB/s-650MiB/s (682MB/s-682MB/s), io=19.1GiB (20.5GB), run=30016-30016msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5020MiB (5264MB), run=30016-30016msec Disk stats (read/write): vdi: ios=19340/4985, merge=0/0, ticks=5045/80683, in_queue=95260, util=95.81% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976332: Fri Jan 7 14:35:38 2022 read: IOPS=161, BW=647MiB/s (679MB/s)(18.0GiB/30015msec) slat (usec): min=72, max=243, avg=93.08, stdev=18.08 clat (usec): min=227, max=22299, avg=717.29, stdev=1543.82 lat (usec): min=409, max=22384, avg=810.63, stdev=1545.52 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 343], 10.00th=[ 347], 20.00th=[ 355], | 30.00th=[ 363], 40.00th=[ 367], 50.00th=[ 375], 60.00th=[ 388], | 70.00th=[ 437], 80.00th=[ 1012], 90.00th=[ 1221], 95.00th=[ 1270], | 99.00th=[ 9503], 99.50th=[14222], 99.90th=[20579], 99.95th=[21103], | 99.99th=[22414] bw ( KiB/s): min=303104, max=1015808, per=100.00%, avg=663029.12, stdev=171608.81, samples=60 iops : min= 74, max= 248, avg=161.87, stdev=41.90, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5008MiB/30015msec) slat (usec): min=267, max=1657, avg=640.06, stdev=311.28 clat (msec): min=11, max=212, avg=20.18, stdev= 9.09 lat (msec): min=11, max=214, avg=20.82, stdev= 9.12 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 27], | 99.00th=[ 57], 99.50th=[ 73], 99.90th=[ 128], 99.95th=[ 213], | 99.99th=[ 213] bw ( KiB/s): min=106496, max=196608, per=99.97%, avg=170809.48, stdev=17389.31, samples=60 iops : min= 26, max= 48, avg=41.70, stdev= 4.24, samples=60 lat (usec) : 250=0.02%, 500=58.82%, 750=2.65%, 1000=1.98% lat (msec) : 2=15.01%, 10=0.26%, 20=12.90%, 50=8.14%, 100=0.16% lat (msec) : 250=0.05% cpu : usr=2.57%, sys=1.92%, ctx=6116, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4856,1252,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=647MiB/s (679MB/s), 647MiB/s-647MiB/s (679MB/s-679MB/s), io=18.0GiB (20.4GB), run=30015-30015msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5008MiB (5251MB), run=30015-30015msec Disk stats (read/write): vdi: ios=19320/4976, merge=0/0, ticks=4618/80970, in_queue=95140, util=96.13% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977097: Fri Jan 7 14:36:15 2022 read: IOPS=164, BW=659MiB/s (691MB/s)(19.3GiB/30014msec) clat (usec): min=378, max=22400, avg=770.57, stdev=1573.03 lat (usec): min=378, max=22400, avg=770.78, stdev=1573.03 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 416], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 445], | 70.00th=[ 494], 80.00th=[ 1037], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[10028], 99.50th=[14484], 99.90th=[20841], 99.95th=[21103], | 99.99th=[22414] bw ( KiB/s): min=327680, max=1073152, per=100.00%, avg=675360.33, stdev=164392.67, samples=60 iops : min= 80, max= 262, avg=164.87, stdev=40.14, samples=60 write: IOPS=42, BW=170MiB/s (178MB/s)(5100MiB/30014msec) clat (msec): min=10, max=113, avg=19.97, stdev= 6.02 lat (msec): min=11, max=114, avg=20.54, stdev= 6.05 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 27], | 99.00th=[ 46], 99.50th=[ 58], 99.90th=[ 69], 99.95th=[ 114], | 99.99th=[ 114] bw ( KiB/s): min=139264, max=196608, per=99.98%, avg=173961.23, stdev=11024.52, samples=60 iops : min= 34, max= 48, avg=42.47, stdev= 2.69, samples=60 lat (usec) : 500=55.94%, 750=5.35%, 1000=1.96% lat (msec) : 2=15.24%, 4=0.02%, 10=0.21%, 20=12.97%, 50=8.17% lat (msec) : 100=0.13%, 250=0.02% cpu : usr=2.52%, sys=1.73%, ctx=6229, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4946,1275,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=659MiB/s (691MB/s), 659MiB/s-659MiB/s (691MB/s-691MB/s), io=19.3GiB (20.7GB), run=30014-30014msec WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=5100MiB (5348MB), run=30014-30014msec Disk stats (read/write): vdi: ios=19636/5056, merge=0/0, ticks=4560/81626, in_queue=95808, util=96.66% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977963: Fri Jan 7 14:36:52 2022 read: IOPS=178, BW=713MiB/s (748MB/s)(21.0GiB/30187msec) slat (usec): min=40, max=421, avg=60.09, stdev=16.95 clat (usec): min=290, max=2527.0k, avg=5625.64, stdev=66750.94 lat (usec): min=381, max=2528.0k, avg=5686.00, stdev=66751.06 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 343], 10.00th=[ 347], | 20.00th=[ 355], 30.00th=[ 363], 40.00th=[ 371], | 50.00th=[ 379], 60.00th=[ 392], 70.00th=[ 449], | 80.00th=[ 1237], 90.00th=[ 1287], 95.00th=[ 1336], | 99.00th=[ 69731], 99.50th=[ 316670], 99.90th=[1098908], | 99.95th=[1249903], 99.99th=[2533360] bw ( KiB/s): min=352256, max=1286144, per=100.00%, avg=735054.93, stdev=187302.21, samples=60 iops : min= 86, max= 314, avg=179.43, stdev=45.73, samples=60 write: IOPS=47, BW=189MiB/s (198MB/s)(5692MiB/30187msec) slat (usec): min=246, max=1999, avg=753.36, stdev=383.39 clat (msec): min=15, max=2515, avg=189.73, stdev=255.89 lat (msec): min=16, max=2516, avg=190.48, stdev=255.91 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 55], | 30.00th=[ 66], 40.00th=[ 80], 50.00th=[ 99], 60.00th=[ 131], | 70.00th=[ 171], 80.00th=[ 251], 90.00th=[ 439], 95.00th=[ 684], | 99.00th=[ 1200], 99.50th=[ 1519], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min=147456, max=221184, per=99.93%, avg=192948.12, stdev=15275.34, samples=60 iops : min= 36, max= 54, avg=47.10, stdev= 3.73, samples=60 lat (usec) : 500=56.71%, 750=2.04%, 1000=0.71% lat (msec) : 2=17.98%, 4=0.26%, 10=0.18%, 20=0.22%, 50=3.57% lat (msec) : 100=7.38%, 250=6.32%, 500=2.64%, 750=0.90%, 1000=0.57% lat (msec) : 2000=0.43%, >=2000=0.09% cpu : usr=3.49%, sys=1.47%, ctx=6771, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5383,1423,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=713MiB/s (748MB/s), 713MiB/s-713MiB/s (748MB/s-748MB/s), io=21.0GiB (22.6GB), run=30187-30187msec WRITE: bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=5692MiB (5968MB), run=30187-30187msec Disk stats (read/write): vdi: ios=21532/5674, merge=0/0, ticks=38877/1027155, in_queue=1087306, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978556: Fri Jan 7 14:37:29 2022 read: IOPS=162, BW=650MiB/s (682MB/s)(19.1GiB/30016msec) clat (usec): min=378, max=36456, avg=778.51, stdev=1732.54 lat (usec): min=378, max=36456, avg=778.70, stdev=1732.55 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 416], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 441], | 70.00th=[ 474], 80.00th=[ 979], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[10028], 99.50th=[14877], 99.90th=[21627], 99.95th=[24511], | 99.99th=[36439] bw ( KiB/s): min=352256, max=1040384, per=100.00%, avg=666419.20, stdev=167538.97, samples=60 iops : min= 86, max= 254, avg=162.70, stdev=40.90, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5020MiB/30016msec) clat (msec): min=10, max=165, avg=20.31, stdev= 7.60 lat (msec): min=11, max=165, avg=20.88, stdev= 7.63 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 28], | 99.00th=[ 44], 99.50th=[ 66], 99.90th=[ 124], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min=114688, max=196608, per=99.97%, avg=171212.80, stdev=15797.28, samples=60 iops : min= 28, max= 48, avg=41.80, stdev= 3.86, samples=60 lat (usec) : 500=57.38%, 750=4.11%, 1000=2.36% lat (msec) : 2=14.67%, 4=0.02%, 10=0.21%, 20=12.53%, 50=8.54% lat (msec) : 100=0.15%, 250=0.03% cpu : usr=2.54%, sys=1.79%, ctx=6142, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4881,1255,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=650MiB/s (682MB/s), 650MiB/s-650MiB/s (682MB/s-682MB/s), io=19.1GiB (20.5GB), run=30016-30016msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5020MiB (5264MB), run=30016-30016msec Disk stats (read/write): vdi: ios=19440/5008, merge=0/0, ticks=4358/82112, in_queue=95951, util=96.31% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979528: Fri Jan 7 14:38:07 2022 read: IOPS=178, BW=716MiB/s (751MB/s)(21.4GiB/30593msec) slat (usec): min=46, max=497605, avg=4218.67, stdev=15710.60 clat (msec): min=17, max=2780, avg=84.61, stdev=104.80 lat (msec): min=17, max=2780, avg=88.83, stdev=106.30 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 26], 10.00th=[ 30], 20.00th=[ 40], | 30.00th=[ 47], 40.00th=[ 55], 50.00th=[ 65], 60.00th=[ 77], | 70.00th=[ 88], 80.00th=[ 105], 90.00th=[ 131], 95.00th=[ 165], | 99.00th=[ 527], 99.50th=[ 642], 99.90th=[ 1284], 99.95th=[ 1838], | 99.99th=[ 2769] bw ( KiB/s): min=90112, max=1469306, per=100.00%, avg=745852.30, stdev=308039.22, samples=60 iops : min= 22, max= 358, avg=182.07, stdev=75.17, samples=60 write: IOPS=47, BW=189MiB/s (198MB/s)(5772MiB/30593msec) slat (usec): min=1131, max=480898, avg=4772.11, stdev=15100.14 clat (msec): min=89, max=3581, avg=717.14, stdev=319.17 lat (msec): min=91, max=3582, avg=721.91, stdev=319.34 clat percentiles (msec): | 1.00th=[ 317], 5.00th=[ 426], 10.00th=[ 485], 20.00th=[ 542], | 30.00th=[ 575], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 676], | 70.00th=[ 735], 80.00th=[ 852], 90.00th=[ 1028], 95.00th=[ 1217], | 99.00th=[ 2567], 99.50th=[ 2903], 99.90th=[ 3104], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 8192, max=344753, per=97.89%, avg=189112.49, stdev=67206.83, samples=61 iops : min= 2, max= 84, avg=46.16, stdev=16.40, samples=61 lat (msec) : 20=0.07%, 50=27.07%, 100=34.34%, 250=15.55%, 500=3.48% lat (msec) : 750=13.35%, 1000=3.58%, 2000=2.30%, >=2000=0.25% cpu : usr=6.47%, sys=1.57%, ctx=2550, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5476,1443,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=716MiB/s (751MB/s), 716MiB/s-716MiB/s (751MB/s-751MB/s), io=21.4GiB (22.0GB), run=30593-30593msec WRITE: bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=5772MiB (6052MB), run=30593-30593msec Disk stats (read/write): vdi: ios=21904/5746, merge=0/0, ticks=136664/3600632, in_queue=3766335, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980491: Fri Jan 7 14:38:43 2022 read: IOPS=164, BW=658MiB/s (690MB/s)(19.3GiB/30001msec) clat (usec): min=377, max=32738, avg=778.72, stdev=1693.01 lat (usec): min=377, max=32738, avg=778.92, stdev=1693.02 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 400], 10.00th=[ 404], 20.00th=[ 412], | 30.00th=[ 420], 40.00th=[ 424], 50.00th=[ 437], 60.00th=[ 445], | 70.00th=[ 486], 80.00th=[ 971], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[10552], 99.50th=[14615], 99.90th=[21627], 99.95th=[26870], | 99.99th=[32637] bw ( KiB/s): min=360448, max=1056768, per=99.76%, avg=672045.07, stdev=163396.69, samples=59 iops : min= 88, max= 258, avg=164.07, stdev=39.89, samples=59 write: IOPS=42, BW=169MiB/s (178MB/s)(5080MiB/30001msec) clat (msec): min=10, max=115, avg=20.04, stdev= 6.48 lat (msec): min=11, max=116, avg=20.59, stdev= 6.51 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 27], | 99.00th=[ 36], 99.50th=[ 51], 99.90th=[ 109], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min=131072, max=196608, per=100.00%, avg=173426.86, stdev=13018.43, samples=59 iops : min= 32, max= 48, avg=42.34, stdev= 3.18, samples=59 lat (usec) : 500=56.48%, 750=5.06%, 1000=2.50% lat (msec) : 2=14.47%, 10=0.18%, 20=12.77%, 50=8.43%, 100=0.08% lat (msec) : 250=0.03% cpu : usr=2.44%, sys=1.89%, ctx=6210, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4934,1270,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=658MiB/s (690MB/s), 658MiB/s-658MiB/s (690MB/s-690MB/s), io=19.3GiB (20.7GB), run=30001-30001msec WRITE: bw=169MiB/s (178MB/s), 169MiB/s-169MiB/s (178MB/s-178MB/s), io=5080MiB (5327MB), run=30001-30001msec Disk stats (read/write): vdi: ios=19612/5048, merge=0/0, ticks=4356/81482, in_queue=95559, util=96.14% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981075: Fri Jan 7 14:39:21 2022 read: IOPS=174, BW=698MiB/s (732MB/s)(20.0GiB/30729msec) slat (usec): min=45, max=524424, avg=4256.59, stdev=16514.84 clat (msec): min=84, max=1886, avg=312.52, stdev=153.56 lat (msec): min=84, max=1887, avg=316.78, stdev=154.42 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 115], 10.00th=[ 167], 20.00th=[ 205], | 30.00th=[ 239], 40.00th=[ 268], 50.00th=[ 296], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 380], 90.00th=[ 456], 95.00th=[ 617], | 99.00th=[ 894], 99.50th=[ 1150], 99.90th=[ 1385], 99.95th=[ 1452], | 99.99th=[ 1888] bw ( KiB/s): min= 8192, max=1581056, per=99.91%, avg=714315.54, stdev=321462.75, samples=61 iops : min= 2, max= 386, avg=174.39, stdev=78.48, samples=61 write: IOPS=46, BW=185MiB/s (194MB/s)(5676MiB/30729msec) slat (usec): min=1143, max=467190, avg=5040.65, stdev=15141.92 clat (msec): min=321, max=3607, avg=959.18, stdev=404.56 lat (msec): min=322, max=3608, avg=964.22, stdev=404.96 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 550], 10.00th=[ 667], 20.00th=[ 735], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1133], 90.00th=[ 1334], 95.00th=[ 1569], | 99.00th=[ 3171], 99.50th=[ 3272], 99.90th=[ 3406], 99.95th=[ 3608], | 99.99th=[ 3608] bw ( KiB/s): min= 8192, max=360448, per=100.00%, avg=189526.78, stdev=78934.23, samples=59 iops : min= 2, max= 88, avg=46.27, stdev=19.27, samples=59 lat (msec) : 100=1.62%, 250=25.59%, 500=46.59%, 750=9.29%, 1000=10.61% lat (msec) : 2000=5.81%, >=2000=0.49% cpu : usr=6.30%, sys=1.58%, ctx=2535, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5364,1419,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=698MiB/s (732MB/s), 698MiB/s-698MiB/s (732MB/s-732MB/s), io=20.0GiB (22.5GB), run=30729-30729msec WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5676MiB (5952MB), run=30729-30729msec Disk stats (read/write): vdi: ios=21456/5661, merge=0/0, ticks=120916/3619857, in_queue=3766939, util=99.08% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981892: Fri Jan 7 14:39:57 2022 read: IOPS=164, BW=657MiB/s (689MB/s)(19.3GiB/30023msec) clat (usec): min=381, max=90380, avg=783.06, stdev=2048.94 lat (usec): min=381, max=90380, avg=783.25, stdev=2048.94 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 400], 10.00th=[ 404], 20.00th=[ 412], | 30.00th=[ 420], 40.00th=[ 424], 50.00th=[ 433], 60.00th=[ 445], | 70.00th=[ 478], 80.00th=[ 955], 90.00th=[ 1237], 95.00th=[ 1319], | 99.00th=[ 9765], 99.50th=[14615], 99.90th=[21103], 99.95th=[21627], | 99.99th=[90702] bw ( KiB/s): min=286720, max=1081344, per=100.00%, avg=673713.18, stdev=174767.77, samples=60 iops : min= 70, max= 264, avg=164.47, stdev=42.68, samples=60 write: IOPS=42, BW=169MiB/s (177MB/s)(5080MiB/30023msec) clat (usec): min=10853, max=76717, avg=20029.35, stdev=5824.21 lat (usec): min=11255, max=77304, avg=20586.10, stdev=5864.45 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14091], 10.00th=[15008], 20.00th=[16057], | 30.00th=[16909], 40.00th=[17957], 50.00th=[18744], 60.00th=[20055], | 70.00th=[22152], 80.00th=[23725], 90.00th=[24773], 95.00th=[26346], | 99.00th=[45876], 99.50th=[56886], 99.90th=[73925], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min=131072, max=196608, per=100.00%, avg=173278.57, stdev=12519.21, samples=60 iops : min= 32, max= 48, avg=42.30, stdev= 3.05, samples=60 lat (usec) : 500=56.90%, 750=4.71%, 1000=2.72% lat (msec) : 2=14.17%, 4=0.02%, 10=0.24%, 20=12.81%, 50=8.25% lat (msec) : 100=0.18% cpu : usr=2.41%, sys=1.98%, ctx=6210, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4934,1270,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=657MiB/s (689MB/s), 657MiB/s-657MiB/s (689MB/s-689MB/s), io=19.3GiB (20.7GB), run=30023-30023msec WRITE: bw=169MiB/s (177MB/s), 169MiB/s-169MiB/s (177MB/s-177MB/s), io=5080MiB (5327MB), run=30023-30023msec Disk stats (read/write): vdi: ios=19628/5052, merge=0/0, ticks=4265/81655, in_queue=95630, util=95.37% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983243: Fri Jan 7 14:40:35 2022 read: IOPS=178, BW=714MiB/s (749MB/s)(21.3GiB/30568msec) slat (usec): min=46, max=627751, avg=4238.50, stdev=17048.60 clat (msec): min=258, max=2541, avg=749.69, stdev=193.23 lat (msec): min=258, max=2541, avg=753.92, stdev=194.00 clat percentiles (msec): | 1.00th=[ 397], 5.00th=[ 477], 10.00th=[ 523], 20.00th=[ 592], | 30.00th=[ 651], 40.00th=[ 693], 50.00th=[ 735], 60.00th=[ 776], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 978], 95.00th=[ 1062], | 99.00th=[ 1301], 99.50th=[ 1435], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2534] bw ( KiB/s): min=81920, max=1409024, per=100.00%, avg=739918.10, stdev=281993.61, samples=59 iops : min= 20, max= 344, avg=180.64, stdev=68.85, samples=59 write: IOPS=47, BW=188MiB/s (197MB/s)(5748MiB/30568msec) slat (usec): min=1126, max=476617, avg=4755.37, stdev=14897.92 clat (msec): min=375, max=3624, avg=1357.66, stdev=359.96 lat (msec): min=376, max=3629, avg=1362.42, stdev=359.44 clat percentiles (msec): | 1.00th=[ 718], 5.00th=[ 877], 10.00th=[ 978], 20.00th=[ 1099], | 30.00th=[ 1200], 40.00th=[ 1250], 50.00th=[ 1318], 60.00th=[ 1385], | 70.00th=[ 1452], 80.00th=[ 1569], 90.00th=[ 1770], 95.00th=[ 1955], | 99.00th=[ 2769], 99.50th=[ 3037], 99.90th=[ 3608], 99.95th=[ 3641], | 99.99th=[ 3641] bw ( KiB/s): min= 8192, max=516096, per=96.93%, avg=186641.07, stdev=110845.92, samples=60 iops : min= 2, max= 126, avg=45.57, stdev=27.06, samples=60 lat (msec) : 500=5.31%, 750=37.01%, 1000=32.86%, 2000=23.87%, >=2000=0.94% cpu : usr=6.40%, sys=1.65%, ctx=2494, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5458,1437,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=714MiB/s (749MB/s), 714MiB/s-714MiB/s (749MB/s-749MB/s), io=21.3GiB (22.9GB), run=30568-30568msec WRITE: bw=188MiB/s (197MB/s), 188MiB/s-188MiB/s (197MB/s-197MB/s), io=5748MiB (6027MB), run=30568-30568msec Disk stats (read/write): vdi: ios=21832/5706, merge=0/0, ticks=142978/3572321, in_queue=3753394, util=98.23%