30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890564: Fri Jan 7 13:33:20 2022 read: IOPS=2461, BW=9848KiB/s (10.1MB/s)(289MiB/30001msec) clat (usec): min=25, max=22600, avg=59.34, stdev=388.53 lat (usec): min=25, max=22600, avg=59.48, stdev=388.53 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 66], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 725], 99.95th=[12256], | 99.99th=[16188] bw ( KiB/s): min= 7032, max=14256, per=100.00%, avg=9852.20, stdev=1085.15, samples=59 iops : min= 1758, max= 3564, avg=2463.05, stdev=271.29, samples=59 write: IOPS=614, BW=2460KiB/s (2519kB/s)(72.1MiB/30001msec) clat (usec): min=192, max=45495, avg=1379.03, stdev=1628.95 lat (usec): min=192, max=45496, avg=1379.31, stdev=1628.95 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 578], 10.00th=[ 791], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1467], 90.00th=[ 1663], 95.00th=[ 1942], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[35390], 99.95th=[40109], | 99.99th=[44827] bw ( KiB/s): min= 1808, max= 3648, per=100.00%, avg=2464.88, stdev=259.88, samples=59 iops : min= 452, max= 912, avg=616.22, stdev=64.97, samples=59 lat (usec) : 50=52.91%, 100=26.77%, 250=0.34%, 500=0.71%, 750=0.99% lat (usec) : 1000=2.29% lat (msec) : 2=15.03%, 4=0.59%, 10=0.24%, 20=0.09%, 50=0.02% cpu : usr=1.39%, sys=3.99%, ctx=92315, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73859,18450,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9848KiB/s (10.1MB/s), 9848KiB/s-9848KiB/s (10.1MB/s-10.1MB/s), io=289MiB (303MB), run=30001-30001msec WRITE: bw=2460KiB/s (2519kB/s), 2460KiB/s-2460KiB/s (2519kB/s-2519kB/s), io=72.1MiB (75.6MB), run=30001-30001msec Disk stats (read/write): vdh: ios=73420/18344, merge=0/0, ticks=792/17552, in_queue=28091, util=93.70% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891408: Fri Jan 7 13:33:57 2022 read: IOPS=2449, BW=9797KiB/s (10.0MB/s)(287MiB/30001msec) slat (usec): min=3, max=135, avg= 9.55, stdev= 4.01 clat (nsec): min=675, max=27148k, avg=46447.50, stdev=393225.41 lat (usec): min=23, max=27157, avg=56.15, stdev=393.25 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 36], | 70.00th=[ 38], 80.00th=[ 41], 90.00th=[ 45], 95.00th=[ 49], | 99.00th=[ 62], 99.50th=[ 71], 99.90th=[ 367], 99.95th=[11731], | 99.99th=[16450] bw ( KiB/s): min= 7256, max=13488, per=100.00%, avg=9797.45, stdev=1067.02, samples=60 iops : min= 1814, max= 3372, avg=2449.35, stdev=266.73, samples=60 write: IOPS=611, BW=2448KiB/s (2506kB/s)(71.7MiB/30001msec) slat (usec): min=6, max=290, avg=10.60, stdev= 4.76 clat (usec): min=176, max=51474, avg=1390.61, stdev=1634.95 lat (usec): min=185, max=51487, avg=1401.37, stdev=1635.01 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 594], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1942], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[35914], 99.95th=[40109], | 99.99th=[48497] bw ( KiB/s): min= 1880, max= 3504, per=100.00%, avg=2447.68, stdev=230.50, samples=60 iops : min= 470, max= 876, avg=611.92, stdev=57.62, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.12%, 4=0.01%, 10=0.04%, 20=0.24%, 50=76.30% lat (usec) : 100=3.08%, 250=0.20%, 500=0.66%, 750=0.84%, 1000=2.13% lat (msec) : 2=15.36%, 4=0.59%, 10=0.23%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.34%, sys=5.94%, ctx=91585, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73478,18358,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9797KiB/s (10.0MB/s), 9797KiB/s-9797KiB/s (10.0MB/s-10.0MB/s), io=287MiB (301MB), run=30001-30001msec WRITE: bw=2448KiB/s (2506kB/s), 2448KiB/s-2448KiB/s (2506kB/s-2506kB/s), io=71.7MiB (75.2MB), run=30001-30001msec Disk stats (read/write): vdh: ios=73091/18270, merge=0/0, ticks=764/17959, in_queue=28123, util=93.80% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892537: Fri Jan 7 13:34:34 2022 read: IOPS=2468, BW=9874KiB/s (10.1MB/s)(289MiB/30001msec) clat (usec): min=25, max=23325, avg=58.56, stdev=377.33 lat (usec): min=25, max=23325, avg=58.70, stdev=377.33 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 416], 99.95th=[11469], | 99.99th=[15401] bw ( KiB/s): min= 7848, max=14520, per=100.00%, avg=9874.98, stdev=1060.44, samples=60 iops : min= 1962, max= 3630, avg=2468.73, stdev=265.11, samples=60 write: IOPS=616, BW=2465KiB/s (2525kB/s)(72.2MiB/30001msec) clat (usec): min=198, max=44713, avg=1378.56, stdev=1601.22 lat (usec): min=198, max=44713, avg=1378.84, stdev=1601.22 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 578], 10.00th=[ 799], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1680], 95.00th=[ 1958], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min= 1976, max= 3752, per=100.00%, avg=2465.40, stdev=262.22, samples=60 iops : min= 494, max= 938, avg=616.35, stdev=65.56, samples=60 lat (usec) : 50=53.82%, 100=25.89%, 250=0.28%, 500=0.76%, 750=0.94% lat (usec) : 1000=2.29% lat (msec) : 2=15.06%, 4=0.61%, 10=0.21%, 20=0.10%, 50=0.02% cpu : usr=1.49%, sys=3.86%, ctx=92561, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74060,18491,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9874KiB/s (10.1MB/s), 9874KiB/s-9874KiB/s (10.1MB/s-10.1MB/s), io=289MiB (303MB), run=30001-30001msec WRITE: bw=2465KiB/s (2525kB/s), 2465KiB/s-2465KiB/s (2525kB/s-2525kB/s), io=72.2MiB (75.7MB), run=30001-30001msec Disk stats (read/write): vdh: ios=73678/18404, merge=0/0, ticks=765/17554, in_queue=28067, util=93.64% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892990: Fri Jan 7 13:35:11 2022 read: IOPS=2511, BW=9.81MiB/s (10.3MB/s)(295MiB/30025msec) slat (usec): min=2, max=159, avg= 9.56, stdev= 3.82 clat (nsec): min=667, max=2495.4M, avg=1019644.81, stdev=41869513.25 lat (usec): min=17, max=2495.5k, avg=1029.35, stdev=41869.52 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 33], 40.00th=[ 35], | 50.00th=[ 36], 60.00th=[ 38], 70.00th=[ 40], | 80.00th=[ 44], 90.00th=[ 49], 95.00th=[ 56], | 99.00th=[ 80], 99.50th=[ 97], 99.90th=[ 848], | 99.95th=[ 742392], 99.99th=[2499806] bw ( KiB/s): min= 8224, max=17472, per=100.00%, avg=10052.73, stdev=1272.90, samples=60 iops : min= 2056, max= 4368, avg=2513.17, stdev=318.23, samples=60 write: IOPS=626, BW=2505KiB/s (2565kB/s)(73.4MiB/30025msec) slat (usec): min=3, max=148, avg=10.76, stdev= 3.87 clat (usec): min=521, max=68709, avg=11814.41, stdev=6037.92 lat (usec): min=530, max=68722, avg=11825.33, stdev=6037.98 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 4228], 10.00th=[ 6980], 20.00th=[ 8586], | 30.00th=[ 9765], 40.00th=[10552], 50.00th=[11338], 60.00th=[12256], | 70.00th=[12911], 80.00th=[13698], 90.00th=[15926], 95.00th=[20055], | 99.00th=[28443], 99.50th=[61604], 99.90th=[64226], 99.95th=[65274], | 99.99th=[68682] bw ( KiB/s): min= 2144, max= 4432, per=100.00%, avg=2506.13, stdev=300.17, samples=60 iops : min= 536, max= 1108, avg=626.53, stdev=75.04, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.13%, 50=73.05% lat (usec) : 100=6.39%, 250=0.27%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.30%, 4=0.66%, 10=5.68%, 20=12.31%, 50=0.87% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=1.42%, sys=6.04%, ctx=92162, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=75393,18803,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9.81MiB/s (10.3MB/s), 9.81MiB/s-9.81MiB/s (10.3MB/s-10.3MB/s), io=295MiB (309MB), run=30025-30025msec WRITE: bw=2505KiB/s (2565kB/s), 2505KiB/s-2505KiB/s (2565kB/s-2565kB/s), io=73.4MiB (77.0MB), run=30025-30025msec Disk stats (read/write): vdh: ios=74841/18668, merge=0/0, ticks=72914/211069, in_queue=296424, util=99.49% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894296: Fri Jan 7 13:35:48 2022 read: IOPS=2456, BW=9826KiB/s (10.1MB/s)(288MiB/30001msec) clat (usec): min=25, max=21580, avg=57.75, stdev=379.31 lat (usec): min=25, max=21580, avg=57.89, stdev=379.31 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 64], | 99.00th=[ 79], 99.50th=[ 89], 99.90th=[ 392], 99.95th=[12125], | 99.99th=[16319] bw ( KiB/s): min= 7920, max=14496, per=100.00%, avg=9826.13, stdev=1151.88, samples=60 iops : min= 1980, max= 3624, avg=2456.53, stdev=287.97, samples=60 write: IOPS=613, BW=2455KiB/s (2514kB/s)(71.9MiB/30001msec) clat (usec): min=193, max=45069, avg=1389.11, stdev=1606.16 lat (usec): min=193, max=45069, avg=1389.38, stdev=1606.16 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 594], 10.00th=[ 816], 20.00th=[ 1004], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1680], 95.00th=[ 1942], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[34341], 99.95th=[38011], | 99.99th=[43779] bw ( KiB/s): min= 2120, max= 3728, per=100.00%, avg=2454.80, stdev=253.96, samples=60 iops : min= 530, max= 932, avg=613.70, stdev=63.49, samples=60 lat (usec) : 50=56.74%, 100=23.04%, 250=0.25%, 500=0.67%, 750=0.92% lat (usec) : 1000=2.25% lat (msec) : 2=15.19%, 4=0.60%, 10=0.24%, 20=0.09%, 50=0.02% cpu : usr=1.33%, sys=3.86%, ctx=92114, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73696,18412,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9826KiB/s (10.1MB/s), 9826KiB/s-9826KiB/s (10.1MB/s-10.1MB/s), io=288MiB (302MB), run=30001-30001msec WRITE: bw=2455KiB/s (2514kB/s), 2455KiB/s-2455KiB/s (2514kB/s-2514kB/s), io=71.9MiB (75.4MB), run=30001-30001msec Disk stats (read/write): vdh: ios=73223/18300, merge=0/0, ticks=773/17720, in_queue=28190, util=94.03% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894963: Fri Jan 7 13:36:25 2022 read: IOPS=2500, BW=9.77MiB/s (10.2MB/s)(294MiB/30099msec) slat (usec): min=2, max=179, avg= 9.52, stdev= 3.83 clat (nsec): min=719, max=2533.0M, avg=936059.44, stdev=38241707.25 lat (usec): min=20, max=2533.0k, avg=945.73, stdev=38241.71 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 37], 60.00th=[ 40], 70.00th=[ 42], | 80.00th=[ 47], 90.00th=[ 58], 95.00th=[ 74], | 99.00th=[ 159], 99.50th=[ 293], 99.90th=[ 1369], | 99.95th=[ 700449], 99.99th=[2499806] bw ( KiB/s): min= 7608, max=17432, per=100.00%, avg=10034.57, stdev=1371.01, samples=60 iops : min= 1902, max= 4358, avg=2508.62, stdev=342.75, samples=60 write: IOPS=623, BW=2495KiB/s (2555kB/s)(73.3MiB/30099msec) slat (usec): min=2, max=183, avg=10.80, stdev= 4.29 clat (msec): min=6, max=2300, avg=76.29, stdev=69.15 lat (msec): min=6, max=2300, avg=76.30, stdev=69.15 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 54], 10.00th=[ 57], 20.00th=[ 61], | 30.00th=[ 64], 40.00th=[ 66], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 79], 90.00th=[ 89], 95.00th=[ 116], | 99.00th=[ 236], 99.50th=[ 518], 99.90th=[ 1053], 99.95th=[ 1351], | 99.99th=[ 2299] bw ( KiB/s): min= 2076, max= 4064, per=100.00%, avg=2497.73, stdev=281.17, samples=60 iops : min= 519, max= 1016, avg=624.43, stdev=70.29, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.11%, 50=67.57% lat (usec) : 100=10.57%, 250=1.22%, 500=0.28%, 750=0.07%, 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.17%, 20=0.10%, 50=0.36% lat (msec) : 100=17.88%, 250=1.28%, 500=0.09%, 750=0.05%, 1000=0.05% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=1.38%, sys=6.00%, ctx=90075, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=75262,18775,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9.77MiB/s (10.2MB/s), 9.77MiB/s-9.77MiB/s (10.2MB/s-10.2MB/s), io=294MiB (308MB), run=30099-30099msec WRITE: bw=2495KiB/s (2555kB/s), 2495KiB/s-2495KiB/s (2555kB/s-2555kB/s), io=73.3MiB (76.9MB), run=30099-30099msec Disk stats (read/write): vdh: ios=75257/18775, merge=0/0, ticks=61740/1420371, in_queue=1499478, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895779: Fri Jan 7 13:37:02 2022 read: IOPS=2470, BW=9883KiB/s (10.1MB/s)(290MiB/30001msec) clat (usec): min=25, max=22071, avg=58.49, stdev=386.99 lat (usec): min=25, max=22071, avg=58.63, stdev=386.99 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 59], 95.00th=[ 65], | 99.00th=[ 81], 99.50th=[ 92], 99.90th=[ 502], 99.95th=[11994], | 99.99th=[15926] bw ( KiB/s): min= 7711, max=15064, per=100.00%, avg=9885.27, stdev=1131.08, samples=59 iops : min= 1927, max= 3766, avg=2471.31, stdev=282.79, samples=59 write: IOPS=616, BW=2467KiB/s (2526kB/s)(72.3MiB/30001msec) clat (usec): min=197, max=57413, avg=1377.41, stdev=1607.64 lat (usec): min=197, max=57413, avg=1377.67, stdev=1607.64 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 578], 10.00th=[ 807], 20.00th=[ 1012], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1680], 95.00th=[ 1926], | 99.00th=[ 9110], 99.50th=[ 9634], 99.90th=[35390], 99.95th=[39584], | 99.99th=[52167] bw ( KiB/s): min= 2100, max= 3880, per=100.00%, avg=2470.44, stdev=245.89, samples=59 iops : min= 525, max= 970, avg=617.61, stdev=61.47, samples=59 lat (usec) : 50=54.96%, 100=24.79%, 250=0.28%, 500=0.73%, 750=0.89% lat (usec) : 1000=2.16% lat (msec) : 2=15.28%, 4=0.59%, 10=0.20%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.44%, sys=3.84%, ctx=92633, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74124,18505,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9883KiB/s (10.1MB/s), 9883KiB/s-9883KiB/s (10.1MB/s-10.1MB/s), io=290MiB (304MB), run=30001-30001msec WRITE: bw=2467KiB/s (2526kB/s), 2467KiB/s-2467KiB/s (2526kB/s-2526kB/s), io=72.3MiB (75.8MB), run=30001-30001msec Disk stats (read/write): vdh: ios=73613/18391, merge=0/0, ticks=795/17636, in_queue=28114, util=93.79% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896939: Fri Jan 7 13:37:39 2022 read: IOPS=2511, BW=9.81MiB/s (10.3MB/s)(296MiB/30173msec) slat (usec): min=2, max=190, avg= 9.43, stdev= 4.28 clat (nsec): min=683, max=2514.8M, avg=963760.34, stdev=38315714.81 lat (usec): min=23, max=2514.8k, avg=973.33, stdev=38315.71 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 49], 90.00th=[ 65], 95.00th=[ 124], | 99.00th=[ 873], 99.50th=[ 1827], 99.90th=[ 4555], | 99.95th=[ 633340], 99.99th=[2499806] bw ( KiB/s): min= 7688, max=19816, per=100.00%, avg=10102.53, stdev=1625.93, samples=60 iops : min= 1922, max= 4954, avg=2525.63, stdev=406.48, samples=60 write: IOPS=626, BW=2506KiB/s (2566kB/s)(73.8MiB/30173msec) slat (usec): min=2, max=365, avg=10.71, stdev= 5.38 clat (msec): min=7, max=2472, avg=155.64, stdev=85.18 lat (msec): min=7, max=2472, avg=155.65, stdev=85.18 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 114], 10.00th=[ 124], 20.00th=[ 133], | 30.00th=[ 138], 40.00th=[ 142], 50.00th=[ 146], 60.00th=[ 150], | 70.00th=[ 155], 80.00th=[ 163], 90.00th=[ 188], 95.00th=[ 209], | 99.00th=[ 451], 99.50th=[ 793], 99.90th=[ 1234], 99.95th=[ 1586], | 99.99th=[ 2333] bw ( KiB/s): min= 2088, max= 4328, per=100.00%, avg=2508.00, stdev=291.70, samples=60 iops : min= 522, max= 1082, avg=627.00, stdev=72.92, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.15%, 50=65.06% lat (usec) : 100=9.75%, 250=2.90%, 500=0.73%, 750=0.37%, 1000=0.27% lat (msec) : 2=0.32%, 4=0.20%, 10=0.18%, 20=0.18%, 50=0.12% lat (msec) : 100=0.24%, 250=18.90%, 500=0.31%, 750=0.07%, 1000=0.07% lat (msec) : 2000=0.06%, >=2000=0.02% cpu : usr=1.49%, sys=5.75%, ctx=88290, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75778,18901,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9.81MiB/s (10.3MB/s), 9.81MiB/s-9.81MiB/s (10.3MB/s-10.3MB/s), io=296MiB (310MB), run=30173-30173msec WRITE: bw=2506KiB/s (2566kB/s), 2506KiB/s-2506KiB/s (2566kB/s-2566kB/s), io=73.8MiB (77.4MB), run=30173-30173msec Disk stats (read/write): vdh: ios=75769/18858, merge=0/0, ticks=57446/2908707, in_queue=3002481, util=99.48% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897388: Fri Jan 7 13:38:16 2022 read: IOPS=2485, BW=9941KiB/s (10.2MB/s)(291MiB/30001msec) clat (usec): min=24, max=22191, avg=61.11, stdev=395.77 lat (usec): min=24, max=22191, avg=61.25, stdev=395.77 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 515], 99.95th=[11863], | 99.99th=[16581] bw ( KiB/s): min= 7824, max=14920, per=100.00%, avg=9941.22, stdev=1146.48, samples=60 iops : min= 1956, max= 3730, avg=2485.30, stdev=286.62, samples=60 write: IOPS=619, BW=2479KiB/s (2539kB/s)(72.6MiB/30001msec) clat (usec): min=186, max=51867, avg=1358.03, stdev=1580.05 lat (usec): min=187, max=51868, avg=1358.33, stdev=1580.05 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 553], 10.00th=[ 783], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1467], 90.00th=[ 1663], 95.00th=[ 1926], | 99.00th=[ 8717], 99.50th=[ 9634], 99.90th=[35914], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min= 1944, max= 3840, per=100.00%, avg=2479.40, stdev=266.92, samples=60 iops : min= 486, max= 960, avg=619.85, stdev=66.73, samples=60 lat (usec) : 50=47.29%, 100=32.33%, 250=0.40%, 500=0.79%, 750=0.98% lat (usec) : 1000=2.32% lat (msec) : 2=14.99%, 4=0.60%, 10=0.19%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.43%, sys=4.19%, ctx=93161, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74557,18596,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9941KiB/s (10.2MB/s), 9941KiB/s-9941KiB/s (10.2MB/s-10.2MB/s), io=291MiB (305MB), run=30001-30001msec WRITE: bw=2479KiB/s (2539kB/s), 2479KiB/s-2479KiB/s (2539kB/s-2539kB/s), io=72.6MiB (76.2MB), run=30001-30001msec Disk stats (read/write): vdh: ios=74158/18515, merge=0/0, ticks=817/17313, in_queue=27997, util=93.38% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898158: Fri Jan 7 13:38:53 2022 read: IOPS=2488, BW=9955KiB/s (10.2MB/s)(294MiB/30202msec) slat (usec): min=2, max=200195, avg=321.93, stdev=2128.56 clat (usec): min=471, max=2594.9k, avg=25875.44, stdev=40089.62 lat (usec): min=475, max=2594.9k, avg=26197.50, stdev=40154.55 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 8455], 10.00th=[ 14091], | 20.00th=[ 17433], 30.00th=[ 19530], 40.00th=[ 21365], | 50.00th=[ 23200], 60.00th=[ 25297], 70.00th=[ 27132], | 80.00th=[ 30016], 90.00th=[ 34866], 95.00th=[ 40633], | 99.00th=[ 82314], 99.50th=[ 185598], 99.90th=[ 223347], | 99.95th=[ 666895], 99.99th=[2164261] bw ( KiB/s): min= 8224, max=17984, per=100.00%, avg=10013.77, stdev=1423.89, samples=60 iops : min= 2056, max= 4496, avg=2503.43, stdev=355.98, samples=60 write: IOPS=620, BW=2484KiB/s (2543kB/s)(73.2MiB/30202msec) slat (usec): min=2, max=56641, avg=299.73, stdev=1417.24 clat (msec): min=4, max=2495, avg=216.63, stdev=109.95 lat (msec): min=4, max=2495, avg=216.93, stdev=109.95 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 161], 10.00th=[ 176], 20.00th=[ 186], | 30.00th=[ 192], 40.00th=[ 199], 50.00th=[ 203], 60.00th=[ 209], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 284], | 99.00th=[ 651], 99.50th=[ 969], 99.90th=[ 1687], 99.95th=[ 1888], | 99.99th=[ 2433] bw ( KiB/s): min= 2000, max= 3680, per=99.99%, avg=2482.75, stdev=252.47, samples=60 iops : min= 500, max= 920, avg=620.68, stdev=63.12, samples=60 lat (usec) : 500=0.01%, 750=0.39%, 1000=0.62% lat (msec) : 2=1.44%, 4=1.11%, 10=0.85%, 20=21.33%, 50=52.45% lat (msec) : 100=1.84%, 250=17.58%, 500=2.08%, 750=0.11%, 1000=0.07% lat (msec) : 2000=0.10%, >=2000=0.02% cpu : usr=0.67%, sys=1.97%, ctx=10358, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75169,18752,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9955KiB/s (10.2MB/s), 9955KiB/s-9955KiB/s (10.2MB/s-10.2MB/s), io=294MiB (308MB), run=30202-30202msec WRITE: bw=2484KiB/s (2543kB/s), 2484KiB/s-2484KiB/s (2543kB/s-2543kB/s), io=73.2MiB (76.8MB), run=30202-30202msec Disk stats (read/write): vdh: ios=75166/18684, merge=0/0, ticks=68353/3573349, in_queue=3669006, util=99.48% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898946: Fri Jan 7 13:39:30 2022 read: IOPS=2377, BW=37.1MiB/s (38.0MB/s)(1114MiB/30002msec) clat (usec): min=26, max=27218, avg=62.99, stdev=395.19 lat (usec): min=26, max=27218, avg=63.13, stdev=395.19 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 90], 99.50th=[ 102], 99.90th=[ 502], 99.95th=[12518], | 99.99th=[16188] bw ( KiB/s): min=27200, max=53952, per=100.00%, avg=38041.07, stdev=4228.44, samples=60 iops : min= 1700, max= 3372, avg=2377.57, stdev=264.28, samples=60 write: IOPS=594, BW=9512KiB/s (9741kB/s)(279MiB/30002msec) clat (usec): min=244, max=44134, avg=1419.97, stdev=1597.04 lat (usec): min=246, max=44135, avg=1421.42, stdev=1597.04 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 578], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1156], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 2008], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[38011], | 99.99th=[42730] bw ( KiB/s): min= 7744, max=14016, per=100.00%, avg=9512.53, stdev=944.52, samples=60 iops : min= 484, max= 876, avg=594.53, stdev=59.03, samples=60 lat (usec) : 50=41.60%, 100=37.95%, 250=0.35%, 500=0.75%, 750=0.95% lat (usec) : 1000=1.90% lat (msec) : 2=15.41%, 4=0.75%, 10=0.21%, 20=0.10%, 50=0.02% cpu : usr=1.35%, sys=3.99%, ctx=89174, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71327,17837,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.0MB/s), 37.1MiB/s-37.1MiB/s (38.0MB/s-38.0MB/s), io=1114MiB (1169MB), run=30002-30002msec WRITE: bw=9512KiB/s (9741kB/s), 9512KiB/s-9512KiB/s (9741kB/s-9741kB/s), io=279MiB (292MB), run=30002-30002msec Disk stats (read/write): vdh: ios=70998/17760, merge=0/0, ticks=756/17592, in_queue=28189, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899724: Fri Jan 7 13:40:07 2022 read: IOPS=2391, BW=37.4MiB/s (39.2MB/s)(1122MiB/30024msec) slat (usec): min=5, max=305, avg=10.31, stdev= 5.04 clat (nsec): min=673, max=19801k, avg=50249.51, stdev=400012.50 lat (usec): min=24, max=19809, avg=60.70, stdev=400.05 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 30], 10.00th=[ 31], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 37], 60.00th=[ 39], | 70.00th=[ 42], 80.00th=[ 44], 90.00th=[ 49], 95.00th=[ 54], | 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 375], 99.95th=[12780], | 99.99th=[17433] bw ( KiB/s): min=29248, max=53888, per=100.00%, avg=38296.00, stdev=3777.82, samples=60 iops : min= 1828, max= 3368, avg=2393.50, stdev=236.11, samples=60 write: IOPS=598, BW=9581KiB/s (9811kB/s)(281MiB/30024msec) slat (usec): min=8, max=466, avg=12.55, stdev= 5.89 clat (usec): min=97, max=48056, avg=1407.34, stdev=1615.43 lat (usec): min=246, max=48066, avg=1420.05, stdev=1615.44 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 594], 10.00th=[ 824], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 1991], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min= 8128, max=14016, per=100.00%, avg=9587.73, stdev=833.73, samples=60 iops : min= 508, max= 876, avg=599.23, stdev=52.11, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.09%, 50=73.24% lat (usec) : 100=6.32%, 250=0.11%, 500=0.73%, 750=0.88%, 1000=1.95% lat (msec) : 2=15.48%, 4=0.74%, 10=0.20%, 20=0.10%, 50=0.02% cpu : usr=1.24%, sys=6.21%, ctx=89644, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71805,17978,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.4MiB/s (39.2MB/s), 37.4MiB/s-37.4MiB/s (39.2MB/s-39.2MB/s), io=1122MiB (1176MB), run=30024-30024msec WRITE: bw=9581KiB/s (9811kB/s), 9581KiB/s-9581KiB/s (9811kB/s-9811kB/s), io=281MiB (295MB), run=30024-30024msec Disk stats (read/write): vdh: ios=71504/17899, merge=0/0, ticks=788/17768, in_queue=28127, util=93.81% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901188: Fri Jan 7 13:40:44 2022 read: IOPS=2395, BW=37.4MiB/s (39.3MB/s)(1123MiB/30002msec) clat (usec): min=26, max=60405, avg=61.58, stdev=449.24 lat (usec): min=27, max=60405, avg=61.72, stdev=449.24 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 97], 99.90th=[ 396], 99.95th=[12256], | 99.99th=[17433] bw ( KiB/s): min=29824, max=58496, per=100.00%, avg=38336.00, stdev=4056.35, samples=60 iops : min= 1864, max= 3656, avg=2396.00, stdev=253.52, samples=60 write: IOPS=599, BW=9597KiB/s (9827kB/s)(281MiB/30002msec) clat (usec): min=243, max=43319, avg=1411.24, stdev=1518.89 lat (usec): min=245, max=43320, avg=1412.70, stdev=1518.87 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 611], 10.00th=[ 848], 20.00th=[ 1057], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 2008], | 99.00th=[ 8717], 99.50th=[ 9634], 99.90th=[17433], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min= 8160, max=15200, per=100.00%, avg=9596.80, stdev=962.70, samples=60 iops : min= 510, max= 950, avg=599.80, stdev=60.17, samples=60 lat (usec) : 50=47.73%, 100=31.91%, 250=0.25%, 500=0.71%, 750=0.82% lat (usec) : 1000=1.77% lat (msec) : 2=15.72%, 4=0.79%, 10=0.19%, 20=0.08%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.43%, sys=3.85%, ctx=89880, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71880,17995,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.4MiB/s (39.3MB/s), 37.4MiB/s-37.4MiB/s (39.3MB/s-39.3MB/s), io=1123MiB (1178MB), run=30002-30002msec WRITE: bw=9597KiB/s (9827kB/s), 9597KiB/s-9597KiB/s (9827kB/s-9827kB/s), io=281MiB (295MB), run=30002-30002msec Disk stats (read/write): vdh: ios=71564/17917, merge=0/0, ticks=818/17713, in_queue=28227, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901831: Fri Jan 7 13:41:21 2022 read: IOPS=2437, BW=38.1MiB/s (39.9MB/s)(1144MiB/30043msec) slat (usec): min=2, max=135, avg=10.84, stdev= 3.97 clat (nsec): min=653, max=2491.2M, avg=771683.20, stdev=33994107.02 lat (usec): min=24, max=2491.2k, avg=782.68, stdev=33994.12 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 48], 90.00th=[ 54], 95.00th=[ 61], | 99.00th=[ 88], 99.50th=[ 119], 99.90th=[ 652], | 99.95th=[ 417334], 99.99th=[2466251] bw ( KiB/s): min=31616, max=69632, per=100.00%, avg=39060.55, stdev=5136.70, samples=60 iops : min= 1976, max= 4352, avg=2441.28, stdev=321.04, samples=60 write: IOPS=609, BW=9749KiB/s (9983kB/s)(286MiB/30043msec) slat (usec): min=3, max=124, avg=13.67, stdev= 4.29 clat (usec): min=404, max=77272, avg=13255.14, stdev=6047.76 lat (usec): min=412, max=77284, avg=13268.98, stdev=6048.16 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 6783], 10.00th=[ 8979], 20.00th=[10290], | 30.00th=[11338], 40.00th=[12125], 50.00th=[12780], 60.00th=[13304], | 70.00th=[13960], 80.00th=[14877], 90.00th=[17171], 95.00th=[21627], | 99.00th=[31327], 99.50th=[62653], 99.90th=[68682], 99.95th=[72877], | 99.99th=[76022] bw ( KiB/s): min= 7936, max=17472, per=100.00%, avg=9761.93, stdev=1188.81, samples=60 iops : min= 496, max= 1092, avg=610.12, stdev=74.30, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.07%, 50=67.94% lat (usec) : 100=11.34%, 250=0.40%, 500=0.06%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.27%, 4=0.20%, 10=2.90%, 20=15.28%, 50=1.19% lat (msec) : 100=0.16%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.39%, sys=6.48%, ctx=90037, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73243,18305,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=38.1MiB/s (39.9MB/s), 38.1MiB/s-38.1MiB/s (39.9MB/s-39.9MB/s), io=1144MiB (1200MB), run=30043-30043msec WRITE: bw=9749KiB/s (9983kB/s), 9749KiB/s-9749KiB/s (9983kB/s-9983kB/s), io=286MiB (300MB), run=30043-30043msec Disk stats (read/write): vdh: ios=72837/18204, merge=0/0, ticks=49303/232599, in_queue=296545, util=99.55% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902759: Fri Jan 7 13:41:58 2022 read: IOPS=2387, BW=37.3MiB/s (39.1MB/s)(1119MiB/30001msec) clat (usec): min=26, max=23826, avg=61.23, stdev=382.72 lat (usec): min=26, max=23827, avg=61.39, stdev=382.72 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 96], 99.90th=[ 310], 99.95th=[11338], | 99.99th=[16712] bw ( KiB/s): min=29600, max=59680, per=100.00%, avg=38201.60, stdev=4585.97, samples=60 iops : min= 1850, max= 3730, avg=2387.60, stdev=286.62, samples=60 write: IOPS=597, BW=9564KiB/s (9794kB/s)(280MiB/30001msec) clat (usec): min=238, max=43319, avg=1418.03, stdev=1571.04 lat (usec): min=239, max=43319, avg=1419.48, stdev=1571.03 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 594], 10.00th=[ 832], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 2008], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 7808, max=15328, per=100.00%, avg=9564.27, stdev=1050.32, samples=60 iops : min= 488, max= 958, avg=597.77, stdev=65.65, samples=60 lat (usec) : 50=45.30%, 100=34.35%, 250=0.25%, 500=0.74%, 750=0.84% lat (usec) : 1000=1.84% lat (msec) : 2=15.60%, 4=0.75%, 10=0.21%, 20=0.10%, 50=0.02% cpu : usr=1.69%, sys=3.65%, ctx=89569, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71628,17934,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.3MiB/s (39.1MB/s), 37.3MiB/s-37.3MiB/s (39.1MB/s-39.1MB/s), io=1119MiB (1174MB), run=30001-30001msec WRITE: bw=9564KiB/s (9794kB/s), 9564KiB/s-9564KiB/s (9794kB/s-9794kB/s), io=280MiB (294MB), run=30001-30001msec Disk stats (read/write): vdh: ios=71358/17847, merge=0/0, ticks=747/17683, in_queue=28151, util=93.90% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903775: Fri Jan 7 13:42:35 2022 read: IOPS=2418, BW=37.8MiB/s (39.6MB/s)(1138MiB/30108msec) slat (usec): min=2, max=172, avg= 9.90, stdev= 3.87 clat (nsec): min=831, max=2519.3M, avg=1001433.53, stdev=40933352.30 lat (usec): min=25, max=2519.3k, avg=1011.47, stdev=40933.33 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 48], 90.00th=[ 58], 95.00th=[ 73], | 99.00th=[ 184], 99.50th=[ 359], 99.90th=[ 996], | 99.95th=[ 700449], 99.99th=[2499806] bw ( KiB/s): min=32064, max=63200, per=100.00%, avg=38838.40, stdev=4759.78, samples=60 iops : min= 2004, max= 3950, avg=2427.40, stdev=297.50, samples=60 write: IOPS=604, BW=9673KiB/s (9906kB/s)(284MiB/30108msec) slat (usec): min=3, max=130, avg=12.91, stdev= 4.47 clat (msec): min=7, max=2479, avg=78.62, stdev=68.68 lat (msec): min=7, max=2479, avg=78.63, stdev=68.68 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 64], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 70], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 82], 90.00th=[ 92], 95.00th=[ 120], | 99.00th=[ 243], 99.50th=[ 510], 99.90th=[ 1028], 99.95th=[ 1368], | 99.99th=[ 1955] bw ( KiB/s): min= 8544, max=14656, per=100.00%, avg=9684.80, stdev=883.73, samples=60 iops : min= 534, max= 916, avg=605.30, stdev=55.23, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.04%, 10=0.01%, 20=0.08%, 50=66.88%, 100=11.23% lat (usec) : 250=1.15%, 500=0.31%, 750=0.13%, 1000=0.07% lat (msec) : 2=0.01%, 4=0.01%, 10=0.03%, 20=0.25%, 50=0.29% lat (msec) : 100=17.77%, 250=1.47%, 500=0.10%, 750=0.05%, 1000=0.04% lat (msec) : 2000=0.04%, >=2000=0.02% cpu : usr=1.41%, sys=5.78%, ctx=88376, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=72828,18203,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.6MB/s), 37.8MiB/s-37.8MiB/s (39.6MB/s-39.6MB/s), io=1138MiB (1193MB), run=30108-30108msec WRITE: bw=9673KiB/s (9906kB/s), 9673KiB/s-9673KiB/s (9906kB/s-9906kB/s), io=284MiB (298MB), run=30108-30108msec Disk stats (read/write): vdh: ios=72406/18076, merge=0/0, ticks=65155/1409982, in_queue=1491781, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904238: Fri Jan 7 13:43:12 2022 read: IOPS=2381, BW=37.2MiB/s (39.0MB/s)(1116MiB/30001msec) clat (usec): min=26, max=25206, avg=61.79, stdev=399.11 lat (usec): min=26, max=25206, avg=61.93, stdev=399.11 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 338], 99.95th=[12125], | 99.99th=[16188] bw ( KiB/s): min=28000, max=56384, per=100.00%, avg=38137.71, stdev=4547.17, samples=59 iops : min= 1750, max= 3524, avg=2383.59, stdev=284.20, samples=59 write: IOPS=595, BW=9534KiB/s (9763kB/s)(279MiB/30001msec) clat (usec): min=239, max=45334, avg=1421.14, stdev=1622.12 lat (usec): min=241, max=45336, avg=1422.72, stdev=1622.11 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 594], 10.00th=[ 824], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 1991], | 99.00th=[ 9110], 99.50th=[ 9634], 99.90th=[36439], 99.95th=[41681], | 99.99th=[42730] bw ( KiB/s): min= 6784, max=14400, per=100.00%, avg=9544.98, stdev=1020.15, samples=59 iops : min= 424, max= 900, avg=596.56, stdev=63.76, samples=59 lat (usec) : 50=45.28%, 100=34.32%, 250=0.30%, 500=0.72%, 750=0.88% lat (usec) : 1000=1.86% lat (msec) : 2=15.59%, 4=0.72%, 10=0.22%, 20=0.09%, 50=0.02% cpu : usr=1.53%, sys=3.81%, ctx=89326, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71434,17877,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1116MiB (1170MB), run=30001-30001msec WRITE: bw=9534KiB/s (9763kB/s), 9534KiB/s-9534KiB/s (9763kB/s-9763kB/s), io=279MiB (293MB), run=30001-30001msec Disk stats (read/write): vdh: ios=71126/17795, merge=0/0, ticks=777/17647, in_queue=28186, util=94.03% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905215: Fri Jan 7 13:43:49 2022 read: IOPS=2456, BW=38.4MiB/s (40.2MB/s)(1158MiB/30177msec) slat (usec): min=2, max=394, avg= 9.75, stdev= 5.02 clat (nsec): min=717, max=2543.6M, avg=1060673.28, stdev=43458429.34 lat (usec): min=23, max=2543.6k, avg=1070.56, stdev=43458.43 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 50], 90.00th=[ 67], 95.00th=[ 122], | 99.00th=[ 816], 99.50th=[ 1319], 99.90th=[ 2212], | 99.95th=[ 549454], 99.99th=[2499806] bw ( KiB/s): min=29216, max=79072, per=100.00%, avg=39531.10, stdev=6488.81, samples=60 iops : min= 1826, max= 4942, avg=2470.68, stdev=405.54, samples=60 write: IOPS=613, BW=9809KiB/s (10.0MB/s)(289MiB/30177msec) slat (usec): min=3, max=159, avg=12.78, stdev= 5.13 clat (msec): min=11, max=2395, avg=158.77, stdev=118.30 lat (msec): min=11, max=2395, avg=158.78, stdev=118.30 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 97], 10.00th=[ 116], 20.00th=[ 130], | 30.00th=[ 140], 40.00th=[ 144], 50.00th=[ 148], 60.00th=[ 153], | 70.00th=[ 157], 80.00th=[ 167], 90.00th=[ 194], 95.00th=[ 213], | 99.00th=[ 430], 99.50th=[ 978], 99.90th=[ 2056], 99.95th=[ 2089], | 99.99th=[ 2366] bw ( KiB/s): min= 7776, max=17216, per=100.00%, avg=9816.43, stdev=1203.72, samples=60 iops : min= 486, max= 1076, avg=613.52, stdev=75.23, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.06%, 4=0.01%, 10=0.04%, 20=0.14%, 50=64.02% lat (usec) : 100=10.70%, 250=2.85%, 500=0.82%, 750=0.45%, 1000=0.27% lat (msec) : 2=0.48%, 4=0.05%, 20=0.17%, 50=0.21%, 100=0.75% lat (msec) : 250=18.32%, 500=0.38%, 750=0.05%, 1000=0.05%, 2000=0.07% lat (msec) : >=2000=0.05% cpu : usr=1.32%, sys=5.78%, ctx=86634, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=74115,18501,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=38.4MiB/s (40.2MB/s), 38.4MiB/s-38.4MiB/s (40.2MB/s-40.2MB/s), io=1158MiB (1214MB), run=30177-30177msec WRITE: bw=9809KiB/s (10.0MB/s), 9809KiB/s-9809KiB/s (10.0MB/s-10.0MB/s), io=289MiB (303MB), run=30177-30177msec Disk stats (read/write): vdh: ios=74113/18476, merge=0/0, ticks=72085/2913879, in_queue=3005090, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905749: Fri Jan 7 13:44:26 2022 read: IOPS=2373, BW=37.1MiB/s (38.9MB/s)(1113MiB/30002msec) clat (usec): min=26, max=22548, avg=62.12, stdev=386.02 lat (usec): min=26, max=22548, avg=62.26, stdev=386.03 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 86], 99.50th=[ 97], 99.90th=[ 351], 99.95th=[11863], | 99.99th=[16057] bw ( KiB/s): min=29184, max=48224, per=100.00%, avg=37981.42, stdev=3859.12, samples=60 iops : min= 1824, max= 3014, avg=2373.80, stdev=241.19, samples=60 write: IOPS=593, BW=9502KiB/s (9730kB/s)(278MiB/30002msec) clat (usec): min=253, max=43567, avg=1425.84, stdev=1638.10 lat (usec): min=255, max=43569, avg=1427.29, stdev=1638.10 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 603], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 2008], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[35914], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 7264, max=12544, per=100.00%, avg=9503.70, stdev=762.96, samples=60 iops : min= 454, max= 784, avg=593.97, stdev=47.68, samples=60 lat (usec) : 50=43.60%, 100=36.03%, 250=0.26%, 500=0.71%, 750=0.96% lat (usec) : 1000=1.95% lat (msec) : 2=15.40%, 4=0.71%, 10=0.26%, 20=0.09%, 50=0.02% cpu : usr=1.52%, sys=3.76%, ctx=89024, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71201,17817,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1113MiB (1167MB), run=30002-30002msec WRITE: bw=9502KiB/s (9730kB/s), 9502KiB/s-9502KiB/s (9730kB/s-9730kB/s), io=278MiB (292MB), run=30002-30002msec Disk stats (read/write): vdh: ios=70929/17739, merge=0/0, ticks=742/17700, in_queue=28214, util=94.18% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906530: Fri Jan 7 13:45:03 2022 read: IOPS=2449, BW=38.3MiB/s (40.1MB/s)(1157MiB/30225msec) slat (usec): min=2, max=196988, avg=315.10, stdev=1750.08 clat (usec): min=643, max=2579.8k, avg=26479.12, stdev=45944.34 lat (usec): min=647, max=2579.8k, avg=26794.36, stdev=45983.40 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 8586], 10.00th=[ 14222], | 20.00th=[ 17695], 30.00th=[ 20055], 40.00th=[ 22152], | 50.00th=[ 23725], 60.00th=[ 25560], 70.00th=[ 27919], | 80.00th=[ 30540], 90.00th=[ 34866], 95.00th=[ 40109], | 99.00th=[ 84411], 99.50th=[ 185598], 99.90th=[ 240124], | 99.95th=[ 792724], 99.99th=[2533360] bw ( KiB/s): min=30240, max=77888, per=100.00%, avg=39453.92, stdev=6334.67, samples=60 iops : min= 1890, max= 4868, avg=2465.87, stdev=395.92, samples=60 write: IOPS=611, BW=9785KiB/s (10.0MB/s)(289MiB/30225msec) slat (usec): min=3, max=199097, avg=351.21, stdev=2894.04 clat (msec): min=5, max=3490, avg=219.29, stdev=156.71 lat (msec): min=12, max=3490, avg=219.64, stdev=156.68 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 201], 60.00th=[ 207], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 259], 95.00th=[ 288], | 99.00th=[ 869], 99.50th=[ 1167], 99.90th=[ 2500], 99.95th=[ 2869], | 99.99th=[ 3171] bw ( KiB/s): min= 7968, max=16320, per=100.00%, avg=9791.33, stdev=1134.97, samples=60 iops : min= 498, max= 1020, avg=611.95, stdev=70.94, samples=60 lat (usec) : 750=0.02%, 1000=0.42% lat (msec) : 2=1.58%, 4=1.31%, 10=1.08%, 20=19.57%, 50=54.00% lat (msec) : 100=1.95%, 250=17.39%, 500=2.31%, 750=0.10%, 1000=0.06% lat (msec) : 2000=0.15%, >=2000=0.06% cpu : usr=0.88%, sys=2.03%, ctx=10359, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=74042,18484,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=1157MiB (1213MB), run=30225-30225msec WRITE: bw=9785KiB/s (10.0MB/s), 9785KiB/s-9785KiB/s (10.0MB/s-10.0MB/s), io=289MiB (303MB), run=30225-30225msec Disk stats (read/write): vdh: ios=74034/18435, merge=0/0, ticks=73667/3563991, in_queue=3673143, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908056: Fri Jan 7 13:45:40 2022 read: IOPS=2286, BW=71.5MiB/s (74.9MB/s)(2144MiB/30002msec) clat (usec): min=31, max=26326, avg=66.19, stdev=420.29 lat (usec): min=31, max=26327, avg=66.33, stdev=420.30 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 89], 99.50th=[ 99], 99.90th=[ 412], 99.95th=[12518], | 99.99th=[17695] bw ( KiB/s): min=56896, max=109760, per=100.00%, avg=73189.98, stdev=8685.85, samples=60 iops : min= 1778, max= 3430, avg=2287.18, stdev=271.43, samples=60 write: IOPS=572, BW=17.9MiB/s (18.8MB/s)(537MiB/30002msec) clat (usec): min=270, max=50946, avg=1471.58, stdev=1621.97 lat (usec): min=273, max=50949, avg=1474.57, stdev=1621.95 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 611], 10.00th=[ 848], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2089], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min=15232, max=28288, per=100.00%, avg=18314.18, stdev=1911.99, samples=60 iops : min= 476, max= 884, avg=572.32, stdev=59.75, samples=60 lat (usec) : 50=33.70%, 100=45.91%, 250=0.28%, 500=0.62%, 750=0.95% lat (usec) : 1000=1.56% lat (msec) : 2=15.64%, 4=1.01%, 10=0.19%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.54%, sys=3.74%, ctx=85789, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68613,17170,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.5MiB/s (74.9MB/s), 71.5MiB/s-71.5MiB/s (74.9MB/s-74.9MB/s), io=2144MiB (2248MB), run=30002-30002msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=537MiB (563MB), run=30002-30002msec Disk stats (read/write): vdh: ios=68326/17095, merge=0/0, ticks=792/17528, in_queue=28171, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908670: Fri Jan 7 13:46:17 2022 read: IOPS=2284, BW=71.4MiB/s (74.9MB/s)(2142MiB/30001msec) slat (usec): min=5, max=174, avg=11.36, stdev= 4.06 clat (nsec): min=783, max=24722k, avg=53806.94, stdev=395279.76 lat (usec): min=27, max=24733, avg=65.32, stdev=395.32 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 45], 80.00th=[ 48], 90.00th=[ 53], 95.00th=[ 58], | 99.00th=[ 72], 99.50th=[ 81], 99.90th=[ 412], 99.95th=[12256], | 99.99th=[16450] bw ( KiB/s): min=57344, max=109952, per=100.00%, avg=73116.80, stdev=8424.95, samples=60 iops : min= 1792, max= 3436, avg=2284.90, stdev=263.28, samples=60 write: IOPS=571, BW=17.9MiB/s (18.7MB/s)(536MiB/30001msec) slat (usec): min=9, max=138, avg=15.04, stdev= 4.40 clat (usec): min=266, max=43746, avg=1464.96, stdev=1600.28 lat (usec): min=278, max=43759, avg=1480.16, stdev=1600.31 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 594], 10.00th=[ 848], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1811], 95.00th=[ 2089], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min=15552, max=28288, per=100.00%, avg=18299.73, stdev=1811.80, samples=60 iops : min= 486, max= 884, avg=571.87, stdev=56.62, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.05%, 50=67.56% lat (usec) : 100=12.09%, 250=0.09%, 500=0.65%, 750=0.91%, 1000=1.57% lat (msec) : 2=15.69%, 4=0.97%, 10=0.20%, 20=0.11%, 50=0.02% cpu : usr=1.31%, sys=6.21%, ctx=85623, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68547,17157,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.4MiB/s (74.9MB/s), 71.4MiB/s-71.4MiB/s (74.9MB/s-74.9MB/s), io=2142MiB (2246MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=536MiB (562MB), run=30001-30001msec Disk stats (read/write): vdh: ios=68266/17082, merge=0/0, ticks=748/17596, in_queue=28093, util=93.72% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909457: Fri Jan 7 13:46:54 2022 read: IOPS=2263, BW=70.7MiB/s (74.2MB/s)(2122MiB/30001msec) clat (usec): min=31, max=24066, avg=65.82, stdev=408.84 lat (usec): min=32, max=24066, avg=65.96, stdev=408.84 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 91], 99.50th=[ 104], 99.90th=[ 486], 99.95th=[12125], | 99.99th=[17171] bw ( KiB/s): min=50176, max=115968, per=100.00%, avg=72438.98, stdev=8873.48, samples=60 iops : min= 1568, max= 3624, avg=2263.70, stdev=277.30, samples=60 write: IOPS=566, BW=17.7MiB/s (18.6MB/s)(531MiB/30001msec) clat (usec): min=269, max=43410, avg=1491.53, stdev=1647.65 lat (usec): min=271, max=43413, avg=1494.42, stdev=1647.63 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 619], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1401], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1844], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[41681], | 99.99th=[43254] bw ( KiB/s): min=13952, max=29760, per=100.00%, avg=18124.43, stdev=2077.64, samples=60 iops : min= 436, max= 930, avg=566.38, stdev=64.92, samples=60 lat (usec) : 50=34.95%, 100=44.54%, 250=0.39%, 500=0.62%, 750=0.86% lat (usec) : 1000=1.47% lat (msec) : 2=15.81%, 4=1.00%, 10=0.21%, 20=0.12%, 50=0.02% cpu : usr=1.33%, sys=3.78%, ctx=84902, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67905,16991,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=70.7MiB/s (74.2MB/s), 70.7MiB/s-70.7MiB/s (74.2MB/s-74.2MB/s), io=2122MiB (2225MB), run=30001-30001msec WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=531MiB (557MB), run=30001-30001msec Disk stats (read/write): vdh: ios=67687/16920, merge=0/0, ticks=770/17669, in_queue=28269, util=94.28% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910041: Fri Jan 7 13:47:31 2022 read: IOPS=2304, BW=72.0MiB/s (75.5MB/s)(2163MiB/30028msec) slat (usec): min=3, max=141, avg=11.21, stdev= 3.91 clat (nsec): min=786, max=2493.3M, avg=900061.46, stdev=39106221.51 lat (usec): min=28, max=2493.3k, avg=911.42, stdev=39106.24 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 56], 95.00th=[ 62], | 99.00th=[ 92], 99.50th=[ 126], 99.90th=[ 1029], | 99.95th=[ 650118], 99.99th=[2499806] bw ( KiB/s): min=57856, max=116160, per=100.00%, avg=73818.27, stdev=8308.31, samples=60 iops : min= 1808, max= 3630, avg=2306.80, stdev=259.63, samples=60 write: IOPS=576, BW=18.0MiB/s (18.9MB/s)(541MiB/30028msec) slat (usec): min=5, max=138, avg=15.53, stdev= 4.73 clat (usec): min=513, max=78018, avg=13672.34, stdev=6134.98 lat (usec): min=526, max=78038, avg=13688.03, stdev=6135.25 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 7308], 10.00th=[ 8717], 20.00th=[10552], | 30.00th=[11731], 40.00th=[12518], 50.00th=[13173], 60.00th=[13698], | 70.00th=[14484], 80.00th=[15270], 90.00th=[17957], 95.00th=[21890], | 99.00th=[33817], 99.50th=[63177], 99.90th=[66847], 99.95th=[71828], | 99.99th=[78119] bw ( KiB/s): min=15808, max=29440, per=100.00%, avg=18454.53, stdev=1850.95, samples=60 iops : min= 494, max= 920, avg=576.70, stdev=57.84, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=65.30% lat (usec) : 100=13.98%, 250=0.49%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.08%, 4=0.25%, 10=2.98%, 20=15.21%, 50=1.34% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.59%, sys=5.91%, ctx=85344, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69201,17309,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=72.0MiB/s (75.5MB/s), 72.0MiB/s-72.0MiB/s (75.5MB/s-75.5MB/s), io=2163MiB (2268MB), run=30028-30028msec WRITE: bw=18.0MiB/s (18.9MB/s), 18.0MiB/s-18.0MiB/s (18.9MB/s-18.9MB/s), io=541MiB (567MB), run=30028-30028msec Disk stats (read/write): vdh: ios=68971/17229, merge=0/0, ticks=59066/226573, in_queue=296900, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910832: Fri Jan 7 13:48:08 2022 read: IOPS=2293, BW=71.7MiB/s (75.2MB/s)(2150MiB/30001msec) clat (usec): min=27, max=23831, avg=65.20, stdev=411.68 lat (usec): min=27, max=23831, avg=65.34, stdev=411.68 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 93], 99.50th=[ 109], 99.90th=[ 725], 99.95th=[12780], | 99.99th=[17171] bw ( KiB/s): min=49792, max=140288, per=100.00%, avg=73397.40, stdev=11432.86, samples=60 iops : min= 1556, max= 4384, avg=2293.67, stdev=357.28, samples=60 write: IOPS=573, BW=17.9MiB/s (18.8MB/s)(538MiB/30001msec) clat (usec): min=269, max=43486, avg=1470.99, stdev=1640.18 lat (usec): min=272, max=43488, avg=1474.04, stdev=1640.15 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 553], 10.00th=[ 799], 20.00th=[ 1057], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2089], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min=14592, max=36352, per=100.00%, avg=18353.62, stdev=2760.22, samples=60 iops : min= 456, max= 1136, avg=573.55, stdev=86.26, samples=60 lat (usec) : 50=37.74%, 100=41.71%, 250=0.42%, 500=0.87%, 750=0.94% lat (usec) : 1000=1.59% lat (msec) : 2=15.41%, 4=0.96%, 10=0.23%, 20=0.11%, 50=0.03% cpu : usr=1.53%, sys=3.95%, ctx=86032, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68809,17206,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=71.7MiB/s (75.2MB/s), 71.7MiB/s-71.7MiB/s (75.2MB/s-75.2MB/s), io=2150MiB (2255MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=538MiB (564MB), run=30001-30001msec Disk stats (read/write): vdh: ios=68496/17137, merge=0/0, ticks=788/17475, in_queue=28165, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911949: Fri Jan 7 13:48:45 2022 read: IOPS=2313, BW=72.3MiB/s (75.8MB/s)(2176MiB/30101msec) slat (usec): min=3, max=161, avg=10.98, stdev= 4.31 clat (nsec): min=702, max=2570.6M, avg=1102622.76, stdev=44438795.72 lat (usec): min=26, max=2570.6k, avg=1113.75, stdev=44438.77 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 35], 30.00th=[ 37], 40.00th=[ 38], | 50.00th=[ 41], 60.00th=[ 44], 70.00th=[ 47], | 80.00th=[ 51], 90.00th=[ 60], 95.00th=[ 72], | 99.00th=[ 225], 99.50th=[ 465], 99.90th=[ 1450], | 99.95th=[ 767558], 99.99th=[2499806] bw ( KiB/s): min=58613, max=139776, per=100.00%, avg=74278.53, stdev=10758.36, samples=60 iops : min= 1831, max= 4368, avg=2321.18, stdev=336.22, samples=60 write: IOPS=578, BW=18.1MiB/s (18.0MB/s)(544MiB/30101msec) slat (usec): min=5, max=116, avg=16.33, stdev= 5.53 clat (msec): min=7, max=2054, avg=81.93, stdev=65.06 lat (msec): min=7, max=2054, avg=81.95, stdev=65.06 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 92], 95.00th=[ 122], | 99.00th=[ 259], 99.50th=[ 527], 99.90th=[ 1003], 99.95th=[ 1150], | 99.99th=[ 1720] bw ( KiB/s): min=15519, max=32832, per=100.00%, avg=18520.60, stdev=2216.29, samples=60 iops : min= 484, max= 1026, avg=578.75, stdev=69.28, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.07%, 50=62.87% lat (usec) : 100=15.11%, 250=1.13%, 500=0.36%, 750=0.14%, 1000=0.09% lat (msec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.25%, 50=0.17% lat (msec) : 100=18.06%, 250=1.30%, 500=0.10%, 750=0.06%, 1000=0.05% lat (msec) : 2000=0.03%, >=2000=0.02% cpu : usr=1.35%, sys=6.05%, ctx=84212, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=69634,17410,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=2176MiB (2282MB), run=30101-30101msec WRITE: bw=18.1MiB/s (18.0MB/s), 18.1MiB/s-18.1MiB/s (18.0MB/s-18.0MB/s), io=544MiB (570MB), run=30101-30101msec Disk stats (read/write): vdh: ios=69335/17286, merge=0/0, ticks=72203/1404348, in_queue=1491539, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912583: Fri Jan 7 13:49:22 2022 read: IOPS=2280, BW=71.3MiB/s (74.7MB/s)(2138MiB/30001msec) clat (usec): min=26, max=65840, avg=63.40, stdev=471.31 lat (usec): min=26, max=65840, avg=63.54, stdev=471.31 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 90], 99.50th=[ 104], 99.90th=[ 611], 99.95th=[12256], | 99.99th=[17433] bw ( KiB/s): min=56192, max=116160, per=100.00%, avg=73182.08, stdev=9067.13, samples=59 iops : min= 1756, max= 3630, avg=2286.93, stdev=283.35, samples=59 write: IOPS=570, BW=17.8MiB/s (18.7MB/s)(535MiB/30001msec) clat (usec): min=285, max=44382, avg=1487.59, stdev=1626.86 lat (usec): min=288, max=44386, avg=1490.56, stdev=1626.84 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 635], 10.00th=[ 889], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1319], 60.00th=[ 1401], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1827], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min=14272, max=29952, per=100.00%, avg=18293.63, stdev=1998.95, samples=59 iops : min= 446, max= 936, avg=571.66, stdev=62.47, samples=59 lat (usec) : 50=44.68%, 100=34.83%, 250=0.34%, 500=0.63%, 750=0.77% lat (usec) : 1000=1.52% lat (msec) : 2=15.90%, 4=0.96%, 10=0.22%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.46%, sys=3.76%, ctx=85539, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68409,17120,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=71.3MiB/s (74.7MB/s), 71.3MiB/s-71.3MiB/s (74.7MB/s-74.7MB/s), io=2138MiB (2242MB), run=30001-30001msec WRITE: bw=17.8MiB/s (18.7MB/s), 17.8MiB/s-17.8MiB/s (18.7MB/s-18.7MB/s), io=535MiB (561MB), run=30001-30001msec Disk stats (read/write): vdh: ios=68136/17054, merge=0/0, ticks=800/17793, in_queue=28224, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913407: Fri Jan 7 13:49:59 2022 read: IOPS=2329, BW=72.8MiB/s (76.3MB/s)(2200MiB/30222msec) slat (usec): min=2, max=897, avg=10.88, stdev= 7.23 clat (nsec): min=698, max=2611.1M, avg=1092999.32, stdev=43413464.69 lat (usec): min=25, max=2611.1k, avg=1104.03, stdev=43413.49 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 45], 70.00th=[ 49], | 80.00th=[ 53], 90.00th=[ 65], 95.00th=[ 94], | 99.00th=[ 791], 99.50th=[ 1614], 99.90th=[ 2671], | 99.95th=[ 616563], 99.99th=[2499806] bw ( KiB/s): min=62528, max=149120, per=100.00%, avg=75103.43, stdev=11565.38, samples=60 iops : min= 1954, max= 4660, avg=2346.97, stdev=361.42, samples=60 write: IOPS=582, BW=18.2MiB/s (19.1MB/s)(551MiB/30222msec) slat (usec): min=4, max=284, avg=17.35, stdev= 7.89 clat (msec): min=12, max=2230, avg=167.07, stdev=103.95 lat (msec): min=12, max=2230, avg=167.08, stdev=103.95 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 140], | 30.00th=[ 146], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 201], 95.00th=[ 215], | 99.00th=[ 472], 99.50th=[ 927], 99.90th=[ 1653], 99.95th=[ 1653], | 99.99th=[ 2232] bw ( KiB/s): min=15006, max=32000, per=100.00%, avg=18693.98, stdev=2165.33, samples=60 iops : min= 468, max= 1000, avg=584.17, stdev=67.69, samples=60 lat (nsec) : 750=0.01%, 1000=0.09% lat (usec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.11%, 50=58.95% lat (usec) : 100=16.97%, 250=1.58%, 500=0.87%, 750=0.45%, 1000=0.20% lat (msec) : 2=0.40%, 4=0.17%, 10=0.01%, 20=0.09%, 50=0.30% lat (msec) : 100=0.23%, 250=18.84%, 500=0.38%, 750=0.06%, 1000=0.05% lat (msec) : 2000=0.10%, >=2000=0.02% cpu : usr=1.67%, sys=5.89%, ctx=83058, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70408,17619,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=72.8MiB/s (76.3MB/s), 72.8MiB/s-72.8MiB/s (76.3MB/s-76.3MB/s), io=2200MiB (2307MB), run=30222-30222msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=551MiB (577MB), run=30222-30222msec Disk stats (read/write): vdh: ios=70405/17530, merge=0/0, ticks=68703/2907130, in_queue=3009603, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914685: Fri Jan 7 13:50:36 2022 read: IOPS=2283, BW=71.4MiB/s (74.8MB/s)(2141MiB/30001msec) clat (usec): min=27, max=18728, avg=62.55, stdev=393.67 lat (usec): min=27, max=18728, avg=62.69, stdev=393.67 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 87], 99.50th=[ 99], 99.90th=[ 594], 99.95th=[11731], | 99.99th=[17171] bw ( KiB/s): min=57216, max=95936, per=100.00%, avg=73087.60, stdev=7862.91, samples=60 iops : min= 1788, max= 2998, avg=2283.98, stdev=245.71, samples=60 write: IOPS=571, BW=17.9MiB/s (18.7MB/s)(536MiB/30001msec) clat (usec): min=277, max=67946, avg=1488.63, stdev=1690.16 lat (usec): min=280, max=67949, avg=1491.69, stdev=1690.13 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 635], 10.00th=[ 898], 20.00th=[ 1123], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1401], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1844], 95.00th=[ 2089], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[34341], 99.95th=[41157], | 99.99th=[51119] bw ( KiB/s): min=15104, max=24896, per=100.00%, avg=18285.43, stdev=1521.66, samples=60 iops : min= 472, max= 778, avg=571.42, stdev=47.55, samples=60 lat (usec) : 50=44.05%, 100=35.56%, 250=0.28%, 500=0.62%, 750=0.73% lat (usec) : 1000=1.37% lat (msec) : 2=16.04%, 4=1.01%, 10=0.20%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.41%, sys=3.84%, ctx=85668, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68517,17143,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=71.4MiB/s (74.8MB/s), 71.4MiB/s-71.4MiB/s (74.8MB/s-74.8MB/s), io=2141MiB (2245MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=536MiB (562MB), run=30001-30001msec Disk stats (read/write): vdh: ios=68197/17069, merge=0/0, ticks=753/17873, in_queue=28232, util=94.18% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915273: Fri Jan 7 13:51:13 2022 read: IOPS=2339, BW=73.1MiB/s (76.7MB/s)(2214MiB/30276msec) slat (usec): min=2, max=250977, avg=335.29, stdev=2112.29 clat (usec): min=713, max=2725.9k, avg=27797.67, stdev=52044.05 lat (usec): min=717, max=2725.9k, avg=28133.10, stdev=52099.97 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 6063], 10.00th=[ 15139], | 20.00th=[ 18744], 30.00th=[ 21103], 40.00th=[ 23200], | 50.00th=[ 25035], 60.00th=[ 26870], 70.00th=[ 28967], | 80.00th=[ 31851], 90.00th=[ 36439], 95.00th=[ 41681], | 99.00th=[ 81265], 99.50th=[ 214959], 99.90th=[ 291505], | 99.95th=[ 792724], 99.99th=[2533360] bw ( KiB/s): min=55808, max=146240, per=100.00%, avg=75489.88, stdev=11825.28, samples=60 iops : min= 1744, max= 4570, avg=2359.05, stdev=369.53, samples=60 write: IOPS=584, BW=18.3MiB/s (19.2MB/s)(553MiB/30276msec) slat (usec): min=4, max=207729, avg=342.69, stdev=2167.93 clat (msec): min=15, max=3738, avg=228.85, stdev=190.49 lat (msec): min=16, max=3738, avg=229.19, stdev=190.48 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 142], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 192], 40.00th=[ 199], 50.00th=[ 207], 60.00th=[ 213], | 70.00th=[ 222], 80.00th=[ 232], 90.00th=[ 259], 95.00th=[ 326], | 99.00th=[ 936], 99.50th=[ 1519], 99.90th=[ 2601], 99.95th=[ 3306], | 99.99th=[ 3742] bw ( KiB/s): min=13440, max=30144, per=100.00%, avg=18749.53, stdev=2200.28, samples=60 iops : min= 420, max= 942, avg=585.92, stdev=68.75, samples=60 lat (usec) : 750=0.01%, 1000=0.10% lat (msec) : 2=1.54%, 4=1.74%, 10=1.17%, 20=14.97%, 50=58.81% lat (msec) : 100=1.61%, 250=17.28%, 500=2.35%, 750=0.11%, 1000=0.10% lat (msec) : 2000=0.11%, >=2000=0.10% cpu : usr=1.07%, sys=1.95%, ctx=9960, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70836,17710,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=73.1MiB/s (76.7MB/s), 73.1MiB/s-73.1MiB/s (76.7MB/s-76.7MB/s), io=2214MiB (2321MB), run=30276-30276msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=553MiB (580MB), run=30276-30276msec Disk stats (read/write): vdh: ios=70835/17639, merge=0/0, ticks=89679/3550337, in_queue=3675874, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916222: Fri Jan 7 13:51:50 2022 read: IOPS=2105, BW=132MiB/s (138MB/s)(3949MiB/30004msec) clat (usec): min=38, max=22250, avg=77.12, stdev=419.70 lat (usec): min=39, max=22250, avg=77.26, stdev=419.70 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 89], | 99.00th=[ 118], 99.50th=[ 139], 99.90th=[ 1319], 99.95th=[12649], | 99.99th=[17171] bw ( KiB/s): min=104064, max=211072, per=100.00%, avg=134801.07, stdev=15358.94, samples=60 iops : min= 1626, max= 3298, avg=2106.27, stdev=239.98, samples=60 write: IOPS=529, BW=33.1MiB/s (34.7MB/s)(993MiB/30004msec) clat (usec): min=382, max=100547, avg=1567.81, stdev=1775.44 lat (usec): min=388, max=100554, avg=1573.94, stdev=1775.40 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 660], 10.00th=[ 865], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1582], 80.00th=[ 1713], 90.00th=[ 1958], 95.00th=[ 2245], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[41157], | 99.99th=[45876] bw ( KiB/s): min=25344, max=54528, per=100.00%, avg=33879.47, stdev=3643.90, samples=60 iops : min= 396, max= 852, avg=529.37, stdev=56.94, samples=60 lat (usec) : 50=4.60%, 100=73.43%, 250=1.70%, 500=0.44%, 750=1.09% lat (usec) : 1000=1.31% lat (msec) : 2=15.57%, 4=1.51%, 10=0.21%, 20=0.11%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=1.58%, sys=3.85%, ctx=79076, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63188,15882,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3949MiB (4141MB), run=30004-30004msec WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=993MiB (1041MB), run=30004-30004msec Disk stats (read/write): vdh: ios=62925/15825, merge=0/0, ticks=726/17035, in_queue=28148, util=93.92% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916799: Fri Jan 7 13:52:27 2022 read: IOPS=2112, BW=132MiB/s (138MB/s)(3960MiB/30001msec) slat (usec): min=6, max=143, avg=13.65, stdev= 4.52 clat (nsec): min=808, max=23752k, avg=64165.48, stdev=437070.06 lat (usec): min=37, max=23763, avg=77.98, stdev=437.13 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 88], 99.50th=[ 103], 99.90th=[ 979], 99.95th=[13435], | 99.99th=[18220] bw ( KiB/s): min=97024, max=192000, per=100.00%, avg=135196.20, stdev=15512.26, samples=59 iops : min= 1516, max= 3000, avg=2112.44, stdev=242.38, samples=59 write: IOPS=530, BW=33.1MiB/s (34.8MB/s)(994MiB/30001msec) slat (usec): min=11, max=122, avg=20.88, stdev= 6.16 clat (usec): min=348, max=45336, avg=1546.06, stdev=1620.80 lat (usec): min=368, max=45355, avg=1567.12, stdev=1620.86 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 635], 10.00th=[ 840], 20.00th=[ 1123], | 30.00th=[ 1254], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1467], | 70.00th=[ 1565], 80.00th=[ 1696], 90.00th=[ 1942], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[38536], | 99.99th=[43254] bw ( KiB/s): min=27392, max=49920, per=100.00%, avg=33965.56, stdev=3141.36, samples=59 iops : min= 428, max= 780, avg=530.71, stdev=49.08, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.02%, 50=43.86% lat (usec) : 100=35.57%, 250=0.31%, 500=0.47%, 750=1.11%, 1000=1.47% lat (msec) : 2=15.41%, 4=1.38%, 10=0.21%, 20=0.12%, 50=0.03% cpu : usr=1.58%, sys=6.38%, ctx=79260, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63366,15911,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3960MiB (4153MB), run=30001-30001msec WRITE: bw=33.1MiB/s (34.8MB/s), 33.1MiB/s-33.1MiB/s (34.8MB/s-34.8MB/s), io=994MiB (1043MB), run=30001-30001msec Disk stats (read/write): vdh: ios=63038/15851, merge=0/0, ticks=776/16858, in_queue=28009, util=93.43% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917782: Fri Jan 7 13:53:04 2022 read: IOPS=2122, BW=133MiB/s (139MB/s)(3980MiB/30002msec) clat (usec): min=32, max=23083, avg=69.84, stdev=426.80 lat (usec): min=32, max=23083, avg=69.97, stdev=426.80 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 102], 99.50th=[ 118], 99.90th=[ 816], 99.95th=[13566], | 99.99th=[17433] bw ( KiB/s): min=104448, max=211712, per=100.00%, avg=135833.58, stdev=15649.68, samples=60 iops : min= 1632, max= 3308, avg=2122.37, stdev=244.52, samples=60 write: IOPS=533, BW=33.3MiB/s (34.0MB/s)(1000MiB/30002msec) clat (usec): min=361, max=50040, avg=1581.81, stdev=1614.14 lat (usec): min=367, max=50056, avg=1588.19, stdev=1614.09 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 693], 10.00th=[ 947], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2245], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[18220], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min=28288, max=54912, per=100.00%, avg=34142.62, stdev=3626.03, samples=60 iops : min= 442, max= 858, avg=533.45, stdev=56.66, samples=60 lat (usec) : 50=25.23%, 100=53.82%, 250=0.72%, 500=0.48%, 750=0.80% lat (usec) : 1000=1.09% lat (msec) : 2=15.99%, 4=1.52%, 10=0.20%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.51%, sys=3.95%, ctx=79690, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63676,16006,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=3980MiB (4173MB), run=30002-30002msec WRITE: bw=33.3MiB/s (34.0MB/s), 33.3MiB/s-33.3MiB/s (34.0MB/s-34.0MB/s), io=1000MiB (1049MB), run=30002-30002msec Disk stats (read/write): vdh: ios=63453/15941, merge=0/0, ticks=764/17404, in_queue=28084, util=93.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918727: Fri Jan 7 13:53:41 2022 read: IOPS=2154, BW=135MiB/s (141MB/s)(4044MiB/30036msec) slat (usec): min=4, max=1453, avg=12.87, stdev= 7.47 clat (nsec): min=782, max=2518.1M, avg=1143384.00, stdev=44136773.16 lat (usec): min=32, max=2518.1k, avg=1156.41, stdev=44136.74 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 37], 10.00th=[ 39], | 20.00th=[ 41], 30.00th=[ 42], 40.00th=[ 44], | 50.00th=[ 46], 60.00th=[ 49], 70.00th=[ 52], | 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 94], 99.50th=[ 115], 99.90th=[ 586], | 99.95th=[ 918553], 99.99th=[2499806] bw ( KiB/s): min=111872, max=236800, per=100.00%, avg=138007.47, stdev=17708.83, samples=60 iops : min= 1748, max= 3700, avg=2156.37, stdev=276.70, samples=60 write: IOPS=540, BW=33.8MiB/s (35.4MB/s)(1015MiB/30036msec) slat (usec): min=8, max=264, avg=20.62, stdev= 7.01 clat (usec): min=1441, max=81990, avg=13854.77, stdev=6387.12 lat (usec): min=1455, max=82016, avg=13875.56, stdev=6387.59 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 5735], 10.00th=[ 7111], 20.00th=[10159], | 30.00th=[11731], 40.00th=[12780], 50.00th=[13698], 60.00th=[14484], | 70.00th=[15270], 80.00th=[16450], 90.00th=[18482], 95.00th=[22676], | 99.00th=[31065], 99.50th=[63701], 99.90th=[71828], 99.95th=[72877], | 99.99th=[82314] bw ( KiB/s): min=28544, max=59904, per=100.00%, avg=34645.33, stdev=3988.39, samples=60 iops : min= 446, max= 936, avg=541.33, stdev=62.31, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.02%, 50=51.74% lat (usec) : 100=27.45%, 250=0.51%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.37%, 10=3.47%, 20=14.68%, 50=1.39% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=1.50%, sys=6.23%, ctx=80032, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=64698,16242,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4044MiB (4240MB), run=30036-30036msec WRITE: bw=33.8MiB/s (35.4MB/s), 33.8MiB/s-33.8MiB/s (35.4MB/s-35.4MB/s), io=1015MiB (1064MB), run=30036-30036msec Disk stats (read/write): vdh: ios=64420/16174, merge=0/0, ticks=64483/216399, in_queue=297023, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919245: Fri Jan 7 13:54:18 2022 read: IOPS=2099, BW=131MiB/s (138MB/s)(3937MiB/30002msec) clat (usec): min=32, max=60868, avg=69.39, stdev=503.91 lat (usec): min=32, max=60868, avg=69.53, stdev=503.91 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 92], 99.50th=[ 105], 99.90th=[ 750], 99.95th=[14091], | 99.99th=[18220] bw ( KiB/s): min=106112, max=199936, per=100.00%, avg=134372.27, stdev=15815.79, samples=60 iops : min= 1658, max= 3124, avg=2099.53, stdev=247.13, samples=60 write: IOPS=527, BW=32.0MiB/s (34.6MB/s)(990MiB/30002msec) clat (usec): min=359, max=46286, avg=1603.94, stdev=1606.94 lat (usec): min=365, max=46289, avg=1609.97, stdev=1606.92 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 701], 10.00th=[ 979], 20.00th=[ 1221], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1450], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[29230], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=26368, max=52352, per=100.00%, avg=33787.73, stdev=3550.91, samples=60 iops : min= 412, max= 818, avg=527.93, stdev=55.48, samples=60 lat (usec) : 50=28.63%, 100=50.79%, 250=0.39%, 500=0.45%, 750=0.72% lat (usec) : 1000=0.98% lat (msec) : 2=15.98%, 4=1.70%, 10=0.20%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.55%, sys=3.67%, ctx=78835, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62987,15839,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3937MiB (4128MB), run=30002-30002msec WRITE: bw=32.0MiB/s (34.6MB/s), 32.0MiB/s-32.0MiB/s (34.6MB/s-34.6MB/s), io=990MiB (1038MB), run=30002-30002msec Disk stats (read/write): vdh: ios=62768/15787, merge=0/0, ticks=820/17598, in_queue=28184, util=94.01% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920089: Fri Jan 7 13:54:55 2022 read: IOPS=2158, BW=135MiB/s (141MB/s)(4061MiB/30101msec) slat (usec): min=2, max=176, avg=10.68, stdev= 4.18 clat (nsec): min=729, max=2525.7M, avg=1147667.61, stdev=45989312.63 lat (usec): min=29, max=2525.7k, avg=1158.49, stdev=45989.32 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 43], 60.00th=[ 46], 70.00th=[ 50], | 80.00th=[ 55], 90.00th=[ 64], 95.00th=[ 74], | 99.00th=[ 289], 99.50th=[ 578], 99.90th=[ 1483], | 99.95th=[ 893387], 99.99th=[2499806] bw ( KiB/s): min=102144, max=244992, per=100.00%, avg=138596.63, stdev=18812.14, samples=60 iops : min= 1596, max= 3828, avg=2165.55, stdev=293.93, samples=60 write: IOPS=541, BW=33.8MiB/s (35.5MB/s)(1019MiB/30101msec) slat (usec): min=6, max=107, avg=21.20, stdev= 7.91 clat (msec): min=4, max=2594, avg=87.68, stdev=82.23 lat (msec): min=4, max=2595, avg=87.70, stdev=82.23 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 62], 10.00th=[ 66], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 79], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 131], | 99.00th=[ 247], 99.50th=[ 592], 99.90th=[ 1167], 99.95th=[ 2165], | 99.99th=[ 2500] bw ( KiB/s): min=26880, max=56960, per=100.00%, avg=34681.68, stdev=3764.46, samples=60 iops : min= 420, max= 890, avg=541.88, stdev=58.82, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.03%, 50=56.23% lat (usec) : 100=21.86%, 250=0.85%, 500=0.45%, 750=0.14%, 1000=0.08% lat (msec) : 2=0.16%, 10=0.03%, 20=0.30%, 50=0.18%, 100=17.81% lat (msec) : 250=1.57%, 500=0.09%, 750=0.06%, 1000=0.03%, 2000=0.04% lat (msec) : >=2000=0.03% cpu : usr=1.79%, sys=5.31%, ctx=78945, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=64973,16300,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4061MiB (4258MB), run=30101-30101msec WRITE: bw=33.8MiB/s (35.5MB/s), 33.8MiB/s-33.8MiB/s (35.5MB/s-35.5MB/s), io=1019MiB (1068MB), run=30101-30101msec Disk stats (read/write): vdh: ios=64707/16203, merge=0/0, ticks=65005/1409761, in_queue=1492381, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921287: Fri Jan 7 13:55:32 2022 read: IOPS=2098, BW=131MiB/s (138MB/s)(3936MiB/30004msec) clat (usec): min=31, max=59542, avg=71.24, stdev=501.16 lat (usec): min=31, max=59542, avg=71.38, stdev=501.16 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 96], 99.50th=[ 111], 99.90th=[ 799], 99.95th=[14353], | 99.99th=[17957] bw ( KiB/s): min=106752, max=203520, per=100.00%, avg=134346.98, stdev=15455.33, samples=60 iops : min= 1668, max= 3180, avg=2099.17, stdev=241.49, samples=60 write: IOPS=527, BW=32.0MiB/s (34.6MB/s)(990MiB/30004msec) clat (usec): min=359, max=53268, avg=1596.78, stdev=1651.60 lat (usec): min=366, max=53274, avg=1602.85, stdev=1651.56 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 676], 10.00th=[ 955], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2245], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[40633], | 99.99th=[51119] bw ( KiB/s): min=27136, max=52736, per=100.00%, avg=33780.30, stdev=3425.68, samples=60 iops : min= 424, max= 824, avg=527.82, stdev=53.53, samples=60 lat (usec) : 50=23.24%, 100=56.01%, 250=0.54%, 500=0.48%, 750=0.80% lat (usec) : 1000=1.02% lat (msec) : 2=15.94%, 4=1.61%, 10=0.21%, 20=0.13%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.45%, sys=3.92%, ctx=78818, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62973,15835,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3936MiB (4127MB), run=30004-30004msec WRITE: bw=32.0MiB/s (34.6MB/s), 32.0MiB/s-32.0MiB/s (34.6MB/s-34.6MB/s), io=990MiB (1038MB), run=30004-30004msec Disk stats (read/write): vdh: ios=62720/15772, merge=0/0, ticks=835/17474, in_queue=28237, util=94.20% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922071: Fri Jan 7 13:56:09 2022 read: IOPS=2157, BW=135MiB/s (141MB/s)(4073MiB/30212msec) slat (usec): min=2, max=239, avg=11.10, stdev= 5.31 clat (nsec): min=765, max=2585.8M, avg=1432547.64, stdev=52903954.38 lat (usec): min=28, max=2585.8k, avg=1443.81, stdev=52903.95 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 45], 60.00th=[ 49], 70.00th=[ 54], | 80.00th=[ 60], 90.00th=[ 73], 95.00th=[ 125], | 99.00th=[ 1549], 99.50th=[ 2474], 99.90th=[ 4080], | 99.95th=[1216349], 99.99th=[2499806] bw ( KiB/s): min=109568, max=290757, per=100.00%, avg=139025.80, stdev=23005.85, samples=60 iops : min= 1712, max= 4543, avg=2172.27, stdev=359.47, samples=60 write: IOPS=541, BW=33.8MiB/s (35.5MB/s)(1022MiB/30212msec) slat (usec): min=6, max=588, avg=21.43, stdev=10.53 clat (msec): min=3, max=3604, avg=179.02, stdev=143.42 lat (msec): min=3, max=3604, avg=179.04, stdev=143.42 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 118], 10.00th=[ 129], 20.00th=[ 142], | 30.00th=[ 159], 40.00th=[ 163], 50.00th=[ 167], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 184], 90.00th=[ 215], 95.00th=[ 243], | 99.00th=[ 460], 99.50th=[ 1036], 99.90th=[ 2668], 99.95th=[ 2769], | 99.99th=[ 3473] bw ( KiB/s): min=29312, max=62460, per=100.00%, avg=34678.23, stdev=4325.16, samples=60 iops : min= 458, max= 975, avg=541.82, stdev=67.50, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.04%, 50=50.23% lat (usec) : 100=24.98%, 250=1.69%, 500=0.88%, 750=0.51%, 1000=0.38% lat (msec) : 2=0.55%, 4=0.52%, 10=0.01%, 20=0.05%, 50=0.34% lat (msec) : 100=0.24%, 250=18.58%, 500=0.66%, 750=0.07%, 1000=0.03% lat (msec) : 2000=0.09%, >=2000=0.06% cpu : usr=1.76%, sys=5.44%, ctx=76396, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=65169,16346,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4073MiB (4271MB), run=30212-30212msec WRITE: bw=33.8MiB/s (35.5MB/s), 33.8MiB/s-33.8MiB/s (35.5MB/s-35.5MB/s), io=1022MiB (1071MB), run=30212-30212msec Disk stats (read/write): vdh: ios=65162/16300, merge=0/0, ticks=80722/2898607, in_queue=3006161, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922889: Fri Jan 7 13:56:46 2022 read: IOPS=2117, BW=132MiB/s (139MB/s)(3970MiB/30001msec) clat (usec): min=31, max=54135, avg=68.31, stdev=490.17 lat (usec): min=31, max=54135, avg=68.45, stdev=490.17 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 89], 99.50th=[ 100], 99.90th=[ 433], 99.95th=[13829], | 99.99th=[18482] bw ( KiB/s): min=108288, max=212992, per=100.00%, avg=135499.67, stdev=15462.65, samples=60 iops : min= 1692, max= 3328, avg=2117.17, stdev=241.58, samples=60 write: IOPS=532, BW=33.2MiB/s (34.9MB/s)(998MiB/30001msec) clat (usec): min=346, max=50060, avg=1593.35, stdev=1572.53 lat (usec): min=352, max=50067, avg=1599.69, stdev=1572.47 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 701], 10.00th=[ 988], 20.00th=[ 1221], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2212], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min=27264, max=55680, per=100.00%, avg=34049.22, stdev=3659.79, samples=60 iops : min= 426, max= 870, avg=532.02, stdev=57.18, samples=60 lat (usec) : 50=32.59%, 100=46.94%, 250=0.29%, 500=0.46%, 750=0.71% lat (usec) : 1000=0.95% lat (msec) : 2=16.14%, 4=1.59%, 10=0.18%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.52%, sys=3.63%, ctx=79482, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63513,15961,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3970MiB (4162MB), run=30001-30001msec WRITE: bw=33.2MiB/s (34.9MB/s), 33.2MiB/s-33.2MiB/s (34.9MB/s-34.9MB/s), io=998MiB (1046MB), run=30001-30001msec Disk stats (read/write): vdh: ios=63265/15894, merge=0/0, ticks=839/17530, in_queue=28276, util=94.30% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923638: Fri Jan 7 13:57:24 2022 read: IOPS=2167, BW=135MiB/s (142MB/s)(4106MiB/30301msec) slat (usec): min=2, max=258839, avg=362.54, stdev=2408.37 clat (usec): min=1006, max=2565.3k, avg=30037.81, stdev=54815.34 lat (usec): min=1010, max=2565.3k, avg=30400.51, stdev=54873.05 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 5866], 10.00th=[ 15926], | 20.00th=[ 20317], 30.00th=[ 22938], 40.00th=[ 25035], | 50.00th=[ 27132], 60.00th=[ 28967], 70.00th=[ 31589], | 80.00th=[ 34341], 90.00th=[ 39584], 95.00th=[ 45351], | 99.00th=[ 88605], 99.50th=[ 208667], 99.90th=[ 333448], | 99.95th=[1249903], 99.99th=[2533360] bw ( KiB/s): min=94720, max=292864, per=100.00%, avg=139997.75, stdev=23941.58, samples=60 iops : min= 1480, max= 4576, avg=2187.45, stdev=374.10, samples=60 write: IOPS=542, BW=33.9MiB/s (35.6MB/s)(1028MiB/30301msec) slat (usec): min=6, max=58622, avg=367.01, stdev=1693.05 clat (msec): min=15, max=3453, avg=246.53, stdev=251.68 lat (msec): min=16, max=3453, avg=246.90, stdev=251.65 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 130], 10.00th=[ 144], 20.00th=[ 165], | 30.00th=[ 194], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 234], | 70.00th=[ 243], 80.00th=[ 253], 90.00th=[ 284], 95.00th=[ 338], | 99.00th=[ 1653], 99.50th=[ 2333], 99.90th=[ 3138], 99.95th=[ 3440], | 99.99th=[ 3440] bw ( KiB/s): min=26752, max=60416, per=100.00%, avg=34821.28, stdev=4354.70, samples=60 iops : min= 418, max= 944, avg=544.07, stdev=68.06, samples=60 lat (msec) : 2=1.17%, 4=1.80%, 10=1.78%, 20=10.38%, 50=62.53% lat (msec) : 100=2.25%, 250=15.41%, 500=4.18%, 750=0.11%, 1000=0.05% lat (msec) : 2000=0.17%, >=2000=0.16% cpu : usr=1.67%, sys=2.01%, ctx=9173, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=65691,16453,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=4106MiB (4305MB), run=30301-30301msec WRITE: bw=33.9MiB/s (35.6MB/s), 33.9MiB/s-33.9MiB/s (35.6MB/s-35.6MB/s), io=1028MiB (1078MB), run=30301-30301msec Disk stats (read/write): vdh: ios=65686/16344, merge=0/0, ticks=87887/3541809, in_queue=3683269, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924559: Fri Jan 7 13:58:00 2022 read: IOPS=2031, BW=159MiB/s (166MB/s)(4761MiB/30001msec) clat (usec): min=36, max=66877, avg=79.77, stdev=517.93 lat (usec): min=36, max=66877, avg=79.92, stdev=517.93 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 89], | 99.00th=[ 119], 99.50th=[ 143], 99.90th=[ 8979], 99.95th=[13435], | 99.99th=[17695] bw ( KiB/s): min=129760, max=265920, per=100.00%, avg=162530.87, stdev=20992.26, samples=60 iops : min= 1622, max= 3324, avg=2031.62, stdev=262.40, samples=60 write: IOPS=510, BW=39.9MiB/s (41.8MB/s)(1196MiB/30001msec) clat (usec): min=401, max=94663, avg=1624.47, stdev=1756.39 lat (usec): min=409, max=94672, avg=1632.87, stdev=1756.43 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 693], 10.00th=[ 906], 20.00th=[ 1205], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2024], 95.00th=[ 2311], | 99.00th=[ 9372], 99.50th=[10028], 99.90th=[35390], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=34240, max=69440, per=100.00%, avg=40834.63, stdev=4751.01, samples=60 iops : min= 428, max= 868, avg=510.42, stdev=59.40, samples=60 lat (usec) : 50=4.74%, 100=73.28%, 250=1.69%, 500=0.34%, 750=1.09% lat (usec) : 1000=1.14% lat (msec) : 2=15.49%, 4=1.86%, 10=0.20%, 20=0.15%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.76%, sys=3.90%, ctx=76263, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60945,15313,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=4761MiB (4993MB), run=30001-30001msec WRITE: bw=39.9MiB/s (41.8MB/s), 39.9MiB/s-39.9MiB/s (41.8MB/s-41.8MB/s), io=1196MiB (1254MB), run=30001-30001msec Disk stats (read/write): vdh: ios=60676/15252, merge=0/0, ticks=879/16967, in_queue=28034, util=93.52% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925576: Fri Jan 7 13:58:37 2022 read: IOPS=2041, BW=160MiB/s (167MB/s)(4786MiB/30005msec) slat (usec): min=7, max=234, avg=14.78, stdev= 4.94 clat (nsec): min=844, max=57905k, avg=69090.71, stdev=491966.72 lat (usec): min=39, max=57921, avg=84.04, stdev=492.01 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 92], 99.50th=[ 105], 99.90th=[ 1516], 99.95th=[13435], | 99.99th=[17433] bw ( KiB/s): min=122560, max=262880, per=100.00%, avg=163362.67, stdev=19337.22, samples=60 iops : min= 1532, max= 3286, avg=2042.03, stdev=241.72, samples=60 write: IOPS=512, BW=40.1MiB/s (42.0MB/s)(1202MiB/30005msec) slat (usec): min=12, max=403, avg=23.02, stdev= 6.84 clat (usec): min=399, max=44637, avg=1584.21, stdev=1589.74 lat (usec): min=417, max=44663, avg=1607.42, stdev=1589.68 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 668], 10.00th=[ 857], 20.00th=[ 1156], | 30.00th=[ 1287], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2311], | 99.00th=[ 9110], 99.50th=[ 9896], 99.90th=[18220], 99.95th=[39584], | 99.99th=[44303] bw ( KiB/s): min=32800, max=67840, per=100.00%, avg=41032.00, stdev=4510.54, samples=60 iops : min= 410, max= 848, avg=512.90, stdev=56.38, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=31.64% lat (usec) : 100=47.71%, 250=0.38%, 500=0.36%, 750=1.15%, 1000=1.30% lat (msec) : 2=15.33%, 4=1.71%, 10=0.20%, 20=0.13%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.56%, sys=6.54%, ctx=76630, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61261,15388,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=160MiB/s (167MB/s), 160MiB/s-160MiB/s (167MB/s-167MB/s), io=4786MiB (5019MB), run=30005-30005msec WRITE: bw=40.1MiB/s (42.0MB/s), 40.1MiB/s-40.1MiB/s (42.0MB/s-42.0MB/s), io=1202MiB (1261MB), run=30005-30005msec Disk stats (read/write): vdh: ios=61013/15330, merge=0/0, ticks=837/16659, in_queue=27863, util=92.95% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926051: Fri Jan 7 13:59:14 2022 read: IOPS=2046, BW=160MiB/s (168MB/s)(4797MiB/30001msec) clat (usec): min=32, max=25229, avg=74.28, stdev=464.60 lat (usec): min=32, max=25230, avg=74.42, stdev=464.60 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 99], 99.50th=[ 114], 99.90th=[ 2999], 99.95th=[13698], | 99.99th=[18744] bw ( KiB/s): min=124160, max=265920, per=100.00%, avg=163759.02, stdev=20107.02, samples=60 iops : min= 1552, max= 3324, avg=2046.97, stdev=251.36, samples=60 write: IOPS=514, BW=40.2MiB/s (42.1MB/s)(1206MiB/30001msec) clat (usec): min=395, max=44978, avg=1632.56, stdev=1613.48 lat (usec): min=402, max=44985, avg=1640.14, stdev=1613.41 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 709], 10.00th=[ 955], 20.00th=[ 1237], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[40109], | 99.99th=[44303] bw ( KiB/s): min=32960, max=69120, per=100.00%, avg=41149.20, stdev=4744.61, samples=60 iops : min= 412, max= 864, avg=514.33, stdev=59.33, samples=60 lat (usec) : 50=13.75%, 100=65.41%, 250=0.64%, 500=0.30%, 750=0.93% lat (usec) : 1000=1.01% lat (msec) : 2=15.69%, 4=1.91%, 10=0.20%, 20=0.13%, 50=0.03% cpu : usr=1.67%, sys=3.83%, ctx=76853, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61406,15431,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=4797MiB (5030MB), run=30001-30001msec WRITE: bw=40.2MiB/s (42.1MB/s), 40.2MiB/s-40.2MiB/s (42.1MB/s-42.1MB/s), io=1206MiB (1264MB), run=30001-30001msec Disk stats (read/write): vdh: ios=61253/15383, merge=0/0, ticks=824/17189, in_queue=28138, util=93.86% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926944: Fri Jan 7 13:59:51 2022 read: IOPS=2064, BW=161MiB/s (169MB/s)(4846MiB/30042msec) slat (usec): min=4, max=116, avg=13.17, stdev= 4.30 clat (nsec): min=813, max=2535.7M, avg=1480469.42, stdev=55272778.03 lat (usec): min=32, max=2535.7k, avg=1493.80, stdev=55272.76 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 36], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 51], | 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 70], | 99.00th=[ 94], 99.50th=[ 117], 99.90th=[ 18220], | 99.95th=[1417675], 99.99th=[2499806] bw ( KiB/s): min=136480, max=298560, per=100.00%, avg=165424.98, stdev=22297.95, samples=60 iops : min= 1706, max= 3732, avg=2067.78, stdev=278.71, samples=60 write: IOPS=518, BW=40.5MiB/s (42.5MB/s)(1217MiB/30042msec) slat (usec): min=8, max=180, avg=22.44, stdev= 6.47 clat (usec): min=586, max=74195, avg=13300.12, stdev=6775.72 lat (usec): min=604, max=74226, avg=13322.75, stdev=6776.08 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 4948], 10.00th=[ 6521], 20.00th=[ 8455], | 30.00th=[10421], 40.00th=[11863], 50.00th=[13042], 60.00th=[14222], | 70.00th=[15401], 80.00th=[16450], 90.00th=[18482], 95.00th=[22414], | 99.00th=[34341], 99.50th=[62653], 99.90th=[69731], 99.95th=[71828], | 99.99th=[72877] bw ( KiB/s): min=32800, max=75520, per=100.00%, avg=41530.97, stdev=5466.82, samples=60 iops : min= 410, max= 944, avg=519.13, stdev=68.33, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.02%, 50=54.40% lat (usec) : 100=24.82%, 250=0.48%, 500=0.05%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.10%, 4=0.53%, 10=4.87%, 20=13.12%, 50=1.31% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.60%, sys=6.05%, ctx=76944, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62032,15581,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4846MiB (5082MB), run=30042-30042msec WRITE: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=1217MiB (1276MB), run=30042-30042msec Disk stats (read/write): vdh: ios=61870/15515, merge=0/0, ticks=88267/198129, in_queue=297101, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927499: Fri Jan 7 14:00:28 2022 read: IOPS=2031, BW=159MiB/s (166MB/s)(4762MiB/30001msec) clat (usec): min=33, max=25717, avg=72.50, stdev=481.90 lat (usec): min=33, max=25717, avg=72.64, stdev=481.90 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 95], 99.50th=[ 111], 99.90th=[ 7373], 99.95th=[14222], | 99.99th=[17957] bw ( KiB/s): min=125600, max=255230, per=100.00%, avg=162655.03, stdev=20429.55, samples=59 iops : min= 1570, max= 3190, avg=2033.15, stdev=255.35, samples=59 write: IOPS=510, BW=39.9MiB/s (41.8MB/s)(1196MiB/30001msec) clat (usec): min=409, max=60678, avg=1654.31, stdev=1678.17 lat (usec): min=416, max=60683, avg=1661.95, stdev=1678.12 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 725], 10.00th=[ 996], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[36439], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=33600, max=66052, per=100.00%, avg=40852.37, stdev=4334.49, samples=59 iops : min= 420, max= 825, avg=510.61, stdev=54.14, samples=59 lat (usec) : 50=21.89%, 100=57.43%, 250=0.48%, 500=0.31%, 750=0.81% lat (usec) : 1000=0.98% lat (msec) : 2=15.74%, 4=1.98%, 10=0.19%, 20=0.17%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.44%, sys=3.80%, ctx=76281, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60952,15315,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=4762MiB (4993MB), run=30001-30001msec WRITE: bw=39.9MiB/s (41.8MB/s), 39.9MiB/s-39.9MiB/s (41.8MB/s-41.8MB/s), io=1196MiB (1255MB), run=30001-30001msec Disk stats (read/write): vdh: ios=60640/15239, merge=0/0, ticks=879/17395, in_queue=28270, util=94.61% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928718: Fri Jan 7 14:01:05 2022 read: IOPS=2073, BW=162MiB/s (170MB/s)(4879MiB/30117msec) slat (usec): min=2, max=374, avg=11.47, stdev= 5.64 clat (nsec): min=736, max=2532.3M, avg=1563548.83, stdev=56916274.67 lat (usec): min=29, max=2532.3k, avg=1575.17, stdev=56916.28 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 44], 60.00th=[ 48], 70.00th=[ 53], | 80.00th=[ 60], 90.00th=[ 69], 95.00th=[ 78], | 99.00th=[ 334], 99.50th=[ 676], 99.90th=[ 18220], | 99.95th=[1635779], 99.99th=[2499806] bw ( KiB/s): min=134720, max=309120, per=100.00%, avg=166543.90, stdev=23528.18, samples=60 iops : min= 1684, max= 3864, avg=2081.77, stdev=294.12, samples=60 write: IOPS=521, BW=40.7MiB/s (42.7MB/s)(1226MiB/30117msec) slat (usec): min=8, max=380, avg=24.27, stdev=11.83 clat (msec): min=5, max=2501, avg=89.61, stdev=77.84 lat (msec): min=5, max=2501, avg=89.63, stdev=77.84 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 83], 60.00th=[ 85], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 125], | 99.00th=[ 224], 99.50th=[ 493], 99.90th=[ 1368], 99.95th=[ 1586], | 99.99th=[ 2265] bw ( KiB/s): min=35040, max=71360, per=100.00%, avg=41745.33, stdev=4758.86, samples=60 iops : min= 438, max= 892, avg=521.78, stdev=59.52, samples=60 lat (nsec) : 750=0.01%, 1000=0.08% lat (usec) : 2=0.09%, 4=0.01%, 10=0.02%, 20=0.08%, 50=51.44% lat (usec) : 100=26.30%, 250=0.90%, 500=0.42%, 750=0.21%, 1000=0.09% lat (msec) : 2=0.19%, 4=0.01%, 10=0.02%, 20=0.28%, 50=0.18% lat (msec) : 100=17.24%, 250=2.19%, 500=0.10%, 750=0.04%, 1000=0.02% lat (msec) : 2000=0.05%, >=2000=0.04% cpu : usr=1.95%, sys=5.27%, ctx=75853, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=62447,15699,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=4879MiB (5116MB), run=30117-30117msec WRITE: bw=40.7MiB/s (42.7MB/s), 40.7MiB/s-40.7MiB/s (42.7MB/s-42.7MB/s), io=1226MiB (1286MB), run=30117-30117msec Disk stats (read/write): vdh: ios=62205/15582, merge=0/0, ticks=91140/1385698, in_queue=1492428, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929902: Fri Jan 7 14:01:42 2022 read: IOPS=2037, BW=159MiB/s (167MB/s)(4777MiB/30001msec) clat (usec): min=35, max=23388, avg=73.99, stdev=445.45 lat (usec): min=35, max=23388, avg=74.13, stdev=445.46 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 100], 99.50th=[ 115], 99.90th=[ 1958], 99.95th=[13829], | 99.99th=[17171] bw ( KiB/s): min=128960, max=239040, per=100.00%, avg=163050.45, stdev=17201.62, samples=60 iops : min= 1612, max= 2988, avg=2038.12, stdev=215.03, samples=60 write: IOPS=512, BW=40.0MiB/s (41.9MB/s)(1200MiB/30001msec) clat (usec): min=406, max=43659, avg=1641.83, stdev=1605.83 lat (usec): min=412, max=43667, avg=1649.84, stdev=1605.80 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 701], 10.00th=[ 971], 20.00th=[ 1237], | 30.00th=[ 1336], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2311], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[17433], 99.95th=[40633], | 99.99th=[42206] bw ( KiB/s): min=33600, max=61920, per=100.00%, avg=40961.32, stdev=3913.88, samples=60 iops : min= 420, max= 774, avg=512.00, stdev=48.93, samples=60 lat (usec) : 50=12.20%, 100=66.92%, 250=0.68%, 500=0.33%, 750=0.90% lat (usec) : 1000=0.95% lat (msec) : 2=15.73%, 4=1.91%, 10=0.21%, 20=0.15%, 50=0.02% cpu : usr=1.77%, sys=3.78%, ctx=76512, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61142,15361,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4777MiB (5009MB), run=30001-30001msec WRITE: bw=40.0MiB/s (41.9MB/s), 40.0MiB/s-40.0MiB/s (41.9MB/s-41.9MB/s), io=1200MiB (1258MB), run=30001-30001msec Disk stats (read/write): vdh: ios=60871/15296, merge=0/0, ticks=815/17266, in_queue=28078, util=93.69% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930575: Fri Jan 7 14:02:19 2022 read: IOPS=2083, BW=163MiB/s (171MB/s)(4917MiB/30212msec) slat (usec): min=2, max=287, avg=11.12, stdev= 5.51 clat (nsec): min=769, max=2631.1M, avg=1621993.95, stdev=57602945.52 lat (usec): min=29, max=2631.1k, avg=1633.27, stdev=57602.93 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 45], 60.00th=[ 50], 70.00th=[ 57], | 80.00th=[ 63], 90.00th=[ 77], 95.00th=[ 145], | 99.00th=[ 1647], 99.50th=[ 2835], 99.90th=[ 14222], | 99.95th=[1669333], 99.99th=[2499806] bw ( KiB/s): min=132320, max=350400, per=100.00%, avg=167851.43, stdev=28904.81, samples=60 iops : min= 1654, max= 4380, avg=2098.13, stdev=361.31, samples=60 write: IOPS=523, BW=40.9MiB/s (42.9MB/s)(1237MiB/30212msec) slat (usec): min=7, max=320, avg=23.81, stdev=10.43 clat (msec): min=12, max=3334, avg=184.32, stdev=178.08 lat (msec): min=12, max=3334, avg=184.34, stdev=178.08 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 107], 10.00th=[ 127], 20.00th=[ 136], | 30.00th=[ 150], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 218], 95.00th=[ 236], | 99.00th=[ 768], 99.50th=[ 1905], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 3205] bw ( KiB/s): min=36000, max=74880, per=100.00%, avg=41954.93, stdev=5133.01, samples=60 iops : min= 450, max= 936, avg=524.43, stdev=64.16, samples=60 lat (nsec) : 1000=0.04% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.05%, 50=48.45% lat (usec) : 100=26.30%, 250=1.90%, 500=0.95%, 750=0.57%, 1000=0.36% lat (msec) : 2=0.59%, 4=0.55%, 10=0.02%, 20=0.04%, 50=0.40% lat (msec) : 100=0.32%, 250=18.57%, 500=0.55%, 750=0.04%, 1000=0.05% lat (msec) : 2000=0.13%, >=2000=0.08% cpu : usr=1.97%, sys=5.13%, ctx=73623, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62938,15829,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4917MiB (5156MB), run=30212-30212msec WRITE: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=1237MiB (1297MB), run=30212-30212msec Disk stats (read/write): vdh: ios=62936/15752, merge=0/0, ticks=95483/2878212, in_queue=3008768, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931403: Fri Jan 7 14:02:56 2022 read: IOPS=2036, BW=159MiB/s (167MB/s)(4773MiB/30001msec) clat (usec): min=32, max=21611, avg=70.94, stdev=440.95 lat (usec): min=32, max=21611, avg=71.08, stdev=440.95 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 96], 99.50th=[ 110], 99.90th=[ 1549], 99.95th=[13304], | 99.99th=[16909] bw ( KiB/s): min=124160, max=245440, per=100.00%, avg=162925.33, stdev=19676.31, samples=60 iops : min= 1552, max= 3068, avg=2036.57, stdev=245.95, samples=60 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1199MiB/30001msec) clat (usec): min=403, max=134693, avg=1655.45, stdev=1906.00 lat (usec): min=410, max=134700, avg=1662.91, stdev=1905.98 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 717], 10.00th=[ 1012], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2024], 95.00th=[ 2343], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[35914], 99.95th=[39584], | 99.99th=[44303] bw ( KiB/s): min=32960, max=64160, per=100.00%, avg=40930.67, stdev=4224.22, samples=60 iops : min= 412, max= 802, avg=511.63, stdev=52.80, samples=60 lat (usec) : 50=24.58%, 100=54.73%, 250=0.49%, 500=0.32%, 750=0.82% lat (usec) : 1000=0.84% lat (msec) : 2=15.96%, 4=1.90%, 10=0.20%, 20=0.15%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=1.69%, sys=3.50%, ctx=76458, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61097,15350,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4773MiB (5005MB), run=30001-30001msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1199MiB (1257MB), run=30001-30001msec Disk stats (read/write): vdh: ios=60816/15286, merge=0/0, ticks=812/17426, in_queue=28261, util=94.28% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932274: Fri Jan 7 14:03:34 2022 read: IOPS=2081, BW=163MiB/s (171MB/s)(4929MiB/30310msec) slat (usec): min=2, max=275082, avg=372.57, stdev=2388.00 clat (usec): min=1033, max=2676.1k, avg=31539.97, stdev=61110.73 lat (usec): min=1039, max=2676.1k, avg=31912.69, stdev=61172.65 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 6587], 10.00th=[ 16581], | 20.00th=[ 21365], 30.00th=[ 23725], 40.00th=[ 25822], | 50.00th=[ 27919], 60.00th=[ 30278], 70.00th=[ 32637], | 80.00th=[ 35914], 90.00th=[ 41157], 95.00th=[ 47449], | 99.00th=[ 90702], 99.50th=[ 223347], 99.90th=[ 337642], | 99.95th=[1451230], 99.99th=[2566915] bw ( KiB/s): min=143200, max=330080, per=100.00%, avg=168045.33, stdev=25618.01, samples=60 iops : min= 1790, max= 4126, avg=2100.57, stdev=320.23, samples=60 write: IOPS=523, BW=40.9MiB/s (42.9MB/s)(1239MiB/30310msec) slat (usec): min=7, max=221648, avg=399.56, stdev=2499.16 clat (msec): min=20, max=3382, avg=254.65, stdev=161.50 lat (msec): min=20, max=3382, avg=255.05, stdev=161.48 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 180], 10.00th=[ 201], 20.00th=[ 213], | 30.00th=[ 224], 40.00th=[ 232], 50.00th=[ 241], 60.00th=[ 247], | 70.00th=[ 255], 80.00th=[ 268], 90.00th=[ 292], 95.00th=[ 338], | 99.00th=[ 600], 99.50th=[ 1368], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3373] bw ( KiB/s): min=35360, max=65440, per=100.00%, avg=41949.33, stdev=4081.82, samples=60 iops : min= 442, max= 818, avg=524.37, stdev=51.02, samples=60 lat (msec) : 2=1.02%, 4=1.71%, 10=2.06%, 20=8.28%, 50=64.22% lat (msec) : 100=2.43%, 250=12.94%, 500=6.92%, 750=0.18%, 1000=0.04% lat (msec) : 2000=0.10%, >=2000=0.09% cpu : usr=1.69%, sys=2.00%, ctx=8858, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=63086,15862,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4929MiB (5168MB), run=30310-30310msec WRITE: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=1239MiB (1299MB), run=30310-30310msec Disk stats (read/write): vdh: ios=63083/15774, merge=0/0, ticks=108127/3526431, in_queue=3681564, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932828: Fri Jan 7 14:04:10 2022 read: IOPS=1798, BW=225MiB/s (236MB/s)(6743MiB/30001msec) clat (usec): min=49, max=25438, avg=98.22, stdev=503.18 lat (usec): min=49, max=25439, avg=98.37, stdev=503.18 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 69], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 94], 90.00th=[ 103], 95.00th=[ 113], | 99.00th=[ 141], 99.50th=[ 159], 99.90th=[11469], 99.95th=[15664], | 99.99th=[19006] bw ( KiB/s): min=122112, max=370176, per=99.90%, avg=229940.07, stdev=31627.49, samples=59 iops : min= 954, max= 2892, avg=1796.41, stdev=247.09, samples=59 write: IOPS=451, BW=56.5MiB/s (59.2MB/s)(1694MiB/30001msec) clat (usec): min=533, max=208860, avg=1800.83, stdev=2506.89 lat (usec): min=545, max=208873, avg=1813.51, stdev=2506.88 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 906], 10.00th=[ 1074], 20.00th=[ 1221], | 30.00th=[ 1369], 40.00th=[ 1500], 50.00th=[ 1598], 60.00th=[ 1696], | 70.00th=[ 1811], 80.00th=[ 1958], 90.00th=[ 2180], 95.00th=[ 2540], | 99.00th=[ 9241], 99.50th=[10159], 99.90th=[36963], 99.95th=[39584], | 99.99th=[59507] bw ( KiB/s): min=32768, max=96000, per=99.85%, avg=57738.85, stdev=7253.30, samples=59 iops : min= 256, max= 750, avg=451.08, stdev=56.67, samples=59 lat (usec) : 50=0.01%, 100=70.07%, 250=9.67%, 500=0.07%, 750=0.52% lat (usec) : 1000=0.96% lat (msec) : 2=15.15%, 4=2.99%, 10=0.35%, 20=0.18%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.70%, sys=3.74%, ctx=67510, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53946,13553,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6743MiB (7071MB), run=30001-30001msec WRITE: bw=56.5MiB/s (59.2MB/s), 56.5MiB/s-56.5MiB/s (59.2MB/s-59.2MB/s), io=1694MiB (1776MB), run=30001-30001msec Disk stats (read/write): vdh: ios=53714/13495, merge=0/0, ticks=848/17193, in_queue=28060, util=93.59% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933687: Fri Jan 7 14:04:47 2022 read: IOPS=1840, BW=230MiB/s (241MB/s)(6901MiB/30001msec) slat (usec): min=8, max=233, avg=15.63, stdev= 5.53 clat (nsec): min=834, max=32254k, avg=77498.39, stdev=514993.29 lat (usec): min=48, max=32266, avg=93.29, stdev=515.08 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 52], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 83], | 99.00th=[ 99], 99.50th=[ 112], 99.90th=[11076], 99.95th=[14877], | 99.99th=[19006] bw ( KiB/s): min=168960, max=382464, per=99.99%, avg=235533.02, stdev=29932.05, samples=59 iops : min= 1320, max= 2988, avg=1840.10, stdev=233.84, samples=59 write: IOPS=462, BW=57.8MiB/s (60.6MB/s)(1734MiB/30001msec) slat (usec): min=14, max=178, avg=29.49, stdev= 9.60 clat (usec): min=520, max=48025, avg=1754.29, stdev=1631.95 lat (usec): min=547, max=48066, avg=1783.97, stdev=1632.05 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 783], 10.00th=[ 914], 20.00th=[ 1254], | 30.00th=[ 1434], 40.00th=[ 1532], 50.00th=[ 1614], 60.00th=[ 1713], | 70.00th=[ 1827], 80.00th=[ 1975], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 8979], 99.50th=[10159], 99.90th=[36439], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=50432, max=99072, per=100.00%, avg=59192.41, stdev=6425.03, samples=59 iops : min= 394, max= 774, avg=462.44, stdev=50.20, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 10=0.01%, 20=0.01%, 50=8.64%, 100=70.48% lat (usec) : 250=0.63%, 500=0.02%, 750=0.79%, 1000=1.65% lat (msec) : 2=13.85%, 4=3.50%, 10=0.19%, 20=0.18%, 50=0.03% cpu : usr=1.78%, sys=5.70%, ctx=69060, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55208,13870,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6901MiB (7236MB), run=30001-30001msec WRITE: bw=57.8MiB/s (60.6MB/s), 57.8MiB/s-57.8MiB/s (60.6MB/s-60.6MB/s), io=1734MiB (1818MB), run=30001-30001msec Disk stats (read/write): vdh: ios=54987/13817, merge=0/0, ticks=892/16644, in_queue=28056, util=93.58% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934407: Fri Jan 7 14:05:24 2022 read: IOPS=1830, BW=229MiB/s (240MB/s)(6864MiB/30002msec) clat (usec): min=37, max=64859, avg=83.87, stdev=611.70 lat (usec): min=37, max=64859, avg=84.01, stdev=611.70 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 56], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 108], 99.50th=[ 125], 99.90th=[11469], 99.95th=[16188], | 99.99th=[24249] bw ( KiB/s): min=163840, max=385536, per=100.00%, avg=234308.43, stdev=30016.55, samples=60 iops : min= 1280, max= 3012, avg=1830.53, stdev=234.50, samples=60 write: IOPS=460, BW=57.5MiB/s (60.3MB/s)(1726MiB/30002msec) clat (usec): min=535, max=146912, avg=1818.79, stdev=2007.35 lat (usec): min=547, max=146925, avg=1831.23, stdev=2007.29 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 791], 10.00th=[ 1057], 20.00th=[ 1369], | 30.00th=[ 1500], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[35914], 99.95th=[39584], | 99.99th=[44827] bw ( KiB/s): min=41216, max=100096, per=100.00%, avg=58899.22, stdev=6863.14, samples=60 iops : min= 322, max= 782, avg=460.15, stdev=53.62, samples=60 lat (usec) : 50=1.78%, 100=76.92%, 250=1.07%, 500=0.02%, 750=0.79% lat (usec) : 1000=1.03% lat (msec) : 2=14.18%, 4=3.77%, 10=0.21%, 20=0.19%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.60%, sys=3.54%, ctx=68729, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54914,13805,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6864MiB (7198MB), run=30002-30002msec WRITE: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=1726MiB (1809MB), run=30002-30002msec Disk stats (read/write): vdh: ios=54718/13749, merge=0/0, ticks=1014/17227, in_queue=28301, util=94.39% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935576: Fri Jan 7 14:06:01 2022 read: IOPS=1867, BW=233MiB/s (245MB/s)(7008MiB/30025msec) slat (nsec): min=6036, max=97345, avg=15025.76, stdev=4408.78 clat (nsec): min=803, max=2499.5M, avg=1530358.69, stdev=55333395.14 lat (usec): min=43, max=2499.5k, avg=1545.54, stdev=55333.37 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 47], 10.00th=[ 48], | 20.00th=[ 51], 30.00th=[ 52], 40.00th=[ 54], | 50.00th=[ 57], 60.00th=[ 61], 70.00th=[ 66], | 80.00th=[ 73], 90.00th=[ 83], 95.00th=[ 92], | 99.00th=[ 130], 99.50th=[ 190], 99.90th=[ 64226], | 99.95th=[1518339], 99.99th=[2499806] bw ( KiB/s): min=183552, max=444928, per=100.00%, avg=239185.07, stdev=34309.55, samples=60 iops : min= 1434, max= 3476, avg=1868.63, stdev=268.04, samples=60 write: IOPS=469, BW=58.7MiB/s (61.5MB/s)(1762MiB/30025msec) slat (usec): min=12, max=157, avg=30.33, stdev= 9.38 clat (usec): min=1296, max=86792, avg=15115.41, stdev=6536.96 lat (usec): min=1314, max=86820, avg=15145.92, stdev=6537.33 clat percentiles (usec): | 1.00th=[ 3130], 5.00th=[ 7046], 10.00th=[ 9241], 20.00th=[11076], | 30.00th=[12780], 40.00th=[13960], 50.00th=[14746], 60.00th=[15795], | 70.00th=[16712], 80.00th=[17957], 90.00th=[20055], 95.00th=[24249], | 99.00th=[34341], 99.50th=[64226], 99.90th=[70779], 99.95th=[85459], | 99.99th=[86508] bw ( KiB/s): min=49920, max=111616, per=100.00%, avg=60100.27, stdev=7603.89, samples=60 iops : min= 390, max= 872, avg=469.53, stdev=59.41, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=15.05%, 100=62.41%, 250=2.14% lat (usec) : 500=0.13%, 750=0.04%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.57%, 10=2.11%, 20=15.35%, 50=1.92% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.73%, sys=5.71%, ctx=69346, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=56060,14094,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=233MiB/s (245MB/s), 233MiB/s-233MiB/s (245MB/s-245MB/s), io=7008MiB (7348MB), run=30025-30025msec WRITE: bw=58.7MiB/s (61.5MB/s), 58.7MiB/s-58.7MiB/s (61.5MB/s-61.5MB/s), io=1762MiB (1847MB), run=30025-30025msec Disk stats (read/write): vdh: ios=55903/14039, merge=0/0, ticks=81089/204615, in_queue=297095, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936740: Fri Jan 7 14:06:38 2022 read: IOPS=1822, BW=228MiB/s (239MB/s)(6835MiB/30001msec) clat (usec): min=41, max=25835, avg=83.53, stdev=531.16 lat (usec): min=41, max=25835, avg=83.67, stdev=531.16 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 88], | 99.00th=[ 108], 99.50th=[ 119], 99.90th=[11469], 99.95th=[15664], | 99.99th=[20317] bw ( KiB/s): min=179968, max=378368, per=100.00%, avg=233786.71, stdev=29285.34, samples=59 iops : min= 1406, max= 2956, avg=1826.44, stdev=228.80, samples=59 write: IOPS=457, BW=57.2MiB/s (60.0MB/s)(1717MiB/30001msec) clat (usec): min=521, max=45611, avg=1829.83, stdev=1682.51 lat (usec): min=535, max=45623, avg=1843.05, stdev=1682.38 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 807], 10.00th=[ 1090], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2245], 95.00th=[ 2540], | 99.00th=[ 9765], 99.50th=[10290], 99.90th=[37487], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=44800, max=98816, per=100.00%, avg=58680.05, stdev=6748.98, samples=59 iops : min= 350, max= 772, avg=458.41, stdev=52.73, samples=59 lat (usec) : 50=1.18%, 100=77.53%, 250=1.10%, 500=0.02%, 750=0.77% lat (usec) : 1000=0.94% lat (msec) : 2=14.31%, 4=3.69%, 10=0.21%, 20=0.21%, 50=0.03% cpu : usr=1.77%, sys=3.58%, ctx=68424, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54676,13736,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6835MiB (7166MB), run=30001-30001msec WRITE: bw=57.2MiB/s (60.0MB/s), 57.2MiB/s-57.2MiB/s (60.0MB/s-60.0MB/s), io=1717MiB (1800MB), run=30001-30001msec Disk stats (read/write): vdh: ios=54427/13677, merge=0/0, ticks=921/17267, in_queue=28237, util=94.19% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937252: Fri Jan 7 14:07:16 2022 read: IOPS=1871, BW=234MiB/s (245MB/s)(7046MiB/30120msec) slat (usec): min=3, max=176, avg=12.99, stdev= 5.00 clat (nsec): min=837, max=2567.6M, avg=1700593.51, stdev=57026957.76 lat (usec): min=34, max=2567.6k, avg=1713.75, stdev=57026.94 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 40], 10.00th=[ 41], | 20.00th=[ 44], 30.00th=[ 45], 40.00th=[ 48], | 50.00th=[ 51], 60.00th=[ 55], 70.00th=[ 63], | 80.00th=[ 76], 90.00th=[ 86], 95.00th=[ 100], | 99.00th=[ 510], 99.50th=[ 1237], 99.90th=[ 325059], | 99.95th=[1702888], 99.99th=[2533360] bw ( KiB/s): min=187648, max=460800, per=100.00%, avg=240525.10, stdev=35014.71, samples=60 iops : min= 1466, max= 3600, avg=1879.05, stdev=273.56, samples=60 write: IOPS=470, BW=58.8MiB/s (61.7MB/s)(1772MiB/30120msec) slat (usec): min=12, max=249, avg=32.56, stdev=13.94 clat (msec): min=11, max=2503, avg=99.37, stdev=71.12 lat (msec): min=11, max=2503, avg=99.40, stdev=71.12 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 73], 10.00th=[ 80], 20.00th=[ 85], | 30.00th=[ 88], 40.00th=[ 90], 50.00th=[ 92], 60.00th=[ 95], | 70.00th=[ 97], 80.00th=[ 103], 90.00th=[ 113], 95.00th=[ 140], | 99.00th=[ 228], 99.50th=[ 575], 99.90th=[ 1083], 99.95th=[ 1385], | 99.99th=[ 2089] bw ( KiB/s): min=49408, max=105984, per=100.00%, avg=60295.17, stdev=6980.42, samples=60 iops : min= 386, max= 828, avg=470.97, stdev=54.57, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=38.84% lat (usec) : 100=37.07%, 250=2.65%, 500=0.47%, 750=0.21%, 1000=0.11% lat (msec) : 2=0.31%, 4=0.08%, 10=0.01%, 20=0.27%, 50=0.22% lat (msec) : 100=14.88%, 250=4.55%, 500=0.07%, 750=0.07%, 1000=0.05% lat (msec) : 2000=0.05%, >=2000=0.03% cpu : usr=2.03%, sys=5.27%, ctx=68334, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=56367,14173,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=7046MiB (7388MB), run=30120-30120msec WRITE: bw=58.8MiB/s (61.7MB/s), 58.8MiB/s-58.8MiB/s (61.7MB/s-61.7MB/s), io=1772MiB (1858MB), run=30120-30120msec Disk stats (read/write): vdh: ios=56362/14168, merge=0/0, ticks=86789/1396587, in_queue=1501257, util=99.53% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938160: Fri Jan 7 14:07:52 2022 read: IOPS=1829, BW=229MiB/s (240MB/s)(6862MiB/30001msec) clat (usec): min=48, max=26531, avg=90.16, stdev=520.00 lat (usec): min=48, max=26531, avg=90.30, stdev=520.00 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 60], 10.00th=[ 62], 20.00th=[ 64], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 89], 95.00th=[ 96], | 99.00th=[ 112], 99.50th=[ 122], 99.90th=[11731], 99.95th=[15795], | 99.99th=[18482] bw ( KiB/s): min=182784, max=377344, per=100.00%, avg=234416.78, stdev=27749.15, samples=59 iops : min= 1428, max= 2948, avg=1831.37, stdev=216.79, samples=59 write: IOPS=460, BW=57.5MiB/s (60.3MB/s)(1725MiB/30001msec) clat (usec): min=548, max=61749, avg=1794.29, stdev=1711.90 lat (usec): min=555, max=61762, avg=1806.43, stdev=1711.79 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 799], 10.00th=[ 1012], 20.00th=[ 1352], | 30.00th=[ 1483], 40.00th=[ 1549], 50.00th=[ 1631], 60.00th=[ 1729], | 70.00th=[ 1844], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 9503], 99.50th=[10159], 99.90th=[37487], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=47872, max=98560, per=100.00%, avg=58899.44, stdev=6537.86, samples=59 iops : min= 374, max= 770, avg=460.14, stdev=51.07, samples=59 lat (usec) : 50=0.01%, 100=77.55%, 250=2.25%, 500=0.01%, 750=0.74% lat (usec) : 1000=1.24% lat (msec) : 2=14.29%, 4=3.52%, 10=0.19%, 20=0.19%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.64%, sys=3.37%, ctx=68710, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54898,13803,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6862MiB (7196MB), run=30001-30001msec WRITE: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=1725MiB (1809MB), run=30001-30001msec Disk stats (read/write): vdh: ios=54696/13742, merge=0/0, ticks=905/16851, in_queue=28287, util=94.35% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938713: Fri Jan 7 14:08:30 2022 read: IOPS=1852, BW=232MiB/s (243MB/s)(6996MiB/30213msec) slat (usec): min=3, max=259, avg=11.93, stdev= 5.50 clat (nsec): min=924, max=2586.7M, avg=1684337.24, stdev=55127071.03 lat (usec): min=35, max=2586.7k, avg=1696.43, stdev=55127.07 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 55], 70.00th=[ 65], | 80.00th=[ 77], 90.00th=[ 91], 95.00th=[ 176], | 99.00th=[ 2212], 99.50th=[ 3425], 99.90th=[ 291505], | 99.95th=[1568670], 99.99th=[2499806] bw ( KiB/s): min=191744, max=482560, per=100.00%, avg=238771.20, stdev=38437.51, samples=60 iops : min= 1498, max= 3770, avg=1865.40, stdev=300.29, samples=60 write: IOPS=465, BW=58.2MiB/s (61.0MB/s)(1758MiB/30213msec) slat (usec): min=10, max=378, avg=33.78, stdev=14.85 clat (msec): min=5, max=2596, avg=207.99, stdev=103.07 lat (msec): min=5, max=2596, avg=208.02, stdev=103.07 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 146], 10.00th=[ 165], 20.00th=[ 186], | 30.00th=[ 192], 40.00th=[ 197], 50.00th=[ 201], 60.00th=[ 205], | 70.00th=[ 209], 80.00th=[ 215], 90.00th=[ 249], 95.00th=[ 259], | 99.00th=[ 435], 99.50th=[ 902], 99.90th=[ 1754], 99.95th=[ 1754], | 99.99th=[ 2165] bw ( KiB/s): min=47360, max=99584, per=100.00%, avg=59601.07, stdev=6443.76, samples=60 iops : min= 370, max= 778, avg=465.63, stdev=50.34, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=40.35% lat (usec) : 100=33.30%, 250=2.75%, 500=1.01%, 750=0.55%, 1000=0.41% lat (msec) : 2=0.65%, 4=0.60%, 10=0.16%, 20=0.05%, 50=0.35% lat (msec) : 100=0.24%, 250=17.76%, 500=1.51%, 750=0.06%, 1000=0.06% lat (msec) : 2000=0.10%, >=2000=0.03% cpu : usr=2.25%, sys=4.71%, ctx=65326, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=55966,14065,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=6996MiB (7336MB), run=30213-30213msec WRITE: bw=58.2MiB/s (61.0MB/s), 58.2MiB/s-58.2MiB/s (61.0MB/s-61.0MB/s), io=1758MiB (1844MB), run=30213-30213msec Disk stats (read/write): vdh: ios=55962/14036, merge=0/0, ticks=83251/2897659, in_queue=3006268, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939639: Fri Jan 7 14:09:06 2022 read: IOPS=1847, BW=231MiB/s (242MB/s)(6927MiB/30001msec) clat (usec): min=37, max=71872, avg=80.14, stdev=589.50 lat (usec): min=37, max=71872, avg=80.29, stdev=589.50 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 104], 99.50th=[ 120], 99.90th=[10814], 99.95th=[15270], | 99.99th=[19792] bw ( KiB/s): min=165376, max=366080, per=100.00%, avg=236477.18, stdev=29130.54, samples=60 iops : min= 1292, max= 2860, avg=1847.45, stdev=227.59, samples=60 write: IOPS=464, BW=58.0MiB/s (60.9MB/s)(1742MiB/30001msec) clat (usec): min=513, max=44246, avg=1813.08, stdev=1580.71 lat (usec): min=525, max=44256, avg=1825.40, stdev=1580.59 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 824], 10.00th=[ 1123], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 9503], 99.50th=[10159], 99.90th=[18220], 99.95th=[41681], | 99.99th=[43779] bw ( KiB/s): min=47616, max=94208, per=100.00%, avg=59445.02, stdev=6242.90, samples=60 iops : min= 372, max= 736, avg=464.37, stdev=48.76, samples=60 lat (usec) : 50=4.00%, 100=74.91%, 250=0.87%, 500=0.03%, 750=0.74% lat (usec) : 1000=0.84% lat (msec) : 2=14.42%, 4=3.80%, 10=0.18%, 20=0.18%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.60%, sys=3.55%, ctx=69360, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55419,13932,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6927MiB (7264MB), run=30001-30001msec WRITE: bw=58.0MiB/s (60.9MB/s), 58.0MiB/s-58.0MiB/s (60.9MB/s-60.9MB/s), io=1742MiB (1826MB), run=30001-30001msec Disk stats (read/write): vdh: ios=55235/13881, merge=0/0, ticks=945/17260, in_queue=28205, util=94.10% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940607: Fri Jan 7 14:09:44 2022 read: IOPS=1874, BW=234MiB/s (246MB/s)(7096MiB/30286msec) slat (usec): min=3, max=296106, avg=412.52, stdev=2697.53 clat (usec): min=1497, max=2647.2k, avg=34847.97, stdev=58842.20 lat (usec): min=1502, max=2647.2k, avg=35260.63, stdev=58914.39 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 52], | 99.00th=[ 100], 99.50th=[ 243], 99.90th=[ 334], 99.95th=[ 1586], | 99.99th=[ 2534] bw ( KiB/s): min=172544, max=513280, per=100.00%, avg=241954.50, stdev=44005.29, samples=60 iops : min= 1348, max= 4010, avg=1890.23, stdev=343.78, samples=60 write: IOPS=471, BW=58.0MiB/s (61.8MB/s)(1786MiB/30286msec) slat (usec): min=13, max=222424, avg=449.64, stdev=2606.60 clat (msec): min=6, max=2858, avg=283.13, stdev=135.06 lat (msec): min=6, max=2858, avg=283.58, stdev=135.06 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 178], 10.00th=[ 213], 20.00th=[ 241], | 30.00th=[ 257], 40.00th=[ 266], 50.00th=[ 271], 60.00th=[ 279], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 380], | 99.00th=[ 919], 99.50th=[ 1284], 99.90th=[ 1703], 99.95th=[ 1737], | 99.99th=[ 2433] bw ( KiB/s): min=43520, max=101120, per=100.00%, avg=60402.78, stdev=6971.41, samples=60 iops : min= 340, max= 790, avg=471.85, stdev=54.47, samples=60 lat (msec) : 2=0.06%, 4=1.20%, 10=3.87%, 20=4.64%, 50=65.83% lat (msec) : 100=4.01%, 250=4.85%, 500=14.94%, 750=0.28%, 1000=0.08% lat (msec) : 2000=0.19%, >=2000=0.03% cpu : usr=2.38%, sys=1.94%, ctx=7912, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56766,14289,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=7096MiB (7440MB), run=30286-30286msec WRITE: bw=58.0MiB/s (61.8MB/s), 58.0MiB/s-58.0MiB/s (61.8MB/s-61.8MB/s), io=1786MiB (1873MB), run=30286-30286msec Disk stats (read/write): vdh: ios=56761/14190, merge=0/0, ticks=91341/3531043, in_queue=3683537, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941089: Fri Jan 7 14:10:20 2022 read: IOPS=1252, BW=313MiB/s (328MB/s)(9393MiB/30003msec) clat (usec): min=70, max=25037, avg=125.26, stdev=526.99 lat (usec): min=70, max=25037, avg=125.41, stdev=526.99 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 84], 10.00th=[ 86], 20.00th=[ 89], | 30.00th=[ 93], 40.00th=[ 97], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 133], 95.00th=[ 145], | 99.00th=[ 184], 99.50th=[ 206], 99.90th=[10159], 99.95th=[15664], | 99.99th=[20579] bw ( KiB/s): min=195584, max=548352, per=100.00%, avg=320614.40, stdev=58254.07, samples=60 iops : min= 764, max= 2142, avg=1252.37, stdev=227.52, samples=60 write: IOPS=315, BW=78.0MiB/s (82.8MB/s)(2370MiB/30003msec) clat (usec): min=887, max=98933, avg=2633.09, stdev=2787.69 lat (usec): min=910, max=98958, avg=2659.50, stdev=2787.55 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1123], 10.00th=[ 1352], 20.00th=[ 1598], | 30.00th=[ 1811], 40.00th=[ 1958], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2671], 90.00th=[ 4359], 95.00th=[ 5538], | 99.00th=[11207], 99.50th=[13042], 99.90th=[44303], 99.95th=[53740], | 99.99th=[99091] bw ( KiB/s): min=50176, max=143872, per=100.00%, avg=80887.47, stdev=14274.66, samples=60 iops : min= 196, max= 562, avg=315.97, stdev=55.76, samples=60 lat (usec) : 100=37.57%, 250=42.04%, 500=0.09%, 750=0.03%, 1000=0.43% lat (msec) : 2=8.32%, 4=9.20%, 10=1.73%, 20=0.56%, 50=0.02% lat (msec) : 100=0.02% cpu : usr=1.64%, sys=2.95%, ctx=47062, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=37572,9480,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=9393MiB (9849MB), run=30003-30003msec WRITE: bw=78.0MiB/s (82.8MB/s), 78.0MiB/s-78.0MiB/s (82.8MB/s-82.8MB/s), io=2370MiB (2485MB), run=30003-30003msec Disk stats (read/write): vdh: ios=37447/9446, merge=0/0, ticks=677/19954, in_queue=28426, util=94.81% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942438: Fri Jan 7 14:10:57 2022 read: IOPS=1307, BW=327MiB/s (343MB/s)(9805MiB/30002msec) slat (nsec): min=11888, max=84298, avg=19818.47, stdev=5147.16 clat (usec): min=20, max=18429, avg=89.31, stdev=350.24 lat (usec): min=65, max=18446, avg=109.30, stdev=350.40 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 62], 10.00th=[ 64], 20.00th=[ 66], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 125], 99.50th=[ 137], 99.90th=[ 6521], 99.95th=[ 9503], | 99.99th=[13698] bw ( KiB/s): min=250880, max=555520, per=100.00%, avg=334669.93, stdev=44433.43, samples=60 iops : min= 980, max= 2170, avg=1307.40, stdev=173.61, samples=60 write: IOPS=329, BW=82.5MiB/s (86.5MB/s)(2474MiB/30002msec) slat (usec): min=24, max=140, avg=43.81, stdev= 9.29 clat (usec): min=849, max=60762, avg=2546.42, stdev=2461.07 lat (usec): min=886, max=60820, avg=2590.43, stdev=2461.13 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1549], | 30.00th=[ 1876], 40.00th=[ 1991], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2671], 90.00th=[ 4015], 95.00th=[ 4883], | 99.00th=[10945], 99.50th=[12649], 99.90th=[45876], 99.95th=[53740], | 99.99th=[60556] bw ( KiB/s): min=69120, max=147968, per=100.00%, avg=84440.07, stdev=10161.06, samples=60 iops : min= 270, max= 578, avg=329.83, stdev=39.69, samples=60 lat (usec) : 50=0.07%, 100=74.37%, 250=5.28%, 500=0.02%, 750=0.01% lat (usec) : 1000=0.63% lat (msec) : 2=7.68%, 4=9.83%, 10=1.54%, 20=0.56%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.68%, sys=4.70%, ctx=49134, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39221,9896,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=9805MiB (10.3GB), run=30002-30002msec WRITE: bw=82.5MiB/s (86.5MB/s), 82.5MiB/s-82.5MiB/s (86.5MB/s-86.5MB/s), io=2474MiB (2594MB), run=30002-30002msec Disk stats (read/write): vdh: ios=39104/9865, merge=0/0, ticks=462/20561, in_queue=28316, util=94.46% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943462: Fri Jan 7 14:11:34 2022 read: IOPS=1360, BW=340MiB/s (357MB/s)(9.96GiB/30001msec) clat (usec): min=51, max=21153, avg=92.21, stdev=391.26 lat (usec): min=51, max=21153, avg=92.36, stdev=391.26 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 64], 10.00th=[ 66], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 126], 99.50th=[ 139], 99.90th=[ 7504], 99.95th=[10683], | 99.99th=[13435] bw ( KiB/s): min=262656, max=595968, per=100.00%, avg=348611.25, stdev=46978.72, samples=59 iops : min= 1026, max= 2328, avg=1361.76, stdev=183.51, samples=59 write: IOPS=343, BW=85.9MiB/s (90.0MB/s)(2576MiB/30001msec) clat (usec): min=846, max=60848, avg=2512.74, stdev=2274.87 lat (usec): min=869, max=60876, avg=2537.85, stdev=2274.88 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1057], 10.00th=[ 1188], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2057], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2442], 80.00th=[ 2671], 90.00th=[ 3326], 95.00th=[ 4555], | 99.00th=[10552], 99.50th=[11338], 99.90th=[15270], 99.95th=[53740], | 99.99th=[55837] bw ( KiB/s): min=73216, max=156160, per=100.00%, avg=88003.25, stdev=10762.12, samples=59 iops : min= 286, max= 610, avg=343.76, stdev=42.04, samples=59 lat (usec) : 100=74.62%, 250=5.08%, 500=0.02%, 750=0.01%, 1000=0.56% lat (msec) : 2=6.69%, 4=11.28%, 10=1.25%, 20=0.48%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.69%, sys=2.97%, ctx=51119, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40804,10303,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=340MiB/s (357MB/s), 340MiB/s-340MiB/s (357MB/s-357MB/s), io=9.96GiB (10.7GB), run=30001-30001msec WRITE: bw=85.9MiB/s (90.0MB/s), 85.9MiB/s-85.9MiB/s (90.0MB/s-90.0MB/s), io=2576MiB (2701MB), run=30001-30001msec Disk stats (read/write): vdh: ios=40702/10266, merge=0/0, ticks=552/20753, in_queue=28419, util=94.79% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943989: Fri Jan 7 14:12:11 2022 read: IOPS=1415, BW=354MiB/s (371MB/s)(10.4GiB/30032msec) slat (usec): min=8, max=192, avg=18.35, stdev= 5.29 clat (usec): min=25, max=97574, avg=138.78, stdev=1536.96 lat (usec): min=49, max=97589, avg=157.29, stdev=1536.96 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 52], 10.00th=[ 54], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 76], | 70.00th=[ 87], 80.00th=[ 105], 90.00th=[ 128], 95.00th=[ 151], | 99.00th=[ 245], 99.50th=[ 392], 99.90th=[32637], 99.95th=[40633], | 99.99th=[49546] bw ( KiB/s): min=269824, max=717312, per=100.00%, avg=362786.13, stdev=59205.61, samples=60 iops : min= 1054, max= 2802, avg=1417.13, stdev=231.27, samples=60 write: IOPS=357, BW=89.3MiB/s (93.6MB/s)(2682MiB/30032msec) slat (usec): min=20, max=258, avg=50.10, stdev=18.51 clat (usec): min=1695, max=103300, avg=27311.26, stdev=11358.95 lat (usec): min=1720, max=103375, avg=27361.54, stdev=11359.61 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 12], 20.00th=[ 19], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 30], | 70.00th=[ 33], 80.00th=[ 35], 90.00th=[ 40], 95.00th=[ 44], | 99.00th=[ 62], 99.50th=[ 83], 99.90th=[ 97], 99.95th=[ 102], | 99.99th=[ 104] bw ( KiB/s): min=73728, max=181760, per=100.00%, avg=91460.27, stdev=13045.31, samples=60 iops : min= 288, max= 710, avg=357.27, stdev=50.96, samples=60 lat (usec) : 50=1.90%, 100=59.93%, 250=17.26%, 500=0.47%, 750=0.10% lat (usec) : 1000=0.05% lat (msec) : 2=0.02%, 4=0.03%, 10=1.46%, 20=2.93%, 50=15.45% lat (msec) : 100=0.38%, 250=0.02% cpu : usr=2.15%, sys=4.62%, ctx=48927, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=42514,10727,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=354MiB/s (371MB/s), 354MiB/s-354MiB/s (371MB/s-371MB/s), io=10.4GiB (11.1GB), run=30032-30032msec WRITE: bw=89.3MiB/s (93.6MB/s), 89.3MiB/s-89.3MiB/s (93.6MB/s-93.6MB/s), io=2682MiB (2812MB), run=30032-30032msec Disk stats (read/write): vdh: ios=42301/10670, merge=0/0, ticks=2335/286039, in_queue=297107, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944961: Fri Jan 7 14:12:48 2022 read: IOPS=1360, BW=340MiB/s (357MB/s)(9.97GiB/30002msec) clat (usec): min=52, max=19411, avg=95.07, stdev=393.47 lat (usec): min=52, max=19411, avg=95.21, stdev=393.47 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 91], 90.00th=[ 98], 95.00th=[ 106], | 99.00th=[ 131], 99.50th=[ 147], 99.90th=[ 7832], 99.95th=[11731], | 99.99th=[14091] bw ( KiB/s): min=278528, max=580608, per=100.00%, avg=348369.60, stdev=41634.21, samples=60 iops : min= 1088, max= 2268, avg=1360.80, stdev=162.64, samples=60 write: IOPS=343, BW=85.9MiB/s (90.0MB/s)(2577MiB/30002msec) clat (usec): min=842, max=61293, avg=2500.19, stdev=2254.91 lat (usec): min=866, max=61317, avg=2526.24, stdev=2254.41 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2638], 90.00th=[ 3261], 95.00th=[ 4555], | 99.00th=[10552], 99.50th=[11469], 99.90th=[16581], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=75776, max=154112, per=100.00%, avg=87941.33, stdev=9942.13, samples=60 iops : min= 296, max= 602, avg=343.50, stdev=38.85, samples=60 lat (usec) : 100=73.16%, 250=6.53%, 500=0.02%, 750=0.01%, 1000=0.53% lat (msec) : 2=7.03%, 4=11.03%, 10=1.18%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.72%, sys=3.18%, ctx=51135, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40822,10306,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=340MiB/s (357MB/s), 340MiB/s-340MiB/s (357MB/s-357MB/s), io=9.97GiB (10.7GB), run=30002-30002msec WRITE: bw=85.9MiB/s (90.0MB/s), 85.9MiB/s-85.9MiB/s (90.0MB/s-90.0MB/s), io=2577MiB (2702MB), run=30002-30002msec Disk stats (read/write): vdh: ios=40694/10265, merge=0/0, ticks=548/20590, in_queue=28292, util=94.39% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945587: Fri Jan 7 14:13:26 2022 read: IOPS=1406, BW=352MiB/s (369MB/s)(10.4GiB/30164msec) slat (usec): min=4, max=240, avg=13.04, stdev= 5.60 clat (nsec): min=786, max=2541.9M, avg=2300337.70, stdev=66491689.20 lat (usec): min=45, max=2541.0k, avg=2313.53, stdev=66491.69 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 59], | 50.00th=[ 64], 60.00th=[ 72], 70.00th=[ 93], | 80.00th=[ 115], 90.00th=[ 135], 95.00th=[ 172], | 99.00th=[ 1188], 99.50th=[ 2606], 99.90th=[ 859833], | 99.95th=[2499806], 99.99th=[2533360] bw ( KiB/s): min=279552, max=776192, per=100.00%, avg=362128.58, stdev=65053.15, samples=60 iops : min= 1092, max= 3032, avg=1414.55, stdev=254.11, samples=60 write: IOPS=354, BW=88.7MiB/s (93.1MB/s)(2677MiB/30164msec) slat (usec): min=19, max=330, avg=53.35, stdev=25.30 clat (msec): min=12, max=1896, avg=131.60, stdev=63.77 lat (msec): min=12, max=1896, avg=131.65, stdev=63.77 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 92], 10.00th=[ 104], 20.00th=[ 111], | 30.00th=[ 117], 40.00th=[ 123], 50.00th=[ 128], 60.00th=[ 132], | 70.00th=[ 138], 80.00th=[ 146], 90.00th=[ 159], 95.00th=[ 174], | 99.00th=[ 253], 99.50th=[ 514], 99.90th=[ 995], 99.95th=[ 1435], | 99.99th=[ 1854] bw ( KiB/s): min=78336, max=177152, per=100.00%, avg=90988.62, stdev=12439.64, samples=60 iops : min= 306, max= 692, avg=355.40, stdev=48.59, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=5.32% lat (usec) : 100=51.94%, 250=20.25%, 500=0.82%, 750=0.32%, 1000=0.21% lat (msec) : 2=0.41%, 4=0.42%, 10=0.01%, 20=0.03%, 50=0.56% lat (msec) : 100=1.03%, 250=18.32%, 500=0.13%, 750=0.08%, 1000=0.03% lat (msec) : 2000=0.05%, >=2000=0.05% cpu : usr=2.41%, sys=3.77%, ctx=48040, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=42438,10708,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=10.4GiB (11.1GB), run=30164-30164msec WRITE: bw=88.7MiB/s (93.1MB/s), 88.7MiB/s-88.7MiB/s (93.1MB/s-93.1MB/s), io=2677MiB (2807MB), run=30164-30164msec Disk stats (read/write): vdh: ios=42434/10708, merge=0/0, ticks=90278/1399778, in_queue=1501962, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946366: Fri Jan 7 14:14:02 2022 read: IOPS=1371, BW=343MiB/s (360MB/s)(10.0GiB/30002msec) clat (usec): min=49, max=14581, avg=91.18, stdev=373.11 lat (usec): min=49, max=14582, avg=91.32, stdev=373.11 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 66], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 123], 99.50th=[ 141], 99.90th=[ 8094], 99.95th=[11207], | 99.99th=[13042] bw ( KiB/s): min=272896, max=598016, per=100.00%, avg=351197.87, stdev=45020.57, samples=60 iops : min= 1066, max= 2336, avg=1371.87, stdev=175.86, samples=60 write: IOPS=346, BW=86.6MiB/s (90.8MB/s)(2599MiB/30002msec) clat (usec): min=796, max=61797, avg=2491.77, stdev=2278.00 lat (usec): min=820, max=61822, avg=2516.30, stdev=2277.83 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1074], 10.00th=[ 1205], 20.00th=[ 1811], | 30.00th=[ 1942], 40.00th=[ 2057], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2474], 80.00th=[ 2671], 90.00th=[ 3195], 95.00th=[ 4490], | 99.00th=[10421], 99.50th=[11076], 99.90th=[19006], 99.95th=[53740], | 99.99th=[61604] bw ( KiB/s): min=75264, max=157696, per=100.00%, avg=88695.47, stdev=10414.40, samples=60 iops : min= 294, max= 616, avg=346.47, stdev=40.68, samples=60 lat (usec) : 50=0.01%, 100=75.25%, 250=4.45%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.55% lat (msec) : 2=6.52%, 4=11.65%, 10=1.12%, 20=0.43%, 100=0.02% cpu : usr=1.67%, sys=3.05%, ctx=51559, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41156,10395,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=343MiB/s (360MB/s), 343MiB/s-343MiB/s (360MB/s-360MB/s), io=10.0GiB (10.8GB), run=30002-30002msec WRITE: bw=86.6MiB/s (90.8MB/s), 86.6MiB/s-86.6MiB/s (90.8MB/s-90.8MB/s), io=2599MiB (2725MB), run=30002-30002msec Disk stats (read/write): vdh: ios=40990/10351, merge=0/0, ticks=529/20736, in_queue=28371, util=94.64% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947436: Fri Jan 7 14:14:40 2022 read: IOPS=1413, BW=353MiB/s (370MB/s)(10.5GiB/30305msec) slat (usec): min=4, max=301, avg=12.33, stdev= 6.66 clat (nsec): min=1043, max=2726.8M, avg=2660992.75, stdev=71244816.67 lat (usec): min=47, max=2726.8k, avg=2673.48, stdev=71244.80 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 51], | 20.00th=[ 54], 30.00th=[ 57], 40.00th=[ 61], | 50.00th=[ 67], 60.00th=[ 77], 70.00th=[ 106], | 80.00th=[ 121], 90.00th=[ 165], 95.00th=[ 594], | 99.00th=[ 6194], 99.50th=[ 7767], 99.90th=[1082131], | 99.95th=[2499806], 99.99th=[2533360] bw ( KiB/s): min=288256, max=844288, per=100.00%, avg=365425.45, stdev=70469.85, samples=60 iops : min= 1126, max= 3298, avg=1427.43, stdev=275.28, samples=60 write: IOPS=356, BW=89.1MiB/s (93.4MB/s)(2699MiB/30305msec) slat (usec): min=19, max=309, avg=52.52, stdev=24.87 clat (msec): min=12, max=3340, avg=269.92, stdev=158.09 lat (msec): min=12, max=3340, avg=269.97, stdev=158.09 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 169], 10.00th=[ 211], 20.00th=[ 236], | 30.00th=[ 249], 40.00th=[ 255], 50.00th=[ 262], 60.00th=[ 268], | 70.00th=[ 275], 80.00th=[ 288], 90.00th=[ 309], 95.00th=[ 355], | 99.00th=[ 584], 99.50th=[ 709], 99.90th=[ 3339], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min=79872, max=167424, per=100.00%, avg=91292.35, stdev=11559.04, samples=60 iops : min= 312, max= 654, avg=356.60, stdev=45.16, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=4.99%, 100=49.24% lat (usec) : 250=19.80%, 500=1.56%, 750=0.62%, 1000=0.44% lat (msec) : 2=1.10%, 4=0.81%, 10=1.14%, 20=0.03%, 50=0.14% lat (msec) : 100=0.52%, 250=6.01%, 500=13.11%, 750=0.26%, 1000=0.04% lat (msec) : 2000=0.06%, >=2000=0.09% cpu : usr=2.32%, sys=3.61%, ctx=45998, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42825,10795,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=353MiB/s (370MB/s), 353MiB/s-353MiB/s (370MB/s-370MB/s), io=10.5GiB (11.2GB), run=30305-30305msec WRITE: bw=89.1MiB/s (93.4MB/s), 89.1MiB/s-89.1MiB/s (93.4MB/s-93.4MB/s), io=2699MiB (2830MB), run=30305-30305msec Disk stats (read/write): vdh: ios=42822/10726, merge=0/0, ticks=104747/2872661, in_queue=3005556, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948010: Fri Jan 7 14:15:16 2022 read: IOPS=1361, BW=340MiB/s (357MB/s)(9.97GiB/30001msec) clat (usec): min=51, max=23242, avg=96.35, stdev=471.42 lat (usec): min=51, max=23242, avg=96.49, stdev=471.42 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 137], 99.50th=[ 161], 99.90th=[ 8848], 99.95th=[11994], | 99.99th=[19792] bw ( KiB/s): min=259072, max=591872, per=100.00%, avg=348475.73, stdev=45383.79, samples=60 iops : min= 1012, max= 2312, avg=1361.20, stdev=177.29, samples=60 write: IOPS=343, BW=85.9MiB/s (90.1MB/s)(2578MiB/30001msec) clat (usec): min=803, max=62211, avg=2493.28, stdev=2312.84 lat (usec): min=828, max=62224, avg=2519.39, stdev=2312.50 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1057], 10.00th=[ 1205], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2638], 90.00th=[ 3130], 95.00th=[ 4490], | 99.00th=[10683], 99.50th=[11600], 99.90th=[45351], 99.95th=[53216], | 99.99th=[54789] bw ( KiB/s): min=65536, max=155648, per=99.99%, avg=87970.13, stdev=10841.30, samples=60 iops : min= 256, max= 608, avg=343.63, stdev=42.35, samples=60 lat (usec) : 100=73.83%, 250=5.80%, 500=0.06%, 750=0.02%, 1000=0.59% lat (msec) : 2=6.71%, 4=11.40%, 10=1.06%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.75%, sys=3.03%, ctx=51159, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40837,10310,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=340MiB/s (357MB/s), 340MiB/s-340MiB/s (357MB/s-357MB/s), io=9.97GiB (10.7GB), run=30001-30001msec WRITE: bw=85.9MiB/s (90.1MB/s), 85.9MiB/s-85.9MiB/s (90.1MB/s-90.1MB/s), io=2578MiB (2703MB), run=30001-30001msec Disk stats (read/write): vdh: ios=40666/10259, merge=0/0, ticks=650/20537, in_queue=28419, util=94.81% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949342: Fri Jan 7 14:15:54 2022 read: IOPS=1409, BW=352MiB/s (369MB/s)(10.4GiB/30355msec) slat (usec): min=4, max=372983, avg=547.67, stdev=4808.93 clat (msec): min=2, max=2786, avg=46.91, stdev=73.05 lat (msec): min=2, max=2786, avg=47.46, stdev=73.18 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 13], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 39], 50.00th=[ 42], 60.00th=[ 46], | 70.00th=[ 50], 80.00th=[ 56], 90.00th=[ 67], 95.00th=[ 81], | 99.00th=[ 313], 99.50th=[ 359], 99.90th=[ 953], 99.95th=[ 1972], | 99.99th=[ 2567] bw ( KiB/s): min=264192, max=894464, per=100.00%, avg=364523.77, stdev=80478.18, samples=60 iops : min= 1032, max= 3494, avg=1423.88, stdev=314.37, samples=60 write: IOPS=355, BW=88.8MiB/s (93.1MB/s)(2696MiB/30355msec) slat (usec): min=19, max=87653, avg=597.73, stdev=3324.15 clat (msec): min=21, max=3055, avg=373.93, stdev=172.73 lat (msec): min=21, max=3055, avg=374.53, stdev=172.74 clat percentiles (msec): | 1.00th=[ 68], 5.00th=[ 209], 10.00th=[ 292], 20.00th=[ 326], | 30.00th=[ 347], 40.00th=[ 355], 50.00th=[ 368], 60.00th=[ 376], | 70.00th=[ 388], 80.00th=[ 401], 90.00th=[ 435], 95.00th=[ 523], | 99.00th=[ 768], 99.50th=[ 1217], 99.90th=[ 2970], 99.95th=[ 2970], | 99.99th=[ 2970] bw ( KiB/s): min=72704, max=167424, per=99.97%, avg=90928.23, stdev=12186.64, samples=60 iops : min= 284, max= 654, avg=355.13, stdev=47.63, samples=60 lat (msec) : 4=0.27%, 10=5.59%, 20=5.51%, 50=45.21%, 100=22.33% lat (msec) : 250=1.41%, 500=18.47%, 750=0.92%, 1000=0.06%, 2000=0.14% lat (msec) : >=2000=0.09% cpu : usr=3.25%, sys=1.88%, ctx=5706, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42779,10785,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=10.4GiB (11.2GB), run=30355-30355msec WRITE: bw=88.8MiB/s (93.1MB/s), 88.8MiB/s-88.8MiB/s (93.1MB/s-93.1MB/s), io=2696MiB (2827MB), run=30355-30355msec Disk stats (read/write): vdh: ios=42776/10706, merge=0/0, ticks=107462/3534769, in_queue=3692500, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950045: Fri Jan 7 14:16:30 2022 read: IOPS=913, BW=457MiB/s (479MB/s)(13.4GiB/30001msec) clat (usec): min=117, max=18631, avg=186.24, stdev=516.50 lat (usec): min=117, max=18631, avg=186.39, stdev=516.50 clat percentiles (usec): | 1.00th=[ 129], 5.00th=[ 135], 10.00th=[ 137], 20.00th=[ 139], | 30.00th=[ 145], 40.00th=[ 149], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 172], 80.00th=[ 184], 90.00th=[ 202], 95.00th=[ 221], | 99.00th=[ 273], 99.50th=[ 306], 99.90th=[10945], 99.95th=[13304], | 99.99th=[16450] bw ( KiB/s): min=376832, max=624640, per=100.00%, avg=467765.20, stdev=57448.03, samples=60 iops : min= 736, max= 1220, avg=913.60, stdev=112.20, samples=60 write: IOPS=230, BW=115MiB/s (121MB/s)(3452MiB/30001msec) clat (usec): min=1446, max=62061, avg=3547.20, stdev=2575.16 lat (usec): min=1488, max=62107, avg=3597.04, stdev=2574.78 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1713], 10.00th=[ 1795], 20.00th=[ 2311], | 30.00th=[ 2606], 40.00th=[ 2868], 50.00th=[ 3064], 60.00th=[ 3228], | 70.00th=[ 3458], 80.00th=[ 3851], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[11731], 99.50th=[14484], 99.90th=[53740], 99.95th=[54264], | 99.99th=[62129] bw ( KiB/s): min=92160, max=171008, per=100.00%, avg=117807.42, stdev=11970.55, samples=60 iops : min= 180, max= 334, avg=230.08, stdev=23.38, samples=60 lat (usec) : 250=78.51%, 500=1.17%, 750=0.02%, 1000=0.01% lat (msec) : 2=2.91%, 4=13.61%, 10=3.16%, 20=0.60%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.86%, sys=2.55%, ctx=34322, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27410,6903,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=457MiB/s (479MB/s), 457MiB/s-457MiB/s (479MB/s-479MB/s), io=13.4GiB (14.4GB), run=30001-30001msec WRITE: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=3452MiB (3619MB), run=30001-30001msec Disk stats (read/write): vdh: ios=27293/6879, merge=0/0, ticks=604/20649, in_queue=28445, util=94.90% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950865: Fri Jan 7 14:17:07 2022 read: IOPS=973, BW=487MiB/s (511MB/s)(14.3GiB/30002msec) slat (usec): min=17, max=111, avg=27.89, stdev= 6.32 clat (usec): min=41, max=24010, avg=128.76, stdev=563.14 lat (usec): min=95, max=24033, avg=156.84, stdev=563.23 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 85], 10.00th=[ 87], 20.00th=[ 89], | 30.00th=[ 92], 40.00th=[ 95], 50.00th=[ 99], 60.00th=[ 104], | 70.00th=[ 111], 80.00th=[ 118], 90.00th=[ 128], 95.00th=[ 139], | 99.00th=[ 176], 99.50th=[ 204], 99.90th=[11731], 99.95th=[14222], | 99.99th=[16712] bw ( KiB/s): min=372736, max=658432, per=100.00%, avg=498685.25, stdev=57242.19, samples=60 iops : min= 728, max= 1286, avg=973.98, stdev=111.81, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3691MiB/30002msec) slat (usec): min=39, max=347, avg=75.34, stdev=19.63 clat (usec): min=1369, max=55878, avg=3359.68, stdev=2334.29 lat (usec): min=1444, max=55951, avg=3435.24, stdev=2333.97 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2311], | 30.00th=[ 2573], 40.00th=[ 2835], 50.00th=[ 3032], 60.00th=[ 3228], | 70.00th=[ 3392], 80.00th=[ 3687], 90.00th=[ 4817], 95.00th=[ 6521], | 99.00th=[11469], 99.50th=[12256], 99.90th=[20055], 99.95th=[54264], | 99.99th=[55837] bw ( KiB/s): min=103424, max=190464, per=100.00%, avg=125972.95, stdev=11319.03, samples=60 iops : min= 202, max= 372, avg=246.03, stdev=22.11, samples=60 lat (usec) : 50=0.01%, 100=42.52%, 250=37.06%, 500=0.06% lat (msec) : 2=3.29%, 4=14.10%, 10=2.44%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=2.07%, sys=4.28%, ctx=36616, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29219,7382,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=487MiB/s (511MB/s), 487MiB/s-487MiB/s (511MB/s-511MB/s), io=14.3GiB (15.3GB), run=30002-30002msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3691MiB (3870MB), run=30002-30002msec Disk stats (read/write): vdh: ios=29130/7357, merge=0/0, ticks=708/20896, in_queue=28339, util=94.54% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951813: Fri Jan 7 14:17:44 2022 read: IOPS=984, BW=492MiB/s (516MB/s)(14.4GiB/30001msec) clat (usec): min=78, max=48993, avg=135.54, stdev=607.14 lat (usec): min=78, max=48993, avg=135.69, stdev=607.15 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 96], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 125], 90.00th=[ 135], 95.00th=[ 145], | 99.00th=[ 180], 99.50th=[ 204], 99.90th=[11207], 99.95th=[14222], | 99.99th=[18220] bw ( KiB/s): min=350208, max=632832, per=100.00%, avg=504029.87, stdev=57608.41, samples=60 iops : min= 684, max= 1236, avg=984.43, stdev=112.52, samples=60 write: IOPS=248, BW=124MiB/s (130MB/s)(3728MiB/30001msec) clat (usec): min=1403, max=58874, avg=3427.70, stdev=2397.15 lat (usec): min=1451, max=58921, avg=3478.32, stdev=2396.88 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3752], 90.00th=[ 4424], 95.00th=[ 6325], | 99.00th=[11338], 99.50th=[12256], 99.90th=[45876], 99.95th=[54264], | 99.99th=[58983] bw ( KiB/s): min=108544, max=179200, per=99.99%, avg=127214.93, stdev=9957.30, samples=60 iops : min= 212, max= 350, avg=248.47, stdev=19.45, samples=60 lat (usec) : 100=29.26%, 250=50.37%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.80%, 4=14.46%, 10=2.58%, 20=0.44%, 50=0.02% lat (msec) : 100=0.02% cpu : usr=1.92%, sys=2.87%, ctx=37003, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29533,7455,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=492MiB/s (516MB/s), 492MiB/s-492MiB/s (516MB/s-516MB/s), io=14.4GiB (15.5GB), run=30001-30001msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3728MiB (3909MB), run=30001-30001msec Disk stats (read/write): vdh: ios=29447/7430, merge=0/0, ticks=715/21482, in_queue=28353, util=94.58% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952334: Fri Jan 7 14:18:21 2022 read: IOPS=999, BW=500MiB/s (524MB/s)(14.7GiB/30033msec) slat (usec): min=6, max=115, avg=21.90, stdev= 8.01 clat (nsec): min=1241, max=73687k, avg=189099.92, stdev=1616081.70 lat (usec): min=69, max=73707, avg=211.16, stdev=1616.15 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 71], 10.00th=[ 72], 20.00th=[ 75], | 30.00th=[ 78], 40.00th=[ 82], 50.00th=[ 88], 60.00th=[ 99], | 70.00th=[ 117], 80.00th=[ 180], 90.00th=[ 202], 95.00th=[ 233], | 99.00th=[ 375], 99.50th=[ 685], 99.90th=[31851], 99.95th=[41157], | 99.99th=[55313] bw ( KiB/s): min=373760, max=732160, per=100.00%, avg=512205.10, stdev=59082.06, samples=60 iops : min= 730, max= 1430, avg=1000.40, stdev=115.33, samples=60 write: IOPS=252, BW=126MiB/s (132MB/s)(3788MiB/30033msec) slat (usec): min=35, max=321, avg=92.60, stdev=41.67 clat (msec): min=2, max=133, avg=38.70, stdev=15.31 lat (msec): min=2, max=133, avg=38.79, stdev=15.31 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 20], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 46], 80.00th=[ 51], 90.00th=[ 57], 95.00th=[ 63], | 99.00th=[ 85], 99.50th=[ 99], 99.90th=[ 115], 99.95th=[ 122], | 99.99th=[ 134] bw ( KiB/s): min=106496, max=202752, per=99.98%, avg=129130.83, stdev=12353.99, samples=60 iops : min= 208, max= 396, avg=252.20, stdev=24.14, samples=60 lat (usec) : 2=0.01%, 50=0.02%, 100=48.48%, 250=28.59%, 500=2.21% lat (usec) : 750=0.17%, 1000=0.08% lat (msec) : 2=0.12%, 4=0.01%, 10=0.59%, 20=1.50%, 50=14.21% lat (msec) : 100=3.94%, 250=0.08% cpu : usr=2.65%, sys=3.61%, ctx=35712, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=30011,7576,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=14.7GiB (15.7GB), run=30033-30033msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3788MiB (3972MB), run=30033-30033msec Disk stats (read/write): vdh: ios=29897/7540, merge=0/0, ticks=2095/287890, in_queue=297289, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953206: Fri Jan 7 14:18:59 2022 read: IOPS=990, BW=495MiB/s (520MB/s)(14.5GiB/30002msec) clat (usec): min=77, max=61425, avg=135.13, stdev=644.95 lat (usec): min=77, max=61425, avg=135.28, stdev=644.95 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 109], | 70.00th=[ 116], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 141], | 99.00th=[ 169], 99.50th=[ 192], 99.90th=[11600], 99.95th=[14222], | 99.99th=[16450] bw ( KiB/s): min=406528, max=655360, per=100.00%, avg=507426.53, stdev=50023.59, samples=60 iops : min= 794, max= 1280, avg=991.07, stdev=97.70, samples=60 write: IOPS=250, BW=125MiB/s (131MB/s)(3752MiB/30002msec) clat (usec): min=1381, max=62680, avg=3405.39, stdev=2314.26 lat (usec): min=1408, max=62729, avg=3454.13, stdev=2313.86 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1647], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3458], 80.00th=[ 3720], 90.00th=[ 4424], 95.00th=[ 6325], | 99.00th=[11469], 99.50th=[13829], 99.90th=[32900], 99.95th=[53740], | 99.99th=[62653] bw ( KiB/s): min=104448, max=188416, per=100.00%, avg=128038.37, stdev=11091.07, samples=60 iops : min= 204, max= 368, avg=250.07, stdev=21.66, samples=60 lat (usec) : 100=30.82%, 250=48.80%, 500=0.04%, 750=0.01% lat (msec) : 2=2.83%, 4=14.55%, 10=2.47%, 20=0.46%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.77%, sys=2.93%, ctx=37244, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29731,7503,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=495MiB/s (520MB/s), 495MiB/s-495MiB/s (520MB/s-520MB/s), io=14.5GiB (15.6GB), run=30002-30002msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3752MiB (3934MB), run=30002-30002msec Disk stats (read/write): vdh: ios=29603/7467, merge=0/0, ticks=746/21458, in_queue=28359, util=94.59% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954251: Fri Jan 7 14:19:36 2022 read: IOPS=1003, BW=502MiB/s (526MB/s)(14.8GiB/30211msec) slat (usec): min=5, max=230, avg=15.75, stdev= 6.82 clat (usec): min=22, max=191685, avg=367.01, stdev=4188.10 lat (usec): min=71, max=191698, avg=382.91, stdev=4188.10 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 72], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 87], | 70.00th=[ 104], 80.00th=[ 184], 90.00th=[ 206], 95.00th=[ 277], | 99.00th=[ 3916], 99.50th=[ 6718], 99.90th=[ 80217], 99.95th=[100140], | 99.99th=[145753] bw ( KiB/s): min=392192, max=991232, per=100.00%, avg=517444.40, stdev=85544.34, samples=60 iops : min= 766, max= 1936, avg=1010.63, stdev=167.08, samples=60 write: IOPS=253, BW=127MiB/s (133MB/s)(3823MiB/30211msec) slat (usec): min=34, max=515, avg=92.00, stdev=45.30 clat (msec): min=16, max=2478, avg=195.91, stdev=107.17 lat (msec): min=16, max=2478, avg=196.00, stdev=107.17 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 73], 10.00th=[ 84], 20.00th=[ 107], | 30.00th=[ 136], 40.00th=[ 169], 50.00th=[ 203], 60.00th=[ 224], | 70.00th=[ 241], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 317], | 99.00th=[ 430], 99.50th=[ 481], 99.90th=[ 1385], 99.95th=[ 2039], | 99.99th=[ 2467] bw ( KiB/s): min=108544, max=221184, per=100.00%, avg=129642.83, stdev=14323.74, samples=60 iops : min= 212, max= 432, avg=253.20, stdev=27.97, samples=60 lat (usec) : 50=0.03%, 100=54.90%, 250=20.44%, 500=1.66%, 750=0.42% lat (usec) : 1000=0.33% lat (msec) : 2=0.90%, 4=0.37%, 10=0.63%, 20=0.02%, 50=0.21% lat (msec) : 100=3.38%, 250=11.51%, 500=5.12%, 750=0.02%, 1000=0.02% lat (msec) : 2000=0.03%, >=2000=0.01% cpu : usr=2.72%, sys=2.86%, ctx=35424, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=30318,7646,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=502MiB/s (526MB/s), 502MiB/s-502MiB/s (526MB/s-526MB/s), io=14.8GiB (15.9GB), run=30211-30211msec WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=3823MiB (4009MB), run=30211-30211msec Disk stats (read/write): vdh: ios=30318/7600, merge=0/0, ticks=6934/1478743, in_queue=1503730, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954764: Fri Jan 7 14:20:12 2022 read: IOPS=980, BW=490MiB/s (514MB/s)(14.4GiB/30001msec) clat (usec): min=86, max=17070, avg=143.54, stdev=504.87 lat (usec): min=86, max=17070, avg=143.69, stdev=504.87 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 101], 10.00th=[ 103], 20.00th=[ 106], | 30.00th=[ 109], 40.00th=[ 112], 50.00th=[ 116], 60.00th=[ 122], | 70.00th=[ 129], 80.00th=[ 137], 90.00th=[ 147], 95.00th=[ 157], | 99.00th=[ 192], 99.50th=[ 217], 99.90th=[11338], 99.95th=[12911], | 99.99th=[16319] bw ( KiB/s): min=366592, max=655360, per=100.00%, avg=501930.67, stdev=57467.58, samples=60 iops : min= 716, max= 1280, avg=980.33, stdev=112.24, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3712MiB/30001msec) clat (usec): min=1474, max=56041, avg=3413.76, stdev=2337.33 lat (usec): min=1523, max=56094, avg=3463.45, stdev=2336.87 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2376], | 30.00th=[ 2671], 40.00th=[ 2933], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3752], 90.00th=[ 4555], 95.00th=[ 6456], | 99.00th=[11469], 99.50th=[12125], 99.90th=[23462], 99.95th=[54789], | 99.99th=[55837] bw ( KiB/s): min=105472, max=188416, per=99.99%, avg=126685.87, stdev=10552.99, samples=60 iops : min= 206, max= 368, avg=247.43, stdev=20.61, samples=60 lat (usec) : 100=3.48%, 250=76.10%, 500=0.08%, 750=0.01% lat (msec) : 2=3.12%, 4=14.10%, 10=2.60%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.82%, sys=2.95%, ctx=36845, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29410,7424,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=490MiB/s (514MB/s), 490MiB/s-490MiB/s (514MB/s-514MB/s), io=14.4GiB (15.4GB), run=30001-30001msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3712MiB (3892MB), run=30001-30001msec Disk stats (read/write): vdh: ios=29233/7386, merge=0/0, ticks=630/21194, in_queue=28324, util=94.49% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=955884: Fri Jan 7 14:20:50 2022 read: IOPS=1005, BW=503MiB/s (527MB/s)(14.9GiB/30447msec) slat (usec): min=5, max=460, avg=14.43, stdev=10.18 clat (nsec): min=1670, max=456184k, avg=858058.81, stdev=5738432.74 lat (usec): min=69, max=456191, avg=872.64, stdev=5738.37 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 70], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 83], 60.00th=[ 96], | 70.00th=[ 182], 80.00th=[ 200], 90.00th=[ 766], 95.00th=[ 4424], | 99.00th=[ 12911], 99.50th=[ 15401], 99.90th=[ 92799], 99.95th=[130548], | 99.99th=[175113] bw ( KiB/s): min=142336, max=897024, per=100.00%, avg=522545.72, stdev=139558.44, samples=60 iops : min= 278, max= 1752, avg=1020.58, stdev=272.58, samples=60 write: IOPS=254, BW=127MiB/s (133MB/s)(3872MiB/30447msec) slat (usec): min=31, max=628, avg=93.03, stdev=47.15 clat (msec): min=13, max=2647, avg=389.41, stdev=136.28 lat (msec): min=13, max=2648, avg=389.51, stdev=136.28 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 222], 10.00th=[ 264], 20.00th=[ 296], | 30.00th=[ 326], 40.00th=[ 355], 50.00th=[ 384], 60.00th=[ 414], | 70.00th=[ 439], 80.00th=[ 468], 90.00th=[ 514], 95.00th=[ 584], | 99.00th=[ 793], 99.50th=[ 877], 99.90th=[ 1636], 99.95th=[ 1888], | 99.99th=[ 2635] bw ( KiB/s): min=44032, max=214016, per=100.00%, avg=130466.57, stdev=30924.36, samples=60 iops : min= 86, max= 418, avg=254.80, stdev=60.40, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=0.03%, 100=48.92%, 250=18.69% lat (usec) : 500=2.70%, 750=1.41%, 1000=0.77% lat (msec) : 2=1.53%, 4=1.50%, 10=2.71%, 20=1.39%, 50=0.05% lat (msec) : 100=0.35%, 250=1.20%, 500=16.34%, 750=2.12%, 1000=0.22% lat (msec) : 2000=0.06%, >=2000=0.01% cpu : usr=2.65%, sys=2.65%, ctx=32006, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30616,7744,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=503MiB/s (527MB/s), 503MiB/s-503MiB/s (527MB/s-527MB/s), io=14.9GiB (16.1GB), run=30447-30447msec WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=3872MiB (4060MB), run=30447-30447msec Disk stats (read/write): vdh: ios=30616/7699, merge=0/0, ticks=19088/2967853, in_queue=3016819, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956758: Fri Jan 7 14:21:27 2022 read: IOPS=995, BW=498MiB/s (522MB/s)(14.6GiB/30001msec) clat (usec): min=77, max=20117, avg=131.54, stdev=546.81 lat (usec): min=77, max=20117, avg=131.69, stdev=546.81 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 91], 10.00th=[ 92], 20.00th=[ 94], | 30.00th=[ 96], 40.00th=[ 98], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 131], 95.00th=[ 141], | 99.00th=[ 169], 99.50th=[ 194], 99.90th=[11731], 99.95th=[14091], | 99.99th=[17171] bw ( KiB/s): min=364544, max=645120, per=100.00%, avg=509712.88, stdev=53188.37, samples=60 iops : min= 712, max= 1260, avg=995.52, stdev=103.89, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3770MiB/30001msec) clat (usec): min=1451, max=70955, avg=3399.70, stdev=2288.18 lat (usec): min=1502, max=70988, avg=3447.76, stdev=2287.59 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1647], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3458], 80.00th=[ 3720], 90.00th=[ 4359], 95.00th=[ 6325], | 99.00th=[11338], 99.50th=[12387], 99.90th=[17171], 99.95th=[54264], | 99.99th=[70779] bw ( KiB/s): min=109568, max=184320, per=100.00%, avg=128673.85, stdev=10769.61, samples=60 iops : min= 214, max= 360, avg=251.30, stdev=21.04, samples=60 lat (usec) : 100=37.12%, 250=42.48%, 500=0.06%, 750=0.01% lat (msec) : 2=2.71%, 4=14.74%, 10=2.41%, 20=0.45%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.87%, sys=2.80%, ctx=37414, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29864,7540,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=14.6GiB (15.7GB), run=30001-30001msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3770MiB (3953MB), run=30001-30001msec Disk stats (read/write): vdh: ios=29718/7494, merge=0/0, ticks=687/21496, in_queue=28360, util=94.60% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957719: Fri Jan 7 14:22:04 2022 read: IOPS=1004, BW=502MiB/s (527MB/s)(14.9GiB/30444msec) slat (usec): min=6, max=462837, avg=738.51, stdev=5816.96 clat (msec): min=9, max=547, avg=63.88, stdev=54.70 lat (msec): min=9, max=556, avg=64.61, stdev=55.02 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 18], 10.00th=[ 24], 20.00th=[ 36], | 30.00th=[ 45], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 64], | 70.00th=[ 71], 80.00th=[ 81], 90.00th=[ 92], 95.00th=[ 110], | 99.00th=[ 468], 99.50th=[ 493], 99.90th=[ 542], 99.95th=[ 542], | 99.99th=[ 550] bw ( KiB/s): min=34816, max=1233920, per=100.00%, avg=521023.78, stdev=196896.17, samples=60 iops : min= 68, max= 2410, avg=1017.62, stdev=384.57, samples=60 write: IOPS=254, BW=127MiB/s (133MB/s)(3869MiB/30444msec) slat (usec): min=47, max=485484, avg=947.35, stdev=8322.29 clat (msec): min=19, max=2680, avg=530.17, stdev=174.39 lat (msec): min=19, max=2680, avg=531.11, stdev=174.24 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 288], 10.00th=[ 376], 20.00th=[ 426], | 30.00th=[ 464], 40.00th=[ 502], 50.00th=[ 531], 60.00th=[ 558], | 70.00th=[ 584], 80.00th=[ 609], 90.00th=[ 659], 95.00th=[ 818], | 99.00th=[ 1028], 99.50th=[ 1183], 99.90th=[ 1972], 99.95th=[ 2668], | 99.99th=[ 2668] bw ( KiB/s): min=10240, max=226304, per=99.61%, avg=129624.95, stdev=35563.95, samples=60 iops : min= 20, max= 442, avg=253.17, stdev=69.47, samples=60 lat (msec) : 10=0.01%, 20=5.56%, 50=25.05%, 100=43.92%, 250=5.12% lat (msec) : 500=7.82%, 750=11.22%, 1000=1.04%, 2000=0.25%, >=2000=0.02% cpu : usr=4.42%, sys=1.84%, ctx=3921, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30585,7738,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=502MiB/s (527MB/s), 502MiB/s-502MiB/s (527MB/s-527MB/s), io=14.9GiB (16.0GB), run=30444-30444msec WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=3869MiB (4057MB), run=30444-30444msec Disk stats (read/write): vdh: ios=30585/7697, merge=0/0, ticks=66825/3608306, in_queue=3715407, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958706: Fri Jan 7 14:22:41 2022 read: IOPS=609, BW=609MiB/s (639MB/s)(17.8GiB/30001msec) clat (usec): min=138, max=30888, avg=243.05, stdev=986.75 lat (usec): min=138, max=30888, avg=243.21, stdev=986.75 clat percentiles (usec): | 1.00th=[ 149], 5.00th=[ 155], 10.00th=[ 157], 20.00th=[ 159], | 30.00th=[ 163], 40.00th=[ 167], 50.00th=[ 172], 60.00th=[ 178], | 70.00th=[ 190], 80.00th=[ 204], 90.00th=[ 223], 95.00th=[ 241], | 99.00th=[ 334], 99.50th=[ 478], 99.90th=[17957], 99.95th=[19530], | 99.99th=[25560] bw ( KiB/s): min=421888, max=886784, per=100.00%, avg=624258.17, stdev=100975.90, samples=59 iops : min= 412, max= 866, avg=609.63, stdev=98.61, samples=59 write: IOPS=154, BW=154MiB/s (162MB/s)(4627MiB/30001msec) clat (usec): min=2552, max=63817, avg=5414.68, stdev=2968.31 lat (usec): min=2617, max=63921, avg=5514.51, stdev=2967.13 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2704], 10.00th=[ 2737], 20.00th=[ 3490], | 30.00th=[ 4359], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8586], 95.00th=[11076], | 99.00th=[15795], 99.50th=[16909], 99.90th=[20055], 99.95th=[55837], | 99.99th=[63701] bw ( KiB/s): min=122880, max=221184, per=99.81%, avg=157626.58, stdev=13552.74, samples=59 iops : min= 120, max= 216, avg=153.93, stdev=13.24, samples=59 lat (usec) : 250=76.82%, 500=2.58%, 750=0.05% lat (msec) : 2=0.01%, 4=5.14%, 10=13.73%, 20=1.62%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.98%, sys=2.49%, ctx=22908, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18274,4627,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=609MiB/s (639MB/s), 609MiB/s-609MiB/s (639MB/s-639MB/s), io=17.8GiB (19.2GB), run=30001-30001msec WRITE: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=4627MiB (4852MB), run=30001-30001msec Disk stats (read/write): vdh: ios=18213/4606, merge=0/0, ticks=1080/22368, in_queue=28544, util=95.25% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959283: Fri Jan 7 14:23:18 2022 read: IOPS=605, BW=605MiB/s (635MB/s)(17.7GiB/30004msec) slat (usec): min=29, max=131, avg=43.15, stdev= 8.97 clat (usec): min=118, max=27540, avg=232.42, stdev=956.48 lat (usec): min=162, max=27580, avg=275.77, stdev=956.64 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 172], | 70.00th=[ 184], 80.00th=[ 196], 90.00th=[ 215], 95.00th=[ 237], | 99.00th=[ 347], 99.50th=[ 586], 99.90th=[17433], 99.95th=[20055], | 99.99th=[22676] bw ( KiB/s): min=434176, max=784384, per=100.00%, avg=620051.15, stdev=82281.09, samples=60 iops : min= 424, max= 766, avg=605.52, stdev=80.35, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4590MiB/30004msec) slat (usec): min=69, max=478, avg=134.60, stdev=29.37 clat (usec): min=2550, max=59833, avg=5301.41, stdev=3033.26 lat (usec): min=2669, max=59912, avg=5436.24, stdev=3033.19 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3228], | 30.00th=[ 4113], 40.00th=[ 4621], 50.00th=[ 4948], 60.00th=[ 5211], | 70.00th=[ 5473], 80.00th=[ 5800], 90.00th=[ 8455], 95.00th=[10945], | 99.00th=[15270], 99.50th=[16909], 99.90th=[53740], 99.95th=[55837], | 99.99th=[60031] bw ( KiB/s): min=133120, max=217088, per=100.00%, avg=156643.20, stdev=12401.68, samples=60 iops : min= 130, max= 212, avg=152.97, stdev=12.11, samples=60 lat (usec) : 250=76.93%, 500=2.47%, 750=0.07%, 1000=0.01% lat (msec) : 2=0.02%, 4=5.75%, 10=13.19%, 20=1.50%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=2.09%, sys=3.70%, ctx=22764, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18165,4590,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=605MiB/s (635MB/s), 605MiB/s-605MiB/s (635MB/s-635MB/s), io=17.7GiB (19.0GB), run=30004-30004msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4590MiB (4813MB), run=30004-30004msec Disk stats (read/write): vdh: ios=18098/4570, merge=0/0, ticks=1039/21781, in_queue=28511, util=95.11% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960013: Fri Jan 7 14:23:55 2022 read: IOPS=615, BW=616MiB/s (646MB/s)(18.0GiB/30004msec) clat (usec): min=130, max=54628, avg=236.85, stdev=1139.20 lat (usec): min=130, max=54628, avg=237.01, stdev=1139.20 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 208], 95.00th=[ 229], | 99.00th=[ 326], 99.50th=[ 506], 99.90th=[19268], 99.95th=[19792], | 99.99th=[48497] bw ( KiB/s): min=415744, max=868352, per=100.00%, avg=630634.32, stdev=94205.67, samples=60 iops : min= 406, max= 848, avg=615.85, stdev=92.00, samples=60 write: IOPS=155, BW=156MiB/s (163MB/s)(4675MiB/30004msec) clat (usec): min=2539, max=64656, avg=5374.69, stdev=2786.59 lat (usec): min=2599, max=64767, avg=5472.33, stdev=2785.58 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3589], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5866], 90.00th=[ 8029], 95.00th=[10683], | 99.00th=[15270], 99.50th=[16450], 99.90th=[20579], 99.95th=[55313], | 99.99th=[64750] bw ( KiB/s): min=131072, max=210944, per=100.00%, avg=159544.12, stdev=12153.39, samples=60 iops : min= 128, max= 206, avg=155.80, stdev=11.87, samples=60 lat (usec) : 250=77.31%, 500=2.10%, 750=0.06% lat (msec) : 2=0.01%, 4=4.89%, 10=14.05%, 20=1.53%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.95%, sys=2.51%, ctx=23163, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18475,4675,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=616MiB/s (646MB/s), 616MiB/s-616MiB/s (646MB/s-646MB/s), io=18.0GiB (19.4GB), run=30004-30004msec WRITE: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4675MiB (4902MB), run=30004-30004msec Disk stats (read/write): vdh: ios=18369/4656, merge=0/0, ticks=1217/22422, in_queue=28511, util=95.09% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961049: Fri Jan 7 14:24:32 2022 read: IOPS=574, BW=574MiB/s (602MB/s)(16.9GiB/30071msec) slat (usec): min=9, max=137, avg=25.02, stdev=13.65 clat (usec): min=47, max=90922, avg=339.01, stdev=2494.60 lat (usec): min=112, max=90973, avg=364.20, stdev=2494.78 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 113], | 30.00th=[ 116], 40.00th=[ 120], 50.00th=[ 127], 60.00th=[ 149], | 70.00th=[ 188], 80.00th=[ 326], 90.00th=[ 347], 95.00th=[ 404], | 99.00th=[ 611], 99.50th=[ 2409], 99.90th=[44827], 99.95th=[52691], | 99.99th=[76022] bw ( KiB/s): min=421888, max=948224, per=100.00%, avg=589569.67, stdev=94114.39, samples=60 iops : min= 412, max= 926, avg=575.75, stdev=91.91, samples=60 write: IOPS=145, BW=146MiB/s (153MB/s)(4383MiB/30071msec) slat (usec): min=61, max=644, avg=169.67, stdev=85.77 clat (msec): min=4, max=203, avg=66.98, stdev=26.42 lat (msec): min=4, max=203, avg=67.15, stdev=26.42 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 19], 10.00th=[ 27], 20.00th=[ 45], | 30.00th=[ 58], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 75], | 70.00th=[ 81], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 108], | 99.00th=[ 133], 99.50th=[ 144], 99.90th=[ 169], 99.95th=[ 176], | 99.99th=[ 203] bw ( KiB/s): min=126976, max=239616, per=100.00%, avg=149269.72, stdev=14444.37, samples=60 iops : min= 124, max= 234, avg=145.77, stdev=14.11, samples=60 lat (usec) : 50=0.01%, 100=0.17%, 250=61.33%, 500=16.90%, 750=0.68% lat (usec) : 1000=0.08% lat (msec) : 2=0.10%, 4=0.14%, 10=0.06%, 20=1.19%, 50=3.59% lat (msec) : 100=14.07%, 250=1.68% cpu : usr=2.64%, sys=2.25%, ctx=21123, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17272,4383,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=574MiB/s (602MB/s), 574MiB/s-574MiB/s (602MB/s-602MB/s), io=16.9GiB (18.1GB), run=30071-30071msec WRITE: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=4383MiB (4596MB), run=30071-30071msec Disk stats (read/write): vdh: ios=18867/4782, merge=0/0, ticks=2729/317900, in_queue=327107, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961507: Fri Jan 7 14:25:09 2022 read: IOPS=616, BW=616MiB/s (646MB/s)(18.1GiB/30016msec) clat (usec): min=129, max=27406, avg=229.09, stdev=1021.86 lat (usec): min=129, max=27407, avg=229.24, stdev=1021.86 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 153], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 172], 80.00th=[ 186], 90.00th=[ 198], 95.00th=[ 210], | 99.00th=[ 265], 99.50th=[ 429], 99.90th=[18220], 99.95th=[19792], | 99.99th=[24511] bw ( KiB/s): min=399360, max=902925, per=100.00%, avg=631228.12, stdev=101774.50, samples=60 iops : min= 390, max= 881, avg=616.42, stdev=99.36, samples=60 write: IOPS=155, BW=156MiB/s (163MB/s)(4678MiB/30016msec) clat (usec): min=2553, max=56426, avg=5404.84, stdev=2820.46 lat (usec): min=2616, max=56527, avg=5501.86, stdev=2819.30 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3720], | 30.00th=[ 4490], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5538], 80.00th=[ 5866], 90.00th=[ 7767], 95.00th=[10683], | 99.00th=[15008], 99.50th=[17171], 99.90th=[21627], 99.95th=[55837], | 99.99th=[56361] bw ( KiB/s): min=126976, max=217088, per=100.00%, avg=159657.80, stdev=12465.82, samples=60 iops : min= 124, max= 212, avg=155.90, stdev=12.17, samples=60 lat (usec) : 250=78.83%, 500=0.63%, 750=0.01% lat (msec) : 2=0.01%, 4=4.67%, 10=14.32%, 20=1.49%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.95%, sys=2.43%, ctx=23185, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18491,4678,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=616MiB/s (646MB/s), 616MiB/s-616MiB/s (646MB/s-646MB/s), io=18.1GiB (19.4GB), run=30016-30016msec WRITE: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4678MiB (4905MB), run=30016-30016msec Disk stats (read/write): vdh: ios=18343/4653, merge=0/0, ticks=1155/22542, in_queue=28526, util=95.16% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962973: Fri Jan 7 14:25:46 2022 read: IOPS=598, BW=599MiB/s (628MB/s)(17.7GiB/30340msec) slat (usec): min=9, max=329, avg=20.51, stdev=11.21 clat (nsec): min=1141, max=363907k, avg=1197913.87, stdev=10653966.94 lat (usec): min=112, max=363922, avg=1218.61, stdev=10654.00 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 126], 60.00th=[ 141], | 70.00th=[ 255], 80.00th=[ 343], 90.00th=[ 494], 95.00th=[ 2868], | 99.00th=[ 13173], 99.50th=[ 15926], 99.90th=[175113], 99.95th=[221250], | 99.99th=[350225] bw ( KiB/s): min=143360, max=1208320, per=100.00%, avg=620187.53, stdev=142275.78, samples=60 iops : min= 140, max= 1180, avg=605.65, stdev=138.94, samples=60 write: IOPS=151, BW=151MiB/s (159MB/s)(4593MiB/30340msec) slat (usec): min=55, max=623, avg=182.71, stdev=93.31 clat (msec): min=12, max=1598, avg=325.17, stdev=162.03 lat (msec): min=12, max=1598, avg=325.35, stdev=162.03 clat percentiles (msec): | 1.00th=[ 56], 5.00th=[ 124], 10.00th=[ 142], 20.00th=[ 171], | 30.00th=[ 207], 40.00th=[ 253], 50.00th=[ 305], 60.00th=[ 359], | 70.00th=[ 414], 80.00th=[ 468], 90.00th=[ 535], 95.00th=[ 609], | 99.00th=[ 760], 99.50th=[ 835], 99.90th=[ 1036], 99.95th=[ 1083], | 99.99th=[ 1603] bw ( KiB/s): min=36864, max=241664, per=100.00%, avg=155175.32, stdev=24101.10, samples=60 iops : min= 36, max= 236, avg=151.53, stdev=23.54, samples=60 lat (usec) : 2=0.01%, 50=0.01%, 100=0.23%, 250=55.57%, 500=16.08% lat (usec) : 750=1.17%, 1000=0.67% lat (msec) : 2=1.42%, 4=1.30%, 10=1.82%, 20=1.21%, 50=0.16% lat (msec) : 100=0.43%, 250=7.71%, 500=9.30%, 750=2.70%, 1000=0.19% lat (msec) : 2000=0.02% cpu : usr=2.95%, sys=2.02%, ctx=20234, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=18171,4593,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=599MiB/s (628MB/s), 599MiB/s-599MiB/s (628MB/s-628MB/s), io=17.7GiB (19.1GB), run=30340-30340msec WRITE: bw=151MiB/s (159MB/s), 151MiB/s-151MiB/s (159MB/s-159MB/s), io=4593MiB (4816MB), run=30340-30340msec Disk stats (read/write): vdh: ios=18606/4652, merge=0/0, ticks=16368/1501628, in_queue=1531254, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963420: Fri Jan 7 14:26:23 2022 read: IOPS=613, BW=614MiB/s (644MB/s)(17.0GiB/30002msec) clat (usec): min=116, max=64513, avg=221.85, stdev=1143.58 lat (usec): min=116, max=64513, avg=222.01, stdev=1143.58 clat percentiles (usec): | 1.00th=[ 123], 5.00th=[ 128], 10.00th=[ 130], 20.00th=[ 133], | 30.00th=[ 137], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 149], | 70.00th=[ 161], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 202], | 99.00th=[ 273], 99.50th=[ 424], 99.90th=[18482], 99.95th=[20579], | 99.99th=[27919] bw ( KiB/s): min=389120, max=864256, per=100.00%, avg=628634.05, stdev=100341.18, samples=60 iops : min= 380, max= 844, avg=613.88, stdev=98.02, samples=60 write: IOPS=155, BW=155MiB/s (163MB/s)(4661MiB/30002msec) clat (usec): min=2563, max=64999, avg=5454.79, stdev=2894.38 lat (usec): min=2633, max=65092, avg=5550.63, stdev=2893.31 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3785], | 30.00th=[ 4555], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5342], | 70.00th=[ 5538], 80.00th=[ 5866], 90.00th=[ 7963], 95.00th=[10814], | 99.00th=[15533], 99.50th=[16909], 99.90th=[29492], 99.95th=[55837], | 99.99th=[64750] bw ( KiB/s): min=133120, max=217088, per=99.99%, avg=159071.65, stdev=12809.66, samples=60 iops : min= 130, max= 212, avg=155.33, stdev=12.51, samples=60 lat (usec) : 250=78.71%, 500=0.70%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.41%, 10=14.52%, 20=1.53%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.95%, sys=1.58%, ctx=23089, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18416,4661,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=614MiB/s (644MB/s), 614MiB/s-614MiB/s (644MB/s-644MB/s), io=17.0GiB (19.3GB), run=30002-30002msec WRITE: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=4661MiB (4887MB), run=30002-30002msec Disk stats (read/write): vdh: ios=18294/4632, merge=0/0, ticks=1228/22748, in_queue=28755, util=95.91% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964370: Fri Jan 7 14:27:00 2022 read: IOPS=597, BW=598MiB/s (627MB/s)(17.9GiB/30672msec) slat (usec): min=9, max=714, avg=19.55, stdev=19.90 clat (usec): min=57, max=728099, avg=2672.89, stdev=13648.14 lat (usec): min=110, max=728116, avg=2692.60, stdev=13648.43 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 113], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 133], 60.00th=[ 221], | 70.00th=[ 343], 80.00th=[ 529], 90.00th=[ 6980], 95.00th=[ 16581], | 99.00th=[ 29230], 99.50th=[ 32900], 99.90th=[175113], 99.95th=[248513], | 99.99th=[683672] bw ( KiB/s): min=22528, max=1445888, per=100.00%, avg=625775.07, stdev=245503.49, samples=60 iops : min= 22, max= 1412, avg=611.08, stdev=239.75, samples=60 write: IOPS=151, BW=152MiB/s (159MB/s)(4651MiB/30672msec) slat (usec): min=61, max=659, avg=201.19, stdev=105.32 clat (msec): min=25, max=2568, avg=648.39, stdev=262.25 lat (msec): min=25, max=2568, avg=648.59, stdev=262.23 clat percentiles (msec): | 1.00th=[ 118], 5.00th=[ 300], 10.00th=[ 401], 20.00th=[ 472], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 600], 60.00th=[ 667], | 70.00th=[ 735], 80.00th=[ 810], 90.00th=[ 944], 95.00th=[ 1099], | 99.00th=[ 1485], 99.50th=[ 1636], 99.90th=[ 2567], 99.95th=[ 2567], | 99.99th=[ 2567] bw ( KiB/s): min= 2048, max=258048, per=98.43%, avg=152839.36, stdev=53753.67, samples=61 iops : min= 2, max= 252, avg=149.25, stdev=52.49, samples=61 lat (usec) : 100=0.12%, 250=48.78%, 500=14.61%, 750=1.84%, 1000=1.00% lat (msec) : 2=1.87%, 4=1.95%, 10=2.96%, 20=3.90%, 50=2.42% lat (msec) : 100=0.21%, 250=0.80%, 500=4.81%, 750=9.03%, 1000=4.09% lat (msec) : 2000=1.52%, >=2000=0.07% cpu : usr=3.20%, sys=1.87%, ctx=17893, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18332,4651,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=598MiB/s (627MB/s), 598MiB/s-598MiB/s (627MB/s-627MB/s), io=17.9GiB (19.2GB), run=30672-30672msec WRITE: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=4651MiB (4877MB), run=30672-30672msec Disk stats (read/write): vdh: ios=18332/4639, merge=0/0, ticks=39930/2966441, in_queue=3023981, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965435: Fri Jan 7 14:27:37 2022 read: IOPS=615, BW=616MiB/s (646MB/s)(18.0GiB/30002msec) clat (usec): min=114, max=25270, avg=210.43, stdev=985.72 lat (usec): min=114, max=25270, avg=210.59, stdev=985.72 clat percentiles (usec): | 1.00th=[ 120], 5.00th=[ 124], 10.00th=[ 127], 20.00th=[ 130], | 30.00th=[ 133], 40.00th=[ 137], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 155], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 196], | 99.00th=[ 273], 99.50th=[ 433], 99.90th=[18220], 99.95th=[19792], | 99.99th=[23987] bw ( KiB/s): min=423936, max=851968, per=100.00%, avg=630738.73, stdev=98295.37, samples=60 iops : min= 414, max= 832, avg=615.92, stdev=95.93, samples=60 write: IOPS=155, BW=156MiB/s (163MB/s)(4675MiB/30002msec) clat (usec): min=2533, max=64061, avg=5479.43, stdev=2996.48 lat (usec): min=2607, max=64156, avg=5576.63, stdev=2995.25 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3851], | 30.00th=[ 4555], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5932], 90.00th=[ 7963], 95.00th=[10814], | 99.00th=[15270], 99.50th=[17171], 99.90th=[54789], 99.95th=[56886], | 99.99th=[64226] bw ( KiB/s): min=131072, max=213418, per=100.00%, avg=159567.97, stdev=12958.56, samples=60 iops : min= 128, max= 208, avg=155.80, stdev=12.62, samples=60 lat (usec) : 250=78.72%, 500=0.72%, 750=0.01% lat (msec) : 2=0.02%, 4=4.34%, 10=14.54%, 20=1.59%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=2.01%, sys=1.38%, ctx=23163, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18475,4675,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=616MiB/s (646MB/s), 616MiB/s-616MiB/s (646MB/s-646MB/s), io=18.0GiB (19.4GB), run=30002-30002msec WRITE: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4675MiB (4902MB), run=30002-30002msec Disk stats (read/write): vdh: ios=18324/4642, merge=0/0, ticks=1109/22888, in_queue=28673, util=95.65% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966052: Fri Jan 7 14:28:15 2022 read: IOPS=595, BW=596MiB/s (625MB/s)(17.9GiB/30830msec) slat (usec): min=11, max=745081, avg=1178.29, stdev=10839.18 clat (msec): min=16, max=955, avg=109.96, stdev=111.42 lat (msec): min=16, max=955, avg=111.14, stdev=111.96 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 38], 20.00th=[ 55], | 30.00th=[ 70], 40.00th=[ 84], 50.00th=[ 92], 60.00th=[ 104], | 70.00th=[ 117], 80.00th=[ 133], 90.00th=[ 157], 95.00th=[ 192], | 99.00th=[ 810], 99.50th=[ 835], 99.90th=[ 911], 99.95th=[ 919], | 99.99th=[ 944] bw ( KiB/s): min= 2048, max=1488896, per=100.00%, avg=614828.20, stdev=251468.54, samples=61 iops : min= 2, max= 1454, avg=600.39, stdev=245.56, samples=61 write: IOPS=151, BW=151MiB/s (158MB/s)(4657MiB/30830msec) slat (usec): min=274, max=802183, avg=1783.88, stdev=17014.02 clat (msec): min=64, max=2629, avg=882.85, stdev=297.55 lat (msec): min=133, max=2629, avg=884.64, stdev=296.88 clat percentiles (msec): | 1.00th=[ 161], 5.00th=[ 405], 10.00th=[ 575], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 860], 60.00th=[ 919], | 70.00th=[ 986], 80.00th=[ 1053], 90.00th=[ 1234], 95.00th=[ 1435], | 99.00th=[ 1854], 99.50th=[ 1921], 99.90th=[ 2232], 99.95th=[ 2232], | 99.99th=[ 2635] bw ( KiB/s): min= 2048, max=280576, per=99.74%, avg=154272.25, stdev=58488.29, samples=60 iops : min= 2, max= 274, avg=150.63, stdev=57.10, samples=60 lat (msec) : 20=0.12%, 50=14.15%, 100=31.28%, 250=32.20%, 500=1.71% lat (msec) : 750=5.07%, 1000=9.80%, 2000=5.63%, >=2000=0.04% cpu : usr=5.24%, sys=1.41%, ctx=2237, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18371,4657,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=596MiB/s (625MB/s), 596MiB/s-596MiB/s (625MB/s-625MB/s), io=17.9GiB (19.3GB), run=30830-30830msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4657MiB (4883MB), run=30830-30830msec Disk stats (read/write): vdh: ios=18371/4648, merge=0/0, ticks=111051/3617951, in_queue=3748277, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966904: Fri Jan 7 14:28:51 2022 read: IOPS=258, BW=516MiB/s (541MB/s)(15.1GiB/30014msec) clat (usec): min=278, max=16741, avg=495.64, stdev=975.96 lat (usec): min=279, max=16741, avg=495.82, stdev=975.96 clat percentiles (usec): | 1.00th=[ 289], 5.00th=[ 293], 10.00th=[ 297], 20.00th=[ 306], | 30.00th=[ 310], 40.00th=[ 318], 50.00th=[ 330], 60.00th=[ 347], | 70.00th=[ 388], 80.00th=[ 611], 90.00th=[ 750], 95.00th=[ 791], | 99.00th=[ 955], 99.50th=[ 9896], 99.90th=[14615], 99.95th=[15139], | 99.99th=[16712] bw ( KiB/s): min=282624, max=958464, per=100.00%, avg=528674.70, stdev=127993.98, samples=60 iops : min= 138, max= 468, avg=258.13, stdev=62.50, samples=60 write: IOPS=65, BW=131MiB/s (138MB/s)(3940MiB/30014msec) clat (usec): min=5746, max=48109, avg=13001.15, stdev=3322.08 lat (usec): min=5956, max=48310, avg=13274.71, stdev=3343.56 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[10159], | 30.00th=[10945], 40.00th=[11731], 50.00th=[13042], 60.00th=[14222], | 70.00th=[15008], 80.00th=[15795], 90.00th=[16450], 95.00th=[17171], | 99.00th=[21365], 99.50th=[23987], 99.90th=[47449], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min=114688, max=155648, per=100.00%, avg=134421.57, stdev=8312.81, samples=60 iops : min= 56, max= 76, avg=65.63, stdev= 4.06, samples=60 lat (usec) : 500=62.17%, 750=9.92%, 1000=6.94% lat (msec) : 2=0.12%, 10=4.01%, 20=16.55%, 50=0.28% cpu : usr=2.01%, sys=1.94%, ctx=9724, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7744,1970,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=516MiB/s (541MB/s), 516MiB/s-516MiB/s (541MB/s-541MB/s), io=15.1GiB (16.2GB), run=30014-30014msec WRITE: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3940MiB (4131MB), run=30014-30014msec Disk stats (read/write): vdh: ios=15428/3920, merge=0/0, ticks=636/48719, in_queue=56613, util=95.57% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967500: Fri Jan 7 14:29:28 2022 read: IOPS=266, BW=532MiB/s (558MB/s)(15.6GiB/30001msec) slat (usec): min=52, max=203, avg=74.17, stdev=14.17 clat (usec): min=136, max=23254, avg=357.98, stdev=999.63 lat (usec): min=247, max=23320, avg=432.40, stdev=1000.21 clat percentiles (usec): | 1.00th=[ 196], 5.00th=[ 200], 10.00th=[ 202], 20.00th=[ 206], | 30.00th=[ 210], 40.00th=[ 215], 50.00th=[ 221], 60.00th=[ 229], | 70.00th=[ 255], 80.00th=[ 318], 90.00th=[ 519], 95.00th=[ 611], | 99.00th=[ 783], 99.50th=[10290], 99.90th=[14746], 99.95th=[15533], | 99.99th=[23200] bw ( KiB/s): min=249856, max=995328, per=100.00%, avg=545707.36, stdev=132821.85, samples=59 iops : min= 122, max= 486, avg=266.44, stdev=64.85, samples=59 write: IOPS=67, BW=135MiB/s (142MB/s)(4058MiB/30001msec) slat (usec): min=139, max=791, avg=300.57, stdev=112.47 clat (usec): min=4646, max=45436, avg=12772.35, stdev=3552.82 lat (usec): min=4903, max=45694, avg=13073.23, stdev=3573.38 clat percentiles (usec): | 1.00th=[ 4883], 5.00th=[ 7046], 10.00th=[ 8160], 20.00th=[ 9634], | 30.00th=[10683], 40.00th=[11600], 50.00th=[13042], 60.00th=[14353], | 70.00th=[15139], 80.00th=[15926], 90.00th=[16450], 95.00th=[16909], | 99.00th=[20579], 99.50th=[23200], 99.90th=[31589], 99.95th=[45351], | 99.99th=[45351] bw ( KiB/s): min=118784, max=163840, per=100.00%, avg=138648.07, stdev=10958.60, samples=59 iops : min= 58, max= 80, avg=67.69, stdev= 5.35, samples=59 lat (usec) : 250=54.75%, 500=16.38%, 750=7.73%, 1000=0.29% lat (msec) : 2=0.04%, 10=4.79%, 20=15.75%, 50=0.26% cpu : usr=1.87%, sys=2.66%, ctx=10023, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7983,2029,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=532MiB/s (558MB/s), 532MiB/s-532MiB/s (558MB/s-558MB/s), io=15.6GiB (16.7GB), run=30001-30001msec WRITE: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=4058MiB (4255MB), run=30001-30001msec Disk stats (read/write): vdh: ios=15914/4038, merge=0/0, ticks=670/49414, in_queue=56618, util=95.96% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968307: Fri Jan 7 14:30:05 2022 read: IOPS=267, BW=536MiB/s (562MB/s)(15.7GiB/30009msec) clat (usec): min=221, max=16672, avg=383.64, stdev=989.46 lat (usec): min=221, max=16672, avg=383.81, stdev=989.46 clat percentiles (usec): | 1.00th=[ 229], 5.00th=[ 235], 10.00th=[ 239], 20.00th=[ 243], | 30.00th=[ 247], 40.00th=[ 251], 50.00th=[ 258], 60.00th=[ 265], | 70.00th=[ 285], 80.00th=[ 343], 90.00th=[ 482], 95.00th=[ 619], | 99.00th=[ 783], 99.50th=[10290], 99.90th=[14877], 99.95th=[15401], | 99.99th=[16712] bw ( KiB/s): min=323584, max=962560, per=100.00%, avg=548603.52, stdev=122660.32, samples=60 iops : min= 158, max= 470, avg=267.87, stdev=59.90, samples=60 write: IOPS=67, BW=136MiB/s (143MB/s)(4080MiB/30009msec) clat (usec): min=4661, max=46615, avg=12960.96, stdev=3545.63 lat (usec): min=4846, max=46810, avg=13188.38, stdev=3555.68 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7177], 10.00th=[ 8356], 20.00th=[ 9896], | 30.00th=[10945], 40.00th=[11863], 50.00th=[13435], 60.00th=[14615], | 70.00th=[15401], 80.00th=[16057], 90.00th=[16581], 95.00th=[17171], | 99.00th=[19268], 99.50th=[21627], 99.90th=[42206], 99.95th=[44303], | 99.99th=[46400] bw ( KiB/s): min=122880, max=167936, per=99.98%, avg=139200.52, stdev=10653.48, samples=60 iops : min= 60, max= 82, avg=67.97, stdev= 5.20, samples=60 lat (usec) : 250=29.45%, 500=42.97%, 750=6.34%, 1000=0.39% lat (msec) : 2=0.08%, 10=4.38%, 20=16.24%, 50=0.16% cpu : usr=1.73%, sys=1.32%, ctx=10080, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8036,2040,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=15.7GiB (16.9GB), run=30009-30009msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4080MiB (4278MB), run=30009-30009msec Disk stats (read/write): vdh: ios=15992/4064, merge=0/0, ticks=663/41174, in_queue=48020, util=96.72% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969561: Fri Jan 7 14:30:42 2022 read: IOPS=350, BW=701MiB/s (735MB/s)(20.6GiB/30113msec) slat (usec): min=22, max=184, avg=36.75, stdev=13.98 clat (usec): min=128, max=909256, avg=1858.96, stdev=25865.21 lat (usec): min=206, max=909286, avg=1895.93, stdev=25865.32 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 190], 10.00th=[ 192], 20.00th=[ 196], | 30.00th=[ 200], 40.00th=[ 204], 50.00th=[ 210], 60.00th=[ 221], | 70.00th=[ 245], 80.00th=[ 611], 90.00th=[ 660], 95.00th=[ 685], | 99.00th=[ 988], 99.50th=[ 30540], 99.90th=[484443], 99.95th=[530580], | 99.99th=[767558] bw ( KiB/s): min=417792, max=1150976, per=100.00%, avg=720691.20, stdev=140288.16, samples=60 iops : min= 204, max= 562, avg=351.90, stdev=68.50, samples=60 write: IOPS=89, BW=178MiB/s (187MB/s)(5370MiB/30113msec) slat (usec): min=120, max=1059, avg=372.18, stdev=194.49 clat (msec): min=8, max=2503, avg=104.27, stdev=187.75 lat (msec): min=9, max=2503, avg=104.64, stdev=187.75 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 31], | 30.00th=[ 41], 40.00th=[ 52], 50.00th=[ 67], 60.00th=[ 86], | 70.00th=[ 110], 80.00th=[ 142], 90.00th=[ 199], 95.00th=[ 259], | 99.00th=[ 443], 99.50th=[ 1569], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min=151552, max=196608, per=100.00%, avg=182613.33, stdev=8900.44, samples=60 iops : min= 74, max= 96, avg=89.17, stdev= 4.35, samples=60 lat (usec) : 250=56.46%, 500=5.86%, 750=15.78%, 1000=0.86% lat (msec) : 2=0.18%, 4=0.11%, 10=0.02%, 20=1.42%, 50=6.53% lat (msec) : 100=5.63%, 250=5.79%, 500=1.14%, 750=0.10%, 1000=0.02% lat (msec) : 2000=0.01%, >=2000=0.10% cpu : usr=3.40%, sys=1.89%, ctx=13215, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10557,2685,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=701MiB/s (735MB/s), 701MiB/s-701MiB/s (735MB/s-735MB/s), io=20.6GiB (22.1GB), run=30113-30113msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=5370MiB (5631MB), run=30113-30113msec Disk stats (read/write): vdh: ios=21055/5337, merge=0/0, ticks=17394/573148, in_queue=608508, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970140: Fri Jan 7 14:31:19 2022 read: IOPS=269, BW=538MiB/s (564MB/s)(15.8GiB/30007msec) clat (usec): min=227, max=21225, avg=377.19, stdev=1006.95 lat (usec): min=227, max=21225, avg=377.36, stdev=1006.95 clat percentiles (usec): | 1.00th=[ 235], 5.00th=[ 241], 10.00th=[ 245], 20.00th=[ 249], | 30.00th=[ 253], 40.00th=[ 260], 50.00th=[ 265], 60.00th=[ 273], | 70.00th=[ 289], 80.00th=[ 343], 90.00th=[ 396], 95.00th=[ 465], | 99.00th=[ 750], 99.50th=[10290], 99.90th=[15139], 99.95th=[15795], | 99.99th=[21103] bw ( KiB/s): min=282624, max=925696, per=100.00%, avg=551134.17, stdev=125824.25, samples=60 iops : min= 138, max= 452, avg=269.10, stdev=61.44, samples=60 write: IOPS=68, BW=137MiB/s (143MB/s)(4096MiB/30007msec) clat (usec): min=4671, max=60368, avg=12944.32, stdev=3685.03 lat (usec): min=4863, max=60558, avg=13154.57, stdev=3686.40 clat percentiles (usec): | 1.00th=[ 5080], 5.00th=[ 7242], 10.00th=[ 8356], 20.00th=[ 9765], | 30.00th=[10814], 40.00th=[11731], 50.00th=[13173], 60.00th=[14615], | 70.00th=[15401], 80.00th=[16057], 90.00th=[16581], 95.00th=[16909], | 99.00th=[20579], 99.50th=[24249], 99.90th=[43254], 99.95th=[49546], | 99.99th=[60556] bw ( KiB/s): min=110592, max=172032, per=99.98%, avg=139745.97, stdev=12683.06, samples=60 iops : min= 54, max= 84, avg=68.23, stdev= 6.20, samples=60 lat (usec) : 250=17.08%, 500=59.63%, 750=2.26%, 1000=0.23% lat (msec) : 2=0.03%, 10=4.53%, 20=16.01%, 50=0.23%, 100=0.01% cpu : usr=1.59%, sys=1.51%, ctx=10131, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8073,2048,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=538MiB/s (564MB/s), 538MiB/s-538MiB/s (564MB/s-564MB/s), io=15.8GiB (16.9GB), run=30007-30007msec WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4096MiB (4295MB), run=30007-30007msec Disk stats (read/write): vdh: ios=16056/4076, merge=0/0, ticks=677/42905, in_queue=49654, util=96.61% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971036: Fri Jan 7 14:31:56 2022 read: IOPS=362, BW=725MiB/s (760MB/s)(21.6GiB/30542msec) slat (usec): min=20, max=434, avg=34.11, stdev=18.08 clat (usec): min=97, max=2511.6k, avg=4074.96, stdev=57021.74 lat (usec): min=207, max=2511.6k, avg=4109.28, stdev=57021.66 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 192], 10.00th=[ 194], | 20.00th=[ 200], 30.00th=[ 204], 40.00th=[ 208], | 50.00th=[ 215], 60.00th=[ 241], 70.00th=[ 627], | 80.00th=[ 660], 90.00th=[ 1893], 95.00th=[ 9765], | 99.00th=[ 25560], 99.50th=[ 76022], 99.90th=[1098908], | 99.95th=[1249903], 99.99th=[2499806] bw ( KiB/s): min=360448, max=1441792, per=100.00%, avg=756015.43, stdev=179642.61, samples=60 iops : min= 176, max= 704, avg=369.13, stdev=87.70, samples=60 write: IOPS=92, BW=184MiB/s (193MB/s)(5628MiB/30542msec) slat (usec): min=119, max=1225, avg=374.43, stdev=196.77 clat (msec): min=40, max=3322, avg=525.83, stdev=245.77 lat (msec): min=41, max=3322, avg=526.20, stdev=245.78 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 288], 10.00th=[ 334], 20.00th=[ 376], | 30.00th=[ 397], 40.00th=[ 422], 50.00th=[ 456], 60.00th=[ 498], | 70.00th=[ 558], 80.00th=[ 642], 90.00th=[ 818], 95.00th=[ 969], | 99.00th=[ 1452], 99.50th=[ 1720], 99.90th=[ 2165], 99.95th=[ 3071], | 99.99th=[ 3339] bw ( KiB/s): min= 4096, max=274432, per=98.43%, avg=185736.56, stdev=32939.46, samples=61 iops : min= 2, max= 134, avg=90.69, stdev=16.08, samples=61 lat (usec) : 100=0.01%, 250=48.97%, 500=5.95%, 750=14.41%, 1000=1.27% lat (msec) : 2=1.29%, 4=1.32%, 10=2.58%, 20=2.84%, 50=0.62% lat (msec) : 100=0.23%, 250=0.70%, 500=11.72%, 750=5.42%, 1000=1.70% lat (msec) : 2000=0.91%, >=2000=0.06% cpu : usr=3.52%, sys=1.77%, ctx=12315, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=11075,2814,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=725MiB/s (760MB/s), 725MiB/s-725MiB/s (760MB/s-760MB/s), io=21.6GiB (23.2GB), run=30542-30542msec WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5628MiB (5901MB), run=30542-30542msec Disk stats (read/write): vdh: ios=22149/5540, merge=0/0, ticks=48454/2879856, in_queue=2990658, util=99.49% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972121: Fri Jan 7 14:32:33 2022 read: IOPS=267, BW=535MiB/s (561MB/s)(15.7GiB/30013msec) clat (usec): min=241, max=16654, avg=391.01, stdev=987.15 lat (usec): min=241, max=16654, avg=391.19, stdev=987.15 clat percentiles (usec): | 1.00th=[ 249], 5.00th=[ 255], 10.00th=[ 260], 20.00th=[ 265], | 30.00th=[ 269], 40.00th=[ 273], 50.00th=[ 281], 60.00th=[ 289], | 70.00th=[ 310], 80.00th=[ 351], 90.00th=[ 400], 95.00th=[ 482], | 99.00th=[ 791], 99.50th=[10290], 99.90th=[15139], 99.95th=[15270], | 99.99th=[16712] bw ( KiB/s): min=249856, max=1048576, per=100.00%, avg=547976.53, stdev=140147.56, samples=60 iops : min= 122, max= 512, avg=267.57, stdev=68.43, samples=60 write: IOPS=67, BW=136MiB/s (142MB/s)(4076MiB/30013msec) clat (usec): min=4673, max=49789, avg=12965.16, stdev=3623.33 lat (usec): min=4861, max=49994, avg=13175.84, stdev=3624.32 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7177], 10.00th=[ 8291], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11731], 50.00th=[13304], 60.00th=[14615], | 70.00th=[15401], 80.00th=[16057], 90.00th=[16712], 95.00th=[16909], | 99.00th=[19530], 99.50th=[23725], 99.90th=[36439], 99.95th=[44303], | 99.99th=[49546] bw ( KiB/s): min=122880, max=167936, per=99.99%, avg=139059.20, stdev=10515.38, samples=60 iops : min= 60, max= 82, avg=67.90, stdev= 5.13, samples=60 lat (usec) : 250=0.93%, 500=75.37%, 750=2.42%, 1000=0.46% lat (msec) : 2=0.03%, 10=4.48%, 20=16.11%, 50=0.20% cpu : usr=1.53%, sys=1.92%, ctx=10074, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8027,2038,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=15.7GiB (16.8GB), run=30013-30013msec WRITE: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=4076MiB (4274MB), run=30013-30013msec Disk stats (read/write): vdh: ios=15924/4042, merge=0/0, ticks=660/47824, in_queue=54495, util=96.02% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972679: Fri Jan 7 14:33:10 2022 read: IOPS=359, BW=718MiB/s (753MB/s)(21.5GiB/30685msec) slat (usec): min=22, max=523477, avg=1961.65, stdev=10148.92 clat (msec): min=20, max=2679, avg=89.87, stdev=98.38 lat (msec): min=20, max=2679, avg=91.83, stdev=98.96 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 36], 20.00th=[ 47], | 30.00th=[ 56], 40.00th=[ 66], 50.00th=[ 74], 60.00th=[ 84], | 70.00th=[ 93], 80.00th=[ 107], 90.00th=[ 132], 95.00th=[ 163], | 99.00th=[ 558], 99.50th=[ 693], 99.90th=[ 1150], 99.95th=[ 1385], | 99.99th=[ 2601] bw ( KiB/s): min= 4096, max=1716224, per=100.00%, avg=737968.41, stdev=296598.87, samples=61 iops : min= 2, max= 838, avg=360.30, stdev=144.82, samples=61 write: IOPS=91, BW=183MiB/s (192MB/s)(5606MiB/30685msec) slat (usec): min=460, max=487640, avg=2976.15, stdev=14238.11 clat (msec): min=60, max=3501, avg=729.54, stdev=362.90 lat (msec): min=61, max=3501, avg=732.52, stdev=362.64 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 401], 10.00th=[ 456], 20.00th=[ 523], | 30.00th=[ 567], 40.00th=[ 609], 50.00th=[ 642], 60.00th=[ 693], | 70.00th=[ 760], 80.00th=[ 885], 90.00th=[ 1062], 95.00th=[ 1250], | 99.00th=[ 2702], 99.50th=[ 2937], 99.90th=[ 3406], 99.95th=[ 3440], | 99.99th=[ 3507] bw ( KiB/s): min=16384, max=331776, per=99.89%, avg=186864.02, stdev=66582.79, samples=60 iops : min= 8, max= 162, avg=91.23, stdev=32.51, samples=60 lat (msec) : 50=19.15%, 100=40.99%, 250=17.90%, 500=3.54%, 750=11.93% lat (msec) : 1000=3.72%, 2000=2.34%, >=2000=0.43% cpu : usr=6.24%, sys=1.67%, ctx=2710, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11022,2803,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=718MiB/s (753MB/s), 718MiB/s-718MiB/s (753MB/s-753MB/s), io=21.5GiB (23.1GB), run=30685-30685msec WRITE: bw=183MiB/s (192MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=5606MiB (5878MB), run=30685-30685msec Disk stats (read/write): vdh: ios=22074/5591, merge=0/0, ticks=131948/3580757, in_queue=3749759, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973639: Fri Jan 7 14:33:47 2022 read: IOPS=268, BW=536MiB/s (563MB/s)(15.7GiB/30010msec) clat (usec): min=231, max=16513, avg=386.22, stdev=973.78 lat (usec): min=231, max=16513, avg=386.41, stdev=973.78 clat percentiles (usec): | 1.00th=[ 239], 5.00th=[ 247], 10.00th=[ 251], 20.00th=[ 255], | 30.00th=[ 262], 40.00th=[ 269], 50.00th=[ 277], 60.00th=[ 285], | 70.00th=[ 306], 80.00th=[ 355], 90.00th=[ 412], 95.00th=[ 502], | 99.00th=[ 775], 99.50th=[10159], 99.90th=[15008], 99.95th=[15270], | 99.99th=[16450] bw ( KiB/s): min=331776, max=819200, per=100.00%, avg=549546.67, stdev=121448.87, samples=60 iops : min= 162, max= 400, avg=268.33, stdev=59.30, samples=60 write: IOPS=68, BW=136MiB/s (143MB/s)(4084MiB/30010msec) clat (usec): min=4663, max=46926, avg=12948.44, stdev=3609.95 lat (usec): min=4800, max=47160, avg=13162.28, stdev=3613.68 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7111], 10.00th=[ 8225], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11731], 50.00th=[13304], 60.00th=[14615], | 70.00th=[15401], 80.00th=[16057], 90.00th=[16712], 95.00th=[16909], | 99.00th=[19792], 99.50th=[25822], 99.90th=[33817], 99.95th=[45351], | 99.99th=[46924] bw ( KiB/s): min=114688, max=167936, per=99.98%, avg=139332.27, stdev=10678.18, samples=60 iops : min= 56, max= 82, avg=68.03, stdev= 5.21, samples=60 lat (usec) : 250=7.99%, 500=67.79%, 750=3.10%, 1000=0.30% lat (msec) : 2=0.06%, 10=4.54%, 20=16.04%, 50=0.19% cpu : usr=1.56%, sys=1.76%, ctx=10098, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8050,2042,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=536MiB/s (563MB/s), 536MiB/s-536MiB/s (563MB/s-563MB/s), io=15.7GiB (16.9GB), run=30010-30010msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4084MiB (4282MB), run=30010-30010msec Disk stats (read/write): vdh: ios=16054/4074, merge=0/0, ticks=660/42833, in_queue=49670, util=95.91% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974256: Fri Jan 7 14:34:25 2022 read: IOPS=362, BW=726MiB/s (761MB/s)(21.7GiB/30672msec) slat (usec): min=22, max=477421, avg=2042.75, stdev=10807.20 clat (msec): min=81, max=3018, avg=311.24, stdev=146.11 lat (msec): min=81, max=3018, avg=313.28, stdev=146.44 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 125], 10.00th=[ 182], 20.00th=[ 232], | 30.00th=[ 259], 40.00th=[ 279], 50.00th=[ 296], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 422], 95.00th=[ 592], | 99.00th=[ 776], 99.50th=[ 885], 99.90th=[ 1888], 99.95th=[ 2265], | 99.99th=[ 2769] bw ( KiB/s): min= 4096, max=1359872, per=99.53%, avg=739484.39, stdev=268831.80, samples=61 iops : min= 2, max= 664, avg=361.05, stdev=131.26, samples=61 write: IOPS=92, BW=184MiB/s (193MB/s)(5650MiB/30672msec) slat (usec): min=547, max=498185, avg=2558.28, stdev=10883.37 clat (msec): min=291, max=2972, avg=931.02, stdev=281.42 lat (msec): min=292, max=2974, avg=933.58, stdev=281.56 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 600], 10.00th=[ 676], 20.00th=[ 743], | 30.00th=[ 793], 40.00th=[ 827], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 986], 80.00th=[ 1083], 90.00th=[ 1267], 95.00th=[ 1435], | 99.00th=[ 2072], 99.50th=[ 2198], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2970] bw ( KiB/s): min=16384, max=331776, per=100.00%, avg=190389.88, stdev=68216.75, samples=59 iops : min= 8, max= 162, avg=92.95, stdev=33.30, samples=59 lat (msec) : 100=1.33%, 250=20.10%, 500=52.80%, 750=8.61%, 1000=11.40% lat (msec) : 2000=5.47%, >=2000=0.30% cpu : usr=6.29%, sys=1.71%, ctx=2808, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11127,2825,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=726MiB/s (761MB/s), 726MiB/s-726MiB/s (761MB/s-761MB/s), io=21.7GiB (23.3GB), run=30672-30672msec WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5650MiB (5924MB), run=30672-30672msec Disk stats (read/write): vdh: ios=22253/5606, merge=0/0, ticks=138613/3565071, in_queue=3748020, util=99.09% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975073: Fri Jan 7 14:35:01 2022 read: IOPS=160, BW=644MiB/s (675MB/s)(18.9GiB/30003msec) clat (usec): min=428, max=29908, avg=869.39, stdev=1530.45 lat (usec): min=428, max=29908, avg=869.59, stdev=1530.46 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 453], 10.00th=[ 461], 20.00th=[ 469], | 30.00th=[ 478], 40.00th=[ 486], 50.00th=[ 498], 60.00th=[ 523], | 70.00th=[ 619], 80.00th=[ 1319], 90.00th=[ 1385], 95.00th=[ 1450], | 99.00th=[10552], 99.50th=[13698], 99.90th=[18220], 99.95th=[20579], | 99.99th=[30016] bw ( KiB/s): min=278528, max=1056768, per=100.00%, avg=659480.62, stdev=163556.71, samples=60 iops : min= 68, max= 258, avg=161.00, stdev=39.93, samples=60 write: IOPS=41, BW=166MiB/s (174MB/s)(4980MiB/30003msec) clat (usec): min=11096, max=67476, avg=20124.56, stdev=4330.35 lat (usec): min=11511, max=68768, avg=20713.30, stdev=4384.25 clat percentiles (usec): | 1.00th=[12780], 5.00th=[14222], 10.00th=[15008], 20.00th=[16188], | 30.00th=[17433], 40.00th=[18482], 50.00th=[20055], 60.00th=[21627], | 70.00th=[22676], 80.00th=[23725], 90.00th=[24773], 95.00th=[25560], | 99.00th=[31065], 99.50th=[33424], 99.90th=[56361], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=147456, max=188416, per=99.93%, avg=169853.48, stdev=9530.00, samples=60 iops : min= 36, max= 46, avg=41.47, stdev= 2.33, samples=60 lat (usec) : 500=40.43%, 750=19.69%, 1000=1.04% lat (msec) : 2=17.20%, 4=0.07%, 10=0.23%, 20=10.95%, 50=10.37% lat (msec) : 100=0.03% cpu : usr=2.53%, sys=2.14%, ctx=6090, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4830,1245,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=18.9GiB (20.3GB), run=30003-30003msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4980MiB (5222MB), run=30003-30003msec Disk stats (read/write): vdh: ios=19272/4963, merge=0/0, ticks=5068/84321, in_queue=99391, util=95.98% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976331: Fri Jan 7 14:35:38 2022 read: IOPS=162, BW=651MiB/s (682MB/s)(19.1GiB/30014msec) slat (usec): min=73, max=224, avg=94.54, stdev=18.85 clat (usec): min=266, max=23904, avg=715.84, stdev=1491.12 lat (usec): min=408, max=23992, avg=810.65, stdev=1492.28 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 343], 10.00th=[ 347], 20.00th=[ 355], | 30.00th=[ 359], 40.00th=[ 367], 50.00th=[ 375], 60.00th=[ 383], | 70.00th=[ 420], 80.00th=[ 1045], 90.00th=[ 1237], 95.00th=[ 1270], | 99.00th=[10028], 99.50th=[13829], 99.90th=[18220], 99.95th=[19006], | 99.99th=[23987] bw ( KiB/s): min=327680, max=1089536, per=100.00%, avg=666692.27, stdev=163708.68, samples=60 iops : min= 80, max= 266, avg=162.77, stdev=39.97, samples=60 write: IOPS=41, BW=167MiB/s (176MB/s)(5024MiB/30014msec) slat (usec): min=263, max=1822, avg=644.50, stdev=321.12 clat (msec): min=11, max=164, avg=20.09, stdev= 6.10 lat (msec): min=11, max=164, avg=20.73, stdev= 6.13 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 32], 99.50th=[ 37], 99.90th=[ 73], 99.95th=[ 165], | 99.99th=[ 165] bw ( KiB/s): min=114688, max=188416, per=99.97%, avg=171349.33, stdev=10801.95, samples=60 iops : min= 28, max= 46, avg=41.83, stdev= 2.64, samples=60 lat (usec) : 500=59.31%, 750=2.23%, 1000=1.60% lat (msec) : 2=15.31%, 4=0.02%, 10=0.28%, 20=11.08%, 50=10.13% lat (msec) : 100=0.03%, 250=0.02% cpu : usr=2.52%, sys=2.03%, ctx=6148, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4883,1256,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=651MiB/s (682MB/s), 651MiB/s-651MiB/s (682MB/s-682MB/s), io=19.1GiB (20.5GB), run=30014-30014msec WRITE: bw=167MiB/s (176MB/s), 167MiB/s-167MiB/s (176MB/s-176MB/s), io=5024MiB (5268MB), run=30014-30014msec Disk stats (read/write): vdh: ios=19424/5004, merge=0/0, ticks=4713/81495, in_queue=95708, util=96.20% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977086: Fri Jan 7 14:36:15 2022 read: IOPS=164, BW=657MiB/s (689MB/s)(19.3GiB/30023msec) clat (usec): min=373, max=24000, avg=755.47, stdev=1482.30 lat (usec): min=373, max=24000, avg=755.67, stdev=1482.31 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 400], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 486], 80.00th=[ 979], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[ 9896], 99.50th=[13960], 99.90th=[17957], 99.95th=[19006], | 99.99th=[23987] bw ( KiB/s): min=286720, max=1105920, per=100.00%, avg=673655.47, stdev=175169.55, samples=60 iops : min= 70, max= 270, avg=164.47, stdev=42.77, samples=60 write: IOPS=42, BW=169MiB/s (178MB/s)(5084MiB/30023msec) clat (usec): min=10868, max=82276, avg=20116.75, stdev=4863.30 lat (usec): min=11259, max=83485, avg=20676.48, stdev=4912.51 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13698], 10.00th=[14746], 20.00th=[15926], | 30.00th=[17171], 40.00th=[18220], 50.00th=[20055], 60.00th=[21890], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[25822], | 99.00th=[31589], 99.50th=[32375], 99.90th=[77071], 99.95th=[82314], | 99.99th=[82314] bw ( KiB/s): min=147456, max=196608, per=100.00%, avg=173397.33, stdev=9195.08, samples=60 iops : min= 36, max= 48, avg=42.33, stdev= 2.24, samples=60 lat (usec) : 500=56.50%, 750=4.87%, 1000=2.42% lat (msec) : 2=14.65%, 4=0.05%, 10=0.26%, 20=11.02%, 50=10.19% lat (msec) : 100=0.05% cpu : usr=2.49%, sys=1.56%, ctx=6213, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4934,1271,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=657MiB/s (689MB/s), 657MiB/s-657MiB/s (689MB/s-689MB/s), io=19.3GiB (20.7GB), run=30023-30023msec WRITE: bw=169MiB/s (178MB/s), 169MiB/s-169MiB/s (178MB/s-178MB/s), io=5084MiB (5331MB), run=30023-30023msec Disk stats (read/write): vdh: ios=19628/5053, merge=0/0, ticks=4335/81847, in_queue=95884, util=96.78% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977958: Fri Jan 7 14:36:52 2022 read: IOPS=190, BW=763MiB/s (800MB/s)(22.5GiB/30155msec) slat (usec): min=40, max=219, avg=61.72, stdev=17.39 clat (usec): min=284, max=2515.8k, avg=6504.45, stdev=74124.67 lat (usec): min=377, max=2515.9k, avg=6566.44, stdev=74124.88 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 343], 10.00th=[ 347], | 20.00th=[ 355], 30.00th=[ 363], 40.00th=[ 371], | 50.00th=[ 379], 60.00th=[ 396], 70.00th=[ 478], | 80.00th=[ 1237], 90.00th=[ 1287], 95.00th=[ 1352], | 99.00th=[ 106431], 99.50th=[ 387974], 99.90th=[1166017], | 99.95th=[1333789], 99.99th=[2499806] bw ( KiB/s): min=368640, max=1245184, per=100.00%, avg=785547.92, stdev=207448.42, samples=60 iops : min= 90, max= 304, avg=191.77, stdev=50.65, samples=60 write: IOPS=49, BW=199MiB/s (209MB/s)(6000MiB/30155msec) slat (usec): min=229, max=2018, avg=801.55, stdev=390.98 clat (msec): min=17, max=2526, avg=174.92, stdev=169.15 lat (msec): min=17, max=2527, avg=175.72, stdev=169.15 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 36], 10.00th=[ 43], 20.00th=[ 58], | 30.00th=[ 78], 40.00th=[ 100], 50.00th=[ 128], 60.00th=[ 167], | 70.00th=[ 205], 80.00th=[ 264], 90.00th=[ 347], 95.00th=[ 447], | 99.00th=[ 751], 99.50th=[ 860], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=180224, max=221184, per=99.99%, avg=203727.70, stdev=8584.34, samples=60 iops : min= 44, max= 54, avg=49.73, stdev= 2.10, samples=60 lat (usec) : 500=56.24%, 750=2.29%, 1000=0.69% lat (msec) : 2=18.57%, 4=0.15%, 10=0.12%, 20=0.26%, 50=2.62% lat (msec) : 100=5.84%, 250=8.19%, 500=3.97%, 750=0.62%, 1000=0.25% lat (msec) : 2000=0.15%, >=2000=0.04% cpu : usr=3.90%, sys=1.64%, ctx=7231, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5755,1500,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=22.5GiB (24.1GB), run=30155-30155msec WRITE: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=6000MiB (6291MB), run=30155-30155msec Disk stats (read/write): vdh: ios=23017/5992, merge=0/0, ticks=40174/1025500, in_queue=1081573, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978553: Fri Jan 7 14:37:29 2022 read: IOPS=163, BW=652MiB/s (684MB/s)(19.1GiB/30004msec) clat (usec): min=379, max=23999, avg=762.51, stdev=1494.89 lat (usec): min=379, max=23999, avg=762.73, stdev=1494.90 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 437], | 70.00th=[ 469], 80.00th=[ 996], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[10421], 99.50th=[13960], 99.90th=[17957], 99.95th=[19006], | 99.99th=[23987] bw ( KiB/s): min=368640, max=1105920, per=100.00%, avg=667765.98, stdev=161545.67, samples=60 iops : min= 90, max= 270, avg=163.00, stdev=39.47, samples=60 write: IOPS=41, BW=168MiB/s (176MB/s)(5032MiB/30004msec) clat (msec): min=10, max=119, avg=20.31, stdev= 5.86 lat (msec): min=11, max=120, avg=20.87, stdev= 5.90 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 26], | 99.00th=[ 33], 99.50th=[ 35], 99.90th=[ 111], 99.95th=[ 120], | 99.99th=[ 120] bw ( KiB/s): min=139264, max=197002, per=99.93%, avg=171611.22, stdev=9830.56, samples=60 iops : min= 34, max= 48, avg=41.88, stdev= 2.40, samples=60 lat (usec) : 500=57.39%, 750=4.15%, 1000=2.15% lat (msec) : 2=14.73%, 4=0.05%, 10=0.24%, 20=10.81%, 50=10.42% lat (msec) : 100=0.02%, 250=0.03% cpu : usr=2.53%, sys=1.72%, ctx=6159, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4891,1258,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=652MiB/s (684MB/s), 652MiB/s-652MiB/s (684MB/s-684MB/s), io=19.1GiB (20.5GB), run=30004-30004msec WRITE: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=5032MiB (5276MB), run=30004-30004msec Disk stats (read/write): vdh: ios=19360/4997, merge=0/0, ticks=4329/81676, in_queue=95423, util=96.39% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979520: Fri Jan 7 14:38:06 2022 read: IOPS=187, BW=749MiB/s (786MB/s)(22.4GiB/30561msec) slat (usec): min=45, max=635892, avg=4048.14, stdev=16915.18 clat (msec): min=16, max=2704, avg=83.62, stdev=124.02 lat (msec): min=16, max=2704, avg=87.67, stdev=125.02 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 37], | 30.00th=[ 44], 40.00th=[ 53], 50.00th=[ 63], 60.00th=[ 73], | 70.00th=[ 84], 80.00th=[ 101], 90.00th=[ 125], 95.00th=[ 161], | 99.00th=[ 567], 99.50th=[ 667], 99.90th=[ 1871], 99.95th=[ 2567], | 99.99th=[ 2702] bw ( KiB/s): min= 8192, max=1654784, per=99.90%, avg=766366.82, stdev=299877.76, samples=61 iops : min= 2, max= 404, avg=187.08, stdev=73.22, samples=61 write: IOPS=48, BW=195MiB/s (205MB/s)(5968MiB/30561msec) slat (usec): min=1103, max=70606, avg=4555.77, stdev=8619.44 clat (msec): min=78, max=3999, avg=682.18, stdev=291.79 lat (msec): min=79, max=4001, avg=686.73, stdev=291.83 clat percentiles (msec): | 1.00th=[ 305], 5.00th=[ 401], 10.00th=[ 460], 20.00th=[ 531], | 30.00th=[ 558], 40.00th=[ 592], 50.00th=[ 617], 60.00th=[ 659], | 70.00th=[ 709], 80.00th=[ 793], 90.00th=[ 936], 95.00th=[ 1099], | 99.00th=[ 1871], 99.50th=[ 2534], 99.90th=[ 4010], 99.95th=[ 4010], | 99.99th=[ 4010] bw ( KiB/s): min=32768, max=376832, per=99.77%, avg=199501.18, stdev=61206.10, samples=60 iops : min= 8, max= 92, avg=48.70, stdev=14.94, samples=60 lat (msec) : 20=0.50%, 50=29.00%, 100=34.02%, 250=13.53%, 500=4.09% lat (msec) : 750=13.47%, 1000=3.60%, 2000=1.54%, >=2000=0.25% cpu : usr=6.59%, sys=1.70%, ctx=2659, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5724,1492,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=749MiB/s (786MB/s), 749MiB/s-749MiB/s (786MB/s-786MB/s), io=22.4GiB (24.0GB), run=30561-30561msec WRITE: bw=195MiB/s (205MB/s), 195MiB/s-195MiB/s (205MB/s-205MB/s), io=5968MiB (6258MB), run=30561-30561msec Disk stats (read/write): vdh: ios=22896/5964, merge=0/0, ticks=154095/3589403, in_queue=3760032, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980466: Fri Jan 7 14:38:43 2022 read: IOPS=163, BW=654MiB/s (685MB/s)(19.2GiB/30005msec) clat (usec): min=375, max=24679, avg=763.14, stdev=1526.26 lat (usec): min=375, max=24680, avg=763.33, stdev=1526.26 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 482], 80.00th=[ 1004], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[10421], 99.50th=[13960], 99.90th=[18482], 99.95th=[20579], | 99.99th=[24773] bw ( KiB/s): min=294912, max=1171456, per=100.00%, avg=669480.65, stdev=161324.31, samples=60 iops : min= 72, max= 286, avg=163.43, stdev=39.37, samples=60 write: IOPS=42, BW=168MiB/s (177MB/s)(5052MiB/30005msec) clat (usec): min=11313, max=73400, avg=20220.89, stdev=4599.18 lat (usec): min=11698, max=73885, avg=20781.63, stdev=4650.71 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13960], 10.00th=[14877], 20.00th=[15926], | 30.00th=[17171], 40.00th=[18482], 50.00th=[20317], 60.00th=[22152], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[25560], | 99.00th=[28181], 99.50th=[32637], 99.90th=[70779], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min=147456, max=196608, per=99.94%, avg=172316.55, stdev=8919.17, samples=60 iops : min= 36, max= 48, avg=42.07, stdev= 2.18, samples=60 lat (usec) : 500=56.44%, 750=5.06%, 1000=2.01% lat (msec) : 2=14.92%, 4=0.02%, 10=0.24%, 20=10.69%, 50=10.59% lat (msec) : 100=0.03% cpu : usr=2.47%, sys=1.63%, ctx=6175, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4903,1263,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=654MiB/s (685MB/s), 654MiB/s-654MiB/s (685MB/s-685MB/s), io=19.2GiB (20.6GB), run=30005-30005msec WRITE: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=5052MiB (5297MB), run=30005-30005msec Disk stats (read/write): vdh: ios=19564/5033, merge=0/0, ticks=4398/81548, in_queue=95815, util=96.22% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981065: Fri Jan 7 14:39:21 2022 read: IOPS=189, BW=760MiB/s (797MB/s)(22.7GiB/30591msec) slat (usec): min=46, max=608900, avg=4007.11, stdev=16376.53 clat (msec): min=84, max=2785, avg=292.52, stdev=155.30 lat (msec): min=84, max=2785, avg=296.53, stdev=155.92 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 121], 10.00th=[ 161], 20.00th=[ 199], | 30.00th=[ 224], 40.00th=[ 247], 50.00th=[ 268], 60.00th=[ 292], | 70.00th=[ 317], 80.00th=[ 351], 90.00th=[ 414], 95.00th=[ 575], | 99.00th=[ 835], 99.50th=[ 936], 99.90th=[ 1854], 99.95th=[ 2366], | 99.99th=[ 2802] bw ( KiB/s): min= 8192, max=1400832, per=99.39%, avg=773345.87, stdev=277503.72, samples=61 iops : min= 2, max= 342, avg=188.79, stdev=67.75, samples=61 write: IOPS=49, BW=198MiB/s (208MB/s)(6064MiB/30591msec) slat (usec): min=1120, max=195291, avg=4426.16, stdev=10346.64 clat (msec): min=373, max=2926, avg=873.45, stdev=246.59 lat (msec): min=375, max=2927, avg=877.87, stdev=246.62 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 567], 10.00th=[ 634], 20.00th=[ 709], | 30.00th=[ 751], 40.00th=[ 785], 50.00th=[ 827], 60.00th=[ 877], | 70.00th=[ 936], 80.00th=[ 1011], 90.00th=[ 1133], 95.00th=[ 1318], | 99.00th=[ 1720], 99.50th=[ 2005], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min=16384, max=376832, per=98.89%, avg=200725.58, stdev=76133.88, samples=60 iops : min= 4, max= 92, avg=49.00, stdev=18.59, samples=60 lat (msec) : 100=1.16%, 250=31.70%, 500=41.72%, 750=9.68%, 1000=11.04% lat (msec) : 2000=4.52%, >=2000=0.18% cpu : usr=6.76%, sys=1.57%, ctx=2727, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5811,1516,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=760MiB/s (797MB/s), 760MiB/s-760MiB/s (797MB/s-797MB/s), io=22.7GiB (24.4GB), run=30591-30591msec WRITE: bw=198MiB/s (208MB/s), 198MiB/s-198MiB/s (208MB/s-208MB/s), io=6064MiB (6359MB), run=30591-30591msec Disk stats (read/write): vdh: ios=23244/5997, merge=0/0, ticks=143052/3568679, in_queue=3766551, util=99.07% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981897: Fri Jan 7 14:39:57 2022 read: IOPS=164, BW=656MiB/s (688MB/s)(19.2GiB/30013msec) clat (usec): min=373, max=24045, avg=750.65, stdev=1513.95 lat (usec): min=373, max=24045, avg=750.84, stdev=1513.96 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 400], | 30.00th=[ 408], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 474], 80.00th=[ 955], 90.00th=[ 1237], 95.00th=[ 1303], | 99.00th=[10421], 99.50th=[14091], 99.90th=[18220], 99.95th=[19792], | 99.99th=[23987] bw ( KiB/s): min=294912, max=1073152, per=100.00%, avg=672455.12, stdev=162605.72, samples=60 iops : min= 72, max= 262, avg=164.17, stdev=39.69, samples=60 write: IOPS=42, BW=169MiB/s (177MB/s)(5072MiB/30013msec) clat (usec): min=10960, max=91216, avg=20188.14, stdev=4695.11 lat (usec): min=11187, max=91647, avg=20741.85, stdev=4740.35 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13960], 10.00th=[14746], 20.00th=[15926], | 30.00th=[17171], 40.00th=[18220], 50.00th=[20055], 60.00th=[22152], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[25822], | 99.00th=[30540], 99.50th=[32900], 99.90th=[48497], 99.95th=[90702], | 99.99th=[90702] bw ( KiB/s): min=139264, max=188416, per=99.97%, avg=172993.20, stdev=8800.82, samples=60 iops : min= 34, max= 46, avg=42.23, stdev= 2.15, samples=60 lat (usec) : 500=56.77%, 750=4.91%, 1000=2.83% lat (msec) : 2=13.94%, 4=0.02%, 10=0.24%, 20=10.88%, 50=10.40% lat (msec) : 100=0.02% cpu : usr=2.50%, sys=1.47%, ctx=6202, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4925,1268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=656MiB/s (688MB/s), 656MiB/s-656MiB/s (688MB/s-688MB/s), io=19.2GiB (20.7GB), run=30013-30013msec WRITE: bw=169MiB/s (177MB/s), 169MiB/s-169MiB/s (177MB/s-177MB/s), io=5072MiB (5318MB), run=30013-30013msec Disk stats (read/write): vdh: ios=19592/5040, merge=0/0, ticks=4142/81950, in_queue=95675, util=95.09% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983201: Fri Jan 7 14:40:35 2022 read: IOPS=189, BW=759MiB/s (796MB/s)(22.6GiB/30551msec) slat (usec): min=45, max=486599, avg=3899.27, stdev=13675.68 clat (msec): min=363, max=3237, avg=706.41, stdev=175.47 lat (msec): min=363, max=3237, avg=710.31, stdev=175.93 clat percentiles (msec): | 1.00th=[ 426], 5.00th=[ 472], 10.00th=[ 514], 20.00th=[ 575], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 693], 60.00th=[ 726], | 70.00th=[ 760], 80.00th=[ 818], 90.00th=[ 919], 95.00th=[ 986], | 99.00th=[ 1099], 99.50th=[ 1301], 99.90th=[ 2333], 99.95th=[ 3071], | 99.99th=[ 3239] bw ( KiB/s): min=139264, max=1269760, per=100.00%, avg=785321.22, stdev=245813.94, samples=59 iops : min= 34, max= 310, avg=191.73, stdev=60.01, samples=59 write: IOPS=49, BW=198MiB/s (207MB/s)(6044MiB/30551msec) slat (usec): min=1120, max=423732, avg=4873.82, stdev=17462.78 clat (msec): min=585, max=4275, avg=1283.83, stdev=347.79 lat (msec): min=586, max=4276, avg=1288.71, stdev=347.62 clat percentiles (msec): | 1.00th=[ 701], 5.00th=[ 902], 10.00th=[ 995], 20.00th=[ 1083], | 30.00th=[ 1150], 40.00th=[ 1200], 50.00th=[ 1234], 60.00th=[ 1301], | 70.00th=[ 1368], 80.00th=[ 1435], 90.00th=[ 1569], 95.00th=[ 1687], | 99.00th=[ 2299], 99.50th=[ 4245], 99.90th=[ 4279], 99.95th=[ 4279], | 99.99th=[ 4279] bw ( KiB/s): min= 8192, max=434176, per=98.00%, avg=198519.47, stdev=81702.65, samples=60 iops : min= 2, max= 106, avg=48.47, stdev=19.95, samples=60 lat (msec) : 500=6.13%, 750=48.02%, 1000=23.93%, 2000=21.43%, >=2000=0.49% cpu : usr=6.85%, sys=1.68%, ctx=2722, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5798,1511,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=759MiB/s (796MB/s), 759MiB/s-759MiB/s (796MB/s-796MB/s), io=22.6GiB (24.3GB), run=30551-30551msec WRITE: bw=198MiB/s (207MB/s), 198MiB/s-198MiB/s (207MB/s-207MB/s), io=6044MiB (6338MB), run=30551-30551msec Disk stats (read/write): vdh: ios=23192/6032, merge=0/0, ticks=146030/3591190, in_queue=3759108, util=98.43%