30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890668: Fri Jan 7 13:33:20 2022 read: IOPS=2485, BW=9940KiB/s (10.2MB/s)(291MiB/30002msec) clat (usec): min=24, max=24862, avg=59.64, stdev=394.68 lat (usec): min=26, max=24862, avg=59.78, stdev=394.68 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 644], 99.95th=[11994], | 99.99th=[16909] bw ( KiB/s): min= 7872, max=14520, per=100.00%, avg=9942.07, stdev=1078.60, samples=60 iops : min= 1968, max= 3630, avg=2485.50, stdev=269.66, samples=60 write: IOPS=619, BW=2479KiB/s (2539kB/s)(72.6MiB/30002msec) clat (usec): min=192, max=45282, avg=1365.07, stdev=1629.13 lat (usec): min=192, max=45282, avg=1365.36, stdev=1629.12 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 562], 10.00th=[ 766], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1942], | 99.00th=[ 9110], 99.50th=[ 9503], 99.90th=[39060], 99.95th=[42206], | 99.99th=[45351] bw ( KiB/s): min= 2024, max= 3744, per=100.00%, avg=2479.47, stdev=248.48, samples=60 iops : min= 506, max= 936, avg=619.87, stdev=62.12, samples=60 lat (usec) : 50=51.48%, 100=28.20%, 250=0.37%, 500=0.74%, 750=1.06% lat (usec) : 1000=2.34% lat (msec) : 2=14.89%, 4=0.61%, 10=0.20%, 20=0.09%, 50=0.02% cpu : usr=1.37%, sys=3.98%, ctx=93157, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74556,18595,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9940KiB/s (10.2MB/s), 9940KiB/s-9940KiB/s (10.2MB/s-10.2MB/s), io=291MiB (305MB), run=30002-30002msec WRITE: bw=2479KiB/s (2539kB/s), 2479KiB/s-2479KiB/s (2539kB/s-2539kB/s), io=72.6MiB (76.2MB), run=30002-30002msec Disk stats (read/write): vdf: ios=73823/18442, merge=0/0, ticks=792/17353, in_queue=28026, util=93.53% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891410: Fri Jan 7 13:33:57 2022 read: IOPS=2464, BW=9860KiB/s (10.1MB/s)(289MiB/30002msec) slat (usec): min=5, max=254, avg= 9.50, stdev= 4.40 clat (nsec): min=647, max=19973k, avg=46606.56, stdev=379585.44 lat (usec): min=22, max=19980, avg=56.25, stdev=379.64 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 38], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 50], | 99.00th=[ 64], 99.50th=[ 74], 99.90th=[ 553], 99.95th=[11994], | 99.99th=[16581] bw ( KiB/s): min= 8312, max=15064, per=100.00%, avg=9861.17, stdev=1127.83, samples=60 iops : min= 2078, max= 3766, avg=2465.27, stdev=281.92, samples=60 write: IOPS=615, BW=2462KiB/s (2521kB/s)(72.1MiB/30002msec) slat (usec): min=6, max=255, avg=10.61, stdev= 5.10 clat (usec): min=129, max=44925, avg=1379.12, stdev=1664.68 lat (usec): min=197, max=44933, avg=1389.88, stdev=1664.68 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 578], 10.00th=[ 799], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1958], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[37487], 99.95th=[43254], | 99.99th=[44827] bw ( KiB/s): min= 2056, max= 3888, per=100.00%, avg=2462.57, stdev=260.62, samples=60 iops : min= 514, max= 972, avg=615.63, stdev=65.15, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.08%, 4=0.01%, 10=0.03%, 20=0.19%, 50=75.74% lat (usec) : 100=3.75%, 250=0.20%, 500=0.68%, 750=0.91%, 1000=2.33% lat (msec) : 2=15.05%, 4=0.62%, 10=0.23%, 20=0.09%, 50=0.02% cpu : usr=1.53%, sys=5.85%, ctx=92246, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73953,18469,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9860KiB/s (10.1MB/s), 9860KiB/s-9860KiB/s (10.1MB/s-10.1MB/s), io=289MiB (303MB), run=30002-30002msec WRITE: bw=2462KiB/s (2521kB/s), 2462KiB/s-2462KiB/s (2521kB/s-2521kB/s), io=72.1MiB (75.6MB), run=30002-30002msec Disk stats (read/write): vdf: ios=73430/18346, merge=0/0, ticks=766/17802, in_queue=28137, util=93.86% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892451: Fri Jan 7 13:34:34 2022 read: IOPS=2478, BW=9915KiB/s (10.2MB/s)(291MiB/30026msec) clat (usec): min=25, max=58818, avg=59.23, stdev=423.96 lat (usec): min=25, max=58818, avg=59.37, stdev=423.96 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 66], | 99.00th=[ 83], 99.50th=[ 95], 99.90th=[ 392], 99.95th=[11207], | 99.99th=[16188] bw ( KiB/s): min= 7544, max=13784, per=100.00%, avg=9924.22, stdev=1101.63, samples=60 iops : min= 1886, max= 3446, avg=2481.05, stdev=275.40, samples=60 write: IOPS=618, BW=2473KiB/s (2532kB/s)(72.5MiB/30026msec) clat (usec): min=199, max=48296, avg=1370.65, stdev=1632.27 lat (usec): min=200, max=48296, avg=1370.93, stdev=1632.27 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 570], 10.00th=[ 775], 20.00th=[ 979], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1942], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[37487], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min= 2008, max= 3544, per=100.00%, avg=2474.73, stdev=219.28, samples=60 iops : min= 502, max= 886, avg=618.68, stdev=54.82, samples=60 lat (usec) : 50=52.41%, 100=27.32%, 250=0.27%, 500=0.78%, 750=1.02% lat (usec) : 1000=2.41% lat (msec) : 2=14.87%, 4=0.57%, 10=0.23%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.45%, sys=3.93%, ctx=92995, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74429,18561,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9915KiB/s (10.2MB/s), 9915KiB/s-9915KiB/s (10.2MB/s-10.2MB/s), io=291MiB (305MB), run=30026-30026msec WRITE: bw=2473KiB/s (2532kB/s), 2473KiB/s-2473KiB/s (2532kB/s-2532kB/s), io=72.5MiB (76.0MB), run=30026-30026msec Disk stats (read/write): vdf: ios=73893/18455, merge=0/0, ticks=789/17448, in_queue=28161, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892966: Fri Jan 7 13:35:11 2022 read: IOPS=2530, BW=9.88MiB/s (10.4MB/s)(297MiB/30041msec) slat (usec): min=2, max=193, avg= 9.64, stdev= 3.48 clat (nsec): min=711, max=2494.8M, avg=902828.67, stdev=39759891.28 lat (usec): min=23, max=2494.8k, avg=912.63, stdev=39759.90 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 41], | 80.00th=[ 44], 90.00th=[ 50], 95.00th=[ 58], | 99.00th=[ 82], 99.50th=[ 101], 99.90th=[ 766], | 99.95th=[ 446694], 99.99th=[2499806] bw ( KiB/s): min= 8569, max=17472, per=100.00%, avg=10134.97, stdev=1221.48, samples=60 iops : min= 2142, max= 4368, avg=2533.73, stdev=305.38, samples=60 write: IOPS=630, BW=2524KiB/s (2584kB/s)(74.0MiB/30041msec) slat (usec): min=2, max=134, avg=10.86, stdev= 3.65 clat (usec): min=401, max=86196, avg=12162.45, stdev=6113.99 lat (usec): min=406, max=86207, avg=12173.48, stdev=6114.07 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 5473], 10.00th=[ 6980], 20.00th=[ 8586], | 30.00th=[ 9765], 40.00th=[10945], 50.00th=[11863], 60.00th=[12780], | 70.00th=[13435], 80.00th=[14222], 90.00th=[16319], 95.00th=[20055], | 99.00th=[26870], 99.50th=[60556], 99.90th=[71828], 99.95th=[72877], | 99.99th=[85459] bw ( KiB/s): min= 2120, max= 4424, per=100.00%, avg=2526.67, stdev=286.49, samples=60 iops : min= 530, max= 1106, avg=631.67, stdev=71.62, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.10%, 50=71.75% lat (usec) : 100=7.69%, 250=0.30%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.24%, 4=0.15%, 10=6.04%, 20=12.51%, 50=0.87% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.36%, sys=6.16%, ctx=92996, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=76013,18954,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9.88MiB/s (10.4MB/s), 9.88MiB/s-9.88MiB/s (10.4MB/s-10.4MB/s), io=297MiB (311MB), run=30041-30041msec WRITE: bw=2524KiB/s (2584kB/s), 2524KiB/s-2524KiB/s (2584kB/s-2584kB/s), io=74.0MiB (77.6MB), run=30041-30041msec Disk stats (read/write): vdf: ios=75538/18842, merge=0/0, ticks=58762/219893, in_queue=296364, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894404: Fri Jan 7 13:35:48 2022 read: IOPS=2467, BW=9871KiB/s (10.1MB/s)(289MiB/30000msec) clat (usec): min=25, max=69495, avg=59.24, stdev=465.11 lat (usec): min=25, max=69495, avg=59.38, stdev=465.11 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 80], 99.50th=[ 90], 99.90th=[ 465], 99.95th=[12649], | 99.99th=[16909] bw ( KiB/s): min= 7200, max=14256, per=100.00%, avg=9884.95, stdev=1137.04, samples=59 iops : min= 1800, max= 3564, avg=2471.24, stdev=284.26, samples=59 write: IOPS=616, BW=2464KiB/s (2523kB/s)(72.2MiB/30000msec) clat (usec): min=207, max=46068, avg=1375.90, stdev=1643.51 lat (usec): min=208, max=46068, avg=1376.17, stdev=1643.50 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 570], 10.00th=[ 775], 20.00th=[ 979], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1958], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[17957], 99.95th=[42206], | 99.99th=[44827] bw ( KiB/s): min= 1888, max= 3656, per=100.00%, avg=2470.31, stdev=254.11, samples=59 iops : min= 472, max= 914, avg=617.58, stdev=63.58, samples=59 lat (usec) : 50=54.87%, 100=24.90%, 250=0.25%, 500=0.72%, 750=1.08% lat (usec) : 1000=2.39% lat (msec) : 2=14.80%, 4=0.62%, 10=0.25%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.56%, sys=3.70%, ctx=92521, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74035,18481,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9871KiB/s (10.1MB/s), 9871KiB/s-9871KiB/s (10.1MB/s-10.1MB/s), io=289MiB (303MB), run=30000-30000msec WRITE: bw=2464KiB/s (2523kB/s), 2464KiB/s-2464KiB/s (2523kB/s-2523kB/s), io=72.2MiB (75.7MB), run=30000-30000msec Disk stats (read/write): vdf: ios=73480/18359, merge=0/0, ticks=843/17448, in_queue=28226, util=94.19% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894974: Fri Jan 7 13:36:25 2022 read: IOPS=2524, BW=9.86MiB/s (10.3MB/s)(297MiB/30089msec) slat (usec): min=2, max=220, avg= 9.54, stdev= 4.28 clat (nsec): min=661, max=2510.2M, avg=925344.21, stdev=39166995.13 lat (usec): min=19, max=2510.2k, avg=935.03, stdev=39166.98 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 40], 70.00th=[ 43], | 80.00th=[ 47], 90.00th=[ 58], 95.00th=[ 74], | 99.00th=[ 161], 99.50th=[ 314], 99.90th=[ 1090], | 99.95th=[ 541066], 99.99th=[2332034] bw ( KiB/s): min= 7680, max=17720, per=100.00%, avg=10129.90, stdev=1343.64, samples=60 iops : min= 1920, max= 4430, avg=2532.45, stdev=335.89, samples=60 write: IOPS=629, BW=2519KiB/s (2579kB/s)(74.0MiB/30089msec) slat (usec): min=2, max=225, avg=10.82, stdev= 4.90 clat (msec): min=6, max=2393, avg=75.60, stdev=65.02 lat (msec): min=6, max=2393, avg=75.61, stdev=65.02 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 54], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 66], 50.00th=[ 68], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 88], 95.00th=[ 117], | 99.00th=[ 234], 99.50th=[ 472], 99.90th=[ 969], 99.95th=[ 1250], | 99.99th=[ 2366] bw ( KiB/s): min= 1992, max= 4208, per=100.00%, avg=2520.25, stdev=275.39, samples=60 iops : min= 498, max= 1052, avg=630.05, stdev=68.84, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.07%, 4=0.01%, 10=0.04%, 20=0.20%, 50=66.83% lat (usec) : 100=11.08%, 250=1.24%, 500=0.31%, 750=0.09%, 1000=0.04% lat (msec) : 2=0.02%, 4=0.01%, 10=0.17%, 20=0.11%, 50=0.36% lat (msec) : 100=17.84%, 250=1.31%, 500=0.11%, 750=0.05%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.02% cpu : usr=1.38%, sys=6.01%, ctx=91429, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=75970,18946,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9.86MiB/s (10.3MB/s), 9.86MiB/s-9.86MiB/s (10.3MB/s-10.3MB/s), io=297MiB (311MB), run=30089-30089msec WRITE: bw=2519KiB/s (2579kB/s), 2519KiB/s-2519KiB/s (2579kB/s-2579kB/s), io=74.0MiB (77.6MB), run=30089-30089msec Disk stats (read/write): vdf: ios=75966/18939, merge=0/0, ticks=61473/1419970, in_queue=1499631, util=99.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895817: Fri Jan 7 13:37:02 2022 read: IOPS=2477, BW=9911KiB/s (10.1MB/s)(290MiB/30002msec) clat (usec): min=24, max=24433, avg=58.91, stdev=396.38 lat (usec): min=24, max=24433, avg=59.05, stdev=396.38 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 66], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 474], 99.95th=[11994], | 99.99th=[16581] bw ( KiB/s): min= 8248, max=15072, per=100.00%, avg=9912.60, stdev=1095.77, samples=60 iops : min= 2062, max= 3768, avg=2478.13, stdev=273.95, samples=60 write: IOPS=618, BW=2472KiB/s (2532kB/s)(72.4MiB/30002msec) clat (usec): min=184, max=46401, avg=1372.18, stdev=1660.67 lat (usec): min=185, max=46401, avg=1372.46, stdev=1660.66 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 570], 10.00th=[ 783], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1942], | 99.00th=[ 9110], 99.50th=[ 9503], 99.90th=[39060], 99.95th=[41681], | 99.99th=[45876] bw ( KiB/s): min= 2056, max= 3896, per=100.00%, avg=2472.53, stdev=273.44, samples=60 iops : min= 514, max= 974, avg=618.13, stdev=68.36, samples=60 lat (usec) : 50=54.13%, 100=25.55%, 250=0.36%, 500=0.75%, 750=0.98% lat (usec) : 1000=2.36% lat (msec) : 2=14.94%, 4=0.61%, 10=0.21%, 20=0.08%, 50=0.03% cpu : usr=1.62%, sys=3.66%, ctx=92891, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74340,18544,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9911KiB/s (10.1MB/s), 9911KiB/s-9911KiB/s (10.1MB/s-10.1MB/s), io=290MiB (304MB), run=30002-30002msec WRITE: bw=2472KiB/s (2532kB/s), 2472KiB/s-2472KiB/s (2532kB/s-2532kB/s), io=72.4MiB (75.0MB), run=30002-30002msec Disk stats (read/write): vdf: ios=73683/18407, merge=0/0, ticks=786/17480, in_queue=28144, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896941: Fri Jan 7 13:37:39 2022 read: IOPS=2514, BW=9.82MiB/s (10.3MB/s)(296MiB/30180msec) slat (usec): min=2, max=180, avg= 9.35, stdev= 4.23 clat (nsec): min=662, max=2513.3M, avg=907377.49, stdev=38252920.64 lat (usec): min=22, max=2513.3k, avg=916.87, stdev=38252.90 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 50], 90.00th=[ 66], 95.00th=[ 119], | 99.00th=[ 750], 99.50th=[ 1188], 99.90th=[ 2343], | 99.95th=[ 557843], 99.99th=[2499806] bw ( KiB/s): min= 8072, max=18760, per=100.00%, avg=10121.07, stdev=1456.44, samples=60 iops : min= 2018, max= 4690, avg=2530.25, stdev=364.11, samples=60 write: IOPS=627, BW=2509KiB/s (2569kB/s)(73.0MiB/30180msec) slat (usec): min=2, max=194, avg=10.66, stdev= 4.89 clat (usec): min=1795, max=2835.6k, avg=155645.00, stdev=121617.50 lat (usec): min=1894, max=2835.7k, avg=155655.80, stdev=121617.51 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 106], 10.00th=[ 113], 20.00th=[ 130], | 30.00th=[ 138], 40.00th=[ 142], 50.00th=[ 146], 60.00th=[ 148], | 70.00th=[ 155], 80.00th=[ 163], 90.00th=[ 186], 95.00th=[ 209], | 99.00th=[ 372], 99.50th=[ 802], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 2076, max= 4048, per=100.00%, avg=2511.77, stdev=260.02, samples=60 iops : min= 519, max= 1012, avg=627.93, stdev=65.00, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.09%, 50=64.57% lat (usec) : 100=10.53%, 250=3.02%, 500=0.61%, 750=0.32%, 1000=0.25% lat (msec) : 2=0.43%, 4=0.04%, 10=0.01%, 20=0.25%, 50=0.13% lat (msec) : 100=0.21%, 250=18.86%, 500=0.37%, 750=0.06%, 1000=0.05% lat (msec) : 2000=0.06%, >=2000=0.04% cpu : usr=1.40%, sys=5.81%, ctx=88791, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75902,18932,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9.82MiB/s (10.3MB/s), 9.82MiB/s-9.82MiB/s (10.3MB/s-10.3MB/s), io=296MiB (311MB), run=30180-30180msec WRITE: bw=2509KiB/s (2569kB/s), 2509KiB/s-2509KiB/s (2569kB/s-2569kB/s), io=73.0MiB (77.5MB), run=30180-30180msec Disk stats (read/write): vdf: ios=75898/18859, merge=0/0, ticks=58381/2914146, in_queue=3002114, util=99.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897436: Fri Jan 7 13:38:16 2022 read: IOPS=2478, BW=9916KiB/s (10.2MB/s)(291MiB/30001msec) clat (usec): min=25, max=22201, avg=61.09, stdev=387.04 lat (usec): min=25, max=22201, avg=61.22, stdev=387.04 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 88], 99.50th=[ 102], 99.90th=[ 486], 99.95th=[12125], | 99.99th=[15664] bw ( KiB/s): min= 7608, max=15568, per=100.00%, avg=9915.95, stdev=1129.59, samples=60 iops : min= 1902, max= 3892, avg=2478.97, stdev=282.40, samples=60 write: IOPS=618, BW=2473KiB/s (2533kB/s)(72.5MiB/30001msec) clat (usec): min=194, max=45675, avg=1362.69, stdev=1610.90 lat (usec): min=195, max=45676, avg=1362.99, stdev=1610.90 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 553], 10.00th=[ 758], 20.00th=[ 971], | 30.00th=[ 1090], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1467], 90.00th=[ 1680], 95.00th=[ 1926], | 99.00th=[ 8979], 99.50th=[ 9503], 99.90th=[35914], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min= 2056, max= 3992, per=100.00%, avg=2473.30, stdev=276.48, samples=60 iops : min= 514, max= 998, avg=618.32, stdev=69.12, samples=60 lat (usec) : 50=46.71%, 100=32.88%, 250=0.42%, 500=0.78%, 750=1.10% lat (usec) : 1000=2.45% lat (msec) : 2=14.74%, 4=0.57%, 10=0.22%, 20=0.09%, 50=0.02% cpu : usr=1.51%, sys=4.02%, ctx=92925, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74370,18551,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9916KiB/s (10.2MB/s), 9916KiB/s-9916KiB/s (10.2MB/s-10.2MB/s), io=291MiB (305MB), run=30001-30001msec WRITE: bw=2473KiB/s (2533kB/s), 2473KiB/s-2473KiB/s (2533kB/s-2533kB/s), io=72.5MiB (75.0MB), run=30001-30001msec Disk stats (read/write): vdf: ios=73785/18429, merge=0/0, ticks=788/17167, in_queue=27987, util=93.38% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898365: Fri Jan 7 13:38:53 2022 read: IOPS=2493, BW=9973KiB/s (10.2MB/s)(294MiB/30188msec) slat (usec): min=2, max=261750, avg=317.26, stdev=2073.93 clat (usec): min=546, max=2595.2k, avg=25906.89, stdev=44069.58 lat (usec): min=550, max=2600.5k, avg=26224.30, stdev=44126.32 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 8455], 10.00th=[ 14091], | 20.00th=[ 17171], 30.00th=[ 19530], 40.00th=[ 21365], | 50.00th=[ 23200], 60.00th=[ 25297], 70.00th=[ 27132], | 80.00th=[ 30016], 90.00th=[ 34341], 95.00th=[ 39584], | 99.00th=[ 82314], 99.50th=[ 179307], 99.90th=[ 274727], | 99.95th=[ 650118], 99.99th=[2499806] bw ( KiB/s): min= 7968, max=18032, per=100.00%, avg=10026.87, stdev=1403.34, samples=60 iops : min= 1992, max= 4508, avg=2506.72, stdev=350.83, samples=60 write: IOPS=622, BW=2488KiB/s (2548kB/s)(73.4MiB/30188msec) slat (usec): min=2, max=176945, avg=315.68, stdev=1971.95 clat (msec): min=4, max=3393, avg=215.92, stdev=144.09 lat (msec): min=4, max=3393, avg=216.23, stdev=144.09 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 146], 10.00th=[ 163], 20.00th=[ 182], | 30.00th=[ 190], 40.00th=[ 197], 50.00th=[ 201], 60.00th=[ 207], | 70.00th=[ 213], 80.00th=[ 226], 90.00th=[ 251], 95.00th=[ 288], | 99.00th=[ 735], 99.50th=[ 1167], 99.90th=[ 2165], 99.95th=[ 2735], | 99.99th=[ 3406] bw ( KiB/s): min= 1936, max= 3680, per=99.93%, avg=2486.22, stdev=250.52, samples=60 iops : min= 484, max= 920, avg=621.55, stdev=62.63, samples=60 lat (usec) : 750=0.15%, 1000=0.41% lat (msec) : 2=1.50%, 4=1.36%, 10=0.96%, 20=21.40%, 50=52.72% lat (msec) : 100=1.57%, 250=17.73%, 500=1.87%, 750=0.09%, 1000=0.08% lat (msec) : 2000=0.10%, >=2000=0.06% cpu : usr=0.71%, sys=1.96%, ctx=10343, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75267,18778,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9973KiB/s (10.2MB/s), 9973KiB/s-9973KiB/s (10.2MB/s-10.2MB/s), io=294MiB (308MB), run=30188-30188msec WRITE: bw=2488KiB/s (2548kB/s), 2488KiB/s-2488KiB/s (2548kB/s-2548kB/s), io=73.4MiB (76.9MB), run=30188-30188msec Disk stats (read/write): vdf: ios=75262/18702, merge=0/0, ticks=70721/3551538, in_queue=3666712, util=99.49% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898979: Fri Jan 7 13:39:30 2022 read: IOPS=2403, BW=37.6MiB/s (39.4MB/s)(1127MiB/30003msec) clat (usec): min=28, max=28027, avg=64.13, stdev=412.97 lat (usec): min=28, max=28027, avg=64.27, stdev=412.98 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 94], 99.50th=[ 110], 99.90th=[ 725], 99.95th=[12256], | 99.99th=[17433] bw ( KiB/s): min=30816, max=59296, per=100.00%, avg=38468.43, stdev=4396.14, samples=60 iops : min= 1926, max= 3706, avg=2404.27, stdev=274.76, samples=60 write: IOPS=602, BW=9634KiB/s (9865kB/s)(282MiB/30003msec) clat (usec): min=240, max=44308, avg=1394.66, stdev=1594.45 lat (usec): min=241, max=44310, avg=1396.18, stdev=1594.44 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 570], 10.00th=[ 791], 20.00th=[ 1012], | 30.00th=[ 1139], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 1991], | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[17433], 99.95th=[41681], | 99.99th=[43779] bw ( KiB/s): min= 8096, max=15296, per=100.00%, avg=9634.42, stdev=1029.62, samples=60 iops : min= 506, max= 956, avg=602.15, stdev=64.35, samples=60 lat (usec) : 50=39.84%, 100=39.57%, 250=0.45%, 500=0.78%, 750=1.00% lat (usec) : 1000=2.13% lat (msec) : 2=15.17%, 4=0.73%, 10=0.21%, 20=0.09%, 50=0.02% cpu : usr=1.43%, sys=3.96%, ctx=90196, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72126,18065,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.6MiB/s (39.4MB/s), 37.6MiB/s-37.6MiB/s (39.4MB/s-39.4MB/s), io=1127MiB (1182MB), run=30003-30003msec WRITE: bw=9634KiB/s (9865kB/s), 9634KiB/s-9634KiB/s (9865kB/s-9865kB/s), io=282MiB (296MB), run=30003-30003msec Disk stats (read/write): vdf: ios=71801/17974, merge=0/0, ticks=799/17303, in_queue=28134, util=93.90% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899727: Fri Jan 7 13:40:07 2022 read: IOPS=2386, BW=37.3MiB/s (39.1MB/s)(1119MiB/30002msec) slat (usec): min=4, max=249, avg=10.61, stdev= 5.45 clat (nsec): min=656, max=70923k, avg=51026.76, stdev=469241.46 lat (usec): min=24, max=70932, avg=61.80, stdev=469.29 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 31], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 38], 60.00th=[ 40], | 70.00th=[ 42], 80.00th=[ 45], 90.00th=[ 50], 95.00th=[ 56], | 99.00th=[ 71], 99.50th=[ 81], 99.90th=[ 359], 99.95th=[11731], | 99.99th=[16712] bw ( KiB/s): min=29024, max=56128, per=100.00%, avg=38193.60, stdev=4530.36, samples=60 iops : min= 1814, max= 3508, avg=2387.10, stdev=283.15, samples=60 write: IOPS=597, BW=9561KiB/s (9790kB/s)(280MiB/30002msec) slat (usec): min=8, max=255, avg=13.16, stdev= 6.16 clat (usec): min=213, max=45967, avg=1405.27, stdev=1615.58 lat (usec): min=223, max=45978, avg=1418.60, stdev=1615.61 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 594], 10.00th=[ 816], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 1975], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[17957], 99.95th=[42206], | 99.99th=[45351] bw ( KiB/s): min= 7744, max=14336, per=100.00%, avg=9561.07, stdev=1074.05, samples=60 iops : min= 484, max= 896, avg=597.57, stdev=67.13, samples=60 lat (nsec) : 750=0.03%, 1000=0.09% lat (usec) : 2=0.15%, 4=0.01%, 10=0.05%, 20=0.20%, 50=71.43% lat (usec) : 100=7.83%, 250=0.14%, 500=0.71%, 750=0.89%, 1000=2.02% lat (msec) : 2=15.44%, 4=0.67%, 10=0.22%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.55%, sys=6.06%, ctx=89172, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71613,17928,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.3MiB/s (39.1MB/s), 37.3MiB/s-37.3MiB/s (39.1MB/s-39.1MB/s), io=1119MiB (1173MB), run=30002-30002msec WRITE: bw=9561KiB/s (9790kB/s), 9561KiB/s-9561KiB/s (9790kB/s-9790kB/s), io=280MiB (294MB), run=30002-30002msec Disk stats (read/write): vdf: ios=71360/17850, merge=0/0, ticks=809/17627, in_queue=28088, util=93.72% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901187: Fri Jan 7 13:40:44 2022 read: IOPS=2387, BW=37.3MiB/s (39.1MB/s)(1119MiB/30001msec) clat (usec): min=28, max=22155, avg=61.40, stdev=362.45 lat (usec): min=28, max=22155, avg=61.54, stdev=362.45 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 685], 99.95th=[11469], | 99.99th=[15270] bw ( KiB/s): min=29248, max=52288, per=100.00%, avg=38199.47, stdev=4326.64, samples=60 iops : min= 1828, max= 3268, avg=2387.50, stdev=270.45, samples=60 write: IOPS=597, BW=9564KiB/s (9793kB/s)(280MiB/30001msec) clat (usec): min=234, max=45551, avg=1417.85, stdev=1666.95 lat (usec): min=236, max=45553, avg=1419.28, stdev=1666.94 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 586], 10.00th=[ 807], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 1991], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[36963], 99.95th=[41681], | 99.99th=[44303] bw ( KiB/s): min= 7488, max=13472, per=100.00%, avg=9563.73, stdev=915.69, samples=60 iops : min= 468, max= 842, avg=597.73, stdev=57.23, samples=60 lat (usec) : 50=43.61%, 100=35.96%, 250=0.32%, 500=0.74%, 750=0.94% lat (usec) : 1000=1.98% lat (msec) : 2=15.40%, 4=0.71%, 10=0.24%, 20=0.09%, 50=0.02% cpu : usr=1.45%, sys=3.75%, ctx=89563, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71625,17933,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.3MiB/s (39.1MB/s), 37.3MiB/s-37.3MiB/s (39.1MB/s-39.1MB/s), io=1119MiB (1174MB), run=30001-30001msec WRITE: bw=9564KiB/s (9793kB/s), 9564KiB/s-9564KiB/s (9793kB/s-9793kB/s), io=280MiB (294MB), run=30001-30001msec Disk stats (read/write): vdf: ios=71369/17852, merge=0/0, ticks=701/17649, in_queue=28217, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901852: Fri Jan 7 13:41:21 2022 read: IOPS=2429, BW=37.0MiB/s (39.8MB/s)(1140MiB/30040msec) slat (usec): min=2, max=157, avg=10.81, stdev= 3.93 clat (nsec): min=722, max=2531.3M, avg=855388.73, stdev=37696611.23 lat (usec): min=24, max=2531.3k, avg=866.36, stdev=37696.62 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 44], | 80.00th=[ 48], 90.00th=[ 53], 95.00th=[ 60], | 99.00th=[ 86], 99.50th=[ 105], 99.90th=[ 570], | 99.95th=[ 379585], 99.99th=[2499806] bw ( KiB/s): min=30400, max=63968, per=100.00%, avg=38912.53, stdev=4787.02, samples=60 iops : min= 1900, max= 3998, avg=2432.03, stdev=299.19, samples=60 write: IOPS=607, BW=9716KiB/s (9949kB/s)(285MiB/30040msec) slat (nsec): min=4162, max=79262, avg=13591.95, stdev=3922.51 clat (usec): min=401, max=80216, avg=12976.90, stdev=6277.65 lat (usec): min=407, max=80227, avg=12990.65, stdev=6277.92 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 6194], 10.00th=[ 7963], 20.00th=[10028], | 30.00th=[11207], 40.00th=[11731], 50.00th=[12518], 60.00th=[13042], | 70.00th=[13829], 80.00th=[14615], 90.00th=[16712], 95.00th=[20841], | 99.00th=[31327], 99.50th=[62653], 99.90th=[71828], 99.95th=[73925], | 99.99th=[80217] bw ( KiB/s): min= 8288, max=16128, per=100.00%, avg=9727.47, stdev=1049.69, samples=60 iops : min= 518, max= 1008, avg=607.97, stdev=65.61, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.03%, 20=0.11%, 50=68.32% lat (usec) : 100=11.03%, 250=0.35%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.24%, 4=0.13%, 10=3.49%, 20=14.89%, 50=1.08% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.43%, sys=6.41%, ctx=89834, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72968,18242,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.0MiB/s (39.8MB/s), 37.0MiB/s-37.0MiB/s (39.8MB/s-39.8MB/s), io=1140MiB (1196MB), run=30040-30040msec WRITE: bw=9716KiB/s (9949kB/s), 9716KiB/s-9716KiB/s (9949kB/s-9949kB/s), io=285MiB (299MB), run=30040-30040msec Disk stats (read/write): vdf: ios=72620/18170, merge=0/0, ticks=51891/227051, in_queue=296810, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902761: Fri Jan 7 13:41:58 2022 read: IOPS=2369, BW=37.0MiB/s (38.8MB/s)(1111MiB/30002msec) clat (usec): min=25, max=24423, avg=61.37, stdev=382.09 lat (usec): min=25, max=24423, avg=61.52, stdev=382.09 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 71], | 99.00th=[ 89], 99.50th=[ 102], 99.90th=[ 289], 99.95th=[11338], | 99.99th=[16450] bw ( KiB/s): min=29600, max=53120, per=100.00%, avg=37919.43, stdev=4278.46, samples=60 iops : min= 1850, max= 3320, avg=2369.95, stdev=267.41, samples=60 write: IOPS=592, BW=9486KiB/s (9713kB/s)(278MiB/30002msec) clat (usec): min=228, max=46606, avg=1431.14, stdev=1705.21 lat (usec): min=230, max=46608, avg=1432.62, stdev=1705.24 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 603], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2024], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[38536], 99.95th=[42730], | 99.99th=[44827] bw ( KiB/s): min= 7648, max=13792, per=100.00%, avg=9486.80, stdev=924.64, samples=60 iops : min= 478, max= 862, avg=592.92, stdev=57.79, samples=60 lat (usec) : 50=44.43%, 100=35.13%, 250=0.36%, 500=0.72%, 750=0.97% lat (usec) : 1000=1.95% lat (msec) : 2=15.30%, 4=0.78%, 10=0.25%, 20=0.10%, 50=0.02% cpu : usr=1.51%, sys=3.78%, ctx=88886, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71092,17787,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.0MiB/s (38.8MB/s), 37.0MiB/s-37.0MiB/s (38.8MB/s-38.8MB/s), io=1111MiB (1165MB), run=30002-30002msec WRITE: bw=9486KiB/s (9713kB/s), 9486KiB/s-9486KiB/s (9713kB/s-9713kB/s), io=278MiB (291MB), run=30002-30002msec Disk stats (read/write): vdf: ios=70769/17694, merge=0/0, ticks=722/17741, in_queue=28188, util=94.09% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903776: Fri Jan 7 13:42:35 2022 read: IOPS=2420, BW=37.8MiB/s (39.7MB/s)(1139MiB/30101msec) slat (usec): min=2, max=370, avg=10.27, stdev= 7.38 clat (nsec): min=671, max=2515.4M, avg=932716.70, stdev=39702233.20 lat (usec): min=24, max=2515.4k, avg=943.14, stdev=39702.22 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 49], 90.00th=[ 60], 95.00th=[ 78], | 99.00th=[ 221], 99.50th=[ 453], 99.90th=[ 1057], | 99.95th=[ 438305], 99.99th=[2499806] bw ( KiB/s): min=28896, max=70176, per=100.00%, avg=38862.62, stdev=5281.11, samples=60 iops : min= 1806, max= 4386, avg=2428.90, stdev=330.07, samples=60 write: IOPS=605, BW=9682KiB/s (9914kB/s)(285MiB/30101msec) slat (usec): min=3, max=358, avg=13.64, stdev= 8.57 clat (msec): min=6, max=2415, avg=78.82, stdev=69.78 lat (msec): min=6, max=2415, avg=78.83, stdev=69.78 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 92], 95.00th=[ 122], | 99.00th=[ 234], 99.50th=[ 523], 99.90th=[ 1083], 99.95th=[ 1351], | 99.99th=[ 1921] bw ( KiB/s): min= 7456, max=16640, per=100.00%, avg=9690.45, stdev=1146.38, samples=60 iops : min= 466, max= 1040, avg=605.65, stdev=71.65, samples=60 lat (nsec) : 750=0.01%, 1000=0.12% lat (usec) : 2=0.15%, 4=0.02%, 10=0.05%, 20=0.23%, 50=65.00% lat (usec) : 100=12.23%, 250=1.49%, 500=0.37%, 750=0.19%, 1000=0.07% lat (msec) : 2=0.02%, 4=0.01%, 10=0.06%, 20=0.24%, 50=0.28% lat (msec) : 100=17.77%, 250=1.48%, 500=0.10%, 750=0.05%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.02% cpu : usr=1.69%, sys=5.77%, ctx=88164, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=72870,18214,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.7MB/s), 37.8MiB/s-37.8MiB/s (39.7MB/s-39.7MB/s), io=1139MiB (1194MB), run=30101-30101msec WRITE: bw=9682KiB/s (9914kB/s), 9682KiB/s-9682KiB/s (9914kB/s-9914kB/s), io=285MiB (298MB), run=30101-30101msec Disk stats (read/write): vdf: ios=72425/18083, merge=0/0, ticks=60740/1413859, in_queue=1491295, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904052: Fri Jan 7 13:43:12 2022 read: IOPS=2392, BW=37.4MiB/s (39.2MB/s)(1122MiB/30001msec) clat (usec): min=26, max=64326, avg=62.65, stdev=448.90 lat (usec): min=26, max=64326, avg=62.79, stdev=448.90 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 465], 99.95th=[12518], | 99.99th=[16450] bw ( KiB/s): min=29376, max=57824, per=100.00%, avg=38288.95, stdev=4437.96, samples=60 iops : min= 1836, max= 3614, avg=2393.05, stdev=277.36, samples=60 write: IOPS=599, BW=9584KiB/s (9814kB/s)(281MiB/30001msec) clat (usec): min=223, max=54057, avg=1408.94, stdev=1612.26 lat (usec): min=224, max=54059, avg=1410.52, stdev=1612.25 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 578], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 1975], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[18220], 99.95th=[40633], | 99.99th=[45876] bw ( KiB/s): min= 7680, max=14816, per=100.00%, avg=9584.33, stdev=1011.62, samples=60 iops : min= 480, max= 926, avg=599.02, stdev=63.22, samples=60 lat (usec) : 50=42.60%, 100=36.96%, 250=0.33%, 500=0.75%, 750=0.91% lat (usec) : 1000=1.96% lat (msec) : 2=15.48%, 4=0.67%, 10=0.22%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.45%, sys=3.92%, ctx=89767, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71789,17971,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.4MiB/s (39.2MB/s), 37.4MiB/s-37.4MiB/s (39.2MB/s-39.2MB/s), io=1122MiB (1176MB), run=30001-30001msec WRITE: bw=9584KiB/s (9814kB/s), 9584KiB/s-9584KiB/s (9814kB/s-9814kB/s), io=281MiB (294MB), run=30001-30001msec Disk stats (read/write): vdf: ios=71504/17899, merge=0/0, ticks=794/17546, in_queue=28173, util=93.97% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905209: Fri Jan 7 13:43:49 2022 read: IOPS=2440, BW=38.1MiB/s (39.0MB/s)(1151MiB/30191msec) slat (usec): min=2, max=241, avg= 9.89, stdev= 4.90 clat (nsec): min=726, max=2538.1M, avg=994798.87, stdev=40273537.54 lat (usec): min=23, max=2538.2k, avg=1004.83, stdev=40273.51 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 46], | 80.00th=[ 51], 90.00th=[ 68], 95.00th=[ 126], | 99.00th=[ 955], 99.50th=[ 1450], 99.90th=[ 2180], | 99.95th=[ 683672], 99.99th=[2432697] bw ( KiB/s): min=30432, max=77152, per=100.00%, avg=39302.77, stdev=6078.48, samples=60 iops : min= 1902, max= 4822, avg=2456.38, stdev=379.93, samples=60 write: IOPS=609, BW=9757KiB/s (9991kB/s)(288MiB/30191msec) slat (usec): min=3, max=269, avg=13.14, stdev= 5.90 clat (msec): min=9, max=2280, avg=159.91, stdev=110.87 lat (msec): min=9, max=2280, avg=159.92, stdev=110.87 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 111], 10.00th=[ 120], 20.00th=[ 131], | 30.00th=[ 138], 40.00th=[ 144], 50.00th=[ 148], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 194], 95.00th=[ 218], | 99.00th=[ 575], 99.50th=[ 986], 99.90th=[ 1787], 99.95th=[ 1821], | 99.99th=[ 2232] bw ( KiB/s): min= 8384, max=16704, per=100.00%, avg=9771.03, stdev=1124.68, samples=60 iops : min= 524, max= 1044, avg=610.68, stdev=70.29, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.07%, 4=0.01%, 10=0.02%, 20=0.11%, 50=63.11% lat (usec) : 100=11.53%, 250=2.89%, 500=0.85%, 750=0.40%, 1000=0.23% lat (msec) : 2=0.64%, 4=0.05%, 10=0.01%, 20=0.23%, 50=0.15% lat (msec) : 100=0.22%, 250=18.83%, 500=0.38%, 750=0.07%, 1000=0.05% lat (msec) : 2000=0.11%, >=2000=0.02% cpu : usr=1.38%, sys=5.76%, ctx=86416, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73693,18411,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=38.1MiB/s (39.0MB/s), 38.1MiB/s-38.1MiB/s (39.0MB/s-39.0MB/s), io=1151MiB (1207MB), run=30191-30191msec WRITE: bw=9757KiB/s (9991kB/s), 9757KiB/s-9757KiB/s (9991kB/s-9991kB/s), io=288MiB (302MB), run=30191-30191msec Disk stats (read/write): vdf: ios=73686/18379, merge=0/0, ticks=62050/2919951, in_queue=3005468, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905791: Fri Jan 7 13:44:26 2022 read: IOPS=2378, BW=37.2MiB/s (38.0MB/s)(1115MiB/30002msec) clat (usec): min=25, max=22807, avg=61.25, stdev=381.31 lat (usec): min=25, max=22807, avg=61.38, stdev=381.32 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 85], 99.50th=[ 95], 99.90th=[ 375], 99.95th=[11863], | 99.99th=[16319] bw ( KiB/s): min=27296, max=57024, per=100.00%, avg=38061.28, stdev=4500.93, samples=60 iops : min= 1706, max= 3564, avg=2378.80, stdev=281.30, samples=60 write: IOPS=594, BW=9519KiB/s (9747kB/s)(279MiB/30002msec) clat (usec): min=227, max=44140, avg=1426.31, stdev=1631.41 lat (usec): min=228, max=44142, avg=1427.72, stdev=1631.39 clat percentiles (usec): | 1.00th=[ 306], 5.00th=[ 594], 10.00th=[ 824], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2024], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[37487], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min= 7552, max=14624, per=100.00%, avg=9519.87, stdev=1023.56, samples=60 iops : min= 472, max= 914, avg=594.98, stdev=63.97, samples=60 lat (usec) : 50=45.15%, 100=34.53%, 250=0.24%, 500=0.72%, 750=0.91% lat (usec) : 1000=1.97% lat (msec) : 2=15.36%, 4=0.76%, 10=0.24%, 20=0.10%, 50=0.02% cpu : usr=1.40%, sys=3.81%, ctx=89212, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71358,17849,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.2MiB/s (38.0MB/s), 37.2MiB/s-37.2MiB/s (38.0MB/s-38.0MB/s), io=1115MiB (1169MB), run=30002-30002msec WRITE: bw=9519KiB/s (9747kB/s), 9519KiB/s-9519KiB/s (9747kB/s-9747kB/s), io=279MiB (292MB), run=30002-30002msec Disk stats (read/write): vdf: ios=71018/17768, merge=0/0, ticks=723/17713, in_queue=28203, util=94.12% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906612: Fri Jan 7 13:45:03 2022 read: IOPS=2428, BW=37.0MiB/s (39.8MB/s)(1149MiB/30279msec) slat (usec): min=2, max=242165, avg=322.49, stdev=2057.88 clat (usec): min=640, max=2561.1k, avg=26571.02, stdev=43251.17 lat (usec): min=644, max=2561.1k, avg=26893.65, stdev=43304.61 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 8455], 10.00th=[ 14222], | 20.00th=[ 17695], 30.00th=[ 20055], 40.00th=[ 21890], | 50.00th=[ 23725], 60.00th=[ 25822], 70.00th=[ 27919], | 80.00th=[ 30802], 90.00th=[ 35914], 95.00th=[ 41157], | 99.00th=[ 81265], 99.50th=[ 183501], 99.90th=[ 299893], | 99.95th=[ 784335], 99.99th=[2399142] bw ( KiB/s): min=25696, max=75296, per=100.00%, avg=39190.63, stdev=6371.03, samples=60 iops : min= 1606, max= 4706, avg=2449.40, stdev=398.19, samples=60 write: IOPS=606, BW=9711KiB/s (9944kB/s)(287MiB/30279msec) slat (usec): min=3, max=183904, avg=332.42, stdev=1967.52 clat (msec): min=5, max=4622, avg=221.49, stdev=215.72 lat (msec): min=13, max=4622, avg=221.82, stdev=215.70 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 124], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 182], 40.00th=[ 192], 50.00th=[ 201], 60.00th=[ 207], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 257], 95.00th=[ 296], | 99.00th=[ 961], 99.50th=[ 1737], 99.90th=[ 3910], 99.95th=[ 3910], | 99.99th=[ 302] bw ( KiB/s): min= 6720, max=15680, per=100.00%, avg=9731.80, stdev=1134.21, samples=60 iops : min= 420, max= 980, avg=608.23, stdev=70.89, samples=60 lat (usec) : 750=0.06%, 1000=0.37% lat (msec) : 2=1.50%, 4=1.43%, 10=1.15%, 20=19.35%, 50=54.06% lat (msec) : 100=1.99%, 250=17.66%, 500=2.02%, 750=0.13%, 1000=0.07% lat (msec) : 2000=0.14%, >=2000=0.08% cpu : usr=0.80%, sys=1.97%, ctx=10359, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73546,18378,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.0MiB/s (39.8MB/s), 37.0MiB/s-37.0MiB/s (39.8MB/s-39.8MB/s), io=1149MiB (1205MB), run=30279-30279msec WRITE: bw=9711KiB/s (9944kB/s), 9711KiB/s-9711KiB/s (9944kB/s-9944kB/s), io=287MiB (301MB), run=30279-30279msec Disk stats (read/write): vdf: ios=73540/18289, merge=0/0, ticks=68397/3568086, in_queue=3672825, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908074: Fri Jan 7 13:45:40 2022 read: IOPS=2298, BW=71.8MiB/s (75.3MB/s)(2155MiB/30001msec) clat (usec): min=30, max=23511, avg=66.04, stdev=376.44 lat (usec): min=30, max=23511, avg=66.17, stdev=376.44 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 70], 95.00th=[ 75], | 99.00th=[ 93], 99.50th=[ 105], 99.90th=[ 347], 99.95th=[11863], | 99.99th=[15664] bw ( KiB/s): min=57664, max=111232, per=100.00%, avg=73641.22, stdev=8170.32, samples=59 iops : min= 1802, max= 3476, avg=2301.29, stdev=255.32, samples=59 write: IOPS=574, BW=17.0MiB/s (18.8MB/s)(539MiB/30001msec) clat (usec): min=253, max=43821, avg=1465.11, stdev=1616.93 lat (usec): min=255, max=43824, avg=1468.15, stdev=1616.90 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 611], 10.00th=[ 832], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[36963], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=15616, max=28480, per=100.00%, avg=18428.75, stdev=1859.10, samples=59 iops : min= 488, max= 890, avg=575.90, stdev=58.10, samples=59 lat (usec) : 50=30.57%, 100=48.92%, 250=0.43%, 500=0.60%, 750=0.98% lat (usec) : 1000=1.66% lat (msec) : 2=15.52%, 4=0.99%, 10=0.21%, 20=0.10%, 50=0.02% cpu : usr=1.56%, sys=3.72%, ctx=86183, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68944,17233,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.8MiB/s (75.3MB/s), 71.8MiB/s-71.8MiB/s (75.3MB/s-75.3MB/s), io=2155MiB (2259MB), run=30001-30001msec WRITE: bw=17.0MiB/s (18.8MB/s), 17.0MiB/s-17.0MiB/s (18.8MB/s-18.8MB/s), io=539MiB (565MB), run=30001-30001msec Disk stats (read/write): vdf: ios=68547/17156, merge=0/0, ticks=708/17456, in_queue=28175, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908671: Fri Jan 7 13:46:17 2022 read: IOPS=2305, BW=72.0MiB/s (75.5MB/s)(2162MiB/30017msec) slat (usec): min=5, max=229, avg=11.45, stdev= 3.73 clat (nsec): min=761, max=17741k, avg=53985.87, stdev=385709.70 lat (usec): min=27, max=17755, avg=65.59, stdev=385.78 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], 20.00th=[ 37], | 30.00th=[ 38], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 46], 80.00th=[ 49], 90.00th=[ 54], 95.00th=[ 60], | 99.00th=[ 74], 99.50th=[ 83], 99.90th=[ 498], 99.95th=[12649], | 99.99th=[16319] bw ( KiB/s): min=58112, max=142528, per=100.00%, avg=73807.27, stdev=11215.48, samples=60 iops : min= 1816, max= 4454, avg=2306.47, stdev=350.48, samples=60 write: IOPS=576, BW=18.0MiB/s (18.9MB/s)(541MiB/30017msec) slat (usec): min=9, max=145, avg=15.09, stdev= 4.02 clat (usec): min=257, max=45998, avg=1449.45, stdev=1683.55 lat (usec): min=269, max=46015, avg=1464.71, stdev=1683.60 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 537], 10.00th=[ 791], 20.00th=[ 1057], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1811], 95.00th=[ 2057], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[38011], 99.95th=[43254], | 99.99th=[45876] bw ( KiB/s): min=15808, max=36800, per=100.00%, avg=18460.25, stdev=2796.70, samples=60 iops : min= 494, max= 1150, avg=576.87, stdev=87.41, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.04%, 50=66.71% lat (usec) : 100=13.00%, 250=0.09%, 500=0.89%, 750=0.94%, 1000=1.60% lat (msec) : 2=15.44%, 4=0.90%, 10=0.21%, 20=0.10%, 50=0.02% cpu : usr=1.58%, sys=6.07%, ctx=86459, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69192,17307,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=72.0MiB/s (75.5MB/s), 72.0MiB/s-72.0MiB/s (75.5MB/s-75.5MB/s), io=2162MiB (2267MB), run=30017-30017msec WRITE: bw=18.0MiB/s (18.9MB/s), 18.0MiB/s-18.0MiB/s (18.9MB/s-18.9MB/s), io=541MiB (567MB), run=30017-30017msec Disk stats (read/write): vdf: ios=69034/17250, merge=0/0, ticks=734/17474, in_queue=28095, util=93.76% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909508: Fri Jan 7 13:46:54 2022 read: IOPS=2301, BW=71.9MiB/s (75.4MB/s)(2158MiB/30001msec) clat (usec): min=30, max=24027, avg=65.83, stdev=413.62 lat (usec): min=30, max=24027, avg=65.97, stdev=413.63 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 92], 99.50th=[ 102], 99.90th=[ 457], 99.95th=[12780], | 99.99th=[17433] bw ( KiB/s): min=57088, max=113664, per=100.00%, avg=73666.17, stdev=8697.78, samples=59 iops : min= 1784, max= 3552, avg=2302.03, stdev=271.76, samples=59 write: IOPS=575, BW=17.0MiB/s (18.8MB/s)(539MiB/30001msec) clat (usec): min=278, max=45145, avg=1463.41, stdev=1660.63 lat (usec): min=281, max=45148, avg=1466.50, stdev=1660.60 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 603], 10.00th=[ 848], 20.00th=[ 1090], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1811], 95.00th=[ 2040], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[38011], 99.95th=[42206], | 99.99th=[44827] bw ( KiB/s): min=15616, max=28928, per=100.00%, avg=18430.92, stdev=1860.77, samples=59 iops : min= 488, max= 904, avg=575.97, stdev=58.15, samples=59 lat (usec) : 50=34.78%, 100=44.76%, 250=0.36%, 500=0.62%, 750=0.92% lat (usec) : 1000=1.60% lat (msec) : 2=15.74%, 4=0.89%, 10=0.20%, 20=0.10%, 50=0.03% cpu : usr=1.37%, sys=3.83%, ctx=86323, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69055,17259,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=2158MiB (2263MB), run=30001-30001msec WRITE: bw=17.0MiB/s (18.8MB/s), 17.0MiB/s-17.0MiB/s (18.8MB/s-18.8MB/s), io=539MiB (566MB), run=30001-30001msec Disk stats (read/write): vdf: ios=68745/17193, merge=0/0, ticks=773/17484, in_queue=28215, util=94.19% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910035: Fri Jan 7 13:47:31 2022 read: IOPS=2344, BW=73.3MiB/s (76.8MB/s)(2201MiB/30039msec) slat (usec): min=3, max=150, avg=11.19, stdev= 4.19 clat (nsec): min=841, max=2539.0M, avg=941664.81, stdev=38861950.71 lat (usec): min=26, max=2539.0k, avg=953.00, stdev=38861.93 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 37], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 49], 90.00th=[ 56], 95.00th=[ 63], | 99.00th=[ 95], 99.50th=[ 131], 99.90th=[ 906], | 99.95th=[ 641729], 99.99th=[2432697] bw ( KiB/s): min=58688, max=131776, per=100.00%, avg=75108.47, stdev=9992.81, samples=60 iops : min= 1834, max= 4118, avg=2347.13, stdev=312.28, samples=60 write: IOPS=586, BW=18.3MiB/s (19.2MB/s)(551MiB/30039msec) slat (usec): min=4, max=180, avg=15.93, stdev= 5.23 clat (usec): min=331, max=75049, avg=13210.92, stdev=6202.51 lat (usec): min=343, max=75064, avg=13227.00, stdev=6202.84 clat percentiles (usec): | 1.00th=[ 2073], 5.00th=[ 5604], 10.00th=[ 7832], 20.00th=[10028], | 30.00th=[11338], 40.00th=[12256], 50.00th=[13042], 60.00th=[13698], | 70.00th=[14353], 80.00th=[15139], 90.00th=[16909], 95.00th=[21365], | 99.00th=[31327], 99.50th=[63701], 99.90th=[70779], 99.95th=[73925], | 99.99th=[74974] bw ( KiB/s): min=15104, max=33152, per=100.00%, avg=18792.10, stdev=2310.78, samples=60 iops : min= 472, max= 1036, avg=587.25, stdev=72.21, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.08%, 50=65.54% lat (usec) : 100=13.55%, 250=0.54%, 500=0.06%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.18%, 4=0.28%, 10=3.59%, 20=14.70%, 50=1.11% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.48%, sys=6.15%, ctx=86709, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70416,17623,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=73.3MiB/s (76.8MB/s), 73.3MiB/s-73.3MiB/s (76.8MB/s-76.8MB/s), io=2201MiB (2307MB), run=30039-30039msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=551MiB (577MB), run=30039-30039msec Disk stats (read/write): vdf: ios=70123/17549, merge=0/0, ticks=61230/222829, in_queue=296867, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910899: Fri Jan 7 13:48:08 2022 read: IOPS=2314, BW=72.3MiB/s (75.8MB/s)(2170MiB/30001msec) clat (usec): min=27, max=17461, avg=64.19, stdev=391.99 lat (usec): min=27, max=17462, avg=64.33, stdev=391.99 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 73], | 99.00th=[ 95], 99.50th=[ 114], 99.90th=[ 717], 99.95th=[12911], | 99.99th=[16712] bw ( KiB/s): min=60544, max=106880, per=100.00%, avg=74058.45, stdev=7851.44, samples=60 iops : min= 1892, max= 3340, avg=2314.32, stdev=245.34, samples=60 write: IOPS=578, BW=18.1MiB/s (18.0MB/s)(542MiB/30001msec) clat (usec): min=270, max=45459, avg=1460.09, stdev=1587.84 lat (usec): min=273, max=45460, avg=1463.07, stdev=1587.82 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 627], 10.00th=[ 873], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1811], 95.00th=[ 2040], | 99.00th=[ 8029], 99.50th=[ 9765], 99.90th=[36963], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=15360, max=27904, per=100.00%, avg=18515.88, stdev=1777.48, samples=60 iops : min= 480, max= 872, avg=578.62, stdev=55.54, samples=60 lat (usec) : 50=39.68%, 100=39.70%, 250=0.47%, 500=0.66%, 750=0.84% lat (usec) : 1000=1.46% lat (msec) : 2=16.02%, 4=0.88%, 10=0.18%, 20=0.10%, 50=0.02% cpu : usr=1.55%, sys=3.87%, ctx=86791, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69427,17359,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=2170MiB (2275MB), run=30001-30001msec WRITE: bw=18.1MiB/s (18.0MB/s), 18.1MiB/s-18.1MiB/s (18.0MB/s-18.0MB/s), io=542MiB (569MB), run=30001-30001msec Disk stats (read/write): vdf: ios=69056/17260, merge=0/0, ticks=754/17523, in_queue=28179, util=94.00% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911951: Fri Jan 7 13:48:45 2022 read: IOPS=2352, BW=73.5MiB/s (77.1MB/s)(2208MiB/30040msec) slat (usec): min=2, max=261, avg=11.17, stdev= 5.46 clat (nsec): min=711, max=2543.8M, avg=1044416.26, stdev=42990024.49 lat (usec): min=25, max=2543.8k, avg=1055.75, stdev=42990.02 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 37], 40.00th=[ 38], | 50.00th=[ 41], 60.00th=[ 44], 70.00th=[ 47], | 80.00th=[ 51], 90.00th=[ 61], 95.00th=[ 74], | 99.00th=[ 241], 99.50th=[ 494], 99.90th=[ 1237], | 99.95th=[ 675283], 99.99th=[2499806] bw ( KiB/s): min=42688, max=130880, per=100.00%, avg=75371.73, stdev=12365.64, samples=60 iops : min= 1334, max= 4090, avg=2355.33, stdev=386.41, samples=60 write: IOPS=588, BW=18.4MiB/s (19.3MB/s)(552MiB/30040msec) slat (usec): min=5, max=253, avg=16.92, stdev= 7.54 clat (msec): min=8, max=2025, avg=80.74, stdev=61.76 lat (msec): min=8, max=2025, avg=80.76, stdev=61.76 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 58], 10.00th=[ 63], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 73], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 83], 90.00th=[ 92], 95.00th=[ 121], | 99.00th=[ 230], 99.50th=[ 527], 99.90th=[ 995], 99.95th=[ 1083], | 99.99th=[ 1653] bw ( KiB/s): min=13120, max=30336, per=99.88%, avg=18802.13, stdev=2417.89, samples=60 iops : min= 410, max= 948, avg=587.57, stdev=75.56, samples=60 lat (nsec) : 750=0.01%, 1000=0.09% lat (usec) : 2=0.08%, 4=0.01%, 10=0.02%, 20=0.10%, 50=62.21% lat (usec) : 100=15.43%, 250=1.28%, 500=0.38%, 750=0.14%, 1000=0.09% lat (msec) : 2=0.09%, 10=0.03%, 20=0.28%, 50=0.18%, 100=17.97% lat (msec) : 250=1.36%, 500=0.09%, 750=0.06%, 1000=0.04%, 2000=0.03% lat (msec) : >=2000=0.02% cpu : usr=1.49%, sys=6.20%, ctx=85585, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=70666,17672,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=73.5MiB/s (77.1MB/s), 73.5MiB/s-73.5MiB/s (77.1MB/s-77.1MB/s), io=2208MiB (2316MB), run=30040-30040msec WRITE: bw=18.4MiB/s (19.3MB/s), 18.4MiB/s-18.4MiB/s (19.3MB/s-19.3MB/s), io=552MiB (579MB), run=30040-30040msec Disk stats (read/write): vdf: ios=70433/17582, merge=0/0, ticks=65298/1408682, in_queue=1491627, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912584: Fri Jan 7 13:49:22 2022 read: IOPS=2294, BW=71.7MiB/s (75.2MB/s)(2151MiB/30001msec) clat (usec): min=26, max=71372, avg=63.77, stdev=478.82 lat (usec): min=27, max=71372, avg=63.91, stdev=478.82 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 91], 99.50th=[ 105], 99.90th=[ 457], 99.95th=[12649], | 99.99th=[16450] bw ( KiB/s): min=52480, max=120384, per=100.00%, avg=73534.92, stdev=8760.97, samples=59 iops : min= 1640, max= 3762, avg=2297.97, stdev=273.78, samples=59 write: IOPS=573, BW=17.9MiB/s (18.8MB/s)(538MiB/30001msec) clat (usec): min=267, max=83013, avg=1476.05, stdev=1723.20 lat (usec): min=269, max=83015, avg=1479.03, stdev=1723.17 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 881], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1844], 95.00th=[ 2089], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[37487], 99.95th=[41681], | 99.99th=[47973] bw ( KiB/s): min=13760, max=30976, per=100.00%, avg=18397.29, stdev=2142.52, samples=59 iops : min= 430, max= 968, avg=574.92, stdev=66.95, samples=59 lat (usec) : 50=42.79%, 100=36.71%, 250=0.39%, 500=0.64%, 750=0.79% lat (usec) : 1000=1.43% lat (msec) : 2=15.91%, 4=1.02%, 10=0.19%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.35%, sys=3.89%, ctx=86057, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68837,17216,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=71.7MiB/s (75.2MB/s), 71.7MiB/s-71.7MiB/s (75.2MB/s-75.2MB/s), io=2151MiB (2256MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=538MiB (564MB), run=30001-30001msec Disk stats (read/write): vdf: ios=68496/17137, merge=0/0, ticks=805/17789, in_queue=28199, util=94.10% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913304: Fri Jan 7 13:49:59 2022 read: IOPS=2347, BW=73.4MiB/s (76.9MB/s)(2215MiB/30190msec) slat (usec): min=2, max=210, avg=10.92, stdev= 6.24 clat (nsec): min=659, max=2519.8M, avg=1066686.82, stdev=41838835.46 lat (usec): min=26, max=2519.8k, avg=1077.75, stdev=41838.83 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 43], 60.00th=[ 46], 70.00th=[ 50], | 80.00th=[ 56], 90.00th=[ 69], 95.00th=[ 105], | 99.00th=[ 857], 99.50th=[ 1582], 99.90th=[ 2671], | 99.95th=[ 650118], 99.99th=[2499806] bw ( KiB/s): min=58688, max=144576, per=100.00%, avg=75607.70, stdev=10631.80, samples=60 iops : min= 1834, max= 4518, avg=2362.73, stdev=332.25, samples=60 write: IOPS=587, BW=18.3MiB/s (19.2MB/s)(554MiB/30190msec) slat (usec): min=4, max=186, avg=17.17, stdev= 7.75 clat (msec): min=10, max=2625, avg=165.97, stdev=95.54 lat (msec): min=10, max=2625, avg=165.99, stdev=95.54 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 122], 10.00th=[ 136], 20.00th=[ 144], | 30.00th=[ 148], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 161], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 201], 95.00th=[ 215], | 99.00th=[ 388], 99.50th=[ 852], 99.90th=[ 1586], 99.95th=[ 1586], | 99.99th=[ 2635] bw ( KiB/s): min=15360, max=30976, per=100.00%, avg=18804.88, stdev=2017.70, samples=60 iops : min= 480, max= 968, avg=587.65, stdev=63.05, samples=60 lat (nsec) : 750=0.01%, 1000=0.17% lat (usec) : 2=0.05%, 4=0.01%, 10=0.03%, 20=0.08%, 50=56.48% lat (usec) : 100=18.97%, 250=1.94%, 500=0.88%, 750=0.47%, 1000=0.22% lat (msec) : 2=0.45%, 4=0.16%, 20=0.19%, 50=0.21%, 100=0.25% lat (msec) : 250=18.85%, 500=0.36%, 750=0.05%, 1000=0.05%, 2000=0.09% lat (msec) : >=2000=0.02% cpu : usr=1.52%, sys=6.11%, ctx=83182, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70884,17724,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=73.4MiB/s (76.9MB/s), 73.4MiB/s-73.4MiB/s (76.9MB/s-76.9MB/s), io=2215MiB (2323MB), run=30190-30190msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=554MiB (581MB), run=30190-30190msec Disk stats (read/write): vdf: ios=70880/17671, merge=0/0, ticks=67943/2913108, in_queue=3006737, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914718: Fri Jan 7 13:50:36 2022 read: IOPS=2302, BW=71.9MiB/s (75.4MB/s)(2158MiB/30001msec) clat (usec): min=27, max=20449, avg=64.02, stdev=413.76 lat (usec): min=27, max=20449, avg=64.16, stdev=413.76 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 89], 99.50th=[ 101], 99.90th=[ 445], 99.95th=[13698], | 99.99th=[17171] bw ( KiB/s): min=53952, max=106496, per=100.00%, avg=73675.03, stdev=8462.70, samples=60 iops : min= 1686, max= 3328, avg=2302.33, stdev=264.46, samples=60 write: IOPS=575, BW=17.0MiB/s (18.9MB/s)(540MiB/30001msec) clat (usec): min=261, max=65244, avg=1469.78, stdev=1691.96 lat (usec): min=264, max=65248, avg=1472.78, stdev=1691.93 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 635], 10.00th=[ 898], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1827], 95.00th=[ 2073], | 99.00th=[ 8094], 99.50th=[ 9634], 99.90th=[38011], 99.95th=[42206], | 99.99th=[53740] bw ( KiB/s): min=13440, max=27840, per=100.00%, avg=18420.40, stdev=1828.56, samples=60 iops : min= 420, max= 870, avg=575.63, stdev=57.14, samples=60 lat (usec) : 50=40.76%, 100=38.81%, 250=0.34%, 500=0.63%, 750=0.78% lat (usec) : 1000=1.40% lat (msec) : 2=16.00%, 4=0.98%, 10=0.18%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.55%, sys=3.74%, ctx=86347, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69066,17269,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=2158MiB (2263MB), run=30001-30001msec WRITE: bw=17.0MiB/s (18.9MB/s), 17.0MiB/s-17.0MiB/s (18.9MB/s-18.9MB/s), io=540MiB (566MB), run=30001-30001msec Disk stats (read/write): vdf: ios=68807/17205, merge=0/0, ticks=770/17710, in_queue=28162, util=93.96% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915284: Fri Jan 7 13:51:13 2022 read: IOPS=2339, BW=73.1MiB/s (76.7MB/s)(2209MiB/30205msec) slat (usec): min=2, max=211432, avg=335.35, stdev=2204.15 clat (usec): min=768, max=2565.1k, avg=27683.38, stdev=48016.27 lat (usec): min=771, max=2565.1k, avg=28018.88, stdev=48069.83 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 4817], 10.00th=[ 14746], | 20.00th=[ 18744], 30.00th=[ 20841], 40.00th=[ 22938], | 50.00th=[ 25035], 60.00th=[ 26870], 70.00th=[ 28967], | 80.00th=[ 32113], 90.00th=[ 37487], 95.00th=[ 42730], | 99.00th=[ 83362], 99.50th=[ 193987], 99.90th=[ 227541], | 99.95th=[ 717226], 99.99th=[2533360] bw ( KiB/s): min=62528, max=154752, per=100.00%, avg=75325.28, stdev=12702.01, samples=60 iops : min= 1954, max= 4836, avg=2353.90, stdev=396.95, samples=60 write: IOPS=585, BW=18.3MiB/s (19.2MB/s)(552MiB/30205msec) slat (usec): min=4, max=58460, avg=346.78, stdev=1605.61 clat (msec): min=5, max=3757, avg=229.34, stdev=232.96 lat (msec): min=5, max=3758, avg=229.69, stdev=232.95 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 127], 10.00th=[ 148], 20.00th=[ 169], | 30.00th=[ 184], 40.00th=[ 197], 50.00th=[ 205], 60.00th=[ 213], | 70.00th=[ 224], 80.00th=[ 236], 90.00th=[ 264], 95.00th=[ 305], | 99.00th=[ 1133], 99.50th=[ 2106], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 3742] bw ( KiB/s): min=15680, max=32384, per=99.93%, avg=18711.57, stdev=2251.93, samples=60 iops : min= 490, max= 1012, avg=584.73, stdev=70.37, samples=60 lat (usec) : 1000=0.15% lat (msec) : 2=1.73%, 4=1.75%, 10=1.01%, 20=15.85%, 50=57.34% lat (msec) : 100=2.08%, 250=17.17%, 500=2.45%, 750=0.09%, 1000=0.13% lat (msec) : 2000=0.13%, >=2000=0.12% cpu : usr=1.00%, sys=2.06%, ctx=9950, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70679,17675,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=73.1MiB/s (76.7MB/s), 73.1MiB/s-73.1MiB/s (76.7MB/s-76.7MB/s), io=2209MiB (2316MB), run=30205-30205msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=552MiB (579MB), run=30205-30205msec Disk stats (read/write): vdf: ios=70676/17634, merge=0/0, ticks=79891/3567073, in_queue=3674612, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916223: Fri Jan 7 13:51:50 2022 read: IOPS=2118, BW=132MiB/s (139MB/s)(3973MiB/30002msec) clat (usec): min=36, max=62088, avg=77.54, stdev=482.36 lat (usec): min=36, max=62088, avg=77.69, stdev=482.36 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 81], 95.00th=[ 89], | 99.00th=[ 117], 99.50th=[ 139], 99.90th=[ 1188], 99.95th=[13042], | 99.99th=[17957] bw ( KiB/s): min=109056, max=211072, per=100.00%, avg=135620.27, stdev=15893.27, samples=60 iops : min= 1704, max= 3298, avg=2119.07, stdev=248.33, samples=60 write: IOPS=532, BW=33.3MiB/s (34.9MB/s)(999MiB/30002msec) clat (usec): min=349, max=45126, avg=1553.59, stdev=1577.71 lat (usec): min=355, max=45157, avg=1560.01, stdev=1577.69 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 668], 10.00th=[ 873], 20.00th=[ 1156], | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 7963], 99.50th=[ 9896], 99.90th=[17171], 99.95th=[42730], | 99.99th=[44827] bw ( KiB/s): min=28672, max=54656, per=100.00%, avg=34097.07, stdev=3572.13, samples=60 iops : min= 448, max= 854, avg=532.77, stdev=55.81, samples=60 lat (usec) : 50=5.29%, 100=72.77%, 250=1.67%, 500=0.49%, 750=1.03% lat (usec) : 1000=1.30% lat (msec) : 2=15.62%, 4=1.51%, 10=0.19%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.72%, sys=3.75%, ctx=79560, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63572,15984,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3973MiB (4166MB), run=30002-30002msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=999MiB (1048MB), run=30002-30002msec Disk stats (read/write): vdf: ios=63401/15916, merge=0/0, ticks=791/16784, in_queue=28122, util=93.87% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916839: Fri Jan 7 13:52:27 2022 read: IOPS=2085, BW=130MiB/s (137MB/s)(3911MiB/30002msec) slat (usec): min=6, max=442, avg=13.84, stdev= 6.17 clat (nsec): min=668, max=21834k, avg=62170.23, stdev=415947.24 lat (usec): min=33, max=21851, avg=76.18, stdev=416.01 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 693], 99.95th=[12780], | 99.99th=[17433] bw ( KiB/s): min=96512, max=203264, per=100.00%, avg=133501.87, stdev=15892.41, samples=60 iops : min= 1508, max= 3176, avg=2085.97, stdev=248.32, samples=60 write: IOPS=524, BW=32.8MiB/s (34.4MB/s)(983MiB/30002msec) slat (usec): min=11, max=421, avg=20.88, stdev= 8.48 clat (usec): min=341, max=136630, avg=1575.36, stdev=2016.26 lat (usec): min=358, max=136648, avg=1596.43, stdev=2016.06 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 644], 10.00th=[ 881], 20.00th=[ 1156], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1418], 60.00th=[ 1483], | 70.00th=[ 1582], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[39060], 99.95th=[43254], | 99.99th=[52691] bw ( KiB/s): min=22784, max=52608, per=100.00%, avg=33550.93, stdev=3606.71, samples=60 iops : min= 356, max= 822, avg=524.23, stdev=56.35, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.09%, 4=0.01%, 10=0.02%, 20=0.06%, 50=46.43% lat (usec) : 100=32.80%, 250=0.33%, 500=0.48%, 750=0.99%, 1000=1.24% lat (msec) : 2=15.62%, 4=1.52%, 10=0.21%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.44%, sys=6.55%, ctx=78153, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62579,15728,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=130MiB/s (137MB/s), 130MiB/s-130MiB/s (137MB/s-137MB/s), io=3911MiB (4101MB), run=30002-30002msec WRITE: bw=32.8MiB/s (34.4MB/s), 32.8MiB/s-32.8MiB/s (34.4MB/s-34.4MB/s), io=983MiB (1031MB), run=30002-30002msec Disk stats (read/write): vdf: ios=62362/15669, merge=0/0, ticks=703/17161, in_queue=27947, util=93.26% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917789: Fri Jan 7 13:53:04 2022 read: IOPS=2104, BW=132MiB/s (138MB/s)(3947MiB/30012msec) clat (usec): min=33, max=60830, avg=71.23, stdev=490.61 lat (usec): min=33, max=60830, avg=71.37, stdev=490.61 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 685], 99.95th=[14353], | 99.99th=[17433] bw ( KiB/s): min=108288, max=212480, per=100.00%, avg=134722.13, stdev=15070.05, samples=60 iops : min= 1692, max= 3320, avg=2105.03, stdev=235.47, samples=60 write: IOPS=529, BW=33.1MiB/s (34.7MB/s)(992MiB/30012msec) clat (usec): min=343, max=46307, avg=1592.33, stdev=1597.25 lat (usec): min=349, max=46310, avg=1598.48, stdev=1597.19 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 685], 10.00th=[ 963], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1450], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2245], | 99.00th=[ 9241], 99.50th=[ 9896], 99.90th=[18220], 99.95th=[40633], | 99.99th=[45351] bw ( KiB/s): min=27648, max=55168, per=100.00%, avg=33870.93, stdev=3692.62, samples=60 iops : min= 432, max= 862, avg=529.23, stdev=57.70, samples=60 lat (usec) : 50=22.94%, 100=56.04%, 250=0.80%, 500=0.48%, 750=0.80% lat (usec) : 1000=1.02% lat (msec) : 2=15.93%, 4=1.67%, 10=0.20%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.62%, sys=3.76%, ctx=79038, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63152,15877,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3947MiB (4139MB), run=30012-30012msec WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=992MiB (1041MB), run=30012-30012msec Disk stats (read/write): vdf: ios=62858/15814, merge=0/0, ticks=779/17375, in_queue=28139, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918733: Fri Jan 7 13:53:41 2022 read: IOPS=2144, BW=134MiB/s (141MB/s)(4026MiB/30039msec) slat (usec): min=3, max=113, avg=12.34, stdev= 4.50 clat (nsec): min=709, max=2525.2M, avg=1141336.16, stdev=46879710.58 lat (usec): min=29, max=2525.2k, avg=1153.82, stdev=46879.70 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 50], | 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 69], | 99.00th=[ 92], 99.50th=[ 112], 99.90th=[ 898], | 99.95th=[ 616563], 99.99th=[2499806] bw ( KiB/s): min=107392, max=233344, per=100.00%, avg=137429.52, stdev=17690.02, samples=60 iops : min= 1678, max= 3646, avg=2147.33, stdev=276.41, samples=60 write: IOPS=538, BW=33.7MiB/s (35.3MB/s)(1011MiB/30039msec) slat (usec): min=7, max=136, avg=20.50, stdev= 6.90 clat (usec): min=1147, max=73804, avg=13944.03, stdev=6135.75 lat (usec): min=1160, max=73825, avg=13964.69, stdev=6136.11 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 7439], 10.00th=[ 8455], 20.00th=[10159], | 30.00th=[11731], 40.00th=[13042], 50.00th=[13829], 60.00th=[14615], | 70.00th=[15270], 80.00th=[16057], 90.00th=[17957], 95.00th=[22414], | 99.00th=[32637], 99.50th=[63177], 99.90th=[68682], 99.95th=[72877], | 99.99th=[73925] bw ( KiB/s): min=29056, max=59392, per=100.00%, avg=34499.18, stdev=3979.71, samples=60 iops : min= 454, max= 928, avg=539.03, stdev=62.19, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.05%, 50=56.34% lat (usec) : 100=22.83%, 250=0.46%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.08%, 4=0.38%, 10=3.23%, 20=15.04%, 50=1.21% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.70%, sys=5.85%, ctx=79644, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=64423,16175,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=4026MiB (4222MB), run=30039-30039msec WRITE: bw=33.7MiB/s (35.3MB/s), 33.7MiB/s-33.7MiB/s (35.3MB/s-35.3MB/s), io=1011MiB (1060MB), run=30039-30039msec Disk stats (read/write): vdf: ios=64116/16099, merge=0/0, ticks=67584/216616, in_queue=296850, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919313: Fri Jan 7 13:54:18 2022 read: IOPS=2101, BW=131MiB/s (138MB/s)(3941MiB/30002msec) clat (usec): min=32, max=75300, avg=69.91, stdev=517.93 lat (usec): min=33, max=75300, avg=70.05, stdev=517.94 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 95], 99.50th=[ 109], 99.90th=[ 545], 99.95th=[13173], | 99.99th=[17695] bw ( KiB/s): min=93824, max=211072, per=100.00%, avg=134502.40, stdev=16584.70, samples=60 iops : min= 1466, max= 3298, avg=2101.60, stdev=259.14, samples=60 write: IOPS=528, BW=33.0MiB/s (34.6MB/s)(991MiB/30002msec) clat (usec): min=359, max=118513, avg=1599.86, stdev=1836.06 lat (usec): min=364, max=118520, avg=1605.89, stdev=1836.02 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 693], 10.00th=[ 979], 20.00th=[ 1221], | 30.00th=[ 1319], 40.00th=[ 1369], 50.00th=[ 1450], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2212], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[37487], 99.95th=[42206], | 99.99th=[45876] bw ( KiB/s): min=23168, max=54784, per=100.00%, avg=33824.00, stdev=3937.52, samples=60 iops : min= 362, max= 856, avg=528.50, stdev=61.52, samples=60 lat (usec) : 50=25.71%, 100=53.61%, 250=0.48%, 500=0.46%, 750=0.72% lat (usec) : 1000=0.98% lat (msec) : 2=16.11%, 4=1.59%, 10=0.19%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.44%, sys=3.78%, ctx=78915, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63048,15856,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3941MiB (4132MB), run=30002-30002msec WRITE: bw=33.0MiB/s (34.6MB/s), 33.0MiB/s-33.0MiB/s (34.6MB/s-34.6MB/s), io=991MiB (1039MB), run=30002-30002msec Disk stats (read/write): vdf: ios=62755/15786, merge=0/0, ticks=790/17459, in_queue=28235, util=94.25% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920083: Fri Jan 7 13:54:55 2022 read: IOPS=2154, BW=135MiB/s (141MB/s)(4056MiB/30121msec) slat (usec): min=2, max=172, avg=10.90, stdev= 4.33 clat (nsec): min=771, max=2520.9M, avg=1094762.66, stdev=44732993.21 lat (usec): min=28, max=2520.9k, avg=1105.81, stdev=44732.99 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 52], | 80.00th=[ 57], 90.00th=[ 66], 95.00th=[ 77], | 99.00th=[ 306], 99.50th=[ 644], 99.90th=[ 1631], | 99.95th=[ 675283], 99.99th=[2499806] bw ( KiB/s): min=114176, max=274304, per=100.00%, avg=138456.62, stdev=20800.47, samples=60 iops : min= 1784, max= 4286, avg=2163.35, stdev=325.01, samples=60 write: IOPS=540, BW=33.8MiB/s (35.4MB/s)(1018MiB/30121msec) slat (usec): min=6, max=212, avg=21.33, stdev= 8.50 clat (msec): min=6, max=1923, avg=88.02, stdev=75.57 lat (msec): min=6, max=1923, avg=88.04, stdev=75.57 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 65], 10.00th=[ 69], 20.00th=[ 72], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 100], 95.00th=[ 130], | 99.00th=[ 234], 99.50th=[ 592], 99.90th=[ 1318], 99.95th=[ 1502], | 99.99th=[ 1854] bw ( KiB/s): min=29696, max=63872, per=100.00%, avg=34655.15, stdev=4388.64, samples=60 iops : min= 464, max= 998, avg=541.47, stdev=68.59, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.04%, 50=53.57% lat (usec) : 100=24.27%, 250=1.06%, 500=0.43%, 750=0.17%, 1000=0.10% lat (msec) : 2=0.17%, 10=0.02%, 20=0.28%, 50=0.19%, 100=17.66% lat (msec) : 250=1.73%, 500=0.08%, 750=0.06%, 1000=0.03%, 2000=0.06% lat (msec) : >=2000=0.02% cpu : usr=1.61%, sys=5.52%, ctx=78825, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=64901,16287,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4056MiB (4253MB), run=30121-30121msec WRITE: bw=33.8MiB/s (35.4MB/s), 33.8MiB/s-33.8MiB/s (35.4MB/s-35.4MB/s), io=1018MiB (1067MB), run=30121-30121msec Disk stats (read/write): vdf: ios=64614/16181, merge=0/0, ticks=61425/1413656, in_queue=1492554, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921275: Fri Jan 7 13:55:32 2022 read: IOPS=2104, BW=132MiB/s (138MB/s)(3947MiB/30002msec) clat (usec): min=33, max=26076, avg=70.86, stdev=437.44 lat (usec): min=33, max=26076, avg=71.01, stdev=437.44 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 97], 99.50th=[ 111], 99.90th=[ 824], 99.95th=[13173], | 99.99th=[16909] bw ( KiB/s): min=93824, max=220800, per=100.00%, avg=134726.35, stdev=17355.80, samples=60 iops : min= 1466, max= 3450, avg=2105.08, stdev=271.19, samples=60 write: IOPS=529, BW=33.1MiB/s (34.7MB/s)(992MiB/30002msec) clat (usec): min=350, max=112024, avg=1592.88, stdev=1915.90 lat (usec): min=357, max=112029, avg=1599.07, stdev=1915.87 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 676], 10.00th=[ 963], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2212], | 99.00th=[ 9241], 99.50th=[ 9896], 99.90th=[39060], 99.95th=[43779], | 99.99th=[50070] bw ( KiB/s): min=25088, max=56832, per=100.00%, avg=33871.98, stdev=4067.77, samples=60 iops : min= 392, max= 888, avg=529.23, stdev=63.57, samples=60 lat (usec) : 50=21.23%, 100=58.02%, 250=0.55%, 500=0.49%, 750=0.79% lat (usec) : 1000=1.00% lat (msec) : 2=16.02%, 4=1.57%, 10=0.17%, 20=0.12%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.67%, sys=3.75%, ctx=79032, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63152,15877,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3947MiB (4139MB), run=30002-30002msec WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=992MiB (1041MB), run=30002-30002msec Disk stats (read/write): vdf: ios=62947/15830, merge=0/0, ticks=755/17446, in_queue=28177, util=93.98% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922072: Fri Jan 7 13:56:09 2022 read: IOPS=2129, BW=133MiB/s (140MB/s)(4019MiB/30197msec) slat (usec): min=2, max=480, avg=10.90, stdev= 5.67 clat (nsec): min=687, max=2555.0M, avg=1140964.68, stdev=44601951.90 lat (usec): min=28, max=2555.0k, avg=1152.01, stdev=44601.95 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 38], 30.00th=[ 39], 40.00th=[ 42], | 50.00th=[ 45], 60.00th=[ 49], 70.00th=[ 53], | 80.00th=[ 60], 90.00th=[ 72], 95.00th=[ 110], | 99.00th=[ 1287], 99.50th=[ 2311], 99.90th=[ 3556], | 99.95th=[ 666895], 99.99th=[2499806] bw ( KiB/s): min=108160, max=289152, per=100.00%, avg=137192.70, stdev=22687.94, samples=60 iops : min= 1690, max= 4518, avg=2143.62, stdev=354.51, samples=60 write: IOPS=534, BW=33.4MiB/s (35.0MB/s)(1009MiB/30197msec) slat (usec): min=6, max=410, avg=22.04, stdev=10.13 clat (msec): min=10, max=3446, avg=182.39, stdev=105.49 lat (msec): min=11, max=3446, avg=182.41, stdev=105.49 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 138], 10.00th=[ 150], 20.00th=[ 161], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 174], 60.00th=[ 178], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 218], 95.00th=[ 241], | 99.00th=[ 393], 99.50th=[ 684], 99.90th=[ 1921], 99.95th=[ 2198], | 99.99th=[ 3306] bw ( KiB/s): min=29440, max=61824, per=100.00%, avg=34241.13, stdev=4141.12, samples=60 iops : min= 460, max= 966, avg=535.00, stdev=64.72, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.05%, 50=50.57% lat (usec) : 100=24.91%, 250=1.68%, 500=0.81%, 750=0.51%, 1000=0.29% lat (msec) : 2=0.49%, 4=0.45%, 20=0.03%, 50=0.36%, 100=0.23% lat (msec) : 250=18.66%, 500=0.67%, 750=0.06%, 1000=0.03%, 2000=0.06% lat (msec) : >=2000=0.03% cpu : usr=1.75%, sys=5.36%, ctx=75704, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64306,16146,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4019MiB (4214MB), run=30197-30197msec WRITE: bw=33.4MiB/s (35.0MB/s), 33.4MiB/s-33.4MiB/s (35.0MB/s-35.0MB/s), io=1009MiB (1058MB), run=30197-30197msec Disk stats (read/write): vdf: ios=64303/16110, merge=0/0, ticks=66940/2920153, in_queue=3005670, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923041: Fri Jan 7 13:56:46 2022 read: IOPS=2101, BW=131MiB/s (138MB/s)(3940MiB/30001msec) clat (usec): min=31, max=20460, avg=66.32, stdev=408.50 lat (usec): min=31, max=20460, avg=66.45, stdev=408.50 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 89], 99.50th=[ 100], 99.90th=[ 371], 99.95th=[13042], | 99.99th=[16909] bw ( KiB/s): min=101888, max=197760, per=100.00%, avg=134768.81, stdev=14182.11, samples=59 iops : min= 1592, max= 3090, avg=2105.76, stdev=221.60, samples=59 write: IOPS=528, BW=33.0MiB/s (34.6MB/s)(991MiB/30001msec) clat (usec): min=340, max=45514, avg=1614.43, stdev=1647.57 lat (usec): min=345, max=45520, avg=1620.43, stdev=1647.54 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 725], 10.00th=[ 1004], 20.00th=[ 1221], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[38011], 99.95th=[42206], | 99.99th=[45351] bw ( KiB/s): min=28672, max=51456, per=100.00%, avg=33852.75, stdev=3241.43, samples=59 iops : min= 448, max= 804, avg=528.95, stdev=50.65, samples=59 lat (usec) : 50=33.40%, 100=46.11%, 250=0.31%, 500=0.48%, 750=0.61% lat (usec) : 1000=0.91% lat (msec) : 2=16.19%, 4=1.63%, 10=0.22%, 20=0.12%, 50=0.02% cpu : usr=1.48%, sys=3.55%, ctx=78909, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63041,15853,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3940MiB (4131MB), run=30001-30001msec WRITE: bw=33.0MiB/s (34.6MB/s), 33.0MiB/s-33.0MiB/s (34.6MB/s-34.6MB/s), io=991MiB (1039MB), run=30001-30001msec Disk stats (read/write): vdf: ios=62753/15784, merge=0/0, ticks=702/17724, in_queue=28255, util=94.30% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923640: Fri Jan 7 13:57:23 2022 read: IOPS=2130, BW=133MiB/s (140MB/s)(4031MiB/30273msec) slat (usec): min=2, max=219971, avg=361.37, stdev=2156.79 clat (usec): min=1023, max=2554.4k, avg=30245.98, stdev=48379.04 lat (usec): min=1028, max=2558.1k, avg=30607.49, stdev=48437.59 clat percentiles (usec): | 1.00th=[ 1680], 5.00th=[ 5342], 10.00th=[ 15926], | 20.00th=[ 20579], 30.00th=[ 23200], 40.00th=[ 25297], | 50.00th=[ 27395], 60.00th=[ 29492], 70.00th=[ 32113], | 80.00th=[ 34866], 90.00th=[ 40109], 95.00th=[ 46400], | 99.00th=[ 87557], 99.50th=[ 223347], 99.90th=[ 299893], | 99.95th=[ 683672], 99.99th=[2533360] bw ( KiB/s): min=105728, max=276096, per=100.00%, avg=137470.68, stdev=22227.55, samples=60 iops : min= 1652, max= 4314, avg=2147.95, stdev=347.31, samples=60 write: IOPS=535, BW=33.4MiB/s (35.1MB/s)(1013MiB/30273msec) slat (usec): min=6, max=255678, avg=403.44, stdev=3103.38 clat (msec): min=5, max=4131, avg=251.41, stdev=240.62 lat (msec): min=7, max=4131, avg=251.81, stdev=240.60 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 146], 10.00th=[ 165], 20.00th=[ 188], | 30.00th=[ 207], 40.00th=[ 224], 50.00th=[ 232], 60.00th=[ 239], | 70.00th=[ 247], 80.00th=[ 259], 90.00th=[ 292], 95.00th=[ 330], | 99.00th=[ 1167], 99.50th=[ 1720], 99.90th=[ 4077], 99.95th=[ 4077], | 99.99th=[ 4144] bw ( KiB/s): min=24832, max=55168, per=100.00%, avg=34285.05, stdev=3835.35, samples=60 iops : min= 388, max= 862, avg=535.70, stdev=59.93, samples=60 lat (msec) : 2=1.30%, 4=1.86%, 10=1.51%, 20=9.92%, 50=62.57% lat (msec) : 100=2.65%, 250=14.55%, 500=5.21%, 750=0.10%, 1000=0.06% lat (msec) : 2000=0.19%, >=2000=0.08% cpu : usr=1.57%, sys=2.03%, ctx=9025, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64499,16201,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4031MiB (4227MB), run=30273-30273msec WRITE: bw=33.4MiB/s (35.1MB/s), 33.4MiB/s-33.4MiB/s (35.1MB/s-35.1MB/s), io=1013MiB (1062MB), run=30273-30273msec Disk stats (read/write): vdf: ios=64492/16144, merge=0/0, ticks=65417/3577021, in_queue=3679686, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924579: Fri Jan 7 13:58:00 2022 read: IOPS=2040, BW=159MiB/s (167MB/s)(4782MiB/30001msec) clat (usec): min=41, max=19348, avg=83.90, stdev=442.21 lat (usec): min=42, max=19348, avg=84.04, stdev=442.21 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 79], 90.00th=[ 87], 95.00th=[ 96], | 99.00th=[ 127], 99.50th=[ 153], 99.90th=[ 1352], 99.95th=[14222], | 99.99th=[17695] bw ( KiB/s): min=134240, max=249760, per=99.88%, avg=163013.36, stdev=18053.32, samples=59 iops : min= 1678, max= 3122, avg=2037.64, stdev=225.67, samples=59 write: IOPS=512, BW=40.0MiB/s (41.0MB/s)(1201MiB/30001msec) clat (usec): min=399, max=46267, avg=1601.04, stdev=1618.30 lat (usec): min=407, max=46275, avg=1608.91, stdev=1618.23 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 685], 10.00th=[ 873], 20.00th=[ 1172], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2008], 95.00th=[ 2278], | 99.00th=[ 8979], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min=35040, max=65280, per=99.93%, avg=40960.08, stdev=4286.08, samples=59 iops : min= 438, max= 816, avg=512.00, stdev=53.58, samples=59 lat (usec) : 50=0.49%, 100=76.49%, 250=2.76%, 500=0.35%, 750=1.16% lat (usec) : 1000=1.27% lat (msec) : 2=15.34%, 4=1.81%, 10=0.19%, 20=0.13%, 50=0.02% cpu : usr=1.64%, sys=4.03%, ctx=76582, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61205,15372,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4782MiB (5014MB), run=30001-30001msec WRITE: bw=40.0MiB/s (41.0MB/s), 40.0MiB/s-40.0MiB/s (41.0MB/s-41.0MB/s), io=1201MiB (1259MB), run=30001-30001msec Disk stats (read/write): vdf: ios=60907/15302, merge=0/0, ticks=803/16581, in_queue=28094, util=93.76% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925578: Fri Jan 7 13:58:37 2022 read: IOPS=2020, BW=158MiB/s (166MB/s)(4735MiB/30001msec) slat (usec): min=6, max=219, avg=13.86, stdev= 4.33 clat (nsec): min=815, max=24320k, avg=65030.49, stdev=469057.39 lat (usec): min=35, max=24331, avg=79.04, stdev=469.10 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 88], 99.50th=[ 102], 99.90th=[ 1254], 99.95th=[15270], | 99.99th=[18220] bw ( KiB/s): min=123520, max=261600, per=100.00%, avg=161631.72, stdev=20040.48, samples=60 iops : min= 1544, max= 3270, avg=2020.38, stdev=250.50, samples=60 write: IOPS=507, BW=39.7MiB/s (41.6MB/s)(1190MiB/30001msec) slat (usec): min=12, max=219, avg=22.41, stdev= 6.31 clat (usec): min=376, max=46591, avg=1624.81, stdev=1686.35 lat (usec): min=402, max=46619, avg=1647.39, stdev=1686.41 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 685], 10.00th=[ 922], 20.00th=[ 1205], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1795], 90.00th=[ 2024], 95.00th=[ 2311], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[43254], | 99.99th=[45876] bw ( KiB/s): min=32320, max=67520, per=99.99%, avg=40614.60, stdev=4485.30, samples=60 iops : min= 404, max= 844, avg=507.67, stdev=56.06, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=45.30%, 100=34.15% lat (usec) : 250=0.30%, 500=0.41%, 750=0.94%, 1000=1.06% lat (msec) : 2=15.56%, 4=1.87%, 10=0.20%, 20=0.14%, 50=0.02% cpu : usr=1.57%, sys=6.13%, ctx=75834, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60614,15232,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4735MiB (4965MB), run=30001-30001msec WRITE: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=1190MiB (1248MB), run=30001-30001msec Disk stats (read/write): vdf: ios=60351/15162, merge=0/0, ticks=843/17004, in_queue=27911, util=93.15% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926121: Fri Jan 7 13:59:14 2022 read: IOPS=2035, BW=159MiB/s (167MB/s)(4771MiB/30001msec) clat (usec): min=35, max=62868, avg=75.03, stdev=536.35 lat (usec): min=35, max=62868, avg=75.17, stdev=536.35 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 99], 99.50th=[ 114], 99.90th=[ 1090], 99.95th=[14877], | 99.99th=[18220] bw ( KiB/s): min=122560, max=257280, per=100.00%, avg=162888.14, stdev=19543.31, samples=59 iops : min= 1532, max= 3216, avg=2036.10, stdev=244.29, samples=59 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1199MiB/30001msec) clat (usec): min=385, max=51536, avg=1639.89, stdev=1616.96 lat (usec): min=392, max=51544, avg=1648.02, stdev=1616.93 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 701], 10.00th=[ 996], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2040], 95.00th=[ 2311], | 99.00th=[ 8979], 99.50th=[ 9896], 99.90th=[17957], 99.95th=[41681], | 99.99th=[47449] bw ( KiB/s): min=31840, max=66720, per=100.00%, avg=40916.61, stdev=4532.97, samples=59 iops : min= 398, max= 834, avg=511.46, stdev=56.66, samples=59 lat (usec) : 50=14.30%, 100=64.88%, 250=0.62%, 500=0.35%, 750=0.86% lat (usec) : 1000=0.85% lat (msec) : 2=15.80%, 4=1.99%, 10=0.19%, 20=0.13%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.57%, sys=3.79%, ctx=76425, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61074,15345,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4771MiB (5003MB), run=30001-30001msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1199MiB (1257MB), run=30001-30001msec Disk stats (read/write): vdf: ios=60775/15275, merge=0/0, ticks=904/17196, in_queue=28144, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926791: Fri Jan 7 13:59:51 2022 read: IOPS=2064, BW=161MiB/s (169MB/s)(4844MiB/30042msec) slat (usec): min=4, max=492, avg=13.84, stdev= 6.29 clat (nsec): min=816, max=2531.0M, avg=1137088.64, stdev=45467310.54 lat (usec): min=33, max=2531.0k, avg=1151.10, stdev=45467.30 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 46], 40.00th=[ 48], | 50.00th=[ 51], 60.00th=[ 54], 70.00th=[ 58], | 80.00th=[ 63], 90.00th=[ 71], 95.00th=[ 79], | 99.00th=[ 112], 99.50th=[ 143], 99.90th=[ 906], | 99.95th=[ 608175], 99.99th=[2499806] bw ( KiB/s): min=138560, max=290400, per=100.00%, avg=165365.77, stdev=21262.44, samples=60 iops : min= 1732, max= 3630, avg=2067.03, stdev=265.79, samples=60 write: IOPS=518, BW=40.5MiB/s (42.5MB/s)(1216MiB/30042msec) slat (usec): min=9, max=517, avg=24.35, stdev=11.20 clat (usec): min=797, max=77374, avg=14675.64, stdev=6518.66 lat (usec): min=812, max=77394, avg=14700.18, stdev=6519.35 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 6783], 10.00th=[ 8979], 20.00th=[10683], | 30.00th=[11994], 40.00th=[13304], 50.00th=[14353], 60.00th=[15533], | 70.00th=[16581], 80.00th=[17433], 90.00th=[19268], 95.00th=[23462], | 99.00th=[34866], 99.50th=[62653], 99.90th=[72877], 99.95th=[76022], | 99.99th=[77071] bw ( KiB/s): min=36480, max=73600, per=100.00%, avg=41499.95, stdev=4854.23, samples=60 iops : min= 456, max= 920, avg=518.73, stdev=60.69, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.03%, 50=38.94% lat (usec) : 100=39.71%, 250=0.99%, 500=0.06%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.03%, 4=0.39%, 10=2.62%, 20=15.30%, 50=1.58% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.82%, sys=5.94%, ctx=76622, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62008,15569,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4844MiB (5080MB), run=30042-30042msec WRITE: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=1216MiB (1275MB), run=30042-30042msec Disk stats (read/write): vdf: ios=61829/15503, merge=0/0, ticks=66594/219307, in_queue=296871, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927605: Fri Jan 7 14:00:28 2022 read: IOPS=2037, BW=159MiB/s (167MB/s)(4775MiB/30001msec) clat (usec): min=33, max=23515, avg=71.76, stdev=457.50 lat (usec): min=33, max=23515, avg=71.90, stdev=457.51 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 99], 99.50th=[ 113], 99.90th=[ 791], 99.95th=[14222], | 99.99th=[18220] bw ( KiB/s): min=120160, max=244000, per=99.99%, avg=162957.12, stdev=19629.97, samples=59 iops : min= 1502, max= 3050, avg=2036.93, stdev=245.36, samples=59 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1200MiB/30001msec) clat (usec): min=394, max=45266, avg=1652.13, stdev=1614.93 lat (usec): min=401, max=45273, avg=1659.76, stdev=1614.83 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 717], 10.00th=[ 1029], 20.00th=[ 1270], | 30.00th=[ 1352], 40.00th=[ 1434], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1696], 80.00th=[ 1827], 90.00th=[ 2057], 95.00th=[ 2311], | 99.00th=[ 8717], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[41681], | 99.99th=[44827] bw ( KiB/s): min=32800, max=63680, per=99.99%, avg=40939.85, stdev=4056.15, samples=59 iops : min= 410, max= 796, avg=511.73, stdev=50.71, samples=59 lat (usec) : 50=21.57%, 100=57.63%, 250=0.62%, 500=0.32%, 750=0.83% lat (usec) : 1000=0.79% lat (msec) : 2=15.81%, 4=2.10%, 10=0.18%, 20=0.14%, 50=0.03% cpu : usr=1.49%, sys=3.76%, ctx=76483, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61120,15354,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4775MiB (5007MB), run=30001-30001msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1200MiB (1258MB), run=30001-30001msec Disk stats (read/write): vdf: ios=60824/15287, merge=0/0, ticks=820/17403, in_queue=28241, util=94.56% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928858: Fri Jan 7 14:01:05 2022 read: IOPS=2063, BW=161MiB/s (169MB/s)(4856MiB/30127msec) slat (usec): min=2, max=269, avg=11.16, stdev= 4.79 clat (nsec): min=767, max=2544.0M, avg=1278869.09, stdev=50179350.25 lat (usec): min=30, max=2544.0k, avg=1290.18, stdev=50179.35 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 53], | 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 79], | 99.00th=[ 363], 99.50th=[ 766], 99.90th=[ 1975], | 99.95th=[ 742392], 99.99th=[2499806] bw ( KiB/s): min=127615, max=322560, per=100.00%, avg=165782.70, stdev=26165.23, samples=60 iops : min= 1595, max= 4032, avg=2072.27, stdev=327.08, samples=60 write: IOPS=518, BW=40.5MiB/s (42.5MB/s)(1221MiB/30127msec) slat (usec): min=7, max=314, avg=23.92, stdev=11.40 clat (msec): min=4, max=2553, avg=91.24, stdev=82.38 lat (msec): min=4, max=2553, avg=91.27, stdev=82.38 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 65], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 79], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 86], | 70.00th=[ 88], 80.00th=[ 93], 90.00th=[ 104], 95.00th=[ 131], | 99.00th=[ 251], 99.50th=[ 575], 99.90th=[ 1368], 99.95th=[ 1687], | 99.99th=[ 2265] bw ( KiB/s): min=35110, max=75040, per=100.00%, avg=41538.67, stdev=5131.54, samples=60 iops : min= 438, max= 938, avg=519.20, stdev=64.17, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.04%, 50=52.26% lat (usec) : 100=25.35%, 250=1.15%, 500=0.42%, 750=0.20%, 1000=0.12% lat (msec) : 2=0.21%, 10=0.03%, 20=0.25%, 50=0.20%, 100=17.25% lat (msec) : 250=2.18%, 500=0.10%, 750=0.05%, 1000=0.02%, 2000=0.06% lat (msec) : >=2000=0.03% cpu : usr=1.74%, sys=5.29%, ctx=75415, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=62163,15623,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4856MiB (5092MB), run=30127-30127msec WRITE: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=1221MiB (1280MB), run=30127-30127msec Disk stats (read/write): vdf: ios=62161/15623, merge=0/0, ticks=75124/1414669, in_queue=1501432, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929903: Fri Jan 7 14:01:42 2022 read: IOPS=2033, BW=159MiB/s (167MB/s)(4767MiB/30001msec) clat (usec): min=32, max=26131, avg=73.55, stdev=424.42 lat (usec): min=32, max=26131, avg=73.69, stdev=424.42 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 76], 95.00th=[ 83], | 99.00th=[ 105], 99.50th=[ 123], 99.90th=[ 1205], 99.95th=[12780], | 99.99th=[17171] bw ( KiB/s): min=131840, max=261760, per=100.00%, avg=162716.93, stdev=20280.49, samples=60 iops : min= 1648, max= 3272, avg=2033.95, stdev=253.51, samples=60 write: IOPS=511, BW=39.9MiB/s (41.9MB/s)(1198MiB/30001msec) clat (usec): min=384, max=44860, avg=1647.00, stdev=1658.80 lat (usec): min=391, max=44868, avg=1655.39, stdev=1658.72 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 709], 10.00th=[ 979], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2024], 95.00th=[ 2311], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[36963], 99.95th=[42206], | 99.99th=[44303] bw ( KiB/s): min=32480, max=67680, per=100.00%, avg=40885.43, stdev=4679.67, samples=60 iops : min= 406, max= 846, avg=511.07, stdev=58.50, samples=60 lat (usec) : 50=12.51%, 100=66.40%, 250=0.88%, 500=0.37%, 750=0.87% lat (usec) : 1000=0.91% lat (msec) : 2=15.77%, 4=1.91%, 10=0.21%, 20=0.14%, 50=0.02% cpu : usr=1.60%, sys=3.90%, ctx=76353, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61015,15332,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4767MiB (4998MB), run=30001-30001msec WRITE: bw=39.9MiB/s (41.9MB/s), 39.9MiB/s-39.9MiB/s (41.9MB/s-41.9MB/s), io=1198MiB (1256MB), run=30001-30001msec Disk stats (read/write): vdf: ios=60701/15258, merge=0/0, ticks=754/17269, in_queue=28085, util=93.76% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930537: Fri Jan 7 14:02:19 2022 read: IOPS=2064, BW=161MiB/s (169MB/s)(4875MiB/30229msec) slat (usec): min=2, max=1368, avg=11.36, stdev= 7.89 clat (nsec): min=734, max=2562.0M, avg=1156239.69, stdev=44689975.22 lat (usec): min=28, max=2562.0k, avg=1167.75, stdev=44689.99 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 36], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 46], 60.00th=[ 51], 70.00th=[ 58], | 80.00th=[ 65], 90.00th=[ 79], 95.00th=[ 161], | 99.00th=[ 1450], 99.50th=[ 2540], 99.90th=[ 4555], | 99.95th=[ 759170], 99.99th=[2499806] bw ( KiB/s): min=124640, max=348160, per=100.00%, avg=166402.55, stdev=28847.82, samples=60 iops : min= 1558, max= 4352, avg=2080.00, stdev=360.61, samples=60 write: IOPS=518, BW=40.5MiB/s (42.5MB/s)(1225MiB/30229msec) slat (usec): min=8, max=219, avg=24.22, stdev=11.31 clat (msec): min=5, max=4600, avg=188.06, stdev=214.53 lat (msec): min=5, max=4600, avg=188.09, stdev=214.53 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 127], 10.00th=[ 133], 20.00th=[ 142], | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 174], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 220], 95.00th=[ 247], | 99.00th=[ 609], 99.50th=[ 1787], 99.90th=[ 3943], 99.95th=[ 3943], | 99.99th=[ 168] bw ( KiB/s): min=32865, max=72800, per=100.00%, avg=41562.43, stdev=5128.07, samples=60 iops : min= 410, max= 910, avg=519.50, stdev=64.14, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.07%, 50=46.83% lat (usec) : 100=27.57%, 250=2.08%, 500=1.08%, 750=0.61%, 1000=0.39% lat (msec) : 2=0.61%, 4=0.45%, 10=0.05%, 20=0.10%, 50=0.32% lat (msec) : 100=0.18%, 250=18.59%, 500=0.68%, 750=0.08%, 1000=0.04% lat (msec) : 2000=0.06%, >=2000=0.11% cpu : usr=1.84%, sys=5.26%, ctx=72539, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62395,15682,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4875MiB (5111MB), run=30229-30229msec WRITE: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=1225MiB (1285MB), run=30229-30229msec Disk stats (read/write): vdf: ios=62393/15633, merge=0/0, ticks=66110/2918747, in_queue=3006413, util=99.74% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931404: Fri Jan 7 14:02:56 2022 read: IOPS=2039, BW=159MiB/s (167MB/s)(4781MiB/30002msec) clat (usec): min=32, max=19612, avg=69.73, stdev=426.38 lat (usec): min=33, max=19612, avg=69.87, stdev=426.39 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 95], 99.50th=[ 108], 99.90th=[ 668], 99.95th=[13304], | 99.99th=[16450] bw ( KiB/s): min=130080, max=261440, per=100.00%, avg=163192.00, stdev=19620.16, samples=60 iops : min= 1626, max= 3268, avg=2039.90, stdev=245.25, samples=60 write: IOPS=512, BW=40.0MiB/s (41.0MB/s)(1201MiB/30002msec) clat (usec): min=398, max=46662, avg=1658.55, stdev=1659.42 lat (usec): min=402, max=46668, avg=1666.01, stdev=1659.36 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 734], 10.00th=[ 1057], 20.00th=[ 1287], | 30.00th=[ 1369], 40.00th=[ 1434], 50.00th=[ 1516], 60.00th=[ 1598], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2040], 95.00th=[ 2311], | 99.00th=[ 8586], 99.50th=[10028], 99.90th=[39060], 99.95th=[43779], | 99.99th=[46400] bw ( KiB/s): min=33920, max=67520, per=100.00%, avg=40986.67, stdev=4449.11, samples=60 iops : min= 424, max= 844, avg=512.33, stdev=55.61, samples=60 lat (usec) : 50=27.06%, 100=52.29%, 250=0.46%, 500=0.36%, 750=0.74% lat (usec) : 1000=0.72% lat (msec) : 2=16.00%, 4=2.03%, 10=0.16%, 20=0.15%, 50=0.02% cpu : usr=1.49%, sys=3.61%, ctx=76578, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61197,15371,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4781MiB (5013MB), run=30002-30002msec WRITE: bw=40.0MiB/s (41.0MB/s), 40.0MiB/s-40.0MiB/s (41.0MB/s-41.0MB/s), io=1201MiB (1259MB), run=30002-30002msec Disk stats (read/write): vdf: ios=60871/15296, merge=0/0, ticks=767/17473, in_queue=28209, util=94.14% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932341: Fri Jan 7 14:03:34 2022 read: IOPS=2062, BW=161MiB/s (169MB/s)(4874MiB/30247msec) slat (usec): min=2, max=209986, avg=371.57, stdev=2139.96 clat (usec): min=1085, max=2593.5k, avg=31307.37, stdev=51016.88 lat (usec): min=1089, max=2593.5k, avg=31679.08, stdev=51075.34 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 5538], 10.00th=[ 16712], | 20.00th=[ 21365], 30.00th=[ 23987], 40.00th=[ 26346], | 50.00th=[ 28443], 60.00th=[ 30802], 70.00th=[ 33162], | 80.00th=[ 36439], 90.00th=[ 41681], 95.00th=[ 47449], | 99.00th=[ 92799], 99.50th=[ 219153], 99.90th=[ 291505], | 99.95th=[ 826278], 99.99th=[2533360] bw ( KiB/s): min=132320, max=340960, per=100.00%, avg=166230.23, stdev=26770.95, samples=60 iops : min= 1654, max= 4262, avg=2077.90, stdev=334.64, samples=60 write: IOPS=518, BW=40.5MiB/s (42.5MB/s)(1225MiB/30247msec) slat (usec): min=7, max=257165, avg=424.92, stdev=3198.57 clat (msec): min=7, max=3903, avg=259.28, stdev=201.97 lat (msec): min=7, max=3904, avg=259.71, stdev=201.95 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 174], 10.00th=[ 194], 20.00th=[ 211], | 30.00th=[ 222], 40.00th=[ 232], 50.00th=[ 241], 60.00th=[ 249], | 70.00th=[ 257], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 334], | 99.00th=[ 961], 99.50th=[ 1536], 99.90th=[ 3641], 99.95th=[ 3641], | 99.99th=[ 3809] bw ( KiB/s): min=35360, max=68000, per=99.93%, avg=41440.20, stdev=4696.34, samples=60 iops : min= 442, max= 850, avg=518.00, stdev=58.70, samples=60 lat (msec) : 2=1.02%, 4=2.10%, 10=1.77%, 20=7.91%, 50=64.31% lat (msec) : 100=2.65%, 250=12.50%, 500=7.32%, 750=0.11%, 1000=0.07% lat (msec) : 2000=0.14%, >=2000=0.09% cpu : usr=1.83%, sys=1.93%, ctx=8763, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62393,15679,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4874MiB (5111MB), run=30247-30247msec WRITE: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=1225MiB (1284MB), run=30247-30247msec Disk stats (read/write): vdf: ios=62393/15611, merge=0/0, ticks=78189/3566308, in_queue=3679968, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932835: Fri Jan 7 14:04:10 2022 read: IOPS=1780, BW=223MiB/s (233MB/s)(6679MiB/30004msec) clat (usec): min=49, max=68446, avg=99.58, stdev=594.63 lat (usec): min=49, max=68447, avg=99.73, stdev=594.63 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 94], 90.00th=[ 103], 95.00th=[ 113], | 99.00th=[ 141], 99.50th=[ 161], 99.90th=[10028], 99.95th=[16450], | 99.99th=[19268] bw ( KiB/s): min=134400, max=366080, per=100.00%, avg=227992.15, stdev=31238.81, samples=60 iops : min= 1050, max= 2860, avg=1781.18, stdev=244.06, samples=60 write: IOPS=447, BW=55.9MiB/s (58.6MB/s)(1678MiB/30004msec) clat (usec): min=526, max=241160, avg=1817.50, stdev=2663.79 lat (usec): min=538, max=241172, avg=1830.12, stdev=2663.72 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 914], 10.00th=[ 1090], 20.00th=[ 1254], | 30.00th=[ 1401], 40.00th=[ 1516], 50.00th=[ 1631], 60.00th=[ 1729], | 70.00th=[ 1844], 80.00th=[ 1975], 90.00th=[ 2212], 95.00th=[ 2507], | 99.00th=[ 9634], 99.50th=[10290], 99.90th=[33817], 99.95th=[40633], | 99.99th=[45351] bw ( KiB/s): min=34816, max=94464, per=100.00%, avg=57258.28, stdev=7359.77, samples=60 iops : min= 272, max= 738, avg=447.30, stdev=57.49, samples=60 lat (usec) : 50=0.01%, 100=70.09%, 250=9.66%, 500=0.06%, 750=0.51% lat (usec) : 1000=0.90% lat (msec) : 2=15.04%, 4=3.17%, 10=0.35%, 20=0.19%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.77%, sys=3.58%, ctx=66857, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53432,13420,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=223MiB/s (233MB/s), 223MiB/s-223MiB/s (233MB/s-233MB/s), io=6679MiB (7003MB), run=30004-30004msec WRITE: bw=55.9MiB/s (58.6MB/s), 55.9MiB/s-55.9MiB/s (58.6MB/s-58.6MB/s), io=1678MiB (1759MB), run=30004-30004msec Disk stats (read/write): vdf: ios=53240/13361, merge=0/0, ticks=923/17102, in_queue=28151, util=93.92% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933821: Fri Jan 7 14:04:47 2022 read: IOPS=1808, BW=226MiB/s (237MB/s)(6784MiB/30001msec) slat (usec): min=8, max=107, avg=15.49, stdev= 4.50 clat (nsec): min=1034, max=31386k, avg=74074.09, stdev=509753.04 lat (usec): min=45, max=31404, avg=89.72, stdev=509.85 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 95], 99.50th=[ 108], 99.90th=[11207], 99.95th=[14615], | 99.99th=[18482] bw ( KiB/s): min=181248, max=375808, per=100.00%, avg=231551.32, stdev=28406.95, samples=60 iops : min= 1416, max= 2936, avg=1808.97, stdev=221.96, samples=60 write: IOPS=454, BW=56.8MiB/s (59.6MB/s)(1705MiB/30001msec) slat (usec): min=16, max=237, avg=27.68, stdev= 6.10 clat (usec): min=488, max=47899, avg=1807.04, stdev=1665.44 lat (usec): min=524, max=47937, avg=1834.91, stdev=1665.53 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 783], 10.00th=[ 963], 20.00th=[ 1336], | 30.00th=[ 1467], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2245], 95.00th=[ 2573], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[36963], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=49664, max=97792, per=100.00%, avg=58192.90, stdev=6523.12, samples=60 iops : min= 388, max= 764, avg=454.62, stdev=50.98, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=21.42%, 100=57.93% lat (usec) : 250=0.45%, 500=0.01%, 750=0.81%, 1000=1.35% lat (msec) : 2=13.81%, 4=3.77%, 10=0.23%, 20=0.20%, 50=0.02% cpu : usr=1.47%, sys=5.85%, ctx=67924, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54271,13639,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=6784MiB (7113MB), run=30001-30001msec WRITE: bw=56.8MiB/s (59.6MB/s), 56.8MiB/s-56.8MiB/s (59.6MB/s-59.6MB/s), io=1705MiB (1788MB), run=30001-30001msec Disk stats (read/write): vdf: ios=54076/13590, merge=0/0, ticks=864/17000, in_queue=28132, util=93.88% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934413: Fri Jan 7 14:05:24 2022 read: IOPS=1808, BW=226MiB/s (237MB/s)(6784MiB/30002msec) clat (usec): min=37, max=27157, avg=80.82, stdev=512.88 lat (usec): min=37, max=27157, avg=80.96, stdev=512.88 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 80], 95.00th=[ 86], | 99.00th=[ 109], 99.50th=[ 125], 99.90th=[10945], 99.95th=[15401], | 99.99th=[19268] bw ( KiB/s): min=114688, max=395520, per=100.00%, avg=231563.33, stdev=33992.22, samples=60 iops : min= 896, max= 3090, avg=1809.08, stdev=265.57, samples=60 write: IOPS=454, BW=56.8MiB/s (59.6MB/s)(1705MiB/30002msec) clat (usec): min=504, max=135600, avg=1857.36, stdev=2043.24 lat (usec): min=516, max=135612, avg=1869.50, stdev=2043.14 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 807], 10.00th=[ 1090], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1696], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2040], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9765], 99.50th=[10290], 99.90th=[40109], 99.95th=[42206], | 99.99th=[46400] bw ( KiB/s): min=30976, max=102912, per=100.00%, avg=58194.88, stdev=7867.46, samples=60 iops : min= 242, max= 804, avg=454.63, stdev=61.47, samples=60 lat (usec) : 50=2.61%, 100=76.02%, 250=1.15%, 500=0.03%, 750=0.78% lat (usec) : 1000=0.91% lat (msec) : 2=13.92%, 4=4.15%, 10=0.20%, 20=0.21%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.67%, sys=3.37%, ctx=67919, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54271,13640,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=6784MiB (7113MB), run=30002-30002msec WRITE: bw=56.8MiB/s (59.6MB/s), 56.8MiB/s-56.8MiB/s (59.6MB/s-59.6MB/s), io=1705MiB (1788MB), run=30002-30002msec Disk stats (read/write): vdf: ios=54063/13581, merge=0/0, ticks=881/17456, in_queue=28342, util=94.60% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935646: Fri Jan 7 14:06:01 2022 read: IOPS=1849, BW=231MiB/s (242MB/s)(6944MiB/30034msec) slat (usec): min=4, max=176, avg=13.85, stdev= 4.41 clat (nsec): min=834, max=2494.2M, avg=1452508.94, stdev=53105436.81 lat (usec): min=36, max=2494.2k, avg=1466.51, stdev=53105.44 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 44], 40.00th=[ 45], | 50.00th=[ 48], 60.00th=[ 51], 70.00th=[ 56], | 80.00th=[ 62], 90.00th=[ 72], 95.00th=[ 80], | 99.00th=[ 104], 99.50th=[ 133], 99.90th=[ 89654], | 99.95th=[1350566], 99.99th=[2499806] bw ( KiB/s): min=196864, max=462592, per=100.00%, avg=237036.85, stdev=36296.40, samples=60 iops : min= 1538, max= 3614, avg=1851.83, stdev=283.57, samples=60 write: IOPS=464, BW=58.1MiB/s (60.9MB/s)(1745MiB/30034msec) slat (usec): min=12, max=160, avg=29.08, stdev= 9.39 clat (usec): min=1963, max=77892, avg=15637.41, stdev=6615.58 lat (usec): min=1978, max=77916, avg=15666.66, stdev=6616.23 clat percentiles (usec): | 1.00th=[ 3261], 5.00th=[ 8717], 10.00th=[10159], 20.00th=[11600], | 30.00th=[12649], 40.00th=[13960], 50.00th=[15008], 60.00th=[16188], | 70.00th=[17171], 80.00th=[18482], 90.00th=[20579], 95.00th=[24249], | 99.00th=[36963], 99.50th=[65274], 99.90th=[73925], 99.95th=[77071], | 99.99th=[78119] bw ( KiB/s): min=52480, max=116224, per=100.00%, avg=59536.88, stdev=8072.87, samples=60 iops : min= 410, max= 908, avg=465.10, stdev=63.06, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=46.13% lat (usec) : 100=32.79%, 250=0.75%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.35%, 10=1.52%, 20=15.83%, 50=2.23% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.78%, sys=5.34%, ctx=68941, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55554,13961,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6944MiB (7282MB), run=30034-30034msec WRITE: bw=58.1MiB/s (60.9MB/s), 58.1MiB/s-58.1MiB/s (60.9MB/s-60.9MB/s), io=1745MiB (1830MB), run=30034-30034msec Disk stats (read/write): vdf: ios=55293/13896, merge=0/0, ticks=74586/210165, in_queue=297064, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936741: Fri Jan 7 14:06:38 2022 read: IOPS=1800, BW=225MiB/s (236MB/s)(6751MiB/30003msec) clat (usec): min=38, max=22194, avg=83.21, stdev=518.36 lat (usec): min=38, max=22194, avg=83.36, stdev=518.36 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 82], 95.00th=[ 89], | 99.00th=[ 110], 99.50th=[ 123], 99.90th=[11731], 99.95th=[15795], | 99.99th=[18744] bw ( KiB/s): min=114432, max=369920, per=100.00%, avg=230428.83, stdev=31968.00, samples=60 iops : min= 894, max= 2890, avg=1800.22, stdev=249.75, samples=60 write: IOPS=452, BW=56.5MiB/s (59.3MB/s)(1696MiB/30003msec) clat (usec): min=523, max=203887, avg=1858.86, stdev=2451.37 lat (usec): min=535, max=203898, avg=1871.19, stdev=2451.32 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 807], 10.00th=[ 1090], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1680], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2040], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[39060], 99.95th=[44303], | 99.99th=[47449] bw ( KiB/s): min=32256, max=95744, per=100.00%, avg=57900.60, stdev=7165.23, samples=60 iops : min= 252, max= 748, avg=452.33, stdev=55.98, samples=60 lat (usec) : 50=1.54%, 100=76.94%, 250=1.31%, 500=0.02%, 750=0.76% lat (usec) : 1000=0.92% lat (msec) : 2=13.95%, 4=4.12%, 10=0.21%, 20=0.20%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.59%, sys=3.64%, ctx=67591, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54006,13570,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6751MiB (7079MB), run=30003-30003msec WRITE: bw=56.5MiB/s (59.3MB/s), 56.5MiB/s-56.5MiB/s (59.3MB/s-59.3MB/s), io=1696MiB (1779MB), run=30003-30003msec Disk stats (read/write): vdf: ios=53888/13528, merge=0/0, ticks=858/17494, in_queue=28208, util=94.09% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937332: Fri Jan 7 14:07:16 2022 read: IOPS=1860, BW=233MiB/s (244MB/s)(7011MiB/30142msec) slat (usec): min=3, max=200, avg=12.66, stdev= 4.76 clat (nsec): min=700, max=2530.4M, avg=1304350.16, stdev=48391244.27 lat (usec): min=35, max=2530.4k, avg=1317.16, stdev=48391.25 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 55], 70.00th=[ 63], | 80.00th=[ 75], 90.00th=[ 86], 95.00th=[ 100], | 99.00th=[ 537], 99.50th=[ 1123], 99.90th=[ 139461], | 99.95th=[ 910164], 99.99th=[2533360] bw ( KiB/s): min=179712, max=488192, per=100.00%, avg=239321.07, stdev=38578.80, samples=60 iops : min= 1404, max= 3814, avg=1869.67, stdev=301.39, samples=60 write: IOPS=467, BW=58.5MiB/s (61.3MB/s)(1762MiB/30142msec) slat (usec): min=13, max=308, avg=32.91, stdev=13.85 clat (msec): min=2, max=2574, avg=101.62, stdev=75.12 lat (msec): min=2, max=2574, avg=101.65, stdev=75.12 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 87], | 30.00th=[ 90], 40.00th=[ 92], 50.00th=[ 94], 60.00th=[ 97], | 70.00th=[ 101], 80.00th=[ 105], 90.00th=[ 114], 95.00th=[ 138], | 99.00th=[ 253], 99.50th=[ 609], 99.90th=[ 1217], 99.95th=[ 1418], | 99.99th=[ 2123] bw ( KiB/s): min=47872, max=113408, per=100.00%, avg=59950.55, stdev=7861.29, samples=60 iops : min= 374, max= 886, avg=468.33, stdev=61.42, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=40.12% lat (usec) : 100=35.77%, 250=2.62%, 500=0.51%, 750=0.29%, 1000=0.11% lat (msec) : 2=0.27%, 4=0.08%, 10=0.02%, 20=0.31%, 50=0.20% lat (msec) : 100=13.60%, 250=5.78%, 500=0.10%, 750=0.05%, 1000=0.05% lat (msec) : 2000=0.04%, >=2000=0.03% cpu : usr=2.29%, sys=4.92%, ctx=67826, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=56090,14096,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=233MiB/s (244MB/s), 233MiB/s-233MiB/s (244MB/s-244MB/s), io=7011MiB (7352MB), run=30142-30142msec WRITE: bw=58.5MiB/s (61.3MB/s), 58.5MiB/s-58.5MiB/s (61.3MB/s-61.3MB/s), io=1762MiB (1848MB), run=30142-30142msec Disk stats (read/write): vdf: ios=56087/14096, merge=0/0, ticks=66900/1423543, in_queue=1503028, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938184: Fri Jan 7 14:07:52 2022 read: IOPS=1808, BW=226MiB/s (237MB/s)(6782MiB/30002msec) clat (usec): min=37, max=25668, avg=78.85, stdev=518.56 lat (usec): min=37, max=25668, avg=78.99, stdev=518.56 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 98], 99.50th=[ 109], 99.90th=[11207], 99.95th=[15926], | 99.99th=[19530] bw ( KiB/s): min=165120, max=379392, per=100.00%, avg=231486.95, stdev=29582.00, samples=60 iops : min= 1290, max= 2964, avg=1808.48, stdev=231.11, samples=60 write: IOPS=454, BW=56.8MiB/s (59.6MB/s)(1704MiB/30002msec) clat (usec): min=516, max=89732, avg=1866.57, stdev=1952.45 lat (usec): min=526, max=89744, avg=1878.59, stdev=1952.34 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 824], 10.00th=[ 1172], 20.00th=[ 1434], | 30.00th=[ 1532], 40.00th=[ 1614], 50.00th=[ 1696], 60.00th=[ 1795], | 70.00th=[ 1909], 80.00th=[ 2057], 90.00th=[ 2278], 95.00th=[ 2507], | 99.00th=[ 9634], 99.50th=[10290], 99.90th=[39060], 99.95th=[43254], | 99.99th=[81265] bw ( KiB/s): min=40960, max=99072, per=100.00%, avg=58169.43, stdev=7053.17, samples=60 iops : min= 320, max= 774, avg=454.43, stdev=55.10, samples=60 lat (usec) : 50=3.68%, 100=75.58%, 250=0.55%, 500=0.01%, 750=0.75% lat (usec) : 1000=0.71% lat (msec) : 2=14.00%, 4=4.31%, 10=0.17%, 20=0.21%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.53%, sys=3.38%, ctx=67894, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54253,13634,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=6782MiB (7111MB), run=30002-30002msec WRITE: bw=56.8MiB/s (59.6MB/s), 56.8MiB/s-56.8MiB/s (59.6MB/s-59.6MB/s), io=1704MiB (1787MB), run=30002-30002msec Disk stats (read/write): vdf: ios=54024/13574, merge=0/0, ticks=878/17628, in_queue=28333, util=94.55% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938711: Fri Jan 7 14:08:30 2022 read: IOPS=1855, BW=232MiB/s (243MB/s)(7012MiB/30236msec) slat (usec): min=3, max=322, avg=11.94, stdev= 6.06 clat (nsec): min=786, max=2525.5M, avg=1427315.78, stdev=50918475.79 lat (usec): min=35, max=2525.5k, avg=1439.41, stdev=50918.46 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 51], 60.00th=[ 56], 70.00th=[ 68], | 80.00th=[ 78], 90.00th=[ 95], 95.00th=[ 212], | 99.00th=[ 2507], 99.50th=[ 3589], 99.90th=[ 116917], | 99.95th=[1002439], 99.99th=[2499806] bw ( KiB/s): min=193024, max=553728, per=100.00%, avg=239360.73, stdev=46440.55, samples=60 iops : min= 1508, max= 4326, avg=1869.97, stdev=362.82, samples=60 write: IOPS=466, BW=58.3MiB/s (61.1MB/s)(1762MiB/30236msec) slat (usec): min=14, max=1251, avg=33.04, stdev=18.14 clat (msec): min=4, max=2534, avg=208.69, stdev=98.31 lat (msec): min=4, max=2534, avg=208.72, stdev=98.31 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 136], 10.00th=[ 165], 20.00th=[ 184], | 30.00th=[ 190], 40.00th=[ 194], 50.00th=[ 201], 60.00th=[ 205], | 70.00th=[ 209], 80.00th=[ 220], 90.00th=[ 249], 95.00th=[ 271], | 99.00th=[ 567], 99.50th=[ 877], 99.90th=[ 1569], 99.95th=[ 1569], | 99.99th=[ 2299] bw ( KiB/s): min=47616, max=115712, per=100.00%, avg=59743.18, stdev=8328.41, samples=60 iops : min= 372, max= 904, avg=466.68, stdev=65.08, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=38.80% lat (usec) : 100=34.05%, 250=3.25%, 500=1.00%, 750=0.56%, 1000=0.48% lat (msec) : 2=0.79%, 4=0.64%, 10=0.18%, 20=0.05%, 50=0.38% lat (msec) : 100=0.16%, 250=17.60%, 500=1.66%, 750=0.12%, 1000=0.10% lat (msec) : 2000=0.07%, >=2000=0.03% cpu : usr=2.16%, sys=4.75%, ctx=65188, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56094,14097,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=7012MiB (7352MB), run=30236-30236msec WRITE: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=1762MiB (1848MB), run=30236-30236msec Disk stats (read/write): vdf: ios=56091/14054, merge=0/0, ticks=72264/2911761, in_queue=3006840, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939603: Fri Jan 7 14:09:06 2022 read: IOPS=1820, BW=228MiB/s (239MB/s)(6828MiB/30001msec) clat (usec): min=38, max=25627, avg=81.06, stdev=535.57 lat (usec): min=39, max=25627, avg=81.21, stdev=535.57 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 79], 95.00th=[ 86], | 99.00th=[ 106], 99.50th=[ 121], 99.90th=[11731], 99.95th=[15795], | 99.99th=[19792] bw ( KiB/s): min=184832, max=374784, per=100.00%, avg=233064.60, stdev=29242.49, samples=60 iops : min= 1444, max= 2928, avg=1820.82, stdev=228.46, samples=60 write: IOPS=457, BW=57.2MiB/s (59.0MB/s)(1716MiB/30001msec) clat (usec): min=520, max=46641, avg=1842.28, stdev=1682.45 lat (usec): min=533, max=46654, avg=1854.48, stdev=1682.39 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 816], 10.00th=[ 1139], 20.00th=[ 1434], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1696], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2024], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[37487], 99.95th=[43254], | 99.99th=[45351] bw ( KiB/s): min=48640, max=97280, per=100.00%, avg=58561.77, stdev=6400.38, samples=60 iops : min= 380, max= 760, avg=457.50, stdev=50.02, samples=60 lat (usec) : 50=2.90%, 100=75.84%, 250=1.05%, 500=0.03%, 750=0.74% lat (usec) : 1000=0.82% lat (msec) : 2=14.13%, 4=4.09%, 10=0.19%, 20=0.19%, 50=0.03% cpu : usr=1.67%, sys=3.38%, ctx=68357, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54623,13726,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6828MiB (7160MB), run=30001-30001msec WRITE: bw=57.2MiB/s (59.0MB/s), 57.2MiB/s-57.2MiB/s (59.0MB/s-59.0MB/s), io=1716MiB (1799MB), run=30001-30001msec Disk stats (read/write): vdf: ios=54357/13661, merge=0/0, ticks=926/17363, in_queue=28261, util=94.30% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940608: Fri Jan 7 14:09:44 2022 read: IOPS=1856, BW=232MiB/s (243MB/s)(7030MiB/30298msec) slat (usec): min=3, max=254128, avg=420.28, stdev=2705.32 clat (usec): min=1306, max=2605.4k, avg=34941.82, stdev=58061.82 lat (usec): min=1309, max=2605.5k, avg=35362.25, stdev=58152.87 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 28], 40.00th=[ 30], 50.00th=[ 32], 60.00th=[ 35], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 53], | 99.00th=[ 104], 99.50th=[ 245], 99.90th=[ 347], 99.95th=[ 1368], | 99.99th=[ 2534] bw ( KiB/s): min=184320, max=546560, per=100.00%, avg=239672.42, stdev=46267.78, samples=60 iops : min= 1440, max= 4270, avg=1872.42, stdev=361.47, samples=60 write: IOPS=466, BW=58.3MiB/s (61.1MB/s)(1767MiB/30298msec) slat (usec): min=14, max=185629, avg=440.07, stdev=2300.66 clat (msec): min=10, max=2162, avg=287.50, stdev=147.02 lat (msec): min=10, max=2162, avg=287.94, stdev=147.00 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 184], 10.00th=[ 209], 20.00th=[ 243], | 30.00th=[ 262], 40.00th=[ 271], 50.00th=[ 275], 60.00th=[ 284], | 70.00th=[ 292], 80.00th=[ 305], 90.00th=[ 338], 95.00th=[ 380], | 99.00th=[ 936], 99.50th=[ 1334], 99.90th=[ 2056], 99.95th=[ 2056], | 99.99th=[ 2056] bw ( KiB/s): min=48640, max=109312, per=100.00%, avg=59730.62, stdev=7574.80, samples=60 iops : min= 380, max= 854, avg=466.60, stdev=59.17, samples=60 lat (msec) : 2=0.29%, 4=1.77%, 10=3.22%, 20=4.76%, 50=65.11% lat (msec) : 100=4.38%, 250=4.44%, 500=15.48%, 750=0.23%, 1000=0.12% lat (msec) : 2000=0.13%, >=2000=0.07% cpu : usr=2.37%, sys=1.94%, ctx=7831, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56243,14133,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=7030MiB (7372MB), run=30298-30298msec WRITE: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=1767MiB (1852MB), run=30298-30298msec Disk stats (read/write): vdf: ios=56238/14066, merge=0/0, ticks=83908/3550582, in_queue=3682412, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941087: Fri Jan 7 14:10:20 2022 read: IOPS=1179, BW=295MiB/s (309MB/s)(8847MiB/30001msec) clat (usec): min=73, max=21301, avg=126.52, stdev=426.79 lat (usec): min=73, max=21302, avg=126.68, stdev=426.79 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 87], 10.00th=[ 89], 20.00th=[ 92], | 30.00th=[ 96], 40.00th=[ 101], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 119], 80.00th=[ 127], 90.00th=[ 141], 95.00th=[ 153], | 99.00th=[ 196], 99.50th=[ 229], 99.90th=[ 8848], 99.95th=[11731], | 99.99th=[19268] bw ( KiB/s): min=225792, max=539136, per=100.00%, avg=301975.86, stdev=49662.07, samples=59 iops : min= 882, max= 2106, avg=1179.59, stdev=193.99, samples=59 write: IOPS=296, BW=74.1MiB/s (77.7MB/s)(2225MiB/30001msec) clat (usec): min=894, max=63459, avg=2832.14, stdev=2671.24 lat (usec): min=915, max=63486, avg=2858.85, stdev=2670.79 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1106], 10.00th=[ 1319], 20.00th=[ 1647], | 30.00th=[ 1893], 40.00th=[ 2057], 50.00th=[ 2212], 60.00th=[ 2376], | 70.00th=[ 2540], 80.00th=[ 2933], 90.00th=[ 4817], 95.00th=[ 7242], | 99.00th=[11994], 99.50th=[13173], 99.90th=[45876], 99.95th=[54789], | 99.99th=[63701] bw ( KiB/s): min=60416, max=142336, per=99.95%, avg=75888.81, stdev=12136.66, samples=59 iops : min= 236, max= 556, avg=296.44, stdev=47.41, samples=59 lat (usec) : 100=31.09%, 250=48.48%, 500=0.14%, 750=0.05%, 1000=0.42% lat (msec) : 2=6.99%, 4=9.59%, 10=2.48%, 20=0.72%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.60%, sys=2.72%, ctx=44294, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=35387,8898,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=8847MiB (9276MB), run=30001-30001msec WRITE: bw=74.1MiB/s (77.7MB/s), 74.1MiB/s-74.1MiB/s (77.7MB/s-77.7MB/s), io=2225MiB (2333MB), run=30001-30001msec Disk stats (read/write): vdf: ios=35262/8862, merge=0/0, ticks=509/20588, in_queue=28547, util=95.23% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942437: Fri Jan 7 14:10:57 2022 read: IOPS=1309, BW=327MiB/s (343MB/s)(9821MiB/30006msec) slat (nsec): min=12283, max=90985, avg=20030.35, stdev=5120.40 clat (nsec): min=1687, max=20773k, avg=93683.88, stdev=430500.20 lat (usec): min=65, max=20789, avg=113.87, stdev=430.56 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 67], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 98], 95.00th=[ 106], | 99.00th=[ 129], 99.50th=[ 145], 99.90th=[ 8455], 99.95th=[11731], | 99.99th=[17957] bw ( KiB/s): min=237568, max=553472, per=100.00%, avg=335216.75, stdev=44602.46, samples=60 iops : min= 928, max= 2162, avg=1309.43, stdev=174.23, samples=60 write: IOPS=330, BW=82.6MiB/s (86.6MB/s)(2478MiB/30006msec) slat (usec): min=24, max=175, avg=43.96, stdev= 9.71 clat (usec): min=858, max=62136, avg=2523.85, stdev=2369.64 lat (usec): min=898, max=62167, avg=2567.99, stdev=2369.51 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1614], | 30.00th=[ 1893], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2671], 90.00th=[ 3589], 95.00th=[ 4752], | 99.00th=[10683], 99.50th=[11731], 99.90th=[52691], 99.95th=[54264], | 99.99th=[62129] bw ( KiB/s): min=68608, max=146944, per=100.00%, avg=84576.72, stdev=10313.13, samples=60 iops : min= 268, max= 574, avg=330.37, stdev=40.28, samples=60 lat (usec) : 2=0.01%, 50=0.04%, 100=73.45%, 250=6.22%, 500=0.02% lat (usec) : 750=0.01%, 1000=0.57% lat (msec) : 2=7.06%, 4=10.65%, 10=1.47%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.56%, sys=4.84%, ctx=49205, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39282,9912,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=9821MiB (10.3GB), run=30006-30006msec WRITE: bw=82.6MiB/s (86.6MB/s), 82.6MiB/s-82.6MiB/s (86.6MB/s-86.6MB/s), io=2478MiB (2598MB), run=30006-30006msec Disk stats (read/write): vdf: ios=39143/9873, merge=0/0, ticks=585/20344, in_queue=28375, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943440: Fri Jan 7 14:11:34 2022 read: IOPS=1358, BW=340MiB/s (356MB/s)(9.95GiB/30002msec) clat (usec): min=52, max=28848, avg=96.21, stdev=479.78 lat (usec): min=52, max=28849, avg=96.35, stdev=479.78 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 125], 99.50th=[ 143], 99.90th=[10028], 99.95th=[12911], | 99.99th=[17171] bw ( KiB/s): min=268800, max=576512, per=100.00%, avg=347738.87, stdev=45290.47, samples=60 iops : min= 1050, max= 2252, avg=1358.30, stdev=176.91, samples=60 write: IOPS=342, BW=85.7MiB/s (89.8MB/s)(2571MiB/30002msec) clat (usec): min=826, max=112851, avg=2503.93, stdev=2626.51 lat (usec): min=849, max=112863, avg=2528.11, stdev=2626.31 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1074], 10.00th=[ 1237], 20.00th=[ 1827], | 30.00th=[ 1975], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2474], 80.00th=[ 2671], 90.00th=[ 3064], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[11076], 99.90th=[53216], 99.95th=[61604], | 99.99th=[76022] bw ( KiB/s): min=58368, max=152576, per=100.00%, avg=87745.57, stdev=10438.21, samples=60 iops : min= 228, max= 596, avg=342.70, stdev=40.78, samples=60 lat (usec) : 100=74.47%, 250=5.21%, 500=0.02%, 750=0.01%, 1000=0.60% lat (msec) : 2=5.81%, 4=12.50%, 10=0.94%, 20=0.40%, 50=0.01% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.65%, sys=2.99%, ctx=51034, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40744,10282,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=340MiB/s (356MB/s), 340MiB/s-340MiB/s (356MB/s-356MB/s), io=9.95GiB (10.7GB), run=30002-30002msec WRITE: bw=85.7MiB/s (89.8MB/s), 85.7MiB/s-85.7MiB/s (89.8MB/s-89.8MB/s), io=2571MiB (2695MB), run=30002-30002msec Disk stats (read/write): vdf: ios=40606/10246, merge=0/0, ticks=675/20652, in_queue=28412, util=94.77% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943990: Fri Jan 7 14:12:11 2022 read: IOPS=1387, BW=347MiB/s (364MB/s)(10.2GiB/30067msec) slat (usec): min=7, max=1442, avg=17.54, stdev= 8.93 clat (nsec): min=853, max=2512.2M, avg=452336.66, stdev=24525594.58 lat (usec): min=49, max=2512.2k, avg=470.03, stdev=24525.60 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 55], 40.00th=[ 58], | 50.00th=[ 63], 60.00th=[ 68], 70.00th=[ 76], | 80.00th=[ 91], 90.00th=[ 113], 95.00th=[ 130], | 99.00th=[ 190], 99.50th=[ 253], 99.90th=[ 34866], | 99.95th=[ 54264], 99.99th=[1669333] bw ( KiB/s): min=225792, max=712192, per=100.00%, avg=355848.53, stdev=60003.75, samples=60 iops : min= 882, max= 2782, avg=1390.03, stdev=234.39, samples=60 write: IOPS=350, BW=87.6MiB/s (91.9MB/s)(2635MiB/30067msec) slat (usec): min=23, max=1291, avg=50.10, stdev=22.61 clat (msec): min=2, max=149, avg=26.60, stdev=11.77 lat (msec): min=2, max=149, avg=26.65, stdev=11.77 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 12], 20.00th=[ 18], | 30.00th=[ 23], 40.00th=[ 25], 50.00th=[ 27], 60.00th=[ 29], | 70.00th=[ 31], 80.00th=[ 34], 90.00th=[ 39], 95.00th=[ 43], | 99.00th=[ 72], 99.50th=[ 84], 99.90th=[ 107], 99.95th=[ 142], | 99.99th=[ 148] bw ( KiB/s): min=56320, max=179712, per=100.00%, avg=89941.33, stdev=13945.33, samples=60 iops : min= 220, max= 702, avg=351.33, stdev=54.47, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=6.04% lat (usec) : 100=61.16%, 250=12.19%, 500=0.18%, 750=0.07%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.03%, 10=1.36%, 20=3.32%, 50=15.18% lat (msec) : 100=0.34%, 250=0.06%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=2.01%, sys=4.54%, ctx=49192, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41710,10541,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=347MiB/s (364MB/s), 347MiB/s-347MiB/s (364MB/s-364MB/s), io=10.2GiB (10.9GB), run=30067-30067msec WRITE: bw=87.6MiB/s (91.9MB/s), 87.6MiB/s-87.6MiB/s (91.9MB/s-91.9MB/s), io=2635MiB (2763MB), run=30067-30067msec Disk stats (read/write): vdf: ios=41522/10491, merge=0/0, ticks=5683/274805, in_queue=297036, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944960: Fri Jan 7 14:12:48 2022 read: IOPS=1336, BW=334MiB/s (350MB/s)(9.79GiB/30001msec) clat (usec): min=56, max=58904, avg=105.84, stdev=597.64 lat (usec): min=56, max=58904, avg=105.98, stdev=597.64 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 71], 10.00th=[ 73], 20.00th=[ 75], | 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 83], 60.00th=[ 87], | 70.00th=[ 92], 80.00th=[ 97], 90.00th=[ 105], 95.00th=[ 113], | 99.00th=[ 137], 99.50th=[ 153], 99.90th=[ 9110], 99.95th=[13960], | 99.99th=[23200] bw ( KiB/s): min=266752, max=565760, per=99.98%, avg=342148.93, stdev=41485.48, samples=59 iops : min= 1042, max= 2210, avg=1336.51, stdev=162.05, samples=59 write: IOPS=337, BW=84.4MiB/s (88.5MB/s)(2532MiB/30001msec) clat (usec): min=844, max=106171, avg=2509.74, stdev=2510.03 lat (usec): min=868, max=106195, avg=2534.15, stdev=2509.96 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1795], | 30.00th=[ 1942], 40.00th=[ 2073], 50.00th=[ 2212], 60.00th=[ 2311], | 70.00th=[ 2474], 80.00th=[ 2671], 90.00th=[ 3130], 95.00th=[ 4490], | 99.00th=[10552], 99.50th=[11076], 99.90th=[42730], 99.95th=[55313], | 99.99th=[62653] bw ( KiB/s): min=61952, max=151552, per=99.86%, avg=86313.98, stdev=10494.77, samples=59 iops : min= 242, max= 592, avg=337.15, stdev=40.99, samples=59 lat (usec) : 100=67.84%, 250=11.83%, 500=0.03%, 750=0.01%, 1000=0.60% lat (msec) : 2=6.32%, 4=11.83%, 10=1.03%, 20=0.47%, 50=0.02% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.70%, sys=2.88%, ctx=50238, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40104,10129,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=334MiB/s (350MB/s), 334MiB/s-334MiB/s (350MB/s-350MB/s), io=9.79GiB (10.5GB), run=30001-30001msec WRITE: bw=84.4MiB/s (88.5MB/s), 84.4MiB/s-84.4MiB/s (88.5MB/s-88.5MB/s), io=2532MiB (2655MB), run=30001-30001msec Disk stats (read/write): vdf: ios=39899/10080, merge=0/0, ticks=762/20371, in_queue=28439, util=94.90% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945596: Fri Jan 7 14:13:26 2022 read: IOPS=1378, BW=345MiB/s (361MB/s)(10.2GiB/30168msec) slat (usec): min=4, max=187, avg=14.15, stdev= 6.06 clat (nsec): min=812, max=2553.2M, avg=2093121.30, stdev=63401511.64 lat (usec): min=46, max=2553.2k, avg=2107.43, stdev=63401.49 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 50], 10.00th=[ 51], | 20.00th=[ 54], 30.00th=[ 57], 40.00th=[ 60], | 50.00th=[ 64], 60.00th=[ 72], 70.00th=[ 92], | 80.00th=[ 115], 90.00th=[ 133], 95.00th=[ 169], | 99.00th=[ 1139], 99.50th=[ 2671], 99.90th=[ 574620], | 99.95th=[2499806], 99.99th=[2533360] bw ( KiB/s): min=283136, max=785408, per=100.00%, avg=354781.87, stdev=67684.83, samples=60 iops : min= 1106, max= 3068, avg=1385.87, stdev=264.39, samples=60 write: IOPS=348, BW=87.1MiB/s (91.3MB/s)(2627MiB/30168msec) slat (usec): min=22, max=499, avg=56.47, stdev=26.52 clat (msec): min=14, max=3540, avg=135.15, stdev=80.85 lat (msec): min=14, max=3540, avg=135.20, stdev=80.85 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 114], | 30.00th=[ 121], 40.00th=[ 125], 50.00th=[ 130], 60.00th=[ 136], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 161], 95.00th=[ 180], | 99.00th=[ 271], 99.50th=[ 558], 99.90th=[ 1099], 99.95th=[ 1670], | 99.99th=[ 2937] bw ( KiB/s): min=72192, max=180224, per=100.00%, avg=89258.67, stdev=13207.01, samples=60 iops : min= 282, max= 704, avg=348.67, stdev=51.59, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 20=0.01%, 50=5.27%, 100=52.20%, 250=20.15% lat (usec) : 500=0.70%, 750=0.41%, 1000=0.18% lat (msec) : 2=0.32%, 4=0.43%, 10=0.02%, 20=0.06%, 50=0.51% lat (msec) : 100=0.90%, 250=18.46%, 500=0.17%, 750=0.07%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.05% cpu : usr=2.31%, sys=4.00%, ctx=47049, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=41580,10506,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=345MiB/s (361MB/s), 345MiB/s-345MiB/s (361MB/s-361MB/s), io=10.2GiB (10.9GB), run=30168-30168msec WRITE: bw=87.1MiB/s (91.3MB/s), 87.1MiB/s-87.1MiB/s (91.3MB/s-91.3MB/s), io=2627MiB (2754MB), run=30168-30168msec Disk stats (read/write): vdf: ios=41577/10497, merge=0/0, ticks=80218/1409021, in_queue=1501652, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946481: Fri Jan 7 14:14:02 2022 read: IOPS=1364, BW=341MiB/s (358MB/s)(9.00GiB/30002msec) clat (usec): min=48, max=18139, avg=89.09, stdev=399.46 lat (usec): min=48, max=18139, avg=89.23, stdev=399.46 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 69], 50.00th=[ 71], 60.00th=[ 75], | 70.00th=[ 79], 80.00th=[ 84], 90.00th=[ 91], 95.00th=[ 98], | 99.00th=[ 119], 99.50th=[ 135], 99.90th=[ 8717], 99.95th=[10945], | 99.99th=[14615] bw ( KiB/s): min=273408, max=576512, per=100.00%, avg=349371.73, stdev=44459.78, samples=60 iops : min= 1068, max= 2252, avg=1364.73, stdev=173.67, samples=60 write: IOPS=344, BW=86.1MiB/s (90.3MB/s)(2585MiB/30002msec) clat (usec): min=800, max=63218, avg=2516.59, stdev=2194.94 lat (usec): min=828, max=63237, avg=2540.83, stdev=2194.90 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1074], 10.00th=[ 1254], 20.00th=[ 1860], | 30.00th=[ 2008], 40.00th=[ 2114], 50.00th=[ 2245], 60.00th=[ 2376], | 70.00th=[ 2507], 80.00th=[ 2704], 90.00th=[ 3097], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[10945], 99.90th=[16581], 99.95th=[52691], | 99.99th=[61604] bw ( KiB/s): min=75264, max=153088, per=100.00%, avg=88209.07, stdev=9871.99, samples=60 iops : min= 294, max= 598, avg=344.57, stdev=38.56, samples=60 lat (usec) : 50=0.01%, 100=76.59%, 250=3.07%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.56% lat (msec) : 2=5.51%, 4=12.83%, 10=0.94%, 20=0.43%, 100=0.02% cpu : usr=1.67%, sys=2.47%, ctx=51285, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40942,10338,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=341MiB/s (358MB/s), 341MiB/s-341MiB/s (358MB/s-358MB/s), io=9.00GiB (10.7GB), run=30002-30002msec WRITE: bw=86.1MiB/s (90.3MB/s), 86.1MiB/s-86.1MiB/s (90.3MB/s-90.3MB/s), io=2585MiB (2710MB), run=30002-30002msec Disk stats (read/write): vdf: ios=40769/10292, merge=0/0, ticks=578/20902, in_queue=28561, util=95.29% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947437: Fri Jan 7 14:14:40 2022 read: IOPS=1381, BW=345MiB/s (362MB/s)(10.2GiB/30281msec) slat (usec): min=4, max=435, avg=12.56, stdev= 6.84 clat (nsec): min=1145, max=2652.6M, avg=2184296.76, stdev=62142818.79 lat (usec): min=45, max=2652.7k, avg=2197.02, stdev=62142.83 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 60], | 50.00th=[ 65], 60.00th=[ 75], 70.00th=[ 106], | 80.00th=[ 120], 90.00th=[ 165], 95.00th=[ 603], | 99.00th=[ 5800], 99.50th=[ 7570], 99.90th=[ 450888], | 99.95th=[1920992], 99.99th=[2533360] bw ( KiB/s): min=259584, max=838284, per=100.00%, avg=356953.63, stdev=73418.00, samples=60 iops : min= 1014, max= 3274, avg=1394.32, stdev=286.73, samples=60 write: IOPS=349, BW=87.3MiB/s (91.5MB/s)(2644MiB/30281msec) slat (usec): min=19, max=230, avg=51.18, stdev=23.50 clat (msec): min=8, max=3485, avg=277.58, stdev=207.74 lat (msec): min=8, max=3485, avg=277.64, stdev=207.74 clat percentiles (msec): | 1.00th=[ 52], 5.00th=[ 144], 10.00th=[ 174], 20.00th=[ 232], | 30.00th=[ 249], 40.00th=[ 257], 50.00th=[ 266], 60.00th=[ 275], | 70.00th=[ 284], 80.00th=[ 296], 90.00th=[ 321], 95.00th=[ 384], | 99.00th=[ 726], 99.50th=[ 1036], 99.90th=[ 3272], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min=71168, max=164681, per=100.00%, avg=89427.07, stdev=11743.23, samples=60 iops : min= 278, max= 643, avg=349.28, stdev=45.83, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=4.85%, 100=49.56% lat (usec) : 250=19.49%, 500=1.60%, 750=0.65%, 1000=0.39% lat (msec) : 2=1.08%, 4=0.86%, 10=1.19%, 20=0.02%, 50=0.17% lat (msec) : 100=0.52%, 250=5.74%, 500=13.26%, 750=0.34%, 1000=0.09% lat (msec) : 2000=0.06%, >=2000=0.12% cpu : usr=2.33%, sys=3.46%, ctx=44920, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=41826,10574,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=10.2GiB (10.0GB), run=30281-30281msec WRITE: bw=87.3MiB/s (91.5MB/s), 87.3MiB/s-87.3MiB/s (91.5MB/s-91.5MB/s), io=2644MiB (2772MB), run=30281-30281msec Disk stats (read/write): vdf: ios=41824/10525, merge=0/0, ticks=84738/2901286, in_queue=3005180, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948011: Fri Jan 7 14:15:16 2022 read: IOPS=1350, BW=338MiB/s (354MB/s)(9.89GiB/30001msec) clat (usec): min=49, max=25274, avg=95.17, stdev=428.34 lat (usec): min=49, max=25274, avg=95.31, stdev=428.34 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 135], 99.50th=[ 157], 99.90th=[ 7963], 99.95th=[12125], | 99.99th=[14353] bw ( KiB/s): min=263168, max=560640, per=99.92%, avg=345348.34, stdev=46853.51, samples=59 iops : min= 1028, max= 2190, avg=1349.02, stdev=183.02, samples=59 write: IOPS=340, BW=85.2MiB/s (89.4MB/s)(2557MiB/30001msec) clat (usec): min=808, max=54622, avg=2522.63, stdev=2281.07 lat (usec): min=827, max=54646, avg=2547.76, stdev=2280.86 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1074], 10.00th=[ 1237], 20.00th=[ 1844], | 30.00th=[ 1975], 40.00th=[ 2114], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2507], 80.00th=[ 2671], 90.00th=[ 3130], 95.00th=[ 4424], | 99.00th=[10552], 99.50th=[10945], 99.90th=[52167], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=71680, max=150528, per=99.97%, avg=87248.27, stdev=10242.70, samples=59 iops : min= 280, max= 588, avg=340.81, stdev=40.01, samples=59 lat (usec) : 50=0.01%, 100=73.65%, 250=5.99%, 500=0.04%, 750=0.02% lat (usec) : 1000=0.58% lat (msec) : 2=5.82%, 4=12.39%, 10=0.99%, 20=0.48%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.66%, sys=3.06%, ctx=50739, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40503,10228,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=338MiB/s (354MB/s), 338MiB/s-338MiB/s (354MB/s-354MB/s), io=9.89GiB (10.6GB), run=30001-30001msec WRITE: bw=85.2MiB/s (89.4MB/s), 85.2MiB/s-85.2MiB/s (89.4MB/s-89.4MB/s), io=2557MiB (2681MB), run=30001-30001msec Disk stats (read/write): vdf: ios=40283/10171, merge=0/0, ticks=597/20665, in_queue=28363, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949364: Fri Jan 7 14:15:54 2022 read: IOPS=1383, BW=346MiB/s (363MB/s)(10.3GiB/30406msec) slat (usec): min=4, max=350027, avg=557.34, stdev=4588.04 clat (msec): min=5, max=2888, avg=47.83, stdev=76.59 lat (msec): min=5, max=2888, avg=48.38, stdev=76.70 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 13], 20.00th=[ 32], | 30.00th=[ 35], 40.00th=[ 39], 50.00th=[ 42], 60.00th=[ 46], | 70.00th=[ 51], 80.00th=[ 57], 90.00th=[ 69], 95.00th=[ 81], | 99.00th=[ 309], 99.50th=[ 368], 99.90th=[ 642], 99.95th=[ 2534], | 99.99th=[ 2601] bw ( KiB/s): min=279552, max=903680, per=100.00%, avg=358402.72, stdev=80799.33, samples=60 iops : min= 1092, max= 3530, avg=1400.00, stdev=315.63, samples=60 write: IOPS=349, BW=87.4MiB/s (91.6MB/s)(2656MiB/30406msec) slat (usec): min=27, max=300666, avg=606.40, stdev=4422.76 clat (msec): min=31, max=2694, avg=380.07, stdev=130.62 lat (msec): min=31, max=2694, avg=380.68, stdev=130.61 clat percentiles (msec): | 1.00th=[ 57], 5.00th=[ 218], 10.00th=[ 309], 20.00th=[ 342], | 30.00th=[ 359], 40.00th=[ 368], 50.00th=[ 376], 60.00th=[ 384], | 70.00th=[ 397], 80.00th=[ 409], 90.00th=[ 439], 95.00th=[ 531], | 99.00th=[ 768], 99.50th=[ 1070], 99.90th=[ 1770], 99.95th=[ 1770], | 99.99th=[ 2668] bw ( KiB/s): min=69632, max=168960, per=100.00%, avg=89551.58, stdev=12330.55, samples=60 iops : min= 272, max= 660, avg=349.80, stdev=48.17, samples=60 lat (msec) : 10=5.34%, 20=5.53%, 50=43.92%, 100=23.90%, 250=1.53% lat (msec) : 500=18.53%, 750=0.95%, 1000=0.11%, 2000=0.14%, >=2000=0.05% cpu : usr=3.30%, sys=1.72%, ctx=5584, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42068,10624,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=346MiB/s (363MB/s), 346MiB/s-346MiB/s (363MB/s-363MB/s), io=10.3GiB (11.0GB), run=30406-30406msec WRITE: bw=87.4MiB/s (91.6MB/s), 87.4MiB/s-87.4MiB/s (91.6MB/s-91.6MB/s), io=2656MiB (2785MB), run=30406-30406msec Disk stats (read/write): vdf: ios=42067/10574, merge=0/0, ticks=111901/3543618, in_queue=3706066, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950046: Fri Jan 7 14:16:30 2022 read: IOPS=885, BW=443MiB/s (465MB/s)(12.0GiB/30004msec) clat (usec): min=115, max=23788, avg=184.30, stdev=576.17 lat (usec): min=115, max=23788, avg=184.46, stdev=576.17 clat percentiles (usec): | 1.00th=[ 124], 5.00th=[ 129], 10.00th=[ 131], 20.00th=[ 135], | 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 151], 60.00th=[ 159], | 70.00th=[ 167], 80.00th=[ 180], 90.00th=[ 196], 95.00th=[ 217], | 99.00th=[ 269], 99.50th=[ 306], 99.90th=[11863], 99.95th=[14484], | 99.99th=[19792] bw ( KiB/s): min=344064, max=605184, per=100.00%, avg=453697.67, stdev=54500.84, samples=60 iops : min= 672, max= 1182, avg=886.12, stdev=106.45, samples=60 write: IOPS=223, BW=112MiB/s (117MB/s)(3354MiB/30004msec) clat (usec): min=1400, max=56328, avg=3682.53, stdev=2580.06 lat (usec): min=1449, max=56380, avg=3732.79, stdev=2579.63 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 1713], 10.00th=[ 1795], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 2999], 50.00th=[ 3228], 60.00th=[ 3425], | 70.00th=[ 3654], 80.00th=[ 4047], 90.00th=[ 6325], 95.00th=[ 7635], | 99.00th=[11731], 99.50th=[13829], 99.90th=[53740], 99.95th=[54789], | 99.99th=[56361] bw ( KiB/s): min=92160, max=172032, per=100.00%, avg=114469.65, stdev=11831.60, samples=60 iops : min= 180, max= 336, avg=223.57, stdev=23.11, samples=60 lat (usec) : 250=78.53%, 500=1.11%, 750=0.02% lat (msec) : 2=2.70%, 4=13.22%, 10=3.80%, 20=0.58%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.68%, sys=2.62%, ctx=33298, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=26583,6708,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=443MiB/s (465MB/s), 443MiB/s-443MiB/s (465MB/s-465MB/s), io=12.0GiB (13.9GB), run=30004-30004msec WRITE: bw=112MiB/s (117MB/s), 112MiB/s-112MiB/s (117MB/s-117MB/s), io=3354MiB (3517MB), run=30004-30004msec Disk stats (read/write): vdf: ios=26475/6681, merge=0/0, ticks=647/21019, in_queue=28531, util=95.23% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950864: Fri Jan 7 14:17:07 2022 read: IOPS=925, BW=463MiB/s (485MB/s)(13.6GiB/30002msec) slat (usec): min=18, max=124, avg=30.98, stdev= 7.82 clat (usec): min=37, max=16231, avg=149.94, stdev=550.18 lat (usec): min=108, max=16261, avg=181.10, stdev=550.31 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 101], 10.00th=[ 103], 20.00th=[ 106], | 30.00th=[ 110], 40.00th=[ 114], 50.00th=[ 119], 60.00th=[ 126], | 70.00th=[ 133], 80.00th=[ 141], 90.00th=[ 155], 95.00th=[ 169], | 99.00th=[ 217], 99.50th=[ 247], 99.90th=[12256], 99.95th=[13960], | 99.99th=[15926] bw ( KiB/s): min=384000, max=619520, per=100.00%, avg=474112.00, stdev=45294.41, samples=60 iops : min= 750, max= 1210, avg=926.00, stdev=88.47, samples=60 write: IOPS=233, BW=117MiB/s (122MB/s)(3501MiB/30002msec) slat (usec): min=38, max=430, avg=80.99, stdev=25.30 clat (usec): min=1418, max=71207, avg=3476.47, stdev=2548.15 lat (usec): min=1495, max=71317, avg=3557.67, stdev=2547.73 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1663], 10.00th=[ 1713], 20.00th=[ 2114], | 30.00th=[ 2606], 40.00th=[ 2900], 50.00th=[ 3163], 60.00th=[ 3359], | 70.00th=[ 3589], 80.00th=[ 3916], 90.00th=[ 5407], 95.00th=[ 6980], | 99.00th=[11469], 99.50th=[11994], 99.90th=[17433], 99.95th=[61604], | 99.99th=[70779] bw ( KiB/s): min=95232, max=181248, per=99.99%, avg=119483.73, stdev=10942.25, samples=60 iops : min= 186, max= 354, avg=233.37, stdev=21.37, samples=60 lat (usec) : 50=0.01%, 100=3.13%, 250=76.35%, 500=0.18%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=3.78%, 4=12.69%, 10=3.34%, 20=0.48%, 100=0.02% cpu : usr=1.72%, sys=4.69%, ctx=34811, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27780,7002,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=463MiB/s (485MB/s), 463MiB/s-463MiB/s (485MB/s-485MB/s), io=13.6GiB (14.6GB), run=30002-30002msec WRITE: bw=117MiB/s (122MB/s), 117MiB/s-117MiB/s (122MB/s-122MB/s), io=3501MiB (3671MB), run=30002-30002msec Disk stats (read/write): vdf: ios=27675/6971, merge=0/0, ticks=671/20550, in_queue=28376, util=94.70% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951790: Fri Jan 7 14:17:44 2022 read: IOPS=952, BW=476MiB/s (499MB/s)(13.0GiB/30001msec) clat (usec): min=80, max=20752, avg=138.40, stdev=562.68 lat (usec): min=81, max=20752, avg=138.55, stdev=562.68 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 97], | 30.00th=[ 100], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 113], | 70.00th=[ 120], 80.00th=[ 128], 90.00th=[ 139], 95.00th=[ 149], | 99.00th=[ 184], 99.50th=[ 206], 99.90th=[12125], 99.95th=[13960], | 99.99th=[16909] bw ( KiB/s): min=338944, max=616448, per=100.00%, avg=488515.36, stdev=53739.35, samples=59 iops : min= 662, max= 1204, avg=954.10, stdev=104.96, samples=59 write: IOPS=240, BW=120MiB/s (126MB/s)(3606MiB/30001msec) clat (usec): min=1383, max=63132, avg=3551.63, stdev=2396.53 lat (usec): min=1408, max=63162, avg=3602.64, stdev=2395.92 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1663], 10.00th=[ 1762], 20.00th=[ 2540], | 30.00th=[ 2933], 40.00th=[ 3163], 50.00th=[ 3294], 60.00th=[ 3490], | 70.00th=[ 3687], 80.00th=[ 3949], 90.00th=[ 4555], 95.00th=[ 6521], | 99.00th=[11207], 99.50th=[11731], 99.90th=[53740], 99.95th=[56361], | 99.99th=[63177] bw ( KiB/s): min=98304, max=179200, per=100.00%, avg=123270.31, stdev=10127.78, samples=59 iops : min= 192, max= 350, avg=240.75, stdev=19.78, samples=59 lat (usec) : 100=24.74%, 250=54.85%, 500=0.08%, 750=0.01% lat (msec) : 2=2.54%, 4=13.87%, 10=3.48%, 20=0.42%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.94%, sys=2.41%, ctx=35792, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28570,7212,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=13.0GiB (14.0GB), run=30001-30001msec WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=3606MiB (3781MB), run=30001-30001msec Disk stats (read/write): vdf: ios=28479/7185, merge=0/0, ticks=698/21724, in_queue=28520, util=95.17% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952393: Fri Jan 7 14:18:22 2022 read: IOPS=960, BW=480MiB/s (504MB/s)(14.1GiB/30054msec) slat (usec): min=6, max=118, avg=17.39, stdev= 6.73 clat (usec): min=16, max=77435, avg=187.33, stdev=1679.51 lat (usec): min=69, max=77448, avg=204.88, stdev=1679.55 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 70], 10.00th=[ 71], 20.00th=[ 74], | 30.00th=[ 77], 40.00th=[ 81], 50.00th=[ 88], 60.00th=[ 94], | 70.00th=[ 110], 80.00th=[ 174], 90.00th=[ 200], 95.00th=[ 221], | 99.00th=[ 326], 99.50th=[ 570], 99.90th=[34866], 99.95th=[40109], | 99.99th=[60556] bw ( KiB/s): min=362496, max=706560, per=100.00%, avg=492604.55, stdev=57662.14, samples=60 iops : min= 708, max= 1380, avg=962.10, stdev=112.63, samples=60 write: IOPS=242, BW=121MiB/s (127MB/s)(3649MiB/30054msec) slat (usec): min=35, max=1279, avg=89.79, stdev=44.46 clat (msec): min=2, max=159, avg=40.26, stdev=16.31 lat (msec): min=2, max=159, avg=40.35, stdev=16.31 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 18], 20.00th=[ 29], | 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 48], 80.00th=[ 53], 90.00th=[ 60], 95.00th=[ 66], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 124], 99.95th=[ 131], | 99.99th=[ 161] bw ( KiB/s): min=107520, max=197632, per=100.00%, avg=124358.63, stdev=11406.87, samples=60 iops : min= 210, max= 386, avg=242.87, stdev=22.28, samples=60 lat (usec) : 20=0.01%, 50=0.02%, 100=52.22%, 250=25.61%, 500=1.52% lat (usec) : 750=0.12%, 1000=0.06% lat (msec) : 2=0.09%, 4=0.01%, 10=0.63%, 20=1.64%, 50=12.95% lat (msec) : 100=5.03%, 250=0.09% cpu : usr=2.60%, sys=2.91%, ctx=34750, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28873,7298,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=480MiB/s (504MB/s), 480MiB/s-480MiB/s (504MB/s-504MB/s), io=14.1GiB (15.1GB), run=30054-30054msec WRITE: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=3649MiB (3826MB), run=30054-30054msec Disk stats (read/write): vdf: ios=28755/7253, merge=0/0, ticks=2117/288118, in_queue=297274, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953224: Fri Jan 7 14:18:59 2022 read: IOPS=955, BW=478MiB/s (501MB/s)(13.0GiB/30002msec) clat (usec): min=78, max=50006, avg=136.56, stdev=645.87 lat (usec): min=78, max=50006, avg=136.71, stdev=645.87 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 97], 40.00th=[ 100], 50.00th=[ 104], 60.00th=[ 109], | 70.00th=[ 116], 80.00th=[ 123], 90.00th=[ 133], 95.00th=[ 141], | 99.00th=[ 169], 99.50th=[ 190], 99.90th=[12387], 99.95th=[15270], | 99.99th=[17433] bw ( KiB/s): min=370688, max=608256, per=100.00%, avg=489094.43, stdev=49608.63, samples=60 iops : min= 724, max= 1188, avg=955.25, stdev=96.90, samples=60 write: IOPS=241, BW=121MiB/s (126MB/s)(3616MiB/30002msec) clat (usec): min=1387, max=62711, avg=3549.36, stdev=2364.43 lat (usec): min=1414, max=62761, avg=3598.52, stdev=2363.94 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1663], 10.00th=[ 1745], 20.00th=[ 2573], | 30.00th=[ 2933], 40.00th=[ 3130], 50.00th=[ 3294], 60.00th=[ 3490], | 70.00th=[ 3654], 80.00th=[ 3916], 90.00th=[ 4555], 95.00th=[ 6521], | 99.00th=[11469], 99.50th=[12125], 99.90th=[18744], 99.95th=[56886], | 99.99th=[62653] bw ( KiB/s): min=105472, max=173056, per=100.00%, avg=123413.20, stdev=10127.68, samples=60 iops : min= 206, max= 338, avg=241.03, stdev=19.78, samples=60 lat (usec) : 100=32.04%, 250=47.57%, 500=0.04%, 1000=0.01% lat (msec) : 2=2.58%, 4=13.99%, 10=3.29%, 20=0.45%, 100=0.02% cpu : usr=1.91%, sys=2.60%, ctx=35895, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28657,7232,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=478MiB/s (501MB/s), 478MiB/s-478MiB/s (501MB/s-501MB/s), io=13.0GiB (15.0GB), run=30002-30002msec WRITE: bw=121MiB/s (126MB/s), 121MiB/s-121MiB/s (126MB/s-126MB/s), io=3616MiB (3792MB), run=30002-30002msec Disk stats (read/write): vdf: ios=28516/7196, merge=0/0, ticks=764/21683, in_queue=28467, util=95.01% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954178: Fri Jan 7 14:19:36 2022 read: IOPS=963, BW=482MiB/s (505MB/s)(14.2GiB/30247msec) slat (usec): min=5, max=209, avg=15.78, stdev= 6.75 clat (nsec): min=1349, max=263672k, avg=408239.13, stdev=5351201.92 lat (usec): min=70, max=263684, avg=424.17, stdev=5351.12 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 88], | 70.00th=[ 109], 80.00th=[ 184], 90.00th=[ 206], 95.00th=[ 297], | 99.00th=[ 3458], 99.50th=[ 6390], 99.90th=[ 91751], 99.95th=[128451], | 99.99th=[217056] bw ( KiB/s): min=356352, max=966656, per=100.00%, avg=497544.53, stdev=80753.42, samples=60 iops : min= 696, max= 1888, avg=971.77, stdev=157.72, samples=60 write: IOPS=243, BW=122MiB/s (128MB/s)(3683MiB/30247msec) slat (usec): min=35, max=496, avg=91.32, stdev=45.90 clat (msec): min=6, max=2522, avg=203.53, stdev=105.15 lat (msec): min=6, max=2522, avg=203.62, stdev=105.15 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 73], 10.00th=[ 87], 20.00th=[ 112], | 30.00th=[ 140], 40.00th=[ 178], 50.00th=[ 211], 60.00th=[ 232], | 70.00th=[ 251], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 338], | 99.00th=[ 443], 99.50th=[ 506], 99.90th=[ 1301], 99.95th=[ 1552], | 99.99th=[ 2534] bw ( KiB/s): min=104448, max=216064, per=100.00%, avg=124842.67, stdev=13569.57, samples=60 iops : min= 204, max= 422, avg=243.83, stdev=26.50, samples=60 lat (usec) : 2=0.01%, 50=0.03%, 100=54.01%, 250=20.97%, 500=1.91% lat (usec) : 750=0.54%, 1000=0.34% lat (msec) : 2=0.79%, 4=0.46%, 10=0.61%, 20=0.02%, 50=0.20% lat (msec) : 100=3.07%, 250=10.96%, 500=5.97%, 750=0.04%, 1000=0.02% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.62%, sys=2.73%, ctx=33902, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=29153,7365,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=482MiB/s (505MB/s), 482MiB/s-482MiB/s (505MB/s-505MB/s), io=14.2GiB (15.3GB), run=30247-30247msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3683MiB (3861MB), run=30247-30247msec Disk stats (read/write): vdf: ios=29153/7316, merge=0/0, ticks=7883/1471955, in_queue=1504275, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954770: Fri Jan 7 14:20:13 2022 read: IOPS=955, BW=478MiB/s (501MB/s)(13.0GiB/30003msec) clat (usec): min=80, max=57014, avg=139.27, stdev=654.58 lat (usec): min=80, max=57014, avg=139.42, stdev=654.58 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 98], | 30.00th=[ 100], 40.00th=[ 103], 50.00th=[ 108], 60.00th=[ 113], | 70.00th=[ 119], 80.00th=[ 127], 90.00th=[ 137], 95.00th=[ 147], | 99.00th=[ 182], 99.50th=[ 206], 99.90th=[11469], 99.95th=[14484], | 99.99th=[22152] bw ( KiB/s): min=382976, max=656384, per=100.00%, avg=489045.33, stdev=58029.58, samples=60 iops : min= 748, max= 1282, avg=955.17, stdev=113.34, samples=60 write: IOPS=240, BW=120MiB/s (126MB/s)(3615MiB/30003msec) clat (usec): min=1452, max=60515, avg=3538.12, stdev=2213.62 lat (usec): min=1478, max=60566, avg=3588.76, stdev=2212.84 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2474], | 30.00th=[ 2900], 40.00th=[ 3130], 50.00th=[ 3294], 60.00th=[ 3458], | 70.00th=[ 3687], 80.00th=[ 3949], 90.00th=[ 4621], 95.00th=[ 6652], | 99.00th=[11338], 99.50th=[12256], 99.90th=[20317], 99.95th=[54789], | 99.99th=[60556] bw ( KiB/s): min=99328, max=189440, per=100.00%, avg=123374.93, stdev=11723.24, samples=60 iops : min= 194, max= 370, avg=240.97, stdev=22.90, samples=60 lat (usec) : 100=24.08%, 250=55.52%, 500=0.06%, 1000=0.01% lat (msec) : 2=2.81%, 4=13.61%, 10=3.45%, 20=0.43%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.82%, sys=2.86%, ctx=35897, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28655,7230,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=478MiB/s (501MB/s), 478MiB/s-478MiB/s (501MB/s-501MB/s), io=13.0GiB (15.0GB), run=30003-30003msec WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=3615MiB (3791MB), run=30003-30003msec Disk stats (read/write): vdf: ios=28488/7189, merge=0/0, ticks=733/21598, in_queue=28339, util=94.57% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=955998: Fri Jan 7 14:20:50 2022 read: IOPS=973, BW=487MiB/s (511MB/s)(14.5GiB/30401msec) slat (usec): min=5, max=439, avg=14.69, stdev= 9.19 clat (nsec): min=1520, max=426215k, avg=793221.39, stdev=6123267.99 lat (usec): min=69, max=426235, avg=808.05, stdev=6123.17 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 83], 60.00th=[ 93], | 70.00th=[ 176], 80.00th=[ 194], 90.00th=[ 445], 95.00th=[ 2769], | 99.00th=[ 13173], 99.50th=[ 15664], 99.90th=[106431], 99.95th=[129500], | 99.99th=[231736] bw ( KiB/s): min=60416, max=1098752, per=100.00%, avg=505207.47, stdev=131147.50, samples=60 iops : min= 118, max= 2146, avg=986.73, stdev=256.15, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3734MiB/30401msec) slat (usec): min=35, max=489, avg=97.31, stdev=49.58 clat (msec): min=13, max=3071, avg=403.68, stdev=143.10 lat (msec): min=13, max=3071, avg=403.78, stdev=143.10 clat percentiles (msec): | 1.00th=[ 86], 5.00th=[ 205], 10.00th=[ 271], 20.00th=[ 300], | 30.00th=[ 334], 40.00th=[ 372], 50.00th=[ 405], 60.00th=[ 435], | 70.00th=[ 456], 80.00th=[ 485], 90.00th=[ 535], 95.00th=[ 617], | 99.00th=[ 818], 99.50th=[ 869], 99.90th=[ 1603], 99.95th=[ 2072], | 99.99th=[ 3071] bw ( KiB/s): min=106496, max=192512, per=99.99%, avg=125764.27, stdev=11125.87, samples=60 iops : min= 208, max= 376, avg=245.63, stdev=21.73, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 50=0.06%, 100=50.15%, 250=19.65% lat (usec) : 500=2.26%, 750=1.00%, 1000=0.57% lat (msec) : 2=1.46%, 4=1.59%, 10=1.60%, 20=1.30%, 50=0.06% lat (msec) : 100=0.36%, 250=1.21%, 500=15.46%, 750=2.87%, 1000=0.34% lat (msec) : 2000=0.05%, >=2000=0.01% cpu : usr=2.68%, sys=2.63%, ctx=31892, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=29603,7468,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=487MiB/s (511MB/s), 487MiB/s-487MiB/s (511MB/s-511MB/s), io=14.5GiB (15.5GB), run=30401-30401msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3734MiB (3915MB), run=30401-30401msec Disk stats (read/write): vdf: ios=29603/7385, merge=0/0, ticks=16474/2956738, in_queue=3025146, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956701: Fri Jan 7 14:21:27 2022 read: IOPS=951, BW=476MiB/s (499MB/s)(13.9GiB/30003msec) clat (usec): min=81, max=23471, avg=139.14, stdev=600.00 lat (usec): min=81, max=23471, avg=139.29, stdev=600.00 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 94], 10.00th=[ 96], 20.00th=[ 98], | 30.00th=[ 100], 40.00th=[ 102], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 119], 80.00th=[ 126], 90.00th=[ 137], 95.00th=[ 147], | 99.00th=[ 178], 99.50th=[ 206], 99.90th=[13173], 99.95th=[14746], | 99.99th=[16712] bw ( KiB/s): min=376832, max=642048, per=100.00%, avg=487337.32, stdev=50752.51, samples=60 iops : min= 736, max= 1254, avg=951.82, stdev=99.13, samples=60 write: IOPS=240, BW=120MiB/s (126MB/s)(3604MiB/30003msec) clat (usec): min=1455, max=69639, avg=3554.33, stdev=2360.23 lat (usec): min=1497, max=69694, avg=3602.36, stdev=2359.64 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2474], | 30.00th=[ 2868], 40.00th=[ 3130], 50.00th=[ 3294], 60.00th=[ 3490], | 70.00th=[ 3687], 80.00th=[ 3949], 90.00th=[ 4686], 95.00th=[ 6849], | 99.00th=[11469], 99.50th=[12125], 99.90th=[19006], 99.95th=[54264], | 99.99th=[69731] bw ( KiB/s): min=104448, max=178176, per=100.00%, avg=123003.67, stdev=10011.53, samples=60 iops : min= 204, max= 348, avg=240.23, stdev=19.55, samples=60 lat (usec) : 100=26.06%, 250=53.52%, 500=0.06%, 750=0.02%, 1000=0.01% lat (msec) : 2=2.88%, 4=13.51%, 10=3.48%, 20=0.44%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.70%, sys=2.75%, ctx=35765, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28554,7208,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=476MiB/s (499MB/s), 476MiB/s-476MiB/s (499MB/s-499MB/s), io=13.9GiB (14.0GB), run=30003-30003msec WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=3604MiB (3779MB), run=30003-30003msec Disk stats (read/write): vdf: ios=28360/7158, merge=0/0, ticks=730/21598, in_queue=28394, util=94.79% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957721: Fri Jan 7 14:22:04 2022 read: IOPS=969, BW=485MiB/s (508MB/s)(14.4GiB/30438msec) slat (usec): min=6, max=501535, avg=737.67, stdev=5516.15 clat (msec): min=10, max=585, avg=65.99, stdev=56.17 lat (msec): min=10, max=585, avg=66.73, stdev=56.47 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 20], 10.00th=[ 26], 20.00th=[ 38], | 30.00th=[ 47], 40.00th=[ 54], 50.00th=[ 60], 60.00th=[ 67], | 70.00th=[ 74], 80.00th=[ 82], 90.00th=[ 95], 95.00th=[ 110], | 99.00th=[ 472], 99.50th=[ 518], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=19456, max=1045504, per=100.00%, avg=502409.58, stdev=191825.07, samples=60 iops : min= 38, max= 2042, avg=981.30, stdev=374.66, samples=60 write: IOPS=244, BW=122MiB/s (128MB/s)(3723MiB/30438msec) slat (usec): min=43, max=470860, avg=1094.77, stdev=9953.50 clat (msec): min=27, max=3267, avg=551.86, stdev=204.19 lat (msec): min=27, max=3267, avg=552.96, stdev=203.85 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 288], 10.00th=[ 372], 20.00th=[ 430], | 30.00th=[ 472], 40.00th=[ 510], 50.00th=[ 542], 60.00th=[ 575], | 70.00th=[ 609], 80.00th=[ 642], 90.00th=[ 701], 95.00th=[ 860], | 99.00th=[ 1368], 99.50th=[ 1620], 99.90th=[ 2198], 99.95th=[ 3071], | 99.99th=[ 3272] bw ( KiB/s): min= 3072, max=236544, per=99.57%, avg=124693.20, stdev=45710.07, samples=60 iops : min= 6, max= 462, avg=243.53, stdev=89.28, samples=60 lat (msec) : 20=4.63%, 50=23.16%, 100=46.11%, 250=5.70%, 500=7.22% lat (msec) : 750=11.68%, 1000=1.06%, 2000=0.40%, >=2000=0.04% cpu : usr=4.39%, sys=1.65%, ctx=3785, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=29499,7445,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=485MiB/s (508MB/s), 485MiB/s-485MiB/s (508MB/s-508MB/s), io=14.4GiB (15.5GB), run=30438-30438msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3723MiB (3903MB), run=30438-30438msec Disk stats (read/write): vdf: ios=29499/7403, merge=0/0, ticks=64871/3601887, in_queue=3714038, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958705: Fri Jan 7 14:22:41 2022 read: IOPS=556, BW=557MiB/s (584MB/s)(16.3GiB/30001msec) clat (usec): min=180, max=35252, avg=297.43, stdev=1003.17 lat (usec): min=180, max=35252, avg=297.60, stdev=1003.17 clat percentiles (usec): | 1.00th=[ 192], 5.00th=[ 198], 10.00th=[ 200], 20.00th=[ 206], | 30.00th=[ 210], 40.00th=[ 215], 50.00th=[ 223], 60.00th=[ 233], | 70.00th=[ 247], 80.00th=[ 262], 90.00th=[ 285], 95.00th=[ 314], | 99.00th=[ 416], 99.50th=[ 519], 99.90th=[17433], 99.95th=[18744], | 99.99th=[25822] bw ( KiB/s): min=415744, max=759808, per=100.00%, avg=570663.05, stdev=74751.53, samples=59 iops : min= 406, max= 742, avg=557.29, stdev=73.00, samples=59 write: IOPS=141, BW=142MiB/s (149MB/s)(4250MiB/30001msec) clat (usec): min=2481, max=59981, avg=5781.91, stdev=3189.91 lat (usec): min=2577, max=60077, avg=5880.73, stdev=3188.99 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3064], 10.00th=[ 3163], 20.00th=[ 3458], | 30.00th=[ 4178], 40.00th=[ 4817], 50.00th=[ 5211], 60.00th=[ 5538], | 70.00th=[ 5866], 80.00th=[ 6456], 90.00th=[ 9896], 95.00th=[11994], | 99.00th=[15401], 99.50th=[16909], 99.90th=[51643], 99.95th=[52691], | 99.99th=[60031] bw ( KiB/s): min=114688, max=192512, per=99.98%, avg=145026.17, stdev=10939.77, samples=59 iops : min= 112, max= 188, avg=141.63, stdev=10.68, samples=59 lat (usec) : 250=57.71%, 500=21.56%, 750=0.08%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.55%, 10=12.86%, 20=2.16%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.68%, sys=2.67%, ctx=20953, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=16696,4250,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=557MiB/s (584MB/s), 557MiB/s-557MiB/s (584MB/s-584MB/s), io=16.3GiB (17.5GB), run=30001-30001msec WRITE: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=4250MiB (4456MB), run=30001-30001msec Disk stats (read/write): vdf: ios=33280/8468, merge=0/0, ticks=1196/44584, in_queue=56124, util=95.30% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959007: Fri Jan 7 14:23:18 2022 read: IOPS=561, BW=562MiB/s (589MB/s)(16.5GiB/30002msec) slat (usec): min=34, max=146, avg=52.68, stdev=10.63 clat (usec): min=148, max=27415, avg=272.74, stdev=981.11 lat (usec): min=199, max=27460, avg=325.63, stdev=981.63 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 174], 10.00th=[ 176], 20.00th=[ 180], | 30.00th=[ 186], 40.00th=[ 190], 50.00th=[ 198], 60.00th=[ 210], | 70.00th=[ 223], 80.00th=[ 237], 90.00th=[ 265], 95.00th=[ 297], | 99.00th=[ 412], 99.50th=[ 742], 99.90th=[17695], 99.95th=[20055], | 99.99th=[25035] bw ( KiB/s): min=430080, max=765952, per=100.00%, avg=575268.55, stdev=81463.66, samples=60 iops : min= 420, max= 748, avg=561.78, stdev=79.55, samples=60 write: IOPS=142, BW=143MiB/s (150MB/s)(4282MiB/30002msec) slat (usec): min=80, max=402, avg=141.67, stdev=26.28 clat (usec): min=2825, max=64972, avg=5573.81, stdev=3066.41 lat (usec): min=2934, max=65075, avg=5715.73, stdev=3065.23 clat percentiles (usec): | 1.00th=[ 2966], 5.00th=[ 3064], 10.00th=[ 3130], 20.00th=[ 3359], | 30.00th=[ 4015], 40.00th=[ 4621], 50.00th=[ 5080], 60.00th=[ 5407], | 70.00th=[ 5800], 80.00th=[ 6325], 90.00th=[ 9634], 95.00th=[10945], | 99.00th=[14353], 99.50th=[15533], 99.90th=[20579], 99.95th=[60556], | 99.99th=[64750] bw ( KiB/s): min=126976, max=206848, per=99.99%, avg=146129.78, stdev=11211.50, samples=60 iops : min= 124, max= 202, avg=142.70, stdev=10.95, samples=60 lat (usec) : 250=68.41%, 500=10.78%, 750=0.15%, 1000=0.03% lat (msec) : 2=0.01%, 4=6.08%, 10=12.59%, 20=1.87%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=4.12%, ctx=21148, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=16853,4282,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=562MiB/s (589MB/s), 562MiB/s-562MiB/s (589MB/s-589MB/s), io=16.5GiB (17.7GB), run=30002-30002msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4282MiB (4490MB), run=30002-30002msec Disk stats (read/write): vdf: ios=33622/8532, merge=0/0, ticks=1185/43677, in_queue=56146, util=95.55% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960012: Fri Jan 7 14:23:55 2022 read: IOPS=586, BW=587MiB/s (615MB/s)(17.2GiB/30008msec) clat (usec): min=134, max=25012, avg=241.83, stdev=1039.04 lat (usec): min=135, max=25012, avg=242.00, stdev=1039.04 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 147], 10.00th=[ 149], 20.00th=[ 153], | 30.00th=[ 155], 40.00th=[ 159], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 198], 90.00th=[ 215], 95.00th=[ 233], | 99.00th=[ 326], 99.50th=[ 603], 99.90th=[18482], 99.95th=[19530], | 99.99th=[24249] bw ( KiB/s): min=409600, max=800768, per=100.00%, avg=600789.88, stdev=79814.52, samples=60 iops : min= 400, max= 782, avg=586.70, stdev=77.93, samples=60 write: IOPS=148, BW=149MiB/s (156MB/s)(4460MiB/30008msec) clat (usec): min=2531, max=65443, avg=5667.17, stdev=3101.89 lat (usec): min=2629, max=65545, avg=5764.36, stdev=3100.74 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3687], | 30.00th=[ 4621], 40.00th=[ 5080], 50.00th=[ 5407], 60.00th=[ 5669], | 70.00th=[ 5932], 80.00th=[ 6259], 90.00th=[ 8094], 95.00th=[11469], | 99.00th=[15270], 99.50th=[16909], 99.90th=[56361], 99.95th=[56886], | 99.99th=[65274] bw ( KiB/s): min=131072, max=192512, per=100.00%, avg=152211.05, stdev=9740.79, samples=60 iops : min= 128, max= 188, avg=148.63, stdev= 9.51, samples=60 lat (usec) : 250=77.19%, 500=2.17%, 750=0.04%, 1000=0.02% lat (msec) : 2=0.01%, 4=4.51%, 10=14.21%, 20=1.78%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.77%, sys=2.52%, ctx=22065, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17600,4460,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=587MiB/s (615MB/s), 587MiB/s-587MiB/s (615MB/s-615MB/s), io=17.2GiB (18.5GB), run=30008-30008msec WRITE: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=4460MiB (4677MB), run=30008-30008msec Disk stats (read/write): vdf: ios=17502/4440, merge=0/0, ticks=1150/22652, in_queue=28519, util=95.14% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961051: Fri Jan 7 14:24:32 2022 read: IOPS=558, BW=559MiB/s (586MB/s)(16.4GiB/30071msec) slat (usec): min=9, max=110, avg=22.93, stdev= 8.77 clat (usec): min=62, max=92441, avg=330.56, stdev=2564.86 lat (usec): min=114, max=92457, avg=353.66, stdev=2564.93 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 110], 20.00th=[ 112], | 30.00th=[ 115], 40.00th=[ 117], 50.00th=[ 121], 60.00th=[ 126], | 70.00th=[ 141], 80.00th=[ 326], 90.00th=[ 343], 95.00th=[ 359], | 99.00th=[ 734], 99.50th=[ 2704], 99.90th=[42206], 99.95th=[52691], | 99.99th=[86508] bw ( KiB/s): min=428032, max=815104, per=100.00%, avg=573405.87, stdev=87194.32, samples=60 iops : min= 418, max= 796, avg=559.97, stdev=85.15, samples=60 write: IOPS=141, BW=142MiB/s (149MB/s)(4266MiB/30071msec) slat (usec): min=61, max=514, avg=166.59, stdev=85.06 clat (msec): min=3, max=211, avg=68.91, stdev=27.74 lat (msec): min=3, max=211, avg=69.07, stdev=27.75 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 20], 10.00th=[ 26], 20.00th=[ 44], | 30.00th=[ 59], 40.00th=[ 67], 50.00th=[ 72], 60.00th=[ 78], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 103], 95.00th=[ 112], | 99.00th=[ 134], 99.50th=[ 142], 99.90th=[ 159], 99.95th=[ 167], | 99.99th=[ 211] bw ( KiB/s): min=124928, max=217088, per=100.00%, avg=145271.47, stdev=11988.53, samples=60 iops : min= 122, max= 212, avg=141.87, stdev=11.71, samples=60 lat (usec) : 100=0.11%, 250=62.07%, 500=16.43%, 750=0.35%, 1000=0.15% lat (msec) : 2=0.16%, 4=0.14%, 10=0.04%, 20=1.14%, 50=3.76% lat (msec) : 100=13.39%, 250=2.25% cpu : usr=2.56%, sys=2.05%, ctx=20628, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=16799,4266,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=559MiB/s (586MB/s), 559MiB/s-559MiB/s (586MB/s-586MB/s), io=16.4GiB (17.6GB), run=30071-30071msec WRITE: bw=142MiB/s (149MB/s), 142MiB/s-142MiB/s (149MB/s-149MB/s), io=4266MiB (4473MB), run=30071-30071msec Disk stats (read/write): vdf: ios=16678/4237, merge=0/0, ticks=2590/289426, in_queue=297516, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961499: Fri Jan 7 14:25:09 2022 read: IOPS=593, BW=593MiB/s (622MB/s)(17.4GiB/30001msec) clat (usec): min=116, max=26792, avg=214.94, stdev=1008.94 lat (usec): min=116, max=26793, avg=215.09, stdev=1008.95 clat percentiles (usec): | 1.00th=[ 124], 5.00th=[ 129], 10.00th=[ 131], 20.00th=[ 133], | 30.00th=[ 137], 40.00th=[ 139], 50.00th=[ 143], 60.00th=[ 147], | 70.00th=[ 157], 80.00th=[ 172], 90.00th=[ 186], 95.00th=[ 198], | 99.00th=[ 249], 99.50th=[ 433], 99.90th=[18220], 99.95th=[19530], | 99.99th=[25297] bw ( KiB/s): min=450560, max=782336, per=100.00%, avg=607334.40, stdev=79904.86, samples=60 iops : min= 440, max= 764, avg=593.10, stdev=78.03, samples=60 write: IOPS=150, BW=150MiB/s (157MB/s)(4504MiB/30001msec) clat (usec): min=2521, max=65148, avg=5707.61, stdev=2975.69 lat (usec): min=2589, max=65248, avg=5802.84, stdev=2975.08 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3982], | 30.00th=[ 4817], 40.00th=[ 5145], 50.00th=[ 5407], 60.00th=[ 5669], | 70.00th=[ 5932], 80.00th=[ 6259], 90.00th=[ 7963], 95.00th=[11469], | 99.00th=[15401], 99.50th=[16712], 99.90th=[24249], 99.95th=[56886], | 99.99th=[65274] bw ( KiB/s): min=133120, max=221184, per=99.98%, avg=153702.40, stdev=11770.45, samples=60 iops : min= 130, max= 216, avg=150.10, stdev=11.49, samples=60 lat (usec) : 250=79.01%, 500=0.42%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.09%, 10=14.66%, 20=1.75%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.73%, sys=1.67%, ctx=22310, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17793,4504,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=593MiB/s (622MB/s), 593MiB/s-593MiB/s (622MB/s-622MB/s), io=17.4GiB (18.7GB), run=30001-30001msec WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=4504MiB (4723MB), run=30001-30001msec Disk stats (read/write): vdf: ios=17708/4484, merge=0/0, ticks=1098/23122, in_queue=28823, util=96.14% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962978: Fri Jan 7 14:25:46 2022 read: IOPS=565, BW=566MiB/s (593MB/s)(16.8GiB/30336msec) slat (usec): min=9, max=371, avg=20.39, stdev=10.46 clat (usec): min=56, max=499978, avg=1192.40, stdev=11615.49 lat (usec): min=111, max=499993, avg=1212.97, stdev=11615.40 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 127], 60.00th=[ 139], | 70.00th=[ 237], 80.00th=[ 343], 90.00th=[ 449], 95.00th=[ 2868], | 99.00th=[ 12518], 99.50th=[ 14877], 99.90th=[179307], 99.95th=[235930], | 99.99th=[438305] bw ( KiB/s): min=374784, max=1150976, per=100.00%, avg=585982.12, stdev=126281.36, samples=60 iops : min= 366, max= 1124, avg=572.23, stdev=123.34, samples=60 write: IOPS=143, BW=143MiB/s (150MB/s)(4352MiB/30336msec) slat (usec): min=61, max=575, avg=186.23, stdev=94.72 clat (msec): min=11, max=1797, avg=343.44, stdev=176.31 lat (msec): min=11, max=1798, avg=343.63, stdev=176.30 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 126], 10.00th=[ 142], 20.00th=[ 174], | 30.00th=[ 215], 40.00th=[ 264], 50.00th=[ 321], 60.00th=[ 388], | 70.00th=[ 443], 80.00th=[ 493], 90.00th=[ 567], 95.00th=[ 659], | 99.00th=[ 810], 99.50th=[ 885], 99.90th=[ 1053], 99.95th=[ 1318], | 99.99th=[ 1804] bw ( KiB/s): min=108544, max=227328, per=100.00%, avg=146913.68, stdev=19334.08, samples=60 iops : min= 106, max= 222, avg=143.47, stdev=18.89, samples=60 lat (usec) : 100=0.16%, 250=55.96%, 500=16.08%, 750=1.01%, 1000=0.45% lat (msec) : 2=1.48%, 4=1.56%, 10=1.79%, 20=0.99%, 50=0.16% lat (msec) : 100=0.45%, 250=7.31%, 500=8.93%, 750=3.23%, 1000=0.41% lat (msec) : 2000=0.03% cpu : usr=2.88%, sys=1.88%, ctx=19149, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=17168,4352,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=566MiB/s (593MB/s), 566MiB/s-566MiB/s (593MB/s-593MB/s), io=16.8GiB (18.0GB), run=30336-30336msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4352MiB (4563MB), run=30336-30336msec Disk stats (read/write): vdf: ios=17167/4315, merge=0/0, ticks=15620/1470273, in_queue=1501178, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963422: Fri Jan 7 14:26:23 2022 read: IOPS=577, BW=577MiB/s (605MB/s)(16.9GiB/30006msec) clat (usec): min=128, max=68775, avg=235.44, stdev=1133.59 lat (usec): min=128, max=68775, avg=235.60, stdev=1133.59 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 206], 95.00th=[ 221], | 99.00th=[ 293], 99.50th=[ 474], 99.90th=[18220], 99.95th=[20055], | 99.99th=[27657] bw ( KiB/s): min=331776, max=768000, per=100.00%, avg=591343.03, stdev=83712.36, samples=60 iops : min= 324, max= 750, avg=577.48, stdev=81.75, samples=60 write: IOPS=146, BW=147MiB/s (154MB/s)(4402MiB/30006msec) clat (usec): min=2571, max=65150, avg=5782.97, stdev=3088.11 lat (usec): min=2647, max=65254, avg=5880.28, stdev=3086.73 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3589], | 30.00th=[ 4621], 40.00th=[ 5080], 50.00th=[ 5407], 60.00th=[ 5669], | 70.00th=[ 5997], 80.00th=[ 6521], 90.00th=[ 9241], 95.00th=[12125], | 99.00th=[16057], 99.50th=[16712], 99.90th=[20317], 99.95th=[53740], | 99.99th=[65274] bw ( KiB/s): min=124928, max=206848, per=100.00%, avg=150225.98, stdev=11506.66, samples=60 iops : min= 122, max= 202, avg=146.70, stdev=11.23, samples=60 lat (usec) : 250=78.04%, 500=1.30%, 750=0.02% lat (msec) : 2=0.02%, 4=4.67%, 10=13.87%, 20=2.00%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.82%, sys=2.36%, ctx=21734, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17324,4402,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=577MiB/s (605MB/s), 577MiB/s-577MiB/s (605MB/s-605MB/s), io=16.9GiB (18.2GB), run=30006-30006msec WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=4402MiB (4616MB), run=30006-30006msec Disk stats (read/write): vdf: ios=17255/4375, merge=0/0, ticks=1130/22829, in_queue=28505, util=95.10% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964371: Fri Jan 7 14:27:00 2022 read: IOPS=569, BW=570MiB/s (597MB/s)(17.1GiB/30649msec) slat (usec): min=9, max=681, avg=19.72, stdev=20.64 clat (usec): min=64, max=585116, avg=2916.46, stdev=13374.52 lat (usec): min=113, max=585133, avg=2936.34, stdev=13374.88 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 117], 40.00th=[ 122], 50.00th=[ 133], 60.00th=[ 243], | 70.00th=[ 343], 80.00th=[ 685], 90.00th=[ 8455], 95.00th=[ 17957], | 99.00th=[ 31327], 99.50th=[ 41681], 99.90th=[181404], 99.95th=[246416], | 99.99th=[484443] bw ( KiB/s): min=22528, max=1359872, per=100.00%, avg=596138.58, stdev=215739.38, samples=60 iops : min= 22, max= 1328, avg=582.15, stdev=210.67, samples=60 write: IOPS=144, BW=145MiB/s (152MB/s)(4431MiB/30649msec) slat (usec): min=61, max=668, avg=200.74, stdev=107.27 clat (msec): min=25, max=2415, avg=679.65, stdev=261.20 lat (msec): min=26, max=2416, avg=679.85, stdev=261.17 clat percentiles (msec): | 1.00th=[ 117], 5.00th=[ 313], 10.00th=[ 422], 20.00th=[ 493], | 30.00th=[ 535], 40.00th=[ 575], 50.00th=[ 634], 60.00th=[ 709], | 70.00th=[ 776], 80.00th=[ 852], 90.00th=[ 1003], 95.00th=[ 1167], | 99.00th=[ 1485], 99.50th=[ 1620], 99.90th=[ 1888], 99.95th=[ 2400], | 99.99th=[ 2433] bw ( KiB/s): min= 2048, max=256000, per=98.25%, avg=145458.02, stdev=51634.27, samples=61 iops : min= 2, max= 250, avg=142.03, stdev=50.41, samples=61 lat (usec) : 100=0.12%, 250=47.91%, 500=14.56%, 750=1.60%, 1000=1.06% lat (msec) : 2=2.03%, 4=2.01%, 10=3.32%, 20=3.76%, 50=3.10% lat (msec) : 100=0.23%, 250=0.74%, 500=3.58%, 750=9.23%, 1000=4.68% lat (msec) : 2000=2.05%, >=2000=0.02% cpu : usr=3.03%, sys=1.81%, ctx=16663, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=17463,4431,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=570MiB/s (597MB/s), 570MiB/s-570MiB/s (597MB/s-597MB/s), io=17.1GiB (18.3GB), run=30649-30649msec WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=4431MiB (4646MB), run=30649-30649msec Disk stats (read/write): vdf: ios=17463/4419, merge=0/0, ticks=41797/2965611, in_queue=3023161, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965442: Fri Jan 7 14:27:37 2022 read: IOPS=590, BW=591MiB/s (620MB/s)(17.3GiB/30003msec) clat (usec): min=129, max=26508, avg=233.56, stdev=1037.11 lat (usec): min=130, max=26508, avg=233.73, stdev=1037.11 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 174], 80.00th=[ 190], 90.00th=[ 202], 95.00th=[ 217], | 99.00th=[ 289], 99.50th=[ 529], 99.90th=[18220], 99.95th=[19792], | 99.99th=[25560] bw ( KiB/s): min=424785, max=776192, per=100.00%, avg=605209.55, stdev=83406.81, samples=60 iops : min= 414, max= 758, avg=591.00, stdev=81.47, samples=60 write: IOPS=149, BW=150MiB/s (157MB/s)(4493MiB/30003msec) clat (usec): min=2528, max=65002, avg=5650.55, stdev=2972.49 lat (usec): min=2630, max=65122, avg=5746.89, stdev=2971.65 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3851], | 30.00th=[ 4752], 40.00th=[ 5145], 50.00th=[ 5407], 60.00th=[ 5604], | 70.00th=[ 5866], 80.00th=[ 6194], 90.00th=[ 7767], 95.00th=[11469], | 99.00th=[15401], 99.50th=[16712], 99.90th=[21365], 99.95th=[56361], | 99.99th=[64750] bw ( KiB/s): min=124928, max=212992, per=100.00%, avg=153342.10, stdev=11990.87, samples=60 iops : min= 122, max= 208, avg=149.73, stdev=11.71, samples=60 lat (usec) : 250=78.29%, 500=1.08%, 750=0.05% lat (msec) : 2=0.01%, 4=4.25%, 10=14.57%, 20=1.67%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.82%, sys=2.38%, ctx=22235, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17729,4493,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=591MiB/s (620MB/s), 591MiB/s-591MiB/s (620MB/s-620MB/s), io=17.3GiB (18.6GB), run=30003-30003msec WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=4493MiB (4711MB), run=30003-30003msec Disk stats (read/write): vdf: ios=17586/4457, merge=0/0, ticks=1146/22675, in_queue=28369, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966047: Fri Jan 7 14:28:15 2022 read: IOPS=568, BW=568MiB/s (596MB/s)(17.1GiB/30762msec) slat (usec): min=10, max=909247, avg=1324.56, stdev=14860.11 clat (msec): min=18, max=979, avg=116.03, stdev=120.49 lat (msec): min=18, max=997, avg=117.36, stdev=121.36 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 39], 20.00th=[ 54], | 30.00th=[ 71], 40.00th=[ 85], 50.00th=[ 96], 60.00th=[ 109], | 70.00th=[ 123], 80.00th=[ 140], 90.00th=[ 171], 95.00th=[ 213], | 99.00th=[ 844], 99.50th=[ 877], 99.90th=[ 969], 99.95th=[ 978], | 99.99th=[ 978] bw ( KiB/s): min=34816, max=1272304, per=100.00%, avg=594314.02, stdev=229587.17, samples=60 iops : min= 34, max= 1242, avg=580.37, stdev=224.19, samples=60 write: IOPS=144, BW=144MiB/s (151MB/s)(4434MiB/30762msec) slat (usec): min=64, max=130621, avg=1532.99, stdev=6452.58 clat (msec): min=39, max=2646, avg=922.55, stdev=342.58 lat (msec): min=40, max=2647, avg=924.08, stdev=342.38 clat percentiles (msec): | 1.00th=[ 169], 5.00th=[ 418], 10.00th=[ 550], 20.00th=[ 684], | 30.00th=[ 760], 40.00th=[ 818], 50.00th=[ 877], 60.00th=[ 944], | 70.00th=[ 1011], 80.00th=[ 1116], 90.00th=[ 1368], 95.00th=[ 1603], | 99.00th=[ 2072], 99.50th=[ 2140], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2635] bw ( KiB/s): min=10240, max=311296, per=99.48%, avg=146823.78, stdev=62349.95, samples=60 iops : min= 10, max= 304, avg=143.37, stdev=60.89, samples=60 lat (msec) : 20=0.01%, 50=13.77%, 100=28.69%, 250=34.99%, 500=2.00% lat (msec) : 750=4.25%, 1000=9.85%, 2000=6.18%, >=2000=0.26% cpu : usr=4.97%, sys=1.33%, ctx=2067, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=17475,4434,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=568MiB/s (596MB/s), 568MiB/s-568MiB/s (596MB/s-596MB/s), io=17.1GiB (18.3GB), run=30762-30762msec WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=4434MiB (4649MB), run=30762-30762msec Disk stats (read/write): vdf: ios=17473/4397, merge=0/0, ticks=114561/3582448, in_queue=3738199, util=99.36% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966905: Fri Jan 7 14:28:51 2022 read: IOPS=277, BW=556MiB/s (583MB/s)(16.3GiB/30004msec) clat (usec): min=246, max=18584, avg=455.96, stdev=1059.56 lat (usec): min=246, max=18585, avg=456.14, stdev=1059.57 clat percentiles (usec): | 1.00th=[ 253], 5.00th=[ 260], 10.00th=[ 265], 20.00th=[ 269], | 30.00th=[ 277], 40.00th=[ 281], 50.00th=[ 289], 60.00th=[ 306], | 70.00th=[ 334], 80.00th=[ 553], 90.00th=[ 693], 95.00th=[ 725], | 99.00th=[ 857], 99.50th=[10945], 99.90th=[16057], 99.95th=[16712], | 99.99th=[18482] bw ( KiB/s): min=278528, max=856064, per=100.00%, avg=569070.93, stdev=117453.59, samples=60 iops : min= 136, max= 418, avg=277.87, stdev=57.35, samples=60 write: IOPS=70, BW=141MiB/s (148MB/s)(4234MiB/30004msec) clat (usec): min=5202, max=59841, avg=12090.06, stdev=3077.14 lat (usec): min=5705, max=60031, avg=12365.50, stdev=3092.97 clat percentiles (usec): | 1.00th=[ 6980], 5.00th=[ 8586], 10.00th=[ 8979], 20.00th=[ 9896], | 30.00th=[10552], 40.00th=[11207], 50.00th=[11731], 60.00th=[12125], | 70.00th=[12911], 80.00th=[13960], 90.00th=[15795], 95.00th=[16909], | 99.00th=[20841], 99.50th=[24249], 99.90th=[33162], 99.95th=[50070], | 99.99th=[60031] bw ( KiB/s): min=126976, max=163840, per=99.97%, avg=144452.27, stdev=7795.97, samples=60 iops : min= 62, max= 80, avg=70.53, stdev= 3.81, samples=60 lat (usec) : 250=0.21%, 500=62.77%, 750=14.15%, 1000=2.04% lat (msec) : 2=0.03%, 10=4.63%, 20=15.95%, 50=0.21%, 100=0.02% cpu : usr=2.17%, sys=2.02%, ctx=10464, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8336,2117,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=556MiB/s (583MB/s), 556MiB/s-556MiB/s (583MB/s-583MB/s), io=16.3GiB (17.5GB), run=30004-30004msec WRITE: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=4234MiB (4440MB), run=30004-30004msec Disk stats (read/write): vdf: ios=16620/4218, merge=0/0, ticks=737/46302, in_queue=54183, util=95.38% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967501: Fri Jan 7 14:29:28 2022 read: IOPS=286, BW=573MiB/s (601MB/s)(16.8GiB/30006msec) slat (usec): min=53, max=381, avg=74.37, stdev=13.65 clat (usec): min=143, max=18665, avg=363.11, stdev=1065.08 lat (usec): min=251, max=18732, avg=437.71, stdev=1065.44 clat percentiles (usec): | 1.00th=[ 200], 5.00th=[ 204], 10.00th=[ 206], 20.00th=[ 210], | 30.00th=[ 215], 40.00th=[ 219], 50.00th=[ 225], 60.00th=[ 233], | 70.00th=[ 260], 80.00th=[ 318], 90.00th=[ 502], 95.00th=[ 594], | 99.00th=[ 775], 99.50th=[10814], 99.90th=[16319], 99.95th=[16581], | 99.99th=[18744] bw ( KiB/s): min=303104, max=929792, per=100.00%, avg=586568.95, stdev=117259.79, samples=60 iops : min= 148, max= 454, avg=286.40, stdev=57.25, samples=60 write: IOPS=72, BW=145MiB/s (152MB/s)(4358MiB/30006msec) slat (usec): min=139, max=1043, avg=298.14, stdev=111.70 clat (usec): min=4643, max=47583, avg=11735.51, stdev=3040.56 lat (usec): min=4876, max=47852, avg=12033.94, stdev=3054.21 clat percentiles (usec): | 1.00th=[ 5866], 5.00th=[ 7373], 10.00th=[ 8356], 20.00th=[ 9503], | 30.00th=[10290], 40.00th=[10945], 50.00th=[11469], 60.00th=[11994], | 70.00th=[12649], 80.00th=[13829], 90.00th=[15664], 95.00th=[16909], | 99.00th=[18744], 99.50th=[21627], 99.90th=[31065], 99.95th=[44303], | 99.99th=[47449] bw ( KiB/s): min=131072, max=163840, per=99.98%, avg=148689.72, stdev=6629.57, samples=60 iops : min= 64, max= 80, avg=72.60, stdev= 3.22, samples=60 lat (usec) : 250=53.97%, 500=17.76%, 750=7.11%, 1000=0.39% lat (msec) : 2=0.01%, 4=0.01%, 10=5.48%, 20=15.13%, 50=0.14% cpu : usr=2.00%, sys=2.84%, ctx=10784, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8592,2179,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=573MiB/s (601MB/s), 573MiB/s-573MiB/s (601MB/s-601MB/s), io=16.8GiB (18.0GB), run=30006-30006msec WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=4358MiB (4570MB), run=30006-30006msec Disk stats (read/write): vdf: ios=17086/4338, merge=0/0, ticks=776/48718, in_queue=56494, util=95.77% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968292: Fri Jan 7 14:30:05 2022 read: IOPS=286, BW=573MiB/s (601MB/s)(16.8GiB/30001msec) clat (usec): min=253, max=51981, avg=427.43, stdev=1186.09 lat (usec): min=253, max=51981, avg=427.60, stdev=1186.10 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 273], 20.00th=[ 277], | 30.00th=[ 285], 40.00th=[ 289], 50.00th=[ 293], 60.00th=[ 302], | 70.00th=[ 326], 80.00th=[ 379], 90.00th=[ 498], 95.00th=[ 627], | 99.00th=[ 824], 99.50th=[10683], 99.90th=[16057], 99.95th=[17433], | 99.99th=[52167] bw ( KiB/s): min=278528, max=999424, per=100.00%, avg=586587.00, stdev=126447.49, samples=60 iops : min= 136, max= 488, avg=286.40, stdev=61.74, samples=60 write: IOPS=72, BW=145MiB/s (152MB/s)(4358MiB/30001msec) clat (usec): min=4702, max=53453, avg=11848.61, stdev=3057.90 lat (usec): min=4872, max=53652, avg=12072.29, stdev=3066.35 clat percentiles (usec): | 1.00th=[ 5997], 5.00th=[ 7570], 10.00th=[ 8455], 20.00th=[ 9634], | 30.00th=[10421], 40.00th=[11076], 50.00th=[11600], 60.00th=[12125], | 70.00th=[12780], 80.00th=[13829], 90.00th=[15664], 95.00th=[16909], | 99.00th=[18744], 99.50th=[20055], 99.90th=[33817], 99.95th=[52691], | 99.99th=[53216] bw ( KiB/s): min=131072, max=163840, per=99.96%, avg=148694.78, stdev=6760.32, samples=60 iops : min= 64, max= 80, avg=72.60, stdev= 3.30, samples=60 lat (usec) : 500=71.90%, 750=6.41%, 1000=0.91% lat (msec) : 2=0.03%, 4=0.01%, 10=5.15%, 20=15.50%, 50=0.07% lat (msec) : 100=0.03% cpu : usr=1.77%, sys=1.74%, ctx=10776, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8592,2179,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=573MiB/s (601MB/s), 573MiB/s-573MiB/s (601MB/s-601MB/s), io=16.8GiB (18.0GB), run=30001-30001msec WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=4358MiB (4570MB), run=30001-30001msec Disk stats (read/write): vdf: ios=17080/4338, merge=0/0, ticks=810/48941, in_queue=56762, util=96.16% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969563: Fri Jan 7 14:30:42 2022 read: IOPS=339, BW=680MiB/s (713MB/s)(19.0GiB/30102msec) slat (usec): min=21, max=207, avg=36.87, stdev=14.97 clat (usec): min=132, max=1569.4k, avg=1654.45, stdev=30618.28 lat (usec): min=206, max=1569.4k, avg=1691.53, stdev=30618.33 clat percentiles (usec): | 1.00th=[ 184], 5.00th=[ 188], 10.00th=[ 190], | 20.00th=[ 194], 30.00th=[ 198], 40.00th=[ 202], | 50.00th=[ 208], 60.00th=[ 217], 70.00th=[ 243], | 80.00th=[ 619], 90.00th=[ 652], 95.00th=[ 685], | 99.00th=[ 889], 99.50th=[ 2573], 99.90th=[ 480248], | 99.95th=[ 658506], 99.99th=[1182794] bw ( KiB/s): min=401408, max=1191936, per=100.00%, avg=698325.03, stdev=138750.42, samples=60 iops : min= 196, max= 582, avg=340.97, stdev=67.74, samples=60 write: IOPS=86, BW=173MiB/s (181MB/s)(5206MiB/30102msec) slat (usec): min=120, max=965, avg=344.85, stdev=181.19 clat (msec): min=10, max=1893, avg=108.60, stdev=125.09 lat (msec): min=10, max=1893, avg=108.94, stdev=125.09 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 21], 10.00th=[ 23], 20.00th=[ 32], | 30.00th=[ 37], 40.00th=[ 48], 50.00th=[ 65], 60.00th=[ 86], | 70.00th=[ 113], 80.00th=[ 159], 90.00th=[ 257], 95.00th=[ 334], | 99.00th=[ 617], 99.50th=[ 760], 99.90th=[ 969], 99.95th=[ 1301], | 99.99th=[ 1888] bw ( KiB/s): min=155648, max=184320, per=100.00%, avg=177089.75, stdev=6006.53, samples=60 iops : min= 76, max= 90, avg=86.47, stdev= 2.93, samples=60 lat (usec) : 250=56.71%, 500=5.79%, 750=15.63%, 1000=0.97% lat (msec) : 2=0.14%, 4=0.11%, 10=0.10%, 20=0.92%, 50=7.39% lat (msec) : 100=5.00%, 250=5.01%, 500=1.80%, 750=0.25%, 1000=0.14% lat (msec) : 2000=0.03% cpu : usr=3.10%, sys=1.78%, ctx=12807, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10230,2603,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=680MiB/s (713MB/s), 680MiB/s-680MiB/s (713MB/s-713MB/s), io=19.0GiB (21.5GB), run=30102-30102msec WRITE: bw=173MiB/s (181MB/s), 173MiB/s-173MiB/s (181MB/s-181MB/s), io=5206MiB (5459MB), run=30102-30102msec Disk stats (read/write): vdf: ios=20460/5204, merge=0/0, ticks=13807/561523, in_queue=584493, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970153: Fri Jan 7 14:31:19 2022 read: IOPS=288, BW=577MiB/s (605MB/s)(16.9GiB/30013msec) clat (usec): min=218, max=18876, avg=371.88, stdev=1087.09 lat (usec): min=218, max=18877, avg=372.06, stdev=1087.09 clat percentiles (usec): | 1.00th=[ 227], 5.00th=[ 233], 10.00th=[ 237], 20.00th=[ 241], | 30.00th=[ 245], 40.00th=[ 249], 50.00th=[ 253], 60.00th=[ 262], | 70.00th=[ 281], 80.00th=[ 330], 90.00th=[ 375], 95.00th=[ 441], | 99.00th=[ 742], 99.50th=[11207], 99.90th=[16581], 99.95th=[16909], | 99.99th=[19006] bw ( KiB/s): min=286720, max=958464, per=100.00%, avg=590748.80, stdev=126087.16, samples=60 iops : min= 140, max= 468, avg=288.43, stdev=61.57, samples=60 write: IOPS=73, BW=147MiB/s (154MB/s)(4400MiB/30013msec) clat (usec): min=4684, max=46658, avg=11961.40, stdev=2933.72 lat (usec): min=4849, max=46878, avg=12169.50, stdev=2934.61 clat percentiles (usec): | 1.00th=[ 6194], 5.00th=[ 7832], 10.00th=[ 8717], 20.00th=[ 9765], | 30.00th=[10552], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256], | 70.00th=[12780], 80.00th=[13829], 90.00th=[15795], 95.00th=[16909], | 99.00th=[18744], 99.50th=[19530], 99.90th=[40109], 99.95th=[44827], | 99.99th=[46400] bw ( KiB/s): min=135168, max=163840, per=100.00%, avg=150128.63, stdev=6352.61, samples=60 iops : min= 66, max= 80, avg=73.30, stdev= 3.10, samples=60 lat (usec) : 250=34.96%, 500=42.05%, 750=1.96%, 1000=0.22% lat (msec) : 4=0.01%, 10=4.66%, 20=16.04%, 50=0.09% cpu : usr=1.71%, sys=1.36%, ctx=10867, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8653,2200,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=577MiB/s (605MB/s), 577MiB/s-577MiB/s (605MB/s-605MB/s), io=16.9GiB (18.1GB), run=30013-30013msec WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=4400MiB (4614MB), run=30013-30013msec Disk stats (read/write): vdf: ios=17238/4376, merge=0/0, ticks=792/42656, in_queue=49699, util=96.33% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971089: Fri Jan 7 14:31:56 2022 read: IOPS=339, BW=680MiB/s (713MB/s)(20.3GiB/30546msec) slat (usec): min=19, max=1977, avg=33.61, stdev=26.06 clat (usec): min=90, max=2553.3k, avg=7222.34, stdev=92372.12 lat (usec): min=206, max=2553.3k, avg=7256.16, stdev=92371.99 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 192], 10.00th=[ 194], | 20.00th=[ 200], 30.00th=[ 204], 40.00th=[ 208], | 50.00th=[ 217], 60.00th=[ 245], 70.00th=[ 627], | 80.00th=[ 660], 90.00th=[ 1532], 95.00th=[ 9372], | 99.00th=[ 24511], 99.50th=[ 371196], 99.90th=[1501561], | 99.95th=[2332034], 99.99th=[2533360] bw ( KiB/s): min= 4096, max=1183744, per=100.00%, avg=697103.82, stdev=182065.82, samples=61 iops : min= 2, max= 578, avg=340.36, stdev=88.90, samples=61 write: IOPS=86, BW=174MiB/s (182MB/s)(5302MiB/30546msec) slat (usec): min=122, max=5511, avg=388.09, stdev=225.42 clat (msec): min=56, max=3297, avg=547.01, stdev=365.59 lat (msec): min=57, max=3297, avg=547.39, stdev=365.61 clat percentiles (msec): | 1.00th=[ 155], 5.00th=[ 268], 10.00th=[ 305], 20.00th=[ 359], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 485], | 70.00th=[ 550], 80.00th=[ 659], 90.00th=[ 835], 95.00th=[ 1062], | 99.00th=[ 2500], 99.50th=[ 2668], 99.90th=[ 2937], 99.95th=[ 3037], | 99.99th=[ 3306] bw ( KiB/s): min=49152, max=225280, per=100.00%, avg=177779.67, stdev=23393.76, samples=60 iops : min= 24, max= 110, avg=86.80, stdev=11.42, samples=60 lat (usec) : 100=0.02%, 250=48.25%, 500=6.50%, 750=14.12%, 1000=1.72% lat (msec) : 2=1.54%, 4=1.50%, 10=2.22%, 20=2.51%, 50=0.73% lat (msec) : 100=0.11%, 250=0.70%, 500=12.07%, 750=5.01%, 1000=1.67% lat (msec) : 2000=0.91%, >=2000=0.44% cpu : usr=3.40%, sys=1.70%, ctx=11561, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=10383,2651,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=680MiB/s (713MB/s), 680MiB/s-680MiB/s (713MB/s-713MB/s), io=20.3GiB (21.8GB), run=30546-30546msec WRITE: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=5302MiB (5560MB), run=30546-30546msec Disk stats (read/write): vdf: ios=20765/5222, merge=0/0, ticks=78185/2822051, in_queue=2961830, util=99.56% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972155: Fri Jan 7 14:32:33 2022 read: IOPS=285, BW=572MiB/s (600MB/s)(16.8GiB/30001msec) clat (usec): min=260, max=18839, avg=414.87, stdev=1073.10 lat (usec): min=261, max=18839, avg=415.04, stdev=1073.10 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 277], 10.00th=[ 281], 20.00th=[ 285], | 30.00th=[ 289], 40.00th=[ 293], 50.00th=[ 302], 60.00th=[ 310], | 70.00th=[ 330], 80.00th=[ 371], 90.00th=[ 408], 95.00th=[ 478], | 99.00th=[ 799], 99.50th=[11207], 99.90th=[16188], 99.95th=[16909], | 99.99th=[18744] bw ( KiB/s): min=282624, max=929792, per=100.00%, avg=585648.77, stdev=120068.32, samples=60 iops : min= 138, max= 454, avg=285.93, stdev=58.63, samples=60 write: IOPS=72, BW=145MiB/s (152MB/s)(4352MiB/30001msec) clat (usec): min=4701, max=54277, avg=11932.49, stdev=3114.28 lat (usec): min=4908, max=54461, avg=12140.92, stdev=3113.76 clat percentiles (usec): | 1.00th=[ 5932], 5.00th=[ 7570], 10.00th=[ 8586], 20.00th=[ 9634], | 30.00th=[10421], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256], | 70.00th=[12780], 80.00th=[14091], 90.00th=[15926], 95.00th=[16909], | 99.00th=[18744], 99.50th=[20841], 99.90th=[33817], 99.95th=[48497], | 99.99th=[54264] bw ( KiB/s): min=135168, max=163840, per=100.00%, avg=148562.88, stdev=6818.75, samples=60 iops : min= 66, max= 80, avg=72.53, stdev= 3.33, samples=60 lat (usec) : 500=76.35%, 750=2.43%, 1000=0.44% lat (msec) : 2=0.03%, 10=5.05%, 20=15.58%, 50=0.12%, 100=0.01% cpu : usr=1.63%, sys=2.08%, ctx=10767, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8579,2176,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=572MiB/s (600MB/s), 572MiB/s-572MiB/s (600MB/s-600MB/s), io=16.8GiB (17.0GB), run=30001-30001msec WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=4352MiB (4563MB), run=30001-30001msec Disk stats (read/write): vdf: ios=17038/4318, merge=0/0, ticks=775/49062, in_queue=56439, util=95.57% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972500: Fri Jan 7 14:33:10 2022 read: IOPS=345, BW=691MiB/s (725MB/s)(20.7GiB/30687msec) slat (usec): min=22, max=565083, avg=2147.98, stdev=11773.71 clat (msec): min=18, max=3195, avg=93.10, stdev=115.30 lat (msec): min=18, max=3195, avg=95.25, stdev=116.04 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 36], 20.00th=[ 48], | 30.00th=[ 57], 40.00th=[ 68], 50.00th=[ 77], 60.00th=[ 86], | 70.00th=[ 96], 80.00th=[ 110], 90.00th=[ 131], 95.00th=[ 161], | 99.00th=[ 558], 99.50th=[ 701], 99.90th=[ 1620], 99.95th=[ 2500], | 99.99th=[ 3171] bw ( KiB/s): min= 4096, max=1343488, per=100.00%, avg=710172.43, stdev=265462.55, samples=61 iops : min= 2, max= 656, avg=346.75, stdev=129.62, samples=61 write: IOPS=87, BW=176MiB/s (184MB/s)(5394MiB/30687msec) slat (usec): min=558, max=483717, avg=2660.42, stdev=10844.53 clat (msec): min=78, max=3362, avg=759.58, stdev=347.07 lat (msec): min=79, max=3363, avg=762.25, stdev=346.86 clat percentiles (msec): | 1.00th=[ 313], 5.00th=[ 409], 10.00th=[ 481], 20.00th=[ 550], | 30.00th=[ 600], 40.00th=[ 625], 50.00th=[ 659], 60.00th=[ 709], | 70.00th=[ 785], 80.00th=[ 927], 90.00th=[ 1133], 95.00th=[ 1368], | 99.00th=[ 2299], 99.50th=[ 2601], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3373] bw ( KiB/s): min=12288, max=319488, per=99.76%, avg=179560.73, stdev=58939.49, samples=60 iops : min= 6, max= 156, avg=87.67, stdev=28.78, samples=60 lat (msec) : 20=0.01%, 50=18.12%, 100=40.42%, 250=19.12%, 500=3.22% lat (msec) : 750=12.06%, 1000=3.65%, 2000=3.01%, >=2000=0.39% cpu : usr=6.04%, sys=1.59%, ctx=2662, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10607,2697,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=691MiB/s (725MB/s), 691MiB/s-691MiB/s (725MB/s-725MB/s), io=20.7GiB (22.2GB), run=30687-30687msec WRITE: bw=176MiB/s (184MB/s), 176MiB/s-176MiB/s (184MB/s-184MB/s), io=5394MiB (5656MB), run=30687-30687msec Disk stats (read/write): vdf: ios=21212/5370, merge=0/0, ticks=127433/3597032, in_queue=3758754, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973641: Fri Jan 7 14:33:47 2022 read: IOPS=286, BW=574MiB/s (602MB/s)(16.8GiB/30001msec) clat (usec): min=232, max=18977, avg=392.67, stdev=1073.04 lat (usec): min=232, max=18978, avg=392.85, stdev=1073.04 clat percentiles (usec): | 1.00th=[ 239], 5.00th=[ 247], 10.00th=[ 251], 20.00th=[ 258], | 30.00th=[ 262], 40.00th=[ 269], 50.00th=[ 277], 60.00th=[ 285], | 70.00th=[ 310], 80.00th=[ 351], 90.00th=[ 404], 95.00th=[ 486], | 99.00th=[ 742], 99.50th=[10814], 99.90th=[16581], 99.95th=[17171], | 99.99th=[19006] bw ( KiB/s): min=303104, max=917504, per=100.00%, avg=587411.56, stdev=123238.56, samples=59 iops : min= 148, max= 448, avg=286.81, stdev=60.18, samples=59 write: IOPS=72, BW=145MiB/s (153MB/s)(4364MiB/30001msec) clat (usec): min=4670, max=51531, avg=11978.78, stdev=3086.52 lat (usec): min=4817, max=51766, avg=12189.84, stdev=3087.76 clat percentiles (usec): | 1.00th=[ 5932], 5.00th=[ 7635], 10.00th=[ 8586], 20.00th=[ 9765], | 30.00th=[10552], 40.00th=[11207], 50.00th=[11731], 60.00th=[12256], | 70.00th=[12911], 80.00th=[14091], 90.00th=[15926], 95.00th=[17171], | 99.00th=[19006], 99.50th=[20579], 99.90th=[39584], 99.95th=[45876], | 99.99th=[51643] bw ( KiB/s): min=135168, max=163840, per=100.00%, avg=148988.32, stdev=6266.32, samples=59 iops : min= 66, max= 80, avg=72.75, stdev= 3.06, samples=59 lat (usec) : 250=7.17%, 500=68.98%, 750=2.85%, 1000=0.21% lat (msec) : 2=0.03%, 4=0.01%, 10=4.71%, 20=15.93%, 50=0.11% lat (msec) : 100=0.01% cpu : usr=1.78%, sys=1.74%, ctx=10789, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8605,2182,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=574MiB/s (602MB/s), 574MiB/s-574MiB/s (602MB/s-602MB/s), io=16.8GiB (18.0GB), run=30001-30001msec WRITE: bw=145MiB/s (153MB/s), 145MiB/s-145MiB/s (153MB/s-153MB/s), io=4364MiB (4576MB), run=30001-30001msec Disk stats (read/write): vdf: ios=17154/4350, merge=0/0, ticks=795/43836, in_queue=51042, util=95.48% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974263: Fri Jan 7 14:34:25 2022 read: IOPS=342, BW=686MiB/s (719MB/s)(20.6GiB/30716msec) slat (usec): min=22, max=530169, avg=2166.30, stdev=11817.63 clat (msec): min=84, max=3304, avg=330.74, stdev=164.47 lat (msec): min=84, max=3304, avg=332.90, stdev=164.79 clat percentiles (msec): | 1.00th=[ 105], 5.00th=[ 148], 10.00th=[ 186], 20.00th=[ 239], | 30.00th=[ 271], 40.00th=[ 292], 50.00th=[ 309], 60.00th=[ 326], | 70.00th=[ 347], 80.00th=[ 376], 90.00th=[ 472], 95.00th=[ 651], | 99.00th=[ 877], 99.50th=[ 936], 99.90th=[ 2366], 99.95th=[ 2668], | 99.99th=[ 3104] bw ( KiB/s): min= 4096, max=1167360, per=99.58%, avg=699095.92, stdev=254215.63, samples=61 iops : min= 2, max= 570, avg=341.34, stdev=124.13, samples=61 write: IOPS=87, BW=175MiB/s (183MB/s)(5360MiB/30716msec) slat (usec): min=542, max=490769, avg=2671.94, stdev=11317.09 clat (msec): min=345, max=3964, avg=977.46, stdev=359.71 lat (msec): min=346, max=3965, avg=980.13, stdev=359.59 clat percentiles (msec): | 1.00th=[ 523], 5.00th=[ 634], 10.00th=[ 684], 20.00th=[ 751], | 30.00th=[ 810], 40.00th=[ 852], 50.00th=[ 885], 60.00th=[ 944], | 70.00th=[ 1020], 80.00th=[ 1133], 90.00th=[ 1334], 95.00th=[ 1552], | 99.00th=[ 2635], 99.50th=[ 3171], 99.90th=[ 3876], 99.95th=[ 3876], | 99.99th=[ 3977] bw ( KiB/s): min=12288, max=323584, per=100.00%, avg=180438.24, stdev=69275.23, samples=59 iops : min= 6, max= 158, avg=88.10, stdev=33.83, samples=59 lat (msec) : 100=0.59%, 250=17.40%, 500=54.79%, 750=8.61%, 1000=11.77% lat (msec) : 2000=6.39%, >=2000=0.45% cpu : usr=6.04%, sys=1.66%, ctx=2636, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10529,2680,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=686MiB/s (719MB/s), 686MiB/s-686MiB/s (719MB/s-719MB/s), io=20.6GiB (22.1GB), run=30716-30716msec WRITE: bw=175MiB/s (183MB/s), 175MiB/s-175MiB/s (183MB/s-183MB/s), io=5360MiB (5620MB), run=30716-30716msec Disk stats (read/write): vdf: ios=21057/5352, merge=0/0, ticks=135354/3594726, in_queue=3753000, util=99.02% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975107: Fri Jan 7 14:35:01 2022 read: IOPS=164, BW=659MiB/s (691MB/s)(19.3GiB/30007msec) clat (usec): min=424, max=22547, avg=877.13, stdev=1588.61 lat (usec): min=424, max=22547, avg=877.34, stdev=1588.62 clat percentiles (usec): | 1.00th=[ 437], 5.00th=[ 449], 10.00th=[ 453], 20.00th=[ 461], | 30.00th=[ 474], 40.00th=[ 486], 50.00th=[ 498], 60.00th=[ 523], | 70.00th=[ 627], 80.00th=[ 1336], 90.00th=[ 1401], 95.00th=[ 1467], | 99.00th=[11076], 99.50th=[14615], 99.90th=[20579], 99.95th=[20841], | 99.99th=[22676] bw ( KiB/s): min=303104, max=1073152, per=100.00%, avg=674970.37, stdev=156986.93, samples=60 iops : min= 74, max= 262, avg=164.75, stdev=38.28, samples=60 write: IOPS=42, BW=170MiB/s (178MB/s)(5096MiB/30007msec) clat (usec): min=11801, max=57948, avg=19545.21, stdev=3544.53 lat (usec): min=12171, max=58388, avg=20136.30, stdev=3602.86 clat percentiles (usec): | 1.00th=[13829], 5.00th=[14877], 10.00th=[15664], 20.00th=[16712], | 30.00th=[17433], 40.00th=[18220], 50.00th=[19006], 60.00th=[19792], | 70.00th=[21103], 80.00th=[22152], 90.00th=[23987], 95.00th=[25035], | 99.00th=[30016], 99.50th=[31327], 99.90th=[40109], 99.95th=[57934], | 99.99th=[57934] bw ( KiB/s): min=147456, max=196608, per=99.94%, avg=173801.77, stdev=9794.84, samples=60 iops : min= 36, max= 48, avg=42.42, stdev= 2.40, samples=60 lat (usec) : 500=40.87%, 750=19.31%, 1000=0.95% lat (msec) : 2=17.16%, 4=0.18%, 10=0.16%, 20=13.27%, 50=8.09% lat (msec) : 100=0.02% cpu : usr=2.68%, sys=2.13%, ctx=6225, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4944,1274,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=659MiB/s (691MB/s), 659MiB/s-659MiB/s (691MB/s-691MB/s), io=19.3GiB (20.7GB), run=30007-30007msec WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=5096MiB (5344MB), run=30007-30007msec Disk stats (read/write): vdf: ios=19716/5072, merge=0/0, ticks=5281/82622, in_queue=98320, util=96.19% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976333: Fri Jan 7 14:35:38 2022 read: IOPS=167, BW=671MiB/s (704MB/s)(19.7GiB/30002msec) slat (usec): min=72, max=260, avg=94.40, stdev=18.63 clat (usec): min=247, max=22537, avg=719.93, stdev=1597.23 lat (usec): min=398, max=22619, avg=814.61, stdev=1597.98 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 330], 10.00th=[ 338], 20.00th=[ 343], | 30.00th=[ 351], 40.00th=[ 359], 50.00th=[ 363], 60.00th=[ 375], | 70.00th=[ 412], 80.00th=[ 1037], 90.00th=[ 1221], 95.00th=[ 1254], | 99.00th=[10814], 99.50th=[14484], 99.90th=[20317], 99.95th=[21627], | 99.99th=[22414] bw ( KiB/s): min=360448, max=1081344, per=100.00%, avg=687383.18, stdev=152343.68, samples=60 iops : min= 88, max= 264, avg=167.80, stdev=37.19, samples=60 write: IOPS=43, BW=175MiB/s (183MB/s)(5244MiB/30002msec) slat (usec): min=253, max=1768, avg=639.16, stdev=313.76 clat (usec): min=11949, max=64256, avg=19105.65, stdev=3946.88 lat (usec): min=12292, max=65525, avg=19745.19, stdev=4022.99 clat percentiles (usec): | 1.00th=[13304], 5.00th=[14484], 10.00th=[15270], 20.00th=[16188], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18482], 60.00th=[19268], | 70.00th=[20317], 80.00th=[21627], 90.00th=[23462], 95.00th=[25297], | 99.00th=[28443], 99.50th=[34866], 99.90th=[58459], 99.95th=[64226], | 99.99th=[64226] bw ( KiB/s): min=147456, max=196608, per=99.94%, avg=178876.70, stdev=10026.93, samples=60 iops : min= 36, max= 48, avg=43.67, stdev= 2.45, samples=60 lat (usec) : 250=0.02%, 500=59.15%, 750=1.86%, 1000=1.94% lat (msec) : 2=15.30%, 10=0.20%, 20=14.61%, 50=6.87%, 100=0.05% cpu : usr=2.66%, sys=2.03%, ctx=6353, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5034,1311,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=671MiB/s (704MB/s), 671MiB/s-671MiB/s (704MB/s-704MB/s), io=19.7GiB (21.1GB), run=30002-30002msec WRITE: bw=175MiB/s (183MB/s), 175MiB/s-175MiB/s (183MB/s-183MB/s), io=5244MiB (5499MB), run=30002-30002msec Disk stats (read/write): vdf: ios=20108/5217, merge=0/0, ticks=4995/81058, in_queue=95993, util=96.41% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977093: Fri Jan 7 14:36:15 2022 read: IOPS=167, BW=669MiB/s (702MB/s)(19.6GiB/30017msec) clat (usec): min=393, max=41457, avg=793.39, stdev=1649.49 lat (usec): min=393, max=41457, avg=793.58, stdev=1649.50 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 408], 10.00th=[ 412], 20.00th=[ 420], | 30.00th=[ 424], 40.00th=[ 433], 50.00th=[ 441], 60.00th=[ 457], | 70.00th=[ 498], 80.00th=[ 1045], 90.00th=[ 1303], 95.00th=[ 1352], | 99.00th=[10683], 99.50th=[14484], 99.90th=[18482], 99.95th=[20579], | 99.99th=[41681] bw ( KiB/s): min=401408, max=1064960, per=100.00%, avg=685716.08, stdev=166315.17, samples=60 iops : min= 98, max= 260, avg=167.40, stdev=40.60, samples=60 write: IOPS=43, BW=174MiB/s (182MB/s)(5212MiB/30017msec) clat (usec): min=11858, max=80678, avg=19401.13, stdev=4586.38 lat (usec): min=12231, max=81116, avg=19967.32, stdev=4629.27 clat percentiles (usec): | 1.00th=[13042], 5.00th=[14615], 10.00th=[15401], 20.00th=[16319], | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[19530], | 70.00th=[20579], 80.00th=[21890], 90.00th=[23987], 95.00th=[25560], | 99.00th=[27919], 99.50th=[46924], 99.90th=[80217], 99.95th=[80217], | 99.99th=[80217] bw ( KiB/s): min=131072, max=196608, per=99.99%, avg=177778.70, stdev=13772.00, samples=60 iops : min= 32, max= 48, avg=43.40, stdev= 3.36, samples=60 lat (usec) : 500=55.78%, 750=5.30%, 1000=1.96% lat (msec) : 2=15.29%, 4=0.03%, 10=0.19%, 20=14.01%, 50=7.35% lat (msec) : 100=0.09% cpu : usr=2.58%, sys=1.65%, ctx=6329, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5022,1303,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=669MiB/s (702MB/s), 669MiB/s-669MiB/s (702MB/s-702MB/s), io=19.6GiB (21.1GB), run=30017-30017msec WRITE: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=5212MiB (5465MB), run=30017-30017msec Disk stats (read/write): vdf: ios=19936/5177, merge=0/0, ticks=4712/81279, in_queue=96280, util=96.84% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977961: Fri Jan 7 14:36:52 2022 read: IOPS=179, BW=716MiB/s (751MB/s)(21.1GiB/30189msec) slat (usec): min=41, max=357, avg=61.37, stdev=17.66 clat (usec): min=201, max=2497.5k, avg=7650.20, stdev=90230.12 lat (usec): min=374, max=2497.6k, avg=7711.84, stdev=90230.08 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 338], 10.00th=[ 343], | 20.00th=[ 351], 30.00th=[ 359], 40.00th=[ 367], | 50.00th=[ 375], 60.00th=[ 396], 70.00th=[ 478], | 80.00th=[ 1237], 90.00th=[ 1287], 95.00th=[ 1352], | 99.00th=[ 52691], 99.50th=[ 591397], 99.90th=[1400898], | 99.95th=[2071987], 99.99th=[2499806] bw ( KiB/s): min=327680, max=1228800, per=100.00%, avg=738285.52, stdev=191902.56, samples=60 iops : min= 80, max= 300, avg=180.23, stdev=46.85, samples=60 write: IOPS=47, BW=189MiB/s (198MB/s)(5708MiB/30189msec) slat (usec): min=244, max=1892, avg=792.22, stdev=393.30 clat (msec): min=16, max=2536, avg=181.39, stdev=201.62 lat (msec): min=16, max=2536, avg=182.18, stdev=201.61 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 41], 20.00th=[ 58], | 30.00th=[ 73], 40.00th=[ 88], 50.00th=[ 109], 60.00th=[ 140], | 70.00th=[ 192], 80.00th=[ 264], 90.00th=[ 401], 95.00th=[ 567], | 99.00th=[ 936], 99.50th=[ 1011], 99.90th=[ 2500], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=172032, max=205210, per=99.93%, avg=193480.85, stdev=7096.04, samples=60 iops : min= 42, max= 50, avg=47.23, stdev= 1.73, samples=60 lat (usec) : 250=0.01%, 500=56.00%, 750=2.33%, 1000=0.73% lat (msec) : 2=18.44%, 4=0.23%, 10=0.09%, 20=0.25%, 50=3.18% lat (msec) : 100=6.91%, 250=6.73%, 500=3.32%, 750=1.01%, 1000=0.47% lat (msec) : 2000=0.23%, >=2000=0.07% cpu : usr=3.75%, sys=1.45%, ctx=6752, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5407,1427,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=716MiB/s (751MB/s), 716MiB/s-716MiB/s (751MB/s-751MB/s), io=21.1GiB (22.7GB), run=30189-30189msec WRITE: bw=189MiB/s (198MB/s), 189MiB/s-189MiB/s (198MB/s-198MB/s), io=5708MiB (5985MB), run=30189-30189msec Disk stats (read/write): vdf: ios=21628/5693, merge=0/0, ticks=49784/1007083, in_queue=1070174, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978554: Fri Jan 7 14:37:29 2022 read: IOPS=167, BW=671MiB/s (703MB/s)(19.7GiB/30007msec) clat (usec): min=367, max=50074, avg=772.40, stdev=1735.17 lat (usec): min=368, max=50074, avg=772.62, stdev=1735.18 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 396], | 30.00th=[ 404], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 478], 80.00th=[ 963], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[10683], 99.50th=[14746], 99.90th=[20579], 99.95th=[22676], | 99.99th=[50070] bw ( KiB/s): min=344064, max=1212416, per=100.00%, avg=686950.42, stdev=172269.74, samples=60 iops : min= 84, max= 296, avg=167.68, stdev=42.06, samples=60 write: IOPS=43, BW=174MiB/s (183MB/s)(5228MiB/30007msec) clat (msec): min=11, max=103, avg=19.41, stdev= 4.71 lat (msec): min=11, max=104, avg=19.97, stdev= 4.76 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 21], 80.00th=[ 22], 90.00th=[ 24], 95.00th=[ 26], | 99.00th=[ 30], 99.50th=[ 33], 99.90th=[ 89], 99.95th=[ 105], | 99.99th=[ 105] bw ( KiB/s): min=139264, max=204800, per=99.95%, avg=178325.93, stdev=12938.03, samples=60 iops : min= 34, max= 50, avg=43.52, stdev= 3.20, samples=60 lat (usec) : 500=56.80%, 750=4.51%, 1000=2.62% lat (msec) : 2=14.34%, 4=0.03%, 10=0.21%, 20=14.07%, 50=7.34% lat (msec) : 100=0.06%, 250=0.02% cpu : usr=2.63%, sys=1.27%, ctx=6347, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5031,1307,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=671MiB/s (703MB/s), 671MiB/s-671MiB/s (703MB/s-703MB/s), io=19.7GiB (21.1GB), run=30007-30007msec WRITE: bw=174MiB/s (183MB/s), 174MiB/s-174MiB/s (183MB/s-183MB/s), io=5228MiB (5482MB), run=30007-30007msec Disk stats (read/write): vdf: ios=20124/5225, merge=0/0, ticks=4505/82321, in_queue=96738, util=96.25% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979522: Fri Jan 7 14:38:07 2022 read: IOPS=177, BW=710MiB/s (744MB/s)(21.2GiB/30661msec) slat (usec): min=40, max=626605, avg=4235.80, stdev=17775.29 clat (msec): min=15, max=2524, avg=87.30, stdev=110.99 lat (msec): min=15, max=2524, avg=91.54, stdev=112.53 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 40], | 30.00th=[ 47], 40.00th=[ 56], 50.00th=[ 67], 60.00th=[ 79], | 70.00th=[ 90], 80.00th=[ 106], 90.00th=[ 136], 95.00th=[ 171], | 99.00th=[ 634], 99.50th=[ 701], 99.90th=[ 1469], 99.95th=[ 1989], | 99.99th=[ 2534] bw ( KiB/s): min= 8192, max=1376456, per=100.00%, avg=731044.84, stdev=302754.28, samples=61 iops : min= 2, max= 336, avg=178.39, stdev=73.88, samples=61 write: IOPS=46, BW=187MiB/s (196MB/s)(5732MiB/30661msec) slat (usec): min=1103, max=56024, avg=4852.67, stdev=8820.17 clat (msec): min=191, max=3273, avg=716.40, stdev=344.25 lat (msec): min=211, max=3314, avg=721.26, stdev=344.29 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 409], 10.00th=[ 481], 20.00th=[ 542], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 634], 60.00th=[ 667], | 70.00th=[ 735], 80.00th=[ 835], 90.00th=[ 1020], 95.00th=[ 1217], | 99.00th=[ 2601], 99.50th=[ 3004], 99.90th=[ 3239], 99.95th=[ 3272], | 99.99th=[ 3272] bw ( KiB/s): min= 8192, max=352256, per=100.00%, avg=191644.50, stdev=68985.38, samples=60 iops : min= 2, max= 86, avg=46.68, stdev=16.83, samples=60 lat (msec) : 20=0.38%, 50=26.32%, 100=34.15%, 250=16.30%, 500=3.30% lat (msec) : 750=13.23%, 1000=3.96%, 2000=2.02%, >=2000=0.33% cpu : usr=6.22%, sys=1.62%, ctx=2527, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.5%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5439,1433,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=710MiB/s (744MB/s), 710MiB/s-710MiB/s (744MB/s-744MB/s), io=21.2GiB (22.8GB), run=30661-30661msec WRITE: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=5732MiB (6010MB), run=30661-30661msec Disk stats (read/write): vdf: ios=21815/5714, merge=0/0, ticks=143231/3572898, in_queue=3761580, util=99.30% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980486: Fri Jan 7 14:38:43 2022 read: IOPS=167, BW=671MiB/s (704MB/s)(19.7GiB/30004msec) clat (usec): min=382, max=24517, avg=771.33, stdev=1580.13 lat (usec): min=382, max=24517, avg=771.53, stdev=1580.14 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 400], 10.00th=[ 404], 20.00th=[ 408], | 30.00th=[ 416], 40.00th=[ 424], 50.00th=[ 433], 60.00th=[ 445], | 70.00th=[ 482], 80.00th=[ 988], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[10552], 99.50th=[14746], 99.90th=[18482], 99.95th=[20579], | 99.99th=[24511] bw ( KiB/s): min=360448, max=1122304, per=100.00%, avg=687172.27, stdev=173103.68, samples=60 iops : min= 88, max= 274, avg=167.77, stdev=42.26, samples=60 write: IOPS=43, BW=175MiB/s (183MB/s)(5240MiB/30004msec) clat (usec): min=12144, max=84384, avg=19377.03, stdev=4073.30 lat (usec): min=12550, max=85389, avg=19928.70, stdev=4141.99 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14746], 10.00th=[15533], 20.00th=[16450], | 30.00th=[17171], 40.00th=[17957], 50.00th=[19006], 60.00th=[19530], | 70.00th=[20579], 80.00th=[21890], 90.00th=[23987], 95.00th=[25560], | 99.00th=[27919], 99.50th=[31065], 99.90th=[63177], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=147456, max=204800, per=99.94%, avg=178722.13, stdev=11436.19, samples=60 iops : min= 36, max= 50, avg=43.63, stdev= 2.79, samples=60 lat (usec) : 500=56.71%, 750=4.70%, 1000=2.32% lat (msec) : 2=14.52%, 4=0.06%, 10=0.19%, 20=14.19%, 50=7.27% lat (msec) : 100=0.05% cpu : usr=2.54%, sys=1.84%, ctx=6357, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5033,1310,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=671MiB/s (704MB/s), 671MiB/s-671MiB/s (704MB/s-704MB/s), io=19.7GiB (21.1GB), run=30004-30004msec WRITE: bw=175MiB/s (183MB/s), 175MiB/s-175MiB/s (183MB/s-183MB/s), io=5240MiB (5495MB), run=30004-30004msec Disk stats (read/write): vdf: ios=20124/5224, merge=0/0, ticks=4452/81953, in_queue=96425, util=96.08% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980997: Fri Jan 7 14:39:21 2022 read: IOPS=177, BW=709MiB/s (743MB/s)(21.2GiB/30641msec) slat (usec): min=46, max=479572, avg=4116.49, stdev=15425.29 clat (msec): min=84, max=2829, avg=312.64, stdev=170.59 lat (msec): min=86, max=2829, avg=316.76, stdev=171.18 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 127], 10.00th=[ 169], 20.00th=[ 205], | 30.00th=[ 236], 40.00th=[ 268], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 376], 90.00th=[ 439], 95.00th=[ 617], | 99.00th=[ 760], 99.50th=[ 885], 99.90th=[ 2265], 99.95th=[ 2534], | 99.99th=[ 2836] bw ( KiB/s): min= 8192, max=1490944, per=99.38%, avg=721257.67, stdev=303908.85, samples=61 iops : min= 2, max= 364, avg=176.07, stdev=74.20, samples=61 write: IOPS=46, BW=187MiB/s (196MB/s)(5720MiB/30641msec) slat (usec): min=1138, max=513176, avg=5327.49, stdev=16574.62 clat (msec): min=311, max=3403, avg=931.16, stdev=330.46 lat (msec): min=313, max=3405, avg=936.49, stdev=330.52 clat percentiles (msec): | 1.00th=[ 460], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 827], 50.00th=[ 869], 60.00th=[ 919], | 70.00th=[ 978], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1418], | 99.00th=[ 2400], 99.50th=[ 3071], 99.90th=[ 3306], 99.95th=[ 3406], | 99.99th=[ 3406] bw ( KiB/s): min=16384, max=376832, per=100.00%, avg=193027.69, stdev=76653.38, samples=59 iops : min= 4, max= 92, avg=47.12, stdev=18.71, samples=59 lat (msec) : 100=1.24%, 250=25.89%, 500=46.33%, 750=9.59%, 1000=11.05% lat (msec) : 2000=5.44%, >=2000=0.45% cpu : usr=6.42%, sys=1.54%, ctx=2586, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5429,1430,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=709MiB/s (743MB/s), 709MiB/s-709MiB/s (743MB/s-743MB/s), io=21.2GiB (22.8GB), run=30641-30641msec WRITE: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=5720MiB (5998MB), run=30641-30641msec Disk stats (read/write): vdf: ios=21716/5673, merge=0/0, ticks=151662/3568388, in_queue=3760049, util=99.02% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981878: Fri Jan 7 14:39:57 2022 read: IOPS=167, BW=672MiB/s (704MB/s)(19.7GiB/30010msec) clat (usec): min=373, max=23012, avg=763.49, stdev=1566.27 lat (usec): min=373, max=23012, avg=763.68, stdev=1566.28 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 400], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 433], | 70.00th=[ 469], 80.00th=[ 979], 90.00th=[ 1237], 95.00th=[ 1303], | 99.00th=[10552], 99.50th=[14746], 99.90th=[17957], 99.95th=[20579], | 99.99th=[22938] bw ( KiB/s): min=311296, max=1220608, per=100.00%, avg=688013.35, stdev=170760.26, samples=60 iops : min= 76, max= 298, avg=167.97, stdev=41.69, samples=60 write: IOPS=43, BW=175MiB/s (184MB/s)(5260MiB/30010msec) clat (usec): min=11534, max=69222, avg=19334.26, stdev=3864.39 lat (usec): min=11951, max=70141, avg=19883.83, stdev=3921.45 clat percentiles (usec): | 1.00th=[13435], 5.00th=[14615], 10.00th=[15664], 20.00th=[16450], | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[19530], | 70.00th=[20579], 80.00th=[21890], 90.00th=[23725], 95.00th=[25297], | 99.00th=[28181], 99.50th=[35914], 99.90th=[60031], 99.95th=[69731], | 99.99th=[69731] bw ( KiB/s): min=155648, max=196608, per=99.96%, avg=179410.53, stdev=10079.83, samples=60 iops : min= 38, max= 48, avg=43.80, stdev= 2.46, samples=60 lat (usec) : 500=56.80%, 750=4.17%, 1000=2.75% lat (msec) : 2=14.48%, 4=0.03%, 10=0.20%, 20=14.37%, 50=7.15% lat (msec) : 100=0.05% cpu : usr=2.49%, sys=1.70%, ctx=6365, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5039,1315,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=672MiB/s (704MB/s), 672MiB/s-672MiB/s (704MB/s-704MB/s), io=19.7GiB (21.1GB), run=30010-30010msec WRITE: bw=175MiB/s (184MB/s), 175MiB/s-175MiB/s (184MB/s-184MB/s), io=5260MiB (5516MB), run=30010-30010msec Disk stats (read/write): vdf: ios=20132/5237, merge=0/0, ticks=4413/82075, in_queue=96305, util=95.12% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983232: Fri Jan 7 14:40:35 2022 read: IOPS=177, BW=708MiB/s (743MB/s)(21.2GiB/30633msec) slat (usec): min=46, max=465638, avg=3981.03, stdev=12433.21 clat (msec): min=322, max=3295, avg=754.71, stdev=210.33 lat (msec): min=322, max=3295, avg=758.69, stdev=211.04 clat percentiles (msec): | 1.00th=[ 393], 5.00th=[ 468], 10.00th=[ 510], 20.00th=[ 592], | 30.00th=[ 659], 40.00th=[ 709], 50.00th=[ 751], 60.00th=[ 785], | 70.00th=[ 818], 80.00th=[ 869], 90.00th=[ 986], 95.00th=[ 1099], | 99.00th=[ 1368], 99.50th=[ 1536], 99.90th=[ 2400], 99.95th=[ 2702], | 99.99th=[ 3306] bw ( KiB/s): min= 8192, max=1337971, per=99.75%, avg=723591.30, stdev=293152.84, samples=60 iops : min= 2, max= 326, avg=176.63, stdev=71.55, samples=60 write: IOPS=46, BW=187MiB/s (196MB/s)(5716MiB/30633msec) slat (usec): min=1132, max=504480, avg=5860.55, stdev=24122.43 clat (msec): min=635, max=3781, avg=1370.34, stdev=325.81 lat (msec): min=637, max=3806, avg=1376.20, stdev=324.82 clat percentiles (msec): | 1.00th=[ 743], 5.00th=[ 877], 10.00th=[ 986], 20.00th=[ 1150], | 30.00th=[ 1234], 40.00th=[ 1284], 50.00th=[ 1334], 60.00th=[ 1401], | 70.00th=[ 1469], 80.00th=[ 1569], 90.00th=[ 1720], 95.00th=[ 1905], | 99.00th=[ 2400], 99.50th=[ 2769], 99.90th=[ 3574], 99.95th=[ 3775], | 99.99th=[ 3775] bw ( KiB/s): min=24576, max=565248, per=98.55%, avg=188310.78, stdev=103168.96, samples=59 iops : min= 6, max= 138, avg=45.97, stdev=25.18, samples=59 lat (msec) : 500=7.13%, 750=33.06%, 1000=33.78%, 2000=25.04%, >=2000=0.99% cpu : usr=6.38%, sys=1.64%, ctx=2515, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5425,1429,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=708MiB/s (743MB/s), 708MiB/s-708MiB/s (743MB/s-743MB/s), io=21.2GiB (22.8GB), run=30633-30633msec WRITE: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=5716MiB (5994MB), run=30633-30633msec Disk stats (read/write): vdf: ios=21700/5684, merge=0/0, ticks=152323/3577200, in_queue=3761368, util=98.43%