30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890670: Fri Jan 7 13:33:20 2022 read: IOPS=2346, BW=9385KiB/s (9610kB/s)(275MiB/30002msec) clat (usec): min=25, max=60379, avg=78.08, stdev=656.51 lat (usec): min=25, max=60379, avg=78.23, stdev=656.51 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 69], | 99.00th=[ 89], 99.50th=[ 106], 99.90th=[12911], 99.95th=[15008], | 99.99th=[17957] bw ( KiB/s): min= 6320, max=14304, per=100.00%, avg=9386.35, stdev=1158.48, samples=60 iops : min= 1580, max= 3576, avg=2346.57, stdev=289.63, samples=60 write: IOPS=587, BW=2349KiB/s (2405kB/s)(68.8MiB/30002msec) clat (usec): min=177, max=51435, avg=1381.62, stdev=1672.02 lat (usec): min=177, max=51436, avg=1381.91, stdev=1672.01 clat percentiles (usec): | 1.00th=[ 255], 5.00th=[ 429], 10.00th=[ 685], 20.00th=[ 947], | 30.00th=[ 1090], 40.00th=[ 1156], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1352], 80.00th=[ 1467], 90.00th=[ 1696], 95.00th=[ 2024], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[23725], 99.95th=[40109], | 99.99th=[44827] bw ( KiB/s): min= 1776, max= 3680, per=100.00%, avg=2348.75, stdev=269.77, samples=60 iops : min= 444, max= 920, avg=587.18, stdev=67.44, samples=60 lat (usec) : 50=50.84%, 100=28.66%, 250=0.44%, 500=1.12%, 750=1.04% lat (usec) : 1000=2.34% lat (msec) : 2=14.33%, 4=0.63%, 10=0.38%, 20=0.18%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.35%, sys=3.80%, ctx=88013, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70391,17615,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9385KiB/s (9610kB/s), 9385KiB/s-9385KiB/s (9610kB/s-9610kB/s), io=275MiB (288MB), run=30002-30002msec WRITE: bw=2349KiB/s (2405kB/s), 2349KiB/s-2349KiB/s (2405kB/s-2405kB/s), io=68.8MiB (72.2MB), run=30002-30002msec Disk stats (read/write): vdd: ios=69980/17517, merge=0/0, ticks=1937/16842, in_queue=28205, util=94.17% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891398: Fri Jan 7 13:33:57 2022 read: IOPS=2379, BW=9519KiB/s (9747kB/s)(279MiB/30005msec) slat (usec): min=5, max=202, avg= 9.27, stdev= 3.47 clat (nsec): min=790, max=25570k, avg=63895.62, stdev=602692.00 lat (usec): min=22, max=25577, avg=73.31, stdev=602.71 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 47], 95.00th=[ 52], | 99.00th=[ 68], 99.50th=[ 80], 99.90th=[12256], 99.95th=[14746], | 99.99th=[17433] bw ( KiB/s): min= 6968, max=16064, per=100.00%, avg=9520.63, stdev=1273.13, samples=60 iops : min= 1742, max= 4016, avg=2380.15, stdev=318.27, samples=60 write: IOPS=595, BW=2382KiB/s (2439kB/s)(69.8MiB/30005msec) slat (usec): min=7, max=132, avg=10.31, stdev= 3.47 clat (usec): min=171, max=44026, avg=1367.52, stdev=1639.64 lat (usec): min=179, max=44035, avg=1377.97, stdev=1639.69 clat percentiles (usec): | 1.00th=[ 245], 5.00th=[ 412], 10.00th=[ 693], 20.00th=[ 963], | 30.00th=[ 1090], 40.00th=[ 1156], 50.00th=[ 1205], 60.00th=[ 1270], | 70.00th=[ 1352], 80.00th=[ 1467], 90.00th=[ 1696], 95.00th=[ 1975], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 1896, max= 4128, per=100.00%, avg=2382.07, stdev=317.02, samples=60 iops : min= 474, max= 1032, avg=595.52, stdev=79.26, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.09%, 50=74.85% lat (usec) : 100=4.68%, 250=0.29%, 500=1.06%, 750=0.96%, 1000=2.31% lat (msec) : 2=14.50%, 4=0.61%, 10=0.36%, 20=0.16%, 50=0.03% cpu : usr=1.34%, sys=5.61%, ctx=89182, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71403,17865,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9519KiB/s (9747kB/s), 9519KiB/s-9519KiB/s (9747kB/s-9747kB/s), io=279MiB (292MB), run=30005-30005msec WRITE: bw=2382KiB/s (2439kB/s), 2382KiB/s-2382KiB/s (2439kB/s-2439kB/s), io=69.8MiB (73.2MB), run=30005-30005msec Disk stats (read/write): vdd: ios=71038/17771, merge=0/0, ticks=1890/17094, in_queue=28208, util=94.09% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892538: Fri Jan 7 13:34:34 2022 read: IOPS=2369, BW=9479KiB/s (9706kB/s)(278MiB/30001msec) clat (usec): min=25, max=63005, avg=78.49, stdev=668.91 lat (usec): min=25, max=63006, avg=78.63, stdev=668.91 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 87], 99.50th=[ 105], 99.90th=[13173], 99.95th=[15139], | 99.99th=[19792] bw ( KiB/s): min= 7616, max=15064, per=100.00%, avg=9508.88, stdev=1169.69, samples=59 iops : min= 1904, max= 3766, avg=2377.22, stdev=292.42, samples=59 write: IOPS=592, BW=2372KiB/s (2429kB/s)(69.5MiB/30001msec) clat (usec): min=194, max=44528, avg=1363.63, stdev=1621.82 lat (usec): min=194, max=44529, avg=1363.91, stdev=1621.81 clat percentiles (usec): | 1.00th=[ 262], 5.00th=[ 433], 10.00th=[ 685], 20.00th=[ 947], | 30.00th=[ 1090], 40.00th=[ 1156], 50.00th=[ 1221], 60.00th=[ 1270], | 70.00th=[ 1352], 80.00th=[ 1467], 90.00th=[ 1680], 95.00th=[ 1991], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[34341], 99.95th=[37487], | 99.99th=[43254] bw ( KiB/s): min= 1936, max= 3888, per=100.00%, avg=2380.75, stdev=286.35, samples=59 iops : min= 484, max= 972, avg=595.19, stdev=71.59, samples=59 lat (usec) : 50=50.87%, 100=28.65%, 250=0.39%, 500=1.13%, 750=1.08% lat (usec) : 1000=2.34% lat (msec) : 2=14.35%, 4=0.65%, 10=0.31%, 20=0.19%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.34%, sys=3.79%, ctx=88888, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71092,17788,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9479KiB/s (9706kB/s), 9479KiB/s-9479KiB/s (9706kB/s-9706kB/s), io=278MiB (291MB), run=30001-30001msec WRITE: bw=2372KiB/s (2429kB/s), 2372KiB/s-2372KiB/s (2429kB/s-2429kB/s), io=69.5MiB (72.9MB), run=30001-30001msec Disk stats (read/write): vdd: ios=70856/17713, merge=0/0, ticks=1989/16697, in_queue=28146, util=93.97% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892989: Fri Jan 7 13:35:11 2022 read: IOPS=2501, BW=9.77MiB/s (10.2MB/s)(293MiB/30032msec) slat (usec): min=2, max=464, avg=10.03, stdev= 5.49 clat (nsec): min=670, max=2490.8M, avg=1720609.42, stdev=46517505.24 lat (usec): min=23, max=2490.8k, avg=1730.80, stdev=46517.49 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 38], 60.00th=[ 40], 70.00th=[ 43], | 80.00th=[ 46], 90.00th=[ 53], 95.00th=[ 61], | 99.00th=[ 98], 99.50th=[ 133], 99.90th=[ 666895], | 99.95th=[1149240], 99.99th=[2298479] bw ( KiB/s): min= 8640, max=16976, per=100.00%, avg=10016.42, stdev=1108.64, samples=60 iops : min= 2160, max= 4244, avg=2504.08, stdev=277.17, samples=60 write: IOPS=624, BW=2496KiB/s (2556kB/s)(73.2MiB/30032msec) slat (usec): min=2, max=244, avg=11.18, stdev= 5.31 clat (usec): min=260, max=69867, avg=9059.12, stdev=6310.61 lat (usec): min=274, max=69877, avg=9070.47, stdev=6310.82 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1287], 10.00th=[ 2245], 20.00th=[ 4178], | 30.00th=[ 6128], 40.00th=[ 7373], 50.00th=[ 8586], 60.00th=[ 9765], | 70.00th=[11076], 80.00th=[12649], 90.00th=[14746], 95.00th=[18482], | 99.00th=[25035], 99.50th=[54264], 99.90th=[64226], 99.95th=[65274], | 99.99th=[69731] bw ( KiB/s): min= 2136, max= 4264, per=100.00%, avg=2498.33, stdev=284.22, samples=60 iops : min= 534, max= 1066, avg=624.58, stdev=71.05, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.12%, 4=0.01%, 10=0.06%, 20=0.18%, 50=69.47% lat (usec) : 100=9.36%, 250=0.53%, 500=0.06%, 750=0.10%, 1000=0.20% lat (msec) : 2=1.52%, 4=1.95%, 10=8.64%, 20=6.88%, 50=0.58% lat (msec) : 100=0.12%, 250=0.02%, 500=0.03%, 750=0.03%, 1000=0.01% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=1.54%, sys=6.11%, ctx=92293, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=75119,18742,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9.77MiB/s (10.2MB/s), 9.77MiB/s-9.77MiB/s (10.2MB/s-10.2MB/s), io=293MiB (308MB), run=30032-30032msec WRITE: bw=2496KiB/s (2556kB/s), 2496KiB/s-2496KiB/s (2556kB/s-2556kB/s), io=73.2MiB (76.8MB), run=30032-30032msec Disk stats (read/write): vdd: ios=74698/18632, merge=0/0, ticks=121636/158986, in_queue=296363, util=99.47% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894362: Fri Jan 7 13:35:48 2022 read: IOPS=2378, BW=9512KiB/s (9741kB/s)(279MiB/30001msec) clat (usec): min=25, max=57400, avg=77.66, stdev=665.58 lat (usec): min=25, max=57400, avg=77.79, stdev=665.58 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 85], 99.50th=[ 99], 99.90th=[13698], 99.95th=[15401], | 99.99th=[20579] bw ( KiB/s): min= 7552, max=13811, per=100.00%, avg=9513.55, stdev=1162.75, samples=60 iops : min= 1888, max= 3452, avg=2378.35, stdev=290.66, samples=60 write: IOPS=594, BW=2379KiB/s (2436kB/s)(69.7MiB/30001msec) clat (usec): min=195, max=43240, avg=1360.42, stdev=1599.53 lat (usec): min=195, max=43241, avg=1360.68, stdev=1599.53 clat percentiles (usec): | 1.00th=[ 258], 5.00th=[ 433], 10.00th=[ 693], 20.00th=[ 955], | 30.00th=[ 1090], 40.00th=[ 1156], 50.00th=[ 1221], 60.00th=[ 1270], | 70.00th=[ 1352], 80.00th=[ 1450], 90.00th=[ 1680], 95.00th=[ 2008], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[34866], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min= 1912, max= 3551, per=100.00%, avg=2379.20, stdev=261.54, samples=60 iops : min= 478, max= 887, avg=594.78, stdev=65.33, samples=60 lat (usec) : 50=54.20%, 100=25.42%, 250=0.33%, 500=1.12%, 750=1.04% lat (usec) : 1000=2.24% lat (msec) : 2=14.46%, 4=0.66%, 10=0.35%, 20=0.17%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.53%, sys=3.61%, ctx=89192, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71344,17843,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9512KiB/s (9741kB/s), 9512KiB/s-9512KiB/s (9741kB/s-9741kB/s), io=279MiB (292MB), run=30001-30001msec WRITE: bw=2379KiB/s (2436kB/s), 2379KiB/s-2379KiB/s (2436kB/s-2436kB/s), io=69.7MiB (73.1MB), run=30001-30001msec Disk stats (read/write): vdd: ios=70936/17743, merge=0/0, ticks=1987/16869, in_queue=28230, util=94.22% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894975: Fri Jan 7 13:36:25 2022 read: IOPS=2498, BW=9993KiB/s (10.2MB/s)(294MiB/30088msec) slat (usec): min=2, max=131, avg= 9.61, stdev= 3.83 clat (nsec): min=635, max=2523.6M, avg=2027996.91, stdev=52583614.87 lat (usec): min=23, max=2523.6k, avg=2037.75, stdev=52583.61 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 50], 90.00th=[ 61], 95.00th=[ 79], | 99.00th=[ 208], 99.50th=[ 498], 99.90th=[ 801113], | 99.95th=[1266680], 99.99th=[2499806] bw ( KiB/s): min= 7704, max=17432, per=100.00%, avg=10021.25, stdev=1314.03, samples=60 iops : min= 1926, max= 4358, avg=2505.30, stdev=328.51, samples=60 write: IOPS=623, BW=2493KiB/s (2553kB/s)(73.2MiB/30088msec) slat (usec): min=2, max=189, avg=10.80, stdev= 4.33 clat (msec): min=7, max=2145, avg=72.00, stdev=54.21 lat (msec): min=7, max=2145, avg=72.01, stdev=54.21 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 51], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 62], 40.00th=[ 64], 50.00th=[ 66], 60.00th=[ 69], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 113], | 99.00th=[ 178], 99.50th=[ 351], 99.90th=[ 885], 99.95th=[ 1053], | 99.99th=[ 1787] bw ( KiB/s): min= 1872, max= 4064, per=100.00%, avg=2494.87, stdev=270.59, samples=60 iops : min= 468, max= 1016, avg=623.72, stdev=67.65, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.11%, 50=64.70% lat (usec) : 100=12.88%, 250=1.54%, 500=0.30%, 750=0.12%, 1000=0.06% lat (msec) : 2=0.03%, 10=0.15%, 20=0.09%, 50=0.74%, 100=17.66% lat (msec) : 250=1.21%, 500=0.13%, 750=0.06%, 1000=0.04%, 2000=0.05% lat (msec) : >=2000=0.02% cpu : usr=1.34%, sys=6.04%, ctx=91173, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=75166,18752,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9993KiB/s (10.2MB/s), 9993KiB/s-9993KiB/s (10.2MB/s-10.2MB/s), io=294MiB (308MB), run=30088-30088msec WRITE: bw=2493KiB/s (2553kB/s), 2493KiB/s-2493KiB/s (2553kB/s-2553kB/s), io=73.2MiB (76.8MB), run=30088-30088msec Disk stats (read/write): vdd: ios=75159/18742, merge=0/0, ticks=144246/1337961, in_queue=1499233, util=99.47% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895735: Fri Jan 7 13:37:02 2022 read: IOPS=2362, BW=9450KiB/s (9677kB/s)(277MiB/30001msec) clat (usec): min=25, max=23563, avg=76.42, stdev=609.74 lat (usec): min=25, max=23563, avg=76.56, stdev=609.74 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 68], | 99.00th=[ 88], 99.50th=[ 110], 99.90th=[12780], 99.95th=[15270], | 99.99th=[17171] bw ( KiB/s): min= 7720, max=14472, per=100.00%, avg=9450.17, stdev=1171.78, samples=60 iops : min= 1930, max= 3618, avg=2362.53, stdev=292.94, samples=60 write: IOPS=590, BW=2363KiB/s (2419kB/s)(69.2MiB/30001msec) clat (usec): min=181, max=42901, avg=1377.76, stdev=1637.60 lat (usec): min=181, max=42902, avg=1378.04, stdev=1637.60 clat percentiles (usec): | 1.00th=[ 255], 5.00th=[ 429], 10.00th=[ 676], 20.00th=[ 947], | 30.00th=[ 1090], 40.00th=[ 1156], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1713], 95.00th=[ 2024], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[35390], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min= 1840, max= 3720, per=100.00%, avg=2362.63, stdev=272.44, samples=60 iops : min= 460, max= 930, avg=590.65, stdev=68.10, samples=60 lat (usec) : 50=53.30%, 100=26.19%, 250=0.46%, 500=1.10%, 750=1.10% lat (usec) : 1000=2.32% lat (msec) : 2=14.30%, 4=0.68%, 10=0.35%, 20=0.18%, 50=0.03% cpu : usr=1.38%, sys=3.68%, ctx=88606, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70877,17721,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9450KiB/s (9677kB/s), 9450KiB/s-9450KiB/s (9677kB/s-9677kB/s), io=277MiB (290MB), run=30001-30001msec WRITE: bw=2363KiB/s (2419kB/s), 2363KiB/s-2363KiB/s (2419kB/s-2419kB/s), io=69.2MiB (72.6MB), run=30001-30001msec Disk stats (read/write): vdd: ios=70388/17612, merge=0/0, ticks=1913/16846, in_queue=28255, util=94.24% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896942: Fri Jan 7 13:37:39 2022 read: IOPS=2505, BW=9.79MiB/s (10.3MB/s)(294MiB/30062msec) slat (usec): min=2, max=467, avg= 9.67, stdev= 4.94 clat (nsec): min=712, max=2506.7M, avg=2138798.58, stdev=53762773.74 lat (usec): min=22, max=2506.7k, avg=2148.61, stdev=53762.77 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 52], 90.00th=[ 72], 95.00th=[ 129], | 99.00th=[ 816], 99.50th=[ 1516], 99.90th=[ 843056], | 99.95th=[1350566], 99.99th=[2499806] bw ( KiB/s): min= 6264, max=19448, per=100.00%, avg=10043.53, stdev=1703.04, samples=60 iops : min= 1566, max= 4862, avg=2510.87, stdev=425.75, samples=60 write: IOPS=625, BW=2500KiB/s (2560kB/s)(73.4MiB/30062msec) slat (usec): min=2, max=255, avg=10.80, stdev= 4.66 clat (msec): min=8, max=2147, avg=151.29, stdev=67.24 lat (msec): min=8, max=2147, avg=151.30, stdev=67.24 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 103], 10.00th=[ 122], 20.00th=[ 132], | 30.00th=[ 136], 40.00th=[ 140], 50.00th=[ 144], 60.00th=[ 148], | 70.00th=[ 153], 80.00th=[ 161], 90.00th=[ 184], 95.00th=[ 207], | 99.00th=[ 388], 99.50th=[ 634], 99.90th=[ 894], 99.95th=[ 1053], | 99.99th=[ 1754] bw ( KiB/s): min= 1624, max= 4248, per=99.76%, avg=2494.07, stdev=317.61, samples=60 iops : min= 406, max= 1062, avg=623.50, stdev=79.40, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.11%, 50=61.99% lat (usec) : 100=12.62%, 250=3.29%, 500=0.79%, 750=0.29%, 1000=0.20% lat (msec) : 2=0.42%, 4=0.06%, 10=0.04%, 20=0.16%, 50=0.22% lat (msec) : 100=0.52%, 250=18.49%, 500=0.49%, 750=0.10%, 1000=0.06% lat (msec) : 2000=0.06%, >=2000=0.02% cpu : usr=1.37%, sys=6.00%, ctx=88505, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75329,18790,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9.79MiB/s (10.3MB/s), 9.79MiB/s-9.79MiB/s (10.3MB/s-10.3MB/s), io=294MiB (309MB), run=30062-30062msec WRITE: bw=2500KiB/s (2560kB/s), 2500KiB/s-2500KiB/s (2560kB/s-2560kB/s), io=73.4MiB (76.0MB), run=30062-30062msec Disk stats (read/write): vdd: ios=75316/18703, merge=0/0, ticks=144527/2806628, in_queue=3001159, util=99.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897502: Fri Jan 7 13:38:16 2022 read: IOPS=2345, BW=9380KiB/s (9605kB/s)(275MiB/30001msec) clat (usec): min=25, max=30783, avg=79.30, stdev=634.93 lat (usec): min=25, max=30783, avg=79.44, stdev=634.93 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 93], 99.50th=[ 113], 99.90th=[13042], 99.95th=[15139], | 99.99th=[21627] bw ( KiB/s): min= 7192, max=14456, per=100.00%, avg=9380.97, stdev=1191.88, samples=60 iops : min= 1798, max= 3614, avg=2345.22, stdev=297.98, samples=60 write: IOPS=586, BW=2347KiB/s (2404kB/s)(68.8MiB/30001msec) clat (usec): min=184, max=57106, avg=1377.59, stdev=1721.83 lat (usec): min=185, max=57106, avg=1377.92, stdev=1721.83 clat percentiles (usec): | 1.00th=[ 258], 5.00th=[ 424], 10.00th=[ 660], 20.00th=[ 930], | 30.00th=[ 1074], 40.00th=[ 1156], 50.00th=[ 1221], 60.00th=[ 1270], | 70.00th=[ 1352], 80.00th=[ 1467], 90.00th=[ 1696], 95.00th=[ 2040], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[35914], 99.95th=[40109], | 99.99th=[44827] bw ( KiB/s): min= 1888, max= 3704, per=100.00%, avg=2347.35, stdev=288.07, samples=60 iops : min= 472, max= 926, avg=586.83, stdev=72.02, samples=60 lat (usec) : 50=45.75%, 100=33.65%, 250=0.53%, 500=1.19%, 750=1.13% lat (usec) : 1000=2.31% lat (msec) : 2=14.17%, 4=0.70%, 10=0.37%, 20=0.16%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.39%, sys=3.86%, ctx=87971, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70353,17605,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9380KiB/s (9605kB/s), 9380KiB/s-9380KiB/s (9605kB/s-9605kB/s), io=275MiB (288MB), run=30001-30001msec WRITE: bw=2347KiB/s (2404kB/s), 2347KiB/s-2347KiB/s (2404kB/s-2404kB/s), io=68.8MiB (72.1MB), run=30001-30001msec Disk stats (read/write): vdd: ios=69847/17475, merge=0/0, ticks=1928/16734, in_queue=28025, util=93.59% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898159: Fri Jan 7 13:38:53 2022 read: IOPS=2463, BW=9854KiB/s (10.1MB/s)(290MiB/30186msec) slat (usec): min=2, max=191235, avg=319.28, stdev=1747.67 clat (usec): min=537, max=2576.1k, avg=27365.79, stdev=57568.80 lat (usec): min=540, max=2576.1k, avg=27685.20, stdev=57599.08 clat percentiles (usec): | 1.00th=[ 1045], 5.00th=[ 8717], 10.00th=[ 14091], | 20.00th=[ 17433], 30.00th=[ 19792], 40.00th=[ 21627], | 50.00th=[ 23462], 60.00th=[ 25297], 70.00th=[ 27919], | 80.00th=[ 30802], 90.00th=[ 35914], 95.00th=[ 42206], | 99.00th=[ 86508], 99.50th=[ 187696], 99.90th=[ 834667], | 99.95th=[1367344], 99.99th=[2533360] bw ( KiB/s): min= 7376, max=18072, per=100.00%, avg=9905.80, stdev=1428.17, samples=60 iops : min= 1844, max= 4518, avg=2476.45, stdev=357.04, samples=60 write: IOPS=614, BW=2458KiB/s (2517kB/s)(72.5MiB/30186msec) slat (usec): min=2, max=177966, avg=327.46, stdev=2540.72 clat (msec): min=13, max=2526, avg=214.00, stdev=98.63 lat (msec): min=13, max=2529, avg=214.33, stdev=98.63 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 142], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 190], 40.00th=[ 197], 50.00th=[ 203], 60.00th=[ 211], | 70.00th=[ 222], 80.00th=[ 239], 90.00th=[ 259], 95.00th=[ 284], | 99.00th=[ 542], 99.50th=[ 877], 99.90th=[ 1234], 99.95th=[ 1888], | 99.99th=[ 2534] bw ( KiB/s): min= 1936, max= 3704, per=99.96%, avg=2456.08, stdev=250.84, samples=60 iops : min= 484, max= 926, avg=614.02, stdev=62.71, samples=60 lat (usec) : 750=0.20%, 1000=0.50% lat (msec) : 2=1.15%, 4=1.46%, 10=0.95%, 20=21.12%, 50=52.56% lat (msec) : 100=2.04%, 250=17.07%, 500=2.61%, 750=0.12%, 1000=0.10% lat (msec) : 2000=0.10%, >=2000=0.03% cpu : usr=0.70%, sys=1.91%, ctx=10355, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=74363,18548,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9854KiB/s (10.1MB/s), 9854KiB/s-9854KiB/s (10.1MB/s-10.1MB/s), io=290MiB (305MB), run=30186-30186msec WRITE: bw=2458KiB/s (2517kB/s), 2458KiB/s-2458KiB/s (2517kB/s-2517kB/s), io=72.5MiB (75.0MB), run=30186-30186msec Disk stats (read/write): vdd: ios=74361/18477, merge=0/0, ticks=158765/3476843, in_queue=3667423, util=99.48% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898983: Fri Jan 7 13:39:30 2022 read: IOPS=2312, BW=36.1MiB/s (37.9MB/s)(1085MiB/30013msec) clat (usec): min=25, max=65234, avg=82.41, stdev=693.15 lat (usec): min=26, max=65234, avg=82.56, stdev=693.15 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 74], | 99.00th=[ 97], 99.50th=[ 118], 99.90th=[12649], 99.95th=[15139], | 99.99th=[20055] bw ( KiB/s): min=30560, max=57056, per=100.00%, avg=37019.20, stdev=4211.43, samples=60 iops : min= 1910, max= 3566, avg=2313.70, stdev=263.21, samples=60 write: IOPS=578, BW=9250KiB/s (9472kB/s)(271MiB/30013msec) clat (usec): min=205, max=43214, avg=1389.75, stdev=1574.71 lat (usec): min=209, max=43215, avg=1391.23, stdev=1574.70 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 457], 10.00th=[ 701], 20.00th=[ 971], | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1745], 95.00th=[ 2073], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[16909], 99.95th=[40109], | 99.99th=[42206] bw ( KiB/s): min= 7040, max=14656, per=100.00%, avg=9254.40, stdev=1046.56, samples=60 iops : min= 440, max= 916, avg=578.40, stdev=65.41, samples=60 lat (usec) : 50=39.46%, 100=39.86%, 250=0.50%, 500=1.14%, 750=1.08% lat (usec) : 1000=2.10% lat (msec) : 2=14.56%, 4=0.78%, 10=0.33%, 20=0.17%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.37%, sys=3.82%, ctx=86772, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69412,17352,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.1MiB/s (37.9MB/s), 36.1MiB/s-36.1MiB/s (37.9MB/s-37.9MB/s), io=1085MiB (1137MB), run=30013-30013msec WRITE: bw=9250KiB/s (9472kB/s), 9250KiB/s-9250KiB/s (9472kB/s-9472kB/s), io=271MiB (284MB), run=30013-30013msec Disk stats (read/write): vdd: ios=69150/17293, merge=0/0, ticks=1902/16675, in_queue=28219, util=94.23% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899802: Fri Jan 7 13:40:07 2022 read: IOPS=2292, BW=35.8MiB/s (37.6MB/s)(1075MiB/30002msec) slat (usec): min=5, max=315, avg=10.43, stdev= 5.06 clat (nsec): min=679, max=28931k, avg=68370.56, stdev=639577.37 lat (usec): min=23, max=28939, avg=78.95, stdev=639.58 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 30], 10.00th=[ 31], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 38], 60.00th=[ 40], | 70.00th=[ 42], 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 57], | 99.00th=[ 75], 99.50th=[ 90], 99.90th=[12649], 99.95th=[15401], | 99.99th=[22938] bw ( KiB/s): min=29632, max=59680, per=100.00%, avg=36689.73, stdev=4815.49, samples=60 iops : min= 1852, max= 3730, avg=2293.10, stdev=300.97, samples=60 write: IOPS=573, BW=9174KiB/s (9395kB/s)(269MiB/30002msec) slat (usec): min=8, max=275, avg=12.70, stdev= 5.66 clat (usec): min=201, max=42856, avg=1407.49, stdev=1650.31 lat (usec): min=211, max=42866, avg=1420.35, stdev=1650.36 clat percentiles (usec): | 1.00th=[ 289], 5.00th=[ 449], 10.00th=[ 709], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1745], 95.00th=[ 2057], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[16712], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 7328, max=15360, per=100.00%, avg=9174.70, stdev=1221.31, samples=60 iops : min= 458, max= 960, avg=573.42, stdev=76.33, samples=60 lat (nsec) : 750=0.02%, 1000=0.08% lat (usec) : 2=0.11%, 4=0.01%, 10=0.03%, 20=0.15%, 50=71.01% lat (usec) : 100=8.26%, 250=0.16%, 500=1.16%, 750=1.00%, 1000=1.99% lat (msec) : 2=14.73%, 4=0.72%, 10=0.36%, 20=0.18%, 50=0.03% cpu : usr=1.28%, sys=5.93%, ctx=85751, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68791,17203,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=35.8MiB/s (37.6MB/s), 35.8MiB/s-35.8MiB/s (37.6MB/s-37.6MB/s), io=1075MiB (1127MB), run=30002-30002msec WRITE: bw=9174KiB/s (9395kB/s), 9174KiB/s-9174KiB/s (9395kB/s-9395kB/s), io=269MiB (282MB), run=30002-30002msec Disk stats (read/write): vdd: ios=68413/17122, merge=0/0, ticks=1903/17083, in_queue=28306, util=94.50% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901189: Fri Jan 7 13:40:44 2022 read: IOPS=2304, BW=36.0MiB/s (37.8MB/s)(1081MiB/30003msec) clat (usec): min=25, max=25023, avg=78.09, stdev=603.54 lat (usec): min=25, max=25024, avg=78.23, stdev=603.54 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 91], 99.50th=[ 111], 99.90th=[12649], 99.95th=[14484], | 99.99th=[17433] bw ( KiB/s): min=28768, max=57216, per=100.00%, avg=36882.97, stdev=4515.22, samples=60 iops : min= 1798, max= 3576, avg=2305.18, stdev=282.20, samples=60 write: IOPS=576, BW=9224KiB/s (9446kB/s)(270MiB/30003msec) clat (usec): min=211, max=42671, avg=1412.08, stdev=1669.16 lat (usec): min=212, max=42672, avg=1413.75, stdev=1669.14 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 461], 10.00th=[ 717], 20.00th=[ 988], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1745], 95.00th=[ 2073], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[38536], | 99.99th=[42730] bw ( KiB/s): min= 7040, max=14720, per=100.00%, avg=9224.88, stdev=1113.73, samples=60 iops : min= 440, max= 920, avg=576.55, stdev=69.60, samples=60 lat (usec) : 50=46.91%, 100=32.56%, 250=0.34%, 500=1.12%, 750=1.02% lat (usec) : 1000=2.02% lat (msec) : 2=14.72%, 4=0.74%, 10=0.36%, 20=0.18%, 50=0.03% cpu : usr=1.38%, sys=3.71%, ctx=86458, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69153,17297,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=36.0MiB/s (37.8MB/s), 36.0MiB/s-36.0MiB/s (37.8MB/s-37.8MB/s), io=1081MiB (1133MB), run=30003-30003msec WRITE: bw=9224KiB/s (9446kB/s), 9224KiB/s-9224KiB/s (9446kB/s-9446kB/s), io=270MiB (283MB), run=30003-30003msec Disk stats (read/write): vdd: ios=68989/17242, merge=0/0, ticks=1831/17060, in_queue=28254, util=94.33% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901836: Fri Jan 7 13:41:21 2022 read: IOPS=2416, BW=37.8MiB/s (39.6MB/s)(1135MiB/30055msec) slat (usec): min=2, max=208, avg=10.89, stdev= 4.01 clat (nsec): min=754, max=2491.4M, avg=1758901.01, stdev=48002915.23 lat (usec): min=27, max=2491.4k, avg=1769.94, stdev=48002.91 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 44], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 56], 95.00th=[ 64], | 99.00th=[ 91], 99.50th=[ 131], 99.90th=[ 717226], | 99.95th=[1233126], 99.99th=[2365588] bw ( KiB/s): min=31680, max=70208, per=100.00%, avg=38741.88, stdev=5102.88, samples=60 iops : min= 1980, max= 4388, avg=2421.37, stdev=318.93, samples=60 write: IOPS=604, BW=9675KiB/s (9907kB/s)(284MiB/30055msec) slat (usec): min=4, max=666, avg=13.49, stdev= 6.43 clat (usec): min=296, max=84537, avg=9437.83, stdev=6120.29 lat (usec): min=306, max=84548, avg=9451.49, stdev=6120.59 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 1532], 10.00th=[ 2999], 20.00th=[ 4817], | 30.00th=[ 6521], 40.00th=[ 7832], 50.00th=[ 9372], 60.00th=[10552], | 70.00th=[11600], 80.00th=[12649], 90.00th=[14484], 95.00th=[17695], | 99.00th=[23725], 99.50th=[52167], 99.90th=[65799], 99.95th=[70779], | 99.99th=[83362] bw ( KiB/s): min= 8000, max=17952, per=100.00%, avg=9690.25, stdev=1306.10, samples=60 iops : min= 500, max= 1122, avg=605.63, stdev=81.63, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.06%, 50=64.99% lat (usec) : 100=14.25%, 250=0.37%, 500=0.07%, 750=0.06%, 1000=0.12% lat (msec) : 2=1.07%, 4=1.67%, 10=8.23%, 20=8.28%, 50=0.49% lat (msec) : 100=0.12%, 250=0.02%, 500=0.03%, 750=0.03%, 1000=0.02% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=1.52%, sys=6.30%, ctx=89613, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72640,18174,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.6MB/s), 37.8MiB/s-37.8MiB/s (39.6MB/s-39.6MB/s), io=1135MiB (1190MB), run=30055-30055msec WRITE: bw=9675KiB/s (9907kB/s), 9675KiB/s-9675KiB/s (9907kB/s-9907kB/s), io=284MiB (298MB), run=30055-30055msec Disk stats (read/write): vdd: ios=72260/18083, merge=0/0, ticks=119624/161157, in_queue=296540, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902716: Fri Jan 7 13:41:58 2022 read: IOPS=2290, BW=35.8MiB/s (37.5MB/s)(1074MiB/30004msec) clat (usec): min=26, max=28982, avg=80.39, stdev=644.00 lat (usec): min=26, max=28982, avg=80.54, stdev=644.00 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 93], 99.50th=[ 111], 99.90th=[13698], 99.95th=[15664], | 99.99th=[20317] bw ( KiB/s): min=30016, max=63200, per=100.00%, avg=36661.92, stdev=5127.58, samples=60 iops : min= 1876, max= 3950, avg=2291.37, stdev=320.48, samples=60 write: IOPS=572, BW=9167KiB/s (9387kB/s)(269MiB/30004msec) clat (usec): min=207, max=44271, avg=1413.64, stdev=1662.64 lat (usec): min=208, max=44272, avg=1415.09, stdev=1662.63 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 465], 10.00th=[ 709], 20.00th=[ 979], | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1516], 90.00th=[ 1762], 95.00th=[ 2073], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[17171], 99.95th=[40633], | 99.99th=[42730] bw ( KiB/s): min= 7264, max=16224, per=100.00%, avg=9168.25, stdev=1304.47, samples=60 iops : min= 454, max= 1014, avg=573.00, stdev=81.55, samples=60 lat (usec) : 50=43.09%, 100=36.34%, 250=0.42%, 500=1.12%, 750=1.02% lat (usec) : 1000=2.06% lat (msec) : 2=14.58%, 4=0.81%, 10=0.34%, 20=0.19%, 50=0.03% cpu : usr=1.45%, sys=3.68%, ctx=85939, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68739,17191,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=35.8MiB/s (37.5MB/s), 35.8MiB/s-35.8MiB/s (37.5MB/s-37.5MB/s), io=1074MiB (1126MB), run=30004-30004msec WRITE: bw=9167KiB/s (9387kB/s), 9167KiB/s-9167KiB/s (9387kB/s-9387kB/s), io=269MiB (282MB), run=30004-30004msec Disk stats (read/write): vdd: ios=68393/17113, merge=0/0, ticks=1917/16942, in_queue=28225, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903780: Fri Jan 7 13:42:35 2022 read: IOPS=2399, BW=37.5MiB/s (39.3MB/s)(1130MiB/30134msec) slat (usec): min=2, max=394, avg=10.07, stdev= 5.78 clat (nsec): min=654, max=2616.5M, avg=2106385.82, stdev=55918364.58 lat (usec): min=23, max=2616.5k, avg=2116.61, stdev=55918.35 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 51], 90.00th=[ 63], 95.00th=[ 82], | 99.00th=[ 265], 99.50th=[ 537], 99.90th=[ 775947], | 99.95th=[1451230], 99.99th=[2499806] bw ( KiB/s): min=29184, max=70802, per=100.00%, avg=38562.60, stdev=5326.22, samples=60 iops : min= 1824, max= 4425, avg=2410.13, stdev=332.88, samples=60 write: IOPS=600, BW=9609KiB/s (9839kB/s)(283MiB/30134msec) slat (usec): min=4, max=262, avg=12.94, stdev= 5.88 clat (msec): min=7, max=2160, avg=74.76, stdev=52.81 lat (msec): min=7, max=2160, avg=74.77, stdev=52.81 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 51], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 67], 50.00th=[ 69], 60.00th=[ 71], | 70.00th=[ 74], 80.00th=[ 78], 90.00th=[ 89], 95.00th=[ 113], | 99.00th=[ 205], 99.50th=[ 368], 99.90th=[ 827], 99.95th=[ 944], | 99.99th=[ 1888] bw ( KiB/s): min= 7951, max=16798, per=100.00%, avg=9628.87, stdev=1156.51, samples=60 iops : min= 496, max= 1049, avg=601.77, stdev=72.21, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.08%, 4=0.01%, 10=0.03%, 20=0.12%, 50=62.66% lat (usec) : 100=14.56%, 250=1.65%, 500=0.42%, 750=0.15%, 1000=0.06% lat (msec) : 2=0.03%, 4=0.01%, 10=0.06%, 20=0.18%, 50=0.73% lat (msec) : 100=17.68%, 250=1.25%, 500=0.13%, 750=0.06%, 1000=0.03% lat (msec) : 2000=0.05%, >=2000=0.02% cpu : usr=1.30%, sys=5.92%, ctx=87636, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=72306,18097,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=1130MiB (1185MB), run=30134-30134msec WRITE: bw=9609KiB/s (9839kB/s), 9609KiB/s-9609KiB/s (9839kB/s-9839kB/s), io=283MiB (297MB), run=30134-30134msec Disk stats (read/write): vdd: ios=72302/18096, merge=0/0, ticks=142722/1340135, in_queue=1500648, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904242: Fri Jan 7 13:43:12 2022 read: IOPS=2297, BW=35.9MiB/s (37.6MB/s)(1077MiB/30001msec) clat (usec): min=25, max=60376, avg=80.77, stdev=675.16 lat (usec): min=25, max=60377, avg=80.91, stdev=675.16 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 94], 99.50th=[ 114], 99.90th=[13173], 99.95th=[15664], | 99.99th=[22676] bw ( KiB/s): min=29984, max=60256, per=100.00%, avg=36775.73, stdev=4590.03, samples=59 iops : min= 1874, max= 3766, avg=2298.47, stdev=286.88, samples=59 write: IOPS=574, BW=9188KiB/s (9409kB/s)(269MiB/30001msec) clat (usec): min=213, max=51979, avg=1407.86, stdev=1633.48 lat (usec): min=215, max=51980, avg=1409.34, stdev=1633.46 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 469], 10.00th=[ 725], 20.00th=[ 996], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2073], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[17695], 99.95th=[40109], | 99.99th=[44303] bw ( KiB/s): min= 7648, max=15520, per=100.00%, avg=9201.10, stdev=1122.01, samples=59 iops : min= 478, max= 970, avg=575.07, stdev=70.13, samples=59 lat (usec) : 50=43.98%, 100=35.41%, 250=0.43%, 500=1.10%, 750=1.05% lat (usec) : 1000=1.93% lat (msec) : 2=14.78%, 4=0.79%, 10=0.33%, 20=0.18%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.50%, sys=3.64%, ctx=86167, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68930,17229,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=35.9MiB/s (37.6MB/s), 35.9MiB/s-35.9MiB/s (37.6MB/s-37.6MB/s), io=1077MiB (1129MB), run=30001-30001msec WRITE: bw=9188KiB/s (9409kB/s), 9188KiB/s-9188KiB/s (9409kB/s-9409kB/s), io=269MiB (282MB), run=30001-30001msec Disk stats (read/write): vdd: ios=68547/17156, merge=0/0, ticks=1941/16903, in_queue=28275, util=94.41% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905214: Fri Jan 7 13:43:49 2022 read: IOPS=2422, BW=37.8MiB/s (39.7MB/s)(1143MiB/30205msec) slat (usec): min=2, max=283, avg= 9.84, stdev= 4.88 clat (nsec): min=704, max=2516.8M, avg=2307903.98, stdev=59176353.75 lat (usec): min=19, max=2516.8k, avg=2317.88, stdev=59176.35 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 47], | 80.00th=[ 53], 90.00th=[ 74], 95.00th=[ 137], | 99.00th=[ 1090], 99.50th=[ 1926], 99.90th=[ 868221], | 99.95th=[1585447], 99.99th=[2499806] bw ( KiB/s): min=31264, max=95232, per=100.00%, avg=39024.87, stdev=8190.60, samples=60 iops : min= 1954, max= 5952, avg=2439.03, stdev=511.91, samples=60 write: IOPS=605, BW=9687KiB/s (9920kB/s)(286MiB/30205msec) slat (usec): min=3, max=275, avg=12.81, stdev= 5.79 clat (msec): min=4, max=1917, avg=155.84, stdev=55.97 lat (msec): min=4, max=1917, avg=155.85, stdev=55.97 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 109], 10.00th=[ 131], 20.00th=[ 138], | 30.00th=[ 142], 40.00th=[ 146], 50.00th=[ 150], 60.00th=[ 155], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 194], 95.00th=[ 211], | 99.00th=[ 317], 99.50th=[ 485], 99.90th=[ 735], 99.95th=[ 885], | 99.99th=[ 1770] bw ( KiB/s): min= 8064, max=21344, per=100.00%, avg=9703.88, stdev=1664.77, samples=60 iops : min= 504, max= 1334, avg=606.48, stdev=104.05, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.16%, 50=60.87% lat (usec) : 100=13.17%, 250=3.19%, 500=0.99%, 750=0.46%, 1000=0.18% lat (msec) : 2=0.48%, 4=0.17%, 10=0.01%, 20=0.11%, 50=0.44% lat (msec) : 100=0.34%, 250=18.65%, 500=0.44%, 750=0.10%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.03% cpu : usr=1.38%, sys=5.68%, ctx=85389, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73169,18288,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.7MB/s), 37.8MiB/s-37.8MiB/s (39.7MB/s-39.7MB/s), io=1143MiB (1199MB), run=30205-30205msec WRITE: bw=9687KiB/s (9920kB/s), 9687KiB/s-9687KiB/s (9920kB/s-9920kB/s), io=286MiB (300MB), run=30205-30205msec Disk stats (read/write): vdd: ios=73164/18238, merge=0/0, ticks=160605/2821811, in_queue=3004746, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905792: Fri Jan 7 13:44:26 2022 read: IOPS=2289, BW=35.8MiB/s (37.5MB/s)(1073MiB/30001msec) clat (usec): min=26, max=57885, avg=79.57, stdev=673.73 lat (usec): min=26, max=57885, avg=79.71, stdev=673.73 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 88], 99.50th=[ 104], 99.90th=[12911], 99.95th=[15270], | 99.99th=[22938] bw ( KiB/s): min=26741, max=62080, per=100.00%, avg=36715.27, stdev=5138.43, samples=59 iops : min= 1671, max= 3880, avg=2294.69, stdev=321.16, samples=59 write: IOPS=572, BW=9164KiB/s (9384kB/s)(269MiB/30001msec) clat (usec): min=213, max=63536, avg=1417.94, stdev=1680.44 lat (usec): min=214, max=63538, avg=1419.36, stdev=1680.43 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 461], 10.00th=[ 725], 20.00th=[ 1004], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1762], 95.00th=[ 2073], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[33817], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min= 7168, max=15936, per=100.00%, avg=9178.41, stdev=1224.93, samples=59 iops : min= 448, max= 996, avg=573.64, stdev=76.55, samples=59 lat (usec) : 50=46.99%, 100=32.54%, 250=0.28%, 500=1.12%, 750=0.99% lat (usec) : 1000=1.89% lat (msec) : 2=14.85%, 4=0.78%, 10=0.36%, 20=0.17%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.32%, sys=3.68%, ctx=85879, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68688,17184,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=35.8MiB/s (37.5MB/s), 35.8MiB/s-35.8MiB/s (37.5MB/s-37.5MB/s), io=1073MiB (1125MB), run=30001-30001msec WRITE: bw=9164KiB/s (9384kB/s), 9164KiB/s-9164KiB/s (9384kB/s-9384kB/s), io=269MiB (282MB), run=30001-30001msec Disk stats (read/write): vdd: ios=68304/17088, merge=0/0, ticks=1950/17031, in_queue=28306, util=94.49% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906613: Fri Jan 7 13:45:03 2022 read: IOPS=2421, BW=37.8MiB/s (39.7MB/s)(1142MiB/30190msec) slat (usec): min=2, max=184572, avg=322.61, stdev=1850.87 clat (usec): min=575, max=2630.2k, avg=27826.32, stdev=60245.43 lat (usec): min=577, max=2630.2k, avg=28149.06, stdev=60279.73 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 8586], 10.00th=[ 14484], | 20.00th=[ 18220], 30.00th=[ 20317], 40.00th=[ 22414], | 50.00th=[ 23987], 60.00th=[ 26084], 70.00th=[ 28181], | 80.00th=[ 31065], 90.00th=[ 35914], 95.00th=[ 42206], | 99.00th=[ 82314], 99.50th=[ 181404], 99.90th=[ 834667], | 99.95th=[1468007], 99.99th=[2533360] bw ( KiB/s): min=29280, max=82944, per=100.00%, avg=38951.35, stdev=6842.10, samples=60 iops : min= 1830, max= 5184, avg=2434.45, stdev=427.63, samples=60 write: IOPS=605, BW=9684KiB/s (9917kB/s)(286MiB/30190msec) slat (usec): min=3, max=173672, avg=342.00, stdev=2295.18 clat (msec): min=5, max=2222, avg=217.31, stdev=81.78 lat (msec): min=5, max=2222, avg=217.65, stdev=81.76 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 150], 10.00th=[ 178], 20.00th=[ 190], | 30.00th=[ 199], 40.00th=[ 203], 50.00th=[ 209], 60.00th=[ 215], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 259], 95.00th=[ 279], | 99.00th=[ 527], 99.50th=[ 735], 99.90th=[ 1083], 99.95th=[ 1200], | 99.99th=[ 2232] bw ( KiB/s): min= 8032, max=17184, per=99.91%, avg=9675.28, stdev=1203.80, samples=60 iops : min= 502, max= 1074, avg=604.70, stdev=75.24, samples=60 lat (usec) : 750=0.14%, 1000=0.48% lat (msec) : 2=1.30%, 4=1.26%, 10=1.14%, 20=18.04%, 50=55.47% lat (msec) : 100=2.03%, 250=17.24%, 500=2.58%, 750=0.16%, 1000=0.05% lat (msec) : 2000=0.09%, >=2000=0.03% cpu : usr=0.79%, sys=1.94%, ctx=10305, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73096,18273,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.7MB/s), 37.8MiB/s-37.8MiB/s (39.7MB/s-39.7MB/s), io=1142MiB (1198MB), run=30190-30190msec WRITE: bw=9684KiB/s (9917kB/s), 9684KiB/s-9684KiB/s (9917kB/s-9917kB/s), io=286MiB (299MB), run=30190-30190msec Disk stats (read/write): vdd: ios=73096/18202, merge=0/0, ticks=158808/3481835, in_queue=3669931, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908060: Fri Jan 7 13:45:40 2022 read: IOPS=2209, BW=69.1MiB/s (72.4MB/s)(2072MiB/30001msec) clat (usec): min=31, max=24405, avg=84.99, stdev=644.81 lat (usec): min=31, max=24406, avg=85.13, stdev=644.81 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 63], 90.00th=[ 70], 95.00th=[ 77], | 99.00th=[ 97], 99.50th=[ 116], 99.90th=[13960], 99.95th=[15795], | 99.99th=[20055] bw ( KiB/s): min=55872, max=116160, per=99.84%, avg=70600.20, stdev=9822.81, samples=59 iops : min= 1746, max= 3630, avg=2206.24, stdev=306.95, samples=59 write: IOPS=553, BW=17.3MiB/s (18.1MB/s)(518MiB/30001msec) clat (usec): min=255, max=46301, avg=1457.25, stdev=1689.39 lat (usec): min=258, max=46304, avg=1460.37, stdev=1689.37 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 506], 10.00th=[ 734], 20.00th=[ 1012], | 30.00th=[ 1156], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1811], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min=14528, max=30016, per=100.00%, avg=17707.56, stdev=2304.77, samples=59 iops : min= 454, max= 938, avg=553.36, stdev=72.02, samples=59 lat (usec) : 50=28.93%, 100=50.38%, 250=0.47%, 500=0.99%, 750=1.14% lat (usec) : 1000=1.76% lat (msec) : 2=14.88%, 4=0.90%, 10=0.31%, 20=0.20%, 50=0.03% cpu : usr=1.41%, sys=3.73%, ctx=82893, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66295,16591,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=69.1MiB/s (72.4MB/s), 69.1MiB/s-69.1MiB/s (72.4MB/s-72.4MB/s), io=2072MiB (2172MB), run=30001-30001msec WRITE: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=518MiB (544MB), run=30001-30001msec Disk stats (read/write): vdd: ios=65928/16506, merge=0/0, ticks=1865/16742, in_queue=28249, util=94.30% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908672: Fri Jan 7 13:46:17 2022 read: IOPS=2225, BW=69.5MiB/s (72.9MB/s)(2086MiB/30002msec) slat (usec): min=6, max=151, avg=11.34, stdev= 4.48 clat (nsec): min=747, max=25261k, avg=70425.24, stdev=626448.01 lat (usec): min=26, max=25270, avg=81.92, stdev=626.48 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 38], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 45], 80.00th=[ 48], 90.00th=[ 54], 95.00th=[ 60], | 99.00th=[ 76], 99.50th=[ 91], 99.90th=[12780], 99.95th=[15401], | 99.99th=[19530] bw ( KiB/s): min=55296, max=122496, per=100.00%, avg=71209.68, stdev=8951.90, samples=60 iops : min= 1728, max= 3828, avg=2225.30, stdev=279.75, samples=60 write: IOPS=556, BW=17.4MiB/s (18.2MB/s)(522MiB/30002msec) slat (usec): min=9, max=131, avg=15.07, stdev= 4.76 clat (usec): min=241, max=47967, avg=1445.42, stdev=1658.75 lat (usec): min=253, max=47982, avg=1460.65, stdev=1658.82 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 490], 10.00th=[ 750], 20.00th=[ 1029], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1811], 95.00th=[ 2089], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=13504, max=31552, per=100.00%, avg=17822.42, stdev=2329.10, samples=60 iops : min= 422, max= 986, avg=556.95, stdev=72.78, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.09%, 4=0.01%, 10=0.02%, 20=0.09%, 50=67.49% lat (usec) : 100=11.92%, 250=0.11%, 500=1.06%, 750=0.97%, 1000=1.65% lat (msec) : 2=15.13%, 4=0.87%, 10=0.31%, 20=0.20%, 50=0.03% cpu : usr=1.39%, sys=5.95%, ctx=83297, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66757,16709,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=69.5MiB/s (72.9MB/s), 69.5MiB/s-69.5MiB/s (72.9MB/s-72.9MB/s), io=2086MiB (2187MB), run=30002-30002msec WRITE: bw=17.4MiB/s (18.2MB/s), 17.4MiB/s-17.4MiB/s (18.2MB/s-18.2MB/s), io=522MiB (548MB), run=30002-30002msec Disk stats (read/write): vdd: ios=66451/16632, merge=0/0, ticks=1815/16949, in_queue=28146, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909510: Fri Jan 7 13:46:54 2022 read: IOPS=2208, BW=69.0MiB/s (72.4MB/s)(2070MiB/30001msec) clat (usec): min=28, max=66277, avg=81.42, stdev=683.11 lat (usec): min=28, max=66277, avg=81.56, stdev=683.11 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 93], 99.50th=[ 112], 99.90th=[13566], 99.95th=[15664], | 99.99th=[19268] bw ( KiB/s): min=41152, max=106496, per=99.93%, avg=70614.44, stdev=9851.19, samples=59 iops : min= 1286, max= 3328, avg=2206.69, stdev=307.85, samples=59 write: IOPS=552, BW=17.3MiB/s (18.1MB/s)(518MiB/30001msec) clat (usec): min=248, max=105142, avg=1472.51, stdev=1843.04 lat (usec): min=251, max=105145, avg=1475.57, stdev=1843.01 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 498], 10.00th=[ 766], 20.00th=[ 1057], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=11136, max=27776, per=100.00%, avg=17714.05, stdev=2276.48, samples=59 iops : min= 348, max= 868, avg=553.56, stdev=71.14, samples=59 lat (usec) : 50=42.96%, 100=36.44%, 250=0.38%, 500=1.02%, 750=0.93% lat (usec) : 1000=1.50% lat (msec) : 2=15.30%, 4=0.93%, 10=0.32%, 20=0.20%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.40%, sys=3.59%, ctx=82840, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66253,16582,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=69.0MiB/s (72.4MB/s), 69.0MiB/s-69.0MiB/s (72.4MB/s-72.4MB/s), io=2070MiB (2171MB), run=30001-30001msec WRITE: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=518MiB (543MB), run=30001-30001msec Disk stats (read/write): vdd: ios=65937/16513, merge=0/0, ticks=1908/17106, in_queue=28322, util=94.61% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910036: Fri Jan 7 13:47:31 2022 read: IOPS=2342, BW=73.2MiB/s (76.8MB/s)(2199MiB/30039msec) slat (usec): min=3, max=243, avg=11.21, stdev= 4.04 clat (nsec): min=724, max=2492.5M, avg=1770687.30, stdev=48996764.56 lat (usec): min=29, max=2492.5k, avg=1782.05, stdev=48996.74 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 45], 70.00th=[ 48], | 80.00th=[ 51], 90.00th=[ 59], 95.00th=[ 67], | 99.00th=[ 100], 99.50th=[ 145], 99.90th=[ 666895], | 99.95th=[1061159], 99.99th=[2499806] bw ( KiB/s): min=63040, max=135936, per=100.00%, avg=75069.97, stdev=9816.39, samples=60 iops : min= 1970, max= 4248, avg=2345.90, stdev=306.76, samples=60 write: IOPS=586, BW=18.3MiB/s (19.2MB/s)(550MiB/30039msec) slat (usec): min=6, max=132, avg=15.28, stdev= 4.58 clat (usec): min=394, max=72028, avg=9909.55, stdev=6382.51 lat (usec): min=405, max=72049, avg=9924.98, stdev=6382.91 clat percentiles (usec): | 1.00th=[ 1123], 5.00th=[ 2114], 10.00th=[ 2966], 20.00th=[ 5014], | 30.00th=[ 6456], 40.00th=[ 8225], 50.00th=[ 9896], 60.00th=[11338], | 70.00th=[12387], 80.00th=[13304], 90.00th=[15008], 95.00th=[18482], | 99.00th=[25297], 99.50th=[56361], 99.90th=[66323], 99.95th=[70779], | 99.99th=[71828] bw ( KiB/s): min=15680, max=34304, per=100.00%, avg=18780.58, stdev=2352.69, samples=60 iops : min= 490, max= 1072, avg=586.88, stdev=73.52, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.04%, 50=61.85% lat (usec) : 100=17.24%, 250=0.56%, 500=0.04%, 750=0.05%, 1000=0.08% lat (msec) : 2=0.83%, 4=1.98%, 10=7.22%, 20=9.10%, 50=0.68% lat (msec) : 100=0.13%, 250=0.03%, 500=0.03%, 750=0.02%, 1000=0.03% lat (msec) : 2000=0.03%, >=2000=0.02% cpu : usr=1.55%, sys=6.07%, ctx=86927, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70375,17611,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=73.2MiB/s (76.8MB/s), 73.2MiB/s-73.2MiB/s (76.8MB/s-76.8MB/s), io=2199MiB (2306MB), run=30039-30039msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=550MiB (577MB), run=30039-30039msec Disk stats (read/write): vdd: ios=70010/17524, merge=0/0, ticks=114985/165219, in_queue=296783, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910900: Fri Jan 7 13:48:08 2022 read: IOPS=2209, BW=69.1MiB/s (72.4MB/s)(2072MiB/30002msec) clat (usec): min=29, max=60334, avg=84.55, stdev=693.13 lat (usec): min=29, max=60334, avg=84.69, stdev=693.13 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 75], | 99.00th=[ 100], 99.50th=[ 125], 99.90th=[13698], 99.95th=[15664], | 99.99th=[21627] bw ( KiB/s): min=54336, max=112192, per=100.00%, avg=70715.15, stdev=9152.67, samples=60 iops : min= 1698, max= 3506, avg=2209.83, stdev=286.00, samples=60 write: IOPS=553, BW=17.3MiB/s (18.1MB/s)(519MiB/30002msec) clat (usec): min=252, max=43739, avg=1458.45, stdev=1627.99 lat (usec): min=255, max=43742, avg=1461.49, stdev=1627.96 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 502], 10.00th=[ 750], 20.00th=[ 1029], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1811], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[17695], 99.95th=[39060], | 99.99th=[42730] bw ( KiB/s): min=14400, max=28608, per=100.00%, avg=17699.60, stdev=2064.49, samples=60 iops : min= 450, max= 894, avg=553.10, stdev=64.50, samples=60 lat (usec) : 50=35.86%, 100=43.31%, 250=0.58%, 500=1.03%, 750=1.03% lat (usec) : 1000=1.65% lat (msec) : 2=15.07%, 4=0.92%, 10=0.33%, 20=0.19%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.61%, sys=3.63%, ctx=82900, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66298,16595,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=69.1MiB/s (72.4MB/s), 69.1MiB/s-69.1MiB/s (72.4MB/s-72.4MB/s), io=2072MiB (2172MB), run=30002-30002msec WRITE: bw=17.3MiB/s (18.1MB/s), 17.3MiB/s-17.3MiB/s (18.1MB/s-18.1MB/s), io=519MiB (544MB), run=30002-30002msec Disk stats (read/write): vdd: ios=65923/16505, merge=0/0, ticks=1959/16811, in_queue=28216, util=94.13% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911954: Fri Jan 7 13:48:45 2022 read: IOPS=2328, BW=72.8MiB/s (76.3MB/s)(2192MiB/30130msec) slat (usec): min=2, max=389, avg=10.86, stdev= 4.51 clat (nsec): min=686, max=2552.8M, avg=2319472.79, stdev=59452056.30 lat (usec): min=24, max=2552.8k, avg=2330.48, stdev=59452.06 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 37], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 44], 70.00th=[ 47], | 80.00th=[ 52], 90.00th=[ 61], 95.00th=[ 74], | 99.00th=[ 269], 99.50th=[ 562], 99.90th=[ 910164], | 99.95th=[1518339], 99.99th=[2499806] bw ( KiB/s): min=56576, max=136320, per=100.00%, avg=74813.58, stdev=10788.83, samples=60 iops : min= 1768, max= 4260, avg=2337.90, stdev=337.15, samples=60 write: IOPS=583, BW=18.2MiB/s (19.1MB/s)(549MiB/30130msec) slat (usec): min=5, max=223, avg=16.00, stdev= 5.82 clat (msec): min=3, max=1829, avg=76.42, stdev=49.56 lat (msec): min=3, max=1829, avg=76.43, stdev=49.56 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 93], 95.00th=[ 115], | 99.00th=[ 180], 99.50th=[ 359], 99.90th=[ 860], 99.95th=[ 1053], | 99.99th=[ 1351] bw ( KiB/s): min=14528, max=31744, per=100.00%, avg=18703.13, stdev=2271.49, samples=60 iops : min= 454, max= 992, avg=584.47, stdev=70.98, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.07%, 50=61.67% lat (usec) : 100=16.03%, 250=1.26%, 500=0.40%, 750=0.16%, 1000=0.07% lat (msec) : 2=0.03%, 4=0.01%, 10=0.03%, 20=0.30%, 50=0.34% lat (msec) : 100=17.92%, 250=1.33%, 500=0.11%, 750=0.06%, 1000=0.04% lat (msec) : 2000=0.05%, >=2000=0.03% cpu : usr=1.56%, sys=5.93%, ctx=85256, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=70150,17566,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=72.8MiB/s (76.3MB/s), 72.8MiB/s-72.8MiB/s (76.3MB/s-76.3MB/s), io=2192MiB (2299MB), run=30130-30130msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=549MiB (576MB), run=30130-30130msec Disk stats (read/write): vdd: ios=70147/17566, merge=0/0, ticks=154508/1330218, in_queue=1500542, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912585: Fri Jan 7 13:49:22 2022 read: IOPS=2235, BW=69.9MiB/s (73.3MB/s)(2097MiB/30011msec) clat (usec): min=26, max=28186, avg=80.58, stdev=633.26 lat (usec): min=26, max=28186, avg=80.72, stdev=633.26 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 95], 99.50th=[ 117], 99.90th=[13173], 99.95th=[15401], | 99.99th=[18744] bw ( KiB/s): min=53888, max=150656, per=100.00%, avg=71586.90, stdev=12488.81, samples=60 iops : min= 1684, max= 4708, avg=2237.05, stdev=390.28, samples=60 write: IOPS=559, BW=17.5MiB/s (18.3MB/s)(525MiB/30011msec) clat (usec): min=252, max=46421, avg=1453.90, stdev=1651.91 lat (usec): min=255, max=46424, avg=1456.91, stdev=1651.92 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 433], 10.00th=[ 709], 20.00th=[ 1029], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1811], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min=13120, max=39168, per=100.00%, avg=17913.30, stdev=3263.76, samples=60 iops : min= 410, max= 1224, avg=559.77, stdev=102.00, samples=60 lat (usec) : 50=43.00%, 100=36.34%, 250=0.40%, 500=1.28%, 750=0.94% lat (usec) : 1000=1.55% lat (msec) : 2=15.04%, 4=0.88%, 10=0.34%, 20=0.19%, 50=0.03% cpu : usr=1.29%, sys=3.79%, ctx=83897, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67103,16791,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=69.9MiB/s (73.3MB/s), 69.9MiB/s-69.9MiB/s (73.3MB/s-73.3MB/s), io=2097MiB (2199MB), run=30011-30011msec WRITE: bw=17.5MiB/s (18.3MB/s), 17.5MiB/s-17.5MiB/s (18.3MB/s-18.3MB/s), io=525MiB (550MB), run=30011-30011msec Disk stats (read/write): vdd: ios=66743/16703, merge=0/0, ticks=1852/17025, in_queue=28326, util=94.56% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913419: Fri Jan 7 13:49:59 2022 read: IOPS=2345, BW=73.3MiB/s (76.9MB/s)(2217MiB/30238msec) slat (usec): min=2, max=279, avg=11.31, stdev= 4.84 clat (nsec): min=797, max=2517.3M, avg=1950424.98, stdev=49729356.54 lat (usec): min=26, max=2517.3k, avg=1961.88, stdev=49729.34 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 52], | 80.00th=[ 58], 90.00th=[ 71], 95.00th=[ 113], | 99.00th=[ 1336], 99.50th=[ 2245], 99.90th=[ 750781], | 99.95th=[1199571], 99.99th=[2499806] bw ( KiB/s): min=56384, max=151680, per=100.00%, avg=75652.27, stdev=12022.10, samples=60 iops : min= 1762, max= 4740, avg=2364.13, stdev=375.69, samples=60 write: IOPS=586, BW=18.3MiB/s (19.2MB/s)(554MiB/30238msec) slat (usec): min=5, max=226, avg=16.75, stdev= 6.40 clat (msec): min=18, max=1826, avg=162.53, stdev=79.58 lat (msec): min=18, max=1826, avg=162.54, stdev=79.58 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 121], 10.00th=[ 130], 20.00th=[ 140], | 30.00th=[ 146], 40.00th=[ 150], 50.00th=[ 155], 60.00th=[ 159], | 70.00th=[ 163], 80.00th=[ 169], 90.00th=[ 194], 95.00th=[ 220], | 99.00th=[ 380], 99.50th=[ 693], 99.90th=[ 1418], 99.95th=[ 1418], | 99.99th=[ 1821] bw ( KiB/s): min=14976, max=33088, per=100.00%, avg=18830.93, stdev=2300.51, samples=60 iops : min= 468, max= 1034, avg=588.47, stdev=71.89, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.05%, 50=53.58% lat (usec) : 100=21.89%, 250=1.86%, 500=0.83%, 750=0.46%, 1000=0.24% lat (msec) : 2=0.52%, 4=0.31%, 20=0.05%, 50=0.34%, 100=0.25% lat (msec) : 250=18.75%, 500=0.54%, 750=0.10%, 1000=0.07%, 2000=0.08% lat (msec) : >=2000=0.01% cpu : usr=1.49%, sys=6.19%, ctx=83535, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70935,17743,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=73.3MiB/s (76.9MB/s), 73.3MiB/s-73.3MiB/s (76.9MB/s-76.9MB/s), io=2217MiB (2324MB), run=30238-30238msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=554MiB (581MB), run=30238-30238msec Disk stats (read/write): vdd: ios=70924/17673, merge=0/0, ticks=124133/2849147, in_queue=3006452, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914731: Fri Jan 7 13:50:36 2022 read: IOPS=2218, BW=69.3MiB/s (72.7MB/s)(2080MiB/30011msec) clat (usec): min=28, max=64205, avg=82.88, stdev=735.46 lat (usec): min=28, max=64205, avg=83.01, stdev=735.46 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 90], 99.50th=[ 105], 99.90th=[13435], 99.95th=[15795], | 99.99th=[22938] bw ( KiB/s): min=52224, max=119552, per=100.00%, avg=71009.07, stdev=9527.25, samples=60 iops : min= 1632, max= 3736, avg=2219.03, stdev=297.73, samples=60 write: IOPS=555, BW=17.4MiB/s (18.2MB/s)(521MiB/30011msec) clat (usec): min=253, max=80707, avg=1458.27, stdev=1717.55 lat (usec): min=256, max=80710, avg=1461.28, stdev=1717.53 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 502], 10.00th=[ 766], 20.00th=[ 1057], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1287], 60.00th=[ 1352], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1811], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[28967], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=12096, max=30784, per=100.00%, avg=17779.20, stdev=2360.02, samples=60 iops : min= 378, max= 962, avg=555.60, stdev=73.75, samples=60 lat (usec) : 50=41.89%, 100=37.61%, 250=0.29%, 500=0.99%, 750=0.92% lat (usec) : 1000=1.54% lat (msec) : 2=15.32%, 4=0.90%, 10=0.31%, 20=0.20%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.37%, sys=3.71%, ctx=83248, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66572,16668,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=69.3MiB/s (72.7MB/s), 69.3MiB/s-69.3MiB/s (72.7MB/s-72.7MB/s), io=2080MiB (2181MB), run=30011-30011msec WRITE: bw=17.4MiB/s (18.2MB/s), 17.4MiB/s-17.4MiB/s (18.2MB/s-18.2MB/s), io=521MiB (546MB), run=30011-30011msec Disk stats (read/write): vdd: ios=66258/16584, merge=0/0, ticks=1980/17062, in_queue=28268, util=94.41% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915266: Fri Jan 7 13:51:13 2022 read: IOPS=2326, BW=72.7MiB/s (76.2MB/s)(2198MiB/30228msec) slat (usec): min=2, max=247793, avg=342.37, stdev=2304.06 clat (usec): min=792, max=2543.8k, avg=29130.16, stdev=62219.10 lat (usec): min=794, max=2543.0k, avg=29472.67, stdev=62264.37 clat percentiles (usec): | 1.00th=[ 1254], 5.00th=[ 7373], 10.00th=[ 15139], | 20.00th=[ 19006], 30.00th=[ 21365], 40.00th=[ 23200], | 50.00th=[ 25035], 60.00th=[ 26870], 70.00th=[ 29230], | 80.00th=[ 32113], 90.00th=[ 36963], 95.00th=[ 42730], | 99.00th=[ 89654], 99.50th=[ 219153], 99.90th=[1098908], | 99.95th=[1568670], 99.99th=[2499806] bw ( KiB/s): min=53440, max=156032, per=100.00%, avg=74936.77, stdev=13136.42, samples=60 iops : min= 1670, max= 4876, avg=2341.77, stdev=410.52, samples=60 write: IOPS=582, BW=18.2MiB/s (19.1MB/s)(550MiB/30228msec) slat (usec): min=4, max=56198, avg=327.14, stdev=1435.79 clat (msec): min=19, max=1867, avg=225.34, stdev=104.20 lat (msec): min=19, max=1867, avg=225.67, stdev=104.20 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 157], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 201], 40.00th=[ 207], 50.00th=[ 213], 60.00th=[ 220], | 70.00th=[ 228], 80.00th=[ 239], 90.00th=[ 268], 95.00th=[ 309], | 99.00th=[ 693], 99.50th=[ 961], 99.90th=[ 1653], 99.95th=[ 1770], | 99.99th=[ 1871] bw ( KiB/s): min=13440, max=32704, per=100.00%, avg=18643.80, stdev=2488.19, samples=60 iops : min= 420, max= 1022, avg=582.60, stdev=77.78, samples=60 lat (usec) : 1000=0.24% lat (msec) : 2=1.75%, 4=1.37%, 10=0.97%, 20=15.17%, 50=58.44% lat (msec) : 100=1.83%, 250=16.95%, 500=2.86%, 750=0.16%, 1000=0.12% lat (msec) : 2000=0.13%, >=2000=0.03% cpu : usr=1.14%, sys=2.04%, ctx=9904, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70323,17601,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=72.7MiB/s (76.2MB/s), 72.7MiB/s-72.7MiB/s (76.2MB/s-76.2MB/s), io=2198MiB (2304MB), run=30228-30228msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=550MiB (577MB), run=30228-30228msec Disk stats (read/write): vdd: ios=70308/17573, merge=0/0, ticks=158235/3479358, in_queue=3676193, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916221: Fri Jan 7 13:51:50 2022 read: IOPS=2078, BW=130MiB/s (136MB/s)(3897MiB/30001msec) clat (usec): min=32, max=27577, avg=89.52, stdev=649.70 lat (usec): min=32, max=27578, avg=89.66, stdev=649.70 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 76], 95.00th=[ 84], | 99.00th=[ 115], 99.50th=[ 157], 99.90th=[13698], 99.95th=[16057], | 99.99th=[18220] bw ( KiB/s): min=95232, max=212480, per=100.00%, avg=133002.67, stdev=17824.37, samples=60 iops : min= 1488, max= 3320, avg=2078.17, stdev=278.51, samples=60 write: IOPS=522, BW=32.6MiB/s (34.2MB/s)(979MiB/30001msec) clat (usec): min=335, max=43459, avg=1543.68, stdev=1641.19 lat (usec): min=345, max=43464, avg=1550.21, stdev=1641.11 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 586], 10.00th=[ 799], 20.00th=[ 1106], | 30.00th=[ 1254], 40.00th=[ 1336], 50.00th=[ 1385], 60.00th=[ 1450], | 70.00th=[ 1549], 80.00th=[ 1680], 90.00th=[ 1909], 95.00th=[ 2180], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[34866], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min=27136, max=55168, per=100.00%, avg=33410.13, stdev=4052.82, samples=60 iops : min= 424, max= 862, avg=522.03, stdev=63.33, samples=60 lat (usec) : 50=18.03%, 100=60.44%, 250=1.16%, 500=0.73%, 750=1.17% lat (usec) : 1000=1.25% lat (msec) : 2=15.46%, 4=1.21%, 10=0.30%, 20=0.22%, 50=0.03% cpu : usr=1.67%, sys=3.71%, ctx=78019, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62345,15662,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3897MiB (4086MB), run=30001-30001msec WRITE: bw=32.6MiB/s (34.2MB/s), 32.6MiB/s-32.6MiB/s (34.2MB/s-34.2MB/s), io=979MiB (1026MB), run=30001-30001msec Disk stats (read/write): vdd: ios=62082/15595, merge=0/0, ticks=1772/16591, in_queue=28202, util=94.12% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916837: Fri Jan 7 13:52:27 2022 read: IOPS=2061, BW=129MiB/s (135MB/s)(3867MiB/30009msec) slat (usec): min=6, max=123, avg=13.46, stdev= 4.38 clat (nsec): min=744, max=22955k, avg=78633.00, stdev=617597.50 lat (usec): min=34, max=22965, avg=92.25, stdev=617.62 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 71], | 99.00th=[ 92], 99.50th=[ 114], 99.90th=[13042], 99.95th=[15270], | 99.99th=[17957] bw ( KiB/s): min=105216, max=212352, per=100.00%, avg=131993.60, stdev=15890.04, samples=60 iops : min= 1644, max= 3318, avg=2062.40, stdev=248.28, samples=60 write: IOPS=517, BW=32.3MiB/s (33.9MB/s)(970MiB/30009msec) slat (usec): min=11, max=178, avg=20.28, stdev= 5.59 clat (usec): min=316, max=49175, avg=1537.26, stdev=1715.74 lat (usec): min=332, max=49197, avg=1557.71, stdev=1715.78 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 570], 10.00th=[ 758], 20.00th=[ 1074], | 30.00th=[ 1221], 40.00th=[ 1303], 50.00th=[ 1369], 60.00th=[ 1434], | 70.00th=[ 1532], 80.00th=[ 1663], 90.00th=[ 1893], 95.00th=[ 2180], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36439], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=27776, max=55040, per=100.00%, avg=33113.60, stdev=3899.37, samples=60 iops : min= 434, max= 860, avg=517.40, stdev=60.93, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.03%, 50=46.74% lat (usec) : 100=32.54%, 250=0.35%, 500=0.75%, 750=1.26%, 1000=1.35% lat (msec) : 2=15.22%, 4=1.13%, 10=0.32%, 20=0.22%, 50=0.03% cpu : usr=1.48%, sys=6.20%, ctx=77361, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61872,15523,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3867MiB (4055MB), run=30009-30009msec WRITE: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=970MiB (1017MB), run=30009-30009msec Disk stats (read/write): vdd: ios=61574/15459, merge=0/0, ticks=1674/16501, in_queue=27980, util=93.40% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917781: Fri Jan 7 13:53:04 2022 read: IOPS=2062, BW=129MiB/s (135MB/s)(3867MiB/30001msec) clat (usec): min=37, max=33377, avg=94.55, stdev=667.01 lat (usec): min=37, max=33377, avg=94.69, stdev=667.01 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 80], 95.00th=[ 88], | 99.00th=[ 119], 99.50th=[ 151], 99.90th=[13960], 99.95th=[15664], | 99.99th=[20317] bw ( KiB/s): min=96256, max=226816, per=99.78%, avg=131695.46, stdev=17314.03, samples=59 iops : min= 1504, max= 3544, avg=2057.76, stdev=270.55, samples=59 write: IOPS=517, BW=32.3MiB/s (33.9MB/s)(970MiB/30001msec) clat (usec): min=347, max=50600, avg=1540.74, stdev=1677.51 lat (usec): min=353, max=50603, avg=1547.08, stdev=1677.48 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 586], 10.00th=[ 766], 20.00th=[ 1074], | 30.00th=[ 1237], 40.00th=[ 1319], 50.00th=[ 1385], 60.00th=[ 1467], | 70.00th=[ 1549], 80.00th=[ 1680], 90.00th=[ 1909], 95.00th=[ 2180], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[34866], 99.95th=[40109], | 99.99th=[44303] bw ( KiB/s): min=24704, max=57728, per=99.99%, avg=33105.66, stdev=4245.63, samples=59 iops : min= 386, max= 902, avg=517.27, stdev=66.34, samples=59 lat (usec) : 50=4.71%, 100=73.49%, 250=1.49%, 500=0.69%, 750=1.28% lat (usec) : 1000=1.48% lat (msec) : 2=15.04%, 4=1.28%, 10=0.29%, 20=0.22%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.58%, sys=3.71%, ctx=77396, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61868,15521,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3867MiB (4055MB), run=30001-30001msec WRITE: bw=32.3MiB/s (33.9MB/s), 32.3MiB/s-32.3MiB/s (33.9MB/s-33.9MB/s), io=970MiB (1017MB), run=30001-30001msec Disk stats (read/write): vdd: ios=61499/15447, merge=0/0, ticks=1804/16259, in_queue=28225, util=94.23% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918735: Fri Jan 7 13:53:41 2022 read: IOPS=2155, BW=135MiB/s (141MB/s)(4048MiB/30042msec) slat (usec): min=3, max=253, avg=12.23, stdev= 4.49 clat (nsec): min=758, max=2491.7M, avg=1975324.10, stdev=54540827.56 lat (usec): min=29, max=2491.7k, avg=1987.70, stdev=54540.82 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 46], 70.00th=[ 49], | 80.00th=[ 53], 90.00th=[ 60], 95.00th=[ 68], | 99.00th=[ 93], 99.50th=[ 124], 99.90th=[ 759170], | 99.95th=[1233126], 99.99th=[2499806] bw ( KiB/s): min=102912, max=234880, per=100.00%, avg=138167.47, stdev=18163.24, samples=60 iops : min= 1608, max= 3670, avg=2158.87, stdev=283.80, samples=60 write: IOPS=541, BW=33.8MiB/s (35.5MB/s)(1016MiB/30042msec) slat (usec): min=7, max=202, avg=19.53, stdev= 6.07 clat (usec): min=468, max=71819, avg=10530.34, stdev=6576.54 lat (usec): min=480, max=71837, avg=10550.03, stdev=6577.06 clat percentiles (usec): | 1.00th=[ 1188], 5.00th=[ 1876], 10.00th=[ 3064], 20.00th=[ 4883], | 30.00th=[ 6915], 40.00th=[ 9110], 50.00th=[10552], 60.00th=[11863], | 70.00th=[13173], 80.00th=[14615], 90.00th=[16319], 95.00th=[19792], | 99.00th=[27395], 99.50th=[55313], 99.90th=[64226], 99.95th=[64226], | 99.99th=[71828] bw ( KiB/s): min=29184, max=59776, per=100.00%, avg=34666.67, stdev=3839.94, samples=60 iops : min= 456, max= 934, avg=541.67, stdev=60.00, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.04%, 50=59.05% lat (usec) : 100=20.12%, 250=0.41%, 500=0.04%, 750=0.04%, 1000=0.07% lat (msec) : 2=0.97%, 4=1.78%, 10=6.33%, 20=9.94%, 50=0.85% lat (msec) : 100=0.13%, 250=0.02%, 500=0.02%, 750=0.03%, 1000=0.03% lat (msec) : 2000=0.02%, >=2000=0.03% cpu : usr=1.51%, sys=5.99%, ctx=80228, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=64770,16256,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4048MiB (4245MB), run=30042-30042msec WRITE: bw=33.8MiB/s (35.5MB/s), 33.8MiB/s-33.8MiB/s (35.5MB/s-35.5MB/s), io=1016MiB (1065MB), run=30042-30042msec Disk stats (read/write): vdd: ios=64398/16171, merge=0/0, ticks=119592/162499, in_queue=296765, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919312: Fri Jan 7 13:54:18 2022 read: IOPS=2070, BW=129MiB/s (136MB/s)(3882MiB/30001msec) clat (usec): min=31, max=28128, avg=85.98, stdev=656.72 lat (usec): min=31, max=28128, avg=86.12, stdev=656.72 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 49], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 97], 99.50th=[ 115], 99.90th=[13960], 99.95th=[16057], | 99.99th=[19268] bw ( KiB/s): min=77952, max=212480, per=100.00%, avg=132507.73, stdev=19206.81, samples=60 iops : min= 1218, max= 3320, avg=2070.43, stdev=300.11, samples=60 write: IOPS=520, BW=32.5MiB/s (34.1MB/s)(975MiB/30001msec) clat (usec): min=339, max=123742, avg=1565.94, stdev=1925.42 lat (usec): min=344, max=123749, avg=1572.17, stdev=1925.37 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 586], 10.00th=[ 840], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1467], | 70.00th=[ 1549], 80.00th=[ 1680], 90.00th=[ 1926], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=19584, max=55168, per=100.00%, avg=33284.27, stdev=4471.22, samples=60 iops : min= 306, max= 862, avg=520.07, stdev=69.86, samples=60 lat (usec) : 50=27.31%, 100=51.93%, 250=0.47%, 500=0.71%, 750=1.00% lat (usec) : 1000=1.11% lat (msec) : 2=15.67%, 4=1.26%, 10=0.29%, 20=0.22%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.55%, sys=3.45%, ctx=77724, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62113,15603,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=3882MiB (4071MB), run=30001-30001msec WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=975MiB (1023MB), run=30001-30001msec Disk stats (read/write): vdd: ios=61885/15530, merge=0/0, ticks=1792/16871, in_queue=28312, util=94.54% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920124: Fri Jan 7 13:54:55 2022 read: IOPS=2183, BW=136MiB/s (143MB/s)(4108MiB/30104msec) slat (usec): min=2, max=223, avg=11.00, stdev= 5.03 clat (nsec): min=714, max=2537.9M, avg=2515655.18, stdev=63982438.09 lat (usec): min=27, max=2537.9k, avg=2526.81, stdev=63982.47 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 48], 70.00th=[ 52], | 80.00th=[ 58], 90.00th=[ 67], 95.00th=[ 78], | 99.00th=[ 322], 99.50th=[ 848], 99.90th=[ 985662], | 99.95th=[1786774], 99.99th=[2499806] bw ( KiB/s): min=105170, max=278912, per=100.00%, avg=140217.87, stdev=22348.54, samples=60 iops : min= 1643, max= 4358, avg=2190.88, stdev=349.20, samples=60 write: IOPS=546, BW=34.2MiB/s (35.8MB/s)(1029MiB/30104msec) slat (usec): min=6, max=236, avg=21.25, stdev=10.01 clat (msec): min=7, max=1993, avg=81.32, stdev=51.02 lat (msec): min=7, max=1993, avg=81.34, stdev=51.02 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 57], 10.00th=[ 63], 20.00th=[ 67], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 77], 60.00th=[ 79], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 97], 95.00th=[ 122], | 99.00th=[ 190], 99.50th=[ 393], 99.90th=[ 793], 99.95th=[ 1062], | 99.99th=[ 1401] bw ( KiB/s): min=29056, max=65024, per=100.00%, avg=35027.40, stdev=4546.46, samples=60 iops : min= 454, max= 1016, avg=547.28, stdev=71.05, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.07%, 50=52.45% lat (usec) : 100=25.31%, 250=1.05%, 500=0.37%, 750=0.13%, 1000=0.10% lat (msec) : 2=0.15%, 10=0.02%, 20=0.28%, 50=0.32%, 100=17.70% lat (msec) : 250=1.59%, 500=0.10%, 750=0.07%, 1000=0.04%, 2000=0.06% lat (msec) : >=2000=0.03% cpu : usr=1.76%, sys=5.59%, ctx=79977, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=65726,16459,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4108MiB (4307MB), run=30104-30104msec WRITE: bw=34.2MiB/s (35.8MB/s), 34.2MiB/s-34.2MiB/s (35.8MB/s-35.8MB/s), io=1029MiB (1079MB), run=30104-30104msec Disk stats (read/write): vdd: ios=65346/16342, merge=0/0, ticks=157675/1316614, in_queue=1491314, util=99.72% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921271: Fri Jan 7 13:55:32 2022 read: IOPS=2064, BW=129MiB/s (135MB/s)(3872MiB/30001msec) clat (usec): min=33, max=60698, avg=89.93, stdev=745.45 lat (usec): min=33, max=60698, avg=90.08, stdev=745.45 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 99], 99.50th=[ 118], 99.90th=[14222], 99.95th=[16188], | 99.99th=[21627] bw ( KiB/s): min=89600, max=232320, per=100.00%, avg=132163.90, stdev=18562.84, samples=60 iops : min= 1400, max= 3630, avg=2065.05, stdev=290.05, samples=60 write: IOPS=518, BW=32.4MiB/s (33.0MB/s)(972MiB/30001msec) clat (usec): min=337, max=93700, avg=1555.98, stdev=1776.41 lat (usec): min=344, max=93706, avg=1562.25, stdev=1776.36 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 578], 10.00th=[ 816], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1385], 60.00th=[ 1467], | 70.00th=[ 1549], 80.00th=[ 1680], 90.00th=[ 1909], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[30540], 99.95th=[39060], | 99.99th=[47449] bw ( KiB/s): min=20736, max=59904, per=100.00%, avg=33176.48, stdev=4569.15, samples=60 iops : min= 324, max= 936, avg=518.37, stdev=71.40, samples=60 lat (usec) : 50=21.07%, 100=58.12%, 250=0.52%, 500=0.73%, 750=1.04% lat (usec) : 1000=1.18% lat (msec) : 2=15.57%, 4=1.23%, 10=0.27%, 20=0.23%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.61%, sys=3.66%, ctx=77512, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61950,15552,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3872MiB (4060MB), run=30001-30001msec WRITE: bw=32.4MiB/s (33.0MB/s), 32.4MiB/s-32.4MiB/s (33.0MB/s-33.0MB/s), io=972MiB (1019MB), run=30001-30001msec Disk stats (read/write): vdd: ios=61737/15494, merge=0/0, ticks=1912/16651, in_queue=28261, util=94.29% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922073: Fri Jan 7 13:56:10 2022 read: IOPS=2168, BW=136MiB/s (142MB/s)(4095MiB/30212msec) slat (usec): min=2, max=688, avg=11.75, stdev= 8.80 clat (nsec): min=769, max=2561.3M, avg=2463883.91, stdev=62577510.39 lat (usec): min=27, max=2561.3k, avg=2475.81, stdev=62577.55 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 46], 60.00th=[ 50], 70.00th=[ 56], | 80.00th=[ 62], 90.00th=[ 76], 95.00th=[ 133], | 99.00th=[ 1418], 99.50th=[ 2671], 99.90th=[ 960496], | 99.95th=[1686111], 99.99th=[2499806] bw ( KiB/s): min=116096, max=301056, per=100.00%, avg=139753.20, stdev=23944.35, samples=60 iops : min= 1814, max= 4704, avg=2183.63, stdev=374.13, samples=60 write: IOPS=543, BW=33.0MiB/s (35.6MB/s)(1026MiB/30212msec) slat (usec): min=8, max=731, avg=23.54, stdev=17.01 clat (msec): min=2, max=1976, avg=174.08, stdev=69.04 lat (msec): min=2, max=1976, avg=174.10, stdev=69.04 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 121], 10.00th=[ 146], 20.00th=[ 155], | 30.00th=[ 159], 40.00th=[ 163], 50.00th=[ 167], 60.00th=[ 171], | 70.00th=[ 178], 80.00th=[ 188], 90.00th=[ 215], 95.00th=[ 230], | 99.00th=[ 351], 99.50th=[ 609], 99.90th=[ 1003], 99.95th=[ 1318], | 99.99th=[ 1603] bw ( KiB/s): min=27520, max=65408, per=100.00%, avg=34828.95, stdev=4709.30, samples=60 iops : min= 430, max= 1022, avg=544.20, stdev=73.58, samples=60 lat (nsec) : 1000=0.11% lat (usec) : 2=0.20%, 4=0.03%, 10=0.05%, 20=0.16%, 50=47.55% lat (usec) : 100=26.95%, 250=2.08%, 500=0.85%, 750=0.54%, 1000=0.31% lat (msec) : 2=0.56%, 4=0.41%, 10=0.01%, 20=0.08%, 50=0.37% lat (msec) : 100=0.33%, 250=18.62%, 500=0.57%, 750=0.09%, 1000=0.06% lat (msec) : 2000=0.07%, >=2000=0.03% cpu : usr=1.84%, sys=5.82%, ctx=76806, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=65512,16418,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=4095MiB (4293MB), run=30212-30212msec WRITE: bw=33.0MiB/s (35.6MB/s), 33.0MiB/s-33.0MiB/s (35.6MB/s-35.6MB/s), io=1026MiB (1076MB), run=30212-30212msec Disk stats (read/write): vdd: ios=65505/16379, merge=0/0, ticks=152502/2831646, in_queue=3004603, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923033: Fri Jan 7 13:56:46 2022 read: IOPS=2077, BW=130MiB/s (136MB/s)(3896MiB/30001msec) clat (usec): min=32, max=28014, avg=85.07, stdev=676.46 lat (usec): min=32, max=28014, avg=85.20, stdev=676.47 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 92], 99.50th=[ 108], 99.90th=[14222], 99.95th=[16450], | 99.99th=[21365] bw ( KiB/s): min=105856, max=212992, per=100.00%, avg=133128.68, stdev=17131.28, samples=59 iops : min= 1654, max= 3328, avg=2080.14, stdev=267.68, samples=59 write: IOPS=521, BW=32.6MiB/s (34.2MB/s)(979MiB/30001msec) clat (usec): min=333, max=44321, avg=1563.14, stdev=1652.28 lat (usec): min=341, max=44326, avg=1569.19, stdev=1652.22 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 586], 10.00th=[ 865], 20.00th=[ 1156], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1483], | 70.00th=[ 1565], 80.00th=[ 1696], 90.00th=[ 1926], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=27776, max=55552, per=100.00%, avg=33449.22, stdev=4016.85, samples=59 iops : min= 434, max= 868, avg=522.64, stdev=62.76, samples=59 lat (usec) : 50=32.38%, 100=47.01%, 250=0.32%, 500=0.71%, 750=0.90% lat (usec) : 1000=1.05% lat (msec) : 2=15.83%, 4=1.28%, 10=0.27%, 20=0.21%, 50=0.03% cpu : usr=1.28%, sys=3.67%, ctx=78000, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62330,15660,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3896MiB (4085MB), run=30001-30001msec WRITE: bw=32.6MiB/s (34.2MB/s), 32.6MiB/s-32.6MiB/s (34.2MB/s-34.2MB/s), io=979MiB (1026MB), run=30001-30001msec Disk stats (read/write): vdd: ios=62056/15585, merge=0/0, ticks=1831/16899, in_queue=28244, util=94.29% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923708: Fri Jan 7 13:57:23 2022 read: IOPS=2178, BW=136MiB/s (143MB/s)(4116MiB/30231msec) slat (usec): min=2, max=236253, avg=357.67, stdev=2274.59 clat (usec): min=1003, max=2551.9k, avg=31016.31, stdev=65588.63 lat (usec): min=1008, max=2554.0k, avg=31374.12, stdev=65641.02 clat percentiles (usec): | 1.00th=[ 1762], 5.00th=[ 7373], 10.00th=[ 16188], | 20.00th=[ 20317], 30.00th=[ 22938], 40.00th=[ 25035], | 50.00th=[ 26870], 60.00th=[ 28967], 70.00th=[ 31327], | 80.00th=[ 34341], 90.00th=[ 39060], 95.00th=[ 45351], | 99.00th=[ 87557], 99.50th=[ 221250], 99.90th=[1044382], | 99.95th=[1635779], 99.99th=[2533360] bw ( KiB/s): min=98432, max=308608, per=100.00%, avg=140343.47, stdev=25861.79, samples=60 iops : min= 1538, max= 4822, avg=2192.87, stdev=404.09, samples=60 write: IOPS=545, BW=34.1MiB/s (35.8MB/s)(1031MiB/30231msec) slat (usec): min=6, max=205003, avg=380.55, stdev=2169.17 clat (msec): min=3, max=1977, avg=240.87, stdev=87.00 lat (msec): min=3, max=1977, avg=241.25, stdev=86.99 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 171], 10.00th=[ 201], 20.00th=[ 215], | 30.00th=[ 222], 40.00th=[ 228], 50.00th=[ 234], 60.00th=[ 241], | 70.00th=[ 247], 80.00th=[ 259], 90.00th=[ 284], 95.00th=[ 309], | 99.00th=[ 481], 99.50th=[ 659], 99.90th=[ 1435], 99.95th=[ 1435], | 99.99th=[ 1972] bw ( KiB/s): min=26624, max=64256, per=99.98%, avg=34909.87, stdev=4562.21, samples=60 iops : min= 416, max= 1004, avg=545.47, stdev=71.28, samples=60 lat (msec) : 2=1.05%, 4=1.64%, 10=1.51%, 20=10.73%, 50=62.50% lat (msec) : 100=2.32%, 250=14.81%, 500=5.14%, 750=0.12%, 1000=0.04% lat (msec) : 2000=0.11%, >=2000=0.03% cpu : usr=1.69%, sys=1.93%, ctx=9306, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=65856,16494,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4116MiB (4316MB), run=30231-30231msec WRITE: bw=34.1MiB/s (35.8MB/s), 34.1MiB/s-34.1MiB/s (35.8MB/s-35.8MB/s), io=1031MiB (1081MB), run=30231-30231msec Disk stats (read/write): vdd: ios=65849/16443, merge=0/0, ticks=159795/3480039, in_queue=3677788, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924568: Fri Jan 7 13:58:00 2022 read: IOPS=2008, BW=157MiB/s (165MB/s)(4708MiB/30001msec) clat (usec): min=34, max=62722, avg=98.08, stdev=772.88 lat (usec): min=34, max=62722, avg=98.22, stdev=772.89 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 88], | 99.00th=[ 125], 99.50th=[ 176], 99.90th=[13960], 99.95th=[16188], | 99.99th=[23200] bw ( KiB/s): min=132960, max=266240, per=100.00%, avg=160677.27, stdev=20695.82, samples=60 iops : min= 1662, max= 3328, avg=2008.45, stdev=258.70, samples=60 write: IOPS=504, BW=39.4MiB/s (41.3MB/s)(1183MiB/30001msec) clat (usec): min=368, max=56060, avg=1574.59, stdev=1584.46 lat (usec): min=376, max=56067, avg=1582.68, stdev=1584.41 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 611], 10.00th=[ 799], 20.00th=[ 1139], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[17957], 99.95th=[37487], | 99.99th=[41157] bw ( KiB/s): min=30560, max=69440, per=99.99%, avg=40366.72, stdev=4996.94, samples=60 iops : min= 382, max= 868, avg=504.58, stdev=62.46, samples=60 lat (usec) : 50=6.84%, 100=71.19%, 250=1.58%, 500=0.51%, 750=1.37% lat (usec) : 1000=1.16% lat (msec) : 2=15.41%, 4=1.41%, 10=0.27%, 20=0.22%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.68%, sys=3.87%, ctx=75405, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60256,15139,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=157MiB/s (165MB/s), 157MiB/s-157MiB/s (165MB/s-165MB/s), io=4708MiB (4936MB), run=30001-30001msec WRITE: bw=39.4MiB/s (41.3MB/s), 39.4MiB/s-39.4MiB/s (41.3MB/s-41.3MB/s), io=1183MiB (1240MB), run=30001-30001msec Disk stats (read/write): vdd: ios=60007/15072, merge=0/0, ticks=1995/16126, in_queue=28124, util=93.83% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925579: Fri Jan 7 13:58:37 2022 read: IOPS=1990, BW=156MiB/s (163MB/s)(4666MiB/30001msec) slat (usec): min=7, max=242, avg=14.50, stdev= 5.37 clat (nsec): min=787, max=26399k, avg=88926.05, stdev=716565.72 lat (usec): min=37, max=26408, avg=103.59, stdev=716.57 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 70], 95.00th=[ 77], | 99.00th=[ 99], 99.50th=[ 118], 99.90th=[15008], 99.95th=[16909], | 99.99th=[22676] bw ( KiB/s): min=126880, max=255360, per=99.84%, avg=158999.22, stdev=20208.08, samples=59 iops : min= 1586, max= 3192, avg=1987.47, stdev=252.61, samples=59 write: IOPS=499, BW=39.0MiB/s (40.9MB/s)(1171MiB/30001msec) slat (usec): min=12, max=444, avg=23.15, stdev= 8.17 clat (usec): min=373, max=42407, avg=1556.80, stdev=1652.94 lat (usec): min=390, max=42427, avg=1580.14, stdev=1652.87 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 594], 10.00th=[ 750], 20.00th=[ 1090], | 30.00th=[ 1254], 40.00th=[ 1336], 50.00th=[ 1418], 60.00th=[ 1483], | 70.00th=[ 1565], 80.00th=[ 1696], 90.00th=[ 1926], 95.00th=[ 2245], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[35914], 99.95th=[40109], | 99.99th=[42206] bw ( KiB/s): min=30240, max=66400, per=99.90%, avg=39945.66, stdev=5106.77, samples=59 iops : min= 378, max= 830, avg=499.29, stdev=63.84, samples=59 lat (nsec) : 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.03%, 50=31.79% lat (usec) : 100=47.29%, 250=0.53%, 500=0.53%, 750=1.50%, 1000=1.37% lat (msec) : 2=15.01%, 4=1.32%, 10=0.28%, 20=0.25%, 50=0.03% cpu : usr=1.50%, sys=6.28%, ctx=74675, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59724,14995,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4666MiB (4893MB), run=30001-30001msec WRITE: bw=39.0MiB/s (40.9MB/s), 39.0MiB/s-39.0MiB/s (40.9MB/s-40.9MB/s), io=1171MiB (1228MB), run=30001-30001msec Disk stats (read/write): vdd: ios=59298/14912, merge=0/0, ticks=1952/15909, in_queue=27975, util=93.41% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926020: Fri Jan 7 13:59:14 2022 read: IOPS=1979, BW=155MiB/s (162MB/s)(4639MiB/30001msec) clat (usec): min=32, max=27740, avg=93.73, stdev=715.75 lat (usec): min=32, max=27740, avg=93.87, stdev=715.75 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 75], 95.00th=[ 82], | 99.00th=[ 105], 99.50th=[ 128], 99.90th=[14091], 99.95th=[16188], | 99.99th=[24511] bw ( KiB/s): min=130720, max=232000, per=100.00%, avg=158723.73, stdev=19297.92, samples=59 iops : min= 1634, max= 2900, avg=1984.03, stdev=241.21, samples=59 write: IOPS=497, BW=38.9MiB/s (40.8MB/s)(1167MiB/30001msec) clat (usec): min=370, max=43865, avg=1619.77, stdev=1711.35 lat (usec): min=378, max=43874, avg=1627.42, stdev=1711.32 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 611], 10.00th=[ 832], 20.00th=[ 1172], | 30.00th=[ 1303], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1729], 90.00th=[ 1975], 95.00th=[ 2311], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[36439], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min=30240, max=60320, per=100.00%, avg=39858.98, stdev=4388.42, samples=59 iops : min= 378, max= 754, avg=498.22, stdev=54.86, samples=59 lat (usec) : 50=14.47%, 100=64.41%, 250=0.78%, 500=0.51%, 750=1.21% lat (usec) : 1000=1.06% lat (msec) : 2=15.45%, 4=1.51%, 10=0.31%, 20=0.24%, 50=0.04% cpu : usr=1.42%, sys=3.83%, ctx=74321, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59378,14935,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4639MiB (4864MB), run=30001-30001msec WRITE: bw=38.9MiB/s (40.8MB/s), 38.9MiB/s-38.9MiB/s (40.8MB/s-40.8MB/s), io=1167MiB (1223MB), run=30001-30001msec Disk stats (read/write): vdd: ios=59112/14863, merge=0/0, ticks=1931/16613, in_queue=28258, util=94.27% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926957: Fri Jan 7 13:59:51 2022 read: IOPS=2093, BW=164MiB/s (171MB/s)(4909MiB/30016msec) slat (usec): min=3, max=290, avg=13.52, stdev= 5.67 clat (nsec): min=730, max=2491.0M, avg=2018391.65, stdev=54884600.21 lat (usec): min=30, max=2491.0k, avg=2032.08, stdev=54884.57 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 36], 10.00th=[ 38], | 20.00th=[ 41], 30.00th=[ 43], 40.00th=[ 45], | 50.00th=[ 48], 60.00th=[ 50], 70.00th=[ 54], | 80.00th=[ 59], 90.00th=[ 66], 95.00th=[ 75], | 99.00th=[ 115], 99.50th=[ 165], 99.90th=[ 675283], | 99.95th=[1468007], 99.99th=[2499806] bw ( KiB/s): min=129120, max=314080, per=100.00%, avg=167562.83, stdev=23755.92, samples=60 iops : min= 1614, max= 3926, avg=2094.53, stdev=296.95, samples=60 write: IOPS=526, BW=41.1MiB/s (43.1MB/s)(1235MiB/30016msec) slat (usec): min=9, max=241, avg=22.80, stdev= 8.92 clat (usec): min=485, max=72226, avg=10881.19, stdev=6668.39 lat (usec): min=503, max=72246, avg=10904.17, stdev=6669.00 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1647], 10.00th=[ 3097], 20.00th=[ 5473], | 30.00th=[ 7963], 40.00th=[ 9110], 50.00th=[10290], 60.00th=[11863], | 70.00th=[13566], 80.00th=[15401], 90.00th=[17171], 95.00th=[19792], | 99.00th=[25560], 99.50th=[52691], 99.90th=[67634], 99.95th=[67634], | 99.99th=[71828] bw ( KiB/s): min=36800, max=79040, per=100.00%, avg=42118.82, stdev=5355.41, samples=60 iops : min= 460, max= 988, avg=526.48, stdev=66.94, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.06%, 50=48.06% lat (usec) : 100=30.48%, 250=0.88%, 500=0.06%, 750=0.06%, 1000=0.10% lat (msec) : 2=1.17%, 4=1.39%, 10=6.80%, 20=9.64%, 50=0.87% lat (msec) : 100=0.12%, 250=0.02%, 500=0.03%, 750=0.04%, 1000=0.02% lat (msec) : 2000=0.03%, >=2000=0.03% cpu : usr=1.82%, sys=6.06%, ctx=77631, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62834,15803,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=164MiB/s (171MB/s), 164MiB/s-164MiB/s (171MB/s-171MB/s), io=4909MiB (5147MB), run=30016-30016msec WRITE: bw=41.1MiB/s (43.1MB/s), 41.1MiB/s-41.1MiB/s (43.1MB/s-43.1MB/s), io=1235MiB (1295MB), run=30016-30016msec Disk stats (read/write): vdd: ios=62611/15726, merge=0/0, ticks=123605/162615, in_queue=296897, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927604: Fri Jan 7 14:00:28 2022 read: IOPS=1991, BW=156MiB/s (163MB/s)(4668MiB/30002msec) clat (usec): min=33, max=64928, avg=92.98, stdev=764.80 lat (usec): min=33, max=64928, avg=93.12, stdev=764.80 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 101], 99.50th=[ 125], 99.90th=[14746], 99.95th=[16450], | 99.99th=[22414] bw ( KiB/s): min=124640, max=249920, per=100.00%, avg=159344.93, stdev=19800.09, samples=60 iops : min= 1558, max= 3124, avg=1991.77, stdev=247.49, samples=60 write: IOPS=500, BW=39.1MiB/s (40.0MB/s)(1172MiB/30002msec) clat (usec): min=385, max=42773, avg=1612.57, stdev=1645.55 lat (usec): min=394, max=42780, avg=1620.40, stdev=1645.50 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 611], 10.00th=[ 865], 20.00th=[ 1188], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[17695], 99.95th=[40633], | 99.99th=[42730] bw ( KiB/s): min=31200, max=65440, per=100.00%, avg=40020.10, stdev=4842.75, samples=60 iops : min= 390, max= 818, avg=500.23, stdev=60.54, samples=60 lat (usec) : 50=19.59%, 100=59.48%, 250=0.62%, 500=0.50%, 750=1.16% lat (usec) : 1000=0.92% lat (msec) : 2=15.62%, 4=1.54%, 10=0.28%, 20=0.24%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.49%, sys=3.61%, ctx=74757, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59744,15006,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4668MiB (4894MB), run=30002-30002msec WRITE: bw=39.1MiB/s (40.0MB/s), 39.1MiB/s-39.1MiB/s (40.0MB/s-40.0MB/s), io=1172MiB (1229MB), run=30002-30002msec Disk stats (read/write): vdd: ios=59378/14934, merge=0/0, ticks=2012/16557, in_queue=28318, util=94.83% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928849: Fri Jan 7 14:01:05 2022 read: IOPS=2096, BW=164MiB/s (172MB/s)(4934MiB/30116msec) slat (usec): min=2, max=163, avg=11.38, stdev= 4.79 clat (nsec): min=675, max=2527.5M, avg=2531491.46, stdev=66141628.09 lat (usec): min=29, max=2527.5k, avg=2543.02, stdev=66141.60 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 45], 60.00th=[ 49], 70.00th=[ 55], | 80.00th=[ 60], 90.00th=[ 70], 95.00th=[ 81], | 99.00th=[ 375], 99.50th=[ 963], 99.90th=[ 918553], | 99.95th=[2231370], 99.99th=[2499806] bw ( KiB/s): min=140000, max=339040, per=100.00%, avg=168395.58, stdev=26218.21, samples=60 iops : min= 1750, max= 4238, avg=2104.92, stdev=327.73, samples=60 write: IOPS=527, BW=41.2MiB/s (43.2MB/s)(1240MiB/30116msec) slat (usec): min=8, max=227, avg=23.48, stdev=10.17 clat (msec): min=4, max=2574, avg=84.69, stdev=57.36 lat (msec): min=4, max=2575, avg=84.72, stdev=57.36 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 125], | 99.00th=[ 192], 99.50th=[ 384], 99.90th=[ 927], 99.95th=[ 1167], | 99.99th=[ 2140] bw ( KiB/s): min=37120, max=78720, per=100.00%, avg=42222.75, stdev=5385.34, samples=60 iops : min= 464, max= 984, avg=527.77, stdev=67.33, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.06%, 50=50.49% lat (usec) : 100=27.03%, 250=1.18%, 500=0.43%, 750=0.17%, 1000=0.08% lat (msec) : 2=0.20%, 10=0.03%, 20=0.31%, 50=0.14%, 100=17.82% lat (msec) : 250=1.67%, 500=0.11%, 750=0.06%, 1000=0.04%, 2000=0.04% lat (msec) : >=2000=0.05% cpu : usr=1.78%, sys=5.45%, ctx=76736, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=63149,15875,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=4934MiB (5173MB), run=30116-30116msec WRITE: bw=41.2MiB/s (43.2MB/s), 41.2MiB/s-41.2MiB/s (43.2MB/s-43.2MB/s), io=1240MiB (1300MB), run=30116-30116msec Disk stats (read/write): vdd: ios=63142/15875, merge=0/0, ticks=149112/1335028, in_queue=1502310, util=99.93% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929904: Fri Jan 7 14:01:42 2022 read: IOPS=2001, BW=156MiB/s (164MB/s)(4692MiB/30001msec) clat (usec): min=34, max=24778, avg=92.99, stdev=679.61 lat (usec): min=34, max=24778, avg=93.14, stdev=679.61 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 52], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 76], 95.00th=[ 83], | 99.00th=[ 108], 99.50th=[ 129], 99.90th=[14353], 99.95th=[16188], | 99.99th=[19006] bw ( KiB/s): min=122240, max=272000, per=100.00%, avg=160146.73, stdev=21638.08, samples=60 iops : min= 1528, max= 3400, avg=2001.83, stdev=270.48, samples=60 write: IOPS=502, BW=39.3MiB/s (41.2MB/s)(1178MiB/30001msec) clat (usec): min=379, max=62045, avg=1602.10, stdev=1686.78 lat (usec): min=387, max=62052, avg=1609.99, stdev=1686.71 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 619], 10.00th=[ 840], 20.00th=[ 1188], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[34866], 99.95th=[40109], | 99.99th=[44303] bw ( KiB/s): min=32160, max=70400, per=100.00%, avg=40222.67, stdev=5278.77, samples=60 iops : min= 402, max= 880, avg=502.78, stdev=65.98, samples=60 lat (usec) : 50=11.38%, 100=67.43%, 250=0.88%, 500=0.47%, 750=1.26% lat (usec) : 1000=0.99% lat (msec) : 2=15.51%, 4=1.54%, 10=0.27%, 20=0.24%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.60%, sys=3.82%, ctx=75145, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60053,15084,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4692MiB (4920MB), run=30001-30001msec WRITE: bw=39.3MiB/s (41.2MB/s), 39.3MiB/s-39.3MiB/s (41.2MB/s-41.2MB/s), io=1178MiB (1236MB), run=30001-30001msec Disk stats (read/write): vdd: ios=59725/14999, merge=0/0, ticks=1858/16483, in_queue=28029, util=93.64% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930571: Fri Jan 7 14:02:19 2022 read: IOPS=2082, BW=163MiB/s (171MB/s)(4915MiB/30201msec) slat (usec): min=2, max=172, avg=10.95, stdev= 4.74 clat (nsec): min=746, max=2605.5M, avg=2365146.01, stdev=62736537.94 lat (usec): min=29, max=2605.5k, avg=2376.25, stdev=62736.53 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 47], 60.00th=[ 51], 70.00th=[ 58], | 80.00th=[ 65], 90.00th=[ 79], 95.00th=[ 128], | 99.00th=[ 1549], 99.50th=[ 2933], 99.90th=[ 750781], | 99.95th=[1870660], 99.99th=[2499806] bw ( KiB/s): min=117760, max=340000, per=100.00%, avg=167762.80, stdev=28604.00, samples=60 iops : min= 1472, max= 4250, avg=2097.00, stdev=357.56, samples=60 write: IOPS=523, BW=40.9MiB/s (42.9MB/s)(1236MiB/30201msec) slat (usec): min=7, max=290, avg=23.11, stdev= 9.74 clat (msec): min=11, max=3142, avg=181.37, stdev=126.51 lat (msec): min=11, max=3142, avg=181.39, stdev=126.51 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 131], 10.00th=[ 140], 20.00th=[ 159], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 174], 60.00th=[ 178], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 215], 95.00th=[ 228], | 99.00th=[ 393], 99.50th=[ 609], 99.90th=[ 3138], 99.95th=[ 3138], | 99.99th=[ 3138] bw ( KiB/s): min=31360, max=71040, per=100.00%, avg=41939.97, stdev=4699.84, samples=60 iops : min= 392, max= 888, avg=524.22, stdev=58.77, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=46.64% lat (usec) : 100=28.30%, 250=2.11%, 500=0.79%, 750=0.53%, 1000=0.31% lat (msec) : 2=0.45%, 4=0.51%, 10=0.01%, 20=0.08%, 50=0.31% lat (msec) : 100=0.18%, 250=18.89%, 500=0.58%, 750=0.09%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.06% cpu : usr=1.73%, sys=5.29%, ctx=74053, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62908,15823,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4915MiB (5153MB), run=30201-30201msec WRITE: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=1236MiB (1296MB), run=30201-30201msec Disk stats (read/write): vdd: ios=62905/15776, merge=0/0, ticks=142031/2839585, in_queue=3005521, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931405: Fri Jan 7 14:02:56 2022 read: IOPS=2002, BW=156MiB/s (164MB/s)(4695MiB/30003msec) clat (usec): min=32, max=28252, avg=89.65, stdev=695.03 lat (usec): min=32, max=28252, avg=89.79, stdev=695.03 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 78], | 99.00th=[ 101], 99.50th=[ 122], 99.90th=[14222], 99.95th=[16188], | 99.99th=[20317] bw ( KiB/s): min=128960, max=269760, per=100.00%, avg=160243.03, stdev=20851.18, samples=60 iops : min= 1612, max= 3372, avg=2003.03, stdev=260.64, samples=60 write: IOPS=503, BW=39.3MiB/s (41.2MB/s)(1180MiB/30003msec) clat (usec): min=368, max=48620, avg=1614.22, stdev=1634.45 lat (usec): min=375, max=48626, avg=1621.91, stdev=1634.40 clat percentiles (usec): | 1.00th=[ 461], 5.00th=[ 611], 10.00th=[ 873], 20.00th=[ 1205], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[17695], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=28800, max=70080, per=100.00%, avg=40259.92, stdev=5123.14, samples=60 iops : min= 360, max= 876, avg=503.23, stdev=64.03, samples=60 lat (usec) : 50=26.25%, 100=52.86%, 250=0.58%, 500=0.51%, 750=1.10% lat (usec) : 1000=0.94% lat (msec) : 2=15.67%, 4=1.55%, 10=0.27%, 20=0.25%, 50=0.03% cpu : usr=1.46%, sys=3.57%, ctx=75197, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60093,15099,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4695MiB (4923MB), run=30003-30003msec WRITE: bw=39.3MiB/s (41.2MB/s), 39.3MiB/s-39.3MiB/s (41.2MB/s-41.2MB/s), io=1180MiB (1237MB), run=30003-30003msec Disk stats (read/write): vdd: ios=59818/15023, merge=0/0, ticks=1899/16680, in_queue=28348, util=94.65% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932260: Fri Jan 7 14:03:33 2022 read: IOPS=2089, BW=163MiB/s (171MB/s)(4934MiB/30225msec) slat (usec): min=2, max=219798, avg=371.58, stdev=2332.84 clat (usec): min=1107, max=2735.1k, avg=32254.88, stdev=68635.69 lat (usec): min=1110, max=2735.1k, avg=32626.60, stdev=68681.89 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 6259], 10.00th=[ 16909], | 20.00th=[ 21365], 30.00th=[ 23987], 40.00th=[ 26084], | 50.00th=[ 27919], 60.00th=[ 30016], 70.00th=[ 32637], | 80.00th=[ 35914], 90.00th=[ 40633], 95.00th=[ 47449], | 99.00th=[ 96994], 99.50th=[ 223347], 99.90th=[ 918553], | 99.95th=[2164261], 99.99th=[2566915] bw ( KiB/s): min=132000, max=377440, per=100.00%, avg=168254.93, stdev=31232.90, samples=60 iops : min= 1650, max= 4718, avg=2103.17, stdev=390.40, samples=60 write: IOPS=525, BW=41.0MiB/s (43.0MB/s)(1240MiB/30225msec) slat (usec): min=7, max=196389, avg=401.72, stdev=2279.71 clat (msec): min=21, max=3209, avg=250.49, stdev=119.25 lat (msec): min=21, max=3211, avg=250.90, stdev=119.27 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 184], 10.00th=[ 203], 20.00th=[ 222], | 30.00th=[ 228], 40.00th=[ 234], 50.00th=[ 241], 60.00th=[ 247], | 70.00th=[ 253], 80.00th=[ 264], 90.00th=[ 296], 95.00th=[ 317], | 99.00th=[ 510], 99.50th=[ 869], 99.90th=[ 2056], 99.95th=[ 2265], | 99.99th=[ 3205] bw ( KiB/s): min=34880, max=78880, per=99.95%, avg=42001.63, stdev=5678.17, samples=60 iops : min= 436, max= 986, avg=524.98, stdev=70.99, samples=60 lat (msec) : 2=0.97%, 4=1.85%, 10=1.69%, 20=8.23%, 50=64.20% lat (msec) : 100=2.68%, 250=13.62%, 500=6.44%, 750=0.10%, 1000=0.06% lat (msec) : 2000=0.09%, >=2000=0.07% cpu : usr=1.70%, sys=2.13%, ctx=8908, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=63156,15877,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4934MiB (5174MB), run=30225-30225msec WRITE: bw=41.0MiB/s (43.0MB/s), 41.0MiB/s-41.0MiB/s (43.0MB/s-43.0MB/s), io=1240MiB (1301MB), run=30225-30225msec Disk stats (read/write): vdd: ios=63154/15822, merge=0/0, ticks=161581/3485802, in_queue=3679538, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932836: Fri Jan 7 14:04:10 2022 read: IOPS=1773, BW=222MiB/s (232MB/s)(6650MiB/30001msec) clat (usec): min=38, max=25318, avg=106.56, stdev=766.00 lat (usec): min=38, max=25318, avg=106.72, stdev=766.00 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 51], 10.00th=[ 54], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 79], 90.00th=[ 88], 95.00th=[ 97], | 99.00th=[ 128], 99.50th=[ 165], 99.90th=[15795], 99.95th=[17957], | 99.99th=[21627] bw ( KiB/s): min=158208, max=388352, per=100.00%, avg=227132.97, stdev=33141.21, samples=59 iops : min= 1236, max= 3034, avg=1774.44, stdev=258.92, samples=59 write: IOPS=444, BW=55.6MiB/s (58.3MB/s)(1669MiB/30001msec) clat (usec): min=488, max=138857, avg=1800.15, stdev=2081.15 lat (usec): min=500, max=138872, avg=1813.81, stdev=2081.28 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 865], 10.00th=[ 1057], 20.00th=[ 1254], | 30.00th=[ 1401], 40.00th=[ 1500], 50.00th=[ 1598], 60.00th=[ 1696], | 70.00th=[ 1795], 80.00th=[ 1926], 90.00th=[ 2147], 95.00th=[ 2507], | 99.00th=[ 9765], 99.50th=[10290], 99.90th=[35914], 99.95th=[40633], | 99.99th=[45351] bw ( KiB/s): min=36864, max=101376, per=100.00%, avg=57126.02, stdev=7809.96, samples=59 iops : min= 288, max= 792, avg=446.24, stdev=61.01, samples=59 lat (usec) : 50=3.04%, 100=73.72%, 250=2.91%, 500=0.04%, 750=0.64% lat (usec) : 1000=1.00% lat (msec) : 2=15.33%, 4=2.56%, 10=0.44%, 20=0.30%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.75%, sys=3.55%, ctx=66556, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53198,13348,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=222MiB/s (232MB/s), 222MiB/s-222MiB/s (232MB/s-232MB/s), io=6650MiB (6973MB), run=30001-30001msec WRITE: bw=55.6MiB/s (58.3MB/s), 55.6MiB/s-55.6MiB/s (58.3MB/s-58.3MB/s), io=1669MiB (1750MB), run=30001-30001msec Disk stats (read/write): vdd: ios=52929/13295, merge=0/0, ticks=1921/16712, in_queue=28215, util=94.15% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933833: Fri Jan 7 14:04:47 2022 read: IOPS=1801, BW=225MiB/s (236MB/s)(6755MiB/30001msec) slat (usec): min=8, max=160, avg=14.56, stdev= 4.41 clat (nsec): min=911, max=62900k, avg=96530.74, stdev=884014.61 lat (usec): min=39, max=62913, avg=111.23, stdev=884.02 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 98], 99.50th=[ 115], 99.90th=[16909], 99.95th=[18220], | 99.99th=[22938] bw ( KiB/s): min=186368, max=375808, per=100.00%, avg=230998.78, stdev=29274.44, samples=59 iops : min= 1456, max= 2936, avg=1804.68, stdev=228.71, samples=59 write: IOPS=452, BW=56.6MiB/s (59.3MB/s)(1697MiB/30001msec) slat (usec): min=15, max=152, avg=27.45, stdev= 6.27 clat (usec): min=493, max=44024, avg=1731.65, stdev=1561.12 lat (usec): min=517, max=44050, avg=1759.27, stdev=1561.07 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 701], 10.00th=[ 881], 20.00th=[ 1287], | 30.00th=[ 1434], 40.00th=[ 1516], 50.00th=[ 1582], 60.00th=[ 1680], | 70.00th=[ 1778], 80.00th=[ 1909], 90.00th=[ 2147], 95.00th=[ 2442], | 99.00th=[ 9503], 99.50th=[10159], 99.90th=[17695], 99.95th=[39060], | 99.99th=[42206] bw ( KiB/s): min=45568, max=97792, per=100.00%, avg=57964.47, stdev=6967.72, samples=59 iops : min= 356, max= 764, avg=452.85, stdev=54.44, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=33.14%, 100=46.07% lat (usec) : 250=0.46%, 500=0.02%, 750=1.20%, 1000=1.24% lat (msec) : 2=14.62%, 4=2.61%, 10=0.31%, 20=0.28%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.74%, sys=5.31%, ctx=67615, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54038,13577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6755MiB (7083MB), run=30001-30001msec WRITE: bw=56.6MiB/s (59.3MB/s), 56.6MiB/s-56.6MiB/s (59.3MB/s-59.3MB/s), io=1697MiB (1780MB), run=30001-30001msec Disk stats (read/write): vdd: ios=53882/13525, merge=0/0, ticks=2169/15901, in_queue=28212, util=94.17% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934414: Fri Jan 7 14:05:24 2022 read: IOPS=1796, BW=225MiB/s (235MB/s)(6737MiB/30002msec) clat (usec): min=38, max=61640, avg=103.84, stdev=827.07 lat (usec): min=39, max=61640, avg=103.98, stdev=827.07 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 112], 99.50th=[ 135], 99.90th=[16450], 99.95th=[18220], | 99.99th=[21103] bw ( KiB/s): min=172800, max=377344, per=100.00%, avg=229960.53, stdev=28649.60, samples=60 iops : min= 1350, max= 2948, avg=1796.57, stdev=223.83, samples=60 write: IOPS=451, BW=56.4MiB/s (59.1MB/s)(1692MiB/30002msec) clat (usec): min=496, max=43596, avg=1780.92, stdev=1623.96 lat (usec): min=509, max=43611, avg=1794.31, stdev=1623.78 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 709], 10.00th=[ 955], 20.00th=[ 1336], | 30.00th=[ 1467], 40.00th=[ 1549], 50.00th=[ 1614], 60.00th=[ 1696], | 70.00th=[ 1811], 80.00th=[ 1942], 90.00th=[ 2212], 95.00th=[ 2474], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[16909], 99.95th=[41157], | 99.99th=[42730] bw ( KiB/s): min=46848, max=98560, per=100.00%, avg=57753.60, stdev=6844.61, samples=60 iops : min= 366, max= 770, avg=451.20, stdev=53.47, samples=60 lat (usec) : 50=2.52%, 100=75.88%, 250=1.27%, 500=0.03%, 750=1.17% lat (usec) : 1000=1.00% lat (msec) : 2=14.50%, 4=2.99%, 10=0.30%, 20=0.31%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.63%, sys=3.43%, ctx=67448, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53897,13537,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=225MiB/s (235MB/s), 225MiB/s-225MiB/s (235MB/s-235MB/s), io=6737MiB (7064MB), run=30002-30002msec WRITE: bw=56.4MiB/s (59.1MB/s), 56.4MiB/s-56.4MiB/s (59.1MB/s-59.1MB/s), io=1692MiB (1774MB), run=30002-30002msec Disk stats (read/write): vdd: ios=53600/13461, merge=0/0, ticks=2087/16322, in_queue=28223, util=94.24% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935713: Fri Jan 7 14:06:01 2022 read: IOPS=1885, BW=236MiB/s (247MB/s)(7081MiB/30041msec) slat (usec): min=5, max=123, avg=14.25, stdev= 4.45 clat (nsec): min=797, max=2511.5M, avg=2417488.43, stdev=61934146.03 lat (usec): min=36, max=2511.5k, avg=2431.89, stdev=61934.14 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 40], 10.00th=[ 42], | 20.00th=[ 44], 30.00th=[ 47], 40.00th=[ 49], | 50.00th=[ 52], 60.00th=[ 55], 70.00th=[ 59], | 80.00th=[ 65], 90.00th=[ 75], 95.00th=[ 85], | 99.00th=[ 116], 99.50th=[ 155], 99.90th=[ 901776], | 99.95th=[1853883], 99.99th=[2499806] bw ( KiB/s): min=192000, max=455424, per=100.00%, avg=241706.87, stdev=35695.87, samples=60 iops : min= 1500, max= 3558, avg=1888.33, stdev=278.87, samples=60 write: IOPS=474, BW=59.3MiB/s (62.2MB/s)(1782MiB/30041msec) slat (usec): min=12, max=124, avg=28.47, stdev= 8.13 clat (usec): min=626, max=69021, avg=11368.91, stdev=6169.85 lat (usec): min=651, max=69068, avg=11397.55, stdev=6170.75 clat percentiles (usec): | 1.00th=[ 1450], 5.00th=[ 3032], 10.00th=[ 3916], 20.00th=[ 6456], | 30.00th=[ 8094], 40.00th=[ 9765], 50.00th=[11076], 60.00th=[12518], | 70.00th=[13960], 80.00th=[15533], 90.00th=[17433], 95.00th=[20317], | 99.00th=[26346], 99.50th=[36963], 99.90th=[64750], 99.95th=[66847], | 99.99th=[68682] bw ( KiB/s): min=48640, max=114176, per=100.00%, avg=60793.45, stdev=7892.35, samples=60 iops : min= 380, max= 892, avg=474.93, stdev=61.66, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=35.79%, 100=42.50% lat (usec) : 250=1.27%, 500=0.07%, 750=0.02%, 1000=0.03% lat (msec) : 2=0.66%, 4=1.38%, 10=6.21%, 20=10.75%, 50=1.01% lat (msec) : 100=0.08%, 250=0.03%, 500=0.04%, 750=0.02%, 1000=0.02% lat (msec) : 2000=0.04%, >=2000=0.03% cpu : usr=1.75%, sys=5.54%, ctx=70253, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=56650,14256,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=236MiB/s (247MB/s), 236MiB/s-236MiB/s (247MB/s-247MB/s), io=7081MiB (7425MB), run=30041-30041msec WRITE: bw=59.3MiB/s (62.2MB/s), 59.3MiB/s-59.3MiB/s (62.2MB/s-62.2MB/s), io=1782MiB (1869MB), run=30041-30041msec Disk stats (read/write): vdd: ios=56428/14179, merge=0/0, ticks=131663/153529, in_queue=296991, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936596: Fri Jan 7 14:06:38 2022 read: IOPS=1784, BW=223MiB/s (234MB/s)(6694MiB/30002msec) clat (usec): min=38, max=33430, avg=105.28, stdev=799.10 lat (usec): min=38, max=33431, avg=105.42, stdev=799.10 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 55], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 110], 99.50th=[ 133], 99.90th=[16319], 99.95th=[18220], | 99.99th=[23462] bw ( KiB/s): min=135424, max=383744, per=100.00%, avg=228507.18, stdev=32861.31, samples=60 iops : min= 1058, max= 2998, avg=1785.17, stdev=256.72, samples=60 write: IOPS=448, BW=56.0MiB/s (58.7MB/s)(1681MiB/30002msec) clat (usec): min=494, max=135103, avg=1790.75, stdev=2109.36 lat (usec): min=506, max=135114, avg=1803.38, stdev=2109.32 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 709], 10.00th=[ 922], 20.00th=[ 1303], | 30.00th=[ 1467], 40.00th=[ 1532], 50.00th=[ 1598], 60.00th=[ 1680], | 70.00th=[ 1795], 80.00th=[ 1926], 90.00th=[ 2180], 95.00th=[ 2474], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[38011], 99.95th=[41157], | 99.99th=[65274] bw ( KiB/s): min=34048, max=99584, per=100.00%, avg=57373.18, stdev=7805.31, samples=60 iops : min= 266, max= 778, avg=448.20, stdev=60.99, samples=60 lat (usec) : 50=1.50%, 100=77.02%, 250=1.17%, 500=0.02%, 750=1.17% lat (usec) : 1000=1.11% lat (msec) : 2=14.52%, 4=2.83%, 10=0.30%, 20=0.32%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.57%, sys=3.63%, ctx=67010, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53553,13447,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=6694MiB (7019MB), run=30002-30002msec WRITE: bw=56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s), io=1681MiB (1763MB), run=30002-30002msec Disk stats (read/write): vdd: ios=53347/13396, merge=0/0, ticks=2039/16366, in_queue=28249, util=94.25% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937317: Fri Jan 7 14:07:16 2022 read: IOPS=1889, BW=236MiB/s (248MB/s)(7117MiB/30140msec) slat (usec): min=3, max=110, avg=12.90, stdev= 4.66 clat (nsec): min=841, max=2530.9M, avg=3018058.31, stdev=71754951.82 lat (usec): min=36, max=2530.9k, avg=3031.11, stdev=71754.96 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 42], | 20.00th=[ 44], 30.00th=[ 46], 40.00th=[ 48], | 50.00th=[ 52], 60.00th=[ 57], 70.00th=[ 65], | 80.00th=[ 77], 90.00th=[ 88], 95.00th=[ 103], | 99.00th=[ 510], 99.50th=[ 1385], 99.90th=[1249903], | 99.95th=[2231370], 99.99th=[2533360] bw ( KiB/s): min=188160, max=467968, per=100.00%, avg=242905.47, stdev=36378.56, samples=60 iops : min= 1470, max= 3656, avg=1897.68, stdev=284.20, samples=60 write: IOPS=475, BW=59.4MiB/s (62.3MB/s)(1790MiB/30140msec) slat (usec): min=13, max=296, avg=32.33, stdev=13.65 clat (msec): min=3, max=1397, avg=93.11, stdev=52.61 lat (msec): min=3, max=1397, avg=93.14, stdev=52.61 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 63], 10.00th=[ 74], 20.00th=[ 82], | 30.00th=[ 84], 40.00th=[ 87], 50.00th=[ 89], 60.00th=[ 91], | 70.00th=[ 94], 80.00th=[ 99], 90.00th=[ 107], 95.00th=[ 133], | 99.00th=[ 184], 99.50th=[ 368], 99.90th=[ 944], 99.95th=[ 1133], | 99.99th=[ 1351] bw ( KiB/s): min=48384, max=109056, per=100.00%, avg=60925.83, stdev=7273.08, samples=60 iops : min= 378, max= 852, avg=475.97, stdev=56.82, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=36.09% lat (usec) : 100=39.38%, 250=3.08%, 500=0.50%, 750=0.24%, 1000=0.08% lat (msec) : 2=0.25%, 4=0.03%, 10=0.01%, 20=0.29%, 50=0.21% lat (msec) : 100=16.21%, 250=3.27%, 500=0.11%, 750=0.06%, 1000=0.04% lat (msec) : 2000=0.07%, >=2000=0.04% cpu : usr=2.15%, sys=5.21%, ctx=69211, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=56935,14323,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=236MiB/s (248MB/s), 236MiB/s-236MiB/s (248MB/s-248MB/s), io=7117MiB (7463MB), run=30140-30140msec WRITE: bw=59.4MiB/s (62.3MB/s), 59.4MiB/s-59.4MiB/s (62.3MB/s-62.3MB/s), io=1790MiB (1877MB), run=30140-30140msec Disk stats (read/write): vdd: ios=56929/14284, merge=0/0, ticks=159699/1312964, in_queue=1501368, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938220: Fri Jan 7 14:07:52 2022 read: IOPS=1805, BW=226MiB/s (237MB/s)(6773MiB/30004msec) clat (usec): min=38, max=65344, avg=100.82, stdev=821.80 lat (usec): min=38, max=65344, avg=100.96, stdev=821.80 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 100], 99.50th=[ 118], 99.90th=[15926], 99.95th=[17957], | 99.99th=[22414] bw ( KiB/s): min=183040, max=395520, per=100.00%, avg=231192.68, stdev=31584.21, samples=60 iops : min= 1430, max= 3090, avg=1806.18, stdev=246.75, samples=60 write: IOPS=453, BW=56.7MiB/s (59.5MB/s)(1702MiB/30004msec) clat (usec): min=508, max=80700, avg=1782.23, stdev=1780.72 lat (usec): min=514, max=80711, avg=1794.25, stdev=1780.68 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 709], 10.00th=[ 963], 20.00th=[ 1352], | 30.00th=[ 1467], 40.00th=[ 1549], 50.00th=[ 1614], 60.00th=[ 1696], | 70.00th=[ 1795], 80.00th=[ 1942], 90.00th=[ 2180], 95.00th=[ 2474], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[35914], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=46336, max=102912, per=100.00%, avg=58092.87, stdev=7715.35, samples=60 iops : min= 362, max= 804, avg=453.85, stdev=60.28, samples=60 lat (usec) : 50=3.56%, 100=75.54%, 250=0.59%, 500=0.01%, 750=1.15% lat (usec) : 1000=1.01% lat (msec) : 2=14.67%, 4=2.86%, 10=0.27%, 20=0.31%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.53%, sys=3.34%, ctx=67806, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54184,13616,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=6773MiB (7102MB), run=30004-30004msec WRITE: bw=56.7MiB/s (59.5MB/s), 56.7MiB/s-56.7MiB/s (59.5MB/s-59.5MB/s), io=1702MiB (1785MB), run=30004-30004msec Disk stats (read/write): vdd: ios=54011/13572, merge=0/0, ticks=2048/16379, in_queue=28370, util=94.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938822: Fri Jan 7 14:08:30 2022 read: IOPS=1900, BW=238MiB/s (249MB/s)(7192MiB/30277msec) slat (usec): min=3, max=274, avg=11.55, stdev= 5.52 clat (nsec): min=825, max=2777.0M, avg=3102524.87, stdev=72573283.73 lat (usec): min=35, max=2778.0k, avg=3114.23, stdev=72573.28 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 48], | 50.00th=[ 52], 60.00th=[ 58], 70.00th=[ 70], | 80.00th=[ 79], 90.00th=[ 97], 95.00th=[ 208], | 99.00th=[ 3097], 99.50th=[ 4490], 99.90th=[1266680], | 99.95th=[2122318], 99.99th=[2533360] bw ( KiB/s): min=202496, max=558336, per=100.00%, avg=245478.40, stdev=46203.42, samples=60 iops : min= 1582, max= 4362, avg=1917.80, stdev=360.96, samples=60 write: IOPS=477, BW=59.7MiB/s (62.6MB/s)(1807MiB/30277msec) slat (usec): min=10, max=188, avg=31.74, stdev=14.01 clat (msec): min=4, max=1750, avg=196.98, stdev=72.12 lat (msec): min=4, max=1750, avg=197.01, stdev=72.12 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 146], 10.00th=[ 169], 20.00th=[ 178], | 30.00th=[ 182], 40.00th=[ 186], 50.00th=[ 190], 60.00th=[ 194], | 70.00th=[ 201], 80.00th=[ 209], 90.00th=[ 232], 95.00th=[ 253], | 99.00th=[ 422], 99.50th=[ 535], 99.90th=[ 1234], 99.95th=[ 1469], | 99.99th=[ 1703] bw ( KiB/s): min=50176, max=118272, per=100.00%, avg=61265.07, stdev=8567.85, samples=60 iops : min= 392, max= 924, avg=478.63, stdev=66.94, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.02%, 50=36.82% lat (usec) : 100=35.75%, 250=3.64%, 500=0.98%, 750=0.53%, 1000=0.42% lat (msec) : 2=0.69%, 4=0.51%, 10=0.35%, 20=0.04%, 50=0.42% lat (msec) : 100=0.19%, 250=18.32%, 500=1.04%, 750=0.09%, 1000=0.04% lat (msec) : 2000=0.09%, >=2000=0.04% cpu : usr=2.01%, sys=4.92%, ctx=66978, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=57537,14456,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=238MiB/s (249MB/s), 238MiB/s-238MiB/s (249MB/s-249MB/s), io=7192MiB (7541MB), run=30277-30277msec WRITE: bw=59.7MiB/s (62.6MB/s), 59.7MiB/s-59.7MiB/s (62.6MB/s-62.6MB/s), io=1807MiB (1895MB), run=30277-30277msec Disk stats (read/write): vdd: ios=57535/14363, merge=0/0, ticks=170645/2802843, in_queue=3008245, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939644: Fri Jan 7 14:09:06 2022 read: IOPS=1799, BW=225MiB/s (236MB/s)(6750MiB/30002msec) clat (usec): min=38, max=51513, avg=103.55, stdev=834.04 lat (usec): min=38, max=51513, avg=103.69, stdev=834.04 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 86], | 99.00th=[ 111], 99.50th=[ 133], 99.90th=[16909], 99.95th=[18482], | 99.99th=[22676] bw ( KiB/s): min=176896, max=390656, per=100.00%, avg=230410.85, stdev=30695.51, samples=60 iops : min= 1382, max= 3052, avg=1800.08, stdev=239.81, samples=60 write: IOPS=452, BW=56.5MiB/s (59.3MB/s)(1696MiB/30002msec) clat (usec): min=503, max=82434, avg=1778.48, stdev=1793.79 lat (usec): min=515, max=82447, avg=1790.73, stdev=1793.69 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 717], 10.00th=[ 955], 20.00th=[ 1336], | 30.00th=[ 1467], 40.00th=[ 1549], 50.00th=[ 1614], 60.00th=[ 1696], | 70.00th=[ 1811], 80.00th=[ 1942], 90.00th=[ 2180], 95.00th=[ 2474], | 99.00th=[ 9634], 99.50th=[10290], 99.90th=[35914], 99.95th=[41681], | 99.99th=[49021] bw ( KiB/s): min=41728, max=102144, per=100.00%, avg=57887.52, stdev=7626.35, samples=60 iops : min= 326, max= 798, avg=452.23, stdev=59.60, samples=60 lat (usec) : 50=3.49%, 100=75.11%, 250=1.07%, 500=0.03%, 750=1.14% lat (usec) : 1000=1.02% lat (msec) : 2=14.57%, 4=2.96%, 10=0.26%, 20=0.30%, 50=0.04% lat (msec) : 100=0.01% cpu : usr=1.74%, sys=3.22%, ctx=67577, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54001,13568,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6750MiB (7078MB), run=30002-30002msec WRITE: bw=56.5MiB/s (59.3MB/s), 56.5MiB/s-56.5MiB/s (59.3MB/s-59.3MB/s), io=1696MiB (1778MB), run=30002-30002msec Disk stats (read/write): vdd: ios=53735/13499, merge=0/0, ticks=2109/16344, in_queue=28310, util=94.50% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940587: Fri Jan 7 14:09:44 2022 read: IOPS=1889, BW=236MiB/s (248MB/s)(7153MiB/30288msec) slat (usec): min=3, max=252228, avg=417.16, stdev=2797.10 clat (usec): min=1371, max=2752.8k, avg=36184.79, stdev=79939.28 lat (usec): min=1378, max=2752.8k, avg=36602.10, stdev=80008.39 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 8], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 36], 80.00th=[ 40], 90.00th=[ 46], 95.00th=[ 53], | 99.00th=[ 123], 99.50th=[ 251], 99.90th=[ 1485], 99.95th=[ 2500], | 99.99th=[ 2567] bw ( KiB/s): min=166400, max=533504, per=100.00%, avg=243845.37, stdev=45529.31, samples=60 iops : min= 1300, max= 4168, avg=1905.02, stdev=355.72, samples=60 write: IOPS=474, BW=59.3MiB/s (62.2MB/s)(1797MiB/30288msec) slat (usec): min=11, max=54089, avg=415.89, stdev=1686.11 clat (msec): min=8, max=1796, avg=275.25, stdev=90.34 lat (msec): min=8, max=1800, avg=275.66, stdev=90.37 clat percentiles (msec): | 1.00th=[ 45], 5.00th=[ 184], 10.00th=[ 226], 20.00th=[ 245], | 30.00th=[ 253], 40.00th=[ 262], 50.00th=[ 268], 60.00th=[ 275], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 326], 95.00th=[ 372], | 99.00th=[ 558], 99.50th=[ 726], 99.90th=[ 1318], 99.95th=[ 1368], | 99.99th=[ 1653] bw ( KiB/s): min=40960, max=105728, per=100.00%, avg=60786.73, stdev=7935.74, samples=60 iops : min= 320, max= 826, avg=474.87, stdev=62.01, samples=60 lat (msec) : 2=0.23%, 4=1.18%, 10=3.29%, 20=5.05%, 50=65.59% lat (msec) : 100=4.09%, 250=5.37%, 500=14.69%, 750=0.30%, 1000=0.06% lat (msec) : 2000=0.09%, >=2000=0.06% cpu : usr=2.46%, sys=1.91%, ctx=8044, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=57221,14373,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=236MiB/s (248MB/s), 236MiB/s-236MiB/s (248MB/s-248MB/s), io=7153MiB (7500MB), run=30288-30288msec WRITE: bw=59.3MiB/s (62.2MB/s), 59.3MiB/s-59.3MiB/s (62.2MB/s-62.2MB/s), io=1797MiB (1884MB), run=30288-30288msec Disk stats (read/write): vdd: ios=57220/14299, merge=0/0, ticks=187519/3446766, in_queue=3684931, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941158: Fri Jan 7 14:10:20 2022 read: IOPS=1199, BW=300MiB/s (315MB/s)(8999MiB/30002msec) clat (usec): min=65, max=21803, avg=127.37, stdev=519.01 lat (usec): min=65, max=21803, avg=127.52, stdev=519.01 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 80], 10.00th=[ 82], 20.00th=[ 85], | 30.00th=[ 89], 40.00th=[ 94], 50.00th=[ 98], 60.00th=[ 103], | 70.00th=[ 110], 80.00th=[ 117], 90.00th=[ 130], 95.00th=[ 143], | 99.00th=[ 182], 99.50th=[ 219], 99.90th=[10290], 99.95th=[12780], | 99.99th=[16909] bw ( KiB/s): min=229376, max=548352, per=100.00%, avg=307165.87, stdev=48041.78, samples=60 iops : min= 896, max= 2142, avg=1199.87, stdev=187.66, samples=60 write: IOPS=302, BW=75.5MiB/s (79.2MB/s)(2266MiB/30002msec) clat (usec): min=842, max=54654, avg=2767.19, stdev=2601.75 lat (usec): min=868, max=54678, avg=2795.06, stdev=2601.26 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1090], 10.00th=[ 1254], 20.00th=[ 1582], | 30.00th=[ 1844], 40.00th=[ 1991], 50.00th=[ 2147], 60.00th=[ 2311], | 70.00th=[ 2474], 80.00th=[ 2868], 90.00th=[ 4621], 95.00th=[ 7701], | 99.00th=[11731], 99.50th=[13042], 99.90th=[34866], 99.95th=[52691], | 99.99th=[54789] bw ( KiB/s): min=59392, max=144896, per=100.00%, avg=77329.07, stdev=12076.82, samples=60 iops : min= 232, max= 566, avg=302.07, stdev=47.18, samples=60 lat (usec) : 100=42.74%, 250=36.81%, 500=0.08%, 750=0.04%, 1000=0.52% lat (msec) : 2=7.66%, 4=8.76%, 10=2.62%, 20=0.74%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=1.80%, sys=2.68%, ctx=45063, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=35996,9063,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=300MiB/s (315MB/s), 300MiB/s-300MiB/s (315MB/s-315MB/s), io=8999MiB (9436MB), run=30002-30002msec WRITE: bw=75.5MiB/s (79.2MB/s), 75.5MiB/s-75.5MiB/s (79.2MB/s-79.2MB/s), io=2266MiB (2376MB), run=30002-30002msec Disk stats (read/write): vdd: ios=35871/9025, merge=0/0, ticks=839/20468, in_queue=28499, util=95.11% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942439: Fri Jan 7 14:10:57 2022 read: IOPS=1314, BW=329MiB/s (345MB/s)(9858MiB/30004msec) slat (usec): min=12, max=335, avg=20.26, stdev= 5.80 clat (nsec): min=1128, max=27468k, avg=108980.83, stdev=593904.68 lat (usec): min=66, max=27488, avg=129.40, stdev=593.97 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 92], 90.00th=[ 101], 95.00th=[ 111], | 99.00th=[ 137], 99.50th=[ 157], 99.90th=[11731], 99.95th=[13960], | 99.99th=[18744] bw ( KiB/s): min=206848, max=527872, per=100.00%, avg=336514.40, stdev=49321.79, samples=60 iops : min= 808, max= 2062, avg=1314.50, stdev=192.67, samples=60 write: IOPS=331, BW=82.9MiB/s (86.9MB/s)(2487MiB/30004msec) slat (usec): min=25, max=176, avg=44.36, stdev= 8.46 clat (usec): min=770, max=116688, avg=2450.14, stdev=2876.96 lat (usec): min=810, max=116729, avg=2494.69, stdev=2876.71 clat percentiles (usec): | 1.00th=[ 840], 5.00th=[ 1037], 10.00th=[ 1123], 20.00th=[ 1516], | 30.00th=[ 1811], 40.00th=[ 1926], 50.00th=[ 2040], 60.00th=[ 2180], | 70.00th=[ 2311], 80.00th=[ 2540], 90.00th=[ 3228], 95.00th=[ 4752], | 99.00th=[ 10683], 99.50th=[ 12387], 99.90th=[ 52691], 99.95th=[ 54789], | 99.99th=[116917] bw ( KiB/s): min=47104, max=141312, per=100.00%, avg=84894.87, stdev=12318.70, samples=60 iops : min= 184, max= 552, avg=331.60, stdev=48.12, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=0.07%, 100=71.04%, 250=8.48% lat (usec) : 500=0.03%, 750=0.01%, 1000=0.74% lat (msec) : 2=8.64%, 4=9.07%, 10=1.32%, 20=0.57%, 50=0.01% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.60%, sys=4.86%, ctx=49397, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39433,9949,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=9858MiB (10.3GB), run=30004-30004msec WRITE: bw=82.9MiB/s (86.9MB/s), 82.9MiB/s-82.9MiB/s (86.9MB/s-86.9MB/s), io=2487MiB (2608MB), run=30004-30004msec Disk stats (read/write): vdd: ios=39286/9912, merge=0/0, ticks=1076/19583, in_queue=28374, util=94.72% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943464: Fri Jan 7 14:11:34 2022 read: IOPS=1362, BW=341MiB/s (357MB/s)(9.98GiB/30002msec) clat (usec): min=53, max=28008, avg=111.09, stdev=665.24 lat (usec): min=53, max=28009, avg=111.23, stdev=665.24 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 66], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 129], 99.50th=[ 153], 99.90th=[12780], 99.95th=[15926], | 99.99th=[21103] bw ( KiB/s): min=268288, max=598016, per=100.00%, avg=348839.20, stdev=47915.65, samples=60 iops : min= 1048, max= 2336, avg=1362.63, stdev=187.18, samples=60 write: IOPS=343, BW=85.0MiB/s (90.1MB/s)(2579MiB/30002msec) clat (usec): min=811, max=136950, avg=2434.73, stdev=2539.92 lat (usec): min=838, max=136975, avg=2459.33, stdev=2539.75 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1057], 10.00th=[ 1188], 20.00th=[ 1762], | 30.00th=[ 1876], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2376], 80.00th=[ 2540], 90.00th=[ 2999], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[10945], 99.90th=[39060], 99.95th=[53216], | 99.99th=[57934] bw ( KiB/s): min=72192, max=157696, per=100.00%, avg=88036.07, stdev=11275.31, samples=60 iops : min= 282, max= 616, avg=343.87, stdev=44.03, samples=60 lat (usec) : 100=74.32%, 250=5.28%, 500=0.01%, 750=0.01%, 1000=0.68% lat (msec) : 2=7.35%, 4=10.81%, 10=0.96%, 20=0.54%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.71%, sys=2.94%, ctx=51199, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40877,10317,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=341MiB/s (357MB/s), 341MiB/s-341MiB/s (357MB/s-357MB/s), io=9.98GiB (10.7GB), run=30002-30002msec WRITE: bw=85.0MiB/s (90.1MB/s), 85.0MiB/s-85.0MiB/s (90.1MB/s-90.1MB/s), io=2579MiB (2705MB), run=30002-30002msec Disk stats (read/write): vdd: ios=40727/10275, merge=0/0, ticks=1264/19864, in_queue=28404, util=94.84% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943991: Fri Jan 7 14:12:11 2022 read: IOPS=1435, BW=359MiB/s (376MB/s)(10.5GiB/30058msec) slat (usec): min=8, max=682, avg=17.08, stdev= 6.70 clat (nsec): min=933, max=2505.1M, avg=717143.23, stdev=28728225.98 lat (usec): min=49, max=2505.1k, avg=734.37, stdev=28728.27 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 52], | 20.00th=[ 54], 30.00th=[ 56], 40.00th=[ 59], | 50.00th=[ 63], 60.00th=[ 68], 70.00th=[ 75], | 80.00th=[ 90], 90.00th=[ 112], 95.00th=[ 128], | 99.00th=[ 186], 99.50th=[ 273], 99.90th=[ 43254], | 99.95th=[ 299893], 99.99th=[1551893] bw ( KiB/s): min=224768, max=709632, per=100.00%, avg=368138.68, stdev=60422.62, samples=60 iops : min= 878, max= 2772, avg=1438.03, stdev=236.03, samples=60 write: IOPS=362, BW=90.6MiB/s (94.0MB/s)(2722MiB/30058msec) slat (usec): min=21, max=374, avg=47.82, stdev=17.62 clat (usec): min=1450, max=143795, avg=24634.66, stdev=11522.98 lat (usec): min=1491, max=143878, avg=24682.66, stdev=11523.51 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 7], 10.00th=[ 10], 20.00th=[ 16], | 30.00th=[ 22], 40.00th=[ 24], 50.00th=[ 26], 60.00th=[ 27], | 70.00th=[ 29], 80.00th=[ 32], 90.00th=[ 36], 95.00th=[ 40], | 99.00th=[ 69], 99.50th=[ 83], 99.90th=[ 93], 99.95th=[ 136], | 99.99th=[ 142] bw ( KiB/s): min=59904, max=178688, per=100.00%, avg=92896.97, stdev=13178.37, samples=60 iops : min= 234, max= 698, avg=362.87, stdev=51.48, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=4.12%, 100=63.66% lat (usec) : 250=11.61%, 500=0.15%, 750=0.04%, 1000=0.02% lat (msec) : 2=0.05%, 4=0.15%, 10=1.90%, 20=3.57%, 50=14.32% lat (msec) : 100=0.33%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.99%, sys=4.63%, ctx=51092, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=43148,10888,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=359MiB/s (376MB/s), 359MiB/s-359MiB/s (376MB/s-376MB/s), io=10.5GiB (11.3GB), run=30058-30058msec WRITE: bw=90.6MiB/s (94.0MB/s), 90.6MiB/s-90.6MiB/s (94.0MB/s-94.0MB/s), io=2722MiB (2854MB), run=30058-30058msec Disk stats (read/write): vdd: ios=42978/10835, merge=0/0, ticks=19121/262388, in_queue=296905, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944968: Fri Jan 7 14:12:48 2022 read: IOPS=1355, BW=339MiB/s (355MB/s)(9.93GiB/30001msec) clat (usec): min=51, max=63950, avg=112.44, stdev=720.16 lat (usec): min=51, max=63951, avg=112.58, stdev=720.16 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 71], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 81], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 133], 99.50th=[ 159], 99.90th=[11863], 99.95th=[15139], | 99.99th=[23725] bw ( KiB/s): min=179200, max=576512, per=100.00%, avg=347038.53, stdev=49942.20, samples=60 iops : min= 700, max= 2252, avg=1355.60, stdev=195.09, samples=60 write: IOPS=341, BW=85.5MiB/s (89.7MB/s)(2565MiB/30001msec) clat (usec): min=801, max=125158, avg=2444.35, stdev=2609.64 lat (usec): min=828, max=125184, avg=2469.84, stdev=2609.47 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1745], | 30.00th=[ 1893], 40.00th=[ 1991], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2343], 80.00th=[ 2540], 90.00th=[ 2999], 95.00th=[ 4293], | 99.00th=[10552], 99.50th=[11076], 99.90th=[45876], 99.95th=[53740], | 99.99th=[72877] bw ( KiB/s): min=38912, max=153088, per=100.00%, avg=87549.37, stdev=12154.21, samples=60 iops : min= 152, max= 598, avg=341.97, stdev=47.48, samples=60 lat (usec) : 100=73.85%, 250=5.75%, 500=0.03%, 750=0.01%, 1000=0.70% lat (msec) : 2=7.49%, 4=10.64%, 10=0.94%, 20=0.56%, 50=0.02% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.87%, sys=2.87%, ctx=50939, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40666,10260,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=339MiB/s (355MB/s), 339MiB/s-339MiB/s (355MB/s-355MB/s), io=9.93GiB (10.7GB), run=30001-30001msec WRITE: bw=85.5MiB/s (89.7MB/s), 85.5MiB/s-85.5MiB/s (89.7MB/s-89.7MB/s), io=2565MiB (2690MB), run=30001-30001msec Disk stats (read/write): vdd: ios=40393/10201, merge=0/0, ticks=1248/19811, in_queue=28348, util=94.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945594: Fri Jan 7 14:13:26 2022 read: IOPS=1434, BW=359MiB/s (376MB/s)(10.6GiB/30197msec) slat (usec): min=4, max=1516, avg=13.09, stdev= 9.65 clat (nsec): min=898, max=2680.4M, avg=3297004.04, stdev=75893540.90 lat (usec): min=43, max=2680.4k, avg=3310.25, stdev=75893.52 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 52], | 20.00th=[ 55], 30.00th=[ 57], 40.00th=[ 61], | 50.00th=[ 66], 60.00th=[ 74], 70.00th=[ 95], | 80.00th=[ 116], 90.00th=[ 135], 95.00th=[ 172], | 99.00th=[ 1106], 99.50th=[ 2966], 99.90th=[1249903], | 99.95th=[2499806], 99.99th=[2533360] bw ( KiB/s): min=279040, max=819712, per=100.00%, avg=369749.13, stdev=69199.49, samples=60 iops : min= 1090, max= 3202, avg=1444.32, stdev=270.31, samples=60 write: IOPS=361, BW=90.5MiB/s (94.8MB/s)(2732MiB/30197msec) slat (usec): min=20, max=300, avg=51.78, stdev=24.69 clat (msec): min=10, max=1802, avg=124.95, stdev=53.46 lat (msec): min=10, max=1802, avg=125.00, stdev=53.46 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 88], 10.00th=[ 100], 20.00th=[ 107], | 30.00th=[ 112], 40.00th=[ 116], 50.00th=[ 121], 60.00th=[ 126], | 70.00th=[ 131], 80.00th=[ 138], 90.00th=[ 153], 95.00th=[ 169], | 99.00th=[ 266], 99.50th=[ 351], 99.90th=[ 919], 99.95th=[ 1183], | 99.99th=[ 1385] bw ( KiB/s): min=79360, max=187904, per=100.00%, avg=92865.42, stdev=13779.84, samples=60 iops : min= 310, max= 734, avg=362.73, stdev=53.84, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.01%, 50=4.20%, 100=52.74% lat (usec) : 250=20.59%, 500=0.82%, 750=0.32%, 1000=0.28% lat (msec) : 2=0.27%, 4=0.37%, 10=0.01%, 20=0.04%, 50=0.49% lat (msec) : 100=1.70%, 250=17.70%, 500=0.20%, 750=0.07%, 1000=0.04% lat (msec) : 2000=0.06%, >=2000=0.06% cpu : usr=2.42%, sys=3.84%, ctx=49216, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=43332,10926,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=359MiB/s (376MB/s), 359MiB/s-359MiB/s (376MB/s-376MB/s), io=10.6GiB (11.4GB), run=30197-30197msec WRITE: bw=90.5MiB/s (94.8MB/s), 90.5MiB/s-90.5MiB/s (94.8MB/s-94.8MB/s), io=2732MiB (2864MB), run=30197-30197msec Disk stats (read/write): vdd: ios=43327/10889, merge=0/0, ticks=131623/1349537, in_queue=1502675, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946482: Fri Jan 7 14:14:02 2022 read: IOPS=1364, BW=341MiB/s (358MB/s)(9.00GiB/30002msec) clat (usec): min=52, max=27709, avg=106.98, stdev=591.98 lat (usec): min=53, max=27709, avg=107.12, stdev=591.98 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 128], 99.50th=[ 153], 99.90th=[11863], 99.95th=[13566], | 99.99th=[17957] bw ( KiB/s): min=247808, max=578560, per=100.00%, avg=349371.73, stdev=42840.72, samples=60 iops : min= 968, max= 2260, avg=1364.73, stdev=167.35, samples=60 write: IOPS=344, BW=86.2MiB/s (90.3MB/s)(2585MiB/30002msec) clat (usec): min=794, max=53345, avg=2444.67, stdev=2330.95 lat (usec): min=807, max=53369, avg=2469.92, stdev=2330.63 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1045], 10.00th=[ 1172], 20.00th=[ 1778], | 30.00th=[ 1893], 40.00th=[ 1991], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2343], 80.00th=[ 2540], 90.00th=[ 2999], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[10945], 99.90th=[52167], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min=70144, max=153600, per=100.00%, avg=88217.60, stdev=10763.07, samples=60 iops : min= 274, max= 600, avg=344.60, stdev=42.04, samples=60 lat (usec) : 100=75.25%, 250=4.33%, 500=0.03%, 750=0.01%, 1000=0.70% lat (msec) : 2=7.45%, 4=10.67%, 10=1.01%, 20=0.53%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.69%, sys=2.99%, ctx=51293, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40942,10339,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=341MiB/s (358MB/s), 341MiB/s-341MiB/s (358MB/s-358MB/s), io=9.00GiB (10.7GB), run=30002-30002msec WRITE: bw=86.2MiB/s (90.3MB/s), 86.2MiB/s-86.2MiB/s (90.3MB/s-90.3MB/s), io=2585MiB (2710MB), run=30002-30002msec Disk stats (read/write): vdd: ios=40780/10294, merge=0/0, ticks=1120/19960, in_queue=28386, util=94.71% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947438: Fri Jan 7 14:14:40 2022 read: IOPS=1427, BW=357MiB/s (374MB/s)(10.6GiB/30298msec) slat (usec): min=3, max=263, avg=12.37, stdev= 6.17 clat (nsec): min=960, max=2638.6M, avg=3494514.33, stdev=77312243.47 lat (usec): min=47, max=2638.6k, avg=3507.04, stdev=77312.20 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 52], | 20.00th=[ 54], 30.00th=[ 57], 40.00th=[ 62], | 50.00th=[ 68], 60.00th=[ 78], 70.00th=[ 108], | 80.00th=[ 122], 90.00th=[ 163], 95.00th=[ 529], | 99.00th=[ 6128], 99.50th=[ 7570], 99.90th=[1434452], | 99.95th=[2399142], 99.99th=[2533360] bw ( KiB/s): min=284672, max=931840, per=100.00%, avg=368987.95, stdev=81051.65, samples=60 iops : min= 1112, max= 3640, avg=1441.33, stdev=316.61, samples=60 write: IOPS=359, BW=89.0MiB/s (94.4MB/s)(2727MiB/30298msec) slat (usec): min=20, max=282, avg=50.57, stdev=24.02 clat (msec): min=13, max=1744, avg=263.77, stdev=78.47 lat (msec): min=13, max=1744, avg=263.82, stdev=78.47 clat percentiles (msec): | 1.00th=[ 50], 5.00th=[ 161], 10.00th=[ 220], 20.00th=[ 241], | 30.00th=[ 249], 40.00th=[ 255], 50.00th=[ 262], 60.00th=[ 268], | 70.00th=[ 275], 80.00th=[ 288], 90.00th=[ 313], 95.00th=[ 338], | 99.00th=[ 523], 99.50th=[ 567], 99.90th=[ 1011], 99.95th=[ 1385], | 99.99th=[ 1536] bw ( KiB/s): min=75776, max=191488, per=100.00%, avg=92327.98, stdev=14373.33, samples=60 iops : min= 296, max= 748, avg=360.63, stdev=56.15, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 20=0.01%, 50=4.40%, 100=49.67%, 250=20.26% lat (usec) : 500=1.46%, 750=0.54%, 1000=0.42% lat (msec) : 2=0.88%, 4=0.80%, 10=1.19%, 20=0.01%, 50=0.20% lat (msec) : 100=0.46%, 250=5.70%, 500=13.59%, 750=0.21%, 1000=0.07% lat (msec) : 2000=0.06%, >=2000=0.06% cpu : usr=2.35%, sys=3.58%, ctx=46557, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=43250,10907,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=357MiB/s (374MB/s), 357MiB/s-357MiB/s (374MB/s-374MB/s), io=10.6GiB (11.3GB), run=30298-30298msec WRITE: bw=89.0MiB/s (94.4MB/s), 89.0MiB/s-89.0MiB/s (94.4MB/s-94.4MB/s), io=2727MiB (2859MB), run=30298-30298msec Disk stats (read/write): vdd: ios=43238/10856, merge=0/0, ticks=128040/2843351, in_queue=3004911, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948012: Fri Jan 7 14:15:16 2022 read: IOPS=1357, BW=339MiB/s (356MB/s)(9.94GiB/30001msec) clat (usec): min=52, max=17703, avg=105.92, stdev=529.93 lat (usec): min=52, max=17704, avg=106.06, stdev=529.93 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 135], 99.50th=[ 167], 99.90th=[10814], 99.95th=[12518], | 99.99th=[14353] bw ( KiB/s): min=273920, max=576512, per=100.00%, avg=347856.27, stdev=41999.94, samples=59 iops : min= 1070, max= 2252, avg=1358.81, stdev=164.06, samples=59 write: IOPS=342, BW=85.6MiB/s (89.7MB/s)(2568MiB/30001msec) clat (usec): min=767, max=54334, avg=2467.77, stdev=2325.76 lat (usec): min=790, max=54358, avg=2492.33, stdev=2325.47 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1778], | 30.00th=[ 1893], 40.00th=[ 2008], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2376], 80.00th=[ 2573], 90.00th=[ 3097], 95.00th=[ 4490], | 99.00th=[10552], 99.50th=[10945], 99.90th=[52691], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=71680, max=153088, per=100.00%, avg=87855.73, stdev=10363.77, samples=59 iops : min= 280, max= 598, avg=343.19, stdev=40.48, samples=59 lat (usec) : 100=74.21%, 250=5.36%, 500=0.05%, 750=0.01%, 1000=0.65% lat (msec) : 2=7.35%, 4=10.67%, 10=1.11%, 20=0.57%, 100=0.02% cpu : usr=1.63%, sys=3.06%, ctx=51000, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40720,10271,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=339MiB/s (356MB/s), 339MiB/s-339MiB/s (356MB/s-356MB/s), io=9.94GiB (10.7GB), run=30001-30001msec WRITE: bw=85.6MiB/s (89.7MB/s), 85.6MiB/s-85.6MiB/s (89.7MB/s-89.7MB/s), io=2568MiB (2692MB), run=30001-30001msec Disk stats (read/write): vdd: ios=40470/10222, merge=0/0, ticks=998/20149, in_queue=28367, util=94.74% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949365: Fri Jan 7 14:15:54 2022 read: IOPS=1431, BW=358MiB/s (375MB/s)(10.6GiB/30414msec) slat (usec): min=4, max=321804, avg=535.33, stdev=3843.63 clat (msec): min=4, max=2840, avg=47.40, stdev=86.90 lat (msec): min=4, max=2840, avg=47.93, stdev=86.97 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 18], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 49], 80.00th=[ 55], 90.00th=[ 66], 95.00th=[ 74], | 99.00th=[ 148], 99.50th=[ 372], 99.90th=[ 1452], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min=236032, max=926720, per=100.00%, avg=370852.85, stdev=84155.68, samples=60 iops : min= 922, max= 3620, avg=1448.63, stdev=328.74, samples=60 write: IOPS=361, BW=90.3MiB/s (94.7MB/s)(2747MiB/30414msec) slat (usec): min=28, max=349930, avg=599.76, stdev=5323.94 clat (msec): min=15, max=1858, avg=362.93, stdev=99.60 lat (msec): min=15, max=1858, avg=363.53, stdev=99.52 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 232], 10.00th=[ 300], 20.00th=[ 330], | 30.00th=[ 338], 40.00th=[ 351], 50.00th=[ 359], 60.00th=[ 368], | 70.00th=[ 380], 80.00th=[ 393], 90.00th=[ 422], 95.00th=[ 506], | 99.00th=[ 718], 99.50th=[ 751], 99.90th=[ 1133], 99.95th=[ 1469], | 99.99th=[ 1636] bw ( KiB/s): min=75776, max=176128, per=100.00%, avg=92649.35, stdev=13644.86, samples=60 iops : min= 296, max= 688, avg=361.90, stdev=53.30, samples=60 lat (msec) : 10=3.26%, 20=5.55%, 50=49.24%, 100=20.65%, 250=1.48% lat (msec) : 500=18.63%, 750=0.96%, 1000=0.08%, 2000=0.09%, >=2000=0.06% cpu : usr=3.39%, sys=1.82%, ctx=5943, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=43526,10989,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=358MiB/s (375MB/s), 358MiB/s-358MiB/s (375MB/s-375MB/s), io=10.6GiB (11.4GB), run=30414-30414msec WRITE: bw=90.3MiB/s (94.7MB/s), 90.3MiB/s-90.3MiB/s (94.7MB/s-94.7MB/s), io=2747MiB (2881MB), run=30414-30414msec Disk stats (read/write): vdd: ios=43517/10925, merge=0/0, ticks=157200/3483224, in_queue=3708609, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950047: Fri Jan 7 14:16:30 2022 read: IOPS=948, BW=474MiB/s (497MB/s)(13.9GiB/30004msec) clat (usec): min=77, max=22391, avg=149.59, stdev=709.87 lat (usec): min=77, max=22391, avg=149.75, stdev=709.87 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 94], 20.00th=[ 96], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 116], 80.00th=[ 124], 90.00th=[ 137], 95.00th=[ 153], | 99.00th=[ 204], 99.50th=[ 239], 99.90th=[13960], 99.95th=[15664], | 99.99th=[20055] bw ( KiB/s): min=366592, max=667648, per=100.00%, avg=485870.93, stdev=63635.49, samples=60 iops : min= 716, max= 1304, avg=948.97, stdev=124.29, samples=60 write: IOPS=239, BW=120MiB/s (126MB/s)(3592MiB/30004msec) clat (usec): min=1347, max=55966, avg=3523.91, stdev=2478.84 lat (usec): min=1394, max=56000, avg=3574.36, stdev=2478.31 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1663], 10.00th=[ 1827], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3851], 90.00th=[ 5669], 95.00th=[ 6980], | 99.00th=[11469], 99.50th=[13304], 99.90th=[53216], 99.95th=[54264], | 99.99th=[55837] bw ( KiB/s): min=98304, max=194560, per=100.00%, avg=122589.87, stdev=14136.37, samples=60 iops : min= 192, max= 380, avg=239.43, stdev=27.61, samples=60 lat (usec) : 100=29.16%, 250=50.34%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.45%, 4=14.20%, 10=3.11%, 20=0.61%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.83%, sys=2.69%, ctx=35663, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28469,7184,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=474MiB/s (497MB/s), 474MiB/s-474MiB/s (497MB/s-497MB/s), io=13.9GiB (14.9GB), run=30004-30004msec WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=3592MiB (3766MB), run=30004-30004msec Disk stats (read/write): vdd: ios=28345/7152, merge=0/0, ticks=1053/21394, in_queue=28481, util=95.10% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950728: Fri Jan 7 14:17:07 2022 read: IOPS=960, BW=480MiB/s (503MB/s)(14.1GiB/30003msec) slat (usec): min=18, max=141, avg=28.42, stdev= 6.47 clat (usec): min=29, max=22515, avg=148.25, stdev=686.00 lat (usec): min=98, max=22540, avg=176.84, stdev=686.08 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 91], 10.00th=[ 92], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 101], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 125], 90.00th=[ 137], 95.00th=[ 149], | 99.00th=[ 192], 99.50th=[ 245], 99.90th=[13304], 99.95th=[15270], | 99.99th=[20055] bw ( KiB/s): min=355328, max=612352, per=100.00%, avg=491652.85, stdev=53324.72, samples=60 iops : min= 694, max= 1196, avg=960.23, stdev=104.17, samples=60 write: IOPS=242, BW=121MiB/s (127MB/s)(3640MiB/30003msec) slat (usec): min=39, max=236, avg=75.33, stdev=16.08 clat (usec): min=1379, max=67826, avg=3338.37, stdev=2440.93 lat (usec): min=1449, max=67895, avg=3413.91, stdev=2440.29 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1631], 10.00th=[ 1680], 20.00th=[ 2089], | 30.00th=[ 2507], 40.00th=[ 2769], 50.00th=[ 2999], 60.00th=[ 3195], | 70.00th=[ 3392], 80.00th=[ 3720], 90.00th=[ 4948], 95.00th=[ 6390], | 99.00th=[11600], 99.50th=[13829], 99.90th=[28181], 99.95th=[54264], | 99.99th=[67634] bw ( KiB/s): min=106496, max=177152, per=100.00%, avg=124219.05, stdev=10284.34, samples=60 iops : min= 208, max= 346, avg=242.60, stdev=20.10, samples=60 lat (usec) : 50=0.01%, 100=29.14%, 250=50.30%, 500=0.12%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=3.85%, 4=13.26%, 10=2.71%, 20=0.58%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.88%, sys=4.40%, ctx=36098, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28806,7279,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=480MiB/s (503MB/s), 480MiB/s-480MiB/s (503MB/s-503MB/s), io=14.1GiB (15.1GB), run=30003-30003msec WRITE: bw=121MiB/s (127MB/s), 121MiB/s-121MiB/s (127MB/s-127MB/s), io=3640MiB (3816MB), run=30003-30003msec Disk stats (read/write): vdd: ios=28708/7249, merge=0/0, ticks=1040/20378, in_queue=28442, util=94.89% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951814: Fri Jan 7 14:17:44 2022 read: IOPS=977, BW=489MiB/s (512MB/s)(14.3GiB/30005msec) clat (usec): min=80, max=21337, avg=148.78, stdev=679.14 lat (usec): min=80, max=21337, avg=148.93, stdev=679.15 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 98], | 30.00th=[ 101], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 113], | 70.00th=[ 118], 80.00th=[ 125], 90.00th=[ 135], 95.00th=[ 145], | 99.00th=[ 180], 99.50th=[ 206], 99.90th=[13698], 99.95th=[15008], | 99.99th=[17957] bw ( KiB/s): min=381952, max=666624, per=100.00%, avg=500360.53, stdev=55990.05, samples=60 iops : min= 746, max= 1302, avg=977.27, stdev=109.36, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3701MiB/30005msec) clat (usec): min=1396, max=56365, avg=3405.94, stdev=2412.50 lat (usec): min=1440, max=56427, avg=3455.40, stdev=2412.24 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1631], 10.00th=[ 1696], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3458], 80.00th=[ 3720], 90.00th=[ 4555], 95.00th=[ 6456], | 99.00th=[11469], 99.50th=[13960], 99.90th=[52691], 99.95th=[53740], | 99.99th=[56361] bw ( KiB/s): min=98304, max=194560, per=100.00%, avg=126314.60, stdev=12529.72, samples=60 iops : min= 192, max= 380, avg=246.70, stdev=24.47, samples=60 lat (usec) : 100=22.21%, 250=57.32%, 500=0.06%, 750=0.01% lat (msec) : 2=3.09%, 4=14.04%, 10=2.72%, 20=0.54%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.87%, sys=2.89%, ctx=36729, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29317,7402,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=489MiB/s (512MB/s), 489MiB/s-489MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30005-30005msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3701MiB (3881MB), run=30005-30005msec Disk stats (read/write): vdd: ios=29175/7368, merge=0/0, ticks=1035/21138, in_queue=28325, util=94.55% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952394: Fri Jan 7 14:18:22 2022 read: IOPS=997, BW=499MiB/s (523MB/s)(14.6GiB/30054msec) slat (usec): min=6, max=137, avg=17.55, stdev= 7.05 clat (nsec): min=1135, max=103895k, avg=229331.59, stdev=2148294.54 lat (usec): min=70, max=103913, avg=247.05, stdev=2148.36 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 71], 10.00th=[ 73], 20.00th=[ 76], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 113], 80.00th=[ 174], 90.00th=[ 200], 95.00th=[ 221], | 99.00th=[ 322], 99.50th=[ 603], 99.90th=[38536], 99.95th=[44303], | 99.99th=[71828] bw ( KiB/s): min=406528, max=789504, per=100.00%, avg=511724.18, stdev=61032.48, samples=60 iops : min= 794, max= 1542, avg=999.43, stdev=119.22, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3786MiB/30054msec) slat (usec): min=33, max=398, avg=89.84, stdev=42.40 clat (msec): min=2, max=119, avg=38.61, stdev=15.76 lat (msec): min=2, max=119, avg=38.70, stdev=15.76 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 12], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 46], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 65], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 110], 99.95th=[ 117], | 99.99th=[ 121] bw ( KiB/s): min=105472, max=216064, per=100.00%, avg=129054.03, stdev=13653.89, samples=60 iops : min= 206, max= 422, avg=252.05, stdev=26.67, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=0.03%, 100=51.10%, 250=26.72% lat (usec) : 500=1.52%, 750=0.11%, 1000=0.02% lat (msec) : 2=0.06%, 4=0.01%, 10=0.64%, 20=1.72%, 50=13.73% lat (msec) : 100=4.25%, 250=0.08% cpu : usr=2.59%, sys=3.18%, ctx=36133, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29989,7572,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=499MiB/s (523MB/s), 499MiB/s-499MiB/s (523MB/s-523MB/s), io=14.6GiB (15.7GB), run=30054-30054msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3786MiB (3970MB), run=30054-30054msec Disk stats (read/write): vdd: ios=29861/7528, merge=0/0, ticks=3345/286630, in_queue=297182, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953225: Fri Jan 7 14:18:59 2022 read: IOPS=970, BW=485MiB/s (509MB/s)(14.2GiB/30002msec) clat (usec): min=78, max=19124, avg=147.50, stdev=680.05 lat (usec): min=78, max=19124, avg=147.66, stdev=680.05 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 93], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 103], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 116], 80.00th=[ 122], 90.00th=[ 133], 95.00th=[ 143], | 99.00th=[ 172], 99.50th=[ 202], 99.90th=[13566], 99.95th=[15139], | 99.99th=[17433] bw ( KiB/s): min=418816, max=645386, per=100.00%, avg=497173.50, stdev=43514.97, samples=60 iops : min= 818, max= 1260, avg=971.03, stdev=84.96, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3679MiB/30002msec) clat (usec): min=1402, max=63922, avg=3435.68, stdev=2421.49 lat (usec): min=1452, max=63971, avg=3484.74, stdev=2420.93 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1631], 10.00th=[ 1696], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3490], 80.00th=[ 3785], 90.00th=[ 4883], 95.00th=[ 6652], | 99.00th=[11469], 99.50th=[12125], 99.90th=[27657], 99.95th=[55313], | 99.99th=[63701] bw ( KiB/s): min=103424, max=184689, per=100.00%, avg=125565.62, stdev=11049.48, samples=60 iops : min= 202, max= 360, avg=245.23, stdev=21.52, samples=60 lat (usec) : 100=26.02%, 250=53.51%, 500=0.04%, 750=0.01% lat (msec) : 2=3.26%, 4=13.68%, 10=2.87%, 20=0.60%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.97%, sys=2.57%, ctx=36503, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29130,7358,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=485MiB/s (509MB/s), 485MiB/s-485MiB/s (509MB/s-509MB/s), io=14.2GiB (15.3GB), run=30002-30002msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3679MiB (3858MB), run=30002-30002msec Disk stats (read/write): vdd: ios=28939/7316, merge=0/0, ticks=1043/21215, in_queue=28430, util=94.93% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954160: Fri Jan 7 14:19:36 2022 read: IOPS=1006, BW=503MiB/s (528MB/s)(14.8GiB/30189msec) slat (usec): min=5, max=315, avg=15.06, stdev= 6.58 clat (nsec): min=1284, max=303439k, avg=486045.00, stdev=5919981.80 lat (usec): min=69, max=303451, avg=501.25, stdev=5919.92 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 71], 10.00th=[ 73], 20.00th=[ 75], | 30.00th=[ 77], 40.00th=[ 79], 50.00th=[ 84], 60.00th=[ 91], | 70.00th=[ 113], 80.00th=[ 188], 90.00th=[ 208], 95.00th=[ 281], | 99.00th=[ 3884], 99.50th=[ 6718], 99.90th=[105382], 99.95th=[127402], | 99.99th=[189793] bw ( KiB/s): min=396288, max=1020928, per=100.00%, avg=518454.02, stdev=87793.10, samples=60 iops : min= 774, max= 1994, avg=1012.58, stdev=171.47, samples=60 write: IOPS=253, BW=127MiB/s (133MB/s)(3834MiB/30189msec) slat (usec): min=31, max=417, avg=94.58, stdev=46.93 clat (msec): min=6, max=1618, avg=194.75, stdev=96.40 lat (msec): min=6, max=1619, avg=194.85, stdev=96.40 clat percentiles (msec): | 1.00th=[ 48], 5.00th=[ 71], 10.00th=[ 84], 20.00th=[ 105], | 30.00th=[ 131], 40.00th=[ 163], 50.00th=[ 201], 60.00th=[ 224], | 70.00th=[ 245], 80.00th=[ 266], 90.00th=[ 296], 95.00th=[ 326], | 99.00th=[ 443], 99.50th=[ 514], 99.90th=[ 978], 99.95th=[ 1200], | 99.99th=[ 1620] bw ( KiB/s): min=106496, max=225280, per=99.98%, avg=130009.63, stdev=14263.05, samples=60 iops : min= 208, max= 440, avg=253.90, stdev=27.86, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=0.02%, 100=52.59%, 250=22.61% lat (usec) : 500=1.68%, 750=0.51%, 1000=0.44% lat (msec) : 2=0.77%, 4=0.43%, 10=0.53%, 20=0.02%, 50=0.24% lat (msec) : 100=3.55%, 250=11.19%, 500=5.31%, 750=0.05%, 1000=0.03% lat (msec) : 2000=0.02% cpu : usr=2.69%, sys=2.90%, ctx=35584, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=30375,7667,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=503MiB/s (528MB/s), 503MiB/s-503MiB/s (528MB/s-528MB/s), io=14.8GiB (15.9GB), run=30189-30189msec WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=3834MiB (4020MB), run=30189-30189msec Disk stats (read/write): vdd: ios=30375/7633, merge=0/0, ticks=10444/1477350, in_queue=1502411, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954766: Fri Jan 7 14:20:12 2022 read: IOPS=975, BW=488MiB/s (511MB/s)(14.3GiB/30003msec) clat (usec): min=87, max=22140, avg=157.16, stdev=674.89 lat (usec): min=87, max=22140, avg=157.32, stdev=674.89 clat percentiles (usec): | 1.00th=[ 96], 5.00th=[ 101], 10.00th=[ 103], 20.00th=[ 106], | 30.00th=[ 110], 40.00th=[ 113], 50.00th=[ 118], 60.00th=[ 122], | 70.00th=[ 127], 80.00th=[ 135], 90.00th=[ 145], 95.00th=[ 155], | 99.00th=[ 194], 99.50th=[ 249], 99.90th=[13173], 99.95th=[14877], | 99.99th=[19006] bw ( KiB/s): min=320512, max=655360, per=100.00%, avg=499398.83, stdev=54262.63, samples=60 iops : min= 626, max= 1280, avg=975.37, stdev=106.01, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3697MiB/30003msec) clat (usec): min=1363, max=55986, avg=3377.45, stdev=2360.67 lat (usec): min=1409, max=56033, avg=3427.26, stdev=2359.89 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1696], 20.00th=[ 2311], | 30.00th=[ 2606], 40.00th=[ 2868], 50.00th=[ 3064], 60.00th=[ 3228], | 70.00th=[ 3458], 80.00th=[ 3720], 90.00th=[ 4817], 95.00th=[ 6587], | 99.00th=[11469], 99.50th=[12125], 99.90th=[19792], 99.95th=[55837], | 99.99th=[55837] bw ( KiB/s): min=106709, max=188416, per=100.00%, avg=126164.75, stdev=10834.30, samples=60 iops : min= 208, max= 368, avg=246.40, stdev=21.17, samples=60 lat (usec) : 100=3.42%, 250=76.02%, 500=0.13%, 750=0.01% lat (msec) : 2=3.47%, 4=13.59%, 10=2.79%, 20=0.56%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.91%, sys=2.85%, ctx=36663, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29260,7393,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=488MiB/s (511MB/s), 488MiB/s-488MiB/s (511MB/s-511MB/s), io=14.3GiB (15.3GB), run=30003-30003msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3697MiB (3876MB), run=30003-30003msec Disk stats (read/write): vdd: ios=29127/7354, merge=0/0, ticks=1022/20864, in_queue=28356, util=94.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=955997: Fri Jan 7 14:20:50 2022 read: IOPS=1003, BW=502MiB/s (526MB/s)(14.9GiB/30417msec) slat (usec): min=5, max=533, avg=14.68, stdev=10.54 clat (nsec): min=1948, max=427535k, avg=902810.14, stdev=7368743.47 lat (usec): min=70, max=427548, avg=917.64, stdev=7368.65 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 74], | 30.00th=[ 76], 40.00th=[ 80], 50.00th=[ 85], 60.00th=[ 96], | 70.00th=[ 180], 80.00th=[ 196], 90.00th=[ 465], 95.00th=[ 2933], | 99.00th=[ 13173], 99.50th=[ 15926], 99.90th=[126354], 99.95th=[156238], | 99.99th=[252707] bw ( KiB/s): min=200704, max=1096704, per=100.00%, avg=520925.87, stdev=105841.66, samples=60 iops : min= 392, max= 2142, avg=1017.43, stdev=206.72, samples=60 write: IOPS=253, BW=127MiB/s (133MB/s)(3857MiB/30417msec) slat (usec): min=35, max=1016, avg=94.78, stdev=49.01 clat (msec): min=10, max=2397, avg=390.48, stdev=132.02 lat (msec): min=10, max=2397, avg=390.58, stdev=132.01 clat percentiles (msec): | 1.00th=[ 93], 5.00th=[ 220], 10.00th=[ 266], 20.00th=[ 292], | 30.00th=[ 317], 40.00th=[ 347], 50.00th=[ 384], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 481], 90.00th=[ 523], 95.00th=[ 592], | 99.00th=[ 802], 99.50th=[ 869], 99.90th=[ 1351], 99.95th=[ 1536], | 99.99th=[ 2400] bw ( KiB/s): min=104448, max=187392, per=100.00%, avg=129962.67, stdev=12634.98, samples=60 iops : min= 204, max= 366, avg=253.83, stdev=24.68, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05% lat (usec) : 100=49.45%, 250=19.97%, 500=2.51%, 750=1.10%, 1000=0.64% lat (msec) : 2=1.44%, 4=1.33%, 10=1.94%, 20=1.12%, 50=0.03% lat (msec) : 100=0.36%, 250=1.35%, 500=15.76%, 750=2.59%, 1000=0.29% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.88%, sys=2.52%, ctx=32930, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30523,7714,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=502MiB/s (526MB/s), 502MiB/s-502MiB/s (526MB/s-526MB/s), io=14.9GiB (16.0GB), run=30417-30417msec WRITE: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=3857MiB (4044MB), run=30417-30417msec Disk stats (read/write): vdd: ios=30523/7665, merge=0/0, ticks=21014/2966738, in_queue=3018493, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956763: Fri Jan 7 14:21:27 2022 read: IOPS=987, BW=494MiB/s (518MB/s)(14.5GiB/30001msec) clat (usec): min=72, max=23017, avg=139.92, stdev=685.92 lat (usec): min=72, max=23017, avg=140.08, stdev=685.92 clat percentiles (usec): | 1.00th=[ 82], 5.00th=[ 86], 10.00th=[ 88], 20.00th=[ 90], | 30.00th=[ 92], 40.00th=[ 94], 50.00th=[ 98], 60.00th=[ 102], | 70.00th=[ 108], 80.00th=[ 113], 90.00th=[ 123], 95.00th=[ 135], | 99.00th=[ 174], 99.50th=[ 208], 99.90th=[13566], 99.95th=[15139], | 99.99th=[17433] bw ( KiB/s): min=386048, max=674816, per=100.00%, avg=505652.68, stdev=59886.49, samples=60 iops : min= 754, max= 1318, avg=987.55, stdev=116.97, samples=60 write: IOPS=249, BW=125MiB/s (131MB/s)(3736MiB/30001msec) clat (usec): min=1376, max=70924, avg=3403.05, stdev=2399.84 lat (usec): min=1402, max=70973, avg=3451.36, stdev=2399.16 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1631], 10.00th=[ 1713], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3458], 80.00th=[ 3752], 90.00th=[ 4490], 95.00th=[ 6325], | 99.00th=[11469], 99.50th=[11994], 99.90th=[53740], 99.95th=[54789], | 99.99th=[70779] bw ( KiB/s): min=108544, max=197632, per=100.00%, avg=127542.63, stdev=11928.21, samples=60 iops : min= 212, max= 386, avg=249.07, stdev=23.31, samples=60 lat (usec) : 100=44.85%, 250=34.69%, 500=0.04%, 1000=0.01% lat (msec) : 2=2.89%, 4=14.40%, 10=2.59%, 20=0.52%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=1.99%, ctx=37100, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29623,7472,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=14.5GiB (15.5GB), run=30001-30001msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3736MiB (3917MB), run=30001-30001msec Disk stats (read/write): vdd: ios=29435/7426, merge=0/0, ticks=1066/21337, in_queue=28549, util=95.30% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957722: Fri Jan 7 14:22:04 2022 read: IOPS=999, BW=500MiB/s (524MB/s)(14.9GiB/30464msec) slat (usec): min=6, max=451149, avg=740.50, stdev=5553.41 clat (msec): min=9, max=518, avg=64.54, stdev=52.82 lat (msec): min=9, max=549, avg=65.28, stdev=53.12 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 19], 10.00th=[ 26], 20.00th=[ 36], | 30.00th=[ 45], 40.00th=[ 53], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 93], 95.00th=[ 111], | 99.00th=[ 447], 99.50th=[ 477], 99.90th=[ 498], 99.95th=[ 510], | 99.99th=[ 514] bw ( KiB/s): min=87040, max=1183744, per=100.00%, avg=518534.87, stdev=177084.25, samples=60 iops : min= 170, max= 2312, avg=1012.72, stdev=345.85, samples=60 write: IOPS=252, BW=126MiB/s (132MB/s)(3844MiB/30464msec) slat (usec): min=130, max=449980, avg=959.14, stdev=7932.93 clat (msec): min=23, max=2067, avg=532.73, stdev=159.20 lat (msec): min=36, max=2068, avg=533.69, stdev=158.96 clat percentiles (msec): | 1.00th=[ 101], 5.00th=[ 288], 10.00th=[ 376], 20.00th=[ 430], | 30.00th=[ 468], 40.00th=[ 506], 50.00th=[ 542], 60.00th=[ 567], | 70.00th=[ 592], 80.00th=[ 617], 90.00th=[ 676], 95.00th=[ 793], | 99.00th=[ 1036], 99.50th=[ 1116], 99.90th=[ 1536], 99.95th=[ 1653], | 99.99th=[ 2072] bw ( KiB/s): min=27648, max=219136, per=99.72%, avg=128844.80, stdev=25090.68, samples=60 iops : min= 54, max= 428, avg=251.63, stdev=49.01, samples=60 lat (msec) : 10=0.01%, 20=5.05%, 50=25.02%, 100=44.00%, 250=5.59% lat (msec) : 500=8.03%, 750=11.08%, 1000=0.91%, 2000=0.32%, >=2000=0.01% cpu : usr=4.65%, sys=1.70%, ctx=3899, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30442,7688,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=500MiB/s (524MB/s), 500MiB/s-500MiB/s (524MB/s-524MB/s), io=14.9GiB (15.0GB), run=30464-30464msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3844MiB (4031MB), run=30464-30464msec Disk stats (read/write): vdd: ios=30442/7635, merge=0/0, ticks=74468/3598519, in_queue=3712641, util=99.54% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958707: Fri Jan 7 14:22:41 2022 read: IOPS=619, BW=620MiB/s (650MB/s)(18.2GiB/30002msec) clat (usec): min=143, max=24933, avg=259.60, stdev=1094.74 lat (usec): min=143, max=24933, avg=259.76, stdev=1094.74 clat percentiles (usec): | 1.00th=[ 151], 5.00th=[ 157], 10.00th=[ 159], 20.00th=[ 163], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 174], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 200], 90.00th=[ 219], 95.00th=[ 241], | 99.00th=[ 351], 99.50th=[ 5932], 99.90th=[18744], 99.95th=[19792], | 99.99th=[24511] bw ( KiB/s): min=464896, max=847872, per=100.00%, avg=634898.53, stdev=87839.68, samples=60 iops : min= 454, max= 828, avg=620.02, stdev=85.78, samples=60 write: IOPS=156, BW=157MiB/s (164MB/s)(4701MiB/30002msec) clat (usec): min=2518, max=56724, avg=5247.02, stdev=2955.09 lat (usec): min=2595, max=56782, avg=5345.29, stdev=2953.83 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3326], | 30.00th=[ 4146], 40.00th=[ 4621], 50.00th=[ 4948], 60.00th=[ 5145], | 70.00th=[ 5407], 80.00th=[ 5800], 90.00th=[ 8029], 95.00th=[10421], | 99.00th=[14091], 99.50th=[17171], 99.90th=[47449], 99.95th=[56361], | 99.99th=[56886] bw ( KiB/s): min=131334, max=225280, per=99.99%, avg=160431.03, stdev=14248.16, samples=60 iops : min= 128, max= 220, avg=156.67, stdev=13.92, samples=60 lat (usec) : 250=76.71%, 500=2.68%, 750=0.01% lat (msec) : 2=0.01%, 4=5.52%, 10=13.53%, 20=1.46%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=2.51%, ctx=23312, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18600,4701,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=620MiB/s (650MB/s), 620MiB/s-620MiB/s (650MB/s-650MB/s), io=18.2GiB (19.5GB), run=30002-30002msec WRITE: bw=157MiB/s (164MB/s), 157MiB/s-157MiB/s (164MB/s-164MB/s), io=4701MiB (4929MB), run=30002-30002msec Disk stats (read/write): vdd: ios=18492/4679, merge=0/0, ticks=1358/21918, in_queue=28506, util=95.23% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959285: Fri Jan 7 14:23:18 2022 read: IOPS=612, BW=612MiB/s (642MB/s)(17.9GiB/30003msec) slat (usec): min=29, max=143, avg=42.05, stdev= 8.62 clat (usec): min=107, max=30937, avg=243.47, stdev=1073.19 lat (usec): min=160, max=30975, avg=285.72, stdev=1073.40 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 139], 10.00th=[ 141], 20.00th=[ 145], | 30.00th=[ 149], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 167], | 70.00th=[ 176], 80.00th=[ 188], 90.00th=[ 210], 95.00th=[ 235], | 99.00th=[ 371], 99.50th=[ 6325], 99.90th=[18220], 99.95th=[20579], | 99.99th=[24511] bw ( KiB/s): min=448512, max=878592, per=100.00%, avg=627263.05, stdev=95375.01, samples=60 iops : min= 438, max= 858, avg=612.53, stdev=93.15, samples=60 write: IOPS=155, BW=155MiB/s (163MB/s)(4658MiB/30003msec) slat (usec): min=68, max=477, avg=134.63, stdev=30.85 clat (usec): min=2498, max=62979, avg=5170.22, stdev=2910.20 lat (usec): min=2634, max=63062, avg=5305.09, stdev=2908.47 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 2999], | 30.00th=[ 3916], 40.00th=[ 4490], 50.00th=[ 4817], 60.00th=[ 5080], | 70.00th=[ 5407], 80.00th=[ 5800], 90.00th=[ 8455], 95.00th=[10290], | 99.00th=[13960], 99.50th=[16057], 99.90th=[53740], 99.95th=[55837], | 99.99th=[63177] bw ( KiB/s): min=141312, max=223232, per=100.00%, avg=158974.43, stdev=12937.56, samples=60 iops : min= 138, max= 218, avg=155.23, stdev=12.64, samples=60 lat (usec) : 250=76.93%, 500=2.31%, 750=0.08%, 1000=0.01% lat (msec) : 2=0.01%, 4=6.26%, 10=12.90%, 20=1.41%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=2.09%, sys=3.71%, ctx=23046, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18375,4658,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=612MiB/s (642MB/s), 612MiB/s-612MiB/s (642MB/s-642MB/s), io=17.9GiB (19.3GB), run=30003-30003msec WRITE: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=4658MiB (4884MB), run=30003-30003msec Disk stats (read/write): vdd: ios=18299/4634, merge=0/0, ticks=1328/21482, in_queue=28473, util=95.06% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960088: Fri Jan 7 14:23:55 2022 read: IOPS=621, BW=621MiB/s (651MB/s)(18.2GiB/30001msec) clat (usec): min=114, max=24406, avg=228.33, stdev=1077.49 lat (usec): min=114, max=24406, avg=228.50, stdev=1077.49 clat percentiles (usec): | 1.00th=[ 125], 5.00th=[ 130], 10.00th=[ 133], 20.00th=[ 135], | 30.00th=[ 139], 40.00th=[ 141], 50.00th=[ 145], 60.00th=[ 151], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 188], 95.00th=[ 208], | 99.00th=[ 297], 99.50th=[ 6063], 99.90th=[18220], 99.95th=[19792], | 99.99th=[23987] bw ( KiB/s): min=476088, max=782336, per=99.91%, avg=635522.88, stdev=74722.47, samples=59 iops : min= 464, max= 764, avg=620.59, stdev=73.04, samples=59 write: IOPS=157, BW=157MiB/s (165MB/s)(4715MiB/30001msec) clat (usec): min=2508, max=58755, avg=5353.73, stdev=2880.07 lat (usec): min=2614, max=58861, avg=5450.82, stdev=2878.50 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2704], 20.00th=[ 3589], | 30.00th=[ 4359], 40.00th=[ 4752], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5866], 90.00th=[ 8094], 95.00th=[10421], | 99.00th=[13829], 99.50th=[16909], 99.90th=[54264], 99.95th=[55837], | 99.99th=[58983] bw ( KiB/s): min=133386, max=225280, per=100.00%, avg=160973.49, stdev=13678.47, samples=59 iops : min= 130, max= 220, avg=157.19, stdev=13.38, samples=59 lat (usec) : 250=78.30%, 500=1.06%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.82%, 10=14.22%, 20=1.50%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=2.12%, sys=1.45%, ctx=23360, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18637,4715,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=621MiB/s (651MB/s), 621MiB/s-621MiB/s (651MB/s-651MB/s), io=18.2GiB (19.5GB), run=30001-30001msec WRITE: bw=157MiB/s (165MB/s), 157MiB/s-157MiB/s (165MB/s-165MB/s), io=4715MiB (4944MB), run=30001-30001msec Disk stats (read/write): vdd: ios=18555/4694, merge=0/0, ticks=1329/22550, in_queue=28739, util=95.94% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961053: Fri Jan 7 14:24:32 2022 read: IOPS=606, BW=606MiB/s (635MB/s)(17.8GiB/30047msec) slat (usec): min=10, max=219, avg=22.91, stdev= 9.54 clat (usec): min=15, max=95831, avg=375.06, stdev=3012.09 lat (usec): min=115, max=95850, avg=398.15, stdev=3012.15 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 110], 10.00th=[ 112], 20.00th=[ 115], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 126], 60.00th=[ 133], | 70.00th=[ 149], 80.00th=[ 326], 90.00th=[ 347], 95.00th=[ 371], | 99.00th=[ 594], 99.50th=[17171], 99.90th=[50070], 99.95th=[64750], | 99.99th=[94897] bw ( KiB/s): min=405504, max=948224, per=100.00%, avg=621586.47, stdev=102637.51, samples=60 iops : min= 396, max= 926, avg=607.02, stdev=100.23, samples=60 write: IOPS=153, BW=153MiB/s (161MB/s)(4606MiB/30047msec) slat (usec): min=61, max=599, avg=168.49, stdev=86.67 clat (msec): min=3, max=160, avg=63.47, stdev=26.06 lat (msec): min=3, max=160, avg=63.64, stdev=26.06 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 25], 20.00th=[ 41], | 30.00th=[ 54], 40.00th=[ 61], 50.00th=[ 66], 60.00th=[ 71], | 70.00th=[ 77], 80.00th=[ 83], 90.00th=[ 94], 95.00th=[ 107], | 99.00th=[ 133], 99.50th=[ 140], 99.90th=[ 150], 99.95th=[ 153], | 99.99th=[ 161] bw ( KiB/s): min=126976, max=239616, per=99.94%, avg=156882.07, stdev=14044.30, samples=60 iops : min= 124, max= 234, avg=153.20, stdev=13.71, samples=60 lat (usec) : 20=0.01%, 100=0.15%, 250=61.94%, 500=16.64%, 750=0.40% lat (usec) : 1000=0.06% lat (msec) : 2=0.09%, 4=0.11%, 10=0.10%, 20=1.24%, 50=4.19% lat (msec) : 100=13.65%, 250=1.43% cpu : usr=2.69%, sys=2.31%, ctx=22319, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18210,4606,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=606MiB/s (635MB/s), 606MiB/s-606MiB/s (635MB/s-635MB/s), io=17.8GiB (19.1GB), run=30047-30047msec WRITE: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=4606MiB (4830MB), run=30047-30047msec Disk stats (read/write): vdd: ios=18139/4573, merge=0/0, ticks=3579/287831, in_queue=297321, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961576: Fri Jan 7 14:25:09 2022 read: IOPS=612, BW=613MiB/s (642MB/s)(17.9GiB/30001msec) clat (usec): min=132, max=56574, avg=245.41, stdev=1168.67 lat (usec): min=132, max=56574, avg=245.56, stdev=1168.67 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 145], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 163], | 70.00th=[ 172], 80.00th=[ 182], 90.00th=[ 196], 95.00th=[ 210], | 99.00th=[ 302], 99.50th=[ 8848], 99.90th=[18220], 99.95th=[19530], | 99.99th=[24773] bw ( KiB/s): min=444416, max=811008, per=100.00%, avg=627302.40, stdev=92007.76, samples=60 iops : min= 434, max= 792, avg=612.60, stdev=89.85, samples=60 write: IOPS=155, BW=155MiB/s (163MB/s)(4659MiB/30001msec) clat (usec): min=2507, max=59103, avg=5362.27, stdev=2777.75 lat (usec): min=2596, max=59218, avg=5459.88, stdev=2776.46 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2704], 20.00th=[ 3458], | 30.00th=[ 4359], 40.00th=[ 4752], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8455], 95.00th=[10683], | 99.00th=[14091], 99.50th=[17433], 99.90th=[22938], 99.95th=[54789], | 99.99th=[58983] bw ( KiB/s): min=133120, max=221184, per=99.98%, avg=158993.07, stdev=13400.31, samples=60 iops : min= 130, max= 216, avg=155.27, stdev=13.09, samples=60 lat (usec) : 250=78.54%, 500=0.79%, 750=0.01% lat (msec) : 2=0.01%, 4=5.00%, 10=14.00%, 20=1.56%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=2.37%, ctx=23048, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18378,4659,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=613MiB/s (642MB/s), 613MiB/s-613MiB/s (642MB/s-642MB/s), io=17.9GiB (19.3GB), run=30001-30001msec WRITE: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=4659MiB (4885MB), run=30001-30001msec Disk stats (read/write): vdd: ios=18287/4630, merge=0/0, ticks=1422/22265, in_queue=28465, util=95.04% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962975: Fri Jan 7 14:25:46 2022 read: IOPS=625, BW=625MiB/s (655MB/s)(18.5GiB/30319msec) slat (usec): min=9, max=584, avg=20.34, stdev=11.29 clat (usec): min=23, max=814306, avg=1371.79, stdev=15619.08 lat (usec): min=117, max=814322, avg=1392.31, stdev=15619.01 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 111], 10.00th=[ 113], 20.00th=[ 117], | 30.00th=[ 120], 40.00th=[ 123], 50.00th=[ 129], 60.00th=[ 141], | 70.00th=[ 210], 80.00th=[ 343], 90.00th=[ 420], 95.00th=[ 2008], | 99.00th=[ 12125], 99.50th=[ 37487], 99.90th=[235930], 99.95th=[379585], | 99.99th=[624952] bw ( KiB/s): min=411648, max=1236992, per=100.00%, avg=646878.30, stdev=134033.67, samples=60 iops : min= 402, max= 1208, avg=631.75, stdev=130.85, samples=60 write: IOPS=157, BW=158MiB/s (165MB/s)(4778MiB/30319msec) slat (usec): min=62, max=655, avg=185.83, stdev=94.15 clat (msec): min=9, max=1284, avg=311.46, stdev=168.47 lat (msec): min=9, max=1284, avg=311.65, stdev=168.46 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 116], 10.00th=[ 129], 20.00th=[ 155], | 30.00th=[ 190], 40.00th=[ 234], 50.00th=[ 284], 60.00th=[ 338], | 70.00th=[ 393], 80.00th=[ 451], 90.00th=[ 527], 95.00th=[ 600], | 99.00th=[ 818], 99.50th=[ 911], 99.90th=[ 1167], 99.95th=[ 1250], | 99.99th=[ 1284] bw ( KiB/s): min=122880, max=260096, per=100.00%, avg=161421.38, stdev=18264.98, samples=60 iops : min= 120, max= 254, avg=157.63, stdev=17.84, samples=60 lat (usec) : 50=0.01%, 100=0.09%, 250=56.84%, 500=15.84%, 750=1.07% lat (usec) : 1000=0.57% lat (msec) : 2=1.44%, 4=1.43%, 10=1.31%, 20=0.89%, 50=0.15% lat (msec) : 100=0.55%, 250=8.49%, 500=8.78%, 750=2.22%, 1000=0.29% lat (msec) : 2000=0.05% cpu : usr=3.23%, sys=2.01%, ctx=21400, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=18952,4778,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=625MiB/s (655MB/s), 625MiB/s-625MiB/s (655MB/s-655MB/s), io=18.5GiB (19.9GB), run=30319-30319msec WRITE: bw=158MiB/s (165MB/s), 158MiB/s-158MiB/s (165MB/s-165MB/s), io=4778MiB (5010MB), run=30319-30319msec Disk stats (read/write): vdd: ios=18952/4742, merge=0/0, ticks=20982/1463778, in_queue=1501078, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963560: Fri Jan 7 14:26:23 2022 read: IOPS=616, BW=616MiB/s (646MB/s)(18.1GiB/30005msec) clat (usec): min=132, max=23151, avg=246.87, stdev=1099.11 lat (usec): min=132, max=23152, avg=247.03, stdev=1099.11 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 151], | 30.00th=[ 155], 40.00th=[ 159], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 178], 80.00th=[ 186], 90.00th=[ 204], 95.00th=[ 221], | 99.00th=[ 322], 99.50th=[ 5407], 99.90th=[18744], 99.95th=[19792], | 99.99th=[22676] bw ( KiB/s): min=473088, max=778240, per=100.00%, avg=631419.80, stdev=77331.40, samples=60 iops : min= 462, max= 760, avg=616.62, stdev=75.52, samples=60 write: IOPS=155, BW=156MiB/s (164MB/s)(4680MiB/30005msec) clat (usec): min=2527, max=63575, avg=5327.92, stdev=2892.16 lat (usec): min=2602, max=63667, avg=5425.96, stdev=2891.17 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3392], | 30.00th=[ 4293], 40.00th=[ 4686], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5473], 80.00th=[ 5932], 90.00th=[ 8356], 95.00th=[10421], | 99.00th=[14222], 99.50th=[15926], 99.90th=[50594], 99.95th=[55313], | 99.99th=[63701] bw ( KiB/s): min=131072, max=223232, per=100.00%, avg=159715.05, stdev=13544.03, samples=60 iops : min= 128, max= 218, avg=155.97, stdev=13.23, samples=60 lat (usec) : 250=78.10%, 500=1.23%, 750=0.04%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.26%, 10=13.77%, 20=1.51%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=2.07%, sys=2.39%, ctx=23194, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18498,4680,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=616MiB/s (646MB/s), 616MiB/s-616MiB/s (646MB/s-646MB/s), io=18.1GiB (19.4GB), run=30005-30005msec WRITE: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4680MiB (4907MB), run=30005-30005msec Disk stats (read/write): vdd: ios=18331/4649, merge=0/0, ticks=1377/22186, in_queue=28368, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964376: Fri Jan 7 14:27:00 2022 read: IOPS=628, BW=629MiB/s (659MB/s)(18.8GiB/30614msec) slat (usec): min=9, max=798, avg=19.32, stdev=18.44 clat (usec): min=2, max=835623, avg=2909.49, stdev=15174.97 lat (usec): min=113, max=835638, avg=2928.97, stdev=15175.21 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 110], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 122], 50.00th=[ 133], 60.00th=[ 204], | 70.00th=[ 338], 80.00th=[ 469], 90.00th=[ 8225], 95.00th=[ 17957], | 99.00th=[ 28705], 99.50th=[ 35914], 99.90th=[187696], 99.95th=[291505], | 99.99th=[692061] bw ( KiB/s): min=47104, max=1566720, per=100.00%, avg=657062.75, stdev=250640.01, samples=60 iops : min= 46, max= 1530, avg=641.65, stdev=244.76, samples=60 write: IOPS=158, BW=159MiB/s (166MB/s)(4854MiB/30614msec) slat (usec): min=62, max=1316, avg=196.34, stdev=107.85 clat (msec): min=22, max=2159, avg=618.44, stdev=233.54 lat (msec): min=23, max=2159, avg=618.64, stdev=233.52 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 288], 10.00th=[ 372], 20.00th=[ 447], | 30.00th=[ 485], 40.00th=[ 531], 50.00th=[ 584], 60.00th=[ 651], | 70.00th=[ 709], 80.00th=[ 785], 90.00th=[ 911], 95.00th=[ 1045], | 99.00th=[ 1318], 99.50th=[ 1418], 99.90th=[ 1737], 99.95th=[ 1955], | 99.99th=[ 2165] bw ( KiB/s): min= 2048, max=305152, per=98.34%, avg=159659.41, stdev=56686.18, samples=61 iops : min= 2, max= 298, avg=155.90, stdev=55.36, samples=61 lat (usec) : 4=0.01%, 50=0.01%, 100=0.12%, 250=49.24%, 500=14.92% lat (usec) : 750=1.59%, 1000=0.72% lat (msec) : 2=1.49%, 4=1.41%, 10=2.96%, 20=4.45%, 50=2.61% lat (msec) : 100=0.20%, 250=0.97%, 500=5.97%, 750=8.39%, 1000=3.73% lat (msec) : 2000=1.22%, >=2000=0.01% cpu : usr=3.34%, sys=1.88%, ctx=18916, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=19248,4854,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=629MiB/s (659MB/s), 629MiB/s-629MiB/s (659MB/s-659MB/s), io=18.8GiB (20.2GB), run=30614-30614msec WRITE: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=4854MiB (5090MB), run=30614-30614msec Disk stats (read/write): vdd: ios=19248/4845, merge=0/0, ticks=46396/2960567, in_queue=3020633, util=99.54% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965441: Fri Jan 7 14:27:37 2022 read: IOPS=619, BW=620MiB/s (650MB/s)(18.2GiB/30004msec) clat (usec): min=118, max=27595, avg=232.04, stdev=1126.76 lat (usec): min=118, max=27595, avg=232.20, stdev=1126.76 clat percentiles (usec): | 1.00th=[ 127], 5.00th=[ 131], 10.00th=[ 133], 20.00th=[ 137], | 30.00th=[ 139], 40.00th=[ 141], 50.00th=[ 145], 60.00th=[ 151], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 184], 95.00th=[ 200], | 99.00th=[ 310], 99.50th=[ 6390], 99.90th=[18744], 99.95th=[19530], | 99.99th=[22414] bw ( KiB/s): min=434176, max=847872, per=100.00%, avg=634948.27, stdev=92576.73, samples=60 iops : min= 424, max= 828, avg=620.07, stdev=90.41, samples=60 write: IOPS=156, BW=157MiB/s (164MB/s)(4702MiB/30004msec) clat (usec): min=2478, max=57472, avg=5357.48, stdev=2874.07 lat (usec): min=2571, max=57570, avg=5454.07, stdev=2872.69 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2704], 20.00th=[ 3556], | 30.00th=[ 4424], 40.00th=[ 4752], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8160], 95.00th=[10290], | 99.00th=[14222], 99.50th=[16909], 99.90th=[55313], 99.95th=[56361], | 99.99th=[57410] bw ( KiB/s): min=129024, max=221184, per=99.99%, avg=160460.80, stdev=13930.20, samples=60 iops : min= 126, max= 216, avg=156.70, stdev=13.60, samples=60 lat (usec) : 250=78.54%, 500=0.83%, 750=0.03% lat (msec) : 2=0.01%, 4=4.87%, 10=14.23%, 20=1.42%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.96%, sys=1.63%, ctx=23315, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18602,4702,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=620MiB/s (650MB/s), 620MiB/s-620MiB/s (650MB/s-650MB/s), io=18.2GiB (19.5GB), run=30004-30004msec WRITE: bw=157MiB/s (164MB/s), 157MiB/s-157MiB/s (164MB/s-164MB/s), io=4702MiB (4930MB), run=30004-30004msec Disk stats (read/write): vdd: ios=18586/4699, merge=0/0, ticks=1427/22571, in_queue=28858, util=95.54% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966061: Fri Jan 7 14:28:14 2022 read: IOPS=627, BW=627MiB/s (658MB/s)(18.8GiB/30671msec) slat (usec): min=10, max=709800, avg=1096.66, stdev=8997.91 clat (msec): min=18, max=1315, avg=104.78, stdev=102.40 lat (msec): min=18, max=1315, avg=105.87, stdev=102.83 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 41], 20.00th=[ 56], | 30.00th=[ 68], 40.00th=[ 79], 50.00th=[ 89], 60.00th=[ 100], | 70.00th=[ 112], 80.00th=[ 127], 90.00th=[ 150], 95.00th=[ 178], | 99.00th=[ 760], 99.50th=[ 793], 99.90th=[ 818], 99.95th=[ 835], | 99.99th=[ 1045] bw ( KiB/s): min= 2048, max=1499136, per=100.00%, avg=644390.16, stdev=262286.61, samples=61 iops : min= 2, max= 1464, avg=629.28, stdev=256.13, samples=61 write: IOPS=158, BW=158MiB/s (166MB/s)(4853MiB/30671msec) slat (usec): min=270, max=705516, avg=1822.79, stdev=18066.36 clat (msec): min=60, max=2397, avg=841.41, stdev=288.15 lat (msec): min=60, max=2397, avg=843.23, stdev=287.47 clat percentiles (msec): | 1.00th=[ 165], 5.00th=[ 409], 10.00th=[ 535], 20.00th=[ 634], | 30.00th=[ 693], 40.00th=[ 760], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 936], 80.00th=[ 1020], 90.00th=[ 1217], 95.00th=[ 1385], | 99.00th=[ 1720], 99.50th=[ 1804], 99.90th=[ 2165], 99.95th=[ 2265], | 99.99th=[ 2400] bw ( KiB/s): min=22528, max=295503, per=100.00%, avg=163339.76, stdev=56798.94, samples=59 iops : min= 22, max= 288, avg=159.49, stdev=55.45, samples=59 lat (msec) : 20=0.02%, 50=12.53%, 100=36.14%, 250=29.22%, 500=2.22% lat (msec) : 750=6.72%, 1000=8.66%, 2000=4.44%, >=2000=0.04% cpu : usr=5.43%, sys=1.56%, ctx=2335, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=19242,4853,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=627MiB/s (658MB/s), 627MiB/s-627MiB/s (658MB/s-658MB/s), io=18.8GiB (20.2GB), run=30671-30671msec WRITE: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4853MiB (5089MB), run=30671-30671msec Disk stats (read/write): vdd: ios=19242/4810, merge=0/0, ticks=118223/3573843, in_queue=3734530, util=99.35% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966913: Fri Jan 7 14:28:51 2022 read: IOPS=258, BW=517MiB/s (542MB/s)(15.2GiB/30005msec) clat (usec): min=253, max=23590, avg=475.89, stdev=1074.00 lat (usec): min=253, max=23590, avg=476.08, stdev=1074.00 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 269], 10.00th=[ 273], 20.00th=[ 281], | 30.00th=[ 289], 40.00th=[ 297], 50.00th=[ 306], 60.00th=[ 322], | 70.00th=[ 355], 80.00th=[ 586], 90.00th=[ 709], 95.00th=[ 750], | 99.00th=[ 898], 99.50th=[10421], 99.90th=[15533], 99.95th=[17171], | 99.99th=[23462] bw ( KiB/s): min=278528, max=770048, per=100.00%, avg=529625.35, stdev=108164.84, samples=60 iops : min= 136, max= 376, avg=258.57, stdev=52.78, samples=60 write: IOPS=65, BW=131MiB/s (138MB/s)(3944MiB/30005msec) clat (usec): min=5204, max=51731, avg=13057.02, stdev=3145.39 lat (usec): min=5745, max=51932, avg=13331.04, stdev=3164.05 clat percentiles (usec): | 1.00th=[ 6783], 5.00th=[ 8455], 10.00th=[ 9241], 20.00th=[10421], | 30.00th=[11338], 40.00th=[12387], 50.00th=[13435], 60.00th=[14222], | 70.00th=[14746], 80.00th=[15401], 90.00th=[16188], 95.00th=[16909], | 99.00th=[20579], 99.50th=[23200], 99.90th=[49021], 99.95th=[51643], | 99.99th=[51643] bw ( KiB/s): min=122880, max=151552, per=100.00%, avg=134638.53, stdev=6698.56, samples=60 iops : min= 60, max= 74, avg=65.73, stdev= 3.28, samples=60 lat (usec) : 500=62.63%, 750=13.02%, 1000=3.43% lat (msec) : 2=0.05%, 4=0.01%, 10=3.49%, 20=17.10%, 50=0.25% lat (msec) : 100=0.01% cpu : usr=1.94%, sys=2.00%, ctx=9737, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7758,1972,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=517MiB/s (542MB/s), 517MiB/s-517MiB/s (542MB/s-542MB/s), io=15.2GiB (16.3GB), run=30005-30005msec WRITE: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3944MiB (4136MB), run=30005-30005msec Disk stats (read/write): vdd: ios=15454/3928, merge=0/0, ticks=716/47919, in_queue=55658, util=95.85% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967287: Fri Jan 7 14:29:28 2022 read: IOPS=350, BW=701MiB/s (735MB/s)(20.5GiB/30006msec) slat (usec): min=60, max=210, avg=89.19, stdev=17.94 clat (usec): min=243, max=442619, avg=770.50, stdev=9310.33 lat (usec): min=328, max=442708, avg=859.94, stdev=9310.32 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 277], 10.00th=[ 281], 20.00th=[ 289], | 30.00th=[ 293], 40.00th=[ 302], 50.00th=[ 314], 60.00th=[ 334], | 70.00th=[ 383], 80.00th=[ 416], 90.00th=[ 506], 95.00th=[ 644], | 99.00th=[ 947], 99.50th=[ 14353], 99.90th=[154141], 99.95th=[248513], | 99.99th=[354419] bw ( KiB/s): min=77824, max=1167360, per=100.00%, avg=717888.82, stdev=188856.28, samples=60 iops : min= 38, max= 570, avg=350.50, stdev=92.22, samples=60 write: IOPS=89, BW=178MiB/s (187MB/s)(5344MiB/30006msec) slat (usec): min=148, max=798, avg=286.85, stdev=77.71 clat (usec): min=5347, max=58084, avg=7548.28, stdev=3194.65 lat (usec): min=5575, max=58374, avg=7835.46, stdev=3198.92 clat percentiles (usec): | 1.00th=[ 5604], 5.00th=[ 5735], 10.00th=[ 5800], 20.00th=[ 5932], | 30.00th=[ 5997], 40.00th=[ 6063], 50.00th=[ 6259], 60.00th=[ 6718], | 70.00th=[ 7504], 80.00th=[ 8586], 90.00th=[10028], 95.00th=[14877], | 99.00th=[19006], 99.50th=[22938], 99.90th=[31065], 99.95th=[55313], | 99.99th=[57934] bw ( KiB/s): min=20480, max=258048, per=99.99%, avg=182355.30, stdev=40932.20, samples=60 iops : min= 10, max= 126, avg=89.03, stdev=19.99, samples=60 lat (usec) : 250=0.02%, 500=71.34%, 750=6.45%, 1000=1.23% lat (msec) : 2=0.07%, 10=18.34%, 20=2.19%, 50=0.26%, 100=0.03% lat (msec) : 250=0.05%, 500=0.04% cpu : usr=2.29%, sys=4.03%, ctx=13200, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10515,2672,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=701MiB/s (735MB/s), 701MiB/s-701MiB/s (735MB/s-735MB/s), io=20.5GiB (22.1GB), run=30006-30006msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=5344MiB (5604MB), run=30006-30006msec Disk stats (read/write): vdd: ios=31293/7977, merge=0/0, ticks=5008/56078, in_queue=76575, util=94.83% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968306: Fri Jan 7 14:30:05 2022 read: IOPS=265, BW=532MiB/s (558MB/s)(15.6GiB/30006msec) clat (usec): min=238, max=24410, avg=416.31, stdev=1159.93 lat (usec): min=239, max=24410, avg=416.48, stdev=1159.94 clat percentiles (usec): | 1.00th=[ 245], 5.00th=[ 251], 10.00th=[ 255], 20.00th=[ 260], | 30.00th=[ 265], 40.00th=[ 269], 50.00th=[ 277], 60.00th=[ 285], | 70.00th=[ 302], 80.00th=[ 347], 90.00th=[ 494], 95.00th=[ 635], | 99.00th=[ 816], 99.50th=[10814], 99.90th=[16909], 99.95th=[20841], | 99.99th=[24511] bw ( KiB/s): min=311296, max=950272, per=100.00%, avg=544721.20, stdev=123632.32, samples=60 iops : min= 152, max= 464, avg=265.97, stdev=60.36, samples=60 write: IOPS=67, BW=135MiB/s (142MB/s)(4056MiB/30006msec) clat (usec): min=4670, max=45288, avg=12916.72, stdev=3259.40 lat (usec): min=4863, max=45576, avg=13146.89, stdev=3268.68 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7308], 10.00th=[ 8586], 20.00th=[10159], | 30.00th=[11207], 40.00th=[12125], 50.00th=[13566], 60.00th=[14353], | 70.00th=[14877], 80.00th=[15533], 90.00th=[16319], 95.00th=[16909], | 99.00th=[18482], 99.50th=[21103], 99.90th=[33817], 99.95th=[42730], | 99.99th=[45351] bw ( KiB/s): min=122880, max=155648, per=99.97%, avg=138381.18, stdev=6957.22, samples=60 iops : min= 60, max= 76, avg=67.57, stdev= 3.40, samples=60 lat (usec) : 250=2.62%, 500=69.40%, 750=6.69%, 1000=0.41% lat (msec) : 2=0.01%, 10=3.98%, 20=16.72%, 50=0.18% cpu : usr=1.83%, sys=1.63%, ctx=10015, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7979,2028,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=532MiB/s (558MB/s), 532MiB/s-532MiB/s (558MB/s-558MB/s), io=15.6GiB (16.7GB), run=30006-30006msec WRITE: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=4056MiB (4253MB), run=30006-30006msec Disk stats (read/write): vdd: ios=15892/4030, merge=0/0, ticks=886/41789, in_queue=48959, util=96.39% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969565: Fri Jan 7 14:30:42 2022 read: IOPS=362, BW=726MiB/s (761MB/s)(21.4GiB/30139msec) slat (usec): min=20, max=223, avg=37.22, stdev=14.08 clat (usec): min=126, max=2500.7k, avg=3050.54, stdev=57086.04 lat (usec): min=208, max=2500.7k, avg=3087.97, stdev=57086.02 clat percentiles (usec): | 1.00th=[ 188], 5.00th=[ 192], 10.00th=[ 194], | 20.00th=[ 200], 30.00th=[ 204], 40.00th=[ 208], | 50.00th=[ 215], 60.00th=[ 227], 70.00th=[ 260], | 80.00th=[ 627], 90.00th=[ 660], 95.00th=[ 693], | 99.00th=[ 1020], 99.50th=[ 88605], 99.90th=[ 851444], | 99.95th=[1434452], 99.99th=[2365588] bw ( KiB/s): min=503808, max=1245184, per=100.00%, avg=746722.95, stdev=136491.65, samples=60 iops : min= 246, max= 608, avg=364.60, stdev=66.65, samples=60 write: IOPS=92, BW=185MiB/s (194MB/s)(5564MiB/30139msec) slat (usec): min=117, max=1015, avg=350.16, stdev=183.80 clat (msec): min=8, max=2499, avg=95.78, stdev=129.76 lat (msec): min=8, max=2499, avg=96.13, stdev=129.76 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 19], 10.00th=[ 20], 20.00th=[ 29], | 30.00th=[ 37], 40.00th=[ 44], 50.00th=[ 54], 60.00th=[ 71], | 70.00th=[ 92], 80.00th=[ 133], 90.00th=[ 207], 95.00th=[ 296], | 99.00th=[ 709], 99.50th=[ 818], 99.90th=[ 1351], 99.95th=[ 1620], | 99.99th=[ 2500] bw ( KiB/s): min=159744, max=204800, per=100.00%, avg=189377.88, stdev=8372.58, samples=60 iops : min= 78, max= 100, avg=92.47, stdev= 4.09, samples=60 lat (usec) : 250=54.15%, 500=8.02%, 750=15.45%, 1000=1.27% lat (msec) : 2=0.15%, 4=0.04%, 10=0.16%, 20=2.16%, 50=7.20% lat (msec) : 100=5.51%, 250=4.32%, 500=1.07%, 750=0.26%, 1000=0.14% lat (msec) : 2000=0.07%, >=2000=0.03% cpu : usr=3.27%, sys=2.00%, ctx=13703, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10940,2782,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=726MiB/s (761MB/s), 726MiB/s-726MiB/s (761MB/s-761MB/s), io=21.4GiB (22.9GB), run=30139-30139msec WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5564MiB (5834MB), run=30139-30139msec Disk stats (read/write): vdd: ios=21879/5564, merge=0/0, ticks=34083/528275, in_queue=572241, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970099: Fri Jan 7 14:31:19 2022 read: IOPS=266, BW=532MiB/s (558MB/s)(15.6GiB/30008msec) clat (usec): min=235, max=24759, avg=398.22, stdev=1139.50 lat (usec): min=235, max=24759, avg=398.39, stdev=1139.50 clat percentiles (usec): | 1.00th=[ 245], 5.00th=[ 251], 10.00th=[ 253], 20.00th=[ 260], | 30.00th=[ 265], 40.00th=[ 269], 50.00th=[ 277], 60.00th=[ 285], | 70.00th=[ 306], 80.00th=[ 343], 90.00th=[ 400], 95.00th=[ 474], | 99.00th=[ 799], 99.50th=[10683], 99.90th=[16712], 99.95th=[21365], | 99.99th=[24773] bw ( KiB/s): min=311296, max=974848, per=100.00%, avg=545360.90, stdev=122533.21, samples=60 iops : min= 152, max= 476, avg=266.27, stdev=59.84, samples=60 write: IOPS=67, BW=135MiB/s (142MB/s)(4060MiB/30008msec) clat (usec): min=4654, max=48881, avg=12994.05, stdev=3253.44 lat (usec): min=4847, max=49087, avg=13204.73, stdev=3254.63 clat percentiles (usec): | 1.00th=[ 5342], 5.00th=[ 7373], 10.00th=[ 8717], 20.00th=[10290], | 30.00th=[11338], 40.00th=[12256], 50.00th=[13566], 60.00th=[14484], | 70.00th=[15008], 80.00th=[15533], 90.00th=[16450], 95.00th=[17171], | 99.00th=[18744], 99.50th=[21890], 99.90th=[25560], 99.95th=[42730], | 99.99th=[49021] bw ( KiB/s): min=106496, max=151552, per=99.99%, avg=138527.13, stdev=7412.43, samples=60 iops : min= 52, max= 74, avg=67.63, stdev= 3.62, samples=60 lat (usec) : 250=4.00%, 500=72.60%, 750=2.18%, 1000=0.35% lat (msec) : 2=0.03%, 10=3.79%, 20=16.87%, 50=0.18% cpu : usr=1.60%, sys=1.52%, ctx=10027, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7988,2030,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=532MiB/s (558MB/s), 532MiB/s-532MiB/s (558MB/s-558MB/s), io=15.6GiB (16.8GB), run=30008-30008msec WRITE: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=4060MiB (4257MB), run=30008-30008msec Disk stats (read/write): vdd: ios=15882/4030, merge=0/0, ticks=849/43143, in_queue=50080, util=96.44% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971111: Fri Jan 7 14:31:56 2022 read: IOPS=370, BW=741MiB/s (777MB/s)(22.1GiB/30561msec) slat (usec): min=20, max=508, avg=33.55, stdev=17.51 clat (usec): min=141, max=2570.0k, avg=6285.65, stdev=85916.54 lat (usec): min=208, max=2570.0k, avg=6319.39, stdev=85916.57 clat percentiles (usec): | 1.00th=[ 192], 5.00th=[ 198], 10.00th=[ 202], | 20.00th=[ 206], 30.00th=[ 210], 40.00th=[ 217], | 50.00th=[ 225], 60.00th=[ 247], 70.00th=[ 635], | 80.00th=[ 668], 90.00th=[ 1926], 95.00th=[ 9634], | 99.00th=[ 25297], 99.50th=[ 143655], 99.90th=[1686111], | 99.95th=[2298479], 99.99th=[2533360] bw ( KiB/s): min=466944, max=1216512, per=100.00%, avg=772437.33, stdev=167949.19, samples=60 iops : min= 228, max= 594, avg=377.17, stdev=81.95, samples=60 write: IOPS=93, BW=188MiB/s (197MB/s)(5736MiB/30561msec) slat (usec): min=120, max=1117, avg=376.83, stdev=195.17 clat (msec): min=62, max=2913, avg=507.19, stdev=284.27 lat (msec): min=63, max=2914, avg=507.57, stdev=284.27 clat percentiles (msec): | 1.00th=[ 180], 5.00th=[ 253], 10.00th=[ 309], 20.00th=[ 359], | 30.00th=[ 380], 40.00th=[ 397], 50.00th=[ 418], 60.00th=[ 451], | 70.00th=[ 506], 80.00th=[ 600], 90.00th=[ 793], 95.00th=[ 1036], | 99.00th=[ 1821], 99.50th=[ 2056], 99.90th=[ 2500], 99.95th=[ 2869], | 99.99th=[ 2903] bw ( KiB/s): min= 4104, max=258048, per=98.56%, avg=189423.34, stdev=32048.13, samples=61 iops : min= 2, max= 126, avg=92.49, stdev=15.65, samples=61 lat (usec) : 250=48.14%, 500=6.73%, 750=14.20%, 1000=1.46% lat (msec) : 2=1.38%, 4=1.06%, 10=3.04%, 20=2.50%, 50=0.70% lat (msec) : 100=0.15%, 250=1.02%, 500=13.09%, 750=4.02%, 1000=1.23% lat (msec) : 2000=1.10%, >=2000=0.18% cpu : usr=3.65%, sys=1.75%, ctx=12543, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=11317,2868,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=741MiB/s (777MB/s), 741MiB/s-741MiB/s (777MB/s-777MB/s), io=22.1GiB (23.7GB), run=30561-30561msec WRITE: bw=188MiB/s (197MB/s), 188MiB/s-188MiB/s (197MB/s-197MB/s), io=5736MiB (6015MB), run=30561-30561msec Disk stats (read/write): vdd: ios=22634/5648, merge=0/0, ticks=77137/2824250, in_queue=2967017, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972108: Fri Jan 7 14:32:33 2022 read: IOPS=265, BW=531MiB/s (557MB/s)(15.6GiB/30001msec) clat (usec): min=210, max=29689, avg=372.10, stdev=1203.65 lat (usec): min=210, max=29689, avg=372.28, stdev=1203.65 clat percentiles (usec): | 1.00th=[ 219], 5.00th=[ 227], 10.00th=[ 229], 20.00th=[ 235], | 30.00th=[ 239], 40.00th=[ 243], 50.00th=[ 249], 60.00th=[ 258], | 70.00th=[ 273], 80.00th=[ 302], 90.00th=[ 343], 95.00th=[ 404], | 99.00th=[ 775], 99.50th=[10814], 99.90th=[17171], 99.95th=[24511], | 99.99th=[29754] bw ( KiB/s): min=274432, max=937984, per=100.00%, avg=543896.53, stdev=117624.28, samples=60 iops : min= 134, max= 458, avg=265.57, stdev=57.44, samples=60 write: IOPS=67, BW=135MiB/s (141MB/s)(4048MiB/30001msec) clat (usec): min=4645, max=48065, avg=13140.81, stdev=3379.87 lat (usec): min=4831, max=48247, avg=13347.40, stdev=3378.86 clat percentiles (usec): | 1.00th=[ 5342], 5.00th=[ 7373], 10.00th=[ 8717], 20.00th=[10421], | 30.00th=[11469], 40.00th=[12387], 50.00th=[13698], 60.00th=[14484], | 70.00th=[15008], 80.00th=[15664], 90.00th=[16450], 95.00th=[17171], | 99.00th=[18744], 99.50th=[23725], 99.90th=[35390], 99.95th=[45876], | 99.99th=[47973] bw ( KiB/s): min=126976, max=151552, per=99.96%, avg=138107.83, stdev=6308.82, samples=60 iops : min= 62, max= 74, avg=67.43, stdev= 3.08, samples=60 lat (usec) : 250=42.15%, 500=35.25%, 750=1.48%, 1000=0.24% lat (msec) : 2=0.02%, 10=3.58%, 20=17.05%, 50=0.23% cpu : usr=1.49%, sys=1.10%, ctx=9995, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7967,2024,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=531MiB/s (557MB/s), 531MiB/s-531MiB/s (557MB/s-557MB/s), io=15.6GiB (16.7GB), run=30001-30001msec WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4048MiB (4245MB), run=30001-30001msec Disk stats (read/write): vdd: ios=15846/4017, merge=0/0, ticks=903/41176, in_queue=47893, util=96.76% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972501: Fri Jan 7 14:33:10 2022 read: IOPS=367, BW=735MiB/s (771MB/s)(22.0GiB/30720msec) slat (usec): min=21, max=480715, avg=2026.76, stdev=11310.00 clat (msec): min=21, max=2927, avg=88.24, stdev=113.23 lat (msec): min=21, max=2927, avg=90.27, stdev=113.85 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 31], 10.00th=[ 35], 20.00th=[ 45], | 30.00th=[ 54], 40.00th=[ 63], 50.00th=[ 71], 60.00th=[ 81], | 70.00th=[ 91], 80.00th=[ 105], 90.00th=[ 127], 95.00th=[ 155], | 99.00th=[ 527], 99.50th=[ 743], 99.90th=[ 1854], 99.95th=[ 2601], | 99.99th=[ 2702] bw ( KiB/s): min= 4096, max=1388544, per=100.00%, avg=756120.15, stdev=288795.10, samples=61 iops : min= 2, max= 678, avg=369.13, stdev=141.00, samples=61 write: IOPS=93, BW=186MiB/s (195MB/s)(5724MiB/30720msec) slat (usec): min=134, max=75665, avg=2477.34, stdev=5967.01 clat (msec): min=93, max=2882, avg=714.02, stdev=337.67 lat (msec): min=93, max=2884, avg=716.49, stdev=337.38 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 401], 10.00th=[ 439], 20.00th=[ 506], | 30.00th=[ 550], 40.00th=[ 584], 50.00th=[ 617], 60.00th=[ 659], | 70.00th=[ 726], 80.00th=[ 844], 90.00th=[ 1083], 95.00th=[ 1385], | 99.00th=[ 2140], 99.50th=[ 2299], 99.90th=[ 2735], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min=16384, max=319488, per=99.99%, avg=190780.65, stdev=58322.88, samples=60 iops : min= 8, max= 156, avg=93.13, stdev=28.47, samples=60 lat (msec) : 50=20.28%, 100=41.40%, 250=16.33%, 500=4.70%, 750=11.26% lat (msec) : 1000=3.34%, 2000=2.25%, >=2000=0.45% cpu : usr=6.36%, sys=1.68%, ctx=2814, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11289,2862,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=735MiB/s (771MB/s), 735MiB/s-735MiB/s (771MB/s-771MB/s), io=22.0GiB (23.7GB), run=30720-30720msec WRITE: bw=186MiB/s (195MB/s), 186MiB/s-186MiB/s (195MB/s-195MB/s), io=5724MiB (6002MB), run=30720-30720msec Disk stats (read/write): vdd: ios=22578/5698, merge=0/0, ticks=152488/3567299, in_queue=3752246, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973642: Fri Jan 7 14:33:47 2022 read: IOPS=264, BW=530MiB/s (556MB/s)(15.5GiB/30008msec) clat (usec): min=241, max=29653, avg=404.23, stdev=1109.56 lat (usec): min=241, max=29653, avg=404.42, stdev=1109.56 clat percentiles (usec): | 1.00th=[ 247], 5.00th=[ 255], 10.00th=[ 258], 20.00th=[ 265], | 30.00th=[ 269], 40.00th=[ 277], 50.00th=[ 285], 60.00th=[ 297], | 70.00th=[ 314], 80.00th=[ 347], 90.00th=[ 400], 95.00th=[ 486], | 99.00th=[ 791], 99.50th=[10421], 99.90th=[15926], 99.95th=[17171], | 99.99th=[29754] bw ( KiB/s): min=278528, max=811008, per=100.00%, avg=542651.73, stdev=107480.71, samples=60 iops : min= 136, max= 396, avg=264.97, stdev=52.48, samples=60 write: IOPS=67, BW=134MiB/s (141MB/s)(4034MiB/30008msec) clat (usec): min=4660, max=49137, avg=13065.07, stdev=3541.84 lat (usec): min=4851, max=49326, avg=13273.21, stdev=3543.16 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7242], 10.00th=[ 8717], 20.00th=[10290], | 30.00th=[11469], 40.00th=[12256], 50.00th=[13566], 60.00th=[14353], | 70.00th=[14877], 80.00th=[15664], 90.00th=[16450], 95.00th=[17171], | 99.00th=[19530], 99.50th=[24249], 99.90th=[44303], 99.95th=[47449], | 99.99th=[49021] bw ( KiB/s): min=122880, max=155648, per=99.98%, avg=137625.60, stdev=7511.12, samples=60 iops : min= 60, max= 76, avg=67.20, stdev= 3.67, samples=60 lat (usec) : 250=1.55%, 500=74.59%, 750=2.69%, 1000=0.30% lat (msec) : 2=0.03%, 10=3.84%, 20=16.78%, 50=0.22% cpu : usr=1.43%, sys=1.95%, ctx=9976, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7949,2017,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=530MiB/s (556MB/s), 530MiB/s-530MiB/s (556MB/s-556MB/s), io=15.5GiB (16.7GB), run=30008-30008msec WRITE: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=4034MiB (4230MB), run=30008-30008msec Disk stats (read/write): vdd: ios=15846/4014, merge=0/0, ticks=826/42747, in_queue=49550, util=95.50% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974264: Fri Jan 7 14:34:25 2022 read: IOPS=369, BW=739MiB/s (775MB/s)(22.1GiB/30679msec) slat (usec): min=22, max=488438, avg=2024.72, stdev=10474.76 clat (msec): min=84, max=3113, avg=306.49, stdev=151.42 lat (msec): min=84, max=3113, avg=308.52, stdev=151.74 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 129], 10.00th=[ 176], 20.00th=[ 224], | 30.00th=[ 249], 40.00th=[ 271], 50.00th=[ 288], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 355], 90.00th=[ 422], 95.00th=[ 575], | 99.00th=[ 793], 99.50th=[ 894], 99.90th=[ 2165], 99.95th=[ 2802], | 99.99th=[ 2937] bw ( KiB/s): min=81920, max=1335296, per=100.00%, avg=766293.33, stdev=263937.77, samples=60 iops : min= 40, max= 652, avg=374.17, stdev=128.88, samples=60 write: IOPS=93, BW=187MiB/s (197MB/s)(5752MiB/30679msec) slat (usec): min=550, max=470989, avg=2434.50, stdev=10314.92 clat (msec): min=110, max=3720, avg=910.82, stdev=325.86 lat (msec): min=125, max=3721, avg=913.25, stdev=325.55 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 584], 10.00th=[ 634], 20.00th=[ 701], | 30.00th=[ 751], 40.00th=[ 793], 50.00th=[ 835], 60.00th=[ 877], | 70.00th=[ 953], 80.00th=[ 1070], 90.00th=[ 1250], 95.00th=[ 1502], | 99.00th=[ 2265], 99.50th=[ 2635], 99.90th=[ 3373], 99.95th=[ 3540], | 99.99th=[ 3708] bw ( KiB/s): min= 4096, max=352256, per=99.24%, avg=190532.27, stdev=74241.08, samples=60 iops : min= 2, max= 172, avg=93.03, stdev=36.25, samples=60 lat (msec) : 100=1.02%, 250=23.12%, 500=50.45%, 750=10.28%, 1000=9.76% lat (msec) : 2000=4.89%, >=2000=0.47% cpu : usr=6.62%, sys=1.76%, ctx=2803, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11339,2876,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=739MiB/s (775MB/s), 739MiB/s-739MiB/s (775MB/s-775MB/s), io=22.1GiB (23.8GB), run=30679-30679msec WRITE: bw=187MiB/s (197MB/s), 187MiB/s-187MiB/s (197MB/s-197MB/s), io=5752MiB (6031MB), run=30679-30679msec Disk stats (read/write): vdd: ios=22678/5748, merge=0/0, ticks=157517/3573859, in_queue=3751961, util=99.05% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975060: Fri Jan 7 14:35:01 2022 read: IOPS=150, BW=603MiB/s (632MB/s)(17.7GiB/30019msec) clat (usec): min=458, max=33512, avg=905.19, stdev=1537.47 lat (usec): min=458, max=33512, avg=905.40, stdev=1537.48 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 486], 10.00th=[ 494], 20.00th=[ 506], | 30.00th=[ 515], 40.00th=[ 529], 50.00th=[ 545], 60.00th=[ 570], | 70.00th=[ 685], 80.00th=[ 1369], 90.00th=[ 1434], 95.00th=[ 1516], | 99.00th=[ 8225], 99.50th=[14091], 99.90th=[21365], 99.95th=[23987], | 99.99th=[33424] bw ( KiB/s): min=229376, max=1007616, per=100.00%, avg=617458.70, stdev=161528.32, samples=60 iops : min= 56, max= 246, avg=150.73, stdev=39.44, samples=60 write: IOPS=39, BW=156MiB/s (164MB/s)(4696MiB/30019msec) clat (msec): min=10, max=187, avg=21.47, stdev=10.94 lat (msec): min=10, max=188, avg=22.07, stdev=10.97 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 23], 80.00th=[ 25], 90.00th=[ 33], 95.00th=[ 43], | 99.00th=[ 59], 99.50th=[ 66], 99.90th=[ 94], 99.95th=[ 188], | 99.99th=[ 188] bw ( KiB/s): min=114917, max=188416, per=99.99%, avg=160168.62, stdev=13005.34, samples=60 iops : min= 28, max= 46, avg=39.10, stdev= 3.18, samples=60 lat (usec) : 500=12.55%, 750=46.51%, 1000=1.63% lat (msec) : 2=17.50%, 4=0.16%, 10=0.44%, 20=12.34%, 50=8.30% lat (msec) : 100=0.54%, 250=0.02% cpu : usr=2.57%, sys=2.04%, ctx=5707, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4522,1174,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=603MiB/s (632MB/s), 603MiB/s-603MiB/s (632MB/s-632MB/s), io=17.7GiB (18.0GB), run=30019-30019msec WRITE: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4696MiB (4924MB), run=30019-30019msec Disk stats (read/write): vdd: ios=22495/5840, merge=0/0, ticks=5988/113727, in_queue=132680, util=96.50% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976336: Fri Jan 7 14:35:38 2022 read: IOPS=164, BW=660MiB/s (692MB/s)(19.3GiB/30009msec) slat (usec): min=74, max=277, avg=99.07, stdev=18.14 clat (usec): min=273, max=28295, avg=754.29, stdev=1540.61 lat (usec): min=438, max=28394, avg=853.62, stdev=1541.40 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 371], 10.00th=[ 375], 20.00th=[ 383], | 30.00th=[ 392], 40.00th=[ 400], 50.00th=[ 408], 60.00th=[ 420], | 70.00th=[ 482], 80.00th=[ 1074], 90.00th=[ 1270], 95.00th=[ 1303], | 99.00th=[ 9372], 99.50th=[14222], 99.90th=[20055], 99.95th=[24249], | 99.99th=[28181] bw ( KiB/s): min=221184, max=1146880, per=100.00%, avg=675728.63, stdev=169391.91, samples=60 iops : min= 54, max= 280, avg=164.97, stdev=41.35, samples=60 write: IOPS=42, BW=170MiB/s (178MB/s)(5108MiB/30009msec) slat (usec): min=269, max=1691, avg=640.53, stdev=311.29 clat (msec): min=10, max=170, avg=19.54, stdev= 6.02 lat (msec): min=11, max=171, avg=20.18, stdev= 6.06 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 29], 99.50th=[ 32], 99.90th=[ 65], 99.95th=[ 171], | 99.99th=[ 171] bw ( KiB/s): min=122880, max=196608, per=99.96%, avg=174222.27, stdev=11472.40, samples=60 iops : min= 30, max= 48, avg=42.53, stdev= 2.80, samples=60 lat (usec) : 500=56.25%, 750=5.03%, 1000=1.54% lat (msec) : 2=15.56%, 4=0.02%, 10=0.39%, 20=11.47%, 50=9.72% lat (msec) : 100=0.02%, 250=0.02% cpu : usr=2.48%, sys=2.19%, ctx=6239, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4949,1277,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=660MiB/s (692MB/s), 660MiB/s-660MiB/s (692MB/s-692MB/s), io=19.3GiB (20.8GB), run=30009-30009msec WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=5108MiB (5356MB), run=30009-30009msec Disk stats (read/write): vdd: ios=19744/5084, merge=0/0, ticks=4943/90078, in_queue=105069, util=95.92% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977094: Fri Jan 7 14:36:15 2022 read: IOPS=165, BW=662MiB/s (695MB/s)(19.4GiB/30008msec) clat (usec): min=374, max=28372, avg=771.18, stdev=1580.15 lat (usec): min=374, max=28372, avg=771.39, stdev=1580.16 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 433], 60.00th=[ 445], | 70.00th=[ 498], 80.00th=[ 988], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[ 9765], 99.50th=[14222], 99.90th=[23725], 99.95th=[23987], | 99.99th=[28443] bw ( KiB/s): min=344064, max=1220608, per=100.00%, avg=678470.78, stdev=164002.12, samples=60 iops : min= 84, max= 298, avg=165.63, stdev=40.05, samples=60 write: IOPS=42, BW=172MiB/s (180MB/s)(5148MiB/30008msec) clat (msec): min=10, max=201, avg=19.78, stdev= 6.75 lat (msec): min=10, max=201, avg=20.33, stdev= 6.78 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 31], 99.50th=[ 34], 99.90th=[ 74], 99.95th=[ 201], | 99.99th=[ 201] bw ( KiB/s): min=114688, max=196608, per=99.96%, avg=175594.17, stdev=11824.84, samples=60 iops : min= 28, max= 48, avg=42.87, stdev= 2.88, samples=60 lat (usec) : 500=55.79%, 750=5.85%, 1000=1.97% lat (msec) : 2=14.72%, 4=0.02%, 10=0.34%, 20=11.27%, 50=10.02% lat (msec) : 100=0.02%, 250=0.02% cpu : usr=2.50%, sys=1.68%, ctx=6267, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4969,1287,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=662MiB/s (695MB/s), 662MiB/s-662MiB/s (695MB/s-695MB/s), io=19.4GiB (20.8GB), run=30008-30008msec WRITE: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s), io=5148MiB (5398MB), run=30008-30008msec Disk stats (read/write): vdd: ios=19828/5113, merge=0/0, ticks=4519/81278, in_queue=95517, util=96.66% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977786: Fri Jan 7 14:36:52 2022 read: IOPS=196, BW=785MiB/s (823MB/s)(23.1GiB/30153msec) slat (usec): min=41, max=304, avg=61.16, stdev=16.43 clat (usec): min=254, max=2501.7k, avg=7471.24, stdev=91710.41 lat (usec): min=376, max=2501.7k, avg=7532.67, stdev=91709.99 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 343], 10.00th=[ 347], | 20.00th=[ 355], 30.00th=[ 367], 40.00th=[ 375], | 50.00th=[ 383], 60.00th=[ 396], 70.00th=[ 465], | 80.00th=[ 1237], 90.00th=[ 1287], 95.00th=[ 1336], | 99.00th=[ 108528], 99.50th=[ 417334], 99.90th=[1702888], | 99.95th=[2298479], 99.99th=[2499806] bw ( KiB/s): min=368640, max=1515520, per=100.00%, avg=807651.02, stdev=189435.26, samples=60 iops : min= 90, max= 370, avg=197.17, stdev=46.24, samples=60 write: IOPS=51, BW=205MiB/s (215MB/s)(6180MiB/30153msec) slat (usec): min=238, max=2089, avg=768.95, stdev=386.21 clat (msec): min=17, max=2532, avg=165.44, stdev=197.13 lat (msec): min=19, max=2533, avg=166.20, stdev=197.15 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 52], | 30.00th=[ 70], 40.00th=[ 89], 50.00th=[ 107], 60.00th=[ 138], | 70.00th=[ 171], 80.00th=[ 236], 90.00th=[ 347], 95.00th=[ 456], | 99.00th=[ 911], 99.50th=[ 1116], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=180224, max=229376, per=100.00%, avg=209865.95, stdev=10355.37, samples=60 iops : min= 44, max= 56, avg=51.23, stdev= 2.53, samples=60 lat (usec) : 500=56.70%, 750=2.17%, 1000=0.72% lat (msec) : 2=18.16%, 4=0.19%, 10=0.11%, 20=0.25%, 50=3.78% lat (msec) : 100=6.11%, 250=7.50%, 500=3.18%, 750=0.58%, 1000=0.24% lat (msec) : 2000=0.20%, >=2000=0.11% cpu : usr=3.89%, sys=1.63%, ctx=7431, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5917,1545,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=785MiB/s (823MB/s), 785MiB/s-785MiB/s (823MB/s-823MB/s), io=23.1GiB (24.8GB), run=30153-30153msec WRITE: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=6180MiB (6480MB), run=30153-30153msec Disk stats (read/write): vdd: ios=23668/6169, merge=0/0, ticks=53150/992392, in_queue=1060626, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978555: Fri Jan 7 14:37:29 2022 read: IOPS=165, BW=662MiB/s (694MB/s)(19.4GiB/30001msec) clat (usec): min=374, max=28388, avg=764.30, stdev=1574.04 lat (usec): min=374, max=28388, avg=764.49, stdev=1574.05 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 416], 40.00th=[ 420], 50.00th=[ 433], 60.00th=[ 445], | 70.00th=[ 482], 80.00th=[ 930], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[ 9634], 99.50th=[14353], 99.90th=[20317], 99.95th=[24249], | 99.99th=[28443] bw ( KiB/s): min=311296, max=1081344, per=100.00%, avg=680795.51, stdev=174064.48, samples=59 iops : min= 76, max= 264, avg=166.20, stdev=42.49, samples=59 write: IOPS=42, BW=171MiB/s (179MB/s)(5128MiB/30001msec) clat (msec): min=10, max=176, avg=19.88, stdev= 6.34 lat (msec): min=10, max=176, avg=20.43, stdev= 6.37 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 31], 99.50th=[ 40], 99.90th=[ 73], 99.95th=[ 178], | 99.99th=[ 178] bw ( KiB/s): min=114688, max=196608, per=100.00%, avg=175092.47, stdev=12063.11, samples=59 iops : min= 28, max= 48, avg=42.75, stdev= 2.95, samples=59 lat (usec) : 500=56.72%, 750=4.96%, 1000=3.06% lat (msec) : 2=13.62%, 4=0.02%, 10=0.30%, 20=10.97%, 50=10.29% lat (msec) : 100=0.03%, 250=0.02% cpu : usr=2.51%, sys=1.58%, ctx=6254, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4964,1282,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=662MiB/s (694MB/s), 662MiB/s-662MiB/s (694MB/s-694MB/s), io=19.4GiB (20.8GB), run=30001-30001msec WRITE: bw=171MiB/s (179MB/s), 171MiB/s-171MiB/s (179MB/s-179MB/s), io=5128MiB (5377MB), run=30001-30001msec Disk stats (read/write): vdd: ios=19840/5125, merge=0/0, ticks=4280/81885, in_queue=96158, util=96.46% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979526: Fri Jan 7 14:38:06 2022 read: IOPS=192, BW=770MiB/s (808MB/s)(23.0GiB/30612msec) slat (usec): min=45, max=442669, avg=3893.04, stdev=14003.36 clat (msec): min=14, max=2634, avg=79.80, stdev=115.82 lat (msec): min=14, max=2634, avg=83.69, stdev=116.85 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 36], | 30.00th=[ 45], 40.00th=[ 53], 50.00th=[ 61], 60.00th=[ 71], | 70.00th=[ 83], 80.00th=[ 97], 90.00th=[ 117], 95.00th=[ 146], | 99.00th=[ 502], 99.50th=[ 667], 99.90th=[ 1888], 99.95th=[ 2567], | 99.99th=[ 2635] bw ( KiB/s): min= 8192, max=1441792, per=100.00%, avg=789776.52, stdev=319903.91, samples=61 iops : min= 2, max= 352, avg=192.77, stdev=78.09, samples=61 write: IOPS=50, BW=201MiB/s (210MB/s)(6140MiB/30612msec) slat (usec): min=1134, max=464598, avg=4571.54, stdev=14193.63 clat (msec): min=85, max=3872, avg=670.14, stdev=296.52 lat (msec): min=87, max=3873, avg=674.72, stdev=296.61 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 384], 10.00th=[ 464], 20.00th=[ 514], | 30.00th=[ 542], 40.00th=[ 567], 50.00th=[ 600], 60.00th=[ 634], | 70.00th=[ 693], 80.00th=[ 785], 90.00th=[ 969], 95.00th=[ 1116], | 99.00th=[ 1670], 99.50th=[ 2232], 99.90th=[ 3876], 99.95th=[ 3876], | 99.99th=[ 3876] bw ( KiB/s): min=24576, max=417792, per=99.79%, avg=204963.35, stdev=67223.36, samples=60 iops : min= 6, max= 102, avg=50.02, stdev=16.41, samples=60 lat (msec) : 20=0.07%, 50=29.52%, 100=35.34%, 250=12.71%, 500=4.39% lat (msec) : 750=12.84%, 1000=3.16%, 2000=1.78%, >=2000=0.20% cpu : usr=6.85%, sys=1.68%, ctx=2794, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5895,1535,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=770MiB/s (808MB/s), 770MiB/s-770MiB/s (808MB/s-808MB/s), io=23.0GiB (24.7GB), run=30612-30612msec WRITE: bw=201MiB/s (210MB/s), 201MiB/s-201MiB/s (210MB/s-210MB/s), io=6140MiB (6438MB), run=30612-30612msec Disk stats (read/write): vdd: ios=23579/6120, merge=0/0, ticks=138257/3598350, in_queue=3764507, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980488: Fri Jan 7 14:38:43 2022 read: IOPS=165, BW=664MiB/s (696MB/s)(19.5GiB/30017msec) clat (usec): min=377, max=28404, avg=763.82, stdev=1569.21 lat (usec): min=377, max=28404, avg=764.02, stdev=1569.22 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 445], | 70.00th=[ 490], 80.00th=[ 947], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[ 9634], 99.50th=[14222], 99.90th=[22938], 99.95th=[24511], | 99.99th=[28443] bw ( KiB/s): min=294912, max=1081344, per=100.00%, avg=680072.53, stdev=161508.07, samples=60 iops : min= 72, max= 264, avg=166.03, stdev=39.43, samples=60 write: IOPS=43, BW=172MiB/s (181MB/s)(5172MiB/30017msec) clat (usec): min=10314, max=77704, avg=19707.13, stdev=4476.91 lat (usec): min=10741, max=78107, avg=20260.34, stdev=4528.95 clat percentiles (usec): | 1.00th=[11600], 5.00th=[13435], 10.00th=[14222], 20.00th=[15401], | 30.00th=[16712], 40.00th=[17957], 50.00th=[20055], 60.00th=[21627], | 70.00th=[22676], 80.00th=[23462], 90.00th=[24511], 95.00th=[25035], | 99.00th=[28705], 99.50th=[32113], 99.90th=[40109], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min=155648, max=196608, per=99.98%, avg=176401.07, stdev=8854.80, samples=60 iops : min= 38, max= 48, avg=43.07, stdev= 2.16, samples=60 lat (usec) : 500=55.93%, 750=5.55%, 1000=2.84% lat (msec) : 2=13.99%, 10=0.35%, 20=10.93%, 50=10.39%, 100=0.02% cpu : usr=2.56%, sys=1.46%, ctx=6277, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4981,1293,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=664MiB/s (696MB/s), 664MiB/s-664MiB/s (696MB/s-696MB/s), io=19.5GiB (20.9GB), run=30017-30017msec WRITE: bw=172MiB/s (181MB/s), 172MiB/s-172MiB/s (181MB/s-181MB/s), io=5172MiB (5423MB), run=30017-30017msec Disk stats (read/write): vdd: ios=19904/5152, merge=0/0, ticks=4343/81491, in_queue=95935, util=96.16% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981072: Fri Jan 7 14:39:21 2022 read: IOPS=193, BW=775MiB/s (812MB/s)(23.1GiB/30589msec) slat (usec): min=46, max=430533, avg=3647.36, stdev=10023.75 clat (msec): min=81, max=3156, avg=285.97, stdev=156.67 lat (msec): min=81, max=3156, avg=289.62, stdev=157.09 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 128], 10.00th=[ 165], 20.00th=[ 199], | 30.00th=[ 222], 40.00th=[ 245], 50.00th=[ 266], 60.00th=[ 284], | 70.00th=[ 309], 80.00th=[ 334], 90.00th=[ 401], 95.00th=[ 558], | 99.00th=[ 818], 99.50th=[ 894], 99.90th=[ 2702], 99.95th=[ 2836], | 99.99th=[ 3171] bw ( KiB/s): min=106496, max=1318912, per=100.00%, avg=801613.53, stdev=277442.43, samples=60 iops : min= 26, max= 322, avg=195.68, stdev=67.74, samples=60 write: IOPS=50, BW=202MiB/s (212MB/s)(6188MiB/30589msec) slat (usec): min=1135, max=504342, avg=5405.22, stdev=23946.20 clat (msec): min=195, max=3836, avg=858.64, stdev=314.95 lat (msec): min=229, max=3840, avg=864.05, stdev=318.74 clat percentiles (msec): | 1.00th=[ 430], 5.00th=[ 558], 10.00th=[ 625], 20.00th=[ 676], | 30.00th=[ 709], 40.00th=[ 751], 50.00th=[ 793], 60.00th=[ 835], | 70.00th=[ 902], 80.00th=[ 986], 90.00th=[ 1150], 95.00th=[ 1334], | 99.00th=[ 2735], 99.50th=[ 2802], 99.90th=[ 3540], 99.95th=[ 3842], | 99.99th=[ 3842] bw ( KiB/s): min= 8192, max=368640, per=97.31%, avg=201583.39, stdev=76120.27, samples=61 iops : min= 2, max= 90, avg=49.20, stdev=18.59, samples=61 lat (msec) : 100=1.61%, 250=31.94%, 500=41.59%, 750=11.66%, 1000=8.99% lat (msec) : 2000=3.88%, >=2000=0.33% cpu : usr=6.89%, sys=1.72%, ctx=2806, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.2% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5924,1547,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=775MiB/s (812MB/s), 775MiB/s-775MiB/s (812MB/s-812MB/s), io=23.1GiB (24.8GB), run=30589-30589msec WRITE: bw=202MiB/s (212MB/s), 202MiB/s-202MiB/s (212MB/s-212MB/s), io=6188MiB (6489MB), run=30589-30589msec Disk stats (read/write): vdd: ios=23693/6152, merge=0/0, ticks=142869/3578306, in_queue=3759726, util=99.04% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981891: Fri Jan 7 14:39:57 2022 read: IOPS=165, BW=662MiB/s (694MB/s)(19.4GiB/30002msec) clat (usec): min=372, max=40098, avg=773.03, stdev=1700.88 lat (usec): min=372, max=40098, avg=773.24, stdev=1700.89 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 416], 40.00th=[ 424], 50.00th=[ 433], 60.00th=[ 449], | 70.00th=[ 494], 80.00th=[ 930], 90.00th=[ 1237], 95.00th=[ 1303], | 99.00th=[ 9896], 99.50th=[14615], 99.90th=[24511], 99.95th=[28443], | 99.99th=[40109] bw ( KiB/s): min=352256, max=1179648, per=100.00%, avg=678186.50, stdev=174314.53, samples=60 iops : min= 86, max= 288, avg=165.57, stdev=42.55, samples=60 write: IOPS=42, BW=171MiB/s (180MB/s)(5140MiB/30002msec) clat (usec): min=10922, max=91862, avg=19789.76, stdev=4690.22 lat (usec): min=11380, max=92284, avg=20347.51, stdev=4744.36 clat percentiles (usec): | 1.00th=[11600], 5.00th=[13304], 10.00th=[14353], 20.00th=[15533], | 30.00th=[16909], 40.00th=[18220], 50.00th=[20317], 60.00th=[21890], | 70.00th=[22676], 80.00th=[23725], 90.00th=[24773], 95.00th=[25297], | 99.00th=[27657], 99.50th=[31851], 99.90th=[49546], 99.95th=[91751], | 99.99th=[91751] bw ( KiB/s): min=155648, max=196608, per=99.93%, avg=175314.82, stdev=8562.32, samples=60 iops : min= 38, max= 48, avg=42.80, stdev= 2.09, samples=60 lat (usec) : 500=56.05%, 750=5.71%, 1000=2.82% lat (msec) : 2=13.79%, 10=0.34%, 20=10.78%, 50=10.51%, 100=0.02% cpu : usr=2.55%, sys=1.60%, ctx=6262, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4967,1285,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=662MiB/s (694MB/s), 662MiB/s-662MiB/s (694MB/s-694MB/s), io=19.4GiB (20.8GB), run=30002-30002msec WRITE: bw=171MiB/s (180MB/s), 171MiB/s-171MiB/s (180MB/s-180MB/s), io=5140MiB (5390MB), run=30002-30002msec Disk stats (read/write): vdd: ios=19788/5100, merge=0/0, ticks=4279/80953, in_queue=95192, util=94.94% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983091: Fri Jan 7 14:40:35 2022 read: IOPS=193, BW=772MiB/s (810MB/s)(23.1GiB/30571msec) slat (usec): min=45, max=501441, avg=3789.06, stdev=13291.68 clat (msec): min=304, max=3407, avg=694.73, stdev=195.91 lat (msec): min=304, max=3407, avg=698.52, stdev=196.40 clat percentiles (msec): | 1.00th=[ 338], 5.00th=[ 447], 10.00th=[ 510], 20.00th=[ 567], | 30.00th=[ 600], 40.00th=[ 634], 50.00th=[ 667], 60.00th=[ 701], | 70.00th=[ 743], 80.00th=[ 810], 90.00th=[ 919], 95.00th=[ 1020], | 99.00th=[ 1217], 99.50th=[ 1301], 99.90th=[ 2937], 99.95th=[ 3306], | 99.99th=[ 3406] bw ( KiB/s): min= 8192, max=1564672, per=98.01%, avg=774909.26, stdev=292146.64, samples=61 iops : min= 2, max= 382, avg=189.18, stdev=71.32, samples=61 write: IOPS=50, BW=202MiB/s (211MB/s)(6164MiB/30571msec) slat (usec): min=1086, max=449297, avg=4936.44, stdev=17858.38 clat (msec): min=565, max=3932, avg=1263.68, stdev=343.50 lat (msec): min=592, max=3933, avg=1268.62, stdev=342.91 clat percentiles (msec): | 1.00th=[ 709], 5.00th=[ 902], 10.00th=[ 978], 20.00th=[ 1053], | 30.00th=[ 1099], 40.00th=[ 1150], 50.00th=[ 1200], 60.00th=[ 1267], | 70.00th=[ 1334], 80.00th=[ 1435], 90.00th=[ 1603], 95.00th=[ 1737], | 99.00th=[ 2265], 99.50th=[ 3943], 99.90th=[ 3943], 99.95th=[ 3943], | 99.99th=[ 3943] bw ( KiB/s): min=16384, max=499712, per=99.06%, avg=204527.31, stdev=94053.93, samples=59 iops : min= 4, max= 122, avg=49.93, stdev=22.96, samples=59 lat (msec) : 500=7.61%, 750=49.13%, 1000=20.40%, 2000=22.27%, >=2000=0.60% cpu : usr=6.94%, sys=1.72%, ctx=2830, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.2% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5901,1541,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=772MiB/s (810MB/s), 772MiB/s-772MiB/s (810MB/s-810MB/s), io=23.1GiB (24.8GB), run=30571-30571msec WRITE: bw=202MiB/s (211MB/s), 202MiB/s-202MiB/s (211MB/s-211MB/s), io=6164MiB (6463MB), run=30571-30571msec Disk stats (read/write): vdd: ios=23604/6144, merge=0/0, ticks=139706/3594333, in_queue=3760677, util=98.41%