30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890677: Fri Jan 7 13:33:20 2022 read: IOPS=2415, BW=9661KiB/s (9893kB/s)(283MiB/30002msec) clat (usec): min=24, max=53719, avg=58.70, stdev=431.89 lat (usec): min=24, max=53719, avg=58.85, stdev=431.89 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 80], 99.50th=[ 93], 99.90th=[ 297], 99.95th=[11600], | 99.99th=[17171] bw ( KiB/s): min= 7992, max=13312, per=100.00%, avg=9660.78, stdev=949.35, samples=60 iops : min= 1998, max= 3328, avg=2415.18, stdev=237.33, samples=60 write: IOPS=604, BW=2418KiB/s (2476kB/s)(70.8MiB/30002msec) clat (usec): min=191, max=43664, avg=1410.25, stdev=1664.32 lat (usec): min=191, max=43665, avg=1410.53, stdev=1664.31 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 578], 10.00th=[ 807], 20.00th=[ 1004], | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2024], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[33817], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min= 2048, max= 3480, per=100.00%, avg=2417.70, stdev=230.57, samples=60 iops : min= 512, max= 870, avg=604.42, stdev=57.65, samples=60 lat (usec) : 50=54.56%, 100=25.13%, 250=0.32%, 500=0.69%, 750=0.92% lat (usec) : 1000=2.22% lat (msec) : 2=15.02%, 4=0.75%, 10=0.26%, 20=0.09%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.35%, sys=3.91%, ctx=90602, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72461,18135,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9661KiB/s (9893kB/s), 9661KiB/s-9661KiB/s (9893kB/s-9893kB/s), io=283MiB (297MB), run=30002-30002msec WRITE: bw=2418KiB/s (2476kB/s), 2418KiB/s-2418KiB/s (2476kB/s-2476kB/s), io=70.8MiB (74.3MB), run=30002-30002msec Disk stats (read/write): vdj: ios=72423/18127, merge=0/0, ticks=784/17906, in_queue=28369, util=94.09% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891552: Fri Jan 7 13:33:57 2022 read: IOPS=2430, BW=9721KiB/s (9954kB/s)(285MiB/30001msec) slat (usec): min=4, max=199, avg= 9.44, stdev= 3.47 clat (nsec): min=718, max=23740k, avg=45224.49, stdev=375235.38 lat (usec): min=23, max=23747, avg=54.82, stdev=375.27 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 34], 60.00th=[ 35], | 70.00th=[ 37], 80.00th=[ 40], 90.00th=[ 44], 95.00th=[ 47], | 99.00th=[ 59], 99.50th=[ 68], 99.90th=[ 515], 99.95th=[10945], | 99.99th=[16319] bw ( KiB/s): min= 7368, max=15512, per=99.90%, avg=9710.37, stdev=1246.17, samples=59 iops : min= 1842, max= 3878, avg=2427.59, stdev=311.54, samples=59 write: IOPS=607, BW=2430KiB/s (2488kB/s)(71.2MiB/30001msec) slat (usec): min=6, max=123, avg=10.48, stdev= 3.62 clat (usec): min=187, max=43606, avg=1408.16, stdev=1625.89 lat (usec): min=195, max=43614, avg=1418.81, stdev=1625.92 clat percentiles (usec): | 1.00th=[ 260], 5.00th=[ 611], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2040], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[34866], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min= 2000, max= 3976, per=100.00%, avg=2431.05, stdev=282.21, samples=59 iops : min= 500, max= 994, avg=607.76, stdev=70.55, samples=59 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.05%, 4=0.01%, 10=0.03%, 20=0.15%, 50=77.35% lat (usec) : 100=2.25%, 250=0.20%, 500=0.62%, 750=0.82%, 1000=2.12% lat (msec) : 2=15.21%, 4=0.80%, 10=0.25%, 20=0.08%, 50=0.02% cpu : usr=1.21%, sys=5.97%, ctx=91024, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72909,18223,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9721KiB/s (9954kB/s), 9721KiB/s-9721KiB/s (9954kB/s-9954kB/s), io=285MiB (299MB), run=30001-30001msec WRITE: bw=2430KiB/s (2488kB/s), 2430KiB/s-2430KiB/s (2488kB/s-2488kB/s), io=71.2MiB (74.6MB), run=30001-30001msec Disk stats (read/write): vdj: ios=72861/18212, merge=0/0, ticks=738/18205, in_queue=28319, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892547: Fri Jan 7 13:34:34 2022 read: IOPS=2449, BW=9796KiB/s (10.0MB/s)(287MiB/30001msec) clat (usec): min=24, max=68081, avg=59.01, stdev=462.23 lat (usec): min=24, max=68082, avg=59.14, stdev=462.23 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 64], | 99.00th=[ 80], 99.50th=[ 92], 99.90th=[ 482], 99.95th=[11076], | 99.99th=[16712] bw ( KiB/s): min= 7816, max=15062, per=99.85%, avg=9781.71, stdev=1196.08, samples=59 iops : min= 1954, max= 3765, avg=2445.41, stdev=298.98, samples=59 write: IOPS=611, BW=2448KiB/s (2506kB/s)(71.7MiB/30001msec) clat (usec): min=208, max=43467, avg=1388.80, stdev=1529.87 lat (usec): min=208, max=43467, avg=1389.07, stdev=1529.87 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 578], 10.00th=[ 807], 20.00th=[ 1012], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2008], | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[17957], 99.95th=[36963], | 99.99th=[42730] bw ( KiB/s): min= 1944, max= 3871, per=100.00%, avg=2449.97, stdev=268.17, samples=59 iops : min= 486, max= 967, avg=612.47, stdev=66.97, samples=59 lat (usec) : 50=55.73%, 100=23.99%, 250=0.30%, 500=0.72%, 750=0.89% lat (usec) : 1000=2.19% lat (msec) : 2=15.09%, 4=0.77%, 10=0.20%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.44%, sys=3.84%, ctx=91837, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73476,18357,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9796KiB/s (10.0MB/s), 9796KiB/s-9796KiB/s (10.0MB/s-10.0MB/s), io=287MiB (301MB), run=30001-30001msec WRITE: bw=2448KiB/s (2506kB/s), 2448KiB/s-2448KiB/s (2506kB/s-2506kB/s), io=71.7MiB (75.2MB), run=30001-30001msec Disk stats (read/write): vdj: ios=72907/18221, merge=0/0, ticks=798/17557, in_queue=28077, util=93.81% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892998: Fri Jan 7 13:35:11 2022 read: IOPS=2485, BW=9944KiB/s (10.2MB/s)(292MiB/30050msec) slat (usec): min=2, max=247, avg= 9.67, stdev= 4.33 clat (nsec): min=649, max=2491.7M, avg=798974.07, stdev=34215156.27 lat (usec): min=21, max=2491.7k, avg=808.80, stdev=34215.14 clat percentiles (usec): | 1.00th=[ 23], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 31], 30.00th=[ 33], 40.00th=[ 34], | 50.00th=[ 36], 60.00th=[ 38], 70.00th=[ 40], | 80.00th=[ 43], 90.00th=[ 48], 95.00th=[ 54], | 99.00th=[ 80], 99.50th=[ 101], 99.90th=[ 799], | 99.95th=[ 566232], 99.99th=[1920992] bw ( KiB/s): min= 7752, max=16688, per=100.00%, avg=9959.07, stdev=1273.71, samples=60 iops : min= 1938, max= 4172, avg=2489.77, stdev=318.43, samples=60 write: IOPS=620, BW=2481KiB/s (2541kB/s)(72.8MiB/30050msec) slat (usec): min=2, max=184, avg=10.88, stdev= 4.49 clat (usec): min=438, max=68795, avg=12851.15, stdev=5997.13 lat (usec): min=442, max=68819, avg=12862.18, stdev=5997.31 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 6456], 10.00th=[ 8455], 20.00th=[10028], | 30.00th=[11076], 40.00th=[11863], 50.00th=[12387], 60.00th=[12911], | 70.00th=[13566], 80.00th=[14222], 90.00th=[16909], 95.00th=[21103], | 99.00th=[30540], 99.50th=[62129], 99.90th=[64226], 99.95th=[64750], | 99.99th=[68682] bw ( KiB/s): min= 1904, max= 4192, per=100.00%, avg=2485.33, stdev=292.49, samples=60 iops : min= 476, max= 1048, avg=621.33, stdev=73.12, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.09%, 4=0.01%, 10=0.06%, 20=0.30%, 50=73.71% lat (usec) : 100=5.39%, 250=0.31%, 500=0.02%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.28%, 4=0.35%, 10=3.32%, 20=14.67%, 50=1.18% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.45%, sys=5.99%, ctx=91300, majf=0, minf=14 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74702,18641,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9944KiB/s (10.2MB/s), 9944KiB/s-9944KiB/s (10.2MB/s-10.2MB/s), io=292MiB (306MB), run=30050-30050msec WRITE: bw=2481KiB/s (2541kB/s), 2481KiB/s-2481KiB/s (2541kB/s-2541kB/s), io=72.8MiB (76.4MB), run=30050-30050msec Disk stats (read/write): vdj: ios=74607/18613, merge=0/0, ticks=47073/230053, in_queue=297792, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894411: Fri Jan 7 13:35:48 2022 read: IOPS=2433, BW=9735KiB/s (9969kB/s)(285MiB/30001msec) clat (usec): min=25, max=66717, avg=58.99, stdev=513.61 lat (usec): min=25, max=66717, avg=59.13, stdev=513.62 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 78], 99.50th=[ 89], 99.90th=[ 351], 99.95th=[12256], | 99.99th=[16909] bw ( KiB/s): min= 7672, max=15816, per=100.00%, avg=9742.54, stdev=1168.90, samples=59 iops : min= 1918, max= 3954, avg=2435.63, stdev=292.22, samples=59 write: IOPS=608, BW=2434KiB/s (2492kB/s)(71.3MiB/30001msec) clat (usec): min=202, max=50847, avg=1398.31, stdev=1600.40 lat (usec): min=202, max=50848, avg=1398.58, stdev=1600.39 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 578], 10.00th=[ 799], 20.00th=[ 1012], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[17957], 99.95th=[39584], | 99.99th=[50070] bw ( KiB/s): min= 2032, max= 4072, per=100.00%, avg=2438.95, stdev=281.71, samples=59 iops : min= 508, max= 1018, avg=609.73, stdev=70.42, samples=59 lat (usec) : 50=57.96%, 100=21.80%, 250=0.24%, 500=0.71%, 750=0.90% lat (usec) : 1000=2.20% lat (msec) : 2=15.06%, 4=0.78%, 10=0.24%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.37%, sys=3.76%, ctx=91268, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73014,18252,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9735KiB/s (9969kB/s), 9735KiB/s-9735KiB/s (9969kB/s-9969kB/s), io=285MiB (299MB), run=30001-30001msec WRITE: bw=2434KiB/s (2492kB/s), 2434KiB/s-2434KiB/s (2492kB/s-2492kB/s), io=71.3MiB (74.8MB), run=30001-30001msec Disk stats (read/write): vdj: ios=72950/18239, merge=0/0, ticks=881/17812, in_queue=28343, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894983: Fri Jan 7 13:36:25 2022 read: IOPS=2482, BW=9929KiB/s (10.2MB/s)(292MiB/30097msec) slat (usec): min=2, max=467, avg= 9.66, stdev= 5.90 clat (nsec): min=694, max=2559.9M, avg=988735.96, stdev=40357539.36 lat (usec): min=21, max=2559.9k, avg=998.55, stdev=40357.55 clat percentiles (usec): | 1.00th=[ 23], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 33], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 42], | 80.00th=[ 46], 90.00th=[ 57], 95.00th=[ 75], | 99.00th=[ 223], 99.50th=[ 445], 99.90th=[ 1958], | 99.95th=[ 750781], 99.99th=[2499806] bw ( KiB/s): min= 7720, max=16600, per=100.00%, avg=9960.43, stdev=1227.24, samples=60 iops : min= 1930, max= 4150, avg=2490.10, stdev=306.81, samples=60 write: IOPS=619, BW=2478KiB/s (2537kB/s)(72.8MiB/30097msec) slat (usec): min=2, max=198, avg=10.96, stdev= 6.20 clat (msec): min=5, max=2403, avg=76.64, stdev=66.07 lat (msec): min=6, max=2403, avg=76.66, stdev=66.07 clat percentiles (msec): | 1.00th=[ 10], 5.00th=[ 54], 10.00th=[ 57], 20.00th=[ 62], | 30.00th=[ 65], 40.00th=[ 67], 50.00th=[ 69], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 91], 95.00th=[ 115], | 99.00th=[ 257], 99.50th=[ 481], 99.90th=[ 1020], 99.95th=[ 1234], | 99.99th=[ 2400] bw ( KiB/s): min= 2168, max= 3864, per=100.00%, avg=2479.67, stdev=243.39, samples=60 iops : min= 542, max= 966, avg=619.92, stdev=60.85, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.12%, 4=0.01%, 10=0.06%, 20=0.34%, 50=68.05% lat (usec) : 100=9.36%, 250=1.29%, 500=0.38%, 750=0.14%, 1000=0.05% lat (msec) : 2=0.07%, 4=0.01%, 10=0.21%, 20=0.07%, 50=0.39% lat (msec) : 100=17.78%, 250=1.34%, 500=0.11%, 750=0.06%, 1000=0.02% lat (msec) : 2000=0.04%, >=2000=0.02% cpu : usr=1.41%, sys=6.00%, ctx=89232, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=74706,18642,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9929KiB/s (10.2MB/s), 9929KiB/s-9929KiB/s (10.2MB/s-10.2MB/s), io=292MiB (306MB), run=30097-30097msec WRITE: bw=2478KiB/s (2537kB/s), 2478KiB/s-2478KiB/s (2537kB/s-2537kB/s), io=72.8MiB (76.4MB), run=30097-30097msec Disk stats (read/write): vdj: ios=74704/18642, merge=0/0, ticks=67963/1416914, in_queue=1499328, util=99.04% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895825: Fri Jan 7 13:37:02 2022 read: IOPS=2446, BW=9786KiB/s (10.0MB/s)(287MiB/30001msec) clat (usec): min=25, max=22722, avg=58.23, stdev=403.61 lat (usec): min=25, max=22722, avg=58.37, stdev=403.61 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 79], 99.50th=[ 90], 99.90th=[ 510], 99.95th=[12387], | 99.99th=[17171] bw ( KiB/s): min= 7456, max=15376, per=100.00%, avg=9786.18, stdev=1143.41, samples=60 iops : min= 1864, max= 3844, avg=2446.53, stdev=285.86, samples=60 write: IOPS=611, BW=2445KiB/s (2504kB/s)(71.6MiB/30001msec) clat (usec): min=198, max=43639, avg=1393.77, stdev=1590.82 lat (usec): min=198, max=43639, avg=1394.05, stdev=1590.82 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 586], 10.00th=[ 799], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[17957], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 1952, max= 3960, per=100.00%, avg=2445.00, stdev=277.31, samples=60 iops : min= 488, max= 990, avg=611.25, stdev=69.33, samples=60 lat (usec) : 50=57.75%, 100=21.99%, 250=0.26%, 500=0.72%, 750=0.96% lat (usec) : 1000=2.26% lat (msec) : 2=14.90%, 4=0.82%, 10=0.23%, 20=0.09%, 50=0.03% cpu : usr=1.43%, sys=3.76%, ctx=91739, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73394,18338,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9786KiB/s (10.0MB/s), 9786KiB/s-9786KiB/s (10.0MB/s-10.0MB/s), io=287MiB (301MB), run=30001-30001msec WRITE: bw=2445KiB/s (2504kB/s), 2445KiB/s-2445KiB/s (2504kB/s-2504kB/s), io=71.6MiB (75.1MB), run=30001-30001msec Disk stats (read/write): vdj: ios=73201/18294, merge=0/0, ticks=805/17763, in_queue=28285, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896949: Fri Jan 7 13:37:39 2022 read: IOPS=2470, BW=9881KiB/s (10.1MB/s)(291MiB/30177msec) slat (usec): min=2, max=383, avg= 9.49, stdev= 5.20 clat (nsec): min=774, max=2515.5M, avg=994972.62, stdev=40778851.07 lat (usec): min=23, max=2515.5k, avg=1004.61, stdev=40778.85 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 37], 60.00th=[ 40], 70.00th=[ 43], | 80.00th=[ 48], 90.00th=[ 65], 95.00th=[ 119], | 99.00th=[ 775], 99.50th=[ 1401], 99.90th=[ 2409], | 99.95th=[ 608175], 99.99th=[2499806] bw ( KiB/s): min= 7664, max=18224, per=100.00%, avg=9938.13, stdev=1423.39, samples=60 iops : min= 1916, max= 4556, avg=2484.53, stdev=355.85, samples=60 write: IOPS=616, BW=2464KiB/s (2524kB/s)(72.6MiB/30177msec) slat (usec): min=2, max=381, avg=10.66, stdev= 5.46 clat (msec): min=8, max=2925, avg=158.19, stdev=131.46 lat (msec): min=8, max=2925, avg=158.20, stdev=131.46 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 108], 10.00th=[ 116], 20.00th=[ 130], | 30.00th=[ 136], 40.00th=[ 140], 50.00th=[ 144], 60.00th=[ 150], | 70.00th=[ 157], 80.00th=[ 165], 90.00th=[ 190], 95.00th=[ 215], | 99.00th=[ 422], 99.50th=[ 894], 99.90th=[ 2937], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 1984, max= 3928, per=100.00%, avg=2466.80, stdev=269.70, samples=60 iops : min= 496, max= 982, avg=616.70, stdev=67.42, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.10%, 50=66.02% lat (usec) : 100=8.97%, 250=2.96%, 500=0.75%, 750=0.34%, 1000=0.22% lat (msec) : 2=0.43%, 4=0.10%, 10=0.05%, 20=0.22%, 50=0.10% lat (msec) : 100=0.20%, 250=18.80%, 500=0.45%, 750=0.06%, 1000=0.05% lat (msec) : 2000=0.06%, >=2000=0.05% cpu : usr=1.41%, sys=5.77%, ctx=86135, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=74544,18592,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9881KiB/s (10.1MB/s), 9881KiB/s-9881KiB/s (10.1MB/s-10.1MB/s), io=291MiB (305MB), run=30177-30177msec WRITE: bw=2464KiB/s (2524kB/s), 2464KiB/s-2464KiB/s (2524kB/s-2524kB/s), io=72.6MiB (76.2MB), run=30177-30177msec Disk stats (read/write): vdj: ios=74473/18482, merge=0/0, ticks=59797/2903492, in_queue=2993881, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897512: Fri Jan 7 13:38:16 2022 read: IOPS=2403, BW=9612KiB/s (9843kB/s)(282MiB/30007msec) clat (usec): min=24, max=58888, avg=60.77, stdev=438.57 lat (usec): min=24, max=58888, avg=60.91, stdev=438.57 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 84], 99.50th=[ 99], 99.90th=[ 498], 99.95th=[11600], | 99.99th=[16909] bw ( KiB/s): min= 7696, max=15520, per=100.00%, avg=9613.90, stdev=1261.06, samples=60 iops : min= 1924, max= 3880, avg=2403.47, stdev=315.26, samples=60 write: IOPS=601, BW=2407KiB/s (2465kB/s)(70.5MiB/30007msec) clat (usec): min=204, max=43839, avg=1409.20, stdev=1669.63 lat (usec): min=204, max=43840, avg=1409.51, stdev=1669.62 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 545], 10.00th=[ 775], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1369], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[28443], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 1880, max= 3984, per=100.00%, avg=2407.90, stdev=290.39, samples=60 iops : min= 470, max= 996, avg=601.97, stdev=72.59, samples=60 lat (usec) : 50=48.99%, 100=30.60%, 250=0.36%, 500=0.79%, 750=1.01% lat (usec) : 1000=2.41% lat (msec) : 2=14.70%, 4=0.73%, 10=0.27%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.39%, sys=4.03%, ctx=90179, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72108,18060,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9612KiB/s (9843kB/s), 9612KiB/s-9612KiB/s (9843kB/s-9843kB/s), io=282MiB (295MB), run=30007-30007msec WRITE: bw=2407KiB/s (2465kB/s), 2407KiB/s-2407KiB/s (2465kB/s-2465kB/s), io=70.5MiB (73.0MB), run=30007-30007msec Disk stats (read/write): vdj: ios=71943/18022, merge=0/0, ticks=784/17628, in_queue=28191, util=93.67% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898368: Fri Jan 7 13:38:53 2022 read: IOPS=2489, BW=9959KiB/s (10.2MB/s)(294MiB/30231msec) slat (usec): min=2, max=235804, avg=316.31, stdev=1917.95 clat (usec): min=557, max=2725.7k, avg=25896.89, stdev=42697.84 lat (usec): min=561, max=2725.7k, avg=26213.33, stdev=42749.20 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 6587], 10.00th=[ 14091], | 20.00th=[ 17433], 30.00th=[ 19792], 40.00th=[ 21627], | 50.00th=[ 23200], 60.00th=[ 25035], 70.00th=[ 27132], | 80.00th=[ 30016], 90.00th=[ 34866], 95.00th=[ 39584], | 99.00th=[ 79168], 99.50th=[ 183501], 99.90th=[ 265290], | 99.95th=[ 884999], 99.99th=[2332034] bw ( KiB/s): min= 7432, max=18557, per=100.00%, avg=10027.10, stdev=1520.79, samples=60 iops : min= 1858, max= 4639, avg=2506.77, stdev=380.17, samples=60 write: IOPS=621, BW=2485KiB/s (2544kB/s)(73.4MiB/30231msec) slat (usec): min=2, max=190277, avg=319.70, stdev=2347.81 clat (msec): min=4, max=2466, avg=216.38, stdev=134.29 lat (msec): min=4, max=2466, avg=216.70, stdev=134.28 clat percentiles (msec): | 1.00th=[ 27], 5.00th=[ 146], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 194], 50.00th=[ 201], 60.00th=[ 207], | 70.00th=[ 215], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 284], | 99.00th=[ 709], 99.50th=[ 1099], 99.90th=[ 2165], 99.95th=[ 2165], | 99.99th=[ 2467] bw ( KiB/s): min= 2056, max= 3823, per=100.00%, avg=2487.13, stdev=264.63, samples=60 iops : min= 514, max= 955, avg=621.77, stdev=66.09, samples=60 lat (usec) : 750=0.33%, 1000=0.58% lat (msec) : 2=1.46%, 4=1.24%, 10=0.90%, 20=20.41%, 50=53.23% lat (msec) : 100=1.89%, 250=17.41%, 500=2.19%, 750=0.12%, 1000=0.08% lat (msec) : 2000=0.10%, >=2000=0.05% cpu : usr=0.69%, sys=1.96%, ctx=10267, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75269,18779,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9959KiB/s (10.2MB/s), 9959KiB/s-9959KiB/s (10.2MB/s-10.2MB/s), io=294MiB (308MB), run=30231-30231msec WRITE: bw=2485KiB/s (2544kB/s), 2485KiB/s-2485KiB/s (2544kB/s-2544kB/s), io=73.4MiB (76.9MB), run=30231-30231msec Disk stats (read/write): vdj: ios=75268/18736, merge=0/0, ticks=75269/3574663, in_queue=3676026, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898991: Fri Jan 7 13:39:30 2022 read: IOPS=2390, BW=37.4MiB/s (39.2MB/s)(1121MiB/30001msec) clat (usec): min=27, max=24490, avg=62.86, stdev=412.21 lat (usec): min=27, max=24490, avg=63.00, stdev=412.21 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 71], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 519], 99.95th=[11731], | 99.99th=[17695] bw ( KiB/s): min=30048, max=60608, per=100.00%, avg=38257.36, stdev=4574.29, samples=59 iops : min= 1878, max= 3788, avg=2391.08, stdev=285.89, samples=59 write: IOPS=598, BW=9578KiB/s (9808kB/s)(281MiB/30001msec) clat (usec): min=236, max=76434, avg=1409.45, stdev=1654.96 lat (usec): min=240, max=76435, avg=1410.92, stdev=1654.94 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 578], 10.00th=[ 807], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 2024], | 99.00th=[ 8979], 99.50th=[ 9634], 99.90th=[34866], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min= 7520, max=15648, per=99.95%, avg=9571.80, stdev=1091.82, samples=59 iops : min= 470, max= 978, avg=598.24, stdev=68.24, samples=59 lat (usec) : 50=42.97%, 100=36.55%, 250=0.37%, 500=0.74%, 750=0.98% lat (usec) : 1000=1.89% lat (msec) : 2=15.35%, 4=0.84%, 10=0.19%, 20=0.08%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.38%, sys=3.99%, ctx=89681, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71718,17959,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.4MiB/s (39.2MB/s), 37.4MiB/s-37.4MiB/s (39.2MB/s-39.2MB/s), io=1121MiB (1175MB), run=30001-30001msec WRITE: bw=9578KiB/s (9808kB/s), 9578KiB/s-9578KiB/s (9808kB/s-9808kB/s), io=281MiB (294MB), run=30001-30001msec Disk stats (read/write): vdj: ios=71661/17941, merge=0/0, ticks=796/17632, in_queue=28296, util=94.07% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899809: Fri Jan 7 13:40:07 2022 read: IOPS=2363, BW=36.9MiB/s (38.7MB/s)(1108MiB/30001msec) slat (usec): min=5, max=203, avg=10.22, stdev= 4.14 clat (nsec): min=664, max=25260k, avg=48045.67, stdev=395003.94 lat (usec): min=24, max=25268, avg=58.41, stdev=395.06 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 33], 40.00th=[ 35], 50.00th=[ 36], 60.00th=[ 38], | 70.00th=[ 40], 80.00th=[ 42], 90.00th=[ 46], 95.00th=[ 51], | 99.00th=[ 65], 99.50th=[ 76], 99.90th=[ 379], 99.95th=[11731], | 99.99th=[17171] bw ( KiB/s): min=29280, max=56992, per=100.00%, avg=37821.15, stdev=4490.32, samples=60 iops : min= 1830, max= 3562, avg=2363.78, stdev=280.65, samples=60 write: IOPS=591, BW=9457KiB/s (9684kB/s)(277MiB/30001msec) slat (usec): min=7, max=195, avg=12.45, stdev= 4.25 clat (usec): min=216, max=46100, avg=1438.02, stdev=1643.81 lat (usec): min=226, max=46111, avg=1450.62, stdev=1643.83 clat percentiles (usec): | 1.00th=[ 293], 5.00th=[ 603], 10.00th=[ 848], 20.00th=[ 1057], | 30.00th=[ 1156], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1434], 80.00th=[ 1549], 90.00th=[ 1795], 95.00th=[ 2057], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[36439], 99.95th=[39584], | 99.99th=[44303] bw ( KiB/s): min= 8032, max=14560, per=100.00%, avg=9457.32, stdev=938.95, samples=60 iops : min= 502, max= 910, avg=591.07, stdev=58.67, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.10%, 4=0.01%, 10=0.02%, 20=0.14%, 50=75.44% lat (usec) : 100=4.07%, 250=0.13%, 500=0.72%, 750=0.77%, 1000=1.82% lat (msec) : 2=15.53%, 4=0.87%, 10=0.22%, 20=0.10%, 50=0.02% cpu : usr=1.23%, sys=6.13%, ctx=88475, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70913,17733,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=1108MiB (1162MB), run=30001-30001msec WRITE: bw=9457KiB/s (9684kB/s), 9457KiB/s-9457KiB/s (9684kB/s-9684kB/s), io=277MiB (291MB), run=30001-30001msec Disk stats (read/write): vdj: ios=70913/17732, merge=0/0, ticks=753/18132, in_queue=28316, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901197: Fri Jan 7 13:40:44 2022 read: IOPS=2381, BW=37.2MiB/s (39.0MB/s)(1116MiB/30001msec) clat (usec): min=26, max=60142, avg=61.79, stdev=447.60 lat (usec): min=26, max=60142, avg=61.94, stdev=447.60 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 523], 99.95th=[11338], | 99.99th=[16909] bw ( KiB/s): min=30784, max=57312, per=100.00%, avg=38098.13, stdev=4075.25, samples=60 iops : min= 1924, max= 3582, avg=2381.13, stdev=254.70, samples=60 write: IOPS=595, BW=9534KiB/s (9762kB/s)(279MiB/30001msec) clat (usec): min=231, max=50364, avg=1421.40, stdev=1571.61 lat (usec): min=232, max=50365, avg=1422.82, stdev=1571.60 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 586], 10.00th=[ 816], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1778], 95.00th=[ 2073], | 99.00th=[ 9110], 99.50th=[ 9634], 99.90th=[17433], 99.95th=[40633], | 99.99th=[46924] bw ( KiB/s): min= 7776, max=14752, per=100.00%, avg=9533.33, stdev=969.47, samples=60 iops : min= 486, max= 922, avg=595.83, stdev=60.59, samples=60 lat (usec) : 50=47.05%, 100=32.60%, 250=0.24%, 500=0.75%, 750=0.90% lat (usec) : 1000=1.85% lat (msec) : 2=15.38%, 4=0.90%, 10=0.22%, 20=0.08%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.47%, sys=3.73%, ctx=89312, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71434,17876,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1116MiB (1170MB), run=30001-30001msec WRITE: bw=9534KiB/s (9762kB/s), 9534KiB/s-9534KiB/s (9762kB/s-9762kB/s), io=279MiB (293MB), run=30001-30001msec Disk stats (read/write): vdj: ios=70950/17746, merge=0/0, ticks=779/17622, in_queue=28088, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901860: Fri Jan 7 13:41:21 2022 read: IOPS=2427, BW=37.9MiB/s (39.8MB/s)(1140MiB/30042msec) slat (usec): min=2, max=211, avg=10.94, stdev= 4.35 clat (nsec): min=713, max=2491.5M, avg=806554.06, stdev=35461898.44 lat (usec): min=24, max=2491.5k, avg=817.73, stdev=35461.90 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 44], | 80.00th=[ 48], 90.00th=[ 53], 95.00th=[ 60], | 99.00th=[ 90], 99.50th=[ 121], 99.90th=[ 717], | 99.95th=[ 367002], 99.99th=[2298479] bw ( KiB/s): min=29984, max=67904, per=100.00%, avg=38893.48, stdev=5128.98, samples=60 iops : min= 1874, max= 4244, avg=2430.83, stdev=320.56, samples=60 write: IOPS=606, BW=9710KiB/s (9943kB/s)(285MiB/30042msec) slat (usec): min=4, max=136, avg=13.85, stdev= 4.56 clat (usec): min=467, max=76353, avg=13180.70, stdev=6030.85 lat (usec): min=476, max=76370, avg=13194.71, stdev=6031.11 clat percentiles (usec): | 1.00th=[ 1582], 5.00th=[ 7046], 10.00th=[ 8291], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11994], 50.00th=[12911], 60.00th=[13566], | 70.00th=[14222], 80.00th=[15008], 90.00th=[16909], 95.00th=[21627], | 99.00th=[30278], 99.50th=[62129], 99.90th=[66323], 99.95th=[69731], | 99.99th=[74974] bw ( KiB/s): min= 7232, max=17056, per=100.00%, avg=9721.90, stdev=1199.26, samples=60 iops : min= 452, max= 1066, avg=607.62, stdev=74.95, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.09%, 50=68.31% lat (usec) : 100=10.88%, 250=0.47%, 500=0.04%, 750=0.01%, 1000=0.03% lat (msec) : 2=0.28%, 4=0.08%, 10=4.11%, 20=14.09%, 50=1.26% lat (msec) : 100=0.16%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.53%, sys=6.41%, ctx=89560, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72929,18232,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.9MiB/s (39.8MB/s), 37.9MiB/s-37.9MiB/s (39.8MB/s-39.8MB/s), io=1140MiB (1195MB), run=30042-30042msec WRITE: bw=9710KiB/s (9943kB/s), 9710KiB/s-9710KiB/s (9943kB/s-9943kB/s), io=285MiB (299MB), run=30042-30042msec Disk stats (read/write): vdj: ios=72928/18232, merge=0/0, ticks=54509/230975, in_queue=298185, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902765: Fri Jan 7 13:41:58 2022 read: IOPS=2365, BW=36.0MiB/s (38.8MB/s)(1109MiB/30007msec) clat (usec): min=24, max=57978, avg=61.21, stdev=441.46 lat (usec): min=24, max=57978, avg=61.36, stdev=441.46 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 404], 99.95th=[11731], | 99.99th=[16450] bw ( KiB/s): min=27328, max=60128, per=100.00%, avg=37864.88, stdev=4900.82, samples=60 iops : min= 1708, max= 3758, avg=2366.55, stdev=306.30, samples=60 write: IOPS=591, BW=9468KiB/s (9695kB/s)(277MiB/30007msec) clat (usec): min=213, max=86703, avg=1434.89, stdev=1692.18 lat (usec): min=215, max=86704, avg=1436.32, stdev=1692.17 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 603], 10.00th=[ 840], 20.00th=[ 1057], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2057], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[40633], | 99.99th=[45351] bw ( KiB/s): min= 6624, max=15456, per=100.00%, avg=9470.22, stdev=1148.75, samples=60 iops : min= 414, max= 966, avg=591.88, stdev=71.79, samples=60 lat (usec) : 50=47.91%, 100=31.74%, 250=0.25%, 500=0.71%, 750=0.87% lat (usec) : 1000=1.77% lat (msec) : 2=15.49%, 4=0.91%, 10=0.21%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.37%, sys=3.91%, ctx=88759, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70995,17756,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=36.0MiB/s (38.8MB/s), 36.0MiB/s-36.0MiB/s (38.8MB/s-38.8MB/s), io=1109MiB (1163MB), run=30007-30007msec WRITE: bw=9468KiB/s (9695kB/s), 9468KiB/s-9468KiB/s (9695kB/s-9695kB/s), io=277MiB (291MB), run=30007-30007msec Disk stats (read/write): vdj: ios=70491/17638, merge=0/0, ticks=782/17798, in_queue=28132, util=93.97% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903787: Fri Jan 7 13:42:35 2022 read: IOPS=2406, BW=37.6MiB/s (39.4MB/s)(1132MiB/30105msec) slat (usec): min=2, max=225, avg= 9.76, stdev= 4.27 clat (nsec): min=728, max=2517.0M, avg=962677.05, stdev=40041608.43 lat (usec): min=23, max=2517.0k, avg=972.58, stdev=40041.61 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 43], | 80.00th=[ 47], 90.00th=[ 57], 95.00th=[ 72], | 99.00th=[ 241], 99.50th=[ 478], 99.90th=[ 2180], | 99.95th=[ 708838], 99.99th=[2499806] bw ( KiB/s): min=28512, max=68160, per=100.00%, avg=38649.48, stdev=5241.85, samples=60 iops : min= 1782, max= 4260, avg=2415.57, stdev=327.60, samples=60 write: IOPS=602, BW=9637KiB/s (9868kB/s)(283MiB/30105msec) slat (usec): min=4, max=122, avg=12.89, stdev= 4.64 clat (msec): min=8, max=2525, avg=79.09, stdev=69.17 lat (msec): min=8, max=2525, avg=79.10, stdev=69.17 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 56], 10.00th=[ 60], 20.00th=[ 64], | 30.00th=[ 66], 40.00th=[ 68], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 93], 95.00th=[ 118], | 99.00th=[ 249], 99.50th=[ 527], 99.90th=[ 1028], 99.95th=[ 1267], | 99.99th=[ 2467] bw ( KiB/s): min= 7200, max=15872, per=100.00%, avg=9649.65, stdev=1112.38, samples=60 iops : min= 450, max= 992, avg=603.08, stdev=69.51, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.13%, 50=67.60% lat (usec) : 100=10.32%, 250=1.02%, 500=0.42%, 750=0.15%, 1000=0.07% lat (msec) : 2=0.05%, 4=0.03%, 10=0.09%, 20=0.18%, 50=0.28% lat (msec) : 100=17.90%, 250=1.38%, 500=0.09%, 750=0.07%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.02% cpu : usr=1.39%, sys=5.75%, ctx=87741, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=72461,18133,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.6MiB/s (39.4MB/s), 37.6MiB/s-37.6MiB/s (39.4MB/s-39.4MB/s), io=1132MiB (1187MB), run=30105-30105msec WRITE: bw=9637KiB/s (9868kB/s), 9637KiB/s-9637KiB/s (9868kB/s-9868kB/s), io=283MiB (297MB), run=30105-30105msec Disk stats (read/write): vdj: ios=72455/18098, merge=0/0, ticks=59926/1419687, in_queue=1498907, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904250: Fri Jan 7 13:43:12 2022 read: IOPS=2372, BW=37.1MiB/s (38.9MB/s)(1112MiB/30001msec) clat (usec): min=24, max=25834, avg=61.23, stdev=401.71 lat (usec): min=24, max=25834, avg=61.37, stdev=401.71 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 297], 99.95th=[11731], | 99.99th=[16909] bw ( KiB/s): min=31616, max=61504, per=100.00%, avg=38105.92, stdev=4474.52, samples=59 iops : min= 1976, max= 3844, avg=2381.58, stdev=279.68, samples=59 write: IOPS=593, BW=9500KiB/s (9728kB/s)(278MiB/30001msec) clat (usec): min=238, max=42611, avg=1429.40, stdev=1615.17 lat (usec): min=239, max=42612, avg=1430.85, stdev=1615.16 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 586], 10.00th=[ 832], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1434], 80.00th=[ 1549], 90.00th=[ 1778], 95.00th=[ 2057], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[36963], 99.95th=[41157], | 99.99th=[42730] bw ( KiB/s): min= 8064, max=15840, per=100.00%, avg=9537.05, stdev=1095.77, samples=59 iops : min= 504, max= 990, avg=596.02, stdev=68.53, samples=59 lat (usec) : 50=46.60%, 100=33.04%, 250=0.26%, 500=0.76%, 750=0.85% lat (usec) : 1000=1.87% lat (msec) : 2=15.40%, 4=0.88%, 10=0.21%, 20=0.10%, 50=0.03% cpu : usr=1.44%, sys=3.90%, ctx=89007, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71187,17814,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1112MiB (1166MB), run=30001-30001msec WRITE: bw=9500KiB/s (9728kB/s), 9500KiB/s-9500KiB/s (9728kB/s-9728kB/s), io=278MiB (292MB), run=30001-30001msec Disk stats (read/write): vdj: ios=71187/17814, merge=0/0, ticks=772/17815, in_queue=28309, util=94.03% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905223: Fri Jan 7 13:43:49 2022 read: IOPS=2412, BW=37.7MiB/s (39.5MB/s)(1140MiB/30236msec) slat (usec): min=2, max=192, avg= 9.82, stdev= 4.70 clat (nsec): min=683, max=2505.1M, avg=977330.94, stdev=38755895.83 lat (usec): min=23, max=2505.1k, avg=987.28, stdev=38755.88 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 49], 90.00th=[ 64], 95.00th=[ 120], | 99.00th=[ 865], 99.50th=[ 1565], 99.90th=[ 4555], | 99.95th=[ 650118], 99.99th=[2264925] bw ( KiB/s): min=28979, max=78112, per=100.00%, avg=38977.12, stdev=6361.27, samples=60 iops : min= 1811, max= 4882, avg=2435.78, stdev=397.62, samples=60 write: IOPS=603, BW=9649KiB/s (9881kB/s)(285MiB/30236msec) slat (usec): min=3, max=314, avg=12.91, stdev= 6.88 clat (msec): min=4, max=2490, avg=161.81, stdev=84.78 lat (msec): min=4, max=2490, avg=161.82, stdev=84.78 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 120], 10.00th=[ 130], 20.00th=[ 140], | 30.00th=[ 144], 40.00th=[ 148], 50.00th=[ 153], 60.00th=[ 157], | 70.00th=[ 161], 80.00th=[ 171], 90.00th=[ 197], 95.00th=[ 218], | 99.00th=[ 418], 99.50th=[ 776], 99.90th=[ 1284], 99.95th=[ 1552], | 99.99th=[ 2140] bw ( KiB/s): min= 7776, max=17056, per=100.00%, avg=9696.07, stdev=1250.77, samples=60 iops : min= 486, max= 1066, avg=605.78, stdev=78.23, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.03%, 20=0.12%, 50=65.41% lat (usec) : 100=9.51%, 250=2.80%, 500=0.80%, 750=0.36%, 1000=0.22% lat (msec) : 2=0.43%, 4=0.15%, 10=0.04%, 20=0.16%, 50=0.22% lat (msec) : 100=0.27%, 250=18.74%, 500=0.47%, 750=0.07%, 1000=0.06% lat (msec) : 2000=0.07%, >=2000=0.01% cpu : usr=1.44%, sys=5.67%, ctx=85778, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=72934,18235,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.7MiB/s (39.5MB/s), 37.7MiB/s-37.7MiB/s (39.5MB/s-39.5MB/s), io=1140MiB (1195MB), run=30236-30236msec WRITE: bw=9649KiB/s (9881kB/s), 9649KiB/s-9649KiB/s (9881kB/s-9881kB/s), io=285MiB (299MB), run=30236-30236msec Disk stats (read/write): vdj: ios=72926/18142, merge=0/0, ticks=57302/2911554, in_queue=2997376, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905800: Fri Jan 7 13:44:26 2022 read: IOPS=2381, BW=37.2MiB/s (39.0MB/s)(1116MiB/30001msec) clat (usec): min=26, max=64836, avg=61.20, stdev=459.22 lat (usec): min=26, max=64836, avg=61.34, stdev=459.22 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 82], 99.50th=[ 96], 99.90th=[ 461], 99.95th=[11600], | 99.99th=[17433] bw ( KiB/s): min=31008, max=61824, per=100.00%, avg=38125.61, stdev=4592.09, samples=59 iops : min= 1938, max= 3864, avg=2382.85, stdev=287.01, samples=59 write: IOPS=595, BW=9532KiB/s (9761kB/s)(279MiB/30001msec) clat (usec): min=233, max=44192, avg=1424.11, stdev=1568.27 lat (usec): min=235, max=44193, avg=1425.52, stdev=1568.26 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 594], 10.00th=[ 840], 20.00th=[ 1045], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2073], | 99.00th=[ 9110], 99.50th=[ 9765], 99.90th=[29754], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 7232, max=15840, per=100.00%, avg=9542.75, stdev=1169.46, samples=59 iops : min= 452, max= 990, avg=596.41, stdev=73.12, samples=59 lat (usec) : 50=49.01%, 100=30.66%, 250=0.23%, 500=0.74%, 750=0.81% lat (usec) : 1000=1.86% lat (msec) : 2=15.42%, 4=0.96%, 10=0.18%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.36%, sys=3.92%, ctx=89319, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71433,17874,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1116MiB (1170MB), run=30001-30001msec WRITE: bw=9532KiB/s (9761kB/s), 9532KiB/s-9532KiB/s (9761kB/s-9761kB/s), io=279MiB (293MB), run=30001-30001msec Disk stats (read/write): vdj: ios=70998/17760, merge=0/0, ticks=797/17642, in_queue=28103, util=94.07% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906621: Fri Jan 7 13:45:03 2022 read: IOPS=2396, BW=37.5MiB/s (39.3MB/s)(1132MiB/30214msec) slat (usec): min=2, max=197274, avg=331.43, stdev=1949.35 clat (usec): min=598, max=2551.7k, avg=26897.73, stdev=44117.40 lat (usec): min=602, max=2551.7k, avg=27229.29, stdev=44165.85 clat percentiles (usec): | 1.00th=[ 1319], 5.00th=[ 7635], 10.00th=[ 14877], | 20.00th=[ 18482], 30.00th=[ 20579], 40.00th=[ 22414], | 50.00th=[ 24249], 60.00th=[ 26346], 70.00th=[ 28443], | 80.00th=[ 31327], 90.00th=[ 35914], 95.00th=[ 41157], | 99.00th=[ 81265], 99.50th=[ 175113], 99.90th=[ 235930], | 99.95th=[ 509608], 99.99th=[2533360] bw ( KiB/s): min=30528, max=83616, per=100.00%, avg=38588.93, stdev=6678.99, samples=60 iops : min= 1908, max= 5226, avg=2411.80, stdev=417.44, samples=60 write: IOPS=599, BW=9599KiB/s (9829kB/s)(283MiB/30214msec) slat (usec): min=3, max=213251, avg=321.76, stdev=2156.32 clat (msec): min=12, max=2461, avg=224.20, stdev=164.30 lat (msec): min=12, max=2461, avg=224.52, stdev=164.28 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 129], 10.00th=[ 150], 20.00th=[ 180], | 30.00th=[ 194], 40.00th=[ 203], 50.00th=[ 209], 60.00th=[ 215], | 70.00th=[ 222], 80.00th=[ 236], 90.00th=[ 262], 95.00th=[ 296], | 99.00th=[ 936], 99.50th=[ 1720], 99.90th=[ 2198], 99.95th=[ 2265], | 99.99th=[ 2467] bw ( KiB/s): min= 7648, max=17248, per=99.99%, avg=9597.35, stdev=1202.57, samples=60 iops : min= 478, max= 1078, avg=599.83, stdev=75.16, samples=60 lat (usec) : 750=0.07%, 1000=0.21% lat (msec) : 2=1.55%, 4=1.66%, 10=0.82%, 20=17.32%, 50=56.35% lat (msec) : 100=2.00%, 250=17.16%, 500=2.45%, 750=0.11%, 1000=0.08% lat (msec) : 2000=0.14%, >=2000=0.08% cpu : usr=0.79%, sys=2.00%, ctx=10134, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=72418,18126,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.5MiB/s (39.3MB/s), 37.5MiB/s-37.5MiB/s (39.3MB/s-39.3MB/s), io=1132MiB (1186MB), run=30214-30214msec WRITE: bw=9599KiB/s (9829kB/s), 9599KiB/s-9599KiB/s (9829kB/s-9829kB/s), io=283MiB (297MB), run=30214-30214msec Disk stats (read/write): vdj: ios=72410/18010, merge=0/0, ticks=65255/3558572, in_queue=3662570, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908080: Fri Jan 7 13:45:40 2022 read: IOPS=2284, BW=71.4MiB/s (74.9MB/s)(2142MiB/30001msec) clat (usec): min=31, max=23793, avg=66.58, stdev=429.74 lat (usec): min=31, max=23793, avg=66.72, stdev=429.74 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 73], | 99.00th=[ 89], 99.50th=[ 99], 99.90th=[ 351], 99.95th=[13042], | 99.99th=[17695] bw ( KiB/s): min=58112, max=102464, per=100.00%, avg=73106.13, stdev=7903.44, samples=60 iops : min= 1816, max= 3202, avg=2284.57, stdev=246.98, samples=60 write: IOPS=571, BW=17.9MiB/s (18.7MB/s)(536MiB/30001msec) clat (usec): min=268, max=47526, avg=1471.76, stdev=1592.44 lat (usec): min=271, max=47528, avg=1474.72, stdev=1592.43 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 611], 10.00th=[ 857], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1844], 95.00th=[ 2114], | 99.00th=[ 9110], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min=15104, max=26624, per=100.00%, avg=18295.47, stdev=1721.11, samples=60 iops : min= 472, max= 832, avg=571.73, stdev=53.78, samples=60 lat (usec) : 50=33.59%, 100=46.03%, 250=0.28%, 500=0.62%, 750=0.92% lat (usec) : 1000=1.56% lat (msec) : 2=15.61%, 4=1.07%, 10=0.21%, 20=0.09%, 50=0.03% cpu : usr=1.38%, sys=3.89%, ctx=85697, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68537,17153,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.4MiB/s (74.9MB/s), 71.4MiB/s-71.4MiB/s (74.9MB/s-74.9MB/s), io=2142MiB (2246MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=536MiB (562MB), run=30001-30001msec Disk stats (read/write): vdj: ios=68135/17053, merge=0/0, ticks=794/17460, in_queue=28088, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908678: Fri Jan 7 13:46:17 2022 read: IOPS=2315, BW=72.4MiB/s (75.9MB/s)(2171MiB/30001msec) slat (usec): min=5, max=249, avg=11.21, stdev= 4.55 clat (nsec): min=659, max=21671k, avg=50950.18, stdev=394555.47 lat (usec): min=26, max=21680, avg=62.31, stdev=394.60 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 40], | 70.00th=[ 42], 80.00th=[ 45], 90.00th=[ 50], 95.00th=[ 55], | 99.00th=[ 68], 99.50th=[ 80], 99.90th=[ 383], 99.95th=[12125], | 99.99th=[17171] bw ( KiB/s): min=61120, max=142976, per=99.96%, avg=74056.31, stdev=11287.06, samples=59 iops : min= 1910, max= 4468, avg=2314.24, stdev=352.72, samples=59 write: IOPS=578, BW=18.1MiB/s (18.0MB/s)(543MiB/30001msec) slat (usec): min=9, max=176, avg=15.12, stdev= 5.42 clat (usec): min=256, max=45314, avg=1455.23, stdev=1591.44 lat (usec): min=268, max=45326, avg=1470.52, stdev=1591.48 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 537], 10.00th=[ 824], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[36439], 99.95th=[40109], | 99.99th=[42206] bw ( KiB/s): min=14976, max=36928, per=100.00%, avg=18531.90, stdev=2822.20, samples=59 iops : min= 468, max= 1154, avg=579.12, stdev=88.19, samples=59 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.08%, 4=0.01%, 10=0.03%, 20=0.08%, 50=72.47% lat (usec) : 100=7.12%, 250=0.08%, 500=0.91%, 750=0.76%, 1000=1.63% lat (msec) : 2=15.42%, 4=1.04%, 10=0.20%, 20=0.10%, 50=0.02% cpu : usr=1.46%, sys=6.21%, ctx=86715, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69460,17370,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=72.4MiB/s (75.9MB/s), 72.4MiB/s-72.4MiB/s (75.9MB/s-75.9MB/s), io=2171MiB (2276MB), run=30001-30001msec WRITE: bw=18.1MiB/s (18.0MB/s), 18.1MiB/s-18.1MiB/s (18.0MB/s-18.0MB/s), io=543MiB (569MB), run=30001-30001msec Disk stats (read/write): vdj: ios=68974/17238, merge=0/0, ticks=761/17642, in_queue=27917, util=93.41% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909517: Fri Jan 7 13:46:54 2022 read: IOPS=2262, BW=70.7MiB/s (74.1MB/s)(2121MiB/30001msec) clat (usec): min=27, max=19804, avg=61.76, stdev=395.14 lat (usec): min=27, max=19804, avg=61.91, stdev=395.14 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 498], 99.95th=[13042], | 99.99th=[16909] bw ( KiB/s): min=40768, max=109184, per=99.92%, avg=72332.58, stdev=9247.67, samples=59 iops : min= 1274, max= 3412, avg=2260.39, stdev=288.99, samples=59 write: IOPS=566, BW=17.7MiB/s (18.6MB/s)(531MiB/30001msec) clat (usec): min=266, max=133269, avg=1508.02, stdev=1962.37 lat (usec): min=269, max=133272, avg=1510.91, stdev=1962.35 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 881], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1860], 95.00th=[ 2180], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[38011], 99.95th=[42206], | 99.99th=[44827] bw ( KiB/s): min=10752, max=28224, per=99.92%, avg=18104.81, stdev=2104.44, samples=59 iops : min= 336, max= 882, avg=565.76, stdev=65.75, samples=59 lat (usec) : 50=46.01%, 100=33.63%, 250=0.24%, 500=0.65%, 750=0.79% lat (usec) : 1000=1.46% lat (msec) : 2=15.76%, 4=1.08%, 10=0.25%, 20=0.11%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=1.37%, sys=3.76%, ctx=84866, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67872,16988,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=70.7MiB/s (74.1MB/s), 70.7MiB/s-70.7MiB/s (74.1MB/s-74.1MB/s), io=2121MiB (2224MB), run=30001-30001msec WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=531MiB (557MB), run=30001-30001msec Disk stats (read/write): vdj: ios=67374/16840, merge=0/0, ticks=744/17985, in_queue=28142, util=94.27% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910182: Fri Jan 7 13:47:31 2022 read: IOPS=2324, BW=72.6MiB/s (76.2MB/s)(2182MiB/30036msec) slat (usec): min=3, max=226, avg=10.94, stdev= 3.90 clat (nsec): min=812, max=2498.3M, avg=1080484.70, stdev=44335139.70 lat (usec): min=25, max=2498.3k, avg=1091.57, stdev=44335.12 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 43], | 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 57], | 99.00th=[ 81], 99.50th=[ 105], 99.90th=[ 529], | 99.95th=[ 817890], 99.99th=[2499806] bw ( KiB/s): min=61824, max=131776, per=100.00%, avg=74466.53, stdev=9625.76, samples=60 iops : min= 1932, max= 4118, avg=2327.07, stdev=300.85, samples=60 write: IOPS=581, BW=18.2MiB/s (19.0MB/s)(546MiB/30036msec) slat (usec): min=5, max=160, avg=15.37, stdev= 5.03 clat (usec): min=403, max=71928, avg=12811.87, stdev=6300.54 lat (usec): min=412, max=71941, avg=12827.39, stdev=6300.89 clat percentiles (usec): | 1.00th=[ 1319], 5.00th=[ 3326], 10.00th=[ 6783], 20.00th=[ 9634], | 30.00th=[10814], 40.00th=[11863], 50.00th=[12911], 60.00th=[13435], | 70.00th=[14222], 80.00th=[15139], 90.00th=[16909], 95.00th=[21365], | 99.00th=[30278], 99.50th=[62129], 99.90th=[66847], 99.95th=[68682], | 99.99th=[71828] bw ( KiB/s): min=15552, max=33152, per=100.00%, avg=18617.10, stdev=2328.60, samples=60 iops : min= 486, max= 1036, avg=581.78, stdev=72.77, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.08%, 50=70.86% lat (usec) : 100=8.54%, 250=0.34%, 500=0.03%, 750=0.02%, 1000=0.04% lat (msec) : 2=0.53%, 4=0.55%, 10=3.60%, 20=14.00%, 50=1.13% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.51%, sys=6.00%, ctx=86253, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69813,17460,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=72.6MiB/s (76.2MB/s), 72.6MiB/s-72.6MiB/s (76.2MB/s-76.2MB/s), io=2182MiB (2288MB), run=30036-30036msec WRITE: bw=18.2MiB/s (19.0MB/s), 18.2MiB/s-18.2MiB/s (19.0MB/s-19.0MB/s), io=546MiB (572MB), run=30036-30036msec Disk stats (read/write): vdj: ios=69395/17341, merge=0/0, ticks=69464/213423, in_queue=296303, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910981: Fri Jan 7 13:48:08 2022 read: IOPS=2257, BW=70.5MiB/s (73.0MB/s)(2116MiB/30001msec) clat (usec): min=28, max=25019, avg=64.76, stdev=426.54 lat (usec): min=28, max=25020, avg=64.90, stdev=426.55 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 93], 99.50th=[ 110], 99.90th=[ 938], 99.95th=[12780], | 99.99th=[17695] bw ( KiB/s): min=50148, max=113664, per=100.00%, avg=72241.28, stdev=9086.67, samples=60 iops : min= 1567, max= 3552, avg=2257.52, stdev=283.96, samples=60 write: IOPS=564, BW=17.6MiB/s (18.5MB/s)(529MiB/30001msec) clat (usec): min=276, max=64398, avg=1501.52, stdev=1713.37 lat (usec): min=279, max=64399, avg=1504.51, stdev=1713.37 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 619], 10.00th=[ 857], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1401], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min=13402, max=28992, per=100.00%, avg=18060.82, stdev=2099.24, samples=60 iops : min= 418, max= 906, avg=564.38, stdev=65.63, samples=60 lat (usec) : 50=40.99%, 100=38.44%, 250=0.44%, 500=0.67%, 750=0.84% lat (usec) : 1000=1.54% lat (msec) : 2=15.59%, 4=1.13%, 10=0.22%, 20=0.11%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.55%, sys=3.76%, ctx=84654, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67718,16931,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.0MB/s), 70.5MiB/s-70.5MiB/s (73.0MB/s-73.0MB/s), io=2116MiB (2219MB), run=30001-30001msec WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=529MiB (555MB), run=30001-30001msec Disk stats (read/write): vdj: ios=67196/16807, merge=0/0, ticks=804/17716, in_queue=28104, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911961: Fri Jan 7 13:48:45 2022 read: IOPS=2312, BW=72.3MiB/s (75.8MB/s)(2175MiB/30105msec) slat (usec): min=2, max=699, avg=11.20, stdev= 8.39 clat (nsec): min=671, max=2501.9M, avg=1015994.06, stdev=40364571.02 lat (usec): min=24, max=2501.9k, avg=1027.35, stdev=40364.56 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 49], 90.00th=[ 58], 95.00th=[ 75], | 99.00th=[ 494], 99.50th=[ 783], 99.90th=[ 1696], | 99.95th=[ 759170], 99.99th=[2499806] bw ( KiB/s): min=59776, max=138624, per=100.00%, avg=74244.58, stdev=10145.25, samples=60 iops : min= 1868, max= 4332, avg=2320.13, stdev=317.04, samples=60 write: IOPS=577, BW=18.1MiB/s (18.9MB/s)(544MiB/30105msec) slat (usec): min=4, max=423, avg=17.09, stdev=10.91 clat (msec): min=7, max=2038, avg=82.36, stdev=62.75 lat (msec): min=7, max=2038, avg=82.38, stdev=62.75 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 60], 10.00th=[ 64], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 81], 80.00th=[ 85], 90.00th=[ 97], 95.00th=[ 122], | 99.00th=[ 234], 99.50th=[ 518], 99.90th=[ 1003], 99.95th=[ 1133], | 99.99th=[ 1720] bw ( KiB/s): min=15616, max=32256, per=100.00%, avg=18510.50, stdev=2163.56, samples=60 iops : min= 488, max= 1008, avg=578.45, stdev=67.61, samples=60 lat (nsec) : 750=0.01%, 1000=0.10% lat (usec) : 2=0.10%, 4=0.01%, 10=0.04%, 20=0.11%, 50=65.27% lat (usec) : 100=11.75%, 250=1.17%, 500=0.65%, 750=0.36%, 1000=0.18% lat (msec) : 2=0.18%, 10=0.17%, 20=0.10%, 50=0.20%, 100=17.75% lat (msec) : 250=1.60%, 500=0.10%, 750=0.06%, 1000=0.04%, 2000=0.04% lat (msec) : >=2000=0.02% cpu : usr=1.66%, sys=5.97%, ctx=83424, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=69605,17400,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=72.3MiB/s (75.8MB/s), 72.3MiB/s-72.3MiB/s (75.8MB/s-75.8MB/s), io=2175MiB (2281MB), run=30105-30105msec WRITE: bw=18.1MiB/s (18.9MB/s), 18.1MiB/s-18.1MiB/s (18.9MB/s-18.9MB/s), io=544MiB (570MB), run=30105-30105msec Disk stats (read/write): vdj: ios=69602/17380, merge=0/0, ticks=63841/1419466, in_queue=1499202, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912593: Fri Jan 7 13:49:22 2022 read: IOPS=2323, BW=72.6MiB/s (76.1MB/s)(2179MiB/30001msec) clat (usec): min=26, max=26474, avg=63.35, stdev=440.74 lat (usec): min=27, max=26474, avg=63.49, stdev=440.75 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 89], 99.50th=[ 104], 99.90th=[ 562], 99.95th=[13829], | 99.99th=[19006] bw ( KiB/s): min=56000, max=145920, per=100.00%, avg=74371.53, stdev=11620.10, samples=60 iops : min= 1750, max= 4560, avg=2324.10, stdev=363.13, samples=60 write: IOPS=581, BW=18.2MiB/s (19.0MB/s)(545MiB/30001msec) clat (usec): min=268, max=50483, avg=1456.30, stdev=1601.72 lat (usec): min=271, max=50486, avg=1459.24, stdev=1601.71 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 562], 10.00th=[ 840], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2089], | 99.00th=[ 9110], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min=13120, max=37760, per=100.00%, avg=18594.23, stdev=2998.91, samples=60 iops : min= 410, max= 1180, avg=581.07, stdev=93.72, samples=60 lat (usec) : 50=45.81%, 100=33.73%, 250=0.35%, 500=0.86%, 750=0.81% lat (usec) : 1000=1.49% lat (msec) : 2=15.60%, 4=1.05%, 10=0.17%, 20=0.11%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.47%, sys=3.85%, ctx=87156, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69719,17432,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=72.6MiB/s (76.1MB/s), 72.6MiB/s-72.6MiB/s (76.1MB/s-76.1MB/s), io=2179MiB (2285MB), run=30001-30001msec WRITE: bw=18.2MiB/s (19.0MB/s), 18.2MiB/s-18.2MiB/s (19.0MB/s-19.0MB/s), io=545MiB (571MB), run=30001-30001msec Disk stats (read/write): vdj: ios=69179/17302, merge=0/0, ticks=852/17584, in_queue=28092, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913427: Fri Jan 7 13:49:59 2022 read: IOPS=2321, BW=72.6MiB/s (76.1MB/s)(2189MiB/30171msec) slat (usec): min=2, max=832, avg=10.65, stdev= 5.49 clat (nsec): min=780, max=2513.1M, avg=1244388.47, stdev=48009178.70 lat (usec): min=25, max=2513.1k, avg=1255.19, stdev=48009.18 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 44], 70.00th=[ 48], | 80.00th=[ 52], 90.00th=[ 62], 95.00th=[ 86], | 99.00th=[ 848], 99.50th=[ 1762], 99.90th=[ 3032], | 99.95th=[1010828], 99.99th=[2499806] bw ( KiB/s): min=54976, max=150528, per=100.00%, avg=74719.05, stdev=11773.63, samples=60 iops : min= 1718, max= 4704, avg=2335.00, stdev=367.91, samples=60 write: IOPS=581, BW=18.2MiB/s (19.0MB/s)(548MiB/30171msec) slat (usec): min=5, max=172, avg=17.27, stdev= 6.05 clat (msec): min=10, max=2630, avg=166.98, stdev=103.67 lat (msec): min=10, max=2630, avg=167.00, stdev=103.67 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 120], 10.00th=[ 132], 20.00th=[ 144], | 30.00th=[ 148], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 159], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 405], 99.50th=[ 919], 99.90th=[ 1603], 99.95th=[ 2056], | 99.99th=[ 2567] bw ( KiB/s): min=13312, max=32768, per=100.00%, avg=18604.30, stdev=2333.46, samples=60 iops : min= 416, max= 1024, avg=581.38, stdev=72.92, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.06%, 50=61.40% lat (usec) : 100=14.96%, 250=1.37%, 500=0.75%, 750=0.47%, 1000=0.16% lat (msec) : 2=0.43%, 4=0.24%, 10=0.01%, 20=0.11%, 50=0.29% lat (msec) : 100=0.23%, 250=18.80%, 500=0.43%, 750=0.06%, 1000=0.06% lat (msec) : 2000=0.08%, >=2000=0.04% cpu : usr=1.65%, sys=5.87%, ctx=83008, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70051,17537,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=72.6MiB/s (76.1MB/s), 72.6MiB/s-72.6MiB/s (76.1MB/s-76.1MB/s), io=2189MiB (2295MB), run=30171-30171msec WRITE: bw=18.2MiB/s (19.0MB/s), 18.2MiB/s-18.2MiB/s (19.0MB/s-19.0MB/s), io=548MiB (575MB), run=30171-30171msec Disk stats (read/write): vdj: ios=70048/17452, merge=0/0, ticks=79022/2894758, in_queue=2999316, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914739: Fri Jan 7 13:50:36 2022 read: IOPS=2299, BW=71.9MiB/s (75.3MB/s)(2156MiB/30002msec) clat (usec): min=28, max=62031, avg=63.26, stdev=476.58 lat (usec): min=28, max=62031, avg=63.39, stdev=476.58 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 469], 99.95th=[12125], | 99.99th=[18482] bw ( KiB/s): min=54656, max=114624, per=100.00%, avg=73588.27, stdev=9124.19, samples=60 iops : min= 1708, max= 3582, avg=2299.63, stdev=285.13, samples=60 write: IOPS=574, BW=17.0MiB/s (18.8MB/s)(539MiB/30002msec) clat (usec): min=267, max=98292, avg=1475.69, stdev=1690.65 lat (usec): min=269, max=98296, avg=1478.68, stdev=1690.63 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 627], 10.00th=[ 889], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1401], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 7439], 99.50th=[ 9765], 99.90th=[31589], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min=14976, max=29504, per=100.00%, avg=18391.47, stdev=2043.38, samples=60 iops : min= 468, max= 922, avg=574.73, stdev=63.86, samples=60 lat (usec) : 50=45.86%, 100=33.78%, 250=0.27%, 500=0.66%, 750=0.75% lat (usec) : 1000=1.48% lat (msec) : 2=15.72%, 4=1.19%, 10=0.17%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.51%, sys=3.71%, ctx=86238, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68989,17243,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.3MB/s), 71.9MiB/s-71.9MiB/s (75.3MB/s-75.3MB/s), io=2156MiB (2261MB), run=30002-30002msec WRITE: bw=17.0MiB/s (18.8MB/s), 17.0MiB/s-17.0MiB/s (18.8MB/s-18.8MB/s), io=539MiB (565MB), run=30002-30002msec Disk stats (read/write): vdj: ios=68924/17228, merge=0/0, ticks=849/17819, in_queue=28314, util=94.14% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915358: Fri Jan 7 13:51:13 2022 read: IOPS=2323, BW=72.6MiB/s (76.1MB/s)(2195MiB/30221msec) slat (usec): min=2, max=192234, avg=332.06, stdev=1805.89 clat (usec): min=783, max=2560.4k, avg=27716.08, stdev=45119.67 lat (usec): min=788, max=2560.4k, avg=28048.28, stdev=45163.93 clat percentiles (usec): | 1.00th=[ 1418], 5.00th=[ 6783], 10.00th=[ 15139], | 20.00th=[ 19268], 30.00th=[ 21365], 40.00th=[ 23462], | 50.00th=[ 25035], 60.00th=[ 26870], 70.00th=[ 28967], | 80.00th=[ 31851], 90.00th=[ 36963], 95.00th=[ 43254], | 99.00th=[ 83362], 99.50th=[ 177210], 99.90th=[ 233833], | 99.95th=[ 650118], 99.99th=[2533360] bw ( KiB/s): min=59840, max=145408, per=100.00%, avg=74854.92, stdev=11531.02, samples=60 iops : min= 1870, max= 4544, avg=2339.18, stdev=360.34, samples=60 write: IOPS=581, BW=18.2MiB/s (19.1MB/s)(549MiB/30221msec) slat (usec): min=4, max=201580, avg=370.32, stdev=2748.54 clat (msec): min=5, max=3366, avg=231.30, stdev=172.01 lat (msec): min=23, max=3366, avg=231.67, stdev=171.99 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 142], 10.00th=[ 165], 20.00th=[ 186], | 30.00th=[ 199], 40.00th=[ 207], 50.00th=[ 215], 60.00th=[ 222], | 70.00th=[ 230], 80.00th=[ 245], 90.00th=[ 268], 95.00th=[ 296], | 99.00th=[ 969], 99.50th=[ 1569], 99.90th=[ 2500], 99.95th=[ 2635], | 99.99th=[ 3373] bw ( KiB/s): min=14848, max=29824, per=100.00%, avg=18616.08, stdev=2008.10, samples=60 iops : min= 464, max= 932, avg=581.73, stdev=62.74, samples=60 lat (usec) : 1000=0.10% lat (msec) : 2=1.51%, 4=1.69%, 10=1.09%, 20=14.10%, 50=59.42% lat (msec) : 100=2.06%, 250=16.57%, 500=3.04%, 750=0.09%, 1000=0.08% lat (msec) : 2000=0.15%, >=2000=0.08% cpu : usr=1.06%, sys=2.05%, ctx=9878, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70229,17582,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=72.6MiB/s (76.1MB/s), 72.6MiB/s-72.6MiB/s (76.1MB/s-76.1MB/s), io=2195MiB (2301MB), run=30221-30221msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=549MiB (576MB), run=30221-30221msec Disk stats (read/write): vdj: ios=70226/17513, merge=0/0, ticks=69522/3564170, in_queue=3671332, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916230: Fri Jan 7 13:51:50 2022 read: IOPS=2094, BW=131MiB/s (137MB/s)(3928MiB/30002msec) clat (usec): min=32, max=68393, avg=71.23, stdev=506.33 lat (usec): min=32, max=68393, avg=71.38, stdev=506.33 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 109], 99.50th=[ 131], 99.90th=[ 709], 99.95th=[13566], | 99.99th=[16909] bw ( KiB/s): min=103168, max=212736, per=100.00%, avg=134092.97, stdev=17135.52, samples=60 iops : min= 1612, max= 3324, avg=2095.18, stdev=267.75, samples=60 write: IOPS=526, BW=32.9MiB/s (34.5MB/s)(988MiB/30002msec) clat (usec): min=359, max=75728, avg=1599.78, stdev=1678.31 lat (usec): min=366, max=75734, avg=1605.91, stdev=1678.25 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 676], 10.00th=[ 938], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1762], 90.00th=[ 2008], 95.00th=[ 2311], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=25600, max=55424, per=100.00%, avg=33725.85, stdev=3833.93, samples=60 iops : min= 400, max= 866, avg=526.95, stdev=59.92, samples=60 lat (usec) : 50=25.98%, 100=52.77%, 250=0.99%, 500=0.53%, 750=0.85% lat (usec) : 1000=1.07% lat (msec) : 2=15.72%, 4=1.75%, 10=0.21%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.49%, sys=3.92%, ctx=78666, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62852,15809,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3928MiB (4119MB), run=30002-30002msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=988MiB (1036MB), run=30002-30002msec Disk stats (read/write): vdj: ios=62447/15698, merge=0/0, ticks=807/17390, in_queue=27980, util=93.68% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916848: Fri Jan 7 13:52:27 2022 read: IOPS=2102, BW=131MiB/s (138MB/s)(3943MiB/30002msec) slat (usec): min=6, max=217, avg=13.52, stdev= 4.26 clat (nsec): min=795, max=28378k, avg=63193.33, stdev=440500.67 lat (usec): min=36, max=28393, avg=76.86, stdev=440.58 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 86], 99.50th=[ 100], 99.90th=[ 635], 99.95th=[13435], | 99.99th=[17433] bw ( KiB/s): min=104192, max=199808, per=100.00%, avg=134596.27, stdev=16490.37, samples=60 iops : min= 1628, max= 3122, avg=2103.07, stdev=257.66, samples=60 write: IOPS=528, BW=33.0MiB/s (34.7MB/s)(992MiB/30002msec) slat (usec): min=11, max=262, avg=20.20, stdev= 5.91 clat (usec): min=340, max=46472, avg=1557.48, stdev=1642.69 lat (usec): min=357, max=46494, avg=1577.85, stdev=1642.71 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 644], 10.00th=[ 865], 20.00th=[ 1139], | 30.00th=[ 1254], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1483], | 70.00th=[ 1582], 80.00th=[ 1713], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9241], 99.50th=[ 9896], 99.90th=[38011], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=27008, max=52096, per=100.00%, avg=33843.20, stdev=3527.47, samples=60 iops : min= 422, max= 814, avg=528.80, stdev=55.12, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=47.11% lat (usec) : 100=32.37%, 250=0.26%, 500=0.52%, 750=1.07%, 1000=1.30% lat (msec) : 2=15.37%, 4=1.62%, 10=0.20%, 20=0.12%, 50=0.03% cpu : usr=1.43%, sys=6.41%, ctx=78948, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63092,15865,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=131MiB/s (138MB/s), 131MiB/s-131MiB/s (138MB/s-138MB/s), io=3943MiB (4135MB), run=30002-30002msec WRITE: bw=33.0MiB/s (34.7MB/s), 33.0MiB/s-33.0MiB/s (34.7MB/s-34.7MB/s), io=992MiB (1040MB), run=30002-30002msec Disk stats (read/write): vdj: ios=62652/15746, merge=0/0, ticks=772/17024, in_queue=27900, util=93.39% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917796: Fri Jan 7 13:53:04 2022 read: IOPS=2094, BW=131MiB/s (137MB/s)(3928MiB/30001msec) clat (usec): min=30, max=62951, avg=70.72, stdev=496.00 lat (usec): min=30, max=62951, avg=70.86, stdev=496.00 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 78], | 99.00th=[ 102], 99.50th=[ 121], 99.90th=[ 848], 99.95th=[14091], | 99.99th=[17171] bw ( KiB/s): min=105728, max=210816, per=100.00%, avg=134435.73, stdev=16760.48, samples=59 iops : min= 1652, max= 3294, avg=2100.54, stdev=261.87, samples=59 write: IOPS=526, BW=32.9MiB/s (34.5MB/s)(988MiB/30001msec) clat (usec): min=359, max=43017, avg=1601.73, stdev=1579.13 lat (usec): min=365, max=43025, avg=1608.04, stdev=1579.11 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 685], 10.00th=[ 947], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1762], 90.00th=[ 2008], 95.00th=[ 2278], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[17695], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min=24320, max=54272, per=100.00%, avg=33744.56, stdev=3897.57, samples=59 iops : min= 380, max= 848, avg=527.25, stdev=60.90, samples=59 lat (usec) : 50=24.85%, 100=54.16%, 250=0.74%, 500=0.50%, 750=0.80% lat (usec) : 1000=1.06% lat (msec) : 2=15.75%, 4=1.75%, 10=0.20%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.54%, sys=3.86%, ctx=78668, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62852,15810,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3928MiB (4119MB), run=30001-30001msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=988MiB (1036MB), run=30001-30001msec Disk stats (read/write): vdj: ios=62447/15698, merge=0/0, ticks=809/17461, in_queue=27982, util=93.65% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918741: Fri Jan 7 13:53:41 2022 read: IOPS=2145, BW=134MiB/s (141MB/s)(4027MiB/30031msec) slat (usec): min=4, max=171, avg=12.58, stdev= 3.97 clat (nsec): min=752, max=2538.1M, avg=1169343.49, stdev=46594258.57 lat (usec): min=31, max=2538.1k, avg=1182.07, stdev=46594.23 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 36], 10.00th=[ 38], | 20.00th=[ 40], 30.00th=[ 42], 40.00th=[ 43], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 51], | 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 69], | 99.00th=[ 96], 99.50th=[ 131], 99.90th=[ 881], | 99.95th=[ 700449], 99.99th=[2499806] bw ( KiB/s): min=112128, max=258048, per=100.00%, avg=137470.97, stdev=19869.38, samples=60 iops : min= 1752, max= 4032, avg=2147.97, stdev=310.46, samples=60 write: IOPS=538, BW=33.7MiB/s (35.3MB/s)(1011MiB/30031msec) slat (usec): min=9, max=161, avg=20.73, stdev= 6.36 clat (usec): min=618, max=76118, avg=13820.77, stdev=6087.28 lat (usec): min=641, max=76134, avg=13841.66, stdev=6087.62 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 7177], 10.00th=[ 8455], 20.00th=[10290], | 30.00th=[11731], 40.00th=[12780], 50.00th=[13435], 60.00th=[14222], | 70.00th=[15008], 80.00th=[16057], 90.00th=[17957], 95.00th=[21890], | 99.00th=[34866], 99.50th=[61080], 99.90th=[66847], 99.95th=[68682], | 99.99th=[74974] bw ( KiB/s): min=28160, max=65152, per=100.00%, avg=34513.42, stdev=4570.37, samples=60 iops : min= 440, max= 1018, avg=539.27, stdev=71.41, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=54.42% lat (usec) : 100=24.73%, 250=0.58%, 500=0.03%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.09%, 4=0.42%, 10=3.04%, 20=15.11%, 50=1.28% lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.67%, sys=5.91%, ctx=79675, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=64434,16182,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=4027MiB (4223MB), run=30031-30031msec WRITE: bw=33.7MiB/s (35.3MB/s), 33.7MiB/s-33.7MiB/s (35.3MB/s-35.3MB/s), io=1011MiB (1061MB), run=30031-30031msec Disk stats (read/write): vdj: ios=64431/16182, merge=0/0, ticks=70700/215348, in_queue=298291, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919321: Fri Jan 7 13:54:18 2022 read: IOPS=2084, BW=130MiB/s (137MB/s)(3908MiB/30001msec) clat (usec): min=31, max=24625, avg=68.53, stdev=444.92 lat (usec): min=31, max=24625, avg=68.67, stdev=444.92 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 92], 99.50th=[ 105], 99.90th=[ 469], 99.95th=[13829], | 99.99th=[17957] bw ( KiB/s): min=95488, max=198016, per=100.00%, avg=134006.71, stdev=16420.81, samples=59 iops : min= 1492, max= 3094, avg=2093.83, stdev=256.57, samples=59 write: IOPS=523, BW=32.7MiB/s (34.3MB/s)(982MiB/30001msec) clat (usec): min=356, max=91490, avg=1621.55, stdev=1848.64 lat (usec): min=362, max=91496, avg=1627.58, stdev=1848.58 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 693], 10.00th=[ 971], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 2008], 95.00th=[ 2311], | 99.00th=[ 9372], 99.50th=[10028], 99.90th=[38011], 99.95th=[42206], | 99.99th=[44827] bw ( KiB/s): min=24704, max=51584, per=100.00%, avg=33624.90, stdev=3591.01, samples=59 iops : min= 386, max= 806, avg=525.37, stdev=56.11, samples=59 lat (usec) : 50=29.24%, 100=50.16%, 250=0.41%, 500=0.47%, 750=0.72% lat (usec) : 1000=1.01% lat (msec) : 2=15.87%, 4=1.78%, 10=0.19%, 20=0.13%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.51%, sys=3.66%, ctx=78247, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62523,15719,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=130MiB/s (137MB/s), 130MiB/s-130MiB/s (137MB/s-137MB/s), io=3908MiB (4098MB), run=30001-30001msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=982MiB (1030MB), run=30001-30001msec Disk stats (read/write): vdj: ios=62512/15715, merge=0/0, ticks=783/17744, in_queue=28293, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920168: Fri Jan 7 13:54:55 2022 read: IOPS=2155, BW=135MiB/s (141MB/s)(4055MiB/30107msec) slat (usec): min=2, max=167, avg=10.67, stdev= 4.38 clat (nsec): min=746, max=2520.6M, avg=1042984.60, stdev=41600319.71 lat (usec): min=29, max=2520.6k, avg=1053.80, stdev=41600.32 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 45], 70.00th=[ 49], | 80.00th=[ 54], 90.00th=[ 62], 95.00th=[ 72], | 99.00th=[ 281], 99.50th=[ 562], 99.90th=[ 1647], | 99.95th=[ 750781], 99.99th=[2499806] bw ( KiB/s): min=103168, max=268288, per=100.00%, avg=138417.75, stdev=20243.09, samples=60 iops : min= 1612, max= 4192, avg=2162.77, stdev=316.29, samples=60 write: IOPS=540, BW=33.8MiB/s (35.4MB/s)(1018MiB/30107msec) slat (usec): min=7, max=180, avg=20.85, stdev= 7.98 clat (msec): min=4, max=2139, avg=88.22, stdev=67.00 lat (msec): min=4, max=2139, avg=88.24, stdev=67.00 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 64], 10.00th=[ 70], 20.00th=[ 74], | 30.00th=[ 78], 40.00th=[ 80], 50.00th=[ 82], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 100], 95.00th=[ 127], | 99.00th=[ 207], 99.50th=[ 535], 99.90th=[ 1116], 99.95th=[ 1217], | 99.99th=[ 1989] bw ( KiB/s): min=30720, max=62208, per=100.00%, avg=34634.85, stdev=4170.97, samples=60 iops : min= 480, max= 972, avg=541.17, stdev=65.17, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.04%, 50=58.08% lat (usec) : 100=19.85%, 250=1.02%, 500=0.43%, 750=0.14%, 1000=0.06% lat (msec) : 2=0.18%, 10=0.02%, 20=0.28%, 50=0.18%, 100=17.62% lat (msec) : 250=1.80%, 500=0.09%, 750=0.07%, 1000=0.04%, 2000=0.04% lat (msec) : >=2000=0.02% cpu : usr=1.74%, sys=5.36%, ctx=78771, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=64881,16281,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4055MiB (4252MB), run=30107-30107msec WRITE: bw=33.8MiB/s (35.4MB/s), 33.8MiB/s-33.8MiB/s (35.4MB/s-35.4MB/s), io=1018MiB (1067MB), run=30107-30107msec Disk stats (read/write): vdj: ios=64879/16237, merge=0/0, ticks=62855/1421482, in_queue=1499955, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921320: Fri Jan 7 13:55:32 2022 read: IOPS=2115, BW=132MiB/s (139MB/s)(3966MiB/30001msec) clat (usec): min=31, max=32240, avg=70.88, stdev=443.48 lat (usec): min=31, max=32240, avg=71.03, stdev=443.48 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 99], 99.50th=[ 113], 99.90th=[ 1012], 99.95th=[13042], | 99.99th=[16909] bw ( KiB/s): min=94200, max=209280, per=100.00%, avg=135617.60, stdev=16457.18, samples=60 iops : min= 1471, max= 3270, avg=2118.70, stdev=257.27, samples=60 write: IOPS=531, BW=33.2MiB/s (34.8MB/s)(996MiB/30001msec) clat (usec): min=349, max=99441, avg=1584.65, stdev=1778.47 lat (usec): min=355, max=99447, avg=1590.96, stdev=1778.38 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 660], 10.00th=[ 922], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1598], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2245], | 99.00th=[ 9110], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[42730], | 99.99th=[51119] bw ( KiB/s): min=25188, max=54016, per=100.00%, avg=34073.32, stdev=3792.27, samples=60 iops : min= 393, max= 844, avg=532.17, stdev=59.42, samples=60 lat (usec) : 50=22.49%, 100=56.70%, 250=0.63%, 500=0.48%, 750=0.90% lat (usec) : 1000=1.06% lat (msec) : 2=15.85%, 4=1.57%, 10=0.19%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.68%, sys=3.82%, ctx=79406, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63453,15942,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3966MiB (4158MB), run=30001-30001msec WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=996MiB (1045MB), run=30001-30001msec Disk stats (read/write): vdj: ios=63439/15939, merge=0/0, ticks=808/17398, in_queue=28259, util=93.89% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922081: Fri Jan 7 13:56:10 2022 read: IOPS=2134, BW=133MiB/s (140MB/s)(4025MiB/30164msec) slat (usec): min=2, max=357, avg=11.09, stdev= 6.17 clat (nsec): min=743, max=2517.1M, avg=1115350.61, stdev=44596505.71 lat (usec): min=28, max=2517.1k, avg=1126.59, stdev=44596.47 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 53], | 80.00th=[ 58], 90.00th=[ 70], 95.00th=[ 122], | 99.00th=[ 1385], 99.50th=[ 2409], 99.90th=[ 3490], | 99.95th=[ 683672], 99.99th=[2499806] bw ( KiB/s): min=103680, max=278912, per=100.00%, avg=137393.60, stdev=22570.68, samples=60 iops : min= 1620, max= 4358, avg=2146.75, stdev=352.67, samples=60 write: IOPS=536, BW=33.5MiB/s (35.1MB/s)(1011MiB/30164msec) slat (usec): min=8, max=460, avg=22.19, stdev=10.91 clat (msec): min=4, max=2321, avg=182.02, stdev=111.43 lat (msec): min=4, max=2321, avg=182.04, stdev=111.43 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 127], 10.00th=[ 142], 20.00th=[ 157], | 30.00th=[ 163], 40.00th=[ 167], 50.00th=[ 171], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 213], 95.00th=[ 241], | 99.00th=[ 405], 99.50th=[ 911], 99.90th=[ 1888], 99.95th=[ 2022], | 99.99th=[ 2265] bw ( KiB/s): min=28032, max=58880, per=99.93%, avg=34282.73, stdev=4072.79, samples=60 iops : min= 438, max= 920, avg=535.65, stdev=63.65, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.04%, 50=52.54% lat (usec) : 100=22.76%, 250=1.63%, 500=0.83%, 750=0.52%, 1000=0.35% lat (msec) : 2=0.60%, 4=0.46%, 10=0.01%, 20=0.14%, 50=0.30% lat (msec) : 100=0.18%, 250=18.60%, 500=0.71%, 750=0.05%, 1000=0.05% lat (msec) : 2000=0.08%, >=2000=0.04% cpu : usr=1.72%, sys=5.47%, ctx=75567, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64399,16169,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4025MiB (4220MB), run=30164-30164msec WRITE: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=1011MiB (1060MB), run=30164-30164msec Disk stats (read/write): vdj: ios=64399/16087, merge=0/0, ticks=67297/2911527, in_queue=3000621, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923046: Fri Jan 7 13:56:46 2022 read: IOPS=2108, BW=132MiB/s (138MB/s)(3954MiB/30001msec) clat (usec): min=30, max=53259, avg=67.61, stdev=487.83 lat (usec): min=30, max=53260, avg=67.75, stdev=487.83 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 87], 99.50th=[ 97], 99.90th=[ 644], 99.95th=[14222], | 99.99th=[18220] bw ( KiB/s): min=109568, max=212480, per=100.00%, avg=134956.45, stdev=15491.78, samples=60 iops : min= 1712, max= 3320, avg=2108.65, stdev=242.08, samples=60 write: IOPS=529, BW=33.1MiB/s (34.7MB/s)(993MiB/30001msec) clat (usec): min=348, max=43612, avg=1604.20, stdev=1602.15 lat (usec): min=354, max=43615, avg=1610.18, stdev=1602.10 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 693], 10.00th=[ 971], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1778], 90.00th=[ 2024], 95.00th=[ 2311], | 99.00th=[ 9241], 99.50th=[ 9896], 99.90th=[37487], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=28928, max=55168, per=99.99%, avg=33904.82, stdev=3535.50, samples=60 iops : min= 452, max= 862, avg=529.73, stdev=55.23, samples=60 lat (usec) : 50=34.94%, 100=44.66%, 250=0.23%, 500=0.48%, 750=0.70% lat (usec) : 1000=1.03% lat (msec) : 2=15.75%, 4=1.88%, 10=0.19%, 20=0.12%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.54%, sys=3.52%, ctx=79169, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63265,15895,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3954MiB (4146MB), run=30001-30001msec WRITE: bw=33.1MiB/s (34.7MB/s), 33.1MiB/s-33.1MiB/s (34.7MB/s-34.7MB/s), io=993MiB (1042MB), run=30001-30001msec Disk stats (read/write): vdj: ios=63265/15894, merge=0/0, ticks=835/17767, in_queue=28423, util=94.40% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923719: Fri Jan 7 13:57:24 2022 read: IOPS=2152, BW=135MiB/s (141MB/s)(4067MiB/30241msec) slat (usec): min=2, max=218861, avg=364.13, stdev=2314.25 clat (usec): min=1082, max=2551.4k, avg=30008.46, stdev=48246.87 lat (usec): min=1089, max=2551.4k, avg=30372.74, stdev=48310.19 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 5538], 10.00th=[ 16057], | 20.00th=[ 20317], 30.00th=[ 23200], 40.00th=[ 25035], | 50.00th=[ 27132], 60.00th=[ 28967], 70.00th=[ 31589], | 80.00th=[ 34866], 90.00th=[ 40109], 95.00th=[ 45876], | 99.00th=[ 90702], 99.50th=[ 221250], 99.90th=[ 261096], | 99.95th=[ 851444], 99.99th=[2533360] bw ( KiB/s): min=99200, max=292608, per=100.00%, avg=138713.08, stdev=23800.18, samples=60 iops : min= 1550, max= 4572, avg=2167.37, stdev=371.87, samples=60 write: IOPS=539, BW=33.7MiB/s (35.4MB/s)(1020MiB/30241msec) slat (usec): min=7, max=209580, avg=376.05, stdev=2313.12 clat (msec): min=10, max=2243, avg=249.00, stdev=146.47 lat (msec): min=22, max=2243, avg=249.37, stdev=146.45 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 140], 10.00th=[ 174], 20.00th=[ 209], | 30.00th=[ 224], 40.00th=[ 230], 50.00th=[ 236], 60.00th=[ 243], | 70.00th=[ 253], 80.00th=[ 266], 90.00th=[ 288], 95.00th=[ 330], | 99.00th=[ 869], 99.50th=[ 1653], 99.90th=[ 1938], 99.95th=[ 2056], | 99.99th=[ 2232] bw ( KiB/s): min=25472, max=60160, per=99.99%, avg=34541.10, stdev=4329.34, samples=60 iops : min= 398, max= 940, avg=539.68, stdev=67.66, samples=60 lat (msec) : 2=0.84%, 4=2.07%, 10=2.09%, 20=9.83%, 50=62.48% lat (msec) : 100=2.52%, 250=13.58%, 500=6.25%, 750=0.09%, 1000=0.06% lat (msec) : 2000=0.17%, >=2000=0.04% cpu : usr=1.62%, sys=2.05%, ctx=9091, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=65079,16323,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4067MiB (4265MB), run=30241-30241msec WRITE: bw=33.7MiB/s (35.4MB/s), 33.7MiB/s-33.7MiB/s (35.4MB/s-35.4MB/s), io=1020MiB (1070MB), run=30241-30241msec Disk stats (read/write): vdj: ios=65076/16224, merge=0/0, ticks=76218/3558639, in_queue=3671976, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924586: Fri Jan 7 13:58:00 2022 read: IOPS=2016, BW=158MiB/s (165MB/s)(4725MiB/30001msec) clat (usec): min=38, max=25289, avg=82.63, stdev=477.78 lat (usec): min=38, max=25289, avg=82.77, stdev=477.79 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 76], 90.00th=[ 84], 95.00th=[ 92], | 99.00th=[ 122], 99.50th=[ 149], 99.90th=[ 6128], 99.95th=[14353], | 99.99th=[19006] bw ( KiB/s): min=125920, max=235840, per=99.96%, avg=161217.29, stdev=17643.70, samples=59 iops : min= 1574, max= 2948, avg=2015.20, stdev=220.55, samples=59 write: IOPS=506, BW=39.6MiB/s (41.5MB/s)(1187MiB/30001msec) clat (usec): min=396, max=68186, avg=1628.12, stdev=1712.13 lat (usec): min=404, max=68194, avg=1636.47, stdev=1712.19 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 676], 10.00th=[ 873], 20.00th=[ 1172], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[35390], 99.95th=[39060], | 99.99th=[43779] bw ( KiB/s): min=30880, max=61600, per=100.00%, avg=40554.54, stdev=4401.26, samples=59 iops : min= 386, max= 770, avg=506.92, stdev=55.02, samples=59 lat (usec) : 50=1.43%, 100=76.24%, 250=2.04%, 500=0.38%, 750=1.12% lat (usec) : 1000=1.27% lat (msec) : 2=15.19%, 4=1.94%, 10=0.20%, 20=0.15%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.71%, sys=3.87%, ctx=75686, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60483,15198,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=158MiB/s (165MB/s), 158MiB/s-158MiB/s (165MB/s-165MB/s), io=4725MiB (4955MB), run=30001-30001msec WRITE: bw=39.6MiB/s (41.5MB/s), 39.6MiB/s-39.6MiB/s (41.5MB/s-41.5MB/s), io=1187MiB (1245MB), run=30001-30001msec Disk stats (read/write): vdj: ios=60103/15103, merge=0/0, ticks=875/16802, in_queue=28091, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925586: Fri Jan 7 13:58:37 2022 read: IOPS=2030, BW=159MiB/s (166MB/s)(4758MiB/30001msec) slat (usec): min=7, max=179, avg=14.43, stdev= 4.62 clat (nsec): min=803, max=24681k, avg=67173.97, stdev=461957.07 lat (usec): min=36, max=24695, avg=81.76, stdev=462.02 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 72], | 99.00th=[ 90], 99.50th=[ 102], 99.90th=[ 5735], 99.95th=[14091], | 99.99th=[18220] bw ( KiB/s): min=131040, max=259200, per=100.00%, avg=162425.32, stdev=20553.06, samples=60 iops : min= 1638, max= 3240, avg=2030.30, stdev=256.88, samples=60 write: IOPS=510, BW=39.8MiB/s (41.8MB/s)(1196MiB/30001msec) slat (usec): min=11, max=202, avg=22.78, stdev= 6.59 clat (usec): min=386, max=44113, avg=1604.33, stdev=1630.06 lat (usec): min=406, max=44135, avg=1627.29, stdev=1630.09 clat percentiles (usec): | 1.00th=[ 465], 5.00th=[ 660], 10.00th=[ 865], 20.00th=[ 1172], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1778], 90.00th=[ 2024], 95.00th=[ 2343], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[35390], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min=31200, max=67200, per=100.00%, avg=40806.93, stdev=4754.71, samples=60 iops : min= 390, max= 840, avg=510.08, stdev=59.43, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.01%, 50=38.29%, 100=41.15% lat (usec) : 250=0.31%, 500=0.41%, 750=1.14%, 1000=1.20% lat (msec) : 2=15.29%, 4=1.81%, 10=0.20%, 20=0.15%, 50=0.02% cpu : usr=1.56%, sys=6.38%, ctx=76193, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60907,15303,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=159MiB/s (166MB/s), 159MiB/s-159MiB/s (166MB/s-166MB/s), io=4758MiB (4990MB), run=30001-30001msec WRITE: bw=39.8MiB/s (41.8MB/s), 39.8MiB/s-39.8MiB/s (41.8MB/s-41.8MB/s), io=1196MiB (1254MB), run=30001-30001msec Disk stats (read/write): vdj: ios=60460/15187, merge=0/0, ticks=844/16774, in_queue=27781, util=93.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926125: Fri Jan 7 13:59:14 2022 read: IOPS=2024, BW=158MiB/s (166MB/s)(4745MiB/30001msec) clat (usec): min=33, max=24169, avg=73.19, stdev=470.26 lat (usec): min=33, max=24169, avg=73.33, stdev=470.26 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 97], 99.50th=[ 112], 99.90th=[ 7504], 99.95th=[14353], | 99.99th=[18220] bw ( KiB/s): min=133652, max=266240, per=100.00%, avg=161978.65, stdev=19593.81, samples=60 iops : min= 1670, max= 3328, avg=2024.63, stdev=244.95, samples=60 write: IOPS=508, BW=39.8MiB/s (41.7MB/s)(1193MiB/30001msec) clat (usec): min=405, max=43613, avg=1657.54, stdev=1624.10 lat (usec): min=413, max=43622, avg=1665.39, stdev=1624.03 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 709], 10.00th=[ 988], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=33760, max=69600, per=100.00%, avg=40720.02, stdev=4856.04, samples=60 iops : min= 422, max= 870, avg=508.90, stdev=60.74, samples=60 lat (usec) : 50=17.96%, 100=61.27%, 250=0.56%, 500=0.36%, 750=0.84% lat (usec) : 1000=0.90% lat (msec) : 2=15.63%, 4=2.09%, 10=0.22%, 20=0.15%, 50=0.02% cpu : usr=1.52%, sys=3.82%, ctx=76005, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60732,15268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4745MiB (4975MB), run=30001-30001msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1193MiB (1251MB), run=30001-30001msec Disk stats (read/write): vdj: ios=60730/15266, merge=0/0, ticks=868/17475, in_queue=28288, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926964: Fri Jan 7 13:59:51 2022 read: IOPS=2076, BW=162MiB/s (170MB/s)(4871MiB/30031msec) slat (usec): min=4, max=151, avg=12.92, stdev= 4.27 clat (nsec): min=731, max=2511.7M, avg=1171433.87, stdev=43980108.61 lat (usec): min=32, max=2511.7k, avg=1184.51, stdev=43980.12 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 50], | 80.00th=[ 53], 90.00th=[ 60], 95.00th=[ 67], | 99.00th=[ 94], 99.50th=[ 126], 99.90th=[ 22676], | 99.95th=[ 792724], 99.99th=[2499806] bw ( KiB/s): min=125120, max=294188, per=100.00%, avg=166262.57, stdev=21699.41, samples=60 iops : min= 1564, max= 3677, avg=2078.27, stdev=271.22, samples=60 write: IOPS=521, BW=40.7MiB/s (42.7MB/s)(1224MiB/30031msec) slat (usec): min=9, max=240, avg=23.31, stdev= 7.62 clat (usec): min=563, max=77790, avg=14423.44, stdev=6273.73 lat (usec): min=581, max=77811, avg=14446.93, stdev=6274.27 clat percentiles (usec): | 1.00th=[ 2900], 5.00th=[ 6652], 10.00th=[ 8455], 20.00th=[10421], | 30.00th=[11994], 40.00th=[13173], 50.00th=[14222], 60.00th=[15270], | 70.00th=[16188], 80.00th=[16909], 90.00th=[19006], 95.00th=[23200], | 99.00th=[30016], 99.50th=[63701], 99.90th=[69731], 99.95th=[76022], | 99.99th=[77071] bw ( KiB/s): min=35360, max=74869, per=100.00%, avg=41755.73, stdev=5210.32, samples=60 iops : min= 442, max= 935, avg=521.92, stdev=65.05, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=57.43% lat (usec) : 100=21.77%, 250=0.48%, 500=0.06%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.31%, 10=3.08%, 20=14.98%, 50=1.56% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=1.62%, sys=6.03%, ctx=77160, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62345,15664,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=4871MiB (5107MB), run=30031-30031msec WRITE: bw=40.7MiB/s (42.7MB/s), 40.7MiB/s-40.7MiB/s (42.7MB/s-42.7MB/s), io=1224MiB (1283MB), run=30031-30031msec Disk stats (read/write): vdj: ios=61979/15554, merge=0/0, ticks=69231/216267, in_queue=296274, util=99.72% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927607: Fri Jan 7 14:00:28 2022 read: IOPS=2036, BW=159MiB/s (167MB/s)(4773MiB/30002msec) clat (usec): min=33, max=25844, avg=71.37, stdev=448.77 lat (usec): min=33, max=25844, avg=71.51, stdev=448.77 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 96], 99.50th=[ 109], 99.90th=[ 1418], 99.95th=[13829], | 99.99th=[17433] bw ( KiB/s): min=129760, max=265170, per=100.00%, avg=162971.27, stdev=19811.62, samples=60 iops : min= 1622, max= 3314, avg=2037.08, stdev=247.61, samples=60 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1199MiB/30002msec) clat (usec): min=397, max=43514, avg=1654.14, stdev=1615.13 lat (usec): min=404, max=43522, avg=1662.10, stdev=1615.05 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 709], 10.00th=[ 1004], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1827], 90.00th=[ 2073], 95.00th=[ 2343], | 99.00th=[ 9241], 99.50th=[10028], 99.90th=[35390], 99.95th=[41681], | 99.99th=[43254] bw ( KiB/s): min=34720, max=68777, per=100.00%, avg=40942.25, stdev=4549.02, samples=60 iops : min= 434, max= 859, avg=511.72, stdev=56.82, samples=60 lat (usec) : 50=21.48%, 100=57.80%, 250=0.52%, 500=0.36%, 750=0.81% lat (usec) : 1000=0.86% lat (msec) : 2=15.62%, 4=2.19%, 10=0.18%, 20=0.14%, 50=0.02% cpu : usr=1.68%, sys=3.62%, ctx=76454, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61097,15350,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4773MiB (5005MB), run=30002-30002msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1199MiB (1257MB), run=30002-30002msec Disk stats (read/write): vdj: ios=60701/15258, merge=0/0, ticks=809/17439, in_queue=28211, util=94.60% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928865: Fri Jan 7 14:01:05 2022 read: IOPS=2065, BW=161MiB/s (169MB/s)(4858MiB/30111msec) slat (usec): min=2, max=240, avg=11.13, stdev= 4.77 clat (nsec): min=760, max=2524.0M, avg=1277227.45, stdev=48111647.61 lat (usec): min=30, max=2524.0k, avg=1288.50, stdev=48111.65 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 45], 70.00th=[ 51], | 80.00th=[ 57], 90.00th=[ 65], 95.00th=[ 75], | 99.00th=[ 392], 99.50th=[ 775], 99.90th=[ 50594], | 99.95th=[ 968885], 99.99th=[2499806] bw ( KiB/s): min=132640, max=322080, per=100.00%, avg=165861.62, stdev=24703.39, samples=60 iops : min= 1658, max= 4026, avg=2073.20, stdev=308.81, samples=60 write: IOPS=518, BW=40.5MiB/s (42.5MB/s)(1221MiB/30111msec) slat (usec): min=7, max=406, avg=23.76, stdev=10.73 clat (msec): min=6, max=2539, avg=91.17, stdev=78.57 lat (msec): min=6, max=2539, avg=91.20, stdev=78.57 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 77], | 30.00th=[ 80], 40.00th=[ 82], 50.00th=[ 84], 60.00th=[ 87], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 104], 95.00th=[ 131], | 99.00th=[ 201], 99.50th=[ 527], 99.90th=[ 1401], 99.95th=[ 1569], | 99.99th=[ 2265] bw ( KiB/s): min=35840, max=74880, per=100.00%, avg=41552.37, stdev=4995.96, samples=60 iops : min= 448, max= 936, avg=519.38, stdev=62.46, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.07%, 4=0.01%, 10=0.02%, 20=0.03%, 50=55.10% lat (usec) : 100=22.67%, 250=0.90%, 500=0.49%, 750=0.22%, 1000=0.11% lat (msec) : 2=0.21%, 10=0.02%, 20=0.30%, 50=0.17%, 100=17.18% lat (msec) : 250=2.25%, 500=0.08%, 750=0.04%, 1000=0.03%, 2000=0.07% lat (msec) : >=2000=0.03% cpu : usr=1.88%, sys=5.12%, ctx=75486, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=62188,15627,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4858MiB (5094MB), run=30111-30111msec WRITE: bw=40.5MiB/s (42.5MB/s), 40.5MiB/s-40.5MiB/s (42.5MB/s-42.5MB/s), io=1221MiB (1280MB), run=30111-30111msec Disk stats (read/write): vdj: ios=62186/15587, merge=0/0, ticks=75189/1410401, in_queue=1499876, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929912: Fri Jan 7 14:01:42 2022 read: IOPS=2035, BW=159MiB/s (167MB/s)(4770MiB/30001msec) clat (usec): min=33, max=34906, avg=74.63, stdev=467.58 lat (usec): min=34, max=34906, avg=74.77, stdev=467.59 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 101], 99.50th=[ 118], 99.90th=[ 2180], 99.95th=[14353], | 99.99th=[16712] bw ( KiB/s): min=125280, max=254080, per=100.00%, avg=162818.67, stdev=19512.02, samples=60 iops : min= 1566, max= 3176, avg=2035.23, stdev=243.90, samples=60 write: IOPS=511, BW=39.9MiB/s (41.9MB/s)(1198MiB/30001msec) clat (usec): min=395, max=44189, avg=1642.25, stdev=1562.36 lat (usec): min=403, max=44198, avg=1650.14, stdev=1562.28 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 701], 10.00th=[ 963], 20.00th=[ 1237], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9372], 99.50th=[10028], 99.90th=[18220], 99.95th=[37487], | 99.99th=[43254] bw ( KiB/s): min=33120, max=65760, per=100.00%, avg=40901.33, stdev=4515.98, samples=60 iops : min= 414, max= 822, avg=511.27, stdev=56.45, samples=60 lat (usec) : 50=12.54%, 100=66.57%, 250=0.68%, 500=0.35%, 750=0.92% lat (usec) : 1000=0.96% lat (msec) : 2=15.59%, 4=2.04%, 10=0.20%, 20=0.14%, 50=0.02% cpu : usr=1.77%, sys=3.76%, ctx=76403, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61057,15339,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4770MiB (5002MB), run=30001-30001msec WRITE: bw=39.9MiB/s (41.9MB/s), 39.9MiB/s-39.9MiB/s (41.9MB/s-41.9MB/s), io=1198MiB (1257MB), run=30001-30001msec Disk stats (read/write): vdj: ios=61008/15328, merge=0/0, ticks=852/17276, in_queue=28139, util=93.54% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930585: Fri Jan 7 14:02:20 2022 read: IOPS=2057, BW=161MiB/s (169MB/s)(4851MiB/30179msec) slat (usec): min=2, max=203, avg=11.22, stdev= 5.50 clat (nsec): min=697, max=2560.8M, avg=1321283.84, stdev=49198187.62 lat (usec): min=26, max=2560.8k, avg=1332.66, stdev=49198.17 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 44], 60.00th=[ 49], 70.00th=[ 56], | 80.00th=[ 62], 90.00th=[ 74], 95.00th=[ 130], | 99.00th=[ 1614], 99.50th=[ 2966], 99.90th=[ 46924], | 99.95th=[ 926942], 99.99th=[2499806] bw ( KiB/s): min=127040, max=349440, per=100.00%, avg=165580.87, stdev=29385.99, samples=60 iops : min= 1588, max= 4368, avg=2069.73, stdev=367.34, samples=60 write: IOPS=516, BW=40.4MiB/s (42.3MB/s)(1219MiB/30179msec) slat (usec): min=10, max=519, avg=24.20, stdev=12.35 clat (msec): min=10, max=3757, avg=188.12, stdev=177.61 lat (msec): min=10, max=3757, avg=188.15, stdev=177.61 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 131], 10.00th=[ 138], 20.00th=[ 148], | 30.00th=[ 163], 40.00th=[ 169], 50.00th=[ 174], 60.00th=[ 178], | 70.00th=[ 184], 80.00th=[ 194], 90.00th=[ 220], 95.00th=[ 243], | 99.00th=[ 430], 99.50th=[ 1183], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3574] bw ( KiB/s): min=32000, max=74080, per=99.96%, avg=41332.07, stdev=5297.89, samples=60 iops : min= 400, max= 926, avg=516.62, stdev=66.24, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.05%, 50=50.25% lat (usec) : 100=24.75%, 250=1.81%, 500=0.95%, 750=0.51%, 1000=0.32% lat (msec) : 2=0.46%, 4=0.50%, 10=0.10%, 20=0.09%, 50=0.30% lat (msec) : 100=0.19%, 250=18.57%, 500=0.79%, 750=0.04%, 1000=0.03% lat (msec) : 2000=0.07%, >=2000=0.09% cpu : usr=1.90%, sys=5.15%, ctx=72619, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62087,15598,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4851MiB (5086MB), run=30179-30179msec WRITE: bw=40.4MiB/s (42.3MB/s), 40.4MiB/s-40.4MiB/s (42.3MB/s-42.3MB/s), io=1219MiB (1278MB), run=30179-30179msec Disk stats (read/write): vdj: ios=62087/15506, merge=0/0, ticks=77596/2898006, in_queue=2999142, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931413: Fri Jan 7 14:02:56 2022 read: IOPS=2027, BW=158MiB/s (166MB/s)(4752MiB/30001msec) clat (usec): min=33, max=28870, avg=70.58, stdev=460.19 lat (usec): min=33, max=28870, avg=70.72, stdev=460.19 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 94], 99.50th=[ 109], 99.90th=[ 9372], 99.95th=[14353], | 99.99th=[17171] bw ( KiB/s): min=132640, max=244000, per=100.00%, avg=162202.37, stdev=18826.08, samples=60 iops : min= 1658, max= 3050, avg=2027.52, stdev=235.32, samples=60 write: IOPS=509, BW=39.8MiB/s (41.7MB/s)(1194MiB/30001msec) clat (usec): min=393, max=45204, avg=1665.61, stdev=1627.77 lat (usec): min=400, max=45212, avg=1673.16, stdev=1627.71 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 717], 10.00th=[ 1020], 20.00th=[ 1270], | 30.00th=[ 1369], 40.00th=[ 1434], 50.00th=[ 1500], 60.00th=[ 1598], | 70.00th=[ 1696], 80.00th=[ 1827], 90.00th=[ 2073], 95.00th=[ 2343], | 99.00th=[ 9372], 99.50th=[10028], 99.90th=[35914], 99.95th=[41157], | 99.99th=[44827] bw ( KiB/s): min=34880, max=63680, per=99.99%, avg=40765.22, stdev=4140.58, samples=60 iops : min= 436, max= 796, avg=509.55, stdev=51.75, samples=60 lat (usec) : 50=27.97%, 100=51.37%, 250=0.46%, 500=0.36%, 750=0.78% lat (usec) : 1000=0.83% lat (msec) : 2=15.75%, 4=2.13%, 10=0.18%, 20=0.16%, 50=0.02% cpu : usr=1.60%, sys=3.52%, ctx=76120, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60830,15289,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4752MiB (4983MB), run=30001-30001msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1194MiB (1252MB), run=30001-30001msec Disk stats (read/write): vdj: ios=60830/15288, merge=0/0, ticks=841/17562, in_queue=28404, util=94.35% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932355: Fri Jan 7 14:03:34 2022 read: IOPS=2071, BW=162MiB/s (170MB/s)(4893MiB/30229msec) slat (usec): min=2, max=263287, avg=379.10, stdev=2702.08 clat (usec): min=1109, max=2731.8k, avg=31249.86, stdev=50427.42 lat (usec): min=1113, max=2731.8k, avg=31629.12, stdev=50503.37 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 31], | 70.00th=[ 33], 80.00th=[ 36], 90.00th=[ 42], 95.00th=[ 47], | 99.00th=[ 89], 99.50th=[ 249], 99.90th=[ 284], 99.95th=[ 751], | 99.99th=[ 2534] bw ( KiB/s): min=119840, max=361920, per=100.00%, avg=166864.83, stdev=28985.11, samples=60 iops : min= 1498, max= 4524, avg=2085.77, stdev=362.32, samples=60 write: IOPS=520, BW=40.7MiB/s (42.7MB/s)(1230MiB/30229msec) slat (usec): min=7, max=65927, avg=387.43, stdev=1712.08 clat (msec): min=21, max=3824, avg=257.79, stdev=176.05 lat (msec): min=23, max=3824, avg=258.18, stdev=176.03 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 184], 10.00th=[ 201], 20.00th=[ 215], | 30.00th=[ 224], 40.00th=[ 232], 50.00th=[ 241], 60.00th=[ 247], | 70.00th=[ 255], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 342], | 99.00th=[ 676], 99.50th=[ 1519], 99.90th=[ 2567], 99.95th=[ 3406], | 99.99th=[ 3809] bw ( KiB/s): min=27840, max=74080, per=99.92%, avg=41622.68, stdev=5532.50, samples=60 iops : min= 348, max= 926, avg=520.23, stdev=69.18, samples=60 lat (msec) : 2=0.45%, 4=2.13%, 10=2.43%, 20=7.66%, 50=64.39% lat (msec) : 100=2.76%, 250=12.78%, 500=6.95%, 750=0.22%, 1000=0.03% lat (msec) : 2000=0.09%, >=2000=0.10% cpu : usr=1.72%, sys=2.06%, ctx=8683, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62630,15740,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=4893MiB (5131MB), run=30229-30229msec WRITE: bw=40.7MiB/s (42.7MB/s), 40.7MiB/s-40.7MiB/s (42.7MB/s-42.7MB/s), io=1230MiB (1289MB), run=30229-30229msec Disk stats (read/write): vdj: ios=62629/15629, merge=0/0, ticks=80063/3552481, in_queue=3671533, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932856: Fri Jan 7 14:04:10 2022 read: IOPS=1811, BW=226MiB/s (238MB/s)(6796MiB/30003msec) clat (usec): min=39, max=26991, avg=84.11, stdev=504.42 lat (usec): min=39, max=26991, avg=84.26, stdev=504.42 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 56], | 30.00th=[ 59], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 77], 90.00th=[ 85], 95.00th=[ 93], | 99.00th=[ 120], 99.50th=[ 139], 99.90th=[10945], 99.95th=[14484], | 99.99th=[19268] bw ( KiB/s): min=139520, max=367616, per=100.00%, avg=231979.47, stdev=30443.69, samples=60 iops : min= 1090, max= 2872, avg=1812.30, stdev=237.86, samples=60 write: IOPS=455, BW=56.9MiB/s (59.7MB/s)(1709MiB/30003msec) clat (usec): min=514, max=133418, avg=1839.06, stdev=1998.08 lat (usec): min=526, max=133430, avg=1851.82, stdev=1998.11 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 938], 10.00th=[ 1106], 20.00th=[ 1287], | 30.00th=[ 1434], 40.00th=[ 1549], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1844], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2638], | 99.00th=[ 9634], 99.50th=[10421], 99.90th=[35390], 99.95th=[38011], | 99.99th=[42730] bw ( KiB/s): min=33024, max=94976, per=100.00%, avg=58318.13, stdev=6746.23, samples=60 iops : min= 258, max= 742, avg=455.57, stdev=52.72, samples=60 lat (usec) : 50=2.02%, 100=75.48%, 250=2.24%, 500=0.06%, 750=0.50% lat (usec) : 1000=0.82% lat (msec) : 2=14.96%, 4=3.33%, 10=0.36%, 20=0.21%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.82%, sys=3.56%, ctx=68039, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54365,13668,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=226MiB/s (238MB/s), 226MiB/s-226MiB/s (238MB/s-238MB/s), io=6796MiB (7126MB), run=30003-30003msec WRITE: bw=56.9MiB/s (59.7MB/s), 56.9MiB/s-56.9MiB/s (59.7MB/s-59.7MB/s), io=1709MiB (1791MB), run=30003-30003msec Disk stats (read/write): vdj: ios=54075/13589, merge=0/0, ticks=867/17603, in_queue=28093, util=93.96% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933838: Fri Jan 7 14:04:47 2022 read: IOPS=1848, BW=231MiB/s (242MB/s)(6933MiB/30001msec) slat (usec): min=8, max=177, avg=14.50, stdev= 4.61 clat (nsec): min=903, max=23848k, avg=68648.72, stdev=504547.82 lat (usec): min=41, max=23870, avg=83.30, stdev=504.60 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 86], 99.50th=[ 96], 99.90th=[11076], 99.95th=[14746], | 99.99th=[19006] bw ( KiB/s): min=163328, max=388352, per=99.99%, avg=236610.78, stdev=31669.68, samples=59 iops : min= 1276, max= 3034, avg=1848.49, stdev=247.43, samples=59 write: IOPS=464, BW=58.1MiB/s (60.9MB/s)(1743MiB/30001msec) slat (usec): min=15, max=163, avg=27.12, stdev= 6.88 clat (usec): min=490, max=43208, avg=1785.56, stdev=1579.50 lat (usec): min=522, max=43233, avg=1812.86, stdev=1579.46 clat percentiles (usec): | 1.00th=[ 578], 5.00th=[ 783], 10.00th=[ 1020], 20.00th=[ 1352], | 30.00th=[ 1467], 40.00th=[ 1549], 50.00th=[ 1631], 60.00th=[ 1729], | 70.00th=[ 1844], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2540], | 99.00th=[ 8848], 99.50th=[10159], 99.90th=[35390], 99.95th=[40633], | 99.99th=[42730] bw ( KiB/s): min=44800, max=101376, per=99.96%, avg=59458.68, stdev=6642.63, samples=59 iops : min= 350, max= 792, avg=464.47, stdev=51.90, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=41.74% lat (usec) : 100=37.77%, 250=0.23%, 500=0.02%, 750=0.84%, 1000=1.10% lat (msec) : 2=14.20%, 4=3.63%, 10=0.20%, 20=0.18%, 50=0.02% cpu : usr=1.77%, sys=5.44%, ctx=69388, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55462,13942,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6933MiB (7270MB), run=30001-30001msec WRITE: bw=58.1MiB/s (60.9MB/s), 58.1MiB/s-58.1MiB/s (60.9MB/s-60.9MB/s), io=1743MiB (1827MB), run=30001-30001msec Disk stats (read/write): vdj: ios=55071/13839, merge=0/0, ticks=889/17103, in_queue=28067, util=93.96% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934422: Fri Jan 7 14:05:24 2022 read: IOPS=1832, BW=229MiB/s (240MB/s)(6874MiB/30001msec) clat (usec): min=37, max=65600, avg=81.61, stdev=602.57 lat (usec): min=38, max=65601, avg=81.75, stdev=602.58 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 106], 99.50th=[ 121], 99.90th=[11994], 99.95th=[16057], | 99.99th=[19006] bw ( KiB/s): min=178688, max=371456, per=99.93%, avg=234438.07, stdev=29068.22, samples=59 iops : min= 1396, max= 2902, avg=1831.51, stdev=227.09, samples=59 write: IOPS=460, BW=57.6MiB/s (60.4MB/s)(1727MiB/30001msec) clat (usec): min=510, max=145971, avg=1825.87, stdev=1964.00 lat (usec): min=522, max=145982, avg=1838.03, stdev=1963.89 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 799], 10.00th=[ 1074], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2606], | 99.00th=[ 9110], 99.50th=[10159], 99.90th=[36439], 99.95th=[38536], | 99.99th=[43254] bw ( KiB/s): min=44544, max=96768, per=99.92%, avg=58907.58, stdev=6358.58, samples=59 iops : min= 348, max= 756, avg=460.17, stdev=49.68, samples=59 lat (usec) : 50=2.89%, 100=75.92%, 250=0.98%, 500=0.04%, 750=0.78% lat (usec) : 1000=0.98% lat (msec) : 2=14.02%, 4=4.00%, 10=0.18%, 20=0.19%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.69%, sys=3.43%, ctx=68815, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54989,13818,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6874MiB (7208MB), run=30001-30001msec WRITE: bw=57.6MiB/s (60.4MB/s), 57.6MiB/s-57.6MiB/s (60.4MB/s-60.4MB/s), io=1727MiB (1811MB), run=30001-30001msec Disk stats (read/write): vdj: ios=54566/13712, merge=0/0, ticks=993/17281, in_queue=28210, util=94.43% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935725: Fri Jan 7 14:06:01 2022 read: IOPS=1891, BW=236MiB/s (248MB/s)(7098MiB/30023msec) slat (usec): min=5, max=104, avg=13.92, stdev= 4.18 clat (nsec): min=716, max=2496.5M, avg=1374146.96, stdev=50534251.22 lat (usec): min=36, max=2496.5k, avg=1388.22, stdev=50534.24 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 43], 40.00th=[ 44], | 50.00th=[ 46], 60.00th=[ 49], 70.00th=[ 53], | 80.00th=[ 59], 90.00th=[ 69], 95.00th=[ 77], | 99.00th=[ 102], 99.50th=[ 130], 99.90th=[ 123208], | 99.95th=[1019216], 99.99th=[2499806] bw ( KiB/s): min=180480, max=464128, per=100.00%, avg=242281.77, stdev=37189.14, samples=60 iops : min= 1410, max= 3626, avg=1892.82, stdev=290.54, samples=60 write: IOPS=476, BW=59.5MiB/s (62.4MB/s)(1787MiB/30023msec) slat (usec): min=13, max=153, avg=29.05, stdev= 9.25 clat (usec): min=2113, max=78595, avg=15451.38, stdev=6284.61 lat (usec): min=2127, max=78631, avg=15480.60, stdev=6285.82 clat percentiles (usec): | 1.00th=[ 3294], 5.00th=[ 7701], 10.00th=[10290], 20.00th=[11600], | 30.00th=[12649], 40.00th=[13698], 50.00th=[14746], 60.00th=[15926], | 70.00th=[17171], 80.00th=[18744], 90.00th=[20579], 95.00th=[23462], | 99.00th=[34341], 99.50th=[61604], 99.90th=[68682], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min=51456, max=116992, per=100.00%, avg=60946.82, stdev=8281.60, samples=60 iops : min= 402, max= 914, avg=476.13, stdev=64.71, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.02%, 50=50.41% lat (usec) : 100=28.56%, 250=0.67%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.35%, 10=1.44%, 20=15.74%, 50=2.45% lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.86%, sys=5.45%, ctx=70490, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=56783,14293,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=236MiB/s (248MB/s), 236MiB/s-236MiB/s (248MB/s-248MB/s), io=7098MiB (7443MB), run=30023-30023msec WRITE: bw=59.5MiB/s (62.4MB/s), 59.5MiB/s-59.5MiB/s (62.4MB/s-62.4MB/s), io=1787MiB (1873MB), run=30023-30023msec Disk stats (read/write): vdj: ios=56492/14191, merge=0/0, ticks=75124/211275, in_queue=296238, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936744: Fri Jan 7 14:06:38 2022 read: IOPS=1840, BW=230MiB/s (241MB/s)(6901MiB/30002msec) clat (usec): min=39, max=23265, avg=81.47, stdev=497.05 lat (usec): min=39, max=23265, avg=81.61, stdev=497.05 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 56], | 30.00th=[ 58], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 80], 95.00th=[ 86], | 99.00th=[ 104], 99.50th=[ 116], 99.90th=[11600], 99.95th=[15008], | 99.99th=[18220] bw ( KiB/s): min=145408, max=396032, per=100.00%, avg=235549.87, stdev=33287.31, samples=60 iops : min= 1136, max= 3094, avg=1840.23, stdev=260.06, samples=60 write: IOPS=462, BW=57.8MiB/s (60.6MB/s)(1734MiB/30002msec) clat (usec): min=512, max=100091, avg=1817.61, stdev=1910.60 lat (usec): min=523, max=100103, avg=1830.59, stdev=1910.49 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 807], 10.00th=[ 1090], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 9503], 99.50th=[10159], 99.90th=[36963], 99.95th=[42206], | 99.99th=[76022] bw ( KiB/s): min=34816, max=103168, per=100.00%, avg=59165.87, stdev=7410.01, samples=60 iops : min= 272, max= 806, avg=462.23, stdev=57.89, samples=60 lat (usec) : 50=1.70%, 100=77.19%, 250=0.92%, 500=0.01%, 750=0.77% lat (usec) : 1000=0.96% lat (msec) : 2=14.31%, 4=3.73%, 10=0.21%, 20=0.18%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.88%, sys=3.48%, ctx=69084, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55207,13868,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6901MiB (7236MB), run=30002-30002msec WRITE: bw=57.8MiB/s (60.6MB/s), 57.8MiB/s-57.8MiB/s (60.6MB/s-60.6MB/s), io=1734MiB (1818MB), run=30002-30002msec Disk stats (read/write): vdj: ios=54886/13795, merge=0/0, ticks=878/17302, in_queue=28051, util=93.66% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937333: Fri Jan 7 14:07:16 2022 read: IOPS=1873, BW=234MiB/s (246MB/s)(7054MiB/30124msec) slat (usec): min=3, max=381, avg=13.02, stdev= 8.93 clat (nsec): min=776, max=2528.7M, avg=1406048.31, stdev=50904645.36 lat (usec): min=34, max=2528.7k, avg=1419.25, stdev=50904.66 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 46], | 50.00th=[ 49], 60.00th=[ 53], 70.00th=[ 62], | 80.00th=[ 74], 90.00th=[ 85], 95.00th=[ 103], | 99.00th=[ 685], 99.50th=[ 1172], 99.90th=[ 156238], | 99.95th=[ 952108], 99.99th=[2499806] bw ( KiB/s): min=164864, max=499200, per=100.00%, avg=240790.60, stdev=41188.78, samples=60 iops : min= 1288, max= 3900, avg=1881.15, stdev=321.79, samples=60 write: IOPS=471, BW=58.9MiB/s (61.7MB/s)(1774MiB/30124msec) slat (usec): min=10, max=821, avg=37.68, stdev=24.27 clat (msec): min=2, max=2451, avg=100.44, stdev=66.56 lat (msec): min=2, max=2451, avg=100.48, stdev=66.56 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 75], 10.00th=[ 82], 20.00th=[ 87], | 30.00th=[ 89], 40.00th=[ 91], 50.00th=[ 93], 60.00th=[ 96], | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 114], 95.00th=[ 142], | 99.00th=[ 243], 99.50th=[ 558], 99.90th=[ 1011], 99.95th=[ 1116], | 99.99th=[ 2089] bw ( KiB/s): min=50533, max=115712, per=100.00%, avg=60351.12, stdev=8281.42, samples=60 iops : min= 394, max= 904, avg=471.43, stdev=64.72, samples=60 lat (nsec) : 1000=0.04% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.07%, 50=42.60% lat (usec) : 100=32.84%, 250=2.45%, 500=0.72%, 750=0.41%, 1000=0.19% lat (msec) : 2=0.35%, 4=0.08%, 10=0.01%, 20=0.32%, 50=0.22% lat (msec) : 100=13.99%, 250=5.37%, 500=0.10%, 750=0.06%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.03% cpu : usr=2.51%, sys=5.09%, ctx=67702, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=56431,14190,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=7054MiB (7397MB), run=30124-30124msec WRITE: bw=58.9MiB/s (61.7MB/s), 58.9MiB/s-58.9MiB/s (61.7MB/s-61.7MB/s), io=1774MiB (1860MB), run=30124-30124msec Disk stats (read/write): vdj: ios=56428/14155, merge=0/0, ticks=73424/1407311, in_queue=1499099, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938228: Fri Jan 7 14:07:52 2022 read: IOPS=1853, BW=232MiB/s (243MB/s)(6951MiB/30002msec) clat (usec): min=39, max=25163, avg=78.49, stdev=514.19 lat (usec): min=39, max=25163, avg=78.63, stdev=514.19 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 96], 99.50th=[ 106], 99.90th=[11469], 99.95th=[15533], | 99.99th=[19268] bw ( KiB/s): min=193280, max=406528, per=100.00%, avg=237268.20, stdev=30504.84, samples=60 iops : min= 1510, max= 3176, avg=1853.63, stdev=238.32, samples=60 write: IOPS=465, BW=58.2MiB/s (61.0MB/s)(1747MiB/30002msec) clat (usec): min=512, max=43604, avg=1814.45, stdev=1560.67 lat (usec): min=524, max=43617, avg=1826.68, stdev=1560.54 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 824], 10.00th=[ 1139], 20.00th=[ 1418], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2245], 95.00th=[ 2507], | 99.00th=[ 9372], 99.50th=[10159], 99.90th=[35914], 99.95th=[38536], | 99.99th=[42730] bw ( KiB/s): min=50688, max=105216, per=100.00%, avg=59617.87, stdev=7017.22, samples=60 iops : min= 396, max= 822, avg=465.73, stdev=54.82, samples=60 lat (usec) : 50=3.65%, 100=75.69%, 250=0.48%, 500=0.01%, 750=0.71% lat (usec) : 1000=0.83% lat (msec) : 2=14.44%, 4=3.81%, 10=0.17%, 20=0.19%, 50=0.02% cpu : usr=1.68%, sys=3.35%, ctx=69587, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55606,13973,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=6951MiB (7288MB), run=30002-30002msec WRITE: bw=58.2MiB/s (61.0MB/s), 58.2MiB/s-58.2MiB/s (61.0MB/s-61.0MB/s), io=1747MiB (1831MB), run=30002-30002msec Disk stats (read/write): vdj: ios=55177/13861, merge=0/0, ticks=903/17290, in_queue=28217, util=94.45% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938831: Fri Jan 7 14:08:30 2022 read: IOPS=1869, BW=234MiB/s (245MB/s)(7063MiB/30228msec) slat (usec): min=3, max=424, avg=11.76, stdev= 7.01 clat (nsec): min=742, max=2535.4M, avg=1567839.59, stdev=54150112.82 lat (usec): min=33, max=2535.4k, avg=1579.75, stdev=54150.12 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 43], 30.00th=[ 44], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 55], 70.00th=[ 65], | 80.00th=[ 76], 90.00th=[ 91], 95.00th=[ 176], | 99.00th=[ 2343], 99.50th=[ 3621], 99.90th=[ 156238], | 99.95th=[1451230], 99.99th=[2499806] bw ( KiB/s): min=183552, max=511744, per=100.00%, avg=241091.18, stdev=42432.67, samples=60 iops : min= 1434, max= 3998, avg=1883.50, stdev=331.51, samples=60 write: IOPS=470, BW=58.8MiB/s (61.6MB/s)(1776MiB/30228msec) slat (usec): min=12, max=434, avg=34.30, stdev=17.23 clat (msec): min=5, max=3019, avg=206.40, stdev=117.49 lat (msec): min=6, max=3019, avg=206.44, stdev=117.49 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 192], 50.00th=[ 197], 60.00th=[ 201], | 70.00th=[ 207], 80.00th=[ 218], 90.00th=[ 245], 95.00th=[ 268], | 99.00th=[ 567], 99.50th=[ 1028], 99.90th=[ 1770], 99.95th=[ 1972], | 99.99th=[ 2769] bw ( KiB/s): min=50176, max=105984, per=100.00%, avg=60215.18, stdev=7322.51, samples=60 iops : min= 392, max= 828, avg=470.42, stdev=57.22, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.09%, 4=0.01%, 10=0.01%, 20=0.06%, 50=41.06% lat (usec) : 100=32.44%, 250=2.75%, 500=0.82%, 750=0.50%, 1000=0.40% lat (msec) : 2=0.81%, 4=0.59%, 10=0.23%, 20=0.04%, 50=0.38% lat (msec) : 100=0.23%, 250=17.79%, 500=1.46%, 750=0.08%, 1000=0.04% lat (msec) : 2000=0.12%, >=2000=0.04% cpu : usr=2.39%, sys=4.66%, ctx=65825, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56506,14208,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=7063MiB (7406MB), run=30228-30228msec WRITE: bw=58.8MiB/s (61.6MB/s), 58.8MiB/s-58.8MiB/s (61.6MB/s-61.6MB/s), io=1776MiB (1862MB), run=30228-30228msec Disk stats (read/write): vdj: ios=56502/14121, merge=0/0, ticks=79658/2894342, in_queue=2998726, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939650: Fri Jan 7 14:09:06 2022 read: IOPS=1845, BW=231MiB/s (242MB/s)(6920MiB/30001msec) clat (usec): min=38, max=24559, avg=79.51, stdev=532.68 lat (usec): min=38, max=24559, avg=79.65, stdev=532.68 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 104], 99.50th=[ 118], 99.90th=[11469], 99.95th=[15664], | 99.99th=[20317] bw ( KiB/s): min=193536, max=376576, per=100.00%, avg=236183.86, stdev=28137.16, samples=59 iops : min= 1512, max= 2942, avg=1845.19, stdev=219.82, samples=59 write: IOPS=463, BW=57.0MiB/s (60.8MB/s)(1740MiB/30001msec) clat (usec): min=517, max=44224, avg=1818.44, stdev=1596.75 lat (usec): min=528, max=44237, avg=1830.58, stdev=1596.65 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 824], 10.00th=[ 1123], 20.00th=[ 1401], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2540], | 99.00th=[ 9241], 99.50th=[10028], 99.90th=[36963], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=50944, max=98304, per=99.95%, avg=59352.95, stdev=6333.95, samples=59 iops : min= 398, max= 768, avg=463.69, stdev=49.48, samples=59 lat (usec) : 50=4.27%, 100=74.66%, 250=0.86%, 500=0.02%, 750=0.76% lat (usec) : 1000=0.80% lat (msec) : 2=14.30%, 4=3.94%, 10=0.20%, 20=0.17%, 50=0.03% cpu : usr=1.65%, sys=3.44%, ctx=69283, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55356,13919,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6920MiB (7256MB), run=30001-30001msec WRITE: bw=57.0MiB/s (60.8MB/s), 57.0MiB/s-57.0MiB/s (60.8MB/s-60.8MB/s), io=1740MiB (1824MB), run=30001-30001msec Disk stats (read/write): vdj: ios=55024/13824, merge=0/0, ticks=941/17369, in_queue=28229, util=94.39% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940615: Fri Jan 7 14:09:44 2022 read: IOPS=1882, BW=235MiB/s (247MB/s)(7114MiB/30241msec) slat (usec): min=3, max=239215, avg=417.24, stdev=2536.71 clat (usec): min=1401, max=2591.5k, avg=34681.63, stdev=59355.10 lat (usec): min=1405, max=2591.5k, avg=35099.02, stdev=59424.95 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 8], 10.00th=[ 18], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 53], | 99.00th=[ 94], 99.50th=[ 247], 99.90th=[ 321], 99.95th=[ 1418], | 99.99th=[ 2534] bw ( KiB/s): min=190963, max=548977, per=100.00%, avg=245764.45, stdev=46359.21, samples=60 iops : min= 1491, max= 4288, avg=1919.55, stdev=362.16, samples=60 write: IOPS=473, BW=59.2MiB/s (62.1MB/s)(1790MiB/30241msec) slat (usec): min=13, max=244627, avg=426.45, stdev=2610.12 clat (msec): min=9, max=3542, avg=282.37, stdev=162.44 lat (msec): min=11, max=3542, avg=282.79, stdev=162.44 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 190], 10.00th=[ 218], 20.00th=[ 241], | 30.00th=[ 255], 40.00th=[ 264], 50.00th=[ 271], 60.00th=[ 275], | 70.00th=[ 284], 80.00th=[ 300], 90.00th=[ 326], 95.00th=[ 355], | 99.00th=[ 844], 99.50th=[ 1569], 99.90th=[ 2123], 99.95th=[ 3373], | 99.99th=[ 3540] bw ( KiB/s): min=50991, max=109123, per=100.00%, avg=61325.25, stdev=7554.28, samples=60 iops : min= 398, max= 852, avg=478.68, stdev=59.00, samples=60 lat (msec) : 2=0.20%, 4=1.39%, 10=3.17%, 20=5.14%, 50=65.45% lat (msec) : 100=4.38%, 250=5.05%, 500=14.74%, 750=0.16%, 1000=0.09% lat (msec) : 2000=0.16%, >=2000=0.05% cpu : usr=2.31%, sys=1.98%, ctx=7962, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56914,14318,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=7114MiB (7460MB), run=30241-30241msec WRITE: bw=59.2MiB/s (62.1MB/s), 59.2MiB/s-59.2MiB/s (62.1MB/s-62.1MB/s), io=1790MiB (1877MB), run=30241-30241msec Disk stats (read/write): vdj: ios=56885/14192, merge=0/0, ticks=99525/3525670, in_queue=3671143, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941236: Fri Jan 7 14:10:20 2022 read: IOPS=1231, BW=308MiB/s (323MB/s)(9238MiB/30001msec) clat (usec): min=57, max=20852, avg=104.05, stdev=380.98 lat (usec): min=57, max=20852, avg=104.21, stdev=380.99 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 73], 10.00th=[ 74], 20.00th=[ 77], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 96], 80.00th=[ 103], 90.00th=[ 114], 95.00th=[ 125], | 99.00th=[ 161], 99.50th=[ 190], 99.90th=[ 7111], 99.95th=[10552], | 99.99th=[15926] bw ( KiB/s): min=222720, max=559104, per=100.00%, avg=316650.31, stdev=50892.35, samples=59 iops : min= 870, max= 2184, avg=1236.92, stdev=198.80, samples=59 write: IOPS=309, BW=77.4MiB/s (81.2MB/s)(2324MiB/30001msec) clat (usec): min=861, max=61649, avg=2779.60, stdev=2626.65 lat (usec): min=886, max=61674, avg=2805.17, stdev=2626.59 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1106], 10.00th=[ 1369], 20.00th=[ 1663], | 30.00th=[ 1893], 40.00th=[ 2040], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2507], 80.00th=[ 2835], 90.00th=[ 4686], 95.00th=[ 7046], | 99.00th=[11600], 99.50th=[12911], 99.90th=[48497], 99.95th=[53740], | 99.99th=[61604] bw ( KiB/s): min=59904, max=149504, per=100.00%, avg=79542.24, stdev=12542.74, samples=59 iops : min= 234, max= 584, avg=310.71, stdev=49.00, samples=59 lat (usec) : 100=60.87%, 250=18.79%, 500=0.10%, 750=0.02%, 1000=0.53% lat (msec) : 2=6.96%, 4=9.73%, 10=2.30%, 20=0.68%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.57%, sys=2.92%, ctx=46252, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=36951,9294,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=308MiB/s (323MB/s), 308MiB/s-308MiB/s (323MB/s-323MB/s), io=9238MiB (9686MB), run=30001-30001msec WRITE: bw=77.4MiB/s (81.2MB/s), 77.4MiB/s-77.4MiB/s (81.2MB/s-81.2MB/s), io=2324MiB (2436MB), run=30001-30001msec Disk stats (read/write): vdj: ios=36794/9246, merge=0/0, ticks=441/20977, in_queue=28391, util=94.98% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942448: Fri Jan 7 14:10:57 2022 read: IOPS=1378, BW=345MiB/s (361MB/s)(10.1GiB/30002msec) slat (usec): min=10, max=115, avg=18.51, stdev= 4.92 clat (nsec): min=1039, max=71454k, avg=90227.51, stdev=659973.49 lat (usec): min=56, max=71470, avg=108.90, stdev=660.01 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 56], 10.00th=[ 57], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 78], 90.00th=[ 85], 95.00th=[ 92], | 99.00th=[ 112], 99.50th=[ 124], 99.90th=[10421], 99.95th=[16909], | 99.99th=[20579] bw ( KiB/s): min=237056, max=559104, per=100.00%, avg=353032.00, stdev=47683.31, samples=60 iops : min= 926, max= 2184, avg=1379.00, stdev=186.27, samples=60 write: IOPS=348, BW=87.1MiB/s (91.3MB/s)(2613MiB/30002msec) slat (usec): min=23, max=212, avg=43.56, stdev=12.27 clat (usec): min=823, max=137409, avg=2387.60, stdev=2623.44 lat (usec): min=863, max=137452, avg=2431.35, stdev=2623.29 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1057], 10.00th=[ 1188], 20.00th=[ 1663], | 30.00th=[ 1876], 40.00th=[ 1991], 50.00th=[ 2114], 60.00th=[ 2212], | 70.00th=[ 2343], 80.00th=[ 2540], 90.00th=[ 2933], 95.00th=[ 4178], | 99.00th=[10421], 99.50th=[11207], 99.90th=[42206], 99.95th=[53216], | 99.99th=[84411] bw ( KiB/s): min=52736, max=149504, per=100.00%, avg=89190.63, stdev=11096.62, samples=60 iops : min= 206, max= 584, avg=348.37, stdev=43.35, samples=60 lat (usec) : 2=0.01%, 50=0.37%, 100=77.49%, 250=1.83%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.60% lat (msec) : 2=7.77%, 4=10.65%, 10=0.82%, 20=0.42%, 50=0.02% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.75%, sys=4.79%, ctx=51831, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41367,10452,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=345MiB/s (361MB/s), 345MiB/s-345MiB/s (361MB/s-361MB/s), io=10.1GiB (10.8GB), run=30002-30002msec WRITE: bw=87.1MiB/s (91.3MB/s), 87.1MiB/s-87.1MiB/s (91.3MB/s-91.3MB/s), io=2613MiB (2740MB), run=30002-30002msec Disk stats (read/write): vdj: ios=41019/10362, merge=0/0, ticks=856/19888, in_queue=28121, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943470: Fri Jan 7 14:11:35 2022 read: IOPS=1380, BW=345MiB/s (362MB/s)(10.1GiB/30005msec) clat (usec): min=52, max=21520, avg=94.87, stdev=451.96 lat (usec): min=52, max=21521, avg=95.02, stdev=451.97 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 125], 99.50th=[ 145], 99.90th=[ 8979], 99.95th=[12256], | 99.99th=[16909] bw ( KiB/s): min=291328, max=598016, per=100.00%, avg=353343.88, stdev=45063.98, samples=60 iops : min= 1138, max= 2336, avg=1380.27, stdev=175.99, samples=60 write: IOPS=348, BW=87.2MiB/s (91.4MB/s)(2616MiB/30005msec) clat (usec): min=819, max=117843, avg=2458.83, stdev=2607.90 lat (usec): min=837, max=117856, avg=2483.20, stdev=2607.76 clat percentiles (usec): | 1.00th=[ 898], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1795], | 30.00th=[ 1909], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2606], 90.00th=[ 3032], 95.00th=[ 4293], | 99.00th=[10552], 99.50th=[12125], 99.90th=[52691], 99.95th=[53740], | 99.99th=[74974] bw ( KiB/s): min=73728, max=157696, per=100.00%, avg=89304.30, stdev=10641.81, samples=60 iops : min= 288, max= 616, avg=348.83, stdev=41.57, samples=60 lat (usec) : 100=74.55%, 250=5.11%, 500=0.03%, 750=0.01%, 1000=0.57% lat (msec) : 2=7.12%, 4=11.19%, 10=0.94%, 20=0.44%, 50=0.01% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.77%, sys=2.97%, ctx=51886, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41408,10465,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=345MiB/s (362MB/s), 345MiB/s-345MiB/s (362MB/s-362MB/s), io=10.1GiB (10.9GB), run=30005-30005msec WRITE: bw=87.2MiB/s (91.4MB/s), 87.2MiB/s-87.2MiB/s (91.4MB/s-91.4MB/s), io=2616MiB (2743MB), run=30005-30005msec Disk stats (read/write): vdj: ios=41076/10378, merge=0/0, ticks=635/20399, in_queue=28283, util=94.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943999: Fri Jan 7 14:12:11 2022 read: IOPS=1448, BW=362MiB/s (380MB/s)(10.6GiB/30030msec) slat (usec): min=4, max=172, avg=13.19, stdev= 5.06 clat (nsec): min=884, max=1869.3M, avg=328770.27, stdev=15524055.19 lat (usec): min=46, max=1869.3k, avg=342.10, stdev=15524.08 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 50], 20.00th=[ 52], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 64], | 70.00th=[ 72], 80.00th=[ 88], 90.00th=[ 111], 95.00th=[ 127], | 99.00th=[ 194], 99.50th=[ 318], 99.90th=[ 30540], 99.95th=[ 41681], | 99.99th=[734004] bw ( KiB/s): min=292352, max=740352, per=100.00%, avg=371123.20, stdev=58840.24, samples=60 iops : min= 1142, max= 2892, avg=1449.70, stdev=229.84, samples=60 write: IOPS=365, BW=91.4MiB/s (95.8MB/s)(2744MiB/30030msec) slat (usec): min=16, max=179, avg=46.70, stdev=18.37 clat (usec): min=2621, max=94774, avg=25947.20, stdev=10636.08 lat (usec): min=2653, max=94807, avg=25994.07, stdev=10635.94 clat percentiles (usec): | 1.00th=[ 5604], 5.00th=[ 8848], 10.00th=[12256], 20.00th=[17171], | 30.00th=[22414], 40.00th=[24511], 50.00th=[26084], 60.00th=[27657], | 70.00th=[30016], 80.00th=[32900], 90.00th=[36439], 95.00th=[40633], | 99.00th=[63177], 99.50th=[79168], 99.90th=[90702], 99.95th=[91751], | 99.99th=[94897] bw ( KiB/s): min=77824, max=187392, per=100.00%, avg=93576.53, stdev=13545.05, samples=60 iops : min= 304, max= 732, avg=365.53, stdev=52.91, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=8.21%, 100=59.86% lat (usec) : 250=11.20%, 500=0.36%, 750=0.05%, 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=1.29%, 20=3.66%, 50=14.96% lat (msec) : 100=0.32%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=2.23%, sys=3.85%, ctx=51207, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=43491,10976,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=362MiB/s (380MB/s), 362MiB/s-362MiB/s (380MB/s-380MB/s), io=10.6GiB (11.4GB), run=30030-30030msec WRITE: bw=91.4MiB/s (95.8MB/s), 91.4MiB/s-91.4MiB/s (95.8MB/s-95.8MB/s), io=2744MiB (2877MB), run=30030-30030msec Disk stats (read/write): vdj: ios=43491/10973, merge=0/0, ticks=11170/279133, in_queue=298682, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944976: Fri Jan 7 14:12:48 2022 read: IOPS=1369, BW=342MiB/s (359MB/s)(10.0GiB/30002msec) clat (usec): min=51, max=47748, avg=97.84, stdev=532.21 lat (usec): min=51, max=47749, avg=97.98, stdev=532.21 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 128], 99.50th=[ 145], 99.90th=[ 8717], 99.95th=[12780], | 99.99th=[19792] bw ( KiB/s): min=266240, max=598016, per=100.00%, avg=350549.33, stdev=47835.17, samples=60 iops : min= 1040, max= 2336, avg=1369.33, stdev=186.86, samples=60 write: IOPS=345, BW=86.5MiB/s (90.7MB/s)(2595MiB/30002msec) clat (usec): min=801, max=101534, avg=2469.19, stdev=2386.30 lat (usec): min=814, max=101549, avg=2494.53, stdev=2385.89 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1074], 10.00th=[ 1205], 20.00th=[ 1795], | 30.00th=[ 1909], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2606], 90.00th=[ 3032], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[10945], 99.90th=[42206], 99.95th=[53216], | 99.99th=[56886] bw ( KiB/s): min=60416, max=157696, per=100.00%, avg=88558.93, stdev=11190.38, samples=60 iops : min= 236, max= 616, avg=345.93, stdev=43.71, samples=60 lat (usec) : 100=73.69%, 250=5.97%, 500=0.04%, 750=0.01%, 1000=0.59% lat (msec) : 2=7.03%, 4=11.20%, 10=0.96%, 20=0.47%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.86%, sys=3.03%, ctx=51469, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41080,10379,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=342MiB/s (359MB/s), 342MiB/s-342MiB/s (359MB/s-359MB/s), io=10.0GiB (10.8GB), run=30002-30002msec WRITE: bw=86.5MiB/s (90.7MB/s), 86.5MiB/s-86.5MiB/s (90.7MB/s-90.7MB/s), io=2595MiB (2721MB), run=30002-30002msec Disk stats (read/write): vdj: ios=41059/10373, merge=0/0, ticks=709/20394, in_queue=28400, util=94.40% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945603: Fri Jan 7 14:13:26 2022 read: IOPS=1430, BW=358MiB/s (375MB/s)(10.5GiB/30143msec) slat (usec): min=4, max=231, avg=13.63, stdev= 5.68 clat (nsec): min=961, max=2541.8M, avg=1880029.19, stdev=56311876.42 lat (usec): min=44, max=2541.8k, avg=1893.81, stdev=56311.85 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 55], 40.00th=[ 58], | 50.00th=[ 62], 60.00th=[ 68], 70.00th=[ 86], | 80.00th=[ 112], 90.00th=[ 128], 95.00th=[ 159], | 99.00th=[ 1123], 99.50th=[ 2835], 99.90th=[ 476054], | 99.95th=[1451230], 99.99th=[2499806] bw ( KiB/s): min=291328, max=772608, per=100.00%, avg=367871.72, stdev=64740.24, samples=60 iops : min= 1138, max= 3018, avg=1436.97, stdev=252.89, samples=60 write: IOPS=360, BW=90.2MiB/s (94.6MB/s)(2718MiB/30143msec) slat (usec): min=18, max=273, avg=53.78, stdev=24.50 clat (msec): min=14, max=3294, avg=131.03, stdev=70.97 lat (msec): min=15, max=3294, avg=131.08, stdev=70.97 clat percentiles (msec): | 1.00th=[ 36], 5.00th=[ 92], 10.00th=[ 103], 20.00th=[ 111], | 30.00th=[ 116], 40.00th=[ 122], 50.00th=[ 126], 60.00th=[ 131], | 70.00th=[ 138], 80.00th=[ 144], 90.00th=[ 159], 95.00th=[ 180], | 99.00th=[ 234], 99.50th=[ 435], 99.90th=[ 1036], 99.95th=[ 1301], | 99.99th=[ 2735] bw ( KiB/s): min=77312, max=176128, per=100.00%, avg=92411.17, stdev=12040.98, samples=60 iops : min= 302, max= 688, avg=360.93, stdev=47.04, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=6.52%, 100=51.91%, 250=19.17% lat (usec) : 500=0.68%, 750=0.42%, 1000=0.28% lat (msec) : 2=0.30%, 4=0.39%, 10=0.05%, 20=0.05%, 50=0.49% lat (msec) : 100=1.16%, 250=18.29%, 500=0.10%, 750=0.05%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.03% cpu : usr=2.40%, sys=3.96%, ctx=49258, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=43109,10873,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=358MiB/s (375MB/s), 358MiB/s-358MiB/s (375MB/s-375MB/s), io=10.5GiB (11.3GB), run=30143-30143msec WRITE: bw=90.2MiB/s (94.6MB/s), 90.2MiB/s-90.2MiB/s (94.6MB/s-94.6MB/s), io=2718MiB (2850MB), run=30143-30143msec Disk stats (read/write): vdj: ios=43104/10839, merge=0/0, ticks=71376/1411698, in_queue=1499087, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946485: Fri Jan 7 14:14:02 2022 read: IOPS=1372, BW=343MiB/s (360MB/s)(10.1GiB/30001msec) clat (usec): min=51, max=17940, avg=92.28, stdev=397.53 lat (usec): min=51, max=17940, avg=92.42, stdev=397.53 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 66], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 124], 99.50th=[ 141], 99.90th=[ 8455], 99.95th=[11207], | 99.99th=[13829] bw ( KiB/s): min=276480, max=595968, per=100.00%, avg=351515.63, stdev=43793.17, samples=60 iops : min= 1080, max= 2328, avg=1373.10, stdev=171.08, samples=60 write: IOPS=346, BW=86.7MiB/s (90.9MB/s)(2601MiB/30001msec) clat (usec): min=817, max=55260, avg=2484.25, stdev=2231.22 lat (usec): min=839, max=55285, avg=2510.04, stdev=2231.24 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1074], 10.00th=[ 1205], 20.00th=[ 1811], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2638], 90.00th=[ 3130], 95.00th=[ 4424], | 99.00th=[10552], 99.50th=[11207], 99.90th=[15664], 99.95th=[53216], | 99.99th=[54789] bw ( KiB/s): min=72704, max=156160, per=100.00%, avg=88769.58, stdev=10374.42, samples=60 iops : min= 284, max= 610, avg=346.73, stdev=40.53, samples=60 lat (usec) : 100=75.28%, 250=4.40%, 500=0.02%, 750=0.01%, 1000=0.54% lat (msec) : 2=6.86%, 4=11.33%, 10=1.06%, 20=0.48%, 100=0.02% cpu : usr=1.71%, sys=3.02%, ctx=51604, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41191,10403,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=343MiB/s (360MB/s), 343MiB/s-343MiB/s (360MB/s-360MB/s), io=10.1GiB (10.8GB), run=30001-30001msec WRITE: bw=86.7MiB/s (90.9MB/s), 86.7MiB/s-86.7MiB/s (90.9MB/s-90.9MB/s), io=2601MiB (2727MB), run=30001-30001msec Disk stats (read/write): vdj: ios=40972/10347, merge=0/0, ticks=563/20600, in_queue=28274, util=94.43% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947446: Fri Jan 7 14:14:40 2022 read: IOPS=1429, BW=357MiB/s (375MB/s)(10.6GiB/30314msec) slat (usec): min=4, max=380, avg=12.65, stdev= 6.33 clat (nsec): min=911, max=2604.9M, avg=2071550.55, stdev=58666261.24 lat (usec): min=44, max=2604.9k, avg=2084.35, stdev=58666.20 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 55], 40.00th=[ 58], | 50.00th=[ 63], 60.00th=[ 71], 70.00th=[ 100], | 80.00th=[ 117], 90.00th=[ 149], 95.00th=[ 523], | 99.00th=[ 5866], 99.50th=[ 7570], 99.90th=[ 675283], | 99.95th=[1719665], 99.99th=[2533360] bw ( KiB/s): min=268288, max=909312, per=100.00%, avg=369865.78, stdev=81566.45, samples=60 iops : min= 1048, max= 3552, avg=1444.75, stdev=318.63, samples=60 write: IOPS=360, BW=90.1MiB/s (94.5MB/s)(2732MiB/30314msec) slat (usec): min=17, max=340, avg=53.02, stdev=25.23 clat (msec): min=25, max=2774, avg=268.99, stdev=90.80 lat (msec): min=25, max=2774, avg=269.05, stdev=90.80 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 159], 10.00th=[ 228], 20.00th=[ 245], | 30.00th=[ 253], 40.00th=[ 259], 50.00th=[ 266], 60.00th=[ 271], | 70.00th=[ 279], 80.00th=[ 292], 90.00th=[ 317], 95.00th=[ 355], | 99.00th=[ 542], 99.50th=[ 617], 99.90th=[ 1469], 99.95th=[ 1670], | 99.99th=[ 2299] bw ( KiB/s): min=77312, max=181248, per=100.00%, avg=92453.38, stdev=13151.65, samples=60 iops : min= 302, max= 708, avg=361.10, stdev=51.39, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=6.23% lat (usec) : 100=49.67%, 250=18.61%, 500=1.25%, 750=0.61%, 1000=0.46% lat (msec) : 2=1.05%, 4=0.79%, 10=1.01%, 20=0.04%, 50=0.10% lat (msec) : 100=0.62%, 250=4.74%, 500=14.39%, 750=0.28%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.04% cpu : usr=2.48%, sys=3.59%, ctx=47276, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=43342,10929,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=357MiB/s (375MB/s), 357MiB/s-357MiB/s (375MB/s-375MB/s), io=10.6GiB (11.4GB), run=30314-30314msec WRITE: bw=90.1MiB/s (94.5MB/s), 90.1MiB/s-90.1MiB/s (94.5MB/s-94.5MB/s), io=2732MiB (2865MB), run=30314-30314msec Disk stats (read/write): vdj: ios=43339/10929, merge=0/0, ticks=78500/2916693, in_queue=3010217, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948019: Fri Jan 7 14:15:17 2022 read: IOPS=1377, BW=344MiB/s (361MB/s)(10.1GiB/30001msec) clat (usec): min=49, max=23210, avg=91.09, stdev=441.59 lat (usec): min=49, max=23210, avg=91.23, stdev=441.60 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 61], 10.00th=[ 62], 20.00th=[ 64], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 75], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 92], 95.00th=[ 99], | 99.00th=[ 129], 99.50th=[ 155], 99.90th=[ 8717], 99.95th=[11994], | 99.99th=[16188] bw ( KiB/s): min=280576, max=576512, per=100.00%, avg=352661.57, stdev=41807.59, samples=60 iops : min= 1096, max= 2252, avg=1377.58, stdev=163.31, samples=60 write: IOPS=347, BW=86.0MiB/s (91.2MB/s)(2609MiB/30001msec) clat (usec): min=811, max=55346, avg=2481.14, stdev=2267.58 lat (usec): min=824, max=55371, avg=2505.37, stdev=2267.42 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1074], 10.00th=[ 1221], 20.00th=[ 1811], | 30.00th=[ 1942], 40.00th=[ 2057], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2606], 90.00th=[ 3064], 95.00th=[ 4490], | 99.00th=[10552], 99.50th=[10945], 99.90th=[52167], 99.95th=[53216], | 99.99th=[54789] bw ( KiB/s): min=73728, max=153088, per=99.99%, avg=89051.08, stdev=9749.44, samples=60 iops : min= 288, max= 598, avg=347.85, stdev=38.09, samples=60 lat (usec) : 50=0.01%, 100=76.08%, 250=3.55%, 500=0.05%, 750=0.02% lat (usec) : 1000=0.57% lat (msec) : 2=6.60%, 4=11.58%, 10=1.03%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.75%, sys=2.51%, ctx=51775, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41329,10437,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=344MiB/s (361MB/s), 344MiB/s-344MiB/s (361MB/s-361MB/s), io=10.1GiB (10.8GB), run=30001-30001msec WRITE: bw=86.0MiB/s (91.2MB/s), 86.0MiB/s-86.0MiB/s (91.2MB/s-91.2MB/s), io=2609MiB (2736MB), run=30001-30001msec Disk stats (read/write): vdj: ios=41329/10436, merge=0/0, ticks=620/20798, in_queue=28656, util=95.13% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949372: Fri Jan 7 14:15:54 2022 read: IOPS=1424, BW=356MiB/s (374MB/s)(10.6GiB/30352msec) slat (usec): min=4, max=366600, avg=539.93, stdev=4581.92 clat (msec): min=2, max=2604, avg=46.24, stdev=69.97 lat (msec): min=2, max=2604, avg=46.78, stdev=70.11 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 15], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 37], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 49], 80.00th=[ 55], 90.00th=[ 67], 95.00th=[ 78], | 99.00th=[ 268], 99.50th=[ 347], 99.90th=[ 659], 99.95th=[ 1871], | 99.99th=[ 2601] bw ( KiB/s): min=203264, max=903680, per=100.00%, avg=368443.05, stdev=81519.73, samples=60 iops : min= 794, max= 3530, avg=1439.20, stdev=318.43, samples=60 write: IOPS=359, BW=89.8MiB/s (94.2MB/s)(2726MiB/30352msec) slat (usec): min=19, max=315344, avg=599.32, stdev=4249.99 clat (msec): min=11, max=2242, avg=370.35, stdev=112.85 lat (msec): min=11, max=2243, avg=370.95, stdev=112.84 clat percentiles (msec): | 1.00th=[ 77], 5.00th=[ 218], 10.00th=[ 313], 20.00th=[ 338], | 30.00th=[ 347], 40.00th=[ 359], 50.00th=[ 368], 60.00th=[ 376], | 70.00th=[ 384], 80.00th=[ 401], 90.00th=[ 426], 95.00th=[ 510], | 99.00th=[ 718], 99.50th=[ 986], 99.90th=[ 1519], 99.95th=[ 1938], | 99.99th=[ 2232] bw ( KiB/s): min=73728, max=168960, per=99.99%, avg=91969.95, stdev=11997.30, samples=60 iops : min= 288, max= 660, avg=359.23, stdev=46.86, samples=60 lat (msec) : 4=0.08%, 10=5.21%, 20=5.10%, 50=48.21%, 100=20.36% lat (msec) : 250=1.28%, 500=18.61%, 750=0.90%, 1000=0.09%, 2000=0.11% lat (msec) : >=2000=0.04% cpu : usr=3.50%, sys=1.68%, ctx=5774, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=43246,10905,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=356MiB/s (374MB/s), 356MiB/s-356MiB/s (374MB/s-374MB/s), io=10.6GiB (11.3GB), run=30352-30352msec WRITE: bw=89.8MiB/s (94.2MB/s), 89.8MiB/s-89.8MiB/s (94.2MB/s-94.2MB/s), io=2726MiB (2859MB), run=30352-30352msec Disk stats (read/write): vdj: ios=43241/10884, merge=0/0, ticks=98718/3558634, in_queue=3699806, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950055: Fri Jan 7 14:16:31 2022 read: IOPS=950, BW=475MiB/s (498MB/s)(13.9GiB/30001msec) clat (usec): min=78, max=21136, avg=135.50, stdev=547.06 lat (usec): min=78, max=21136, avg=135.66, stdev=547.06 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 92], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 100], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 116], 80.00th=[ 125], 90.00th=[ 137], 95.00th=[ 151], | 99.00th=[ 198], 99.50th=[ 229], 99.90th=[11600], 99.95th=[13566], | 99.99th=[16712] bw ( KiB/s): min=382976, max=642048, per=100.00%, avg=486931.02, stdev=58524.63, samples=59 iops : min= 748, max= 1254, avg=951.00, stdev=114.33, samples=59 write: IOPS=239, BW=120MiB/s (126MB/s)(3597MiB/30001msec) clat (usec): min=1390, max=63253, avg=3574.32, stdev=2439.78 lat (usec): min=1434, max=63304, avg=3624.04, stdev=2439.45 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1696], 10.00th=[ 1991], 20.00th=[ 2474], | 30.00th=[ 2802], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3818], 90.00th=[ 5735], 95.00th=[ 7242], | 99.00th=[11469], 99.50th=[12649], 99.90th=[24511], 99.95th=[54264], | 99.99th=[63177] bw ( KiB/s): min=87040, max=183296, per=100.00%, avg=122926.19, stdev=13949.28, samples=59 iops : min= 170, max= 358, avg=240.07, stdev=27.26, samples=59 lat (usec) : 100=32.26%, 250=47.29%, 500=0.10%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.07%, 4=14.71%, 10=2.99%, 20=0.54%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.90%, sys=2.62%, ctx=35715, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28509,7194,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=475MiB/s (498MB/s), 475MiB/s-475MiB/s (498MB/s-498MB/s), io=13.9GiB (14.9GB), run=30001-30001msec WRITE: bw=120MiB/s (126MB/s), 120MiB/s-120MiB/s (126MB/s-126MB/s), io=3597MiB (3772MB), run=30001-30001msec Disk stats (read/write): vdj: ios=28334/7147, merge=0/0, ticks=666/21790, in_queue=28341, util=94.88% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950872: Fri Jan 7 14:17:08 2022 read: IOPS=970, BW=485MiB/s (509MB/s)(14.2GiB/30002msec) slat (usec): min=17, max=127, avg=26.02, stdev= 5.94 clat (usec): min=36, max=23362, avg=116.92, stdev=546.72 lat (usec): min=84, max=23383, avg=143.11, stdev=546.79 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 76], 10.00th=[ 78], 20.00th=[ 80], | 30.00th=[ 82], 40.00th=[ 85], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 98], 80.00th=[ 105], 90.00th=[ 115], 95.00th=[ 125], | 99.00th=[ 157], 99.50th=[ 190], 99.90th=[11600], 99.95th=[14091], | 99.99th=[16909] bw ( KiB/s): min=378880, max=658432, per=100.00%, avg=496710.52, stdev=59089.76, samples=60 iops : min= 740, max= 1286, avg=970.13, stdev=115.40, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3677MiB/30002msec) slat (usec): min=38, max=255, avg=72.11, stdev=15.13 clat (usec): min=1399, max=55010, avg=3432.69, stdev=2345.33 lat (usec): min=1490, max=55056, avg=3505.02, stdev=2344.77 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1631], 10.00th=[ 1696], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3785], 90.00th=[ 4883], 95.00th=[ 6718], | 99.00th=[11469], 99.50th=[12256], 99.90th=[21627], 99.95th=[53740], | 99.99th=[54789] bw ( KiB/s): min=101376, max=191488, per=100.00%, avg=125495.67, stdev=12286.08, samples=60 iops : min= 198, max= 374, avg=245.10, stdev=23.99, samples=60 lat (usec) : 50=0.02%, 100=58.58%, 250=21.01%, 500=0.03%, 750=0.01% lat (msec) : 2=3.14%, 4=13.76%, 10=2.92%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=4.18%, ctx=36476, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29103,7354,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=485MiB/s (509MB/s), 485MiB/s-485MiB/s (509MB/s-509MB/s), io=14.2GiB (15.3GB), run=30002-30002msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3677MiB (3856MB), run=30002-30002msec Disk stats (read/write): vdj: ios=28902/7300, merge=0/0, ticks=680/21294, in_queue=28229, util=94.51% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951822: Fri Jan 7 14:17:45 2022 read: IOPS=978, BW=489MiB/s (513MB/s)(14.3GiB/30003msec) clat (usec): min=73, max=21623, avg=130.21, stdev=550.60 lat (usec): min=73, max=21623, avg=130.36, stdev=550.61 clat percentiles (usec): | 1.00th=[ 84], 5.00th=[ 88], 10.00th=[ 89], 20.00th=[ 92], | 30.00th=[ 94], 40.00th=[ 97], 50.00th=[ 100], 60.00th=[ 105], | 70.00th=[ 113], 80.00th=[ 120], 90.00th=[ 130], 95.00th=[ 139], | 99.00th=[ 167], 99.50th=[ 188], 99.90th=[11600], 99.95th=[13435], | 99.99th=[17957] bw ( KiB/s): min=393216, max=658432, per=100.00%, avg=500974.93, stdev=54632.94, samples=60 iops : min= 768, max= 1286, avg=978.47, stdev=106.70, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3707MiB/30003msec) clat (usec): min=1360, max=61964, avg=3473.63, stdev=2390.44 lat (usec): min=1386, max=62016, avg=3522.79, stdev=2389.81 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3884], 90.00th=[ 4752], 95.00th=[ 6587], | 99.00th=[11338], 99.50th=[12125], 99.90th=[52691], 99.95th=[54264], | 99.99th=[62129] bw ( KiB/s): min=110592, max=190464, per=100.00%, avg=126498.13, stdev=10808.20, samples=60 iops : min= 216, max= 372, avg=247.07, stdev=21.11, samples=60 lat (usec) : 100=39.59%, 250=40.04%, 500=0.02%, 750=0.01% lat (msec) : 2=2.98%, 4=13.69%, 10=3.23%, 20=0.41%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=2.32%, ctx=36773, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29354,7413,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=489MiB/s (513MB/s), 489MiB/s-489MiB/s (513MB/s-513MB/s), io=14.3GiB (15.4GB), run=30003-30003msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3707MiB (3887MB), run=30003-30003msec Disk stats (read/write): vdj: ios=29103/7353, merge=0/0, ticks=693/21622, in_queue=28380, util=95.01% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952404: Fri Jan 7 14:18:22 2022 read: IOPS=997, BW=499MiB/s (523MB/s)(14.6GiB/30029msec) slat (usec): min=5, max=227, avg=17.23, stdev= 7.33 clat (nsec): min=1617, max=71531k, avg=183944.80, stdev=1753608.59 lat (usec): min=69, max=71542, avg=201.34, stdev=1753.67 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 75], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 88], | 70.00th=[ 100], 80.00th=[ 161], 90.00th=[ 192], 95.00th=[ 210], | 99.00th=[ 302], 99.50th=[ 506], 99.90th=[36439], 99.95th=[42206], | 99.99th=[69731] bw ( KiB/s): min=393216, max=789504, per=100.00%, avg=511131.47, stdev=67806.31, samples=60 iops : min= 768, max= 1542, avg=998.30, stdev=132.43, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3780MiB/30029msec) slat (usec): min=34, max=481, avg=94.29, stdev=46.63 clat (msec): min=2, max=132, avg=38.82, stdev=16.06 lat (msec): min=2, max=132, avg=38.91, stdev=16.06 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 17], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 46], 80.00th=[ 52], 90.00th=[ 58], 95.00th=[ 65], | 99.00th=[ 88], 99.50th=[ 97], 99.90th=[ 114], 99.95th=[ 121], | 99.99th=[ 132] bw ( KiB/s): min=106496, max=216064, per=99.98%, avg=128878.87, stdev=13723.47, samples=60 iops : min= 208, max= 422, avg=251.70, stdev=26.80, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 50=0.07%, 100=55.99%, 250=22.36% lat (usec) : 500=1.02%, 750=0.10%, 1000=0.06% lat (msec) : 2=0.07%, 4=0.01%, 10=0.64%, 20=1.86%, 50=13.45% lat (msec) : 100=4.29%, 250=0.08% cpu : usr=2.76%, sys=3.08%, ctx=36281, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29947,7560,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=499MiB/s (523MB/s), 499MiB/s-499MiB/s (523MB/s-523MB/s), io=14.6GiB (15.7GB), run=30029-30029msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3780MiB (3964MB), run=30029-30029msec Disk stats (read/write): vdj: ios=29809/7509, merge=0/0, ticks=2265/287462, in_queue=296619, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953233: Fri Jan 7 14:18:59 2022 read: IOPS=981, BW=491MiB/s (514MB/s)(14.4GiB/30004msec) clat (usec): min=78, max=25206, avg=132.92, stdev=571.83 lat (usec): min=78, max=25206, avg=133.07, stdev=571.83 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 97], 40.00th=[ 99], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 122], 90.00th=[ 131], 95.00th=[ 139], | 99.00th=[ 165], 99.50th=[ 188], 99.90th=[11994], 99.95th=[14615], | 99.99th=[18744] bw ( KiB/s): min=376832, max=653312, per=100.00%, avg=502424.22, stdev=60431.53, samples=60 iops : min= 736, max= 1276, avg=981.28, stdev=118.04, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3715MiB/30004msec) clat (usec): min=1356, max=54499, avg=3455.33, stdev=2290.04 lat (usec): min=1387, max=54526, avg=3503.21, stdev=2289.51 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3818], 90.00th=[ 4621], 95.00th=[ 6718], | 99.00th=[11469], 99.50th=[11994], 99.90th=[25822], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min=106496, max=186368, per=100.00%, avg=126775.12, stdev=10636.39, samples=60 iops : min= 208, max= 364, avg=247.60, stdev=20.78, samples=60 lat (usec) : 100=33.73%, 250=45.88%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.94%, 4=13.99%, 10=2.91%, 20=0.47%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.84%, sys=2.75%, ctx=36874, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29438,7429,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=491MiB/s (514MB/s), 491MiB/s-491MiB/s (514MB/s-514MB/s), io=14.4GiB (15.4GB), run=30004-30004msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3715MiB (3895MB), run=30004-30004msec Disk stats (read/write): vdj: ios=29162/7366, merge=0/0, ticks=713/21547, in_queue=28378, util=95.00% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954260: Fri Jan 7 14:19:36 2022 read: IOPS=998, BW=499MiB/s (523MB/s)(14.7GiB/30205msec) slat (usec): min=5, max=246, avg=14.95, stdev= 6.18 clat (nsec): min=1834, max=505364k, avg=401963.10, stdev=5745042.90 lat (usec): min=70, max=505378, avg=417.07, stdev=5745.05 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 74], 40.00th=[ 76], 50.00th=[ 80], 60.00th=[ 87], | 70.00th=[ 105], 80.00th=[ 184], 90.00th=[ 202], 95.00th=[ 273], | 99.00th=[ 3687], 99.50th=[ 6587], 99.90th=[ 89654], 99.95th=[115868], | 99.99th=[170918] bw ( KiB/s): min=348160, max=966656, per=100.00%, avg=514519.68, stdev=80449.68, samples=60 iops : min= 680, max= 1888, avg=1004.88, stdev=157.16, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3803MiB/30205msec) slat (usec): min=35, max=560, avg=92.35, stdev=46.15 clat (msec): min=7, max=2732, avg=196.80, stdev=108.59 lat (msec): min=7, max=2732, avg=196.89, stdev=108.59 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 71], 10.00th=[ 84], 20.00th=[ 105], | 30.00th=[ 130], 40.00th=[ 161], 50.00th=[ 199], 60.00th=[ 226], | 70.00th=[ 245], 80.00th=[ 268], 90.00th=[ 300], 95.00th=[ 334], | 99.00th=[ 443], 99.50th=[ 535], 99.90th=[ 1267], 99.95th=[ 1636], | 99.99th=[ 2735] bw ( KiB/s): min=105472, max=217088, per=100.00%, avg=128951.78, stdev=13287.62, samples=60 iops : min= 206, max= 424, avg=251.83, stdev=25.95, samples=60 lat (usec) : 2=0.01%, 50=0.03%, 100=54.59%, 250=20.85%, 500=1.48% lat (usec) : 750=0.54%, 1000=0.39% lat (msec) : 2=0.78%, 4=0.46%, 10=0.58%, 20=0.02%, 50=0.17% lat (msec) : 100=3.58%, 250=11.04%, 500=5.39%, 750=0.04%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.78%, sys=2.69%, ctx=35301, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=30145,7605,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=499MiB/s (523MB/s), 499MiB/s-499MiB/s (523MB/s-523MB/s), io=14.7GiB (15.8GB), run=30205-30205msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3803MiB (3987MB), run=30205-30205msec Disk stats (read/write): vdj: ios=30145/7559, merge=0/0, ticks=7980/1476177, in_queue=1499096, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954784: Fri Jan 7 14:20:13 2022 read: IOPS=976, BW=488MiB/s (512MB/s)(14.3GiB/30002msec) clat (usec): min=78, max=16687, avg=135.71, stdev=543.12 lat (usec): min=79, max=16687, avg=135.86, stdev=543.14 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 100], 40.00th=[ 102], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 118], 80.00th=[ 126], 90.00th=[ 137], 95.00th=[ 147], | 99.00th=[ 182], 99.50th=[ 204], 99.90th=[11863], 99.95th=[13698], | 99.99th=[16319] bw ( KiB/s): min=353280, max=612352, per=100.00%, avg=500105.42, stdev=52160.27, samples=60 iops : min= 690, max= 1196, avg=976.77, stdev=101.88, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3699MiB/30002msec) clat (usec): min=1377, max=57934, avg=3457.80, stdev=2385.48 lat (usec): min=1419, max=58078, avg=3508.83, stdev=2385.08 clat percentiles (usec): | 1.00th=[ 1450], 5.00th=[ 1663], 10.00th=[ 1713], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3163], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3851], 90.00th=[ 4752], 95.00th=[ 6652], | 99.00th=[11338], 99.50th=[11994], 99.90th=[53216], 99.95th=[54264], | 99.99th=[57934] bw ( KiB/s): min=108544, max=177152, per=99.99%, avg=126238.13, stdev=9963.06, samples=60 iops : min= 212, max= 346, avg=246.55, stdev=19.47, samples=60 lat (usec) : 100=25.78%, 250=53.80%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2=3.08%, 4=13.65%, 10=3.13%, 20=0.44%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.97%, sys=2.80%, ctx=36710, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29304,7398,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30002-30002msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3699MiB (3879MB), run=30002-30002msec Disk stats (read/write): vdj: ios=29257/7389, merge=0/0, ticks=676/21559, in_queue=28447, util=94.55% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956010: Fri Jan 7 14:20:50 2022 read: IOPS=988, BW=494MiB/s (518MB/s)(14.7GiB/30394msec) slat (usec): min=5, max=426, avg=14.41, stdev= 8.66 clat (nsec): min=1108, max=478778k, avg=899947.86, stdev=7309431.30 lat (usec): min=68, max=478805, avg=914.50, stdev=7309.33 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 69], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 93], | 70.00th=[ 178], 80.00th=[ 194], 90.00th=[ 478], 95.00th=[ 4424], | 99.00th=[ 13829], 99.50th=[ 15795], 99.90th=[ 99091], 99.95th=[145753], | 99.99th=[371196] bw ( KiB/s): min=125952, max=1153024, per=100.00%, avg=512845.98, stdev=123194.35, samples=60 iops : min= 246, max= 2252, avg=1001.62, stdev=240.58, samples=60 write: IOPS=249, BW=125MiB/s (131MB/s)(3792MiB/30394msec) slat (usec): min=36, max=492, avg=96.45, stdev=49.58 clat (msec): min=14, max=2759, avg=396.94, stdev=144.75 lat (msec): min=15, max=2759, avg=397.03, stdev=144.74 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 211], 10.00th=[ 268], 20.00th=[ 296], | 30.00th=[ 326], 40.00th=[ 359], 50.00th=[ 397], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 477], 90.00th=[ 527], 95.00th=[ 600], | 99.00th=[ 793], 99.50th=[ 852], 99.90th=[ 1972], 99.95th=[ 2601], | 99.99th=[ 2769] bw ( KiB/s): min=107520, max=204800, per=99.98%, avg=127731.70, stdev=13142.10, samples=60 iops : min= 210, max= 400, avg=249.45, stdev=25.68, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.05%, 100=50.06% lat (usec) : 250=19.28%, 500=2.60%, 750=0.98%, 1000=0.58% lat (msec) : 2=1.24%, 4=0.92%, 10=2.40%, 20=1.56%, 50=0.05% lat (msec) : 100=0.35%, 250=1.17%, 500=15.90%, 750=2.54%, 1000=0.26% lat (msec) : 2000=0.04%, >=2000=0.02% cpu : usr=2.76%, sys=2.56%, ctx=32092, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30054,7584,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=14.7GiB (15.8GB), run=30394-30394msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3792MiB (3976MB), run=30394-30394msec Disk stats (read/write): vdj: ios=30054/7547, merge=0/0, ticks=19952/2968087, in_queue=3012843, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956767: Fri Jan 7 14:21:27 2022 read: IOPS=973, BW=487MiB/s (510MB/s)(14.3GiB/30003msec) clat (usec): min=79, max=24064, avg=132.34, stdev=561.45 lat (usec): min=79, max=24065, avg=132.50, stdev=561.45 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 91], 10.00th=[ 92], 20.00th=[ 94], | 30.00th=[ 96], 40.00th=[ 98], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 130], 95.00th=[ 141], | 99.00th=[ 174], 99.50th=[ 206], 99.90th=[11863], 99.95th=[13435], | 99.99th=[17695] bw ( KiB/s): min=396288, max=642048, per=100.00%, avg=498420.47, stdev=51006.98, samples=60 iops : min= 774, max= 1254, avg=973.45, stdev=99.66, samples=60 write: IOPS=245, BW=123MiB/s (129MB/s)(3690MiB/30003msec) clat (usec): min=1460, max=62132, avg=3485.08, stdev=2429.65 lat (usec): min=1503, max=62181, avg=3533.31, stdev=2429.17 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2409], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3359], | 70.00th=[ 3523], 80.00th=[ 3818], 90.00th=[ 4686], 95.00th=[ 6783], | 99.00th=[11600], 99.50th=[12518], 99.90th=[45351], 99.95th=[55837], | 99.99th=[62129] bw ( KiB/s): min=95232, max=183296, per=99.99%, avg=125905.05, stdev=11064.32, samples=60 iops : min= 186, max= 358, avg=245.87, stdev=21.63, samples=60 lat (usec) : 100=37.42%, 250=42.17%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=3.01%, 4=13.87%, 10=2.93%, 20=0.51%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.82%, sys=2.75%, ctx=36595, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29207,7379,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=487MiB/s (510MB/s), 487MiB/s-487MiB/s (510MB/s-510MB/s), io=14.3GiB (15.3GB), run=30003-30003msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3690MiB (3869MB), run=30003-30003msec Disk stats (read/write): vdj: ios=29207/7378, merge=0/0, ticks=700/21736, in_queue=28531, util=94.62% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957730: Fri Jan 7 14:22:04 2022 read: IOPS=996, BW=498MiB/s (522MB/s)(14.8GiB/30508msec) slat (usec): min=6, max=454002, avg=740.71, stdev=6138.92 clat (msec): min=9, max=562, avg=64.47, stdev=53.78 lat (msec): min=9, max=588, avg=65.21, stdev=54.14 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 19], 10.00th=[ 26], 20.00th=[ 37], | 30.00th=[ 45], 40.00th=[ 52], 50.00th=[ 59], 60.00th=[ 66], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 93], 95.00th=[ 107], | 99.00th=[ 443], 99.50th=[ 489], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 558] bw ( KiB/s): min=32768, max=1193984, per=100.00%, avg=517717.33, stdev=168089.41, samples=60 iops : min= 64, max= 2332, avg=1011.17, stdev=328.30, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3837MiB/30508msec) slat (usec): min=136, max=485875, avg=963.99, stdev=6660.73 clat (msec): min=22, max=2939, avg=535.50, stdev=188.72 lat (msec): min=48, max=2939, avg=536.47, stdev=188.62 clat percentiles (msec): | 1.00th=[ 113], 5.00th=[ 288], 10.00th=[ 363], 20.00th=[ 422], | 30.00th=[ 460], 40.00th=[ 498], 50.00th=[ 531], 60.00th=[ 558], | 70.00th=[ 584], 80.00th=[ 617], 90.00th=[ 676], 95.00th=[ 835], | 99.00th=[ 1167], 99.50th=[ 1569], 99.90th=[ 2072], 99.95th=[ 2769], | 99.99th=[ 2937] bw ( KiB/s): min= 1024, max=222208, per=98.22%, avg=126489.18, stdev=34631.09, samples=61 iops : min= 2, max= 434, avg=247.05, stdev=67.64, samples=61 lat (msec) : 10=0.01%, 20=5.02%, 50=25.16%, 100=44.37%, 250=5.06% lat (msec) : 500=8.19%, 750=10.78%, 1000=1.02%, 2000=0.37%, >=2000=0.03% cpu : usr=4.51%, sys=1.78%, ctx=3909, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30395,7674,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=14.8GiB (15.9GB), run=30508-30508msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3837MiB (4023MB), run=30508-30508msec Disk stats (read/write): vdj: ios=30395/7604, merge=0/0, ticks=71073/3599018, in_queue=3729802, util=99.56% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958715: Fri Jan 7 14:22:41 2022 read: IOPS=603, BW=603MiB/s (633MB/s)(17.7GiB/30003msec) clat (usec): min=164, max=30039, avg=267.98, stdev=961.02 lat (usec): min=164, max=30040, avg=268.14, stdev=961.02 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 176], 10.00th=[ 180], 20.00th=[ 184], | 30.00th=[ 186], 40.00th=[ 190], 50.00th=[ 196], 60.00th=[ 204], | 70.00th=[ 219], 80.00th=[ 233], 90.00th=[ 251], 95.00th=[ 273], | 99.00th=[ 359], 99.50th=[ 469], 99.90th=[17433], 99.95th=[18482], | 99.99th=[23200] bw ( KiB/s): min=397312, max=839680, per=100.00%, avg=617915.73, stdev=79603.28, samples=60 iops : min= 388, max= 820, avg=603.43, stdev=77.74, samples=60 write: IOPS=152, BW=152MiB/s (160MB/s)(4572MiB/30003msec) clat (usec): min=2575, max=58665, avg=5392.55, stdev=3091.58 lat (usec): min=2674, max=58755, avg=5491.73, stdev=3090.20 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2704], 10.00th=[ 2769], 20.00th=[ 3130], | 30.00th=[ 4080], 40.00th=[ 4686], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5997], 90.00th=[ 8979], 95.00th=[11076], | 99.00th=[15795], 99.50th=[17433], 99.90th=[41681], 99.95th=[56361], | 99.99th=[58459] bw ( KiB/s): min=133120, max=215040, per=99.99%, avg=156023.47, stdev=12457.45, samples=60 iops : min= 130, max= 210, avg=152.37, stdev=12.17, samples=60 lat (usec) : 250=71.36%, 500=8.10%, 750=0.03% lat (msec) : 2=0.01%, 4=5.83%, 10=12.94%, 20=1.68%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.91%, sys=2.49%, ctx=22687, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18103,4572,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=603MiB/s (633MB/s), 603MiB/s-603MiB/s (633MB/s-633MB/s), io=17.7GiB (18.0GB), run=30003-30003msec WRITE: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=4572MiB (4794MB), run=30003-30003msec Disk stats (read/write): vdj: ios=18087/4567, merge=0/0, ticks=1060/22013, in_queue=28638, util=95.25% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959290: Fri Jan 7 14:23:18 2022 read: IOPS=610, BW=610MiB/s (640MB/s)(17.9GiB/30004msec) slat (usec): min=28, max=144, avg=39.27, stdev= 7.79 clat (usec): min=65, max=22674, avg=194.11, stdev=922.67 lat (usec): min=130, max=22730, avg=233.58, stdev=922.75 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 112], 10.00th=[ 114], 20.00th=[ 117], | 30.00th=[ 119], 40.00th=[ 122], 50.00th=[ 125], 60.00th=[ 130], | 70.00th=[ 141], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 188], | 99.00th=[ 297], 99.50th=[ 482], 99.90th=[16712], 99.95th=[18482], | 99.99th=[21365] bw ( KiB/s): min=409600, max=792576, per=100.00%, avg=625174.77, stdev=79299.30, samples=60 iops : min= 400, max= 774, avg=610.52, stdev=77.44, samples=60 write: IOPS=154, BW=155MiB/s (162MB/s)(4640MiB/30004msec) slat (usec): min=67, max=559, avg=130.89, stdev=31.50 clat (usec): min=2506, max=57066, avg=5404.31, stdev=2961.53 lat (usec): min=2634, max=57277, avg=5535.44, stdev=2960.19 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2671], 20.00th=[ 3425], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8848], 95.00th=[10945], | 99.00th=[15533], 99.50th=[16909], 99.90th=[24511], 99.95th=[55313], | 99.99th=[56886] bw ( KiB/s): min=137216, max=204800, per=100.00%, avg=158349.87, stdev=10426.51, samples=60 iops : min= 134, max= 200, avg=154.63, stdev=10.18, samples=60 lat (usec) : 100=0.08%, 250=78.48%, 500=0.83%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=5.27%, 10=13.60%, 20=1.64%, 50=0.03% lat (msec) : 100=0.02% cpu : usr=2.08%, sys=3.50%, ctx=22966, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18315,4640,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=610MiB/s (640MB/s), 610MiB/s-610MiB/s (640MB/s-640MB/s), io=17.9GiB (19.2GB), run=30004-30004msec WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4640MiB (4865MB), run=30004-30004msec Disk stats (read/write): vdj: ios=18199/4602, merge=0/0, ticks=1042/22432, in_queue=28471, util=95.31% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960095: Fri Jan 7 14:23:55 2022 read: IOPS=610, BW=611MiB/s (640MB/s)(17.9GiB/30001msec) clat (usec): min=128, max=49057, avg=234.18, stdev=1060.82 lat (usec): min=128, max=49058, avg=234.34, stdev=1060.82 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 208], 95.00th=[ 229], | 99.00th=[ 322], 99.50th=[ 429], 99.90th=[18220], 99.95th=[19006], | 99.99th=[32375] bw ( KiB/s): min=440320, max=839680, per=100.00%, avg=626064.49, stdev=88299.04, samples=59 iops : min= 430, max= 820, avg=611.36, stdev=86.22, samples=59 write: IOPS=154, BW=155MiB/s (162MB/s)(4640MiB/30001msec) clat (usec): min=2551, max=64490, avg=5433.82, stdev=3022.38 lat (usec): min=2607, max=64583, avg=5531.75, stdev=3021.03 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3425], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[16057], 99.50th=[17957], 99.90th=[22676], 99.95th=[54789], | 99.99th=[64750] bw ( KiB/s): min=114688, max=219136, per=99.91%, avg=158234.93, stdev=13079.87, samples=59 iops : min= 112, max= 214, avg=154.49, stdev=12.77, samples=59 lat (usec) : 250=77.46%, 500=1.95%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.15%, 10=13.62%, 20=1.71%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.92%, sys=2.52%, ctx=22966, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18316,4640,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=611MiB/s (640MB/s), 611MiB/s-611MiB/s (640MB/s-640MB/s), io=17.9GiB (19.2GB), run=30001-30001msec WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4640MiB (4865MB), run=30001-30001msec Disk stats (read/write): vdj: ios=18188/4600, merge=0/0, ticks=1153/22414, in_queue=28392, util=95.11% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961061: Fri Jan 7 14:24:32 2022 read: IOPS=577, BW=578MiB/s (606MB/s)(16.0GiB/30078msec) slat (usec): min=10, max=147, avg=23.16, stdev= 9.30 clat (usec): min=73, max=86499, avg=326.21, stdev=2460.75 lat (usec): min=114, max=86516, avg=349.54, stdev=2460.86 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 115], 40.00th=[ 118], 50.00th=[ 122], 60.00th=[ 128], | 70.00th=[ 141], 80.00th=[ 326], 90.00th=[ 343], 95.00th=[ 363], | 99.00th=[ 578], 99.50th=[ 2474], 99.90th=[44827], 99.95th=[55313], | 99.99th=[62653] bw ( KiB/s): min=413696, max=948224, per=100.00%, avg=593416.60, stdev=94197.58, samples=60 iops : min= 404, max= 926, avg=579.50, stdev=91.99, samples=60 write: IOPS=146, BW=147MiB/s (154MB/s)(4411MiB/30078msec) slat (usec): min=60, max=582, avg=167.63, stdev=85.75 clat (msec): min=3, max=165, avg=66.62, stdev=25.88 lat (msec): min=3, max=166, avg=66.79, stdev=25.88 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 20], 10.00th=[ 27], 20.00th=[ 46], | 30.00th=[ 58], 40.00th=[ 64], 50.00th=[ 69], 60.00th=[ 74], | 70.00th=[ 80], 80.00th=[ 87], 90.00th=[ 99], 95.00th=[ 107], | 99.00th=[ 130], 99.50th=[ 140], 99.90th=[ 161], 99.95th=[ 163], | 99.99th=[ 167] bw ( KiB/s): min=118784, max=239616, per=100.00%, avg=150231.18, stdev=13929.35, samples=60 iops : min= 116, max= 234, avg=146.70, stdev=13.60, samples=60 lat (usec) : 100=0.15%, 250=62.29%, 500=16.39%, 750=0.27%, 1000=0.06% lat (msec) : 2=0.11%, 4=0.15%, 10=0.06%, 20=1.09%, 50=3.70% lat (msec) : 100=14.03%, 250=1.70% cpu : usr=2.63%, sys=2.16%, ctx=21391, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17384,4411,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=578MiB/s (606MB/s), 578MiB/s-578MiB/s (606MB/s-606MB/s), io=16.0GiB (18.2GB), run=30078-30078msec WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=4411MiB (4625MB), run=30078-30078msec Disk stats (read/write): vdj: ios=17384/4405, merge=0/0, ticks=2643/290858, in_queue=299023, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961583: Fri Jan 7 14:25:09 2022 read: IOPS=612, BW=612MiB/s (642MB/s)(17.9GiB/30002msec) clat (usec): min=131, max=55543, avg=226.40, stdev=1043.90 lat (usec): min=131, max=55543, avg=226.55, stdev=1043.90 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 151], 50.00th=[ 155], 60.00th=[ 161], | 70.00th=[ 169], 80.00th=[ 186], 90.00th=[ 198], 95.00th=[ 210], | 99.00th=[ 262], 99.50th=[ 429], 99.90th=[18220], 99.95th=[19792], | 99.99th=[27657] bw ( KiB/s): min=417792, max=860160, per=100.00%, avg=626892.80, stdev=88527.92, samples=60 iops : min= 408, max= 840, avg=612.20, stdev=86.45, samples=60 write: IOPS=155, BW=155MiB/s (163MB/s)(4656MiB/30002msec) clat (usec): min=2539, max=58889, avg=5446.61, stdev=2829.21 lat (usec): min=2601, max=58996, avg=5541.84, stdev=2827.91 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3589], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5932], 90.00th=[ 8455], 95.00th=[11076], | 99.00th=[15401], 99.50th=[16712], 99.90th=[21890], 99.95th=[50594], | 99.99th=[58983] bw ( KiB/s): min=139264, max=210944, per=99.99%, avg=158890.67, stdev=10949.84, samples=60 iops : min= 136, max= 206, avg=155.17, stdev=10.69, samples=60 lat (usec) : 250=78.83%, 500=0.57%, 750=0.02% lat (msec) : 2=0.01%, 4=4.81%, 10=14.01%, 20=1.68%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.82%, sys=2.50%, ctx=23035, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18366,4656,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=612MiB/s (642MB/s), 612MiB/s-612MiB/s (642MB/s-642MB/s), io=17.9GiB (19.3GB), run=30002-30002msec WRITE: bw=155MiB/s (163MB/s), 155MiB/s-155MiB/s (163MB/s-163MB/s), io=4656MiB (4882MB), run=30002-30002msec Disk stats (read/write): vdj: ios=18332/4650, merge=0/0, ticks=1112/22718, in_queue=28634, util=95.18% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962987: Fri Jan 7 14:25:46 2022 read: IOPS=599, BW=599MiB/s (629MB/s)(17.7GiB/30319msec) slat (usec): min=9, max=414, avg=20.56, stdev=11.22 clat (usec): min=48, max=542678, avg=1121.08, stdev=11363.39 lat (usec): min=111, max=542693, avg=1141.81, stdev=11363.36 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 125], 60.00th=[ 139], | 70.00th=[ 204], 80.00th=[ 338], 90.00th=[ 396], 95.00th=[ 1778], | 99.00th=[ 12911], 99.50th=[ 15139], 99.90th=[173016], 99.95th=[212861], | 99.99th=[450888] bw ( KiB/s): min=396722, max=1218850, per=100.00%, avg=626276.63, stdev=134263.26, samples=60 iops : min= 387, max= 1190, avg=611.35, stdev=131.12, samples=60 write: IOPS=151, BW=152MiB/s (159MB/s)(4596MiB/30319msec) slat (usec): min=61, max=718, avg=183.08, stdev=93.32 clat (msec): min=11, max=1236, avg=325.03, stdev=152.60 lat (msec): min=12, max=1236, avg=325.22, stdev=152.59 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 130], 10.00th=[ 148], 20.00th=[ 178], | 30.00th=[ 215], 40.00th=[ 259], 50.00th=[ 313], 60.00th=[ 368], | 70.00th=[ 409], 80.00th=[ 451], 90.00th=[ 514], 95.00th=[ 584], | 99.00th=[ 760], 99.50th=[ 844], 99.90th=[ 1011], 99.95th=[ 1083], | 99.99th=[ 1234] bw ( KiB/s): min=110533, max=242526, per=100.00%, avg=156657.10, stdev=19310.80, samples=60 iops : min= 107, max= 236, avg=152.65, stdev=18.83, samples=60 lat (usec) : 50=0.01%, 100=0.25%, 250=57.02%, 500=16.02%, 750=0.91% lat (usec) : 1000=0.54% lat (msec) : 2=1.30%, 4=1.24%, 10=1.28%, 20=0.92%, 50=0.14% lat (msec) : 100=0.36%, 250=7.46%, 500=10.20%, 750=2.13%, 1000=0.21% lat (msec) : 2000=0.02% cpu : usr=3.10%, sys=1.91%, ctx=20542, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=18173,4596,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=599MiB/s (629MB/s), 599MiB/s-599MiB/s (629MB/s-629MB/s), io=17.7GiB (19.1GB), run=30319-30319msec WRITE: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=4596MiB (4819MB), run=30319-30319msec Disk stats (read/write): vdj: ios=18561/4677, merge=0/0, ticks=15767/1509618, in_queue=1534829, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963566: Fri Jan 7 14:26:23 2022 read: IOPS=608, BW=609MiB/s (638MB/s)(17.8GiB/30001msec) clat (usec): min=130, max=44870, avg=230.79, stdev=1044.41 lat (usec): min=130, max=44870, avg=230.95, stdev=1044.41 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 204], 95.00th=[ 219], | 99.00th=[ 285], 99.50th=[ 510], 99.90th=[18220], 99.95th=[21890], | 99.99th=[30278] bw ( KiB/s): min=440320, max=794624, per=100.00%, avg=624362.31, stdev=83064.41, samples=59 iops : min= 430, max= 776, avg=609.73, stdev=81.12, samples=59 write: IOPS=154, BW=154MiB/s (162MB/s)(4625MiB/30001msec) clat (usec): min=2560, max=58028, avg=5468.68, stdev=3041.85 lat (usec): min=2636, max=58132, avg=5565.79, stdev=3040.60 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3458], | 30.00th=[ 4359], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5997], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[16057], 99.50th=[18482], 99.90th=[27395], 99.95th=[55837], | 99.99th=[57934] bw ( KiB/s): min=131072, max=210944, per=99.87%, avg=157661.29, stdev=11975.00, samples=59 iops : min= 128, max= 206, avg=153.97, stdev=11.69, samples=59 lat (usec) : 250=78.32%, 500=1.07%, 750=0.06% lat (msec) : 2=0.01%, 4=5.15%, 10=13.60%, 20=1.68%, 50=0.09% lat (msec) : 100=0.02% cpu : usr=2.02%, sys=2.37%, ctx=22898, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18267,4625,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=609MiB/s (638MB/s), 609MiB/s-609MiB/s (638MB/s-638MB/s), io=17.8GiB (19.2GB), run=30001-30001msec WRITE: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=4625MiB (4850MB), run=30001-30001msec Disk stats (read/write): vdj: ios=18262/4621, merge=0/0, ticks=1123/22661, in_queue=28589, util=94.89% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964384: Fri Jan 7 14:27:00 2022 read: IOPS=596, BW=597MiB/s (626MB/s)(17.9GiB/30681msec) slat (usec): min=9, max=689, avg=19.39, stdev=17.43 clat (nsec): min=1691, max=884230k, avg=2848973.20, stdev=14283639.58 lat (usec): min=114, max=884245, avg=2868.51, stdev=14283.89 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 116], 40.00th=[ 120], 50.00th=[ 129], 60.00th=[ 215], | 70.00th=[ 338], 80.00th=[ 537], 90.00th=[ 7046], 95.00th=[ 19006], | 99.00th=[ 30802], 99.50th=[ 36439], 99.90th=[170918], 99.95th=[227541], | 99.99th=[817890] bw ( KiB/s): min=20480, max=1316864, per=100.00%, avg=624878.93, stdev=242571.30, samples=60 iops : min= 20, max= 1286, avg=610.23, stdev=236.89, samples=60 write: IOPS=151, BW=151MiB/s (159MB/s)(4638MiB/30681msec) slat (usec): min=61, max=1405, avg=198.06, stdev=107.21 clat (msec): min=23, max=2273, avg=649.71, stdev=246.26 lat (msec): min=23, max=2273, avg=649.91, stdev=246.25 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 300], 10.00th=[ 376], 20.00th=[ 468], | 30.00th=[ 514], 40.00th=[ 558], 50.00th=[ 609], 60.00th=[ 667], | 70.00th=[ 751], 80.00th=[ 827], 90.00th=[ 961], 95.00th=[ 1099], | 99.00th=[ 1418], 99.50th=[ 1586], 99.90th=[ 1620], 99.95th=[ 2022], | 99.99th=[ 2265] bw ( KiB/s): min= 2048, max=278528, per=98.45%, avg=152391.34, stdev=55257.27, samples=61 iops : min= 2, max= 272, avg=148.82, stdev=53.96, samples=61 lat (usec) : 2=0.01%, 100=0.08%, 250=48.78%, 500=14.62%, 750=1.85% lat (usec) : 1000=1.12% lat (msec) : 2=1.74%, 4=1.94%, 10=2.71%, 20=3.26%, 50=3.37% lat (msec) : 100=0.10%, 250=0.80%, 500=4.86%, 750=8.70%, 1000=4.40% lat (msec) : 2000=1.65%, >=2000=0.01% cpu : usr=3.21%, sys=1.80%, ctx=17784, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18307,4638,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=597MiB/s (626MB/s), 597MiB/s-597MiB/s (626MB/s-626MB/s), io=17.9GiB (19.2GB), run=30681-30681msec WRITE: bw=151MiB/s (159MB/s), 151MiB/s-151MiB/s (159MB/s-159MB/s), io=4638MiB (4863MB), run=30681-30681msec Disk stats (read/write): vdj: ios=18307/4608, merge=0/0, ticks=43120/2952591, in_queue=3021434, util=99.50% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965450: Fri Jan 7 14:27:37 2022 read: IOPS=607, BW=608MiB/s (637MB/s)(17.8GiB/30001msec) clat (usec): min=131, max=59671, avg=231.15, stdev=1071.40 lat (usec): min=131, max=59671, avg=231.30, stdev=1071.40 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 204], 95.00th=[ 219], | 99.00th=[ 293], 99.50th=[ 433], 99.90th=[16909], 99.95th=[19006], | 99.99th=[50594] bw ( KiB/s): min=448512, max=817152, per=100.00%, avg=622318.93, stdev=82243.45, samples=60 iops : min= 438, max= 798, avg=607.73, stdev=80.32, samples=60 write: IOPS=153, BW=154MiB/s (161MB/s)(4615MiB/30001msec) clat (usec): min=2570, max=56305, avg=5480.80, stdev=2857.32 lat (usec): min=2626, max=56401, avg=5578.23, stdev=2856.13 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3425], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[15533], 99.50th=[16712], 99.90th=[20579], 99.95th=[33424], | 99.99th=[56361] bw ( KiB/s): min=129024, max=212992, per=100.00%, avg=157525.33, stdev=13003.59, samples=60 iops : min= 126, max= 208, avg=153.83, stdev=12.70, samples=60 lat (usec) : 250=78.20%, 500=1.23%, 750=0.02% lat (msec) : 2=0.01%, 4=5.17%, 10=13.57%, 20=1.73%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=2.45%, ctx=22858, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18233,4615,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=608MiB/s (637MB/s), 608MiB/s-608MiB/s (637MB/s-637MB/s), io=17.8GiB (19.1GB), run=30001-30001msec WRITE: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4615MiB (4839MB), run=30001-30001msec Disk stats (read/write): vdj: ios=18173/4594, merge=0/0, ticks=1105/22622, in_queue=28531, util=94.75% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966068: Fri Jan 7 14:28:15 2022 read: IOPS=603, BW=604MiB/s (633MB/s)(18.1GiB/30726msec) slat (usec): min=11, max=833844, avg=1268.70, stdev=13453.56 clat (msec): min=18, max=978, avg=108.65, stdev=108.51 lat (msec): min=18, max=978, avg=109.92, stdev=109.41 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 39], 20.00th=[ 56], | 30.00th=[ 70], 40.00th=[ 83], 50.00th=[ 92], 60.00th=[ 103], | 70.00th=[ 114], 80.00th=[ 129], 90.00th=[ 155], 95.00th=[ 188], | 99.00th=[ 785], 99.50th=[ 827], 99.90th=[ 877], 99.95th=[ 894], | 99.99th=[ 961] bw ( KiB/s): min= 2048, max=1445888, per=100.00%, avg=620850.25, stdev=238172.80, samples=61 iops : min= 2, max= 1412, avg=606.28, stdev=232.58, samples=61 write: IOPS=152, BW=153MiB/s (160MB/s)(4693MiB/30726msec) slat (usec): min=271, max=183635, avg=1369.15, stdev=5708.83 clat (msec): min=43, max=2272, avg=872.53, stdev=294.27 lat (msec): min=43, max=2272, avg=873.90, stdev=294.21 clat percentiles (msec): | 1.00th=[ 174], 5.00th=[ 401], 10.00th=[ 558], 20.00th=[ 676], | 30.00th=[ 743], 40.00th=[ 802], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1036], 90.00th=[ 1217], 95.00th=[ 1452], | 99.00th=[ 1770], 99.50th=[ 1955], 99.90th=[ 1989], 99.95th=[ 2022], | 99.99th=[ 2265] bw ( KiB/s): min=12288, max=293450, per=99.45%, avg=155539.93, stdev=57324.46, samples=60 iops : min= 12, max= 286, avg=151.87, stdev=55.96, samples=60 lat (msec) : 20=0.01%, 50=12.73%, 100=33.81%, 250=31.34%, 500=1.82% lat (msec) : 750=5.21%, 1000=9.93%, 2000=5.14%, >=2000=0.02% cpu : usr=5.26%, sys=1.42%, ctx=2241, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18551,4693,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=18.1GiB (19.5GB), run=30726-30726msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4693MiB (4921MB), run=30726-30726msec Disk stats (read/write): vdj: ios=18551/4655, merge=0/0, ticks=118160/3586007, in_queue=3745422, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966920: Fri Jan 7 14:28:51 2022 read: IOPS=259, BW=518MiB/s (544MB/s)(15.2GiB/30010msec) clat (usec): min=252, max=24434, avg=463.44, stdev=1002.45 lat (usec): min=252, max=24435, avg=463.62, stdev=1002.46 clat percentiles (usec): | 1.00th=[ 260], 5.00th=[ 269], 10.00th=[ 273], 20.00th=[ 277], | 30.00th=[ 281], 40.00th=[ 289], 50.00th=[ 297], 60.00th=[ 314], | 70.00th=[ 343], 80.00th=[ 594], 90.00th=[ 717], 95.00th=[ 750], | 99.00th=[ 881], 99.50th=[ 8848], 99.90th=[15270], 99.95th=[17171], | 99.99th=[24511] bw ( KiB/s): min=273066, max=805161, per=100.00%, avg=534242.77, stdev=118322.49, samples=60 iops : min= 133, max= 393, avg=260.43, stdev=57.79, samples=60 write: IOPS=65, BW=132MiB/s (138MB/s)(3958MiB/30010msec) clat (usec): min=5804, max=47740, avg=13055.53, stdev=3179.36 lat (usec): min=5990, max=47941, avg=13330.24, stdev=3199.26 clat percentiles (usec): | 1.00th=[ 6783], 5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[10159], | 30.00th=[11207], 40.00th=[11994], 50.00th=[13173], 60.00th=[14222], | 70.00th=[15008], 80.00th=[15533], 90.00th=[16319], 95.00th=[16909], | 99.00th=[21627], 99.50th=[23462], 99.90th=[41681], 99.95th=[47973], | 99.99th=[47973] bw ( KiB/s): min=119261, max=153082, per=100.00%, avg=135881.68, stdev=8214.11, samples=60 iops : min= 58, max= 74, avg=65.90, stdev= 3.99, samples=60 lat (usec) : 500=62.50%, 750=13.34%, 1000=3.27% lat (msec) : 2=0.03%, 4=0.02%, 10=3.78%, 20=16.77%, 50=0.29% cpu : usr=2.03%, sys=1.89%, ctx=9769, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7779,1979,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=518MiB/s (544MB/s), 518MiB/s-518MiB/s (544MB/s-544MB/s), io=15.2GiB (16.3GB), run=30010-30010msec WRITE: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3958MiB (4150MB), run=30010-30010msec Disk stats (read/write): vdj: ios=15544/3955, merge=0/0, ticks=649/48155, in_queue=55727, util=95.73% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967513: Fri Jan 7 14:29:28 2022 read: IOPS=265, BW=531MiB/s (557MB/s)(15.6GiB/30007msec) slat (usec): min=53, max=233, avg=71.99, stdev=13.69 clat (usec): min=176, max=58098, avg=369.19, stdev=1191.65 lat (usec): min=251, max=58159, avg=441.42, stdev=1191.82 clat percentiles (usec): | 1.00th=[ 198], 5.00th=[ 202], 10.00th=[ 206], 20.00th=[ 210], | 30.00th=[ 215], 40.00th=[ 219], 50.00th=[ 225], 60.00th=[ 235], | 70.00th=[ 265], 80.00th=[ 326], 90.00th=[ 537], 95.00th=[ 627], | 99.00th=[ 783], 99.50th=[ 8848], 99.90th=[15926], 99.95th=[18744], | 99.99th=[57934] bw ( KiB/s): min=274432, max=950272, per=100.00%, avg=543607.47, stdev=120272.29, samples=60 iops : min= 134, max= 464, avg=265.43, stdev=58.73, samples=60 write: IOPS=67, BW=135MiB/s (141MB/s)(4046MiB/30007msec) slat (usec): min=135, max=892, avg=300.52, stdev=115.15 clat (usec): min=4650, max=63061, avg=12783.84, stdev=3625.50 lat (usec): min=4858, max=63668, avg=13084.69, stdev=3641.77 clat percentiles (usec): | 1.00th=[ 5014], 5.00th=[ 7111], 10.00th=[ 8291], 20.00th=[ 9765], | 30.00th=[10814], 40.00th=[11731], 50.00th=[13042], 60.00th=[14222], | 70.00th=[15139], 80.00th=[15795], 90.00th=[16319], 95.00th=[16909], | 99.00th=[19268], 99.50th=[24249], 99.90th=[43254], 99.95th=[45351], | 99.99th=[63177] bw ( KiB/s): min=106496, max=155648, per=99.97%, avg=138035.20, stdev=8771.98, samples=60 iops : min= 52, max= 76, avg=67.40, stdev= 4.28, samples=60 lat (usec) : 250=53.35%, 500=17.30%, 750=8.14%, 1000=0.39% lat (msec) : 2=0.01%, 10=4.67%, 20=15.91%, 50=0.20%, 100=0.02% cpu : usr=1.92%, sys=2.54%, ctx=9995, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7963,2023,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=531MiB/s (557MB/s), 531MiB/s-531MiB/s (557MB/s-557MB/s), io=15.6GiB (16.7GB), run=30007-30007msec WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4046MiB (4243MB), run=30007-30007msec Disk stats (read/write): vdj: ios=15852/4020, merge=0/0, ticks=704/49216, in_queue=56631, util=96.16% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968315: Fri Jan 7 14:30:05 2022 read: IOPS=267, BW=536MiB/s (562MB/s)(15.7GiB/30009msec) clat (usec): min=230, max=24651, avg=391.16, stdev=1014.97 lat (usec): min=230, max=24651, avg=391.33, stdev=1014.97 clat percentiles (usec): | 1.00th=[ 237], 5.00th=[ 243], 10.00th=[ 247], 20.00th=[ 251], | 30.00th=[ 255], 40.00th=[ 260], 50.00th=[ 265], 60.00th=[ 273], | 70.00th=[ 293], 80.00th=[ 347], 90.00th=[ 474], 95.00th=[ 611], | 99.00th=[ 775], 99.50th=[ 9110], 99.90th=[15401], 99.95th=[16712], | 99.99th=[24773] bw ( KiB/s): min=282624, max=815104, per=100.00%, avg=548590.93, stdev=120070.63, samples=60 iops : min= 138, max= 398, avg=267.87, stdev=58.63, samples=60 write: IOPS=68, BW=136MiB/s (143MB/s)(4082MiB/30009msec) clat (usec): min=4682, max=48348, avg=12924.17, stdev=3501.29 lat (usec): min=4885, max=48549, avg=13152.28, stdev=3508.66 clat percentiles (usec): | 1.00th=[ 5080], 5.00th=[ 7111], 10.00th=[ 8455], 20.00th=[ 9896], | 30.00th=[11076], 40.00th=[11863], 50.00th=[13304], 60.00th=[14484], | 70.00th=[15270], 80.00th=[15926], 90.00th=[16450], 95.00th=[16909], | 99.00th=[19268], 99.50th=[21365], 99.90th=[43779], 99.95th=[46400], | 99.99th=[48497] bw ( KiB/s): min=114688, max=167936, per=99.98%, avg=139264.00, stdev=8794.65, samples=60 iops : min= 56, max= 82, avg=68.00, stdev= 4.29, samples=60 lat (usec) : 250=15.34%, 500=57.18%, 750=6.25%, 1000=0.41% lat (msec) : 2=0.03%, 10=4.46%, 20=16.17%, 50=0.17% cpu : usr=1.69%, sys=1.62%, ctx=10084, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8036,2041,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=15.7GiB (16.9GB), run=30009-30009msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4082MiB (4280MB), run=30009-30009msec Disk stats (read/write): vdj: ios=15944/4050, merge=0/0, ticks=669/42259, in_queue=49165, util=96.53% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969574: Fri Jan 7 14:30:42 2022 read: IOPS=355, BW=712MiB/s (746MB/s)(20.9GiB/30113msec) slat (usec): min=20, max=190, avg=37.57, stdev=14.01 clat (usec): min=102, max=803457, avg=1109.92, stdev=20100.96 lat (usec): min=204, max=803486, avg=1147.70, stdev=20101.28 clat percentiles (usec): | 1.00th=[ 184], 5.00th=[ 188], 10.00th=[ 190], 20.00th=[ 196], | 30.00th=[ 200], 40.00th=[ 204], 50.00th=[ 208], 60.00th=[ 217], | 70.00th=[ 235], 80.00th=[ 619], 90.00th=[ 652], 95.00th=[ 676], | 99.00th=[ 840], 99.50th=[ 1745], 99.90th=[362808], 99.95th=[616563], | 99.99th=[734004] bw ( KiB/s): min=430080, max=1028096, per=100.00%, avg=731356.50, stdev=126945.01, samples=60 iops : min= 210, max= 502, avg=357.08, stdev=61.99, samples=60 write: IOPS=90, BW=181MiB/s (190MB/s)(5454MiB/30113msec) slat (usec): min=117, max=1670, avg=344.65, stdev=183.82 clat (msec): min=12, max=1651, avg=105.52, stdev=111.27 lat (msec): min=12, max=1652, avg=105.86, stdev=111.28 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 20], 10.00th=[ 21], 20.00th=[ 30], | 30.00th=[ 40], 40.00th=[ 51], 50.00th=[ 68], 60.00th=[ 91], | 70.00th=[ 118], 80.00th=[ 163], 90.00th=[ 236], 95.00th=[ 313], | 99.00th=[ 498], 99.50th=[ 600], 99.90th=[ 911], 99.95th=[ 1250], | 99.99th=[ 1653] bw ( KiB/s): min=167936, max=196608, per=100.00%, avg=185555.50, stdev=5967.98, samples=60 iops : min= 82, max= 96, avg=90.58, stdev= 2.95, samples=60 lat (usec) : 250=57.83%, 500=4.78%, 750=15.71%, 1000=0.83% lat (msec) : 2=0.17%, 4=0.13%, 10=0.07%, 20=1.79%, 50=6.36% lat (msec) : 100=4.78%, 250=5.71%, 500=1.61%, 750=0.19%, 1000=0.04% lat (msec) : 2000=0.01% cpu : usr=3.15%, sys=1.99%, ctx=13425, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10714,2727,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=712MiB/s (746MB/s), 712MiB/s-712MiB/s (746MB/s-746MB/s), io=20.9GiB (22.5GB), run=30113-30113msec WRITE: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=5454MiB (5719MB), run=30113-30113msec Disk stats (read/write): vdj: ios=21409/5432, merge=0/0, ticks=8396/568151, in_queue=588076, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970157: Fri Jan 7 14:31:19 2022 read: IOPS=269, BW=539MiB/s (565MB/s)(15.8GiB/30014msec) clat (usec): min=230, max=24735, avg=378.95, stdev=1036.76 lat (usec): min=230, max=24735, avg=379.12, stdev=1036.77 clat percentiles (usec): | 1.00th=[ 237], 5.00th=[ 243], 10.00th=[ 247], 20.00th=[ 251], | 30.00th=[ 255], 40.00th=[ 260], 50.00th=[ 265], 60.00th=[ 273], | 70.00th=[ 293], 80.00th=[ 343], 90.00th=[ 396], 95.00th=[ 474], | 99.00th=[ 758], 99.50th=[ 9110], 99.90th=[16188], 99.95th=[17171], | 99.99th=[24773] bw ( KiB/s): min=294912, max=831488, per=100.00%, avg=551918.97, stdev=121848.40, samples=60 iops : min= 144, max= 406, avg=269.48, stdev=59.50, samples=60 write: IOPS=68, BW=137MiB/s (143MB/s)(4100MiB/30014msec) clat (usec): min=4680, max=48258, avg=12924.46, stdev=3357.56 lat (usec): min=4884, max=48631, avg=13135.96, stdev=3363.14 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 7373], 10.00th=[ 8586], 20.00th=[ 9896], | 30.00th=[11076], 40.00th=[11994], 50.00th=[13173], 60.00th=[14484], | 70.00th=[15270], 80.00th=[15926], 90.00th=[16450], 95.00th=[16909], | 99.00th=[19006], 99.50th=[19792], 99.90th=[25560], 99.95th=[47973], | 99.99th=[48497] bw ( KiB/s): min=118784, max=159744, per=99.99%, avg=139873.62, stdev=8915.64, samples=60 iops : min= 58, max= 78, avg=68.28, stdev= 4.35, samples=60 lat (usec) : 250=15.49%, 500=60.99%, 750=2.49%, 1000=0.25% lat (msec) : 2=0.03%, 10=4.33%, 20=16.33%, 50=0.10% cpu : usr=1.63%, sys=1.53%, ctx=10143, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8085,2050,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=15.8GiB (16.0GB), run=30014-30014msec WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4100MiB (4299MB), run=30014-30014msec Disk stats (read/write): vdj: ios=16170/4098, merge=0/0, ticks=684/42578, in_queue=49252, util=96.34% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971118: Fri Jan 7 14:31:56 2022 read: IOPS=362, BW=726MiB/s (761MB/s)(21.6GiB/30526msec) slat (usec): min=19, max=441, avg=34.37, stdev=17.72 clat (usec): min=133, max=2574.3k, avg=4410.29, stdev=64118.94 lat (usec): min=204, max=2574.4k, avg=4444.85, stdev=64119.06 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 188], 10.00th=[ 192], | 20.00th=[ 196], 30.00th=[ 200], 40.00th=[ 206], | 50.00th=[ 217], 60.00th=[ 265], 70.00th=[ 619], | 80.00th=[ 660], 90.00th=[ 1532], 95.00th=[ 10814], | 99.00th=[ 24773], 99.50th=[ 51643], 99.90th=[ 960496], | 99.95th=[1468007], 99.99th=[2533360] bw ( KiB/s): min=401408, max=1404928, per=100.00%, avg=756076.17, stdev=178819.74, samples=60 iops : min= 196, max= 686, avg=369.17, stdev=87.32, samples=60 write: IOPS=92, BW=184MiB/s (193MB/s)(5630MiB/30526msec) slat (usec): min=119, max=1009, avg=373.20, stdev=200.77 clat (msec): min=29, max=3241, avg=524.04, stdev=242.49 lat (msec): min=30, max=3241, avg=524.41, stdev=242.48 clat percentiles (msec): | 1.00th=[ 215], 5.00th=[ 300], 10.00th=[ 347], 20.00th=[ 376], | 30.00th=[ 393], 40.00th=[ 418], 50.00th=[ 447], 60.00th=[ 493], | 70.00th=[ 550], 80.00th=[ 642], 90.00th=[ 802], 95.00th=[ 986], | 99.00th=[ 1318], 99.50th=[ 1552], 99.90th=[ 2903], 99.95th=[ 2970], | 99.99th=[ 3239] bw ( KiB/s): min= 4096, max=233472, per=98.38%, avg=185803.30, stdev=27880.94, samples=61 iops : min= 2, max= 114, avg=90.72, stdev=13.61, samples=61 lat (usec) : 250=46.74%, 500=8.16%, 750=13.75%, 1000=2.13% lat (msec) : 2=1.37%, 4=1.10%, 10=2.22%, 20=2.81%, 50=1.06% lat (msec) : 100=0.14%, 250=0.50%, 500=12.12%, 750=5.26%, 1000=1.61% lat (msec) : 2000=0.94%, >=2000=0.09% cpu : usr=3.43%, sys=1.87%, ctx=12287, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=11076,2815,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=726MiB/s (761MB/s), 726MiB/s-726MiB/s (761MB/s-761MB/s), io=21.6GiB (23.2GB), run=30526-30526msec WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5630MiB (5903MB), run=30526-30526msec Disk stats (read/write): vdj: ios=22151/5620, merge=0/0, ticks=54399/2919981, in_queue=2992299, util=99.51% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972164: Fri Jan 7 14:32:33 2022 read: IOPS=267, BW=535MiB/s (561MB/s)(15.7GiB/30013msec) clat (usec): min=215, max=24722, avg=355.94, stdev=1003.63 lat (usec): min=215, max=24722, avg=356.12, stdev=1003.63 clat percentiles (usec): | 1.00th=[ 221], 5.00th=[ 227], 10.00th=[ 229], 20.00th=[ 233], | 30.00th=[ 239], 40.00th=[ 243], 50.00th=[ 247], 60.00th=[ 255], | 70.00th=[ 273], 80.00th=[ 318], 90.00th=[ 359], 95.00th=[ 433], | 99.00th=[ 734], 99.50th=[ 9241], 99.90th=[15401], 99.95th=[16581], | 99.99th=[24773] bw ( KiB/s): min=262144, max=970752, per=100.00%, avg=547976.53, stdev=124164.78, samples=60 iops : min= 128, max= 474, avg=267.57, stdev=60.63, samples=60 write: IOPS=67, BW=136MiB/s (142MB/s)(4076MiB/30013msec) clat (usec): min=4660, max=52060, avg=13110.04, stdev=3568.66 lat (usec): min=4862, max=52252, avg=13314.32, stdev=3567.51 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 7373], 10.00th=[ 8717], 20.00th=[10159], | 30.00th=[11207], 40.00th=[11994], 50.00th=[13435], 60.00th=[14746], | 70.00th=[15401], 80.00th=[15926], 90.00th=[16581], 95.00th=[17171], | 99.00th=[20579], 99.50th=[24511], 99.90th=[41681], 99.95th=[45876], | 99.99th=[52167] bw ( KiB/s): min=126976, max=163840, per=99.99%, avg=139059.20, stdev=8445.80, samples=60 iops : min= 62, max= 80, avg=67.90, stdev= 4.12, samples=60 lat (usec) : 250=42.93%, 500=34.38%, 750=1.67%, 1000=0.23% lat (msec) : 2=0.01%, 10=4.04%, 20=16.49%, 50=0.24%, 100=0.01% cpu : usr=1.54%, sys=1.14%, ctx=10072, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8027,2038,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=535MiB/s (561MB/s), 535MiB/s-535MiB/s (561MB/s-561MB/s), io=15.7GiB (16.8GB), run=30013-30013msec WRITE: bw=136MiB/s (142MB/s), 136MiB/s-136MiB/s (142MB/s-142MB/s), io=4076MiB (4274MB), run=30013-30013msec Disk stats (read/write): vdj: ios=15958/4054, merge=0/0, ticks=665/41713, in_queue=48283, util=96.67% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972688: Fri Jan 7 14:33:11 2022 read: IOPS=361, BW=722MiB/s (757MB/s)(21.6GiB/30656msec) slat (usec): min=20, max=504882, avg=1993.49, stdev=10785.32 clat (msec): min=17, max=2602, avg=88.00, stdev=93.53 lat (msec): min=17, max=2602, avg=90.00, stdev=94.32 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 35], 20.00th=[ 47], | 30.00th=[ 57], 40.00th=[ 66], 50.00th=[ 74], 60.00th=[ 84], | 70.00th=[ 94], 80.00th=[ 106], 90.00th=[ 125], 95.00th=[ 153], | 99.00th=[ 542], 99.50th=[ 693], 99.90th=[ 1011], 99.95th=[ 1469], | 99.99th=[ 2366] bw ( KiB/s): min= 4096, max=1581056, per=100.00%, avg=741433.67, stdev=288949.16, samples=61 iops : min= 2, max= 772, avg=361.97, stdev=141.08, samples=61 write: IOPS=91, BW=184MiB/s (192MB/s)(5626MiB/30656msec) slat (usec): min=543, max=663364, avg=2807.98, stdev=13854.30 clat (msec): min=99, max=3721, avg=731.70, stdev=283.91 lat (msec): min=193, max=3723, avg=734.51, stdev=283.59 clat percentiles (msec): | 1.00th=[ 296], 5.00th=[ 401], 10.00th=[ 456], 20.00th=[ 558], | 30.00th=[ 592], 40.00th=[ 625], 50.00th=[ 659], 60.00th=[ 709], | 70.00th=[ 785], 80.00th=[ 885], 90.00th=[ 1070], 95.00th=[ 1234], | 99.00th=[ 1821], 99.50th=[ 1938], 99.90th=[ 3071], 99.95th=[ 3306], | 99.99th=[ 3708] bw ( KiB/s): min=16384, max=323584, per=99.77%, avg=187492.50, stdev=61922.54, samples=60 iops : min= 8, max= 158, avg=91.53, stdev=30.23, samples=60 lat (msec) : 20=0.01%, 50=18.58%, 100=41.17%, 250=18.30%, 500=3.13% lat (msec) : 750=11.75%, 1000=4.26%, 2000=2.69%, >=2000=0.11% cpu : usr=6.44%, sys=1.56%, ctx=2753, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11072,2813,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=722MiB/s (757MB/s), 722MiB/s-722MiB/s (757MB/s-757MB/s), io=21.6GiB (23.2GB), run=30656-30656msec WRITE: bw=184MiB/s (192MB/s), 184MiB/s-184MiB/s (192MB/s-192MB/s), io=5626MiB (5899MB), run=30656-30656msec Disk stats (read/write): vdj: ios=22142/5580, merge=0/0, ticks=125727/3579855, in_queue=3748850, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973649: Fri Jan 7 14:33:47 2022 read: IOPS=267, BW=536MiB/s (562MB/s)(15.7GiB/30007msec) clat (usec): min=231, max=33105, avg=383.20, stdev=1047.31 lat (usec): min=231, max=33105, avg=383.37, stdev=1047.31 clat percentiles (usec): | 1.00th=[ 239], 5.00th=[ 245], 10.00th=[ 247], 20.00th=[ 253], | 30.00th=[ 260], 40.00th=[ 265], 50.00th=[ 273], 60.00th=[ 281], | 70.00th=[ 302], 80.00th=[ 343], 90.00th=[ 396], 95.00th=[ 474], | 99.00th=[ 766], 99.50th=[ 9110], 99.90th=[16188], 99.95th=[17171], | 99.99th=[33162] bw ( KiB/s): min=278528, max=909312, per=100.00%, avg=548609.93, stdev=121841.93, samples=60 iops : min= 136, max= 444, avg=267.87, stdev=59.49, samples=60 write: IOPS=67, BW=136MiB/s (143MB/s)(4080MiB/30007msec) clat (usec): min=4666, max=58899, avg=12980.90, stdev=3672.39 lat (usec): min=4866, max=59135, avg=13189.40, stdev=3672.34 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7177], 10.00th=[ 8455], 20.00th=[ 9896], | 30.00th=[11076], 40.00th=[11863], 50.00th=[13173], 60.00th=[14484], | 70.00th=[15270], 80.00th=[15926], 90.00th=[16450], 95.00th=[17171], | 99.00th=[19268], 99.50th=[23725], 99.90th=[48497], 99.95th=[51119], | 99.99th=[58983] bw ( KiB/s): min=122880, max=159744, per=99.98%, avg=139200.78, stdev=7935.19, samples=60 iops : min= 60, max= 78, avg=67.97, stdev= 3.87, samples=60 lat (usec) : 250=11.14%, 500=65.32%, 750=2.47%, 1000=0.22% lat (msec) : 2=0.07%, 10=4.34%, 20=16.25%, 50=0.18%, 100=0.02% cpu : usr=1.56%, sys=1.65%, ctx=10082, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8036,2040,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=15.7GiB (16.9GB), run=30007-30007msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4080MiB (4278MB), run=30007-30007msec Disk stats (read/write): vdj: ios=15934/4046, merge=0/0, ticks=685/42666, in_queue=49341, util=95.83% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974269: Fri Jan 7 14:34:25 2022 read: IOPS=363, BW=726MiB/s (761MB/s)(21.7GiB/30658msec) slat (usec): min=23, max=514958, avg=2089.87, stdev=11087.10 clat (msec): min=84, max=2774, avg=308.70, stdev=132.80 lat (msec): min=84, max=2774, avg=310.79, stdev=133.28 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 129], 10.00th=[ 180], 20.00th=[ 230], | 30.00th=[ 255], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 359], 90.00th=[ 422], 95.00th=[ 592], | 99.00th=[ 776], 99.50th=[ 844], 99.90th=[ 927], 99.95th=[ 1385], | 99.99th=[ 2601] bw ( KiB/s): min= 4096, max=1387222, per=99.50%, avg=739923.08, stdev=276045.07, samples=61 iops : min= 2, max= 677, avg=361.26, stdev=134.78, samples=61 write: IOPS=92, BW=184MiB/s (193MB/s)(5654MiB/30658msec) slat (usec): min=510, max=469260, avg=2366.67, stdev=10138.65 clat (msec): min=107, max=3495, avg=937.42, stdev=287.61 lat (msec): min=278, max=3495, avg=939.78, stdev=287.29 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 617], 10.00th=[ 676], 20.00th=[ 743], | 30.00th=[ 793], 40.00th=[ 835], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 995], 80.00th=[ 1099], 90.00th=[ 1284], 95.00th=[ 1452], | 99.00th=[ 1938], 99.50th=[ 2366], 99.90th=[ 2903], 99.95th=[ 3440], | 99.99th=[ 3507] bw ( KiB/s): min=28672, max=356352, per=100.00%, avg=190381.12, stdev=68372.85, samples=59 iops : min= 14, max= 174, avg=92.95, stdev=33.38, samples=59 lat (msec) : 100=1.11%, 250=21.20%, 500=52.35%, 750=8.01%, 1000=11.40% lat (msec) : 2000=5.73%, >=2000=0.21% cpu : usr=6.44%, sys=1.73%, ctx=2773, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11132,2827,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=726MiB/s (761MB/s), 726MiB/s-726MiB/s (761MB/s-761MB/s), io=21.7GiB (23.3GB), run=30658-30658msec WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5654MiB (5929MB), run=30658-30658msec Disk stats (read/write): vdj: ios=22262/5642, merge=0/0, ticks=120523/3604951, in_queue=3751767, util=99.09% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975115: Fri Jan 7 14:35:01 2022 read: IOPS=160, BW=643MiB/s (674MB/s)(18.8GiB/30006msec) clat (usec): min=389, max=24867, avg=810.44, stdev=1517.51 lat (usec): min=389, max=24867, avg=810.66, stdev=1517.52 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 412], 20.00th=[ 420], | 30.00th=[ 429], 40.00th=[ 437], 50.00th=[ 449], 60.00th=[ 465], | 70.00th=[ 553], 80.00th=[ 1270], 90.00th=[ 1319], 95.00th=[ 1401], | 99.00th=[ 9372], 99.50th=[14091], 99.90th=[19268], 99.95th=[19530], | 99.99th=[24773] bw ( KiB/s): min=262144, max=1056768, per=100.00%, avg=658654.30, stdev=165443.52, samples=60 iops : min= 64, max= 258, avg=160.80, stdev=40.39, samples=60 write: IOPS=41, BW=166MiB/s (174MB/s)(4968MiB/30006msec) clat (usec): min=10660, max=72190, avg=20393.59, stdev=4898.88 lat (usec): min=11088, max=72640, avg=20998.40, stdev=4939.74 clat percentiles (usec): | 1.00th=[12780], 5.00th=[13960], 10.00th=[14877], 20.00th=[16057], | 30.00th=[17171], 40.00th=[19006], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23200], 80.00th=[23725], 90.00th=[24773], 95.00th=[25822], | 99.00th=[33424], 99.50th=[42730], 99.90th=[66847], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min=147456, max=188416, per=100.00%, avg=169580.42, stdev=8711.25, samples=60 iops : min= 36, max= 46, avg=41.40, stdev= 2.12, samples=60 lat (usec) : 500=51.94%, 750=8.72%, 1000=0.82% lat (msec) : 2=16.98%, 4=0.02%, 10=0.28%, 20=10.30%, 50=10.88% lat (msec) : 100=0.07% cpu : usr=2.71%, sys=1.89%, ctx=6073, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4825,1242,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=643MiB/s (674MB/s), 643MiB/s-643MiB/s (674MB/s-674MB/s), io=18.8GiB (20.2GB), run=30006-30006msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4968MiB (5209MB), run=30006-30006msec Disk stats (read/write): vdj: ios=19088/4937, merge=0/0, ticks=4997/81170, in_queue=95446, util=96.32% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976340: Fri Jan 7 14:35:38 2022 read: IOPS=161, BW=645MiB/s (676MB/s)(18.9GiB/30017msec) slat (usec): min=73, max=512, avg=96.61, stdev=19.95 clat (usec): min=281, max=22514, avg=725.11, stdev=1502.06 lat (usec): min=420, max=22627, avg=822.00, stdev=1504.27 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 351], 10.00th=[ 355], 20.00th=[ 359], | 30.00th=[ 367], 40.00th=[ 375], 50.00th=[ 383], 60.00th=[ 396], | 70.00th=[ 449], 80.00th=[ 1074], 90.00th=[ 1237], 95.00th=[ 1287], | 99.00th=[ 8717], 99.50th=[14353], 99.90th=[19268], 99.95th=[20317], | 99.99th=[22414] bw ( KiB/s): min=212992, max=1089536, per=100.00%, avg=660909.67, stdev=167273.89, samples=60 iops : min= 52, max= 266, avg=161.33, stdev=40.84, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5000MiB/30017msec) slat (usec): min=258, max=1538, avg=636.14, stdev=310.24 clat (msec): min=10, max=192, avg=20.18, stdev= 6.78 lat (msec): min=11, max=192, avg=20.82, stdev= 6.81 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 33], 99.50th=[ 37], 99.90th=[ 73], 99.95th=[ 192], | 99.99th=[ 192] bw ( KiB/s): min=114688, max=188416, per=99.99%, avg=170553.63, stdev=11839.18, samples=60 iops : min= 28, max= 46, avg=41.63, stdev= 2.89, samples=60 lat (usec) : 500=57.93%, 750=3.27%, 1000=1.56% lat (msec) : 2=15.67%, 4=0.02%, 10=0.33%, 20=10.61%, 50=10.57% lat (msec) : 100=0.03%, 250=0.02% cpu : usr=2.53%, sys=1.98%, ctx=6101, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4840,1250,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=645MiB/s (676MB/s), 645MiB/s-645MiB/s (676MB/s-676MB/s), io=18.9GiB (20.3GB), run=30017-30017msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5000MiB (5243MB), run=30017-30017msec Disk stats (read/write): vdj: ios=19282/4964, merge=0/0, ticks=4711/80452, in_queue=94976, util=96.30% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977101: Fri Jan 7 14:36:15 2022 read: IOPS=162, BW=651MiB/s (682MB/s)(19.1GiB/30009msec) clat (usec): min=376, max=23380, avg=763.64, stdev=1510.89 lat (usec): min=377, max=23380, avg=763.84, stdev=1510.90 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 388], 10.00th=[ 396], 20.00th=[ 400], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 486], 80.00th=[ 1057], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[ 8848], 99.50th=[14353], 99.90th=[19268], 99.95th=[19530], | 99.99th=[23462] bw ( KiB/s): min=344064, max=1048576, per=100.00%, avg=666742.88, stdev=159527.09, samples=60 iops : min= 84, max= 256, avg=162.77, stdev=38.94, samples=60 write: IOPS=41, BW=167MiB/s (176MB/s)(5024MiB/30009msec) clat (usec): min=10579, max=75380, avg=20350.33, stdev=4982.87 lat (usec): min=11019, max=76329, avg=20911.84, stdev=5019.67 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13829], 10.00th=[14746], 20.00th=[15926], | 30.00th=[16909], 40.00th=[18744], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[25822], | 99.00th=[33424], 99.50th=[40633], 99.90th=[64750], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=147751, max=188416, per=99.96%, avg=171359.98, stdev=8689.47, samples=60 iops : min= 36, max= 46, avg=41.83, stdev= 2.12, samples=60 lat (usec) : 500=56.59%, 750=4.40%, 1000=2.04% lat (msec) : 2=15.41%, 4=0.05%, 10=0.33%, 20=10.21%, 50=10.91% lat (msec) : 100=0.07% cpu : usr=2.53%, sys=1.41%, ctx=6148, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4883,1256,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=651MiB/s (682MB/s), 651MiB/s-651MiB/s (682MB/s-682MB/s), io=19.1GiB (20.5GB), run=30009-30009msec WRITE: bw=167MiB/s (176MB/s), 167MiB/s-167MiB/s (176MB/s-176MB/s), io=5024MiB (5268MB), run=30009-30009msec Disk stats (read/write): vdj: ios=19524/5017, merge=0/0, ticks=4555/82217, in_queue=96356, util=96.71% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977959: Fri Jan 7 14:36:52 2022 read: IOPS=191, BW=764MiB/s (801MB/s)(22.5GiB/30191msec) slat (usec): min=39, max=320, avg=61.80, stdev=17.89 clat (usec): min=264, max=1732.9k, avg=5926.55, stdev=69596.01 lat (usec): min=373, max=1732.0k, avg=5988.59, stdev=69595.78 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 334], 10.00th=[ 343], | 20.00th=[ 351], 30.00th=[ 355], 40.00th=[ 363], | 50.00th=[ 371], 60.00th=[ 388], 70.00th=[ 437], | 80.00th=[ 1221], 90.00th=[ 1270], 95.00th=[ 1336], | 99.00th=[ 58459], 99.50th=[ 396362], 99.90th=[1216349], | 99.95th=[1635779], 99.99th=[1736442] bw ( KiB/s): min=385024, max=1368064, per=100.00%, avg=787757.08, stdev=188530.20, samples=60 iops : min= 94, max= 334, avg=192.30, stdev=46.03, samples=60 write: IOPS=49, BW=199MiB/s (209MB/s)(6012MiB/30191msec) slat (usec): min=247, max=2214, avg=797.38, stdev=387.06 clat (msec): min=17, max=2532, avg=176.95, stdev=176.69 lat (msec): min=17, max=2533, avg=177.75, stdev=176.69 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 36], 10.00th=[ 43], 20.00th=[ 58], | 30.00th=[ 79], 40.00th=[ 99], 50.00th=[ 126], 60.00th=[ 165], | 70.00th=[ 207], 80.00th=[ 253], 90.00th=[ 363], 95.00th=[ 464], | 99.00th=[ 718], 99.50th=[ 911], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=180224, max=221184, per=99.98%, avg=203871.33, stdev=8542.08, samples=60 iops : min= 44, max= 54, avg=49.77, stdev= 2.09, samples=60 lat (usec) : 500=57.49%, 750=1.88%, 1000=0.73% lat (msec) : 2=17.63%, 4=0.32%, 10=0.11%, 20=0.23%, 50=2.65% lat (msec) : 100=6.09%, 250=8.17%, 500=3.49%, 750=0.81%, 1000=0.14% lat (msec) : 2000=0.22%, >=2000=0.03% cpu : usr=3.93%, sys=1.59%, ctx=7225, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5769,1503,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=764MiB/s (801MB/s), 764MiB/s-764MiB/s (801MB/s-801MB/s), io=22.5GiB (24.2GB), run=30191-30191msec WRITE: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=6012MiB (6304MB), run=30191-30191msec Disk stats (read/write): vdj: ios=23076/5981, merge=0/0, ticks=39603/1032074, in_queue=1087777, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978563: Fri Jan 7 14:37:29 2022 read: IOPS=162, BW=652MiB/s (683MB/s)(19.1GiB/30016msec) clat (usec): min=372, max=42553, avg=763.24, stdev=1642.42 lat (usec): min=372, max=42554, avg=763.43, stdev=1642.43 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 400], | 30.00th=[ 408], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 474], 80.00th=[ 963], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[ 9241], 99.50th=[14746], 99.90th=[19530], 99.95th=[20317], | 99.99th=[42730] bw ( KiB/s): min=286720, max=1122304, per=100.00%, avg=667761.63, stdev=156424.99, samples=60 iops : min= 70, max= 274, avg=163.02, stdev=38.19, samples=60 write: IOPS=41, BW=168MiB/s (176MB/s)(5028MiB/30016msec) clat (msec): min=10, max=100, avg=20.33, stdev= 5.57 lat (msec): min=11, max=100, avg=20.90, stdev= 5.62 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 26], | 99.00th=[ 34], 99.50th=[ 36], 99.90th=[ 100], 99.95th=[ 101], | 99.99th=[ 101] bw ( KiB/s): min=131072, max=196608, per=99.97%, avg=171479.87, stdev=10209.63, samples=60 iops : min= 32, max= 48, avg=41.85, stdev= 2.48, samples=60 lat (usec) : 500=56.86%, 750=4.52%, 1000=2.67% lat (msec) : 2=14.43%, 4=0.02%, 10=0.29%, 20=10.33%, 50=10.83% lat (msec) : 100=0.03%, 250=0.02% cpu : usr=2.54%, sys=1.26%, ctx=6151, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4891,1257,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=652MiB/s (683MB/s), 652MiB/s-652MiB/s (683MB/s-683MB/s), io=19.1GiB (20.5GB), run=30016-30016msec WRITE: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=5028MiB (5272MB), run=30016-30016msec Disk stats (read/write): vdj: ios=19484/5012, merge=0/0, ticks=4346/82161, in_queue=96163, util=96.45% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979531: Fri Jan 7 14:38:07 2022 read: IOPS=188, BW=753MiB/s (790MB/s)(22.5GiB/30652msec) slat (usec): min=45, max=526366, avg=3934.10, stdev=15063.56 clat (msec): min=18, max=2664, avg=82.61, stdev=122.28 lat (msec): min=18, max=2664, avg=86.55, stdev=123.31 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 28], 20.00th=[ 36], | 30.00th=[ 45], 40.00th=[ 53], 50.00th=[ 62], 60.00th=[ 72], | 70.00th=[ 84], 80.00th=[ 99], 90.00th=[ 129], 95.00th=[ 155], | 99.00th=[ 558], 99.50th=[ 743], 99.90th=[ 1972], 99.95th=[ 2198], | 99.99th=[ 2668] bw ( KiB/s): min= 8192, max=1490944, per=100.00%, avg=772868.20, stdev=329296.95, samples=61 iops : min= 2, max= 364, avg=188.69, stdev=80.39, samples=61 write: IOPS=49, BW=196MiB/s (206MB/s)(6012MiB/30652msec) slat (usec): min=1038, max=483698, avg=4840.32, stdev=15236.47 clat (msec): min=102, max=4278, avg=681.46, stdev=306.80 lat (msec): min=104, max=4280, avg=686.30, stdev=307.12 clat percentiles (msec): | 1.00th=[ 264], 5.00th=[ 397], 10.00th=[ 460], 20.00th=[ 510], | 30.00th=[ 542], 40.00th=[ 575], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 718], 80.00th=[ 818], 90.00th=[ 1003], 95.00th=[ 1133], | 99.00th=[ 1603], 99.50th=[ 1821], 99.90th=[ 4279], 99.95th=[ 4279], | 99.99th=[ 4279] bw ( KiB/s): min= 8192, max=360448, per=99.93%, avg=200704.00, stdev=65942.60, samples=60 iops : min= 2, max= 88, avg=49.00, stdev=16.10, samples=60 lat (msec) : 20=0.19%, 50=28.59%, 100=35.50%, 250=13.39%, 500=4.08% lat (msec) : 750=12.51%, 1000=3.35%, 2000=2.21%, >=2000=0.16% cpu : usr=6.64%, sys=1.70%, ctx=2698, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5771,1503,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=753MiB/s (790MB/s), 753MiB/s-753MiB/s (790MB/s-790MB/s), io=22.5GiB (24.2GB), run=30652-30652msec WRITE: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=6012MiB (6304MB), run=30652-30652msec Disk stats (read/write): vdj: ios=23084/5966, merge=0/0, ticks=156908/3564076, in_queue=3760272, util=99.40% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980496: Fri Jan 7 14:38:43 2022 read: IOPS=161, BW=647MiB/s (678MB/s)(18.0GiB/30020msec) clat (usec): min=373, max=41065, avg=762.04, stdev=1622.49 lat (usec): min=373, max=41066, avg=762.24, stdev=1622.51 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 486], 80.00th=[ 988], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[ 8848], 99.50th=[13829], 99.90th=[19530], 99.95th=[24249], | 99.99th=[41157] bw ( KiB/s): min=286720, max=1089536, per=100.00%, avg=663005.87, stdev=163595.91, samples=60 iops : min= 70, max= 266, avg=161.87, stdev=39.94, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5008MiB/30020msec) clat (usec): min=10881, max=95452, avg=20447.44, stdev=6064.28 lat (usec): min=11301, max=95854, avg=21010.03, stdev=6096.95 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13698], 10.00th=[14615], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20579], 60.00th=[22152], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[25560], | 99.00th=[36963], 99.50th=[56361], 99.90th=[93848], 99.95th=[95945], | 99.99th=[95945] bw ( KiB/s): min=131072, max=188416, per=99.99%, avg=170803.20, stdev=11568.04, samples=60 iops : min= 32, max= 46, avg=41.70, stdev= 2.82, samples=60 lat (usec) : 500=56.27%, 750=5.19%, 1000=2.26% lat (msec) : 2=14.72%, 4=0.02%, 10=0.31%, 20=10.30%, 50=10.79% lat (msec) : 100=0.15% cpu : usr=2.57%, sys=1.47%, ctx=6118, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4856,1252,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=647MiB/s (678MB/s), 647MiB/s-647MiB/s (678MB/s-678MB/s), io=18.0GiB (20.4GB), run=30020-30020msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5008MiB (5251MB), run=30020-30020msec Disk stats (read/write): vdj: ios=19316/4973, merge=0/0, ticks=4340/81751, in_queue=95429, util=96.09% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981076: Fri Jan 7 14:39:21 2022 read: IOPS=189, BW=758MiB/s (794MB/s)(22.6GiB/30611msec) slat (usec): min=45, max=470497, avg=3968.42, stdev=14850.80 clat (msec): min=79, max=2609, avg=293.27, stdev=146.77 lat (msec): min=79, max=2612, avg=297.24, stdev=147.29 clat percentiles (msec): | 1.00th=[ 92], 5.00th=[ 123], 10.00th=[ 165], 20.00th=[ 203], | 30.00th=[ 228], 40.00th=[ 249], 50.00th=[ 275], 60.00th=[ 296], | 70.00th=[ 317], 80.00th=[ 359], 90.00th=[ 422], 95.00th=[ 550], | 99.00th=[ 751], 99.50th=[ 835], 99.90th=[ 1955], 99.95th=[ 2567], | 99.99th=[ 2601] bw ( KiB/s): min= 8192, max=1531904, per=99.46%, avg=771485.39, stdev=307816.46, samples=61 iops : min= 2, max= 374, avg=188.33, stdev=75.13, samples=61 write: IOPS=49, BW=197MiB/s (207MB/s)(6044MiB/30611msec) slat (usec): min=1131, max=450072, avg=4614.89, stdev=14004.99 clat (msec): min=206, max=3390, avg=877.35, stdev=252.84 lat (msec): min=207, max=3391, avg=881.97, stdev=253.02 clat percentiles (msec): | 1.00th=[ 472], 5.00th=[ 600], 10.00th=[ 651], 20.00th=[ 718], | 30.00th=[ 751], 40.00th=[ 793], 50.00th=[ 827], 60.00th=[ 885], | 70.00th=[ 936], 80.00th=[ 1011], 90.00th=[ 1150], 95.00th=[ 1267], | 99.00th=[ 1620], 99.50th=[ 1787], 99.90th=[ 3272], 99.95th=[ 3406], | 99.99th=[ 3406] bw ( KiB/s): min=32768, max=344064, per=100.00%, avg=203433.78, stdev=66963.12, samples=59 iops : min= 8, max= 84, avg=49.66, stdev=16.35, samples=59 lat (msec) : 100=1.86%, 250=30.19%, 500=41.97%, 750=10.65%, 1000=10.62% lat (msec) : 2000=4.56%, >=2000=0.16% cpu : usr=6.72%, sys=1.67%, ctx=2799, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5797,1511,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=758MiB/s (794MB/s), 758MiB/s-758MiB/s (794MB/s-794MB/s), io=22.6GiB (24.3GB), run=30611-30611msec WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=6044MiB (6338MB), run=30611-30611msec Disk stats (read/write): vdj: ios=23188/6029, merge=0/0, ticks=140207/3603867, in_queue=3770470, util=99.09% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981902: Fri Jan 7 14:39:57 2022 read: IOPS=161, BW=646MiB/s (677MB/s)(18.9GiB/30001msec) clat (usec): min=377, max=21148, avg=753.69, stdev=1495.20 lat (usec): min=377, max=21148, avg=753.88, stdev=1495.21 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 469], 80.00th=[ 996], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[ 9110], 99.50th=[14353], 99.90th=[19006], 99.95th=[19530], | 99.99th=[21103] bw ( KiB/s): min=352256, max=1097728, per=99.82%, avg=659870.71, stdev=155893.51, samples=59 iops : min= 86, max= 268, avg=161.08, stdev=38.06, samples=59 write: IOPS=41, BW=167MiB/s (175MB/s)(5000MiB/30001msec) clat (usec): min=10611, max=73025, avg=20517.69, stdev=5404.62 lat (usec): min=11033, max=73856, avg=21069.41, stdev=5452.17 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13960], 10.00th=[14746], 20.00th=[15926], | 30.00th=[17171], 40.00th=[18482], 50.00th=[20579], 60.00th=[22414], | 70.00th=[23200], 80.00th=[23987], 90.00th=[25035], 95.00th=[25822], | 99.00th=[36963], 99.50th=[47973], 99.90th=[72877], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min=122880, max=196608, per=100.00%, avg=170800.44, stdev=11066.70, samples=59 iops : min= 30, max= 48, avg=41.69, stdev= 2.70, samples=59 lat (usec) : 500=57.06%, 750=4.10%, 1000=2.54% lat (msec) : 2=14.71%, 4=0.02%, 10=0.33%, 20=10.33%, 50=10.82% lat (msec) : 100=0.10% cpu : usr=2.46%, sys=1.76%, ctx=6099, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4842,1250,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=646MiB/s (677MB/s), 646MiB/s-646MiB/s (677MB/s-677MB/s), io=18.9GiB (20.3GB), run=30001-30001msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5000MiB (5243MB), run=30001-30001msec Disk stats (read/write): vdj: ios=19288/4964, merge=0/0, ticks=4171/81732, in_queue=95373, util=95.28% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983249: Fri Jan 7 14:40:35 2022 read: IOPS=188, BW=755MiB/s (792MB/s)(22.6GiB/30584msec) slat (usec): min=45, max=458462, avg=3969.26, stdev=14494.08 clat (msec): min=389, max=3057, avg=711.54, stdev=166.87 lat (msec): min=389, max=3057, avg=715.51, stdev=167.57 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 485], 10.00th=[ 514], 20.00th=[ 584], | 30.00th=[ 634], 40.00th=[ 667], 50.00th=[ 701], 60.00th=[ 735], | 70.00th=[ 768], 80.00th=[ 827], 90.00th=[ 894], 95.00th=[ 986], | 99.00th=[ 1083], 99.50th=[ 1267], 99.90th=[ 2165], 99.95th=[ 2802], | 99.99th=[ 3071] bw ( KiB/s): min= 8192, max=1302528, per=97.81%, avg=756653.70, stdev=284898.36, samples=61 iops : min= 2, max= 318, avg=184.69, stdev=69.54, samples=61 write: IOPS=49, BW=197MiB/s (206MB/s)(6016MiB/30584msec) slat (usec): min=1015, max=418540, avg=4679.96, stdev=13672.98 clat (msec): min=548, max=4231, avg=1289.55, stdev=328.98 lat (msec): min=586, max=4233, avg=1294.23, stdev=328.50 clat percentiles (msec): | 1.00th=[ 718], 5.00th=[ 911], 10.00th=[ 995], 20.00th=[ 1099], | 30.00th=[ 1167], 40.00th=[ 1217], 50.00th=[ 1250], 60.00th=[ 1301], | 70.00th=[ 1368], 80.00th=[ 1435], 90.00th=[ 1552], 95.00th=[ 1703], | 99.00th=[ 2123], 99.50th=[ 3742], 99.90th=[ 4245], 99.95th=[ 4245], | 99.99th=[ 4245] bw ( KiB/s): min=24576, max=434176, per=99.83%, avg=201091.12, stdev=87550.56, samples=59 iops : min= 6, max= 106, avg=49.08, stdev=21.37, samples=59 lat (msec) : 500=6.10%, 750=46.18%, 1000=25.67%, 2000=21.55%, >=2000=0.49% cpu : usr=6.80%, sys=1.76%, ctx=2738, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5776,1504,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=755MiB/s (792MB/s), 755MiB/s-755MiB/s (792MB/s-792MB/s), io=22.6GiB (24.2GB), run=30584-30584msec WRITE: bw=197MiB/s (206MB/s), 197MiB/s-197MiB/s (206MB/s-206MB/s), io=6016MiB (6308MB), run=30584-30584msec Disk stats (read/write): vdj: ios=23104/5976, merge=0/0, ticks=145344/3578687, in_queue=3760855, util=98.41%