30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890672: Fri Jan 7 13:33:20 2022 read: IOPS=2370, BW=9481KiB/s (9708kB/s)(278MiB/30002msec) clat (usec): min=25, max=21907, avg=59.43, stdev=392.61 lat (usec): min=25, max=21907, avg=59.57, stdev=392.61 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 465], 99.95th=[11994], | 99.99th=[16581] bw ( KiB/s): min= 7600, max=14304, per=100.00%, avg=9481.02, stdev=1112.42, samples=60 iops : min= 1900, max= 3576, avg=2370.25, stdev=278.11, samples=60 write: IOPS=593, BW=2372KiB/s (2429kB/s)(69.5MiB/30002msec) clat (usec): min=195, max=48337, avg=1439.37, stdev=1700.84 lat (usec): min=195, max=48337, avg=1439.66, stdev=1700.84 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 562], 10.00th=[ 783], 20.00th=[ 996], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2057], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[40109], | 99.99th=[42730] bw ( KiB/s): min= 1888, max= 3680, per=100.00%, avg=2372.32, stdev=259.14, samples=60 iops : min= 472, max= 920, avg=593.07, stdev=64.78, samples=60 lat (usec) : 50=51.58%, 100=28.06%, 250=0.36%, 500=0.74%, 750=0.99% lat (usec) : 1000=2.23% lat (msec) : 2=14.88%, 4=0.71%, 10=0.33%, 20=0.10%, 50=0.02% cpu : usr=1.30%, sys=3.86%, ctx=88914, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71110,17794,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9481KiB/s (9708kB/s), 9481KiB/s-9481KiB/s (9708kB/s-9708kB/s), io=278MiB (291MB), run=30002-30002msec WRITE: bw=2372KiB/s (2429kB/s), 2372KiB/s-2372KiB/s (2429kB/s-2429kB/s), io=69.5MiB (72.9MB), run=30002-30002msec Disk stats (read/write): vdg: ios=71098/17788, merge=0/0, ticks=741/17942, in_queue=28339, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891548: Fri Jan 7 13:33:57 2022 read: IOPS=2402, BW=9612KiB/s (9842kB/s)(282MiB/30001msec) slat (usec): min=5, max=234, avg= 9.76, stdev= 5.10 clat (nsec): min=645, max=23922k, avg=45954.10, stdev=380967.75 lat (usec): min=23, max=23933, avg=55.86, stdev=381.00 clat percentiles (usec): | 1.00th=[ 19], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 38], 80.00th=[ 41], 90.00th=[ 45], 95.00th=[ 50], | 99.00th=[ 63], 99.50th=[ 72], 99.90th=[ 260], 99.95th=[11863], | 99.99th=[16319] bw ( KiB/s): min= 8152, max=14554, per=100.00%, avg=9613.07, stdev=1011.02, samples=60 iops : min= 2038, max= 3638, avg=2403.22, stdev=252.72, samples=60 write: IOPS=601, BW=2408KiB/s (2465kB/s)(70.5MiB/30001msec) slat (usec): min=6, max=234, avg=10.92, stdev= 6.08 clat (usec): min=180, max=43515, avg=1419.07, stdev=1672.02 lat (usec): min=190, max=43523, avg=1430.15, stdev=1672.02 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 586], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2008], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 2104, max= 3784, per=100.00%, avg=2407.87, stdev=245.68, samples=60 iops : min= 526, max= 946, avg=601.95, stdev=61.42, samples=60 lat (nsec) : 750=0.04%, 1000=0.20% lat (usec) : 2=0.17%, 4=0.02%, 10=0.07%, 20=0.45%, 50=75.32% lat (usec) : 100=3.57%, 250=0.19%, 500=0.67%, 750=0.89%, 1000=2.11% lat (msec) : 2=15.23%, 4=0.67%, 10=0.29%, 20=0.10%, 50=0.02% cpu : usr=1.21%, sys=6.02%, ctx=89660, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72089,18058,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9612KiB/s (9842kB/s), 9612KiB/s-9612KiB/s (9842kB/s-9842kB/s), io=282MiB (295MB), run=30001-30001msec WRITE: bw=2408KiB/s (2465kB/s), 2408KiB/s-2408KiB/s (2465kB/s-2465kB/s), io=70.5MiB (73.0MB), run=30001-30001msec Disk stats (read/write): vdg: ios=72088/18056, merge=0/0, ticks=734/18239, in_queue=28326, util=93.87% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892544: Fri Jan 7 13:34:34 2022 read: IOPS=2388, BW=9553KiB/s (9783kB/s)(280MiB/30006msec) clat (usec): min=25, max=65214, avg=60.31, stdev=497.24 lat (usec): min=25, max=65214, avg=60.45, stdev=497.24 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 66], | 99.00th=[ 84], 99.50th=[ 95], 99.90th=[ 334], 99.95th=[12256], | 99.99th=[15926] bw ( KiB/s): min= 7472, max=15520, per=100.00%, avg=9554.83, stdev=1213.66, samples=60 iops : min= 1868, max= 3880, avg=2388.70, stdev=303.41, samples=60 write: IOPS=597, BW=2392KiB/s (2449kB/s)(70.1MiB/30006msec) clat (usec): min=187, max=44020, avg=1421.87, stdev=1661.99 lat (usec): min=187, max=44020, avg=1422.16, stdev=1661.99 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 586], 10.00th=[ 791], 20.00th=[ 1004], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2008], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[17171], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min= 1872, max= 3984, per=100.00%, avg=2392.17, stdev=291.44, samples=60 iops : min= 468, max= 996, avg=598.03, stdev=72.85, samples=60 lat (usec) : 50=51.33%, 100=28.33%, 250=0.31%, 500=0.72%, 750=0.99% lat (usec) : 1000=2.18% lat (msec) : 2=15.05%, 4=0.67%, 10=0.30%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.34%, sys=3.92%, ctx=89615, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71664,17943,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9553KiB/s (9783kB/s), 9553KiB/s-9553KiB/s (9783kB/s-9783kB/s), io=280MiB (294MB), run=30006-30006msec WRITE: bw=2392KiB/s (2449kB/s), 2392KiB/s-2392KiB/s (2449kB/s-2449kB/s), io=70.1MiB (73.5MB), run=30006-30006msec Disk stats (read/write): vdg: ios=71661/17941, merge=0/0, ticks=822/17789, in_queue=28338, util=94.01% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892996: Fri Jan 7 13:35:11 2022 read: IOPS=2433, BW=9732KiB/s (9966kB/s)(286MiB/30050msec) slat (usec): min=2, max=153, avg= 9.64, stdev= 3.67 clat (nsec): min=665, max=2493.3M, avg=884734.08, stdev=37971747.64 lat (usec): min=22, max=2493.3k, avg=894.52, stdev=37971.77 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 41], | 80.00th=[ 44], 90.00th=[ 50], 95.00th=[ 57], | 99.00th=[ 83], 99.50th=[ 103], 99.90th=[ 506], | 99.95th=[ 549454], 99.99th=[2499806] bw ( KiB/s): min= 7552, max=16960, per=100.00%, avg=9747.50, stdev=1264.28, samples=60 iops : min= 1888, max= 4240, avg=2436.87, stdev=316.07, samples=60 write: IOPS=608, BW=2433KiB/s (2491kB/s)(71.4MiB/30050msec) slat (usec): min=2, max=146, avg=10.82, stdev= 3.98 clat (usec): min=257, max=81303, avg=12835.52, stdev=6598.40 lat (usec): min=263, max=81318, avg=12846.50, stdev=6598.42 clat percentiles (usec): | 1.00th=[ 1270], 5.00th=[ 4293], 10.00th=[ 7242], 20.00th=[ 9503], | 30.00th=[10945], 40.00th=[11731], 50.00th=[12518], 60.00th=[13042], | 70.00th=[13698], 80.00th=[14615], 90.00th=[19006], 95.00th=[21627], | 99.00th=[31327], 99.50th=[63177], 99.90th=[70779], 99.95th=[73925], | 99.99th=[81265] bw ( KiB/s): min= 1976, max= 4264, per=100.00%, avg=2436.73, stdev=300.66, samples=60 iops : min= 494, max= 1066, avg=609.18, stdev=75.16, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.06%, 4=0.01%, 10=0.04%, 20=0.25%, 50=72.03% lat (usec) : 100=7.16%, 250=0.34%, 500=0.03%, 750=0.02%, 1000=0.05% lat (msec) : 2=0.45%, 4=0.42%, 10=3.71%, 20=13.66%, 50=1.53% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.41%, sys=5.84%, ctx=89251, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73113,18276,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9732KiB/s (9966kB/s), 9732KiB/s-9732KiB/s (9966kB/s-9966kB/s), io=286MiB (299MB), run=30050-30050msec WRITE: bw=2433KiB/s (2491kB/s), 2433KiB/s-2433KiB/s (2491kB/s-2491kB/s), io=71.4MiB (74.9MB), run=30050-30050msec Disk stats (read/write): vdg: ios=73043/18262, merge=0/0, ticks=52722/225310, in_queue=297992, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894409: Fri Jan 7 13:35:48 2022 read: IOPS=2383, BW=9536KiB/s (9765kB/s)(279MiB/30001msec) clat (usec): min=25, max=23028, avg=57.73, stdev=370.96 lat (usec): min=25, max=23028, avg=57.87, stdev=370.96 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 81], 99.50th=[ 92], 99.90th=[ 355], 99.95th=[10945], | 99.99th=[16057] bw ( KiB/s): min= 7960, max=15288, per=100.00%, avg=9536.13, stdev=1183.62, samples=60 iops : min= 1990, max= 3822, avg=2384.03, stdev=295.91, samples=60 write: IOPS=596, BW=2387KiB/s (2445kB/s)(69.9MiB/30001msec) clat (usec): min=193, max=43540, avg=1434.08, stdev=1685.34 lat (usec): min=193, max=43540, avg=1434.35, stdev=1685.34 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 578], 10.00th=[ 791], 20.00th=[ 1004], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min= 1864, max= 3944, per=100.00%, avg=2387.33, stdev=285.98, samples=60 iops : min= 466, max= 986, avg=596.83, stdev=71.49, samples=60 lat (usec) : 50=54.73%, 100=24.96%, 250=0.29%, 500=0.70%, 750=0.97% lat (usec) : 1000=2.25% lat (msec) : 2=14.93%, 4=0.73%, 10=0.32%, 20=0.09%, 50=0.02% cpu : usr=1.43%, sys=3.75%, ctx=89437, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71521,17906,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9536KiB/s (9765kB/s), 9536KiB/s-9536KiB/s (9765kB/s-9765kB/s), io=279MiB (293MB), run=30001-30001msec WRITE: bw=2387KiB/s (2445kB/s), 2387KiB/s-2387KiB/s (2445kB/s-2445kB/s), io=69.9MiB (73.3MB), run=30001-30001msec Disk stats (read/write): vdg: ios=71402/17863, merge=0/0, ticks=705/17913, in_queue=28349, util=93.97% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894979: Fri Jan 7 13:36:25 2022 read: IOPS=2403, BW=9615KiB/s (9846kB/s)(283MiB/30099msec) slat (usec): min=2, max=243, avg= 9.57, stdev= 4.58 clat (nsec): min=654, max=2513.3M, avg=795557.43, stdev=34630419.59 lat (usec): min=18, max=2513.3k, avg=805.28, stdev=34630.41 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 37], 60.00th=[ 40], 70.00th=[ 43], | 80.00th=[ 47], 90.00th=[ 58], 95.00th=[ 76], | 99.00th=[ 202], 99.50th=[ 412], 99.90th=[ 1139], | 99.95th=[ 450888], 99.99th=[2365588] bw ( KiB/s): min= 6528, max=17888, per=100.00%, avg=9645.57, stdev=1432.77, samples=60 iops : min= 1632, max= 4472, avg=2411.38, stdev=358.19, samples=60 write: IOPS=601, BW=2406KiB/s (2464kB/s)(70.7MiB/30099msec) slat (usec): min=2, max=135, avg=10.86, stdev= 4.47 clat (msec): min=6, max=2565, avg=79.84, stdev=77.18 lat (msec): min=7, max=2565, avg=79.85, stdev=77.18 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 69], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 93], 95.00th=[ 118], | 99.00th=[ 253], 99.50th=[ 567], 99.90th=[ 1183], 99.95th=[ 1368], | 99.99th=[ 2534] bw ( KiB/s): min= 1776, max= 4232, per=100.00%, avg=2408.22, stdev=307.59, samples=60 iops : min= 444, max= 1058, avg=602.03, stdev=76.89, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.10%, 4=0.01%, 10=0.05%, 20=0.19%, 50=66.99% lat (usec) : 100=10.71%, 250=1.23%, 500=0.38%, 750=0.15%, 1000=0.04% lat (msec) : 2=0.04%, 4=0.01%, 10=0.16%, 20=0.12%, 50=0.21% lat (msec) : 100=17.97%, 250=1.35%, 500=0.11%, 750=0.05%, 1000=0.05% lat (msec) : 2000=0.03%, >=2000=0.02% cpu : usr=1.19%, sys=5.85%, ctx=86945, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=72352,18107,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9615KiB/s (9846kB/s), 9615KiB/s-9615KiB/s (9846kB/s-9846kB/s), io=283MiB (296MB), run=30099-30099msec WRITE: bw=2406KiB/s (2464kB/s), 2406KiB/s-2406KiB/s (2464kB/s-2464kB/s), io=70.7MiB (74.2MB), run=30099-30099msec Disk stats (read/write): vdg: ios=72274/18048, merge=0/0, ticks=50997/1427921, in_queue=1496491, util=99.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895821: Fri Jan 7 13:37:02 2022 read: IOPS=2386, BW=9548KiB/s (9777kB/s)(280MiB/30002msec) clat (usec): min=24, max=62947, avg=59.54, stdev=468.06 lat (usec): min=24, max=62947, avg=59.67, stdev=468.06 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 83], 99.50th=[ 94], 99.90th=[ 371], 99.95th=[11863], | 99.99th=[17171] bw ( KiB/s): min= 7504, max=14520, per=100.00%, avg=9548.27, stdev=1169.56, samples=60 iops : min= 1876, max= 3630, avg=2387.07, stdev=292.39, samples=60 write: IOPS=597, BW=2390KiB/s (2447kB/s)(70.0MiB/30002msec) clat (usec): min=207, max=47828, avg=1426.25, stdev=1661.95 lat (usec): min=207, max=47828, avg=1426.54, stdev=1661.95 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 570], 10.00th=[ 799], 20.00th=[ 1004], | 30.00th=[ 1123], 40.00th=[ 1188], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2040], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[23200], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min= 1976, max= 3752, per=100.00%, avg=2390.13, stdev=262.46, samples=60 iops : min= 494, max= 938, avg=597.53, stdev=65.62, samples=60 lat (usec) : 50=54.37%, 100=25.28%, 250=0.31%, 500=0.73%, 750=0.94% lat (usec) : 1000=2.27% lat (msec) : 2=14.94%, 4=0.72%, 10=0.30%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.26%, sys=3.86%, ctx=89547, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71612,17927,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9548KiB/s (9777kB/s), 9548KiB/s-9548KiB/s (9777kB/s-9777kB/s), io=280MiB (293MB), run=30002-30002msec WRITE: bw=2390KiB/s (2447kB/s), 2390KiB/s-2390KiB/s (2447kB/s-2447kB/s), io=70.0MiB (73.4MB), run=30002-30002msec Disk stats (read/write): vdg: ios=71512/17901, merge=0/0, ticks=823/17872, in_queue=28349, util=94.19% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896946: Fri Jan 7 13:37:39 2022 read: IOPS=2425, BW=9701KiB/s (9934kB/s)(286MiB/30216msec) slat (usec): min=2, max=320, avg= 9.49, stdev= 5.20 clat (nsec): min=709, max=2511.1M, avg=986327.52, stdev=40000090.01 lat (usec): min=20, max=2511.1k, avg=995.96, stdev=40000.09 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 49], 90.00th=[ 66], 95.00th=[ 122], | 99.00th=[ 775], 99.50th=[ 1319], 99.90th=[ 3818], | 99.95th=[ 599786], 99.99th=[2432697] bw ( KiB/s): min= 7608, max=17704, per=100.00%, avg=9772.43, stdev=1355.25, samples=60 iops : min= 1902, max= 4426, avg=2443.08, stdev=338.82, samples=60 write: IOPS=605, BW=2424KiB/s (2482kB/s)(71.5MiB/30216msec) slat (usec): min=2, max=244, avg=10.77, stdev= 5.60 clat (msec): min=6, max=2879, avg=160.93, stdev=135.03 lat (msec): min=6, max=2879, avg=160.94, stdev=135.03 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 107], 10.00th=[ 116], 20.00th=[ 128], | 30.00th=[ 138], 40.00th=[ 142], 50.00th=[ 148], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 192], 95.00th=[ 211], | 99.00th=[ 550], 99.50th=[ 1099], 99.90th=[ 2869], 99.95th=[ 2869], | 99.99th=[ 2869] bw ( KiB/s): min= 2008, max= 3792, per=100.00%, avg=2430.12, stdev=260.58, samples=60 iops : min= 502, max= 948, avg=607.52, stdev=65.14, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.08%, 4=0.01%, 10=0.03%, 20=0.22%, 50=64.49% lat (usec) : 100=10.21%, 250=2.95%, 500=0.77%, 750=0.36%, 1000=0.23% lat (msec) : 2=0.36%, 4=0.16%, 10=0.01%, 20=0.26%, 50=0.16% lat (msec) : 100=0.22%, 250=18.73%, 500=0.43%, 750=0.08%, 1000=0.04% lat (msec) : 2000=0.10%, >=2000=0.04% cpu : usr=1.24%, sys=5.75%, ctx=84997, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73280,18310,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9701KiB/s (9934kB/s), 9701KiB/s-9701KiB/s (9934kB/s-9934kB/s), io=286MiB (300MB), run=30216-30216msec WRITE: bw=2424KiB/s (2482kB/s), 2424KiB/s-2424KiB/s (2482kB/s-2482kB/s), io=71.5MiB (74.0MB), run=30216-30216msec Disk stats (read/write): vdg: ios=73214/18210, merge=0/0, ticks=59609/2909516, in_queue=2995349, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897509: Fri Jan 7 13:38:16 2022 read: IOPS=2373, BW=9493KiB/s (9721kB/s)(278MiB/30001msec) clat (usec): min=24, max=21243, avg=61.24, stdev=402.41 lat (usec): min=24, max=21243, avg=61.39, stdev=402.41 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 388], 99.95th=[12125], | 99.99th=[16909] bw ( KiB/s): min= 7448, max=14360, per=100.00%, avg=9493.47, stdev=1169.22, samples=60 iops : min= 1862, max= 3590, avg=2373.37, stdev=292.31, samples=60 write: IOPS=593, BW=2376KiB/s (2433kB/s)(69.6MiB/30001msec) clat (usec): min=193, max=45776, avg=1429.38, stdev=1731.70 lat (usec): min=194, max=45776, avg=1429.67, stdev=1731.70 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 553], 10.00th=[ 775], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2040], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min= 1840, max= 3704, per=100.00%, avg=2375.60, stdev=265.99, samples=60 iops : min= 460, max= 926, avg=593.90, stdev=66.50, samples=60 lat (usec) : 50=47.10%, 100=32.44%, 250=0.43%, 500=0.79%, 750=1.05% lat (usec) : 1000=2.32% lat (msec) : 2=14.74%, 4=0.69%, 10=0.32%, 20=0.11%, 50=0.02% cpu : usr=1.36%, sys=3.98%, ctx=89028, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71201,17818,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9493KiB/s (9721kB/s), 9493KiB/s-9493KiB/s (9721kB/s-9721kB/s), io=278MiB (292MB), run=30001-30001msec WRITE: bw=2376KiB/s (2433kB/s), 2376KiB/s-2376KiB/s (2433kB/s-2433kB/s), io=69.6MiB (72.0MB), run=30001-30001msec Disk stats (read/write): vdg: ios=71110/17793, merge=0/0, ticks=761/17771, in_queue=28286, util=93.91% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898370: Fri Jan 7 13:38:53 2022 read: IOPS=2428, BW=9714KiB/s (9947kB/s)(286MiB/30201msec) slat (usec): min=2, max=211907, avg=329.75, stdev=2097.49 clat (usec): min=539, max=2549.6k, avg=26516.24, stdev=40999.90 lat (usec): min=542, max=2549.6k, avg=26846.13, stdev=41054.89 clat percentiles (usec): | 1.00th=[ 1045], 5.00th=[ 8586], 10.00th=[ 14222], | 20.00th=[ 17695], 30.00th=[ 20055], 40.00th=[ 21890], | 50.00th=[ 23725], 60.00th=[ 25560], 70.00th=[ 27919], | 80.00th=[ 31065], 90.00th=[ 36439], 95.00th=[ 42730], | 99.00th=[ 85459], 99.50th=[ 189793], 99.90th=[ 250610], | 99.95th=[ 513803], 99.99th=[2499806] bw ( KiB/s): min= 7216, max=19824, per=100.00%, avg=9769.52, stdev=1645.52, samples=60 iops : min= 1804, max= 4956, avg=2442.37, stdev=411.38, samples=60 write: IOPS=606, BW=2427KiB/s (2485kB/s)(71.6MiB/30201msec) slat (usec): min=2, max=169762, avg=307.46, stdev=1854.62 clat (msec): min=13, max=3357, avg=221.68, stdev=174.95 lat (msec): min=13, max=3357, avg=221.99, stdev=174.94 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 140], 10.00th=[ 159], 20.00th=[ 176], | 30.00th=[ 186], 40.00th=[ 194], 50.00th=[ 203], 60.00th=[ 209], | 70.00th=[ 220], 80.00th=[ 234], 90.00th=[ 264], 95.00th=[ 296], | 99.00th=[ 894], 99.50th=[ 1569], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 3373] bw ( KiB/s): min= 1920, max= 4136, per=100.00%, avg=2426.22, stdev=313.96, samples=60 iops : min= 480, max= 1034, avg=606.55, stdev=78.49, samples=60 lat (usec) : 750=0.21%, 1000=0.51% lat (msec) : 2=1.21%, 4=1.61%, 10=0.78%, 20=19.76%, 50=53.63% lat (msec) : 100=2.28%, 250=17.02%, 500=2.62%, 750=0.11%, 1000=0.06% lat (msec) : 2000=0.14%, >=2000=0.07% cpu : usr=0.63%, sys=1.93%, ctx=9979, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73341,18324,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9714KiB/s (9947kB/s), 9714KiB/s-9714KiB/s (9947kB/s-9947kB/s), io=286MiB (300MB), run=30201-30201msec WRITE: bw=2427KiB/s (2485kB/s), 2427KiB/s-2427KiB/s (2485kB/s-2485kB/s), io=71.6MiB (75.1MB), run=30201-30201msec Disk stats (read/write): vdg: ios=73236/18193, merge=0/0, ticks=61229/3562526, in_queue=3657445, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898989: Fri Jan 7 13:39:30 2022 read: IOPS=2339, BW=36.6MiB/s (38.3MB/s)(1097MiB/30002msec) clat (usec): min=25, max=24060, avg=62.77, stdev=383.42 lat (usec): min=25, max=24060, avg=62.90, stdev=383.42 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 72], | 99.00th=[ 91], 99.50th=[ 103], 99.90th=[ 453], 99.95th=[11731], | 99.99th=[16450] bw ( KiB/s): min=28480, max=59296, per=100.00%, avg=37437.48, stdev=4691.76, samples=60 iops : min= 1780, max= 3706, avg=2339.83, stdev=293.24, samples=60 write: IOPS=585, BW=9372KiB/s (9597kB/s)(275MiB/30002msec) clat (usec): min=239, max=45685, avg=1446.45, stdev=1660.55 lat (usec): min=240, max=45687, avg=1447.90, stdev=1660.53 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 578], 10.00th=[ 807], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1778], 95.00th=[ 2057], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min= 7712, max=15296, per=100.00%, avg=9372.27, stdev=1078.22, samples=60 iops : min= 482, max= 956, avg=585.77, stdev=67.39, samples=60 lat (usec) : 50=40.43%, 100=39.09%, 250=0.36%, 500=0.74%, 750=0.97% lat (usec) : 1000=1.92% lat (msec) : 2=15.29%, 4=0.79%, 10=0.30%, 20=0.09%, 50=0.03% cpu : usr=1.37%, sys=3.90%, ctx=87773, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70191,17573,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.6MiB/s (38.3MB/s), 36.6MiB/s-36.6MiB/s (38.3MB/s-38.3MB/s), io=1097MiB (1150MB), run=30002-30002msec WRITE: bw=9372KiB/s (9597kB/s), 9372KiB/s-9372KiB/s (9597kB/s-9597kB/s), io=275MiB (288MB), run=30002-30002msec Disk stats (read/write): vdg: ios=69777/17445, merge=0/0, ticks=717/17713, in_queue=28105, util=94.08% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899805: Fri Jan 7 13:40:07 2022 read: IOPS=2343, BW=36.6MiB/s (38.4MB/s)(1099MiB/30003msec) slat (usec): min=4, max=391, avg=11.61, stdev= 8.70 clat (nsec): min=645, max=28273k, avg=48944.89, stdev=394557.38 lat (usec): min=23, max=28280, avg=60.75, stdev=394.65 clat percentiles (nsec): | 1.00th=[ 1288], 5.00th=[ 26496], 10.00th=[ 28800], | 20.00th=[ 31616], 30.00th=[ 33024], 40.00th=[ 34560], | 50.00th=[ 36608], 60.00th=[ 38656], 70.00th=[ 40704], | 80.00th=[ 44288], 90.00th=[ 49408], 95.00th=[ 55040], | 99.00th=[ 72192], 99.50th=[ 83456], 99.90th=[ 415744], | 99.95th=[12386304], 99.99th=[16187392] bw ( KiB/s): min=26176, max=58164, per=100.00%, avg=37508.88, stdev=5206.99, samples=60 iops : min= 1636, max= 3635, avg=2344.27, stdev=325.40, samples=60 write: IOPS=586, BW=9386KiB/s (9612kB/s)(275MiB/30003msec) slat (usec): min=7, max=350, avg=14.22, stdev= 9.20 clat (usec): min=233, max=43343, avg=1438.52, stdev=1645.68 lat (usec): min=243, max=43361, avg=1452.93, stdev=1645.69 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 578], 10.00th=[ 807], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2040], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[38536], | 99.99th=[42730] bw ( KiB/s): min= 6880, max=14973, per=100.00%, avg=9388.08, stdev=1132.94, samples=60 iops : min= 430, max= 935, avg=586.73, stdev=70.74, samples=60 lat (nsec) : 750=0.08%, 1000=0.43% lat (usec) : 2=0.62%, 4=0.04%, 10=0.19%, 20=0.91%, 50=70.20% lat (usec) : 100=7.27%, 250=0.15%, 500=0.77%, 750=0.93%, 1000=1.99% lat (msec) : 2=15.27%, 4=0.73%, 10=0.29%, 20=0.10%, 50=0.02% cpu : usr=1.56%, sys=6.40%, ctx=86489, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70318,17601,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=36.6MiB/s (38.4MB/s), 36.6MiB/s-36.6MiB/s (38.4MB/s-38.4MB/s), io=1099MiB (1152MB), run=30003-30003msec WRITE: bw=9386KiB/s (9612kB/s), 9386KiB/s-9386KiB/s (9612kB/s-9612kB/s), io=275MiB (288MB), run=30003-30003msec Disk stats (read/write): vdg: ios=69863/17488, merge=0/0, ticks=720/17808, in_queue=27991, util=93.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901193: Fri Jan 7 13:40:44 2022 read: IOPS=2309, BW=36.1MiB/s (37.8MB/s)(1082MiB/30001msec) clat (usec): min=25, max=64247, avg=60.81, stdev=453.15 lat (usec): min=25, max=64247, avg=60.95, stdev=453.16 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 404], 99.95th=[11731], | 99.99th=[17433] bw ( KiB/s): min=27232, max=56352, per=100.00%, avg=36968.51, stdev=4593.99, samples=59 iops : min= 1702, max= 3522, avg=2310.54, stdev=287.15, samples=59 write: IOPS=577, BW=9238KiB/s (9459kB/s)(271MiB/30001msec) clat (usec): min=230, max=45100, avg=1478.87, stdev=1695.49 lat (usec): min=231, max=45102, avg=1480.27, stdev=1695.50 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 611], 10.00th=[ 848], 20.00th=[ 1057], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1352], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1811], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[17695], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min= 7168, max=14368, per=100.00%, avg=9253.80, stdev=1028.67, samples=59 iops : min= 448, max= 898, avg=578.36, stdev=64.29, samples=59 lat (usec) : 50=49.01%, 100=30.66%, 250=0.25%, 500=0.73%, 750=0.79% lat (usec) : 1000=1.76% lat (msec) : 2=15.51%, 4=0.84%, 10=0.33%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.43%, sys=3.63%, ctx=86608, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69279,17321,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=36.1MiB/s (37.8MB/s), 36.1MiB/s-36.1MiB/s (37.8MB/s-37.8MB/s), io=1082MiB (1135MB), run=30001-30001msec WRITE: bw=9238KiB/s (9459kB/s), 9238KiB/s-9238KiB/s (9459kB/s-9459kB/s), io=271MiB (284MB), run=30001-30001msec Disk stats (read/write): vdg: ios=68774/17197, merge=0/0, ticks=743/18093, in_queue=28224, util=94.43% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901854: Fri Jan 7 13:41:21 2022 read: IOPS=2380, BW=37.2MiB/s (39.0MB/s)(1117MiB/30039msec) slat (usec): min=2, max=249, avg=11.35, stdev= 6.15 clat (nsec): min=633, max=2491.8M, avg=779198.02, stdev=34664488.14 lat (usec): min=21, max=2491.8k, avg=790.71, stdev=34664.48 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 37], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 57], 95.00th=[ 66], | 99.00th=[ 112], 99.50th=[ 165], 99.90th=[ 988], | 99.95th=[ 480248], 99.99th=[2332034] bw ( KiB/s): min=28928, max=64672, per=100.00%, avg=38141.23, stdev=4681.32, samples=60 iops : min= 1808, max= 4042, avg=2383.82, stdev=292.59, samples=60 write: IOPS=595, BW=9535KiB/s (9764kB/s)(280MiB/30039msec) slat (usec): min=4, max=494, avg=14.49, stdev= 8.60 clat (usec): min=320, max=82313, avg=13587.92, stdev=6507.64 lat (usec): min=369, max=82338, avg=13602.58, stdev=6507.67 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 6128], 10.00th=[ 7767], 20.00th=[10421], | 30.00th=[11600], 40.00th=[12387], 50.00th=[13173], 60.00th=[13698], | 70.00th=[14353], 80.00th=[15139], 90.00th=[19268], 95.00th=[22152], | 99.00th=[34866], 99.50th=[62653], 99.90th=[76022], 99.95th=[79168], | 99.99th=[82314] bw ( KiB/s): min= 7872, max=16352, per=100.00%, avg=9545.90, stdev=1121.73, samples=60 iops : min= 492, max= 1022, avg=596.62, stdev=70.11, samples=60 lat (nsec) : 750=0.03%, 1000=0.17% lat (usec) : 2=0.10%, 4=0.01%, 10=0.04%, 20=0.16%, 50=64.73% lat (usec) : 100=13.70%, 250=0.81%, 500=0.11%, 750=0.04%, 1000=0.02% lat (msec) : 2=0.22%, 4=0.09%, 10=3.11%, 20=14.74%, 50=1.68% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.65%, sys=6.36%, ctx=87444, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71512,17902,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1117MiB (1172MB), run=30039-30039msec WRITE: bw=9535KiB/s (9764kB/s), 9535KiB/s-9535KiB/s (9764kB/s-9764kB/s), io=280MiB (293MB), run=30039-30039msec Disk stats (read/write): vdg: ios=71095/17782, merge=0/0, ticks=51004/232968, in_queue=296064, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902751: Fri Jan 7 13:41:58 2022 read: IOPS=2307, BW=36.1MiB/s (37.8MB/s)(1082MiB/30002msec) clat (usec): min=25, max=26535, avg=61.76, stdev=384.55 lat (usec): min=25, max=26535, avg=61.90, stdev=384.55 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 71], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 306], 99.95th=[11600], | 99.99th=[16909] bw ( KiB/s): min=29824, max=57824, per=100.00%, avg=36929.05, stdev=4534.48, samples=60 iops : min= 1864, max= 3614, avg=2308.05, stdev=283.42, samples=60 write: IOPS=577, BW=9235KiB/s (9457kB/s)(271MiB/30002msec) clat (usec): min=229, max=45192, avg=1475.24, stdev=1694.94 lat (usec): min=230, max=45193, avg=1476.68, stdev=1694.96 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 603], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1795], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[39060], | 99.99th=[41157] bw ( KiB/s): min= 7424, max=14816, per=100.00%, avg=9235.47, stdev=1080.64, samples=60 iops : min= 464, max= 926, avg=577.22, stdev=67.54, samples=60 lat (usec) : 50=42.60%, 100=36.95%, 250=0.37%, 500=0.70%, 750=0.95% lat (usec) : 1000=1.92% lat (msec) : 2=15.24%, 4=0.83%, 10=0.34%, 20=0.09%, 50=0.02% cpu : usr=1.39%, sys=3.82%, ctx=86565, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69240,17317,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=36.1MiB/s (37.8MB/s), 36.1MiB/s-36.1MiB/s (37.8MB/s-37.8MB/s), io=1082MiB (1134MB), run=30002-30002msec WRITE: bw=9235KiB/s (9457kB/s), 9235KiB/s-9235KiB/s (9457kB/s-9457kB/s), io=271MiB (284MB), run=30002-30002msec Disk stats (read/write): vdg: ios=68941/17232, merge=0/0, ticks=700/17900, in_queue=28207, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903784: Fri Jan 7 13:42:35 2022 read: IOPS=2377, BW=37.1MiB/s (38.9MB/s)(1118MiB/30099msec) slat (usec): min=2, max=233, avg= 9.88, stdev= 3.83 clat (nsec): min=768, max=2525.5M, avg=1044044.80, stdev=42019461.76 lat (usec): min=23, max=2525.5k, avg=1054.06, stdev=42019.48 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 49], 90.00th=[ 60], 95.00th=[ 77], | 99.00th=[ 227], 99.50th=[ 474], 99.90th=[ 1254], | 99.95th=[ 750781], 99.99th=[2499806] bw ( KiB/s): min=29952, max=71488, per=100.00%, avg=38161.23, stdev=5266.40, samples=60 iops : min= 1872, max= 4468, avg=2385.07, stdev=329.15, samples=60 write: IOPS=595, BW=9524KiB/s (9752kB/s)(280MiB/30099msec) slat (usec): min=3, max=105, avg=12.86, stdev= 4.09 clat (msec): min=8, max=2522, avg=79.75, stdev=68.43 lat (msec): min=8, max=2522, avg=79.77, stdev=68.43 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 67], 40.00th=[ 69], 50.00th=[ 72], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 94], 95.00th=[ 121], | 99.00th=[ 262], 99.50th=[ 550], 99.90th=[ 1062], 99.95th=[ 1351], | 99.99th=[ 1989] bw ( KiB/s): min= 7840, max=16896, per=100.00%, avg=9532.05, stdev=1183.85, samples=60 iops : min= 490, max= 1056, avg=595.75, stdev=73.99, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.05%, 50=65.56% lat (usec) : 100=12.24%, 250=1.30%, 500=0.38%, 750=0.19%, 1000=0.09% lat (msec) : 2=0.03%, 4=0.01%, 10=0.04%, 20=0.26%, 50=0.23% lat (msec) : 100=17.82%, 250=1.49%, 500=0.10%, 750=0.06%, 1000=0.04% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=1.35%, sys=5.73%, ctx=86647, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=71556,17916,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1118MiB (1172MB), run=30099-30099msec WRITE: bw=9524KiB/s (9752kB/s), 9524KiB/s-9524KiB/s (9752kB/s-9752kB/s), io=280MiB (294MB), run=30099-30099msec Disk stats (read/write): vdg: ios=71550/17890, merge=0/0, ticks=66549/1415084, in_queue=1499408, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904248: Fri Jan 7 13:43:12 2022 read: IOPS=2324, BW=36.3MiB/s (38.1MB/s)(1090MiB/30001msec) clat (usec): min=26, max=23867, avg=61.83, stdev=385.80 lat (usec): min=26, max=23867, avg=61.97, stdev=385.80 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 91], 99.50th=[ 104], 99.90th=[ 343], 99.95th=[11863], | 99.99th=[16712] bw ( KiB/s): min=29129, max=53088, per=100.00%, avg=37352.45, stdev=4238.80, samples=60 iops : min= 1820, max= 3318, avg=2334.10, stdev=264.99, samples=60 write: IOPS=581, BW=9300KiB/s (9523kB/s)(272MiB/30001msec) clat (usec): min=242, max=43151, avg=1462.36, stdev=1678.29 lat (usec): min=244, max=43152, avg=1463.84, stdev=1678.29 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 594], 10.00th=[ 824], 20.00th=[ 1037], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1434], 80.00th=[ 1549], 90.00th=[ 1795], 95.00th=[ 2073], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[37487], | 99.99th=[42730] bw ( KiB/s): min= 7129, max=13760, per=100.00%, avg=9337.87, stdev=996.63, samples=60 iops : min= 445, max= 860, avg=583.33, stdev=62.36, samples=60 lat (usec) : 50=43.17%, 100=36.34%, 250=0.41%, 500=0.71%, 750=0.92% lat (usec) : 1000=1.87% lat (msec) : 2=15.33%, 4=0.83%, 10=0.30%, 20=0.09%, 50=0.03% cpu : usr=1.58%, sys=3.70%, ctx=87189, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69743,17438,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=36.3MiB/s (38.1MB/s), 36.3MiB/s-36.3MiB/s (38.1MB/s-38.1MB/s), io=1090MiB (1143MB), run=30001-30001msec WRITE: bw=9300KiB/s (9523kB/s), 9300KiB/s-9300KiB/s (9523kB/s-9523kB/s), io=272MiB (286MB), run=30001-30001msec Disk stats (read/write): vdg: ios=69736/17435, merge=0/0, ticks=713/17939, in_queue=28246, util=93.70% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905219: Fri Jan 7 13:43:49 2022 read: IOPS=2350, BW=36.7MiB/s (38.5MB/s)(1109MiB/30197msec) slat (usec): min=2, max=282, avg= 9.79, stdev= 5.52 clat (nsec): min=749, max=2533.5M, avg=999915.62, stdev=40919980.46 lat (usec): min=23, max=2533.5k, avg=1009.84, stdev=40919.98 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 51], 90.00th=[ 69], 95.00th=[ 135], | 99.00th=[ 996], 99.50th=[ 1565], 99.90th=[ 5080], | 99.95th=[ 574620], 99.99th=[2499806] bw ( KiB/s): min=29408, max=75264, per=100.00%, avg=37853.55, stdev=5939.28, samples=60 iops : min= 1838, max= 4704, avg=2365.83, stdev=371.20, samples=60 write: IOPS=587, BW=9406KiB/s (9632kB/s)(277MiB/30197msec) slat (usec): min=4, max=940, avg=12.97, stdev= 9.00 clat (msec): min=11, max=3388, avg=165.98, stdev=136.88 lat (msec): min=11, max=3388, avg=165.99, stdev=136.88 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 116], 10.00th=[ 124], 20.00th=[ 134], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 153], 60.00th=[ 157], | 70.00th=[ 165], 80.00th=[ 176], 90.00th=[ 199], 95.00th=[ 222], | 99.00th=[ 531], 99.50th=[ 844], 99.90th=[ 2802], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min= 7488, max=16352, per=100.00%, avg=9417.38, stdev=1135.51, samples=60 iops : min= 468, max= 1022, avg=588.58, stdev=70.97, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.12%, 50=63.24% lat (usec) : 100=11.00%, 250=3.07%, 500=0.90%, 750=0.46%, 1000=0.27% lat (msec) : 2=0.53%, 4=0.12%, 10=0.08%, 20=0.15%, 50=0.21% lat (msec) : 100=0.24%, 250=18.73%, 500=0.49%, 750=0.09%, 1000=0.04% lat (msec) : 2000=0.07%, >=2000=0.04% cpu : usr=1.31%, sys=5.59%, ctx=82764, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70977,17753,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=36.7MiB/s (38.5MB/s), 36.7MiB/s-36.7MiB/s (38.5MB/s-38.5MB/s), io=1109MiB (1163MB), run=30197-30197msec WRITE: bw=9406KiB/s (9632kB/s), 9406KiB/s-9406KiB/s (9632kB/s-9632kB/s), io=277MiB (291MB), run=30197-30197msec Disk stats (read/write): vdg: ios=70931/17647, merge=0/0, ticks=63097/2908657, in_queue=2995901, util=99.58% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905793: Fri Jan 7 13:44:26 2022 read: IOPS=2327, BW=36.4MiB/s (38.1MB/s)(1091MiB/30001msec) clat (usec): min=27, max=21720, avg=60.38, stdev=383.07 lat (usec): min=27, max=21720, avg=60.52, stdev=383.07 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 84], 99.50th=[ 95], 99.90th=[ 351], 99.95th=[11731], | 99.99th=[17433] bw ( KiB/s): min=31040, max=56832, per=100.00%, avg=37239.15, stdev=4007.70, samples=60 iops : min= 1940, max= 3552, avg=2327.43, stdev=250.48, samples=60 write: IOPS=582, BW=9314KiB/s (9538kB/s)(273MiB/30001msec) clat (usec): min=234, max=43602, avg=1466.66, stdev=1685.16 lat (usec): min=236, max=43603, avg=1468.09, stdev=1685.15 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 611], 10.00th=[ 832], 20.00th=[ 1057], | 30.00th=[ 1156], 40.00th=[ 1221], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1434], 80.00th=[ 1565], 90.00th=[ 1778], 95.00th=[ 2057], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 8032, max=14464, per=100.00%, avg=9314.47, stdev=910.27, samples=60 iops : min= 502, max= 904, avg=582.15, stdev=56.89, samples=60 lat (usec) : 50=47.11%, 100=32.57%, 250=0.24%, 500=0.70%, 750=0.90% lat (usec) : 1000=1.80% lat (msec) : 2=15.47%, 4=0.78%, 10=0.30%, 20=0.10%, 50=0.02% cpu : usr=1.31%, sys=3.85%, ctx=87294, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69821,17465,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=36.4MiB/s (38.1MB/s), 36.4MiB/s-36.4MiB/s (38.1MB/s-38.1MB/s), io=1091MiB (1144MB), run=30001-30001msec WRITE: bw=9314KiB/s (9538kB/s), 9314KiB/s-9314KiB/s (9538kB/s-9538kB/s), io=273MiB (286MB), run=30001-30001msec Disk stats (read/write): vdg: ios=69358/17339, merge=0/0, ticks=705/17954, in_queue=28064, util=93.86% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906617: Fri Jan 7 13:45:03 2022 read: IOPS=2381, BW=37.2MiB/s (39.0MB/s)(1126MiB/30251msec) slat (usec): min=2, max=198603, avg=333.00, stdev=2032.82 clat (usec): min=695, max=2578.2k, avg=27092.97, stdev=42743.68 lat (usec): min=698, max=2578.3k, avg=27426.12, stdev=42799.10 clat percentiles (usec): | 1.00th=[ 1385], 5.00th=[ 6521], 10.00th=[ 14222], | 20.00th=[ 18220], 30.00th=[ 20317], 40.00th=[ 22414], | 50.00th=[ 24249], 60.00th=[ 26346], 70.00th=[ 28705], | 80.00th=[ 31851], 90.00th=[ 36963], 95.00th=[ 42730], | 99.00th=[ 83362], 99.50th=[ 193987], 99.90th=[ 263193], | 99.95th=[ 599786], 99.99th=[2533360] bw ( KiB/s): min=29984, max=78816, per=100.00%, avg=38388.92, stdev=6640.19, samples=60 iops : min= 1874, max= 4926, avg=2399.28, stdev=415.01, samples=60 write: IOPS=596, BW=9545KiB/s (9774kB/s)(282MiB/30251msec) slat (usec): min=3, max=203812, avg=323.42, stdev=2159.69 clat (msec): min=11, max=2582, avg=225.34, stdev=125.48 lat (msec): min=12, max=2582, avg=225.67, stdev=125.48 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 148], 10.00th=[ 171], 20.00th=[ 188], | 30.00th=[ 197], 40.00th=[ 205], 50.00th=[ 211], 60.00th=[ 218], | 70.00th=[ 226], 80.00th=[ 241], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 852], 99.50th=[ 1116], 99.90th=[ 1921], 99.95th=[ 2022], | 99.99th=[ 2500] bw ( KiB/s): min= 7392, max=16544, per=100.00%, avg=9556.13, stdev=1255.72, samples=60 iops : min= 462, max= 1034, avg=597.25, stdev=78.48, samples=60 lat (usec) : 750=0.01%, 1000=0.13% lat (msec) : 2=1.66%, 4=1.65%, 10=1.33%, 20=17.81%, 50=55.24% lat (msec) : 100=1.97%, 250=16.81%, 500=3.03%, 750=0.11%, 1000=0.10% lat (msec) : 2000=0.13%, >=2000=0.03% cpu : usr=0.82%, sys=1.93%, ctx=10000, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=72041,18047,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1126MiB (1180MB), run=30251-30251msec WRITE: bw=9545KiB/s (9774kB/s), 9545KiB/s-9545KiB/s (9774kB/s-9774kB/s), io=282MiB (296MB), run=30251-30251msec Disk stats (read/write): vdg: ios=72035/17935, merge=0/0, ticks=63194/3566323, in_queue=3664830, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908078: Fri Jan 7 13:45:40 2022 read: IOPS=2257, BW=70.5MiB/s (73.0MB/s)(2117MiB/30001msec) clat (usec): min=33, max=28929, avg=66.67, stdev=415.50 lat (usec): min=33, max=28929, avg=66.80, stdev=415.50 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 74], | 99.00th=[ 90], 99.50th=[ 100], 99.90th=[ 502], 99.95th=[12649], | 99.99th=[17695] bw ( KiB/s): min=56832, max=112768, per=100.00%, avg=72247.63, stdev=9114.42, samples=60 iops : min= 1776, max= 3524, avg=2257.73, stdev=284.83, samples=60 write: IOPS=564, BW=17.6MiB/s (18.5MB/s)(529MiB/30001msec) clat (usec): min=272, max=91684, avg=1493.98, stdev=1783.60 lat (usec): min=274, max=91687, avg=1496.91, stdev=1783.57 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 545], 10.00th=[ 799], 20.00th=[ 1057], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1598], 90.00th=[ 1844], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[39584], | 99.99th=[42730] bw ( KiB/s): min=14976, max=28800, per=100.00%, avg=18061.38, stdev=2183.79, samples=60 iops : min= 468, max= 900, avg=564.42, stdev=68.24, samples=60 lat (usec) : 50=31.22%, 100=48.39%, 250=0.30%, 500=0.83%, 750=1.00% lat (usec) : 1000=1.59% lat (msec) : 2=15.28%, 4=0.97%, 10=0.30%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.31%, sys=3.91%, ctx=84673, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67730,16933,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.0MB/s), 70.5MiB/s-70.5MiB/s (73.0MB/s-73.0MB/s), io=2117MiB (2219MB), run=30001-30001msec WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=529MiB (555MB), run=30001-30001msec Disk stats (read/write): vdg: ios=67342/16831, merge=0/0, ticks=766/17581, in_queue=28152, util=94.11% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908674: Fri Jan 7 13:46:17 2022 read: IOPS=2233, BW=69.8MiB/s (73.2MB/s)(2094MiB/30001msec) slat (usec): min=5, max=171, avg=11.24, stdev= 4.30 clat (nsec): min=652, max=22321k, avg=51696.19, stdev=398304.05 lat (usec): min=25, max=22330, avg=63.08, stdev=398.35 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 35], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 41], | 70.00th=[ 43], 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 56], | 99.00th=[ 71], 99.50th=[ 79], 99.90th=[ 412], 99.95th=[13042], | 99.99th=[16188] bw ( KiB/s): min=53952, max=116160, per=100.00%, avg=71489.23, stdev=9299.82, samples=60 iops : min= 1686, max= 3630, avg=2234.02, stdev=290.63, samples=60 write: IOPS=558, BW=17.5MiB/s (18.3MB/s)(524MiB/30001msec) slat (usec): min=9, max=135, avg=14.94, stdev= 4.79 clat (usec): min=254, max=47225, avg=1514.32, stdev=1699.98 lat (usec): min=267, max=47245, avg=1529.43, stdev=1700.04 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 611], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[39060], | 99.99th=[43779] bw ( KiB/s): min=13440, max=30016, per=100.00%, avg=17888.57, stdev=2162.59, samples=60 iops : min= 420, max= 938, avg=559.02, stdev=67.58, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.07%, 4=0.01%, 10=0.02%, 20=0.08%, 50=70.83% lat (usec) : 100=8.78%, 250=0.07%, 500=0.65%, 750=0.86%, 1000=1.56% lat (msec) : 2=15.55%, 4=1.03%, 10=0.32%, 20=0.11%, 50=0.02% cpu : usr=1.44%, sys=5.90%, ctx=83655, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67015,16770,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=69.8MiB/s (73.2MB/s), 69.8MiB/s-69.8MiB/s (73.2MB/s-73.2MB/s), io=2094MiB (2196MB), run=30001-30001msec WRITE: bw=17.5MiB/s (18.3MB/s), 17.5MiB/s-17.5MiB/s (18.3MB/s-18.3MB/s), io=524MiB (550MB), run=30001-30001msec Disk stats (read/write): vdg: ios=66569/16667, merge=0/0, ticks=738/17966, in_queue=28094, util=93.98% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909514: Fri Jan 7 13:46:54 2022 read: IOPS=2256, BW=70.5MiB/s (73.9MB/s)(2115MiB/30001msec) clat (usec): min=28, max=22805, avg=62.70, stdev=396.64 lat (usec): min=28, max=22806, avg=62.84, stdev=396.64 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 86], 99.50th=[ 97], 99.90th=[ 490], 99.95th=[11994], | 99.99th=[17695] bw ( KiB/s): min=52352, max=116288, per=99.84%, avg=72079.58, stdev=9233.80, samples=59 iops : min= 1636, max= 3634, avg=2252.47, stdev=288.54, samples=59 write: IOPS=564, BW=17.6MiB/s (18.5MB/s)(529MiB/30001msec) clat (usec): min=283, max=47135, avg=1510.99, stdev=1665.55 lat (usec): min=286, max=47136, avg=1513.89, stdev=1665.52 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 635], 10.00th=[ 881], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1401], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[39584], | 99.99th=[45351] bw ( KiB/s): min=13248, max=30080, per=99.96%, avg=18040.85, stdev=2199.70, samples=59 iops : min= 414, max= 940, avg=563.76, stdev=68.74, samples=59 lat (usec) : 50=42.79%, 100=36.87%, 250=0.24%, 500=0.62%, 750=0.81% lat (usec) : 1000=1.45% lat (msec) : 2=15.77%, 4=1.06%, 10=0.26%, 20=0.11%, 50=0.02% cpu : usr=1.38%, sys=3.76%, ctx=84618, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67687,16921,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=70.5MiB/s (73.9MB/s), 70.5MiB/s-70.5MiB/s (73.9MB/s-73.9MB/s), io=2115MiB (2218MB), run=30001-30001msec WRITE: bw=17.6MiB/s (18.5MB/s), 17.6MiB/s-17.6MiB/s (18.5MB/s-18.5MB/s), io=529MiB (554MB), run=30001-30001msec Disk stats (read/write): vdg: ios=67687/16921, merge=0/0, ticks=731/18000, in_queue=28433, util=94.46% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910179: Fri Jan 7 13:47:31 2022 read: IOPS=2299, BW=71.9MiB/s (75.4MB/s)(2158MiB/30025msec) slat (usec): min=3, max=161, avg=11.23, stdev= 4.75 clat (nsec): min=707, max=2536.5M, avg=1035653.69, stdev=42685862.49 lat (usec): min=26, max=2536.5k, avg=1047.03, stdev=42685.84 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 37], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 44], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 57], 95.00th=[ 63], | 99.00th=[ 92], 99.50th=[ 120], 99.90th=[ 537], | 99.95th=[ 692061], 99.99th=[2499806] bw ( KiB/s): min=59328, max=120576, per=100.00%, avg=73660.23, stdev=9075.95, samples=60 iops : min= 1854, max= 3768, avg=2301.87, stdev=283.61, samples=60 write: IOPS=574, BW=17.0MiB/s (18.8MB/s)(539MiB/30025msec) slat (usec): min=5, max=183, avg=16.16, stdev= 5.99 clat (usec): min=477, max=75314, avg=13172.92, stdev=6603.93 lat (usec): min=488, max=75335, avg=13189.24, stdev=6604.13 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 4359], 10.00th=[ 6521], 20.00th=[ 9765], | 30.00th=[11207], 40.00th=[12125], 50.00th=[12911], 60.00th=[13566], | 70.00th=[14353], 80.00th=[15401], 90.00th=[18220], 95.00th=[22152], | 99.00th=[32375], 99.50th=[62129], 99.90th=[70779], 99.95th=[72877], | 99.99th=[74974] bw ( KiB/s): min=14912, max=30592, per=100.00%, avg=18403.87, stdev=2049.67, samples=60 iops : min= 466, max= 956, avg=575.12, stdev=64.05, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.10%, 50=64.63% lat (usec) : 100=14.48%, 250=0.53%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.21%, 4=0.66%, 10=3.35%, 20=14.14%, 50=1.50% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.61%, sys=5.84%, ctx=85147, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69056,17261,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=2158MiB (2263MB), run=30025-30025msec WRITE: bw=17.0MiB/s (18.8MB/s), 17.0MiB/s-17.0MiB/s (18.8MB/s-18.8MB/s), io=539MiB (566MB), run=30025-30025msec Disk stats (read/write): vdg: ios=68588/17156, merge=0/0, ticks=68030/217120, in_queue=296456, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910970: Fri Jan 7 13:48:08 2022 read: IOPS=2240, BW=70.0MiB/s (73.4MB/s)(2100MiB/30001msec) clat (usec): min=27, max=31687, avg=64.92, stdev=426.95 lat (usec): min=27, max=31687, avg=65.06, stdev=426.95 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 72], | 99.00th=[ 92], 99.50th=[ 109], 99.90th=[ 660], 99.95th=[12780], | 99.99th=[16712] bw ( KiB/s): min=56192, max=106496, per=100.00%, avg=71689.60, stdev=8510.20, samples=60 iops : min= 1756, max= 3328, avg=2240.30, stdev=265.94, samples=60 write: IOPS=560, BW=17.5MiB/s (18.4MB/s)(525MiB/30001msec) clat (usec): min=270, max=43828, avg=1513.57, stdev=1700.98 lat (usec): min=273, max=43830, avg=1516.53, stdev=1700.96 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 635], 10.00th=[ 857], 20.00th=[ 1090], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1860], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min=14912, max=27840, per=100.00%, avg=17928.53, stdev=1854.09, samples=60 iops : min= 466, max= 870, avg=560.27, stdev=57.94, samples=60 lat (usec) : 50=39.33%, 100=40.13%, 250=0.41%, 500=0.63%, 750=0.88% lat (usec) : 1000=1.58% lat (msec) : 2=15.60%, 4=1.01%, 10=0.29%, 20=0.12%, 50=0.02% cpu : usr=1.45%, sys=3.88%, ctx=84024, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67209,16809,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=70.0MiB/s (73.4MB/s), 70.0MiB/s-70.0MiB/s (73.4MB/s-73.4MB/s), io=2100MiB (2202MB), run=30001-30001msec WRITE: bw=17.5MiB/s (18.4MB/s), 17.5MiB/s-17.5MiB/s (18.4MB/s-18.4MB/s), io=525MiB (551MB), run=30001-30001msec Disk stats (read/write): vdg: ios=66777/16715, merge=0/0, ticks=785/17799, in_queue=28173, util=94.10% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911959: Fri Jan 7 13:48:45 2022 read: IOPS=2281, BW=71.3MiB/s (74.8MB/s)(2147MiB/30102msec) slat (usec): min=2, max=154, avg=10.84, stdev= 4.62 clat (nsec): min=704, max=2526.0M, avg=1000451.62, stdev=41066546.91 lat (usec): min=24, max=2526.0k, avg=1011.45, stdev=41066.56 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 70], | 99.00th=[ 196], 99.50th=[ 449], 99.90th=[ 1303], | 99.95th=[ 574620], 99.99th=[2499806] bw ( KiB/s): min=55872, max=138432, per=100.00%, avg=73276.05, stdev=10692.64, samples=60 iops : min= 1746, max= 4326, avg=2289.85, stdev=334.14, samples=60 write: IOPS=570, BW=17.8MiB/s (18.7MB/s)(537MiB/30102msec) slat (usec): min=5, max=293, avg=16.25, stdev= 6.33 clat (msec): min=8, max=2126, avg=83.49, stdev=65.92 lat (msec): min=8, max=2126, avg=83.51, stdev=65.92 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 61], 10.00th=[ 64], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 75], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 99], 95.00th=[ 125], | 99.00th=[ 230], 99.50th=[ 506], 99.90th=[ 1116], 99.95th=[ 1318], | 99.99th=[ 1888] bw ( KiB/s): min=15296, max=32192, per=100.00%, avg=18280.97, stdev=2177.25, samples=60 iops : min= 478, max= 1006, avg=571.27, stdev=68.03, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.14%, 50=64.36% lat (usec) : 100=13.69%, 250=1.00%, 500=0.31%, 750=0.13%, 1000=0.10% lat (msec) : 2=0.06%, 10=0.02%, 20=0.25%, 50=0.18%, 100=17.74% lat (msec) : 250=1.65%, 500=0.10%, 750=0.06%, 1000=0.03%, 2000=0.04% lat (msec) : >=2000=0.01% cpu : usr=1.58%, sys=5.71%, ctx=83471, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=68688,17184,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=71.3MiB/s (74.8MB/s), 71.3MiB/s-71.3MiB/s (74.8MB/s-74.8MB/s), io=2147MiB (2251MB), run=30102-30102msec WRITE: bw=17.8MiB/s (18.7MB/s), 17.8MiB/s-17.8MiB/s (18.7MB/s-18.7MB/s), io=537MiB (563MB), run=30102-30102msec Disk stats (read/write): vdg: ios=68687/17159, merge=0/0, ticks=65100/1420440, in_queue=1499571, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912589: Fri Jan 7 13:49:22 2022 read: IOPS=2259, BW=70.6MiB/s (74.0MB/s)(2118MiB/30002msec) clat (usec): min=27, max=40623, avg=64.07, stdev=436.82 lat (usec): min=27, max=40623, avg=64.21, stdev=436.82 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 91], 99.50th=[ 104], 99.90th=[ 510], 99.95th=[13435], | 99.99th=[16909] bw ( KiB/s): min=57920, max=124096, per=100.00%, avg=72301.87, stdev=9322.62, samples=60 iops : min= 1810, max= 3878, avg=2259.43, stdev=291.33, samples=60 write: IOPS=565, BW=17.7MiB/s (18.5MB/s)(530MiB/30002msec) clat (usec): min=274, max=43652, avg=1502.07, stdev=1671.10 lat (usec): min=277, max=43654, avg=1505.01, stdev=1671.10 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 627], 10.00th=[ 881], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1614], 90.00th=[ 1827], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[36439], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=15168, max=31808, per=100.00%, avg=18083.20, stdev=2226.41, samples=60 iops : min= 474, max= 994, avg=565.10, stdev=69.58, samples=60 lat (usec) : 50=42.21%, 100=37.29%, 250=0.37%, 500=0.64%, 750=0.86% lat (usec) : 1000=1.42% lat (msec) : 2=15.87%, 4=0.95%, 10=0.27%, 20=0.11%, 50=0.03% cpu : usr=1.34%, sys=3.85%, ctx=84744, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67783,16954,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=70.6MiB/s (74.0MB/s), 70.6MiB/s-70.6MiB/s (74.0MB/s-74.0MB/s), io=2118MiB (2221MB), run=30002-30002msec WRITE: bw=17.7MiB/s (18.5MB/s), 17.7MiB/s-17.7MiB/s (18.5MB/s-18.5MB/s), io=530MiB (556MB), run=30002-30002msec Disk stats (read/write): vdg: ios=67369/16835, merge=0/0, ticks=795/17835, in_queue=28152, util=94.18% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913423: Fri Jan 7 13:49:59 2022 read: IOPS=2269, BW=70.9MiB/s (74.4MB/s)(2141MiB/30187msec) slat (usec): min=2, max=193, avg=11.16, stdev= 4.83 clat (nsec): min=781, max=2519.0M, avg=1122748.42, stdev=44138730.28 lat (usec): min=25, max=2520.0k, avg=1134.06, stdev=44138.71 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 43], 60.00th=[ 46], 70.00th=[ 49], | 80.00th=[ 55], 90.00th=[ 66], 95.00th=[ 96], | 99.00th=[ 971], 99.50th=[ 1860], 99.90th=[ 3097], | 99.95th=[ 750781], 99.99th=[2499806] bw ( KiB/s): min=57728, max=139456, per=100.00%, avg=73085.57, stdev=11312.70, samples=60 iops : min= 1804, max= 4358, avg=2283.90, stdev=353.54, samples=60 write: IOPS=567, BW=17.7MiB/s (18.6MB/s)(536MiB/30187msec) slat (usec): min=6, max=203, avg=17.71, stdev= 6.77 clat (msec): min=11, max=3057, avg=171.52, stdev=154.19 lat (msec): min=11, max=3057, avg=171.54, stdev=154.19 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 97], 10.00th=[ 120], 20.00th=[ 136], | 30.00th=[ 148], 40.00th=[ 153], 50.00th=[ 159], 60.00th=[ 163], | 70.00th=[ 171], 80.00th=[ 180], 90.00th=[ 205], 95.00th=[ 224], | 99.00th=[ 502], 99.50th=[ 1653], 99.90th=[ 2400], 99.95th=[ 2735], | 99.99th=[ 3004] bw ( KiB/s): min=14557, max=29248, per=100.00%, avg=18180.37, stdev=2024.94, samples=60 iops : min= 454, max= 914, avg=568.10, stdev=63.33, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.05%, 50=58.05% lat (usec) : 100=17.94%, 250=1.58%, 500=0.79%, 750=0.46%, 1000=0.22% lat (msec) : 2=0.44%, 4=0.28%, 20=0.12%, 50=0.26%, 100=0.72% lat (msec) : 250=18.31%, 500=0.43%, 750=0.05%, 1000=0.03%, 2000=0.10% lat (msec) : >=2000=0.07% cpu : usr=1.70%, sys=5.79%, ctx=80717, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=68510,17141,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=70.9MiB/s (74.4MB/s), 70.9MiB/s-70.9MiB/s (74.4MB/s-74.4MB/s), io=2141MiB (2245MB), run=30187-30187msec WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=536MiB (562MB), run=30187-30187msec Disk stats (read/write): vdg: ios=68509/17061, merge=0/0, ticks=72303/2904444, in_queue=3001469, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914737: Fri Jan 7 13:50:36 2022 read: IOPS=2238, BW=69.9MiB/s (73.3MB/s)(2099MiB/30002msec) clat (usec): min=27, max=26357, avg=63.85, stdev=420.94 lat (usec): min=27, max=26357, avg=64.00, stdev=420.94 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 88], 99.50th=[ 101], 99.90th=[ 449], 99.95th=[12911], | 99.99th=[18482] bw ( KiB/s): min=54080, max=115968, per=100.00%, avg=71626.08, stdev=9364.09, samples=60 iops : min= 1690, max= 3624, avg=2238.27, stdev=292.62, samples=60 write: IOPS=559, BW=17.5MiB/s (18.3MB/s)(525MiB/30002msec) clat (usec): min=275, max=46769, avg=1519.33, stdev=1673.51 lat (usec): min=279, max=46772, avg=1522.30, stdev=1673.49 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 635], 10.00th=[ 881], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1270], 50.00th=[ 1336], 60.00th=[ 1401], | 70.00th=[ 1500], 80.00th=[ 1631], 90.00th=[ 1876], 95.00th=[ 2147], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=14656, max=29760, per=99.99%, avg=17915.08, stdev=2059.27, samples=60 iops : min= 458, max= 930, avg=559.80, stdev=64.38, samples=60 lat (usec) : 50=41.79%, 100=37.79%, 250=0.30%, 500=0.62%, 750=0.78% lat (usec) : 1000=1.49% lat (msec) : 2=15.67%, 4=1.14%, 10=0.26%, 20=0.12%, 50=0.02% cpu : usr=1.45%, sys=3.71%, ctx=83962, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67156,16798,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=69.9MiB/s (73.3MB/s), 69.9MiB/s-69.9MiB/s (73.3MB/s-73.3MB/s), io=2099MiB (2201MB), run=30002-30002msec WRITE: bw=17.5MiB/s (18.3MB/s), 17.5MiB/s-17.5MiB/s (18.3MB/s-18.3MB/s), io=525MiB (550MB), run=30002-30002msec Disk stats (read/write): vdg: ios=67156/16797, merge=0/0, ticks=777/17981, in_queue=28383, util=94.27% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915353: Fri Jan 7 13:51:13 2022 read: IOPS=2288, BW=71.5MiB/s (74.0MB/s)(2161MiB/30216msec) slat (usec): min=2, max=203315, avg=345.04, stdev=2215.67 clat (usec): min=775, max=2560.1k, avg=28083.34, stdev=41775.60 lat (usec): min=779, max=2562.0k, avg=28428.52, stdev=41847.24 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 5276], 10.00th=[ 15008], | 20.00th=[ 19006], 30.00th=[ 21365], 40.00th=[ 23462], | 50.00th=[ 25297], 60.00th=[ 27395], 70.00th=[ 30016], | 80.00th=[ 33162], 90.00th=[ 38011], 95.00th=[ 44827], | 99.00th=[ 83362], 99.50th=[ 189793], 99.90th=[ 240124], | 99.95th=[ 666895], 99.99th=[2533360] bw ( KiB/s): min=59511, max=136320, per=100.00%, avg=73708.77, stdev=10598.15, samples=60 iops : min= 1859, max= 4260, avg=2303.37, stdev=331.20, samples=60 write: IOPS=572, BW=17.9MiB/s (18.8MB/s)(541MiB/30216msec) slat (usec): min=4, max=66070, avg=345.63, stdev=1584.32 clat (msec): min=7, max=2667, avg=235.31, stdev=152.46 lat (msec): min=7, max=2667, avg=235.65, stdev=152.45 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 132], 10.00th=[ 171], 20.00th=[ 194], | 30.00th=[ 205], 40.00th=[ 213], 50.00th=[ 220], 60.00th=[ 228], | 70.00th=[ 236], 80.00th=[ 249], 90.00th=[ 271], 95.00th=[ 296], | 99.00th=[ 961], 99.50th=[ 1452], 99.90th=[ 2072], 99.95th=[ 2165], | 99.99th=[ 2635] bw ( KiB/s): min=14941, max=27264, per=99.93%, avg=18305.77, stdev=1838.00, samples=60 iops : min= 466, max= 852, avg=572.03, stdev=57.46, samples=60 lat (usec) : 1000=0.06% lat (msec) : 2=1.21%, 4=1.95%, 10=1.59%, 20=14.37%, 50=58.31% lat (msec) : 100=2.43%, 250=16.26%, 500=3.39%, 750=0.13%, 1000=0.09% lat (msec) : 2000=0.18%, >=2000=0.04% cpu : usr=1.09%, sys=1.99%, ctx=9684, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=69158,17297,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=71.5MiB/s (74.0MB/s), 71.5MiB/s-71.5MiB/s (74.0MB/s-74.0MB/s), io=2161MiB (2266MB), run=30216-30216msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=541MiB (567MB), run=30216-30216msec Disk stats (read/write): vdg: ios=69155/17211, merge=0/0, ticks=65968/3569119, in_queue=3668794, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916226: Fri Jan 7 13:51:50 2022 read: IOPS=2082, BW=130MiB/s (137MB/s)(3906MiB/30001msec) clat (usec): min=36, max=22773, avg=76.07, stdev=396.37 lat (usec): min=36, max=22774, avg=76.21, stdev=396.37 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 81], 95.00th=[ 90], | 99.00th=[ 121], 99.50th=[ 143], 99.90th=[ 1004], 99.95th=[12125], | 99.99th=[16712] bw ( KiB/s): min=97664, max=209152, per=100.00%, avg=133668.10, stdev=17622.33, samples=59 iops : min= 1526, max= 3268, avg=2088.54, stdev=275.37, samples=59 write: IOPS=523, BW=32.7MiB/s (34.3MB/s)(982MiB/30001msec) clat (usec): min=351, max=44306, avg=1591.28, stdev=1683.84 lat (usec): min=358, max=44310, avg=1597.80, stdev=1683.86 clat percentiles (usec): | 1.00th=[ 453], 5.00th=[ 660], 10.00th=[ 873], 20.00th=[ 1156], | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9503], 99.50th=[10028], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=25344, max=53888, per=100.00%, avg=33557.73, stdev=3849.53, samples=59 iops : min= 396, max= 842, avg=524.32, stdev=60.16, samples=59 lat (usec) : 50=5.88%, 100=72.04%, 250=1.82%, 500=0.48%, 750=1.04% lat (usec) : 1000=1.28% lat (msec) : 2=15.50%, 4=1.57%, 10=0.24%, 20=0.14%, 50=0.02% cpu : usr=1.48%, sys=3.93%, ctx=78210, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62491,15712,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=130MiB/s (137MB/s), 130MiB/s-130MiB/s (137MB/s-137MB/s), io=3906MiB (4095MB), run=30001-30001msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=982MiB (1030MB), run=30001-30001msec Disk stats (read/write): vdg: ios=62096/15599, merge=0/0, ticks=680/17150, in_queue=28144, util=94.21% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916846: Fri Jan 7 13:52:27 2022 read: IOPS=2070, BW=129MiB/s (136MB/s)(3883MiB/30001msec) slat (usec): min=6, max=378, avg=13.38, stdev= 4.70 clat (nsec): min=750, max=35908k, avg=61784.94, stdev=446785.72 lat (usec): min=30, max=35919, avg=75.31, stdev=446.83 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 101], 99.90th=[ 494], 99.95th=[13042], | 99.99th=[18744] bw ( KiB/s): min=100992, max=224128, per=100.00%, avg=132541.58, stdev=18026.70, samples=60 iops : min= 1578, max= 3502, avg=2070.95, stdev=281.68, samples=60 write: IOPS=520, BW=32.5MiB/s (34.1MB/s)(975MiB/30001msec) slat (usec): min=11, max=181, avg=20.65, stdev= 6.98 clat (usec): min=337, max=44294, avg=1593.86, stdev=1682.84 lat (usec): min=353, max=44317, avg=1614.68, stdev=1682.87 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 652], 10.00th=[ 873], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1483], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1975], 95.00th=[ 2245], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=24960, max=57216, per=100.00%, avg=33293.85, stdev=4209.67, samples=60 iops : min= 390, max= 894, avg=520.20, stdev=65.78, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.03%, 50=49.57%, 100=29.88% lat (usec) : 250=0.30%, 500=0.49%, 750=0.96%, 1000=1.32% lat (msec) : 2=15.43%, 4=1.57%, 10=0.27%, 20=0.13%, 50=0.03% cpu : usr=1.51%, sys=6.24%, ctx=77712, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62127,15607,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=3883MiB (4072MB), run=30001-30001msec WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=975MiB (1023MB), run=30001-30001msec Disk stats (read/write): vdg: ios=61824/15509, merge=0/0, ticks=753/17261, in_queue=27927, util=93.42% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917791: Fri Jan 7 13:53:04 2022 read: IOPS=2080, BW=130MiB/s (136MB/s)(3901MiB/30001msec) clat (usec): min=32, max=25109, avg=70.00, stdev=417.83 lat (usec): min=32, max=25109, avg=70.14, stdev=417.83 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 105], 99.50th=[ 123], 99.90th=[ 857], 99.95th=[12256], | 99.99th=[17171] bw ( KiB/s): min=103040, max=204032, per=100.00%, avg=133359.37, stdev=15453.57, samples=59 iops : min= 1610, max= 3188, avg=2083.71, stdev=241.45, samples=59 write: IOPS=522, BW=32.7MiB/s (34.3MB/s)(981MiB/30001msec) clat (usec): min=366, max=44320, avg=1619.12, stdev=1702.47 lat (usec): min=373, max=44325, avg=1625.43, stdev=1702.40 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 685], 10.00th=[ 947], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[37487], 99.95th=[42206], | 99.99th=[43779] bw ( KiB/s): min=27264, max=52992, per=100.00%, avg=33499.15, stdev=3668.67, samples=59 iops : min= 426, max= 828, avg=523.41, stdev=57.34, samples=59 lat (usec) : 50=22.72%, 100=56.19%, 250=0.86%, 500=0.46%, 750=0.82% lat (usec) : 1000=1.07% lat (msec) : 2=15.84%, 4=1.61%, 10=0.26%, 20=0.14%, 50=0.02% cpu : usr=1.51%, sys=3.85%, ctx=78112, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62418,15688,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3901MiB (4091MB), run=30001-30001msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=981MiB (1028MB), run=30001-30001msec Disk stats (read/write): vdg: ios=62008/15567, merge=0/0, ticks=718/17493, in_queue=28005, util=93.78% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918738: Fri Jan 7 13:53:41 2022 read: IOPS=2113, BW=132MiB/s (139MB/s)(3971MiB/30055msec) slat (usec): min=4, max=701, avg=12.32, stdev= 5.63 clat (nsec): min=695, max=2492.4M, avg=1055113.40, stdev=44112427.71 lat (usec): min=29, max=2492.4k, avg=1067.58, stdev=44112.43 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 45], 70.00th=[ 48], | 80.00th=[ 52], 90.00th=[ 59], 95.00th=[ 66], | 99.00th=[ 92], 99.50th=[ 118], 99.90th=[ 490], | 99.95th=[ 641729], 99.99th=[2499806] bw ( KiB/s): min=109184, max=233344, per=100.00%, avg=135535.22, stdev=16665.91, samples=60 iops : min= 1706, max= 3646, avg=2117.73, stdev=260.40, samples=60 write: IOPS=531, BW=33.2MiB/s (34.8MB/s)(998MiB/30055msec) slat (usec): min=8, max=581, avg=20.80, stdev= 9.09 clat (usec): min=561, max=75767, avg=14541.62, stdev=6520.10 lat (usec): min=582, max=75794, avg=14562.59, stdev=6520.64 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 6849], 10.00th=[ 9241], 20.00th=[11469], | 30.00th=[12387], 40.00th=[13304], 50.00th=[13829], 60.00th=[14615], | 70.00th=[15401], 80.00th=[16581], 90.00th=[20317], 95.00th=[23462], | 99.00th=[32900], 99.50th=[63701], 99.90th=[68682], 99.95th=[73925], | 99.99th=[74974] bw ( KiB/s): min=29440, max=59392, per=100.00%, avg=34047.05, stdev=3842.84, samples=60 iops : min= 460, max= 928, avg=531.98, stdev=60.04, samples=60 lat (nsec) : 750=0.01%, 1000=0.06% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.05%, 50=60.23% lat (usec) : 100=18.90%, 250=0.46%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.08%, 4=0.48%, 10=2.01%, 20=15.44%, 50=1.90% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.70%, sys=5.79%, ctx=78580, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63531,15968,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3971MiB (4164MB), run=30055-30055msec WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=998MiB (1046MB), run=30055-30055msec Disk stats (read/write): vdg: ios=63530/15968, merge=0/0, ticks=63524/223948, in_queue=298533, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919317: Fri Jan 7 13:54:18 2022 read: IOPS=2069, BW=129MiB/s (136MB/s)(3880MiB/30001msec) clat (usec): min=30, max=20844, avg=68.42, stdev=430.70 lat (usec): min=30, max=20844, avg=68.56, stdev=430.70 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 93], 99.50th=[ 105], 99.90th=[ 433], 99.95th=[14091], | 99.99th=[17957] bw ( KiB/s): min=103296, max=199936, per=100.00%, avg=132445.05, stdev=14647.23, samples=60 iops : min= 1614, max= 3124, avg=2069.45, stdev=228.86, samples=60 write: IOPS=519, BW=32.5MiB/s (34.1MB/s)(975MiB/30001msec) clat (usec): min=363, max=46147, avg=1636.82, stdev=1686.06 lat (usec): min=368, max=46151, avg=1642.77, stdev=1686.01 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 701], 10.00th=[ 971], 20.00th=[ 1221], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35390], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min=24960, max=52352, per=99.99%, avg=33270.30, stdev=3552.55, samples=60 iops : min= 390, max= 818, avg=519.83, stdev=55.50, samples=60 lat (usec) : 50=28.32%, 100=51.13%, 250=0.38%, 500=0.46%, 750=0.71% lat (usec) : 1000=1.01% lat (msec) : 2=15.96%, 4=1.61%, 10=0.25%, 20=0.15%, 50=0.02% cpu : usr=1.37%, sys=3.82%, ctx=77693, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62086,15597,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=129MiB/s (136MB/s), 129MiB/s-129MiB/s (136MB/s-136MB/s), io=3880MiB (4069MB), run=30001-30001msec WRITE: bw=32.5MiB/s (34.1MB/s), 32.5MiB/s-32.5MiB/s (34.1MB/s-34.1MB/s), io=975MiB (1022MB), run=30001-30001msec Disk stats (read/write): vdg: ios=62086/15596, merge=0/0, ticks=751/17830, in_queue=28336, util=94.14% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920164: Fri Jan 7 13:54:55 2022 read: IOPS=2112, BW=132MiB/s (138MB/s)(3975MiB/30103msec) slat (usec): min=2, max=335, avg=10.82, stdev= 4.97 clat (nsec): min=691, max=2523.2M, avg=1020823.51, stdev=42469025.63 lat (usec): min=28, max=2523.2k, avg=1031.79, stdev=42469.01 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 46], 70.00th=[ 50], | 80.00th=[ 56], 90.00th=[ 64], 95.00th=[ 75], | 99.00th=[ 306], 99.50th=[ 603], 99.90th=[ 1303], | 99.95th=[ 608175], 99.99th=[2499806] bw ( KiB/s): min=106240, max=255616, per=100.00%, avg=135672.65, stdev=19842.35, samples=60 iops : min= 1660, max= 3994, avg=2119.88, stdev=310.03, samples=60 write: IOPS=531, BW=33.2MiB/s (34.8MB/s)(999MiB/30103msec) slat (usec): min=8, max=281, avg=20.94, stdev= 8.76 clat (msec): min=4, max=2954, avg=89.96, stdev=86.34 lat (msec): min=4, max=2954, avg=89.98, stdev=86.34 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 83], 60.00th=[ 85], | 70.00th=[ 88], 80.00th=[ 92], 90.00th=[ 103], 95.00th=[ 128], | 99.00th=[ 220], 99.50th=[ 567], 99.90th=[ 1586], 99.95th=[ 2022], | 99.99th=[ 2869] bw ( KiB/s): min=26368, max=59136, per=100.00%, avg=34012.95, stdev=3992.72, samples=60 iops : min= 412, max= 924, avg=531.45, stdev=62.38, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.05%, 50=55.82% lat (usec) : 100=21.99%, 250=1.00%, 500=0.42%, 750=0.20%, 1000=0.13% lat (msec) : 2=0.10%, 10=0.03%, 20=0.28%, 50=0.16%, 100=17.43% lat (msec) : 250=2.03%, 500=0.09%, 750=0.06%, 1000=0.03%, 2000=0.04% lat (msec) : >=2000=0.03% cpu : usr=1.80%, sys=5.23%, ctx=77340, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=63596,15991,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3975MiB (4168MB), run=30103-30103msec WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=999MiB (1048MB), run=30103-30103msec Disk stats (read/write): vdg: ios=63594/15965, merge=0/0, ticks=59973/1425277, in_queue=1499533, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921273: Fri Jan 7 13:55:32 2022 read: IOPS=2078, BW=130MiB/s (136MB/s)(3897MiB/30001msec) clat (usec): min=33, max=26159, avg=70.81, stdev=436.26 lat (usec): min=33, max=26159, avg=70.96, stdev=436.26 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 57], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 96], 99.50th=[ 110], 99.90th=[ 766], 99.95th=[12780], | 99.99th=[19006] bw ( KiB/s): min=110592, max=209571, per=100.00%, avg=133014.20, stdev=14722.12, samples=60 iops : min= 1728, max= 3274, avg=2078.33, stdev=229.98, samples=60 write: IOPS=522, BW=32.6MiB/s (34.2MB/s)(979MiB/30001msec) clat (usec): min=340, max=57789, avg=1619.10, stdev=1713.48 lat (usec): min=346, max=57795, avg=1625.13, stdev=1713.46 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 685], 10.00th=[ 938], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35390], 99.95th=[40109], | 99.99th=[49546] bw ( KiB/s): min=27008, max=53995, per=100.00%, avg=33413.10, stdev=3644.59, samples=60 iops : min= 422, max= 843, avg=522.07, stdev=56.88, samples=60 lat (usec) : 50=20.68%, 100=58.62%, 250=0.52%, 500=0.46%, 750=0.78% lat (usec) : 1000=1.11% lat (msec) : 2=15.79%, 4=1.64%, 10=0.24%, 20=0.14%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.48%, sys=3.86%, ctx=78016, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62345,15662,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3897MiB (4086MB), run=30001-30001msec WRITE: bw=32.6MiB/s (34.2MB/s), 32.6MiB/s-32.6MiB/s (34.2MB/s-34.2MB/s), io=979MiB (1026MB), run=30001-30001msec Disk stats (read/write): vdg: ios=62069/15589, merge=0/0, ticks=768/17554, in_queue=28211, util=94.15% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922078: Fri Jan 7 13:56:10 2022 read: IOPS=2118, BW=132MiB/s (139MB/s)(3998MiB/30190msec) slat (usec): min=2, max=387, avg=11.06, stdev= 5.59 clat (nsec): min=725, max=2518.0M, avg=1153946.07, stdev=44884932.13 lat (usec): min=27, max=2518.1k, avg=1165.16, stdev=44884.91 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 38], 30.00th=[ 39], 40.00th=[ 42], | 50.00th=[ 45], 60.00th=[ 49], 70.00th=[ 54], | 80.00th=[ 60], 90.00th=[ 72], 95.00th=[ 121], | 99.00th=[ 1319], 99.50th=[ 2180], 99.90th=[ 3720], | 99.95th=[ 784335], 99.99th=[2499806] bw ( KiB/s): min=108160, max=273024, per=100.00%, avg=136452.93, stdev=21997.69, samples=60 iops : min= 1690, max= 4266, avg=2132.05, stdev=343.71, samples=60 write: IOPS=532, BW=33.3MiB/s (34.9MB/s)(1005MiB/30190msec) slat (usec): min=8, max=397, avg=21.82, stdev= 9.65 clat (msec): min=11, max=3699, avg=183.09, stdev=148.12 lat (msec): min=11, max=3699, avg=183.11, stdev=148.12 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 126], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 165], 50.00th=[ 171], 60.00th=[ 178], | 70.00th=[ 184], 80.00th=[ 192], 90.00th=[ 215], 95.00th=[ 234], | 99.00th=[ 414], 99.50th=[ 1200], 99.90th=[ 2232], 99.95th=[ 3239], | 99.99th=[ 3641] bw ( KiB/s): min=29696, max=57344, per=100.00%, avg=34084.50, stdev=3792.31, samples=60 iops : min= 464, max= 896, avg=532.57, stdev=59.25, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.05%, 50=50.34% lat (usec) : 100=24.88%, 250=1.70%, 500=0.85%, 750=0.54%, 1000=0.37% lat (msec) : 2=0.60%, 4=0.38%, 20=0.06%, 50=0.37%, 100=0.25% lat (msec) : 250=18.71%, 500=0.53%, 750=0.05%, 1000=0.02%, 2000=0.11% lat (msec) : >=2000=0.05% cpu : usr=1.69%, sys=5.40%, ctx=74929, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=63961,16072,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3998MiB (4192MB), run=30190-30190msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1005MiB (1053MB), run=30190-30190msec Disk stats (read/write): vdg: ios=63958/15989, merge=0/0, ticks=66595/2910305, in_queue=2999964, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923047: Fri Jan 7 13:56:46 2022 read: IOPS=2066, BW=129MiB/s (135MB/s)(3876MiB/30006msec) clat (usec): min=31, max=82214, avg=68.21, stdev=529.90 lat (usec): min=32, max=82214, avg=68.35, stdev=529.90 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 74], | 99.00th=[ 88], 99.50th=[ 96], 99.90th=[ 392], 99.95th=[12780], | 99.99th=[18482] bw ( KiB/s): min=108672, max=212777, per=100.00%, avg=132290.82, stdev=15049.31, samples=60 iops : min= 1698, max= 3324, avg=2067.03, stdev=235.09, samples=60 write: IOPS=518, BW=32.4MiB/s (34.0MB/s)(973MiB/30006msec) clat (usec): min=347, max=91244, avg=1641.51, stdev=1782.72 lat (usec): min=351, max=91249, avg=1647.50, stdev=1782.67 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 709], 10.00th=[ 988], 20.00th=[ 1221], | 30.00th=[ 1319], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1762], 90.00th=[ 2024], 95.00th=[ 2311], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[34866], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min=28032, max=55150, per=100.00%, avg=33213.57, stdev=3841.47, samples=60 iops : min= 438, max= 861, avg=518.95, stdev=59.95, samples=60 lat (usec) : 50=30.92%, 100=48.69%, 250=0.23%, 500=0.44%, 750=0.68% lat (usec) : 1000=0.96% lat (msec) : 2=15.91%, 4=1.76%, 10=0.26%, 20=0.13%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.58%, sys=3.36%, ctx=77585, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62008,15569,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3876MiB (4064MB), run=30006-30006msec WRITE: bw=32.4MiB/s (34.0MB/s), 32.4MiB/s-32.4MiB/s (34.0MB/s-34.0MB/s), io=973MiB (1020MB), run=30006-30006msec Disk stats (read/write): vdg: ios=61659/15477, merge=0/0, ticks=765/17813, in_queue=28177, util=94.25% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923713: Fri Jan 7 13:57:24 2022 read: IOPS=2127, BW=133MiB/s (139MB/s)(4018MiB/30223msec) slat (usec): min=2, max=224449, avg=370.02, stdev=2332.05 clat (usec): min=969, max=2570.0k, avg=30387.72, stdev=51489.91 lat (usec): min=972, max=2571.0k, avg=30757.88, stdev=51549.70 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 5932], 10.00th=[ 16188], | 20.00th=[ 20579], 30.00th=[ 23200], 40.00th=[ 25297], | 50.00th=[ 27395], 60.00th=[ 29492], 70.00th=[ 31851], | 80.00th=[ 34866], 90.00th=[ 40109], 95.00th=[ 46924], | 99.00th=[ 91751], 99.50th=[ 210764], 99.90th=[ 248513], | 99.95th=[ 834667], 99.99th=[2533360] bw ( KiB/s): min=116736, max=273408, per=100.00%, avg=137023.57, stdev=21893.21, samples=60 iops : min= 1824, max= 4272, avg=2140.95, stdev=342.11, samples=60 write: IOPS=534, BW=33.4MiB/s (34.0MB/s)(1009MiB/30223msec) slat (usec): min=7, max=164219, avg=374.35, stdev=2015.65 clat (msec): min=17, max=3787, avg=251.45, stdev=182.74 lat (msec): min=17, max=3788, avg=251.82, stdev=182.73 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 155], 10.00th=[ 186], 20.00th=[ 205], | 30.00th=[ 220], 40.00th=[ 228], 50.00th=[ 234], 60.00th=[ 243], | 70.00th=[ 251], 80.00th=[ 266], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 852], 99.50th=[ 1687], 99.90th=[ 3037], 99.95th=[ 3540], | 99.99th=[ 3775] bw ( KiB/s): min=28032, max=54656, per=99.95%, avg=34159.07, stdev=3580.41, samples=60 iops : min= 438, max= 854, avg=533.72, stdev=55.96, samples=60 lat (usec) : 1000=0.01% lat (msec) : 2=0.98%, 4=1.85%, 10=2.00%, 20=9.35%, 50=62.86% lat (msec) : 100=2.72%, 250=13.79%, 500=6.07%, 750=0.11%, 1000=0.05% lat (msec) : 2000=0.12%, >=2000=0.09% cpu : usr=1.52%, sys=2.07%, ctx=9023, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64292,16140,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=4018MiB (4213MB), run=30223-30223msec WRITE: bw=33.4MiB/s (34.0MB/s), 33.4MiB/s-33.4MiB/s (34.0MB/s-34.0MB/s), io=1009MiB (1058MB), run=30223-30223msec Disk stats (read/write): vdg: ios=64287/16054, merge=0/0, ticks=78635/3556853, in_queue=3672736, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924584: Fri Jan 7 13:58:00 2022 read: IOPS=1973, BW=154MiB/s (162MB/s)(4626MiB/30001msec) clat (usec): min=40, max=35236, avg=85.37, stdev=490.99 lat (usec): min=40, max=35236, avg=85.52, stdev=491.00 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 55], 10.00th=[ 56], 20.00th=[ 59], | 30.00th=[ 61], 40.00th=[ 64], 50.00th=[ 68], 60.00th=[ 71], | 70.00th=[ 75], 80.00th=[ 79], 90.00th=[ 87], 95.00th=[ 95], | 99.00th=[ 126], 99.50th=[ 153], 99.90th=[ 7570], 99.95th=[14746], | 99.99th=[18744] bw ( KiB/s): min=127520, max=235360, per=100.00%, avg=158193.05, stdev=16627.09, samples=59 iops : min= 1594, max= 2942, avg=1977.39, stdev=207.85, samples=59 write: IOPS=496, BW=38.8MiB/s (40.7MB/s)(1163MiB/30001msec) clat (usec): min=410, max=86914, avg=1658.66, stdev=1875.11 lat (usec): min=418, max=86921, avg=1666.50, stdev=1875.14 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 676], 10.00th=[ 857], 20.00th=[ 1156], | 30.00th=[ 1319], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1778], 90.00th=[ 2024], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[36439], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=32545, max=61440, per=100.00%, avg=39728.56, stdev=3994.44, samples=59 iops : min= 406, max= 768, avg=496.58, stdev=49.96, samples=59 lat (usec) : 50=0.43%, 100=76.72%, 250=2.55%, 500=0.31%, 750=1.24% lat (usec) : 1000=1.28% lat (msec) : 2=15.20%, 4=1.78%, 10=0.28%, 20=0.17%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.70%, sys=3.76%, ctx=74115, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59219,14891,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=4626MiB (4851MB), run=30001-30001msec WRITE: bw=38.8MiB/s (40.7MB/s), 38.8MiB/s-38.8MiB/s (40.7MB/s-40.7MB/s), io=1163MiB (1220MB), run=30001-30001msec Disk stats (read/write): vdg: ios=58870/14799, merge=0/0, ticks=875/16850, in_queue=28108, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925582: Fri Jan 7 13:58:37 2022 read: IOPS=1992, BW=156MiB/s (163MB/s)(4670MiB/30003msec) slat (usec): min=7, max=129, avg=14.39, stdev= 4.35 clat (nsec): min=920, max=23668k, avg=66866.13, stdev=446372.23 lat (usec): min=38, max=23678, avg=81.41, stdev=446.43 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 90], 99.50th=[ 104], 99.90th=[ 1205], 99.95th=[13960], | 99.99th=[17695] bw ( KiB/s): min=122720, max=238880, per=100.00%, avg=159413.85, stdev=18225.68, samples=60 iops : min= 1534, max= 2986, avg=1992.65, stdev=227.81, samples=60 write: IOPS=500, BW=39.1MiB/s (40.0MB/s)(1173MiB/30003msec) slat (usec): min=12, max=108, avg=22.96, stdev= 6.54 clat (usec): min=393, max=42469, avg=1643.40, stdev=1697.54 lat (usec): min=417, max=42489, avg=1666.54, stdev=1697.61 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 676], 10.00th=[ 881], 20.00th=[ 1172], | 30.00th=[ 1303], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1631], 80.00th=[ 1778], 90.00th=[ 2024], 95.00th=[ 2343], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[18744], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min=33440, max=61920, per=100.00%, avg=40034.67, stdev=3976.33, samples=60 iops : min= 418, max= 774, avg=500.42, stdev=49.71, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=37.31%, 100=42.11% lat (usec) : 250=0.34%, 500=0.36%, 750=1.12%, 1000=1.19% lat (msec) : 2=15.32%, 4=1.75%, 10=0.27%, 20=0.17%, 50=0.03% cpu : usr=1.52%, sys=6.23%, ctx=74794, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59776,15013,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4670MiB (4897MB), run=30003-30003msec WRITE: bw=39.1MiB/s (40.0MB/s), 39.1MiB/s-39.1MiB/s (40.0MB/s-40.0MB/s), io=1173MiB (1230MB), run=30003-30003msec Disk stats (read/write): vdg: ios=59270/14899, merge=0/0, ticks=787/17056, in_queue=27884, util=93.29% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926124: Fri Jan 7 13:59:14 2022 read: IOPS=1997, BW=156MiB/s (164MB/s)(4683MiB/30002msec) clat (usec): min=33, max=25957, avg=73.16, stdev=458.16 lat (usec): min=33, max=25957, avg=73.30, stdev=458.17 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 100], 99.50th=[ 116], 99.90th=[ 2114], 99.95th=[13698], | 99.99th=[19006] bw ( KiB/s): min=122560, max=239840, per=100.00%, avg=159865.30, stdev=18736.05, samples=60 iops : min= 1532, max= 2998, avg=1998.27, stdev=234.23, samples=60 write: IOPS=501, BW=39.2MiB/s (41.1MB/s)(1176MiB/30002msec) clat (usec): min=404, max=45723, avg=1685.03, stdev=1689.48 lat (usec): min=411, max=45731, avg=1692.78, stdev=1689.44 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 709], 10.00th=[ 979], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[17171], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=33280, max=62240, per=100.00%, avg=40155.88, stdev=3955.45, samples=60 iops : min= 416, max= 778, avg=501.90, stdev=49.46, samples=60 lat (usec) : 50=16.65%, 100=62.46%, 250=0.69%, 500=0.33%, 750=0.88% lat (usec) : 1000=0.92% lat (msec) : 2=15.61%, 4=2.00%, 10=0.27%, 20=0.17%, 50=0.03% cpu : usr=1.70%, sys=3.59%, ctx=75002, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59940,15057,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4683MiB (4910MB), run=30002-30002msec WRITE: bw=39.2MiB/s (41.1MB/s), 39.2MiB/s-39.2MiB/s (41.1MB/s-41.1MB/s), io=1176MiB (1233MB), run=30002-30002msec Disk stats (read/write): vdg: ios=59452/14955, merge=0/0, ticks=810/17468, in_queue=28156, util=94.22% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926963: Fri Jan 7 13:59:51 2022 read: IOPS=2038, BW=159MiB/s (167MB/s)(4784MiB/30036msec) slat (usec): min=4, max=193, avg=13.22, stdev= 4.54 clat (nsec): min=777, max=2493.3M, avg=1034236.25, stdev=41821757.91 lat (usec): min=30, max=2493.3k, avg=1047.61, stdev=41821.76 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 36], 10.00th=[ 37], | 20.00th=[ 40], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 51], | 80.00th=[ 56], 90.00th=[ 63], 95.00th=[ 71], | 99.00th=[ 99], 99.50th=[ 128], 99.90th=[ 1254], | 99.95th=[ 591397], 99.99th=[2499806] bw ( KiB/s): min=132320, max=298400, per=100.00%, avg=163320.78, stdev=22602.95, samples=60 iops : min= 1654, max= 3730, avg=2041.47, stdev=282.53, samples=60 write: IOPS=512, BW=40.0MiB/s (41.9MB/s)(1202MiB/30036msec) slat (usec): min=10, max=196, avg=23.33, stdev= 8.04 clat (usec): min=715, max=79712, avg=15322.46, stdev=6602.63 lat (usec): min=734, max=79729, avg=15345.98, stdev=6603.24 clat percentiles (usec): | 1.00th=[ 3032], 5.00th=[ 6783], 10.00th=[ 8717], 20.00th=[11731], | 30.00th=[13173], 40.00th=[14091], 50.00th=[15008], 60.00th=[15795], | 70.00th=[16581], 80.00th=[17695], 90.00th=[21103], 95.00th=[24511], | 99.00th=[35914], 99.50th=[64226], 99.90th=[68682], 99.95th=[73925], | 99.99th=[78119] bw ( KiB/s): min=34880, max=75200, per=100.00%, avg=41005.55, stdev=5104.41, samples=60 iops : min= 436, max= 940, avg=512.57, stdev=63.80, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.04%, 50=53.20% lat (usec) : 100=25.85%, 250=0.62%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.30%, 10=2.33%, 20=15.01%, 50=2.27% lat (msec) : 100=0.17%, 250=0.01%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.77%, sys=5.83%, ctx=75859, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61240,15381,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4784MiB (5017MB), run=30036-30036msec WRITE: bw=40.0MiB/s (41.9MB/s), 40.0MiB/s-40.0MiB/s (41.9MB/s-41.9MB/s), io=1202MiB (1260MB), run=30036-30036msec Disk stats (read/write): vdg: ios=60857/15287, merge=0/0, ticks=56684/226430, in_queue=296384, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927602: Fri Jan 7 14:00:28 2022 read: IOPS=1998, BW=156MiB/s (164MB/s)(4684MiB/30002msec) clat (usec): min=32, max=21056, avg=71.91, stdev=457.39 lat (usec): min=32, max=21056, avg=72.05, stdev=457.39 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 97], 99.50th=[ 112], 99.90th=[ 3752], 99.95th=[14353], | 99.99th=[17957] bw ( KiB/s): min=132480, max=249760, per=100.00%, avg=159918.92, stdev=18851.75, samples=60 iops : min= 1656, max= 3122, avg=1998.95, stdev=235.63, samples=60 write: IOPS=501, BW=39.2MiB/s (41.1MB/s)(1177MiB/30002msec) clat (usec): min=385, max=43765, avg=1689.13, stdev=1701.62 lat (usec): min=393, max=43774, avg=1697.56, stdev=1701.55 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 717], 10.00th=[ 1004], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1827], 90.00th=[ 2073], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[35914], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=33280, max=65280, per=100.00%, avg=40164.10, stdev=4096.31, samples=60 iops : min= 416, max= 816, avg=502.03, stdev=51.22, samples=60 lat (usec) : 50=22.01%, 100=57.27%, 250=0.52%, 500=0.28%, 750=0.88% lat (usec) : 1000=0.87% lat (msec) : 2=15.69%, 4=2.01%, 10=0.27%, 20=0.16%, 50=0.02% cpu : usr=1.49%, sys=3.67%, ctx=75028, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59959,15060,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4684MiB (4912MB), run=30002-30002msec WRITE: bw=39.2MiB/s (41.1MB/s), 39.2MiB/s-39.2MiB/s (41.1MB/s-41.1MB/s), io=1177MiB (1234MB), run=30002-30002msec Disk stats (read/write): vdg: ios=59724/14995, merge=0/0, ticks=825/17596, in_queue=28282, util=94.70% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928863: Fri Jan 7 14:01:05 2022 read: IOPS=2035, BW=159MiB/s (167MB/s)(4786MiB/30101msec) slat (usec): min=2, max=272, avg=11.67, stdev= 6.20 clat (nsec): min=736, max=2555.2M, avg=1123036.94, stdev=44765081.78 lat (usec): min=29, max=2555.2k, avg=1134.86, stdev=44765.08 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 34], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 47], 70.00th=[ 53], | 80.00th=[ 59], 90.00th=[ 67], 95.00th=[ 78], | 99.00th=[ 343], 99.50th=[ 676], 99.90th=[ 1991], | 99.95th=[ 541066], 99.99th=[2499806] bw ( KiB/s): min=120960, max=326720, per=100.00%, avg=163373.90, stdev=26237.75, samples=60 iops : min= 1512, max= 4084, avg=2042.12, stdev=327.98, samples=60 write: IOPS=511, BW=39.9MiB/s (41.9MB/s)(1202MiB/30101msec) slat (usec): min=8, max=438, avg=24.40, stdev=13.03 clat (msec): min=3, max=2612, avg=93.26, stdev=77.71 lat (msec): min=3, max=2612, avg=93.28, stdev=77.71 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 66], 10.00th=[ 71], 20.00th=[ 77], | 30.00th=[ 81], 40.00th=[ 84], 50.00th=[ 87], 60.00th=[ 89], | 70.00th=[ 92], 80.00th=[ 96], 90.00th=[ 107], 95.00th=[ 133], | 99.00th=[ 224], 99.50th=[ 567], 99.90th=[ 1284], 99.95th=[ 1519], | 99.99th=[ 2299] bw ( KiB/s): min=34880, max=75680, per=100.00%, avg=40908.20, stdev=5217.37, samples=60 iops : min= 436, max= 946, avg=511.35, stdev=65.22, samples=60 lat (nsec) : 750=0.01%, 1000=0.08% lat (usec) : 2=0.14%, 4=0.02%, 10=0.04%, 20=0.12%, 50=52.53% lat (usec) : 100=24.96%, 250=1.01%, 500=0.44%, 750=0.20%, 1000=0.08% lat (msec) : 2=0.20%, 4=0.01%, 10=0.03%, 20=0.29%, 50=0.15% lat (msec) : 100=16.78%, 250=2.66%, 500=0.09%, 750=0.04%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.02% cpu : usr=1.76%, sys=5.39%, ctx=74185, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=61256,15386,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4786MiB (5018MB), run=30101-30101msec WRITE: bw=39.9MiB/s (41.9MB/s), 39.9MiB/s-39.9MiB/s (41.9MB/s-41.9MB/s), io=1202MiB (1260MB), run=30101-30101msec Disk stats (read/write): vdg: ios=61255/15371, merge=0/0, ticks=63860/1423908, in_queue=1500998, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929908: Fri Jan 7 14:01:42 2022 read: IOPS=1976, BW=154MiB/s (162MB/s)(4632MiB/30001msec) clat (usec): min=34, max=24870, avg=73.85, stdev=445.78 lat (usec): min=34, max=24870, avg=73.99, stdev=445.79 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 81], | 99.00th=[ 103], 99.50th=[ 120], 99.90th=[ 1631], 99.95th=[12911], | 99.99th=[18482] bw ( KiB/s): min=131360, max=261440, per=100.00%, avg=158626.97, stdev=19784.99, samples=59 iops : min= 1642, max= 3268, avg=1982.81, stdev=247.34, samples=59 write: IOPS=496, BW=38.8MiB/s (40.7MB/s)(1165MiB/30001msec) clat (usec): min=396, max=43827, avg=1701.42, stdev=1746.56 lat (usec): min=404, max=43833, avg=1709.76, stdev=1746.56 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 709], 10.00th=[ 979], 20.00th=[ 1237], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1827], 90.00th=[ 2089], 95.00th=[ 2409], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[35390], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min=34240, max=67200, per=100.00%, avg=39830.64, stdev=4510.48, samples=59 iops : min= 428, max= 840, avg=497.86, stdev=56.39, samples=59 lat (usec) : 50=12.91%, 100=66.08%, 250=0.78%, 500=0.30%, 750=0.93% lat (usec) : 1000=0.94% lat (msec) : 2=15.48%, 4=2.11%, 10=0.27%, 20=0.19%, 50=0.03% cpu : usr=1.61%, sys=3.76%, ctx=74213, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59295,14908,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=4632MiB (4857MB), run=30001-30001msec WRITE: bw=38.8MiB/s (40.7MB/s), 38.8MiB/s-38.8MiB/s (40.7MB/s-40.7MB/s), io=1165MiB (1221MB), run=30001-30001msec Disk stats (read/write): vdg: ios=58806/14787, merge=0/0, ticks=772/17569, in_queue=28007, util=93.83% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930579: Fri Jan 7 14:02:20 2022 read: IOPS=2039, BW=159MiB/s (167MB/s)(4818MiB/30241msec) slat (usec): min=2, max=259, avg=11.05, stdev= 5.65 clat (nsec): min=750, max=2572.0M, avg=1207516.21, stdev=46091806.48 lat (usec): min=30, max=2572.0k, avg=1218.73, stdev=46091.77 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 36], | 20.00th=[ 39], 30.00th=[ 40], 40.00th=[ 43], | 50.00th=[ 46], 60.00th=[ 50], 70.00th=[ 57], | 80.00th=[ 63], 90.00th=[ 77], 95.00th=[ 137], | 99.00th=[ 1254], 99.50th=[ 2737], 99.90th=[ 4293], | 99.95th=[ 717226], 99.99th=[2499806] bw ( KiB/s): min=120480, max=340800, per=100.00%, avg=164453.48, stdev=27733.41, samples=60 iops : min= 1506, max= 4260, avg=2055.65, stdev=346.67, samples=60 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1210MiB/30241msec) slat (usec): min=7, max=357, avg=25.38, stdev=13.18 clat (msec): min=3, max=3368, avg=190.42, stdev=166.82 lat (msec): min=3, max=3368, avg=190.44, stdev=166.82 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 131], 10.00th=[ 140], 20.00th=[ 155], | 30.00th=[ 163], 40.00th=[ 171], 50.00th=[ 176], 60.00th=[ 182], | 70.00th=[ 190], 80.00th=[ 199], 90.00th=[ 222], 95.00th=[ 249], | 99.00th=[ 460], 99.50th=[ 1167], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min=34560, max=71680, per=100.00%, avg=41032.03, stdev=4740.48, samples=60 iops : min= 432, max= 896, avg=512.90, stdev=59.26, samples=60 lat (nsec) : 1000=0.04% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.04%, 50=48.10% lat (usec) : 100=26.62%, 250=2.21%, 500=0.93%, 750=0.58%, 1000=0.35% lat (msec) : 2=0.38%, 4=0.51%, 10=0.03%, 20=0.05%, 50=0.35% lat (msec) : 100=0.18%, 250=18.54%, 500=0.79%, 750=0.05%, 1000=0.03% lat (msec) : 2000=0.07%, >=2000=0.08% cpu : usr=1.99%, sys=5.02%, ctx=71998, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=61670,15482,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4818MiB (5052MB), run=30241-30241msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1210MiB (1268MB), run=30241-30241msec Disk stats (read/write): vdg: ios=61666/15409, merge=0/0, ticks=63293/2905856, in_queue=3002261, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931409: Fri Jan 7 14:02:56 2022 read: IOPS=2006, BW=157MiB/s (164MB/s)(4702MiB/30002msec) clat (usec): min=31, max=27622, avg=71.46, stdev=474.88 lat (usec): min=31, max=27622, avg=71.60, stdev=474.88 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 96], 99.50th=[ 111], 99.90th=[ 1057], 99.95th=[14746], | 99.99th=[18220] bw ( KiB/s): min=120480, max=254720, per=100.00%, avg=160520.35, stdev=19217.16, samples=60 iops : min= 1506, max= 3184, avg=2006.48, stdev=240.22, samples=60 write: IOPS=503, BW=39.4MiB/s (41.3MB/s)(1181MiB/30002msec) clat (usec): min=397, max=54868, avg=1683.55, stdev=1676.06 lat (usec): min=404, max=54876, avg=1691.05, stdev=1676.00 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 725], 10.00th=[ 1029], 20.00th=[ 1270], | 30.00th=[ 1369], 40.00th=[ 1434], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1696], 80.00th=[ 1827], 90.00th=[ 2073], 95.00th=[ 2376], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[34866], 99.95th=[39060], | 99.99th=[44303] bw ( KiB/s): min=30560, max=65920, per=100.00%, avg=40318.60, stdev=4595.84, samples=60 iops : min= 382, max= 824, avg=503.95, stdev=57.46, samples=60 lat (usec) : 50=25.99%, 100=53.31%, 250=0.51%, 500=0.30%, 750=0.82% lat (usec) : 1000=0.80% lat (msec) : 2=15.74%, 4=2.13%, 10=0.23%, 20=0.15%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.49%, sys=3.58%, ctx=75318, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60189,15119,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=157MiB/s (164MB/s), 157MiB/s-157MiB/s (164MB/s-164MB/s), io=4702MiB (4931MB), run=30002-30002msec WRITE: bw=39.4MiB/s (41.3MB/s), 39.4MiB/s-39.4MiB/s (41.3MB/s-41.3MB/s), io=1181MiB (1239MB), run=30002-30002msec Disk stats (read/write): vdg: ios=59776/15013, merge=0/0, ticks=832/17569, in_queue=28156, util=94.24% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932351: Fri Jan 7 14:03:34 2022 read: IOPS=2039, BW=159MiB/s (167MB/s)(4829MiB/30310msec) slat (usec): min=2, max=268939, avg=386.96, stdev=2493.90 clat (usec): min=1149, max=2731.6k, avg=31683.50, stdev=49577.91 lat (usec): min=1158, max=2733.4k, avg=32070.61, stdev=49645.57 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 6], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 31], | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 44], 95.00th=[ 51], | 99.00th=[ 92], 99.50th=[ 234], 99.90th=[ 334], 99.95th=[ 793], | 99.99th=[ 2534] bw ( KiB/s): min=120000, max=363040, per=100.00%, avg=164681.02, stdev=31030.87, samples=60 iops : min= 1500, max= 4538, avg=2058.50, stdev=387.88, samples=60 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1212MiB/30310msec) slat (usec): min=8, max=251832, avg=382.20, stdev=2507.55 clat (msec): min=16, max=3541, avg=262.59, stdev=144.00 lat (msec): min=17, max=3541, avg=262.97, stdev=143.98 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 188], 10.00th=[ 207], 20.00th=[ 224], | 30.00th=[ 234], 40.00th=[ 243], 50.00th=[ 249], 60.00th=[ 255], | 70.00th=[ 264], 80.00th=[ 275], 90.00th=[ 300], 95.00th=[ 342], | 99.00th=[ 768], 99.50th=[ 1250], 99.90th=[ 2140], 99.95th=[ 2366], | 99.99th=[ 3406] bw ( KiB/s): min=34240, max=74400, per=100.00%, avg=41009.60, stdev=5335.88, samples=60 iops : min= 428, max= 930, avg=512.62, stdev=66.70, samples=60 lat (msec) : 2=0.77%, 4=2.13%, 10=2.10%, 20=8.31%, 50=62.98% lat (msec) : 100=3.53%, 250=10.09%, 500=9.71%, 750=0.12%, 1000=0.09% lat (msec) : 2000=0.11%, >=2000=0.06% cpu : usr=1.84%, sys=1.83%, ctx=8619, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=61817,15509,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4829MiB (5064MB), run=30310-30310msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1212MiB (1270MB), run=30310-30310msec Disk stats (read/write): vdg: ios=61815/15483, merge=0/0, ticks=75841/3583965, in_queue=3690213, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932854: Fri Jan 7 14:04:10 2022 read: IOPS=1754, BW=219MiB/s (230MB/s)(6584MiB/30017msec) clat (usec): min=49, max=27072, avg=99.65, stdev=546.42 lat (usec): min=49, max=27073, avg=99.81, stdev=546.42 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 94], 90.00th=[ 103], 95.00th=[ 114], | 99.00th=[ 143], 99.50th=[ 167], 99.90th=[11469], 99.95th=[16581], | 99.99th=[22414] bw ( KiB/s): min=109312, max=367616, per=100.00%, avg=224753.50, stdev=32760.52, samples=60 iops : min= 854, max= 2872, avg=1755.88, stdev=255.94, samples=60 write: IOPS=441, BW=55.2MiB/s (57.9MB/s)(1657MiB/30017msec) clat (usec): min=527, max=203761, avg=1845.05, stdev=2632.98 lat (usec): min=539, max=203769, avg=1859.04, stdev=2632.90 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 898], 10.00th=[ 1057], 20.00th=[ 1237], | 30.00th=[ 1369], 40.00th=[ 1500], 50.00th=[ 1598], 60.00th=[ 1713], | 70.00th=[ 1827], 80.00th=[ 1958], 90.00th=[ 2245], 95.00th=[ 2638], | 99.00th=[ 9896], 99.50th=[ 10421], 99.90th=[ 36439], 99.95th=[ 40109], | 99.99th=[104334] bw ( KiB/s): min=28928, max=94976, per=100.00%, avg=56556.55, stdev=7510.78, samples=60 iops : min= 226, max= 742, avg=441.83, stdev=58.68, samples=60 lat (usec) : 50=0.01%, 100=69.76%, 250=9.93%, 500=0.07%, 750=0.52% lat (usec) : 1000=1.03% lat (msec) : 2=15.01%, 4=3.04%, 10=0.40%, 20=0.21%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.77%, sys=3.57%, ctx=65938, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=52675,13256,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=6584MiB (6904MB), run=30017-30017msec WRITE: bw=55.2MiB/s (57.9MB/s), 55.2MiB/s-55.2MiB/s (57.9MB/s-57.9MB/s), io=1657MiB (1737MB), run=30017-30017msec Disk stats (read/write): vdg: ios=52406/13187, merge=0/0, ticks=868/17305, in_queue=28099, util=93.94% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933839: Fri Jan 7 14:04:47 2022 read: IOPS=1813, BW=227MiB/s (238MB/s)(6799MiB/30001msec) slat (usec): min=8, max=126, avg=15.36, stdev= 4.90 clat (nsec): min=831, max=30957k, avg=74382.48, stdev=526934.82 lat (usec): min=44, max=30966, avg=89.90, stdev=527.01 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 94], 99.50th=[ 106], 99.90th=[11863], 99.95th=[15139], | 99.99th=[19268] bw ( KiB/s): min=177664, max=374784, per=100.00%, avg=232541.88, stdev=27930.41, samples=59 iops : min= 1388, max= 2928, avg=1816.73, stdev=218.21, samples=59 write: IOPS=455, BW=56.0MiB/s (59.7MB/s)(1709MiB/30001msec) slat (usec): min=15, max=195, avg=28.08, stdev= 7.44 clat (usec): min=511, max=49471, avg=1800.40, stdev=1669.88 lat (usec): min=539, max=49509, avg=1828.66, stdev=1670.07 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 775], 10.00th=[ 963], 20.00th=[ 1319], | 30.00th=[ 1467], 40.00th=[ 1549], 50.00th=[ 1631], 60.00th=[ 1713], | 70.00th=[ 1844], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2573], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[35914], 99.95th=[38536], | 99.99th=[41157] bw ( KiB/s): min=47360, max=97280, per=100.00%, avg=58393.61, stdev=6563.80, samples=59 iops : min= 370, max= 760, avg=456.17, stdev=51.28, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=23.59%, 100=55.70% lat (usec) : 250=0.46%, 500=0.02%, 750=0.83%, 1000=1.35% lat (msec) : 2=14.06%, 4=3.47%, 10=0.25%, 20=0.21%, 50=0.02% cpu : usr=1.66%, sys=5.63%, ctx=68053, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54392,13673,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6799MiB (7129MB), run=30001-30001msec WRITE: bw=56.0MiB/s (59.7MB/s), 56.0MiB/s-56.0MiB/s (59.7MB/s-59.7MB/s), io=1709MiB (1792MB), run=30001-30001msec Disk stats (read/write): vdg: ios=54079/13591, merge=0/0, ticks=898/16954, in_queue=28056, util=93.80% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934419: Fri Jan 7 14:05:24 2022 read: IOPS=1797, BW=225MiB/s (236MB/s)(6740MiB/30001msec) clat (usec): min=37, max=36125, avg=81.54, stdev=553.65 lat (usec): min=38, max=36125, avg=81.68, stdev=553.65 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 86], | 99.00th=[ 108], 99.50th=[ 123], 99.90th=[11731], 99.95th=[16057], | 99.99th=[20055] bw ( KiB/s): min=164864, max=377344, per=100.00%, avg=230057.53, stdev=28890.74, samples=60 iops : min= 1288, max= 2948, avg=1797.32, stdev=225.71, samples=60 write: IOPS=451, BW=56.4MiB/s (59.2MB/s)(1693MiB/30001msec) clat (usec): min=518, max=43654, avg=1869.24, stdev=1750.34 lat (usec): min=529, max=43666, avg=1881.58, stdev=1750.19 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 807], 10.00th=[ 1074], 20.00th=[ 1401], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2311], 95.00th=[ 2606], | 99.00th=[10028], 99.50th=[10421], 99.90th=[36963], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=48640, max=98560, per=100.00%, avg=57798.18, stdev=6559.44, samples=60 iops : min= 380, max= 770, avg=451.53, stdev=51.25, samples=60 lat (usec) : 50=3.23%, 100=75.47%, 250=1.09%, 500=0.02%, 750=0.76% lat (usec) : 1000=1.03% lat (msec) : 2=14.03%, 4=3.82%, 10=0.26%, 20=0.25%, 50=0.03% cpu : usr=1.64%, sys=3.39%, ctx=67471, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53918,13547,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6740MiB (7067MB), run=30001-30001msec WRITE: bw=56.4MiB/s (59.2MB/s), 56.4MiB/s-56.4MiB/s (59.2MB/s-59.2MB/s), io=1693MiB (1776MB), run=30001-30001msec Disk stats (read/write): vdg: ios=53579/13453, merge=0/0, ticks=937/17464, in_queue=28199, util=94.36% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935723: Fri Jan 7 14:06:01 2022 read: IOPS=1842, BW=230MiB/s (242MB/s)(6919MiB/30036msec) slat (usec): min=5, max=508, avg=13.80, stdev= 4.63 clat (nsec): min=797, max=2493.3M, avg=1353315.75, stdev=49766386.42 lat (usec): min=37, max=2493.3k, avg=1367.26, stdev=49766.40 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 43], 40.00th=[ 45], | 50.00th=[ 47], 60.00th=[ 50], 70.00th=[ 55], | 80.00th=[ 61], 90.00th=[ 71], 95.00th=[ 78], | 99.00th=[ 101], 99.50th=[ 126], 99.90th=[ 114820], | 99.95th=[1010828], 99.99th=[2499806] bw ( KiB/s): min=184576, max=457472, per=100.00%, avg=236180.98, stdev=35883.09, samples=60 iops : min= 1442, max= 3574, avg=1845.15, stdev=280.33, samples=60 write: IOPS=463, BW=57.9MiB/s (60.7MB/s)(1740MiB/30036msec) slat (usec): min=11, max=163, avg=29.09, stdev= 9.19 clat (usec): min=1418, max=77833, avg=16101.27, stdev=6654.24 lat (usec): min=1438, max=77857, avg=16130.54, stdev=6654.83 clat percentiles (usec): | 1.00th=[ 3490], 5.00th=[ 7832], 10.00th=[10421], 20.00th=[12256], | 30.00th=[13435], 40.00th=[14746], 50.00th=[15795], 60.00th=[16712], | 70.00th=[17433], 80.00th=[18744], 90.00th=[21365], 95.00th=[25297], | 99.00th=[36963], 99.50th=[64226], 99.90th=[69731], 99.95th=[76022], | 99.99th=[77071] bw ( KiB/s): min=51200, max=115200, per=100.00%, avg=59359.90, stdev=8027.29, samples=60 iops : min= 400, max= 900, avg=463.73, stdev=62.71, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=47.82%, 100=31.22% lat (usec) : 250=0.66%, 500=0.06%, 750=0.01% lat (msec) : 2=0.01%, 4=0.39%, 10=1.44%, 20=15.63%, 50=2.47% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.79%, sys=5.31%, ctx=68735, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55355,13919,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=230MiB/s (242MB/s), 230MiB/s-230MiB/s (242MB/s-242MB/s), io=6919MiB (7255MB), run=30036-30036msec WRITE: bw=57.9MiB/s (60.7MB/s), 57.9MiB/s-57.9MiB/s (60.7MB/s-60.7MB/s), io=1740MiB (1824MB), run=30036-30036msec Disk stats (read/write): vdg: ios=55035/13821, merge=0/0, ticks=68553/215423, in_queue=296505, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936743: Fri Jan 7 14:06:38 2022 read: IOPS=1818, BW=227MiB/s (238MB/s)(6821MiB/30002msec) clat (usec): min=37, max=33489, avg=83.96, stdev=564.29 lat (usec): min=37, max=33490, avg=84.11, stdev=564.29 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 52], 10.00th=[ 54], 20.00th=[ 57], | 30.00th=[ 58], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 67], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 88], | 99.00th=[ 108], 99.50th=[ 121], 99.90th=[11338], 99.95th=[16188], | 99.99th=[20579] bw ( KiB/s): min=177664, max=371968, per=100.00%, avg=232844.87, stdev=28153.30, samples=60 iops : min= 1388, max= 2906, avg=1819.07, stdev=219.96, samples=60 write: IOPS=457, BW=57.1MiB/s (59.9MB/s)(1715MiB/30002msec) clat (usec): min=519, max=43991, avg=1831.10, stdev=1662.06 lat (usec): min=531, max=44003, avg=1844.73, stdev=1661.92 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 816], 10.00th=[ 1090], 20.00th=[ 1401], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2540], | 99.00th=[ 9896], 99.50th=[10159], 99.90th=[35390], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=49408, max=97280, per=100.00%, avg=58524.93, stdev=6407.90, samples=60 iops : min= 386, max= 760, avg=457.17, stdev=50.07, samples=60 lat (usec) : 50=1.92%, 100=76.70%, 250=1.18%, 500=0.01%, 750=0.73% lat (usec) : 1000=0.96% lat (msec) : 2=14.44%, 4=3.59%, 10=0.22%, 20=0.22%, 50=0.03% cpu : usr=1.71%, sys=3.64%, ctx=68289, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54566,13716,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6821MiB (7152MB), run=30002-30002msec WRITE: bw=57.1MiB/s (59.9MB/s), 57.1MiB/s-57.1MiB/s (59.9MB/s-59.9MB/s), io=1715MiB (1798MB), run=30002-30002msec Disk stats (read/write): vdg: ios=54348/13657, merge=0/0, ticks=933/17197, in_queue=28216, util=94.20% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937327: Fri Jan 7 14:07:16 2022 read: IOPS=1839, BW=230MiB/s (241MB/s)(6927MiB/30130msec) slat (usec): min=3, max=173, avg=13.02, stdev= 5.24 clat (nsec): min=821, max=2569.9M, avg=1500331.62, stdev=53709983.84 lat (usec): min=35, max=2569.0k, avg=1513.51, stdev=53709.95 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 54], 70.00th=[ 61], | 80.00th=[ 75], 90.00th=[ 85], 95.00th=[ 98], | 99.00th=[ 478], 99.50th=[ 1037], 99.90th=[ 130548], | 99.95th=[1061159], 99.99th=[2533360] bw ( KiB/s): min=166400, max=496384, per=100.00%, avg=236455.93, stdev=39737.16, samples=60 iops : min= 1300, max= 3878, avg=1847.30, stdev=310.45, samples=60 write: IOPS=462, BW=57.8MiB/s (60.6MB/s)(1742MiB/30130msec) slat (usec): min=12, max=245, avg=33.75, stdev=14.46 clat (msec): min=5, max=2474, avg=102.06, stdev=67.94 lat (msec): min=5, max=2474, avg=102.09, stdev=67.93 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 78], 10.00th=[ 83], 20.00th=[ 88], | 30.00th=[ 91], 40.00th=[ 93], 50.00th=[ 95], 60.00th=[ 97], | 70.00th=[ 102], 80.00th=[ 106], 90.00th=[ 117], 95.00th=[ 142], | 99.00th=[ 224], 99.50th=[ 575], 99.90th=[ 1045], 99.95th=[ 1334], | 99.99th=[ 2039] bw ( KiB/s): min=49920, max=114432, per=100.00%, avg=59246.40, stdev=8017.48, samples=60 iops : min= 390, max= 894, avg=462.83, stdev=62.65, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.03%, 50=40.99% lat (usec) : 100=35.16%, 250=2.43%, 500=0.47%, 750=0.25%, 1000=0.11% lat (msec) : 2=0.24%, 4=0.08%, 10=0.02%, 20=0.18%, 50=0.28% lat (msec) : 100=13.03%, 250=6.41%, 500=0.08%, 750=0.06%, 1000=0.05% lat (msec) : 2000=0.04%, >=2000=0.03% cpu : usr=2.17%, sys=5.07%, ctx=67266, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=55419,13932,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6927MiB (7264MB), run=30130-30130msec WRITE: bw=57.8MiB/s (60.6MB/s), 57.8MiB/s-57.8MiB/s (60.6MB/s-60.6MB/s), io=1742MiB (1826MB), run=30130-30130msec Disk stats (read/write): vdg: ios=55416/13909, merge=0/0, ticks=75163/1404405, in_queue=1500141, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938226: Fri Jan 7 14:07:52 2022 read: IOPS=1802, BW=225MiB/s (236MB/s)(6758MiB/30003msec) clat (usec): min=39, max=23230, avg=78.67, stdev=509.07 lat (usec): min=39, max=23231, avg=78.80, stdev=509.07 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 97], 99.50th=[ 106], 99.90th=[11076], 99.95th=[15401], | 99.99th=[18482] bw ( KiB/s): min=168960, max=375808, per=100.00%, avg=230691.53, stdev=30396.53, samples=60 iops : min= 1320, max= 2936, avg=1802.27, stdev=237.49, samples=60 write: IOPS=452, BW=56.6MiB/s (59.3MB/s)(1698MiB/30003msec) clat (usec): min=528, max=45011, avg=1875.29, stdev=1760.90 lat (usec): min=542, max=45024, avg=1887.34, stdev=1760.78 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 824], 10.00th=[ 1156], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1680], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2540], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[37487], 99.95th=[41681], | 99.99th=[44827] bw ( KiB/s): min=45824, max=97792, per=100.00%, avg=57955.92, stdev=6745.67, samples=60 iops : min= 358, max= 764, avg=452.77, stdev=52.71, samples=60 lat (usec) : 50=3.36%, 100=75.95%, 250=0.52%, 500=0.01%, 750=0.75% lat (usec) : 1000=0.75% lat (msec) : 2=14.20%, 4=3.97%, 10=0.23%, 20=0.25%, 50=0.02% cpu : usr=1.54%, sys=3.38%, ctx=67660, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54067,13584,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6758MiB (7087MB), run=30003-30003msec WRITE: bw=56.6MiB/s (59.3MB/s), 56.6MiB/s-56.6MiB/s (59.3MB/s-59.3MB/s), io=1698MiB (1780MB), run=30003-30003msec Disk stats (read/write): vdg: ios=53879/13523, merge=0/0, ticks=839/17656, in_queue=28240, util=94.47% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938829: Fri Jan 7 14:08:30 2022 read: IOPS=1830, BW=229MiB/s (240MB/s)(6913MiB/30218msec) slat (usec): min=3, max=248, avg=11.80, stdev= 5.37 clat (nsec): min=793, max=2578.7M, avg=1665966.05, stdev=56443097.51 lat (usec): min=36, max=2578.7k, avg=1677.93, stdev=56443.07 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 55], 70.00th=[ 66], | 80.00th=[ 77], 90.00th=[ 92], 95.00th=[ 192], | 99.00th=[ 2278], 99.50th=[ 4015], 99.90th=[ 154141], | 99.95th=[1652556], 99.99th=[2533360] bw ( KiB/s): min=184832, max=516096, per=100.00%, avg=235945.92, stdev=43915.58, samples=60 iops : min= 1444, max= 4032, avg=1843.32, stdev=343.09, samples=60 write: IOPS=460, BW=57.5MiB/s (60.3MB/s)(1738MiB/30218msec) slat (usec): min=12, max=285, avg=33.89, stdev=15.04 clat (msec): min=4, max=2586, avg=210.51, stdev=119.27 lat (msec): min=4, max=2586, avg=210.54, stdev=119.27 clat percentiles (msec): | 1.00th=[ 30], 5.00th=[ 138], 10.00th=[ 159], 20.00th=[ 182], | 30.00th=[ 190], 40.00th=[ 197], 50.00th=[ 201], 60.00th=[ 207], | 70.00th=[ 213], 80.00th=[ 224], 90.00th=[ 247], 95.00th=[ 271], | 99.00th=[ 493], 99.50th=[ 1267], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 2232] bw ( KiB/s): min=47104, max=107264, per=100.00%, avg=58911.63, stdev=7581.45, samples=60 iops : min= 368, max= 838, avg=460.23, stdev=59.24, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=40.78% lat (usec) : 100=32.66%, 250=2.81%, 500=1.09%, 750=0.54%, 1000=0.31% lat (msec) : 2=0.76%, 4=0.48%, 10=0.32%, 20=0.01%, 50=0.43% lat (msec) : 100=0.21%, 250=17.73%, 500=1.55%, 750=0.06%, 1000=0.03% lat (msec) : 2000=0.13%, >=2000=0.04% cpu : usr=2.27%, sys=4.53%, ctx=64390, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=55300,13904,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6913MiB (7248MB), run=30218-30218msec WRITE: bw=57.5MiB/s (60.3MB/s), 57.5MiB/s-57.5MiB/s (60.3MB/s-60.3MB/s), io=1738MiB (1822MB), run=30218-30218msec Disk stats (read/write): vdg: ios=55298/13817, merge=0/0, ticks=85687/2891039, in_queue=2998422, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939647: Fri Jan 7 14:09:06 2022 read: IOPS=1800, BW=225MiB/s (236MB/s)(6753MiB/30004msec) clat (usec): min=37, max=32750, avg=79.85, stdev=533.25 lat (usec): min=37, max=32750, avg=79.99, stdev=533.25 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 104], 99.50th=[ 119], 99.90th=[11731], 99.95th=[15533], | 99.99th=[19792] bw ( KiB/s): min=184576, max=376320, per=100.00%, avg=230518.23, stdev=28862.52, samples=60 iops : min= 1442, max= 2940, avg=1800.90, stdev=225.48, samples=60 write: IOPS=452, BW=56.6MiB/s (59.3MB/s)(1697MiB/30004msec) clat (usec): min=510, max=49330, avg=1871.03, stdev=1725.81 lat (usec): min=530, max=49341, avg=1883.20, stdev=1725.71 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 816], 10.00th=[ 1139], 20.00th=[ 1418], | 30.00th=[ 1516], 40.00th=[ 1598], 50.00th=[ 1680], 60.00th=[ 1778], | 70.00th=[ 1893], 80.00th=[ 2040], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9896], 99.50th=[10159], 99.90th=[35914], 99.95th=[41157], | 99.99th=[44303] bw ( KiB/s): min=46336, max=98048, per=100.00%, avg=57919.82, stdev=6623.00, samples=60 iops : min= 362, max= 766, avg=452.47, stdev=51.72, samples=60 lat (usec) : 50=4.47%, 100=74.42%, 250=0.89%, 500=0.02%, 750=0.77% lat (usec) : 1000=0.83% lat (msec) : 2=14.03%, 4=4.07%, 10=0.25%, 20=0.22%, 50=0.03% cpu : usr=1.65%, sys=3.34%, ctx=67608, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54024,13575,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6753MiB (7081MB), run=30004-30004msec WRITE: bw=56.6MiB/s (59.3MB/s), 56.6MiB/s-56.6MiB/s (59.3MB/s-59.3MB/s), io=1697MiB (1779MB), run=30004-30004msec Disk stats (read/write): vdg: ios=53714/13495, merge=0/0, ticks=886/17591, in_queue=28233, util=94.39% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940613: Fri Jan 7 14:09:44 2022 read: IOPS=1855, BW=232MiB/s (243MB/s)(7021MiB/30266msec) slat (usec): min=3, max=253612, avg=418.70, stdev=2716.93 clat (usec): min=1728, max=2588.8k, avg=34953.60, stdev=54667.20 lat (usec): min=1733, max=2588.8k, avg=35372.46, stdev=54750.11 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 8], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 47], 95.00th=[ 53], | 99.00th=[ 101], 99.50th=[ 259], 99.90th=[ 296], 99.95th=[ 1083], | 99.99th=[ 2534] bw ( KiB/s): min=185088, max=530432, per=100.00%, avg=239343.87, stdev=44834.76, samples=60 iops : min= 1446, max= 4144, avg=1869.85, stdev=350.27, samples=60 write: IOPS=466, BW=58.3MiB/s (61.1MB/s)(1764MiB/30266msec) slat (usec): min=13, max=242904, avg=448.86, stdev=2645.56 clat (msec): min=11, max=2805, avg=287.58, stdev=106.54 lat (msec): min=11, max=2805, avg=288.02, stdev=106.51 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 211], 10.00th=[ 239], 20.00th=[ 257], | 30.00th=[ 266], 40.00th=[ 271], 50.00th=[ 279], 60.00th=[ 288], | 70.00th=[ 296], 80.00th=[ 309], 90.00th=[ 330], 95.00th=[ 363], | 99.00th=[ 743], 99.50th=[ 927], 99.90th=[ 1418], 99.95th=[ 1787], | 99.99th=[ 2735] bw ( KiB/s): min=48640, max=105472, per=99.96%, avg=59667.33, stdev=7141.02, samples=60 iops : min= 380, max= 824, avg=466.13, stdev=55.78, samples=60 lat (msec) : 2=0.09%, 4=1.14%, 10=3.51%, 20=4.80%, 50=65.46% lat (msec) : 100=4.62%, 250=2.92%, 500=17.00%, 750=0.21%, 1000=0.13% lat (msec) : 2000=0.10%, >=2000=0.03% cpu : usr=2.32%, sys=1.94%, ctx=7760, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56169,14114,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=7021MiB (7362MB), run=30266-30266msec WRITE: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=1764MiB (1850MB), run=30266-30266msec Disk stats (read/write): vdg: ios=56165/14114, merge=0/0, ticks=82362/3585964, in_queue=3696843, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941232: Fri Jan 7 14:10:20 2022 read: IOPS=1187, BW=297MiB/s (311MB/s)(8906MiB/30001msec) clat (usec): min=66, max=21186, avg=121.34, stdev=416.16 lat (usec): min=67, max=21186, avg=121.49, stdev=416.17 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 85], 10.00th=[ 87], 20.00th=[ 90], | 30.00th=[ 93], 40.00th=[ 97], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 135], 95.00th=[ 147], | 99.00th=[ 188], 99.50th=[ 217], 99.90th=[ 8586], 99.95th=[11863], | 99.99th=[18220] bw ( KiB/s): min=224256, max=527872, per=100.00%, avg=304000.60, stdev=47816.86, samples=60 iops : min= 876, max= 2062, avg=1187.50, stdev=186.79, samples=60 write: IOPS=298, BW=74.7MiB/s (78.3MB/s)(2242MiB/30001msec) clat (usec): min=868, max=91598, avg=2829.17, stdev=2811.21 lat (usec): min=895, max=91622, avg=2854.57, stdev=2810.95 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1106], 10.00th=[ 1303], 20.00th=[ 1631], | 30.00th=[ 1876], 40.00th=[ 2024], 50.00th=[ 2180], 60.00th=[ 2343], | 70.00th=[ 2507], 80.00th=[ 2868], 90.00th=[ 4752], 95.00th=[ 8586], | 99.00th=[12649], 99.50th=[13173], 99.90th=[46400], 99.95th=[54264], | 99.99th=[91751] bw ( KiB/s): min=53248, max=141312, per=100.00%, avg=76504.25, stdev=12227.00, samples=60 iops : min= 208, max= 552, avg=298.83, stdev=47.75, samples=60 lat (usec) : 100=37.27%, 250=42.35%, 500=0.11%, 750=0.04%, 1000=0.43% lat (msec) : 2=7.23%, 4=9.39%, 10=2.35%, 20=0.79%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.55%, sys=2.80%, ctx=44600, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=35624,8966,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=297MiB/s (311MB/s), 297MiB/s-297MiB/s (311MB/s-311MB/s), io=8906MiB (9339MB), run=30001-30001msec WRITE: bw=74.7MiB/s (78.3MB/s), 74.7MiB/s-74.7MiB/s (78.3MB/s-78.3MB/s), io=2242MiB (2350MB), run=30001-30001msec Disk stats (read/write): vdg: ios=35393/8900, merge=0/0, ticks=489/20698, in_queue=28381, util=94.97% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942444: Fri Jan 7 14:10:57 2022 read: IOPS=1306, BW=327MiB/s (342MB/s)(9797MiB/30003msec) slat (nsec): min=12949, max=89091, avg=19794.65, stdev=5065.30 clat (usec): min=2, max=22377, avg=90.83, stdev=376.97 lat (usec): min=64, max=22394, avg=110.79, stdev=377.05 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 66], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 127], 99.50th=[ 141], 99.90th=[ 7242], 99.95th=[10814], | 99.99th=[14091] bw ( KiB/s): min=242176, max=580608, per=100.00%, avg=334423.10, stdev=50028.17, samples=60 iops : min= 946, max= 2268, avg=1306.33, stdev=195.43, samples=60 write: IOPS=329, BW=82.4MiB/s (86.4MB/s)(2471MiB/30003msec) slat (usec): min=24, max=130, avg=43.05, stdev= 7.39 clat (usec): min=848, max=54558, avg=2544.48, stdev=2397.08 lat (usec): min=890, max=54598, avg=2587.71, stdev=2397.01 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1057], 10.00th=[ 1139], 20.00th=[ 1565], | 30.00th=[ 1876], 40.00th=[ 1991], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2671], 90.00th=[ 4080], 95.00th=[ 5014], | 99.00th=[10683], 99.50th=[12256], 99.90th=[52167], 99.95th=[53216], | 99.99th=[54789] bw ( KiB/s): min=59904, max=154112, per=100.00%, avg=84346.27, stdev=11392.52, samples=60 iops : min= 234, max= 602, avg=329.47, stdev=44.50, samples=60 lat (usec) : 4=0.01%, 50=0.06%, 100=73.88%, 250=5.77%, 500=0.02% lat (usec) : 750=0.01%, 1000=0.60% lat (msec) : 2=7.50%, 4=9.97%, 10=1.64%, 20=0.52%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.74%, sys=4.60%, ctx=49092, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39189,9885,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=327MiB/s (342MB/s), 327MiB/s-327MiB/s (342MB/s-342MB/s), io=9797MiB (10.3GB), run=30003-30003msec WRITE: bw=82.4MiB/s (86.4MB/s), 82.4MiB/s-82.4MiB/s (86.4MB/s-86.4MB/s), io=2471MiB (2591MB), run=30003-30003msec Disk stats (read/write): vdg: ios=38999/9836, merge=0/0, ticks=494/20446, in_queue=28297, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943466: Fri Jan 7 14:11:35 2022 read: IOPS=1361, BW=340MiB/s (357MB/s)(9.97GiB/30002msec) clat (usec): min=53, max=66397, avg=98.81, stdev=627.12 lat (usec): min=53, max=66398, avg=98.95, stdev=627.12 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 125], 99.50th=[ 141], 99.90th=[ 9765], 99.95th=[14484], | 99.99th=[21890] bw ( KiB/s): min=240128, max=570880, per=100.00%, avg=348512.80, stdev=45267.87, samples=60 iops : min= 938, max= 2230, avg=1361.37, stdev=176.83, samples=60 write: IOPS=343, BW=85.9MiB/s (90.1MB/s)(2578MiB/30002msec) clat (usec): min=794, max=131941, avg=2485.67, stdev=2479.31 lat (usec): min=819, max=131966, avg=2509.55, stdev=2479.14 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1090], 10.00th=[ 1287], 20.00th=[ 1811], | 30.00th=[ 1942], 40.00th=[ 2057], 50.00th=[ 2180], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2638], 90.00th=[ 2999], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[11076], 99.90th=[18482], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=56320, max=152064, per=100.00%, avg=87990.18, stdev=10844.37, samples=60 iops : min= 220, max= 594, avg=343.70, stdev=42.36, samples=60 lat (usec) : 100=74.57%, 250=5.13%, 500=0.01%, 750=0.01%, 1000=0.58% lat (msec) : 2=6.50%, 4=11.88%, 10=0.80%, 20=0.48%, 50=0.02% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.64%, sys=2.98%, ctx=51161, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40840,10312,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=340MiB/s (357MB/s), 340MiB/s-340MiB/s (357MB/s-357MB/s), io=9.97GiB (10.7GB), run=30002-30002msec WRITE: bw=85.9MiB/s (90.1MB/s), 85.9MiB/s-85.9MiB/s (90.1MB/s-90.1MB/s), io=2578MiB (2703MB), run=30002-30002msec Disk stats (read/write): vdg: ios=40634/10251, merge=0/0, ticks=795/20326, in_queue=28316, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943996: Fri Jan 7 14:12:11 2022 read: IOPS=1407, BW=352MiB/s (369MB/s)(10.3GiB/30037msec) slat (usec): min=8, max=896, avg=17.46, stdev= 7.08 clat (nsec): min=911, max=64067k, avg=127935.38, stdev=1374580.00 lat (usec): min=49, max=64081, avg=145.56, stdev=1374.64 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 80], 80.00th=[ 98], 90.00th=[ 120], 95.00th=[ 139], | 99.00th=[ 229], 99.50th=[ 351], 99.90th=[30016], 99.95th=[36963], | 99.99th=[52167] bw ( KiB/s): min=279552, max=708608, per=100.00%, avg=360677.97, stdev=57640.96, samples=60 iops : min= 1092, max= 2768, avg=1408.87, stdev=225.15, samples=60 write: IOPS=355, BW=88.8MiB/s (93.1MB/s)(2666MiB/30037msec) slat (usec): min=21, max=247, avg=50.44, stdev=19.81 clat (usec): min=1671, max=89855, avg=27528.98, stdev=10643.18 lat (usec): min=1698, max=89895, avg=27579.62, stdev=10643.58 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 8455], 10.00th=[12911], 20.00th=[20579], | 30.00th=[23987], 40.00th=[25822], 50.00th=[27657], 60.00th=[29754], | 70.00th=[32113], 80.00th=[34341], 90.00th=[38536], 95.00th=[42206], | 99.00th=[60556], 99.50th=[79168], 99.90th=[87557], 99.95th=[88605], | 99.99th=[88605] bw ( KiB/s): min=75776, max=177664, per=100.00%, avg=90920.48, stdev=12690.43, samples=60 iops : min= 296, max= 694, avg=355.13, stdev=49.56, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.83% lat (usec) : 100=59.75%, 250=14.60%, 500=0.40%, 750=0.08%, 1000=0.04% lat (msec) : 2=0.02%, 4=0.02%, 10=1.25%, 20=2.60%, 50=16.08% lat (msec) : 100=0.32% cpu : usr=2.20%, sys=4.45%, ctx=49030, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=42264,10664,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=10.3GiB (11.1GB), run=30037-30037msec WRITE: bw=88.8MiB/s (93.1MB/s), 88.8MiB/s-88.8MiB/s (93.1MB/s-93.1MB/s), io=2666MiB (2796MB), run=30037-30037msec Disk stats (read/write): vdg: ios=41846/10568, merge=0/0, ticks=2120/285775, in_queue=296203, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944973: Fri Jan 7 14:12:48 2022 read: IOPS=1348, BW=337MiB/s (353MB/s)(9.87GiB/30002msec) clat (usec): min=50, max=53614, avg=94.36, stdev=485.89 lat (usec): min=50, max=53615, avg=94.50, stdev=485.89 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 65], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 128], 99.50th=[ 147], 99.90th=[ 8979], 99.95th=[11731], | 99.99th=[15139] bw ( KiB/s): min=264721, max=576512, per=100.00%, avg=345142.68, stdev=44945.30, samples=60 iops : min= 1034, max= 2252, avg=1348.15, stdev=175.60, samples=60 write: IOPS=340, BW=85.1MiB/s (89.3MB/s)(2554MiB/30002msec) clat (usec): min=859, max=54152, avg=2529.47, stdev=2237.61 lat (usec): min=890, max=54177, avg=2554.56, stdev=2237.38 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1074], 10.00th=[ 1254], 20.00th=[ 1827], | 30.00th=[ 1958], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2442], 80.00th=[ 2638], 90.00th=[ 3097], 95.00th=[ 4555], | 99.00th=[10683], 99.50th=[11338], 99.90th=[14484], 99.95th=[53216], | 99.99th=[53216] bw ( KiB/s): min=73728, max=153088, per=100.00%, avg=87175.90, stdev=9950.21, samples=60 iops : min= 288, max= 598, avg=340.45, stdev=38.90, samples=60 lat (usec) : 100=74.33%, 250=5.34%, 500=0.03%, 750=0.01%, 1000=0.50% lat (msec) : 2=6.34%, 4=11.91%, 10=0.97%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.69%, sys=2.78%, ctx=50663, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40443,10216,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=337MiB/s (353MB/s), 337MiB/s-337MiB/s (353MB/s-353MB/s), io=9.87GiB (10.6GB), run=30002-30002msec WRITE: bw=85.1MiB/s (89.3MB/s), 85.1MiB/s-85.1MiB/s (89.3MB/s-89.3MB/s), io=2554MiB (2678MB), run=30002-30002msec Disk stats (read/write): vdg: ios=40443/10215, merge=0/0, ticks=620/20842, in_queue=28541, util=94.86% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945595: Fri Jan 7 14:13:26 2022 read: IOPS=1402, BW=351MiB/s (368MB/s)(10.3GiB/30153msec) slat (usec): min=4, max=204, avg=12.87, stdev= 5.12 clat (nsec): min=840, max=2537.2M, avg=2029845.41, stdev=62272463.44 lat (usec): min=44, max=2537.2k, avg=2042.87, stdev=62272.46 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 51], | 20.00th=[ 54], 30.00th=[ 57], 40.00th=[ 60], | 50.00th=[ 64], 60.00th=[ 72], 70.00th=[ 92], | 80.00th=[ 115], 90.00th=[ 135], 95.00th=[ 169], | 99.00th=[ 1123], 99.50th=[ 2769], 99.90th=[ 526386], | 99.95th=[1904215], 99.99th=[2533360] bw ( KiB/s): min=276480, max=795136, per=100.00%, avg=360947.80, stdev=66247.10, samples=60 iops : min= 1080, max= 3106, avg=1409.93, stdev=258.79, samples=60 write: IOPS=354, BW=88.5MiB/s (92.8MB/s)(2670MiB/30153msec) slat (usec): min=22, max=224, avg=55.27, stdev=24.77 clat (msec): min=14, max=3365, avg=132.97, stdev=72.99 lat (msec): min=14, max=3365, avg=133.02, stdev=72.99 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 94], 10.00th=[ 104], 20.00th=[ 112], | 30.00th=[ 118], 40.00th=[ 123], 50.00th=[ 128], 60.00th=[ 133], | 70.00th=[ 140], 80.00th=[ 146], 90.00th=[ 161], 95.00th=[ 182], | 99.00th=[ 243], 99.50th=[ 460], 99.90th=[ 986], 99.95th=[ 1167], | 99.99th=[ 2836] bw ( KiB/s): min=78848, max=182784, per=100.00%, avg=90757.80, stdev=13078.38, samples=60 iops : min= 308, max= 714, avg=354.50, stdev=51.09, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=4.98% lat (usec) : 100=52.63%, 250=19.76%, 500=0.80%, 750=0.41%, 1000=0.34% lat (msec) : 2=0.35%, 4=0.38%, 10=0.04%, 20=0.05%, 50=0.46% lat (msec) : 100=1.05%, 250=18.43%, 500=0.10%, 750=0.05%, 1000=0.05% lat (msec) : 2000=0.04%, >=2000=0.04% cpu : usr=2.64%, sys=3.55%, ctx=47799, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=42301,10679,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=10.3GiB (11.1GB), run=30153-30153msec WRITE: bw=88.5MiB/s (92.8MB/s), 88.5MiB/s-88.5MiB/s (92.8MB/s-92.8MB/s), io=2670MiB (2799MB), run=30153-30153msec Disk stats (read/write): vdg: ios=42296/10664, merge=0/0, ticks=76194/1408357, in_queue=1500489, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946374: Fri Jan 7 14:14:02 2022 read: IOPS=1362, BW=341MiB/s (357MB/s)(9.98GiB/30002msec) clat (usec): min=48, max=18410, avg=91.79, stdev=392.95 lat (usec): min=49, max=18410, avg=91.93, stdev=392.95 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 66], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 83], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 122], 99.50th=[ 137], 99.90th=[ 8848], 99.95th=[11469], | 99.99th=[13829] bw ( KiB/s): min=243200, max=555520, per=100.00%, avg=348725.78, stdev=43036.39, samples=60 iops : min= 950, max= 2170, avg=1362.20, stdev=168.11, samples=60 write: IOPS=343, BW=85.9MiB/s (90.1MB/s)(2579MiB/30002msec) clat (usec): min=796, max=63699, avg=2512.65, stdev=2315.50 lat (usec): min=819, max=63756, avg=2536.64, stdev=2315.41 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1074], 10.00th=[ 1254], 20.00th=[ 1844], | 30.00th=[ 1958], 40.00th=[ 2073], 50.00th=[ 2180], 60.00th=[ 2311], | 70.00th=[ 2442], 80.00th=[ 2638], 90.00th=[ 3064], 95.00th=[ 4490], | 99.00th=[10552], 99.50th=[11076], 99.90th=[52691], 99.95th=[53216], | 99.99th=[53740] bw ( KiB/s): min=73728, max=147456, per=100.00%, avg=88007.07, stdev=9200.35, samples=60 iops : min= 288, max= 576, avg=343.77, stdev=35.94, samples=60 lat (usec) : 50=0.01%, 100=75.41%, 250=4.29%, 500=0.02%, 1000=0.53% lat (msec) : 2=6.22%, 4=12.03%, 10=0.96%, 20=0.51%, 100=0.02% cpu : usr=1.72%, sys=2.73%, ctx=51190, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40865,10314,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=341MiB/s (357MB/s), 341MiB/s-341MiB/s (357MB/s-357MB/s), io=9.98GiB (10.7GB), run=30002-30002msec WRITE: bw=85.9MiB/s (90.1MB/s), 85.9MiB/s-85.9MiB/s (90.1MB/s-90.1MB/s), io=2579MiB (2704MB), run=30002-30002msec Disk stats (read/write): vdg: ios=40690/10263, merge=0/0, ticks=550/20787, in_queue=28400, util=94.80% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947442: Fri Jan 7 14:14:40 2022 read: IOPS=1412, BW=353MiB/s (370MB/s)(10.4GiB/30307msec) slat (usec): min=4, max=347, avg=12.51, stdev= 6.67 clat (nsec): min=957, max=2666.0M, avg=2303863.63, stdev=64155628.90 lat (usec): min=47, max=2666.0k, avg=2316.52, stdev=64155.64 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 59], | 50.00th=[ 65], 60.00th=[ 74], 70.00th=[ 105], | 80.00th=[ 119], 90.00th=[ 159], 95.00th=[ 619], | 99.00th=[ 5735], 99.50th=[ 7570], 99.90th=[ 884999], | 99.95th=[1971323], 99.99th=[2533360] bw ( KiB/s): min=263680, max=909312, per=100.00%, avg=365193.60, stdev=80017.11, samples=60 iops : min= 1030, max= 3552, avg=1426.50, stdev=312.57, samples=60 write: IOPS=355, BW=88.0MiB/s (93.3MB/s)(2697MiB/30307msec) slat (usec): min=21, max=267, avg=51.78, stdev=23.90 clat (msec): min=26, max=1931, avg=271.68, stdev=81.35 lat (msec): min=26, max=1931, avg=271.73, stdev=81.35 clat percentiles (msec): | 1.00th=[ 63], 5.00th=[ 159], 10.00th=[ 232], 20.00th=[ 247], | 30.00th=[ 257], 40.00th=[ 264], 50.00th=[ 268], 60.00th=[ 275], | 70.00th=[ 284], 80.00th=[ 292], 90.00th=[ 321], 95.00th=[ 351], | 99.00th=[ 523], 99.50th=[ 600], 99.90th=[ 1200], 99.95th=[ 1418], | 99.99th=[ 1821] bw ( KiB/s): min=73728, max=181248, per=100.00%, avg=91229.55, stdev=13352.95, samples=60 iops : min= 288, max= 708, avg=356.33, stdev=52.18, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=5.87%, 100=48.98% lat (usec) : 250=19.29%, 500=1.40%, 750=0.57%, 1000=0.52% lat (msec) : 2=1.19%, 4=0.90%, 10=1.02%, 20=0.01%, 50=0.12% lat (msec) : 100=0.60%, 250=3.79%, 500=15.35%, 750=0.25%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.04% cpu : usr=2.24%, sys=3.69%, ctx=46188, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42796,10786,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=353MiB/s (370MB/s), 353MiB/s-353MiB/s (370MB/s-370MB/s), io=10.4GiB (11.2GB), run=30307-30307msec WRITE: bw=88.0MiB/s (93.3MB/s), 88.0MiB/s-88.0MiB/s (93.3MB/s-93.3MB/s), io=2697MiB (2827MB), run=30307-30307msec Disk stats (read/write): vdg: ios=42792/10721, merge=0/0, ticks=87809/2891108, in_queue=3017428, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948016: Fri Jan 7 14:15:17 2022 read: IOPS=1367, BW=342MiB/s (358MB/s)(10.0GiB/30001msec) clat (usec): min=49, max=21775, avg=95.26, stdev=446.45 lat (usec): min=49, max=21775, avg=95.41, stdev=446.45 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 133], 99.50th=[ 153], 99.90th=[ 8160], 99.95th=[12518], | 99.99th=[19006] bw ( KiB/s): min=264704, max=548352, per=100.00%, avg=350035.78, stdev=45645.33, samples=60 iops : min= 1034, max= 2142, avg=1367.30, stdev=178.29, samples=60 write: IOPS=345, BW=86.3MiB/s (90.5MB/s)(2591MiB/30001msec) clat (usec): min=856, max=58247, avg=2485.34, stdev=2293.93 lat (usec): min=879, max=58272, avg=2509.39, stdev=2293.78 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1074], 10.00th=[ 1237], 20.00th=[ 1811], | 30.00th=[ 1942], 40.00th=[ 2057], 50.00th=[ 2180], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2606], 90.00th=[ 2999], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[10945], 99.90th=[46400], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=69632, max=144896, per=100.00%, avg=88419.98, stdev=9792.79, samples=60 iops : min= 272, max= 566, avg=345.37, stdev=38.24, samples=60 lat (usec) : 50=0.01%, 100=73.87%, 250=5.76%, 500=0.05%, 750=0.03% lat (usec) : 1000=0.55% lat (msec) : 2=6.61%, 4=11.70%, 10=0.90%, 20=0.50%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.69%, sys=3.09%, ctx=51401, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41017,10362,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=342MiB/s (358MB/s), 342MiB/s-342MiB/s (358MB/s-358MB/s), io=10.0GiB (10.8GB), run=30001-30001msec WRITE: bw=86.3MiB/s (90.5MB/s), 86.3MiB/s-86.3MiB/s (90.5MB/s-90.5MB/s), io=2591MiB (2716MB), run=30001-30001msec Disk stats (read/write): vdg: ios=40724/10274, merge=0/0, ticks=616/20462, in_queue=28321, util=94.78% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949369: Fri Jan 7 14:15:54 2022 read: IOPS=1407, BW=352MiB/s (369MB/s)(10.4GiB/30369msec) slat (usec): min=4, max=351101, avg=546.47, stdev=4561.67 clat (msec): min=3, max=2794, avg=47.13, stdev=74.76 lat (msec): min=3, max=2794, avg=47.67, stdev=74.86 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 10], 10.00th=[ 14], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 38], 50.00th=[ 42], 60.00th=[ 45], | 70.00th=[ 50], 80.00th=[ 57], 90.00th=[ 68], 95.00th=[ 79], | 99.00th=[ 296], 99.50th=[ 347], 99.90th=[ 995], 99.95th=[ 2123], | 99.99th=[ 2601] bw ( KiB/s): min=255488, max=887296, per=100.00%, avg=364170.93, stdev=76881.13, samples=60 iops : min= 998, max= 3466, avg=1422.53, stdev=300.32, samples=60 write: IOPS=354, BW=88.7MiB/s (93.0MB/s)(2694MiB/30369msec) slat (usec): min=32, max=292388, avg=606.47, stdev=4168.64 clat (msec): min=24, max=3036, avg=373.71, stdev=199.92 lat (msec): min=24, max=3036, avg=374.32, stdev=199.88 clat percentiles (msec): | 1.00th=[ 67], 5.00th=[ 224], 10.00th=[ 279], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 351], 50.00th=[ 359], 60.00th=[ 372], | 70.00th=[ 384], 80.00th=[ 397], 90.00th=[ 426], 95.00th=[ 527], | 99.00th=[ 869], 99.50th=[ 1670], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3037] bw ( KiB/s): min=73216, max=166400, per=100.00%, avg=90857.80, stdev=12633.20, samples=60 iops : min= 286, max= 650, avg=354.90, stdev=49.34, samples=60 lat (msec) : 4=0.01%, 10=4.80%, 20=6.06%, 50=45.64%, 100=22.36% lat (msec) : 250=1.40%, 500=18.50%, 750=0.86%, 1000=0.11%, 2000=0.13% lat (msec) : >=2000=0.13% cpu : usr=3.32%, sys=1.79%, ctx=5696, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42745,10777,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=352MiB/s (369MB/s), 352MiB/s-352MiB/s (369MB/s-369MB/s), io=10.4GiB (11.2GB), run=30369-30369msec WRITE: bw=88.7MiB/s (93.0MB/s), 88.7MiB/s-88.7MiB/s (93.0MB/s-93.0MB/s), io=2694MiB (2825MB), run=30369-30369msec Disk stats (read/write): vdg: ios=42743/10751, merge=0/0, ticks=116161/3540234, in_queue=3698696, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950051: Fri Jan 7 14:16:31 2022 read: IOPS=920, BW=460MiB/s (483MB/s)(13.5GiB/30002msec) clat (usec): min=90, max=17244, avg=150.98, stdev=539.33 lat (usec): min=90, max=17244, avg=151.14, stdev=539.33 clat percentiles (usec): | 1.00th=[ 99], 5.00th=[ 104], 10.00th=[ 106], 20.00th=[ 110], | 30.00th=[ 113], 40.00th=[ 117], 50.00th=[ 121], 60.00th=[ 127], | 70.00th=[ 135], 80.00th=[ 143], 90.00th=[ 157], 95.00th=[ 174], | 99.00th=[ 221], 99.50th=[ 253], 99.90th=[11600], 99.95th=[14615], | 99.99th=[16450] bw ( KiB/s): min=308224, max=642048, per=100.00%, avg=471415.47, stdev=64821.04, samples=60 iops : min= 602, max= 1254, avg=920.73, stdev=126.60, samples=60 write: IOPS=231, BW=116MiB/s (122MB/s)(3479MiB/30002msec) clat (usec): min=1393, max=61925, avg=3653.17, stdev=2593.63 lat (usec): min=1444, max=61974, avg=3703.12, stdev=2593.15 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1696], 10.00th=[ 1958], 20.00th=[ 2474], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3916], 90.00th=[ 6063], 95.00th=[ 7242], | 99.00th=[11863], 99.50th=[14484], 99.90th=[47973], 99.95th=[54789], | 99.99th=[62129] bw ( KiB/s): min=87040, max=165888, per=99.99%, avg=118732.80, stdev=12156.29, samples=60 iops : min= 170, max= 324, avg=231.90, stdev=23.74, samples=60 lat (usec) : 100=1.23%, 250=78.22%, 500=0.23%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.13%, 4=14.33%, 10=3.14%, 20=0.67%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.67%, sys=2.77%, ctx=34597, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27622,6958,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=460MiB/s (483MB/s), 460MiB/s-460MiB/s (483MB/s-483MB/s), io=13.5GiB (14.5GB), run=30002-30002msec WRITE: bw=116MiB/s (122MB/s), 116MiB/s-116MiB/s (122MB/s-122MB/s), io=3479MiB (3648MB), run=30002-30002msec Disk stats (read/write): vdg: ios=27430/6907, merge=0/0, ticks=631/21648, in_queue=28353, util=94.88% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950868: Fri Jan 7 14:17:08 2022 read: IOPS=946, BW=473MiB/s (496MB/s)(13.9GiB/30004msec) slat (usec): min=18, max=125, avg=29.19, stdev= 6.85 clat (usec): min=44, max=24524, avg=139.80, stdev=548.17 lat (usec): min=104, max=24547, avg=169.17, stdev=548.22 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 94], 10.00th=[ 96], 20.00th=[ 99], | 30.00th=[ 101], 40.00th=[ 105], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 131], 90.00th=[ 145], 95.00th=[ 157], | 99.00th=[ 198], 99.50th=[ 223], 99.90th=[11731], 99.95th=[13698], | 99.99th=[16057] bw ( KiB/s): min=395264, max=616448, per=100.00%, avg=484621.67, stdev=46242.01, samples=60 iops : min= 772, max= 1204, avg=946.52, stdev=90.33, samples=60 write: IOPS=239, BW=120MiB/s (125MB/s)(3586MiB/30004msec) slat (usec): min=39, max=266, avg=79.20, stdev=25.00 clat (usec): min=1399, max=55601, avg=3426.48, stdev=2425.71 lat (usec): min=1472, max=55695, avg=3505.90, stdev=2425.29 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1647], 10.00th=[ 1713], 20.00th=[ 2147], | 30.00th=[ 2573], 40.00th=[ 2868], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3490], 80.00th=[ 3818], 90.00th=[ 5473], 95.00th=[ 6652], | 99.00th=[11600], 99.50th=[14222], 99.90th=[17957], 99.95th=[53740], | 99.99th=[55837] bw ( KiB/s): min=108544, max=178176, per=100.00%, avg=122372.27, stdev=9203.27, samples=60 iops : min= 212, max= 348, avg=239.00, stdev=17.97, samples=60 lat (usec) : 50=0.01%, 100=20.69%, 250=58.88%, 500=0.08%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=3.70%, 4=13.23%, 10=2.82%, 20=0.56%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.95%, sys=4.40%, ctx=35580, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=28395,7171,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=473MiB/s (496MB/s), 473MiB/s-473MiB/s (496MB/s-496MB/s), io=13.9GiB (14.9GB), run=30004-30004msec WRITE: bw=120MiB/s (125MB/s), 120MiB/s-120MiB/s (125MB/s-125MB/s), io=3586MiB (3760MB), run=30004-30004msec Disk stats (read/write): vdg: ios=28238/7129, merge=0/0, ticks=652/20658, in_queue=28281, util=94.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951818: Fri Jan 7 14:17:44 2022 read: IOPS=968, BW=484MiB/s (508MB/s)(14.2GiB/30001msec) clat (usec): min=79, max=16088, avg=134.77, stdev=540.29 lat (usec): min=79, max=16088, avg=134.93, stdev=540.29 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 92], 10.00th=[ 93], 20.00th=[ 96], | 30.00th=[ 99], 40.00th=[ 102], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 118], 80.00th=[ 125], 90.00th=[ 137], 95.00th=[ 145], | 99.00th=[ 178], 99.50th=[ 202], 99.90th=[11994], 99.95th=[13960], | 99.99th=[16057] bw ( KiB/s): min=377856, max=618496, per=100.00%, avg=497247.49, stdev=53494.31, samples=59 iops : min= 738, max= 1208, avg=971.19, stdev=104.48, samples=59 write: IOPS=244, BW=122MiB/s (128MB/s)(3672MiB/30001msec) clat (usec): min=1423, max=63167, avg=3494.28, stdev=2406.34 lat (usec): min=1450, max=63222, avg=3543.20, stdev=2405.69 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2802], 40.00th=[ 3032], 50.00th=[ 3195], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3851], 90.00th=[ 4555], 95.00th=[ 6652], | 99.00th=[11600], 99.50th=[11994], 99.90th=[53216], 99.95th=[54264], | 99.99th=[63177] bw ( KiB/s): min=104448, max=180224, per=100.00%, avg=125418.31, stdev=9645.45, samples=59 iops : min= 204, max= 352, avg=244.95, stdev=18.84, samples=59 lat (usec) : 100=27.92%, 250=51.68%, 500=0.04%, 750=0.01% lat (msec) : 2=2.82%, 4=14.15%, 10=2.85%, 20=0.52%, 100=0.02% cpu : usr=1.92%, sys=2.82%, ctx=36407, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29055,7343,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=484MiB/s (508MB/s), 484MiB/s-484MiB/s (508MB/s-508MB/s), io=14.2GiB (15.2GB), run=30001-30001msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3672MiB (3850MB), run=30001-30001msec Disk stats (read/write): vdg: ios=28872/7296, merge=0/0, ticks=666/21560, in_queue=28366, util=94.89% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952400: Fri Jan 7 14:18:22 2022 read: IOPS=984, BW=492MiB/s (516MB/s)(14.4GiB/30034msec) slat (usec): min=6, max=246, avg=20.77, stdev= 7.53 clat (usec): min=21, max=93033, avg=180.03, stdev=1607.97 lat (usec): min=70, max=93053, avg=200.96, stdev=1608.01 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 74], | 30.00th=[ 76], 40.00th=[ 79], 50.00th=[ 83], 60.00th=[ 90], | 70.00th=[ 105], 80.00th=[ 174], 90.00th=[ 196], 95.00th=[ 215], | 99.00th=[ 318], 99.50th=[ 523], 99.90th=[32900], 99.95th=[38536], | 99.99th=[52167] bw ( KiB/s): min=393216, max=756736, per=100.00%, avg=504537.90, stdev=63285.78, samples=60 iops : min= 768, max= 1478, avg=985.37, stdev=123.63, samples=60 write: IOPS=248, BW=124MiB/s (130MB/s)(3730MiB/30034msec) slat (usec): min=36, max=703, avg=91.11, stdev=43.21 clat (msec): min=2, max=123, avg=39.36, stdev=15.82 lat (msec): min=2, max=123, avg=39.45, stdev=15.82 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 12], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 32], 40.00th=[ 36], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 47], 80.00th=[ 52], 90.00th=[ 58], 95.00th=[ 65], | 99.00th=[ 83], 99.50th=[ 97], 99.90th=[ 114], 99.95th=[ 116], | 99.99th=[ 124] bw ( KiB/s): min=112640, max=205824, per=100.00%, avg=127180.18, stdev=11843.42, samples=60 iops : min= 220, max= 402, avg=248.37, stdev=23.14, samples=60 lat (usec) : 50=0.02%, 100=54.19%, 250=23.80%, 500=1.40%, 750=0.09% lat (usec) : 1000=0.04% lat (msec) : 2=0.13%, 4=0.01%, 10=0.65%, 20=1.72%, 50=13.31% lat (msec) : 100=4.55%, 250=0.08% cpu : usr=2.48%, sys=3.54%, ctx=35354, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29559,7460,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=492MiB/s (516MB/s), 492MiB/s-492MiB/s (516MB/s-516MB/s), io=14.4GiB (15.5GB), run=30034-30034msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3730MiB (3911MB), run=30034-30034msec Disk stats (read/write): vdg: ios=29351/7402, merge=0/0, ticks=2012/287616, in_queue=296603, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953231: Fri Jan 7 14:18:59 2022 read: IOPS=977, BW=489MiB/s (512MB/s)(14.3GiB/30003msec) clat (usec): min=77, max=17052, avg=130.34, stdev=529.25 lat (usec): min=77, max=17052, avg=130.49, stdev=529.25 clat percentiles (usec): | 1.00th=[ 85], 5.00th=[ 89], 10.00th=[ 91], 20.00th=[ 93], | 30.00th=[ 96], 40.00th=[ 99], 50.00th=[ 102], 60.00th=[ 106], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 130], 95.00th=[ 139], | 99.00th=[ 165], 99.50th=[ 186], 99.90th=[11338], 99.95th=[14091], | 99.99th=[15401] bw ( KiB/s): min=389120, max=655360, per=100.00%, avg=500363.30, stdev=53357.01, samples=60 iops : min= 760, max= 1280, avg=977.27, stdev=104.20, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3701MiB/30003msec) clat (usec): min=1356, max=62109, avg=3478.55, stdev=2343.28 lat (usec): min=1401, max=62160, avg=3527.18, stdev=2342.99 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1647], 10.00th=[ 1745], 20.00th=[ 2474], | 30.00th=[ 2835], 40.00th=[ 3064], 50.00th=[ 3228], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3818], 90.00th=[ 4424], 95.00th=[ 6390], | 99.00th=[11600], 99.50th=[12387], 99.90th=[36963], 99.95th=[54789], | 99.99th=[62129] bw ( KiB/s): min=104448, max=188416, per=100.00%, avg=126314.87, stdev=11059.17, samples=60 iops : min= 204, max= 368, avg=246.70, stdev=21.59, samples=60 lat (usec) : 100=35.49%, 250=44.13%, 500=0.05%, 1000=0.01% lat (msec) : 2=2.63%, 4=14.46%, 10=2.74%, 20=0.48%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=2.46%, ctx=36728, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29317,7402,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=489MiB/s (512MB/s), 489MiB/s-489MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30003-30003msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3701MiB (3881MB), run=30003-30003msec Disk stats (read/write): vdg: ios=29055/7342, merge=0/0, ticks=661/21647, in_queue=28442, util=95.17% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954254: Fri Jan 7 14:19:36 2022 read: IOPS=984, BW=492MiB/s (516MB/s)(14.5GiB/30223msec) slat (usec): min=6, max=833, avg=15.80, stdev= 8.49 clat (nsec): min=1408, max=252819k, avg=408054.48, stdev=4896597.35 lat (usec): min=70, max=252829, avg=424.00, stdev=4896.62 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 88], | 70.00th=[ 106], 80.00th=[ 184], 90.00th=[ 206], 95.00th=[ 314], | 99.00th=[ 4424], 99.50th=[ 7111], 99.90th=[ 87557], 99.95th=[121111], | 99.99th=[189793] bw ( KiB/s): min=367616, max=987136, per=100.00%, avg=507664.70, stdev=82761.78, samples=60 iops : min= 718, max= 1928, avg=991.52, stdev=161.65, samples=60 write: IOPS=248, BW=124MiB/s (130MB/s)(3753MiB/30223msec) slat (usec): min=33, max=492, avg=92.25, stdev=44.89 clat (msec): min=9, max=2735, avg=199.54, stdev=116.12 lat (msec): min=9, max=2735, avg=199.63, stdev=116.12 clat percentiles (msec): | 1.00th=[ 49], 5.00th=[ 70], 10.00th=[ 84], 20.00th=[ 105], | 30.00th=[ 132], 40.00th=[ 167], 50.00th=[ 205], 60.00th=[ 230], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 330], | 99.00th=[ 468], 99.50th=[ 701], 99.90th=[ 1586], 99.95th=[ 1955], | 99.99th=[ 2735] bw ( KiB/s): min=106496, max=219136, per=100.00%, avg=127240.47, stdev=14115.58, samples=60 iops : min= 208, max= 428, avg=248.50, stdev=27.57, samples=60 lat (usec) : 2=0.01%, 50=0.02%, 100=54.43%, 250=20.62%, 500=1.51% lat (usec) : 750=0.41%, 1000=0.35% lat (msec) : 2=1.05%, 4=0.61%, 10=0.70%, 20=0.02%, 50=0.22% lat (msec) : 100=3.51%, 250=10.84%, 500=5.57%, 750=0.07%, 1000=0.03% lat (msec) : 2000=0.05%, >=2000=0.01% cpu : usr=2.66%, sys=2.80%, ctx=34767, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=29744,7505,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=492MiB/s (516MB/s), 492MiB/s-492MiB/s (516MB/s-516MB/s), io=14.5GiB (15.6GB), run=30223-30223msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3753MiB (3935MB), run=30223-30223msec Disk stats (read/write): vdg: ios=29744/7465, merge=0/0, ticks=7947/1473870, in_queue=1499832, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954777: Fri Jan 7 14:20:13 2022 read: IOPS=974, BW=487MiB/s (511MB/s)(14.3GiB/30001msec) clat (usec): min=78, max=23182, avg=135.04, stdev=533.27 lat (usec): min=78, max=23182, avg=135.19, stdev=533.27 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 97], | 30.00th=[ 100], 40.00th=[ 103], 50.00th=[ 106], 60.00th=[ 112], | 70.00th=[ 119], 80.00th=[ 126], 90.00th=[ 137], 95.00th=[ 145], | 99.00th=[ 180], 99.50th=[ 210], 99.90th=[11469], 99.95th=[12780], | 99.99th=[18744] bw ( KiB/s): min=371430, max=655073, per=100.00%, avg=499490.33, stdev=52359.81, samples=60 iops : min= 725, max= 1279, avg=975.35, stdev=102.26, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3694MiB/30001msec) clat (usec): min=1383, max=60886, avg=3465.90, stdev=2389.23 lat (usec): min=1430, max=60936, avg=3515.83, stdev=2389.19 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2474], | 30.00th=[ 2802], 40.00th=[ 3032], 50.00th=[ 3195], 60.00th=[ 3359], | 70.00th=[ 3556], 80.00th=[ 3785], 90.00th=[ 4424], 95.00th=[ 6390], | 99.00th=[11469], 99.50th=[12256], 99.90th=[53740], 99.95th=[55313], | 99.99th=[61080] bw ( KiB/s): min=106496, max=189061, per=100.00%, avg=126203.07, stdev=10965.67, samples=60 iops : min= 208, max= 369, avg=246.35, stdev=21.43, samples=60 lat (usec) : 100=24.75%, 250=54.82%, 500=0.06%, 750=0.02% lat (msec) : 2=2.77%, 4=14.34%, 10=2.72%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.92%, sys=2.84%, ctx=36636, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29238,7388,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=487MiB/s (511MB/s), 487MiB/s-487MiB/s (511MB/s-511MB/s), io=14.3GiB (15.3GB), run=30001-30001msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3694MiB (3873MB), run=30001-30001msec Disk stats (read/write): vdg: ios=29236/7387, merge=0/0, ticks=655/21674, in_queue=28435, util=94.42% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956005: Fri Jan 7 14:20:50 2022 read: IOPS=980, BW=490MiB/s (514MB/s)(14.6GiB/30449msec) slat (usec): min=5, max=452, avg=14.17, stdev=10.35 clat (nsec): min=1844, max=221645k, avg=849312.73, stdev=5599073.07 lat (usec): min=68, max=221654, avg=863.62, stdev=5599.08 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 72], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 82], 60.00th=[ 93], | 70.00th=[ 180], 80.00th=[ 196], 90.00th=[ 660], 95.00th=[ 4146], | 99.00th=[ 12911], 99.50th=[ 14877], 99.90th=[ 94897], 99.95th=[135267], | 99.99th=[221250] bw ( KiB/s): min=135438, max=1057792, per=100.00%, avg=509689.23, stdev=151156.41, samples=60 iops : min= 264, max= 2066, avg=995.47, stdev=295.24, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3770MiB/30449msec) slat (usec): min=36, max=1350, avg=92.07, stdev=48.17 clat (msec): min=35, max=2683, avg=400.26, stdev=155.62 lat (msec): min=35, max=2683, avg=400.35, stdev=155.61 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 218], 10.00th=[ 264], 20.00th=[ 296], | 30.00th=[ 321], 40.00th=[ 355], 50.00th=[ 397], 60.00th=[ 426], | 70.00th=[ 451], 80.00th=[ 477], 90.00th=[ 531], 95.00th=[ 609], | 99.00th=[ 852], 99.50th=[ 1116], 99.90th=[ 1871], 99.95th=[ 2333], | 99.99th=[ 2668] bw ( KiB/s): min=39936, max=204800, per=100.00%, avg=126966.10, stdev=31132.96, samples=60 iops : min= 78, max= 400, avg=247.97, stdev=60.80, samples=60 lat (usec) : 2=0.01%, 50=0.04%, 100=49.74%, 250=18.71%, 500=2.57% lat (usec) : 750=1.08%, 1000=0.62% lat (msec) : 2=1.65%, 4=1.34%, 10=2.48%, 20=1.43%, 50=0.07% lat (msec) : 100=0.26%, 250=1.33%, 500=15.80%, 750=2.39%, 1000=0.35% lat (msec) : 2000=0.11%, >=2000=0.02% cpu : usr=2.52%, sys=2.59%, ctx=31414, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=29863,7539,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=490MiB/s (514MB/s), 490MiB/s-490MiB/s (514MB/s-514MB/s), io=14.6GiB (15.7GB), run=30449-30449msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3770MiB (3953MB), run=30449-30449msec Disk stats (read/write): vdg: ios=29863/7493, merge=0/0, ticks=18648/2965527, in_queue=3012928, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956743: Fri Jan 7 14:21:27 2022 read: IOPS=985, BW=493MiB/s (516MB/s)(14.4GiB/30001msec) clat (usec): min=78, max=20261, avg=131.84, stdev=551.66 lat (usec): min=78, max=20261, avg=131.99, stdev=551.66 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 90], 10.00th=[ 92], 20.00th=[ 94], | 30.00th=[ 96], 40.00th=[ 98], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 114], 80.00th=[ 121], 90.00th=[ 131], 95.00th=[ 141], | 99.00th=[ 176], 99.50th=[ 208], 99.90th=[11600], 99.95th=[14484], | 99.99th=[16581] bw ( KiB/s): min=387072, max=658432, per=99.97%, avg=504223.44, stdev=54457.56, samples=59 iops : min= 756, max= 1286, avg=984.78, stdev=106.37, samples=59 write: IOPS=248, BW=124MiB/s (130MB/s)(3730MiB/30001msec) clat (usec): min=1393, max=54698, avg=3441.95, stdev=2285.56 lat (usec): min=1440, max=54733, avg=3490.34, stdev=2285.03 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1663], 10.00th=[ 1762], 20.00th=[ 2507], | 30.00th=[ 2835], 40.00th=[ 3032], 50.00th=[ 3195], 60.00th=[ 3359], | 70.00th=[ 3523], 80.00th=[ 3785], 90.00th=[ 4359], 95.00th=[ 6194], | 99.00th=[11469], 99.50th=[11863], 99.90th=[52691], 99.95th=[54264], | 99.99th=[54789] bw ( KiB/s): min=113664, max=191488, per=100.00%, avg=127387.61, stdev=10590.00, samples=59 iops : min= 222, max= 374, avg=248.78, stdev=20.69, samples=59 lat (usec) : 100=37.78%, 250=41.80%, 500=0.07%, 750=0.02%, 1000=0.01% lat (msec) : 2=2.57%, 4=14.71%, 10=2.56%, 20=0.47%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.87%, sys=2.74%, ctx=37024, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29553,7460,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=493MiB/s (516MB/s), 493MiB/s-493MiB/s (516MB/s-516MB/s), io=14.4GiB (15.5GB), run=30001-30001msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3730MiB (3911MB), run=30001-30001msec Disk stats (read/write): vdg: ios=29334/7406, merge=0/0, ticks=695/21538, in_queue=28337, util=94.70% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957726: Fri Jan 7 14:22:04 2022 read: IOPS=985, BW=493MiB/s (517MB/s)(14.7GiB/30477msec) slat (usec): min=6, max=462948, avg=755.36, stdev=5803.49 clat (msec): min=9, max=649, avg=65.11, stdev=55.31 lat (msec): min=9, max=649, avg=65.86, stdev=55.65 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 19], 10.00th=[ 26], 20.00th=[ 37], | 30.00th=[ 46], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 65], | 70.00th=[ 72], 80.00th=[ 82], 90.00th=[ 95], 95.00th=[ 109], | 99.00th=[ 464], 99.50th=[ 502], 99.90th=[ 550], 99.95th=[ 558], | 99.99th=[ 558] bw ( KiB/s): min=29696, max=1170432, per=100.00%, avg=511384.33, stdev=181735.70, samples=60 iops : min= 58, max= 2286, avg=998.78, stdev=354.96, samples=60 write: IOPS=248, BW=124MiB/s (130MB/s)(3790MiB/30477msec) slat (usec): min=129, max=476846, avg=953.04, stdev=8296.34 clat (msec): min=24, max=3935, avg=541.99, stdev=240.47 lat (msec): min=25, max=3935, avg=542.94, stdev=240.27 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 271], 10.00th=[ 359], 20.00th=[ 422], | 30.00th=[ 460], 40.00th=[ 489], 50.00th=[ 518], 60.00th=[ 550], | 70.00th=[ 575], 80.00th=[ 617], 90.00th=[ 701], 95.00th=[ 902], | 99.00th=[ 1418], 99.50th=[ 2106], 99.90th=[ 2500], 99.95th=[ 3943], | 99.99th=[ 3943] bw ( KiB/s): min=11264, max=235520, per=99.78%, avg=127048.57, stdev=33159.10, samples=60 iops : min= 22, max= 460, avg=248.13, stdev=64.76, samples=60 lat (msec) : 10=0.02%, 20=5.02%, 50=24.22%, 100=44.27%, 250=6.21% lat (msec) : 500=8.44%, 750=10.14%, 1000=1.08%, 2000=0.51%, >=2000=0.11% cpu : usr=4.46%, sys=1.77%, ctx=3817, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30028,7579,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=493MiB/s (517MB/s), 493MiB/s-493MiB/s (517MB/s-517MB/s), io=14.7GiB (15.7GB), run=30477-30477msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3790MiB (3974MB), run=30477-30477msec Disk stats (read/write): vdg: ios=30028/7514, merge=0/0, ticks=71052/3595416, in_queue=3709106, util=99.52% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958711: Fri Jan 7 14:22:41 2022 read: IOPS=593, BW=594MiB/s (623MB/s)(17.4GiB/30001msec) clat (usec): min=161, max=27083, avg=262.32, stdev=936.58 lat (usec): min=161, max=27083, avg=262.48, stdev=936.58 clat percentiles (usec): | 1.00th=[ 172], 5.00th=[ 176], 10.00th=[ 178], 20.00th=[ 182], | 30.00th=[ 186], 40.00th=[ 190], 50.00th=[ 194], 60.00th=[ 204], | 70.00th=[ 217], 80.00th=[ 229], 90.00th=[ 247], 95.00th=[ 269], | 99.00th=[ 359], 99.50th=[ 465], 99.90th=[17695], 99.95th=[18744], | 99.99th=[24249] bw ( KiB/s): min=381691, max=827392, per=100.00%, avg=608233.73, stdev=92422.37, samples=60 iops : min= 372, max= 808, avg=593.92, stdev=90.26, samples=60 write: IOPS=150, BW=150MiB/s (158MB/s)(4508MiB/30001msec) clat (usec): min=2574, max=57829, avg=5511.07, stdev=3127.99 lat (usec): min=2643, max=57884, avg=5608.80, stdev=3126.58 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2704], 10.00th=[ 2769], 20.00th=[ 3326], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[16319], 99.50th=[17695], 99.90th=[50594], 99.95th=[55837], | 99.99th=[57934] bw ( KiB/s): min=125178, max=216654, per=99.99%, avg=153858.15, stdev=13662.47, samples=60 iops : min= 122, max= 211, avg=150.20, stdev=13.30, samples=60 lat (usec) : 250=72.63%, 500=6.81%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.33%, 10=13.42%, 20=1.69%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=2.44%, ctx=22334, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17817,4508,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=594MiB/s (623MB/s), 594MiB/s-594MiB/s (623MB/s-623MB/s), io=17.4GiB (18.7GB), run=30001-30001msec WRITE: bw=150MiB/s (158MB/s), 150MiB/s-150MiB/s (158MB/s-158MB/s), io=4508MiB (4727MB), run=30001-30001msec Disk stats (read/write): vdg: ios=17817/4507, merge=0/0, ticks=977/22267, in_queue=28623, util=95.11% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959289: Fri Jan 7 14:23:18 2022 read: IOPS=597, BW=597MiB/s (626MB/s)(17.5GiB/30001msec) slat (usec): min=28, max=180, avg=38.53, stdev= 8.36 clat (usec): min=41, max=32539, avg=200.82, stdev=1001.45 lat (usec): min=136, max=32573, avg=239.55, stdev=1001.58 clat percentiles (usec): | 1.00th=[ 112], 5.00th=[ 116], 10.00th=[ 117], 20.00th=[ 120], | 30.00th=[ 123], 40.00th=[ 126], 50.00th=[ 129], 60.00th=[ 135], | 70.00th=[ 147], 80.00th=[ 159], 90.00th=[ 174], 95.00th=[ 192], | 99.00th=[ 302], 99.50th=[ 490], 99.90th=[17433], 99.95th=[23200], | 99.99th=[27132] bw ( KiB/s): min=430080, max=755712, per=99.77%, avg=610118.66, stdev=77391.32, samples=59 iops : min= 420, max= 738, avg=595.80, stdev=75.60, samples=59 write: IOPS=150, BW=151MiB/s (158MB/s)(4529MiB/30001msec) slat (usec): min=68, max=441, avg=130.18, stdev=28.34 clat (usec): min=2514, max=61073, avg=5536.75, stdev=3117.33 lat (usec): min=2650, max=61204, avg=5667.17, stdev=3116.28 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2704], 20.00th=[ 3490], | 30.00th=[ 4424], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5407], | 70.00th=[ 5735], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[15795], 99.50th=[17171], 99.90th=[55313], 99.95th=[56886], | 99.99th=[61080] bw ( KiB/s): min=131072, max=217088, per=100.00%, avg=154691.56, stdev=12750.59, samples=59 iops : min= 128, max= 212, avg=151.05, stdev=12.45, samples=59 lat (usec) : 50=0.01%, 100=0.10%, 250=78.45%, 500=0.87%, 750=0.04% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=4.92%, 10=13.82%, 20=1.69%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.99%, sys=3.40%, ctx=22462, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17917,4529,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=597MiB/s (626MB/s), 597MiB/s-597MiB/s (626MB/s-626MB/s), io=17.5GiB (18.8GB), run=30001-30001msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4529MiB (4749MB), run=30001-30001msec Disk stats (read/write): vdg: ios=17793/4504, merge=0/0, ticks=1022/22570, in_queue=28635, util=95.79% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960093: Fri Jan 7 14:23:55 2022 read: IOPS=603, BW=603MiB/s (633MB/s)(17.7GiB/30002msec) clat (usec): min=129, max=26527, avg=225.36, stdev=937.22 lat (usec): min=130, max=26527, avg=225.53, stdev=937.22 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 208], 95.00th=[ 227], | 99.00th=[ 318], 99.50th=[ 449], 99.90th=[16909], 99.95th=[18744], | 99.99th=[25560] bw ( KiB/s): min=393216, max=813056, per=100.00%, avg=618018.13, stdev=90225.03, samples=60 iops : min= 384, max= 794, avg=603.53, stdev=88.11, samples=60 write: IOPS=152, BW=152MiB/s (160MB/s)(4574MiB/30002msec) clat (usec): min=2534, max=56392, avg=5558.18, stdev=3010.73 lat (usec): min=2635, max=56489, avg=5657.37, stdev=3008.96 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3523], | 30.00th=[ 4424], 40.00th=[ 4883], 50.00th=[ 5211], 60.00th=[ 5473], | 70.00th=[ 5735], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[15533], 99.50th=[16581], 99.90th=[35390], 99.95th=[55313], | 99.99th=[56361] bw ( KiB/s): min=135168, max=217088, per=99.99%, avg=156091.73, stdev=11864.70, samples=60 iops : min= 132, max= 212, avg=152.43, stdev=11.59, samples=60 lat (usec) : 250=77.53%, 500=1.94%, 750=0.04%, 1000=0.01% lat (msec) : 4=4.94%, 10=13.79%, 20=1.70%, 50=0.04%, 100=0.02% cpu : usr=1.98%, sys=2.41%, ctx=22688, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18106,4574,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=603MiB/s (633MB/s), 603MiB/s-603MiB/s (633MB/s-633MB/s), io=17.7GiB (18.0GB), run=30002-30002msec WRITE: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=4574MiB (4796MB), run=30002-30002msec Disk stats (read/write): vdg: ios=17985/4541, merge=0/0, ticks=992/22681, in_queue=28427, util=95.15% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961059: Fri Jan 7 14:24:32 2022 read: IOPS=565, BW=566MiB/s (593MB/s)(16.6GiB/30067msec) slat (usec): min=9, max=107, avg=22.79, stdev= 9.03 clat (usec): min=61, max=130837, avg=318.96, stdev=2544.36 lat (usec): min=112, max=130873, avg=341.92, stdev=2544.57 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 110], 20.00th=[ 112], | 30.00th=[ 114], 40.00th=[ 117], 50.00th=[ 120], 60.00th=[ 126], | 70.00th=[ 141], 80.00th=[ 326], 90.00th=[ 343], 95.00th=[ 359], | 99.00th=[ 619], 99.50th=[ 2180], 99.90th=[42730], 99.95th=[47449], | 99.99th=[90702] bw ( KiB/s): min=409600, max=956416, per=100.00%, avg=580471.47, stdev=97692.74, samples=60 iops : min= 400, max= 934, avg=566.87, stdev=95.40, samples=60 write: IOPS=143, BW=144MiB/s (151MB/s)(4316MiB/30067msec) slat (usec): min=61, max=531, avg=167.47, stdev=85.74 clat (msec): min=3, max=189, avg=68.12, stdev=28.52 lat (msec): min=3, max=189, avg=68.29, stdev=28.52 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 25], 20.00th=[ 42], | 30.00th=[ 58], 40.00th=[ 66], 50.00th=[ 71], 60.00th=[ 78], | 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 102], 95.00th=[ 112], | 99.00th=[ 134], 99.50th=[ 146], 99.90th=[ 182], 99.95th=[ 186], | 99.99th=[ 190] bw ( KiB/s): min=131072, max=243712, per=99.99%, avg=146978.13, stdev=14561.67, samples=60 iops : min= 128, max= 238, avg=143.53, stdev=14.22, samples=60 lat (usec) : 100=0.17%, 250=62.00%, 500=16.51%, 750=0.41%, 1000=0.08% lat (msec) : 2=0.16%, 4=0.10%, 10=0.08%, 20=1.33%, 50=3.76% lat (msec) : 100=13.15%, 250=2.24% cpu : usr=2.63%, sys=2.04%, ctx=20832, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17006,4316,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=566MiB/s (593MB/s), 566MiB/s-566MiB/s (593MB/s-593MB/s), io=16.6GiB (17.8GB), run=30067-30067msec WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=4316MiB (4526MB), run=30067-30067msec Disk stats (read/write): vdg: ios=17006/4312, merge=0/0, ticks=2467/291283, in_queue=299104, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961580: Fri Jan 7 14:25:09 2022 read: IOPS=603, BW=604MiB/s (633MB/s)(17.7GiB/30008msec) clat (usec): min=119, max=27855, avg=210.28, stdev=949.60 lat (usec): min=119, max=27855, avg=210.44, stdev=949.60 clat percentiles (usec): | 1.00th=[ 125], 5.00th=[ 130], 10.00th=[ 133], 20.00th=[ 135], | 30.00th=[ 137], 40.00th=[ 141], 50.00th=[ 143], 60.00th=[ 149], | 70.00th=[ 157], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 198], | 99.00th=[ 265], 99.50th=[ 392], 99.90th=[17171], 99.95th=[18744], | 99.99th=[23987] bw ( KiB/s): min=413696, max=792576, per=100.00%, avg=618552.77, stdev=88246.59, samples=60 iops : min= 404, max= 774, avg=604.05, stdev=86.17, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4578MiB/30008msec) clat (usec): min=2537, max=56892, avg=5615.49, stdev=3018.22 lat (usec): min=2642, max=56987, avg=5713.37, stdev=3016.54 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3916], | 30.00th=[ 4621], 40.00th=[ 5014], 50.00th=[ 5276], 60.00th=[ 5473], | 70.00th=[ 5735], 80.00th=[ 6128], 90.00th=[ 8160], 95.00th=[11338], | 99.00th=[16057], 99.50th=[17433], 99.90th=[53740], 99.95th=[54789], | 99.99th=[56886] bw ( KiB/s): min=124928, max=221184, per=100.00%, avg=156233.45, stdev=12189.60, samples=60 iops : min= 122, max= 216, avg=152.57, stdev=11.90, samples=60 lat (usec) : 250=78.89%, 500=0.60%, 750=0.01% lat (msec) : 2=0.01%, 4=4.18%, 10=14.63%, 20=1.62%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.88%, sys=1.68%, ctx=22707, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18121,4578,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=17.7GiB (19.0GB), run=30008-30008msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4578MiB (4800MB), run=30008-30008msec Disk stats (read/write): vdg: ios=18120/4576, merge=0/0, ticks=1023/23177, in_queue=28803, util=95.69% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962983: Fri Jan 7 14:25:46 2022 read: IOPS=581, BW=582MiB/s (610MB/s)(17.2GiB/30334msec) slat (usec): min=9, max=387, avg=20.67, stdev=11.85 clat (usec): min=44, max=757382, avg=1229.87, stdev=13463.71 lat (usec): min=113, max=757398, avg=1250.72, stdev=13463.81 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 124], 60.00th=[ 137], | 70.00th=[ 235], 80.00th=[ 338], 90.00th=[ 441], 95.00th=[ 2737], | 99.00th=[ 12387], 99.50th=[ 15139], 99.90th=[179307], 99.95th=[256902], | 99.99th=[633340] bw ( KiB/s): min=190464, max=1171456, per=100.00%, avg=602487.47, stdev=174581.98, samples=60 iops : min= 186, max= 1144, avg=588.37, stdev=170.49, samples=60 write: IOPS=147, BW=147MiB/s (155MB/s)(4472MiB/30334msec) slat (usec): min=61, max=791, avg=179.67, stdev=92.19 clat (msec): min=11, max=1440, avg=333.93, stdev=188.34 lat (msec): min=12, max=1440, avg=334.11, stdev=188.33 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 126], 10.00th=[ 140], 20.00th=[ 165], | 30.00th=[ 197], 40.00th=[ 236], 50.00th=[ 292], 60.00th=[ 359], | 70.00th=[ 414], 80.00th=[ 481], 90.00th=[ 592], 95.00th=[ 684], | 99.00th=[ 919], 99.50th=[ 986], 99.90th=[ 1167], 99.95th=[ 1301], | 99.99th=[ 1435] bw ( KiB/s): min=57344, max=239616, per=99.98%, avg=150937.60, stdev=36369.51, samples=60 iops : min= 56, max= 234, avg=147.40, stdev=35.52, samples=60 lat (usec) : 50=0.01%, 100=0.21%, 250=56.06%, 500=16.06%, 750=0.97% lat (usec) : 1000=0.64% lat (msec) : 2=1.28%, 4=1.31%, 10=1.98%, 20=0.96%, 50=0.06% lat (msec) : 100=0.47%, 250=8.36%, 500=8.03%, 750=2.97%, 1000=0.56% lat (msec) : 2000=0.08% cpu : usr=2.81%, sys=1.99%, ctx=19720, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=17651,4472,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=582MiB/s (610MB/s), 582MiB/s-582MiB/s (610MB/s-610MB/s), io=17.2GiB (18.5GB), run=30334-30334msec WRITE: bw=147MiB/s (155MB/s), 147MiB/s-147MiB/s (155MB/s-155MB/s), io=4472MiB (4689MB), run=30334-30334msec Disk stats (read/write): vdg: ios=17651/4454, merge=0/0, ticks=17159/1475352, in_queue=1504861, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963561: Fri Jan 7 14:26:23 2022 read: IOPS=604, BW=604MiB/s (634MB/s)(17.7GiB/30005msec) clat (usec): min=131, max=45554, avg=231.24, stdev=1038.99 lat (usec): min=131, max=45554, avg=231.40, stdev=1038.99 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 204], 95.00th=[ 221], | 99.00th=[ 306], 99.50th=[ 445], 99.90th=[17695], 99.95th=[19530], | 99.99th=[27919] bw ( KiB/s): min=421888, max=827392, per=100.00%, avg=618992.05, stdev=83034.15, samples=60 iops : min= 412, max= 808, avg=604.47, stdev=81.08, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4580MiB/30005msec) clat (usec): min=2559, max=56387, avg=5526.86, stdev=2884.54 lat (usec): min=2669, max=56479, avg=5626.31, stdev=2883.07 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3720], | 30.00th=[ 4555], 40.00th=[ 4948], 50.00th=[ 5211], 60.00th=[ 5473], | 70.00th=[ 5735], 80.00th=[ 6063], 90.00th=[ 8029], 95.00th=[11076], | 99.00th=[15139], 99.50th=[16450], 99.90th=[24249], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min=131072, max=210944, per=100.00%, avg=156301.80, stdev=11259.36, samples=60 iops : min= 128, max= 206, avg=152.62, stdev=11.00, samples=60 lat (usec) : 250=78.20%, 500=1.26%, 750=0.04% lat (msec) : 2=0.01%, 4=4.53%, 10=14.29%, 20=1.60%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=2.01%, sys=2.41%, ctx=22722, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18134,4580,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=604MiB/s (634MB/s), 604MiB/s-604MiB/s (634MB/s-634MB/s), io=17.7GiB (19.0GB), run=30005-30005msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4580MiB (4802MB), run=30005-30005msec Disk stats (read/write): vdg: ios=18134/4579, merge=0/0, ticks=1104/22745, in_queue=28648, util=95.05% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964379: Fri Jan 7 14:27:00 2022 read: IOPS=577, BW=577MiB/s (605MB/s)(17.3GiB/30689msec) slat (usec): min=9, max=730, avg=19.67, stdev=20.65 clat (usec): min=85, max=1258.7k, avg=3177.99, stdev=18285.76 lat (usec): min=112, max=1258.8k, avg=3197.81, stdev=18286.29 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 110], 10.00th=[ 112], | 20.00th=[ 115], 30.00th=[ 118], 40.00th=[ 123], | 50.00th=[ 135], 60.00th=[ 262], 70.00th=[ 347], | 80.00th=[ 758], 90.00th=[ 9503], 95.00th=[ 20317], | 99.00th=[ 31327], 99.50th=[ 35390], 99.90th=[ 168821], | 99.95th=[ 238027], 99.99th=[1061159] bw ( KiB/s): min=28672, max=1310720, per=100.00%, avg=604483.18, stdev=213013.81, samples=60 iops : min= 28, max= 1280, avg=590.27, stdev=208.00, samples=60 write: IOPS=146, BW=146MiB/s (153MB/s)(4485MiB/30689msec) slat (usec): min=61, max=996, avg=198.20, stdev=106.96 clat (msec): min=29, max=2654, avg=671.14, stdev=291.07 lat (msec): min=29, max=2655, avg=671.34, stdev=291.06 clat percentiles (msec): | 1.00th=[ 126], 5.00th=[ 288], 10.00th=[ 380], 20.00th=[ 472], | 30.00th=[ 514], 40.00th=[ 558], 50.00th=[ 609], 60.00th=[ 693], | 70.00th=[ 768], 80.00th=[ 852], 90.00th=[ 1028], 95.00th=[ 1217], | 99.00th=[ 1804], 99.50th=[ 1821], 99.90th=[ 2039], 99.95th=[ 2198], | 99.99th=[ 2668] bw ( KiB/s): min=12288, max=251904, per=100.00%, avg=149805.07, stdev=45359.05, samples=60 iops : min= 12, max= 246, avg=146.27, stdev=44.28, samples=60 lat (usec) : 100=0.08%, 250=47.64%, 500=14.14%, 750=1.89%, 1000=1.34% lat (msec) : 2=2.40%, 4=1.82%, 10=2.75%, 20=3.60%, 50=3.85% lat (msec) : 100=0.16%, 250=0.82%, 500=4.83%, 750=8.11%, 1000=4.36% lat (msec) : 2000=2.14%, >=2000=0.08% cpu : usr=3.14%, sys=1.71%, ctx=16854, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=17708,4485,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=577MiB/s (605MB/s), 577MiB/s-577MiB/s (605MB/s-605MB/s), io=17.3GiB (18.6GB), run=30689-30689msec WRITE: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=4485MiB (4703MB), run=30689-30689msec Disk stats (read/write): vdg: ios=17708/4458, merge=0/0, ticks=45487/2951038, in_queue=3021778, util=99.58% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965445: Fri Jan 7 14:27:37 2022 read: IOPS=604, BW=605MiB/s (634MB/s)(17.7GiB/30001msec) clat (usec): min=117, max=25385, avg=214.06, stdev=977.77 lat (usec): min=117, max=25385, avg=214.21, stdev=977.77 clat percentiles (usec): | 1.00th=[ 127], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 137], | 30.00th=[ 139], 40.00th=[ 141], 50.00th=[ 145], 60.00th=[ 149], | 70.00th=[ 159], 80.00th=[ 176], 90.00th=[ 188], 95.00th=[ 200], | 99.00th=[ 281], 99.50th=[ 416], 99.90th=[17695], 99.95th=[19792], | 99.99th=[23987] bw ( KiB/s): min=391168, max=780288, per=99.71%, avg=617668.46, stdev=87167.14, samples=59 iops : min= 382, max= 762, avg=603.19, stdev=85.12, samples=59 write: IOPS=152, BW=153MiB/s (160MB/s)(4585MiB/30001msec) clat (usec): min=2502, max=60455, avg=5589.73, stdev=3009.31 lat (usec): min=2602, max=60544, avg=5686.81, stdev=3008.06 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3720], | 30.00th=[ 4555], 40.00th=[ 4948], 50.00th=[ 5211], 60.00th=[ 5473], | 70.00th=[ 5735], 80.00th=[ 6128], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[15926], 99.50th=[17171], 99.90th=[23725], 99.95th=[56361], | 99.99th=[60556] bw ( KiB/s): min=135168, max=221184, per=99.91%, avg=156352.58, stdev=12235.47, samples=59 iops : min= 132, max= 216, avg=152.68, stdev=11.95, samples=59 lat (usec) : 250=78.77%, 500=0.70%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.63%, 10=14.04%, 20=1.74%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.90%, sys=1.72%, ctx=22742, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18149,4585,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=605MiB/s (634MB/s), 605MiB/s-605MiB/s (634MB/s-634MB/s), io=17.7GiB (19.0GB), run=30001-30001msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4585MiB (4808MB), run=30001-30001msec Disk stats (read/write): vdg: ios=18115/4574, merge=0/0, ticks=1061/23016, in_queue=28799, util=95.54% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966065: Fri Jan 7 14:28:15 2022 read: IOPS=579, BW=580MiB/s (608MB/s)(17.4GiB/30741msec) slat (usec): min=10, max=840228, avg=1316.76, stdev=14386.20 clat (msec): min=19, max=1010, avg=112.53, stdev=114.75 lat (msec): min=19, max=1012, avg=113.85, stdev=115.68 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 31], 10.00th=[ 37], 20.00th=[ 56], | 30.00th=[ 70], 40.00th=[ 84], 50.00th=[ 95], 60.00th=[ 108], | 70.00th=[ 120], 80.00th=[ 136], 90.00th=[ 167], 95.00th=[ 201], | 99.00th=[ 827], 99.50th=[ 869], 99.90th=[ 953], 99.95th=[ 953], | 99.99th=[ 1011] bw ( KiB/s): min=53248, max=1452032, per=100.00%, avg=606096.70, stdev=225916.90, samples=60 iops : min= 52, max= 1418, avg=591.88, stdev=220.62, samples=60 write: IOPS=146, BW=147MiB/s (154MB/s)(4512MiB/30741msec) slat (usec): min=248, max=98711, avg=1434.28, stdev=5458.46 clat (msec): min=64, max=2130, avg=910.36, stdev=318.02 lat (msec): min=64, max=2130, avg=911.79, stdev=318.02 clat percentiles (msec): | 1.00th=[ 178], 5.00th=[ 409], 10.00th=[ 550], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 818], 50.00th=[ 885], 60.00th=[ 961], | 70.00th=[ 1020], 80.00th=[ 1099], 90.00th=[ 1318], 95.00th=[ 1536], | 99.00th=[ 1804], 99.50th=[ 1871], 99.90th=[ 2106], 99.95th=[ 2106], | 99.99th=[ 2123] bw ( KiB/s): min=14336, max=305152, per=99.50%, avg=149543.80, stdev=60161.05, samples=60 iops : min= 14, max= 298, avg=146.03, stdev=58.75, samples=60 lat (msec) : 20=0.05%, 50=13.57%, 100=29.78%, 250=34.65%, 500=1.75% lat (msec) : 750=4.59%, 1000=8.85%, 2000=6.74%, >=2000=0.04% cpu : usr=5.23%, sys=1.32%, ctx=2115, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=17824,4512,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=580MiB/s (608MB/s), 580MiB/s-580MiB/s (608MB/s-608MB/s), io=17.4GiB (18.7GB), run=30741-30741msec WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=4512MiB (4731MB), run=30741-30741msec Disk stats (read/write): vdg: ios=17824/4502, merge=0/0, ticks=112164/3607671, in_queue=3739065, util=99.33% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966917: Fri Jan 7 14:28:51 2022 read: IOPS=269, BW=539MiB/s (565MB/s)(15.8GiB/30001msec) clat (usec): min=271, max=23752, avg=482.23, stdev=991.59 lat (usec): min=271, max=23752, avg=482.41, stdev=991.60 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 285], 10.00th=[ 289], 20.00th=[ 297], | 30.00th=[ 302], 40.00th=[ 310], 50.00th=[ 322], 60.00th=[ 334], | 70.00th=[ 371], 80.00th=[ 603], 90.00th=[ 742], 95.00th=[ 775], | 99.00th=[ 914], 99.50th=[ 9765], 99.90th=[15270], 99.95th=[16057], | 99.99th=[23725] bw ( KiB/s): min=311296, max=843776, per=99.71%, avg=550297.81, stdev=126789.55, samples=59 iops : min= 152, max= 412, avg=268.68, stdev=61.89, samples=59 write: IOPS=68, BW=137MiB/s (143MB/s)(4098MiB/30001msec) clat (usec): min=6101, max=53511, avg=12450.08, stdev=3618.58 lat (usec): min=6345, max=53713, avg=12726.99, stdev=3637.80 clat percentiles (usec): | 1.00th=[ 6783], 5.00th=[ 8455], 10.00th=[ 8979], 20.00th=[ 9634], | 30.00th=[10290], 40.00th=[11076], 50.00th=[11731], 60.00th=[12780], | 70.00th=[14222], 80.00th=[15270], 90.00th=[16319], 95.00th=[17171], | 99.00th=[22414], 99.50th=[28443], 99.90th=[47449], 99.95th=[47973], | 99.99th=[53740] bw ( KiB/s): min=114688, max=167936, per=99.77%, avg=139551.00, stdev=11173.48, samples=59 iops : min= 56, max= 82, avg=68.14, stdev= 5.46, samples=59 lat (usec) : 500=62.57%, 750=10.68%, 1000=5.95% lat (msec) : 2=0.05%, 4=0.02%, 10=5.17%, 20=15.23%, 50=0.33% lat (msec) : 100=0.01% cpu : usr=2.10%, sys=2.01%, ctx=10146, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8085,2049,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=15.8GiB (16.0GB), run=30001-30001msec WRITE: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4098MiB (4297MB), run=30001-30001msec Disk stats (read/write): vdg: ios=16004/4068, merge=0/0, ticks=657/48239, in_queue=56422, util=95.66% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967511: Fri Jan 7 14:29:28 2022 read: IOPS=281, BW=562MiB/s (589MB/s)(16.5GiB/30014msec) slat (usec): min=53, max=188, avg=70.86, stdev=12.83 clat (usec): min=164, max=24196, avg=349.95, stdev=1028.68 lat (usec): min=245, max=24286, avg=421.05, stdev=1029.29 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 198], 10.00th=[ 200], 20.00th=[ 206], | 30.00th=[ 210], 40.00th=[ 215], 50.00th=[ 221], 60.00th=[ 229], | 70.00th=[ 253], 80.00th=[ 302], 90.00th=[ 482], 95.00th=[ 586], | 99.00th=[ 758], 99.50th=[10028], 99.90th=[15795], 99.95th=[17433], | 99.99th=[24249] bw ( KiB/s): min=286720, max=1007616, per=100.00%, avg=575903.47, stdev=128659.57, samples=60 iops : min= 140, max= 492, avg=281.17, stdev=62.77, samples=60 write: IOPS=71, BW=143MiB/s (150MB/s)(4288MiB/30014msec) slat (usec): min=134, max=766, avg=285.49, stdev=100.81 clat (usec): min=4644, max=48262, avg=12046.35, stdev=3368.80 lat (usec): min=4878, max=48535, avg=12332.12, stdev=3382.60 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 7177], 10.00th=[ 8094], 20.00th=[ 9241], | 30.00th=[10159], 40.00th=[10945], 50.00th=[11600], 60.00th=[12256], | 70.00th=[13960], 80.00th=[15270], 90.00th=[16319], 95.00th=[16909], | 99.00th=[19530], 99.50th=[22414], 99.90th=[27919], 99.95th=[45876], | 99.99th=[48497] bw ( KiB/s): min=122880, max=176128, per=100.00%, avg=146311.05, stdev=10944.92, samples=60 iops : min= 60, max= 86, avg=71.43, stdev= 5.34, samples=60 lat (usec) : 250=55.17%, 500=17.23%, 750=6.50%, 1000=0.29% lat (msec) : 2=0.03%, 4=0.01%, 10=5.79%, 20=14.78%, 50=0.20% cpu : usr=2.04%, sys=2.54%, ctx=10589, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8435,2144,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=562MiB/s (589MB/s), 562MiB/s-562MiB/s (589MB/s-589MB/s), io=16.5GiB (17.7GB), run=30014-30014msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4288MiB (4496MB), run=30014-30014msec Disk stats (read/write): vdg: ios=16814/4260, merge=0/0, ticks=719/49051, in_queue=56620, util=96.05% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968312: Fri Jan 7 14:30:05 2022 read: IOPS=281, BW=562MiB/s (589MB/s)(16.5GiB/30010msec) clat (usec): min=238, max=24259, avg=396.02, stdev=988.17 lat (usec): min=238, max=24259, avg=396.20, stdev=988.18 clat percentiles (usec): | 1.00th=[ 245], 5.00th=[ 251], 10.00th=[ 255], 20.00th=[ 260], | 30.00th=[ 265], 40.00th=[ 269], 50.00th=[ 273], 60.00th=[ 285], | 70.00th=[ 306], 80.00th=[ 359], 90.00th=[ 486], 95.00th=[ 627], | 99.00th=[ 807], 99.50th=[ 9372], 99.90th=[15008], 99.95th=[17433], | 99.99th=[24249] bw ( KiB/s): min=360448, max=1007616, per=100.00%, avg=575873.38, stdev=127871.33, samples=60 iops : min= 176, max= 492, avg=281.17, stdev=62.41, samples=60 write: IOPS=71, BW=143MiB/s (150MB/s)(4288MiB/30010msec) clat (usec): min=4675, max=52543, avg=12200.94, stdev=3581.75 lat (usec): min=4874, max=52771, avg=12428.08, stdev=3588.88 clat percentiles (usec): | 1.00th=[ 5669], 5.00th=[ 7308], 10.00th=[ 8291], 20.00th=[ 9372], | 30.00th=[10290], 40.00th=[11076], 50.00th=[11600], 60.00th=[12387], | 70.00th=[14091], 80.00th=[15401], 90.00th=[16319], 95.00th=[16909], | 99.00th=[20317], 99.50th=[22676], 99.90th=[49546], 99.95th=[50070], | 99.99th=[52691] bw ( KiB/s): min=122880, max=176128, per=99.99%, avg=146305.57, stdev=10012.99, samples=60 iops : min= 60, max= 86, avg=71.43, stdev= 4.89, samples=60 lat (usec) : 250=2.99%, 500=69.32%, 750=6.30%, 1000=0.59% lat (msec) : 2=0.03%, 4=0.03%, 10=5.50%, 20=15.00%, 50=0.23% lat (msec) : 100=0.02% cpu : usr=1.84%, sys=1.83%, ctx=10589, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8435,2144,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=562MiB/s (589MB/s), 562MiB/s-562MiB/s (589MB/s-589MB/s), io=16.5GiB (17.7GB), run=30010-30010msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4288MiB (4496MB), run=30010-30010msec Disk stats (read/write): vdg: ios=16800/4254, merge=0/0, ticks=692/42861, in_queue=50157, util=96.17% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969570: Fri Jan 7 14:30:42 2022 read: IOPS=351, BW=702MiB/s (737MB/s)(20.7GiB/30118msec) slat (usec): min=20, max=2235, avg=36.98, stdev=25.10 clat (usec): min=131, max=1016.9k, avg=1279.92, stdev=24245.32 lat (usec): min=207, max=1016.0k, avg=1317.12, stdev=24245.25 clat percentiles (usec): | 1.00th=[ 184], 5.00th=[ 188], 10.00th=[ 190], | 20.00th=[ 194], 30.00th=[ 198], 40.00th=[ 202], | 50.00th=[ 206], 60.00th=[ 215], 70.00th=[ 237], | 80.00th=[ 627], 90.00th=[ 652], 95.00th=[ 676], | 99.00th=[ 898], 99.50th=[ 3720], 99.90th=[ 438305], | 99.95th=[ 658506], 99.99th=[1010828] bw ( KiB/s): min=430080, max=1191936, per=100.00%, avg=722031.70, stdev=140871.14, samples=60 iops : min= 210, max= 582, avg=352.53, stdev=68.79, samples=60 write: IOPS=89, BW=179MiB/s (187MB/s)(5378MiB/30118msec) slat (usec): min=119, max=1005, avg=357.68, stdev=185.09 clat (msec): min=10, max=2499, avg=106.41, stdev=125.48 lat (msec): min=11, max=2500, avg=106.76, stdev=125.48 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 31], | 30.00th=[ 40], 40.00th=[ 49], 50.00th=[ 67], 60.00th=[ 90], | 70.00th=[ 122], 80.00th=[ 161], 90.00th=[ 236], 95.00th=[ 313], | 99.00th=[ 514], 99.50th=[ 676], 99.90th=[ 1368], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min=167936, max=192512, per=100.00%, avg=182930.38, stdev=6113.74, samples=60 iops : min= 82, max= 94, avg=89.27, stdev= 3.00, samples=60 lat (usec) : 250=57.54%, 500=4.94%, 750=15.73%, 1000=0.91% lat (msec) : 2=0.15%, 4=0.07%, 10=0.14%, 20=1.43%, 50=6.93% lat (msec) : 100=4.71%, 250=5.57%, 500=1.60%, 750=0.18%, 1000=0.06% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=3.26%, sys=1.91%, ctx=13240, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10578,2689,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=702MiB/s (737MB/s), 702MiB/s-702MiB/s (737MB/s-737MB/s), io=20.7GiB (22.2GB), run=30118-30118msec WRITE: bw=179MiB/s (187MB/s), 179MiB/s-179MiB/s (187MB/s-187MB/s), io=5378MiB (5639MB), run=30118-30118msec Disk stats (read/write): vdg: ios=21156/5360, merge=0/0, ticks=10386/565364, in_queue=586475, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970156: Fri Jan 7 14:31:19 2022 read: IOPS=282, BW=566MiB/s (593MB/s)(16.6GiB/30009msec) clat (usec): min=235, max=24495, avg=389.91, stdev=1022.16 lat (usec): min=235, max=24495, avg=390.09, stdev=1022.16 clat percentiles (usec): | 1.00th=[ 243], 5.00th=[ 249], 10.00th=[ 253], 20.00th=[ 258], | 30.00th=[ 262], 40.00th=[ 269], 50.00th=[ 273], 60.00th=[ 281], | 70.00th=[ 297], 80.00th=[ 363], 90.00th=[ 437], 95.00th=[ 529], | 99.00th=[ 791], 99.50th=[10159], 99.90th=[15795], 99.95th=[17695], | 99.99th=[24511] bw ( KiB/s): min=307200, max=929792, per=100.00%, avg=579621.88, stdev=108587.33, samples=60 iops : min= 150, max= 454, avg=283.00, stdev=53.02, samples=60 write: IOPS=71, BW=144MiB/s (151MB/s)(4310MiB/30009msec) clat (usec): min=4669, max=48194, avg=12158.45, stdev=3369.18 lat (usec): min=4866, max=48408, avg=12378.28, stdev=3372.71 clat percentiles (usec): | 1.00th=[ 5669], 5.00th=[ 7308], 10.00th=[ 8291], 20.00th=[ 9503], | 30.00th=[10290], 40.00th=[11076], 50.00th=[11600], 60.00th=[12256], | 70.00th=[14091], 80.00th=[15533], 90.00th=[16450], 95.00th=[16909], | 99.00th=[19268], 99.50th=[22414], 99.90th=[34341], 99.95th=[45876], | 99.99th=[47973] bw ( KiB/s): min=118784, max=168272, per=99.99%, avg=147057.05, stdev=10455.19, samples=60 iops : min= 58, max= 82, avg=71.80, stdev= 5.10, samples=60 lat (usec) : 250=4.87%, 500=70.09%, 750=3.77%, 1000=0.49% lat (msec) : 2=0.04%, 4=0.01%, 10=5.57%, 20=14.99%, 50=0.18% cpu : usr=1.79%, sys=1.77%, ctx=10652, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8490,2155,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=566MiB/s (593MB/s), 566MiB/s-566MiB/s (593MB/s-593MB/s), io=16.6GiB (17.8GB), run=30009-30009msec WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=4310MiB (4519MB), run=30009-30009msec Disk stats (read/write): vdg: ios=16824/4276, merge=0/0, ticks=725/43234, in_queue=50361, util=96.06% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971115: Fri Jan 7 14:31:56 2022 read: IOPS=357, BW=714MiB/s (749MB/s)(21.3GiB/30558msec) slat (usec): min=19, max=480, avg=34.14, stdev=16.53 clat (usec): min=91, max=2528.9k, avg=3926.11, stdev=58670.94 lat (usec): min=207, max=2528.0k, avg=3960.45, stdev=58671.02 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 190], 10.00th=[ 192], | 20.00th=[ 196], 30.00th=[ 200], 40.00th=[ 204], | 50.00th=[ 210], 60.00th=[ 231], 70.00th=[ 515], | 80.00th=[ 652], 90.00th=[ 1045], 95.00th=[ 8586], | 99.00th=[ 23462], 99.50th=[ 68682], 99.90th=[1149240], | 99.95th=[1451230], 99.99th=[2264925] bw ( KiB/s): min=475503, max=1130362, per=100.00%, avg=750693.65, stdev=140019.52, samples=60 iops : min= 232, max= 551, avg=366.27, stdev=68.36, samples=60 write: IOPS=90, BW=181MiB/s (190MB/s)(5534MiB/30558msec) slat (usec): min=122, max=1037, avg=377.16, stdev=201.81 clat (msec): min=60, max=3131, avg=535.87, stdev=266.46 lat (msec): min=61, max=3132, avg=536.25, stdev=266.47 clat percentiles (msec): | 1.00th=[ 207], 5.00th=[ 279], 10.00th=[ 338], 20.00th=[ 376], | 30.00th=[ 397], 40.00th=[ 418], 50.00th=[ 451], 60.00th=[ 493], | 70.00th=[ 558], 80.00th=[ 659], 90.00th=[ 852], 95.00th=[ 1036], | 99.00th=[ 1586], 99.50th=[ 1821], 99.90th=[ 2970], 99.95th=[ 3037], | 99.99th=[ 3138] bw ( KiB/s): min= 4096, max=249756, per=99.21%, avg=183972.57, stdev=33722.32, samples=61 iops : min= 2, max= 121, avg=89.59, stdev=16.43, samples=61 lat (usec) : 100=0.01%, 250=50.58%, 500=5.17%, 750=14.71%, 1000=1.25% lat (msec) : 2=1.34%, 4=1.32%, 10=1.96%, 20=2.38%, 50=0.66% lat (msec) : 100=0.17%, 250=0.48%, 500=11.96%, 750=5.07%, 1000=1.62% lat (msec) : 2000=1.25%, >=2000=0.08% cpu : usr=3.43%, sys=1.82%, ctx=12180, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=10915,2767,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=714MiB/s (749MB/s), 714MiB/s-714MiB/s (749MB/s-749MB/s), io=21.3GiB (22.9GB), run=30558-30558msec WRITE: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=5534MiB (5803MB), run=30558-30558msec Disk stats (read/write): vdg: ios=21829/5524, merge=0/0, ticks=45641/2937503, in_queue=3000867, util=99.55% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972160: Fri Jan 7 14:32:33 2022 read: IOPS=282, BW=565MiB/s (592MB/s)(16.5GiB/30016msec) clat (usec): min=234, max=24530, avg=379.24, stdev=1025.46 lat (usec): min=234, max=24530, avg=379.43, stdev=1025.46 clat percentiles (usec): | 1.00th=[ 243], 5.00th=[ 249], 10.00th=[ 251], 20.00th=[ 258], | 30.00th=[ 262], 40.00th=[ 265], 50.00th=[ 273], 60.00th=[ 281], | 70.00th=[ 297], 80.00th=[ 343], 90.00th=[ 383], 95.00th=[ 465], | 99.00th=[ 750], 99.50th=[10159], 99.90th=[16057], 99.95th=[17957], | 99.99th=[24511] bw ( KiB/s): min=323584, max=1048576, per=100.00%, avg=578447.33, stdev=131393.91, samples=60 iops : min= 158, max= 512, avg=282.42, stdev=64.14, samples=60 write: IOPS=71, BW=143MiB/s (150MB/s)(4304MiB/30016msec) clat (usec): min=4670, max=56783, avg=12235.61, stdev=3492.70 lat (usec): min=4879, max=57338, avg=12444.41, stdev=3498.87 clat percentiles (usec): | 1.00th=[ 5473], 5.00th=[ 7373], 10.00th=[ 8356], 20.00th=[ 9503], | 30.00th=[10421], 40.00th=[11076], 50.00th=[11731], 60.00th=[12387], | 70.00th=[14091], 80.00th=[15533], 90.00th=[16450], 95.00th=[16909], | 99.00th=[19006], 99.50th=[22676], 99.90th=[44827], 99.95th=[47449], | 99.99th=[56886] bw ( KiB/s): min=126976, max=172032, per=100.00%, avg=146847.77, stdev=10271.56, samples=60 iops : min= 62, max= 84, avg=71.68, stdev= 5.03, samples=60 lat (usec) : 250=5.42%, 500=71.29%, 750=2.23%, 1000=0.28% lat (msec) : 2=0.01%, 4=0.02%, 10=5.35%, 20=15.24%, 50=0.15% lat (msec) : 100=0.01% cpu : usr=1.64%, sys=1.84%, ctx=10635, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8473,2152,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=565MiB/s (592MB/s), 565MiB/s-565MiB/s (592MB/s-592MB/s), io=16.5GiB (17.8GB), run=30016-30016msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4304MiB (4513MB), run=30016-30016msec Disk stats (read/write): vdg: ios=16870/4288, merge=0/0, ticks=726/43306, in_queue=50344, util=95.96% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972687: Fri Jan 7 14:33:11 2022 read: IOPS=355, BW=711MiB/s (746MB/s)(21.3GiB/30657msec) slat (usec): min=22, max=519784, avg=2191.03, stdev=12181.68 clat (msec): min=18, max=2716, avg=90.94, stdev=116.59 lat (msec): min=18, max=2717, avg=93.14, stdev=117.25 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 35], 20.00th=[ 46], | 30.00th=[ 56], 40.00th=[ 66], 50.00th=[ 75], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 107], 90.00th=[ 127], 95.00th=[ 163], | 99.00th=[ 558], 99.50th=[ 676], 99.90th=[ 1770], 99.95th=[ 2601], | 99.99th=[ 2702] bw ( KiB/s): min= 4096, max=1216512, per=100.00%, avg=729607.93, stdev=265933.72, samples=61 iops : min= 2, max= 594, avg=356.23, stdev=129.84, samples=61 write: IOPS=90, BW=180MiB/s (189MB/s)(5530MiB/30657msec) slat (usec): min=560, max=79574, avg=2194.92, stdev=5176.71 clat (msec): min=72, max=3238, avg=738.39, stdev=318.17 lat (msec): min=73, max=3239, avg=740.59, stdev=318.09 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 405], 10.00th=[ 477], 20.00th=[ 542], | 30.00th=[ 584], 40.00th=[ 617], 50.00th=[ 651], 60.00th=[ 709], | 70.00th=[ 785], 80.00th=[ 894], 90.00th=[ 1099], 95.00th=[ 1250], | 99.00th=[ 2056], 99.50th=[ 2903], 99.90th=[ 2903], 99.95th=[ 2937], | 99.99th=[ 3239] bw ( KiB/s): min=16384, max=290816, per=99.91%, avg=184540.25, stdev=60779.96, samples=60 iops : min= 8, max= 142, avg=90.10, stdev=29.67, samples=60 lat (msec) : 20=0.01%, 50=18.97%, 100=41.52%, 250=17.10%, 500=3.40% lat (msec) : 750=11.94%, 1000=4.08%, 2000=2.71%, >=2000=0.29% cpu : usr=6.12%, sys=1.65%, ctx=2718, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10902,2765,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=711MiB/s (746MB/s), 711MiB/s-711MiB/s (746MB/s-746MB/s), io=21.3GiB (22.9GB), run=30657-30657msec WRITE: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=5530MiB (5799MB), run=30657-30657msec Disk stats (read/write): vdg: ios=21803/5492, merge=0/0, ticks=140493/3561994, in_queue=3743101, util=99.32% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973645: Fri Jan 7 14:33:47 2022 read: IOPS=281, BW=564MiB/s (591MB/s)(16.5GiB/30004msec) clat (usec): min=221, max=32891, avg=368.23, stdev=1068.12 lat (usec): min=221, max=32891, avg=368.41, stdev=1068.12 clat percentiles (usec): | 1.00th=[ 227], 5.00th=[ 235], 10.00th=[ 237], 20.00th=[ 243], | 30.00th=[ 247], 40.00th=[ 251], 50.00th=[ 260], 60.00th=[ 269], | 70.00th=[ 289], 80.00th=[ 334], 90.00th=[ 375], 95.00th=[ 449], | 99.00th=[ 709], 99.50th=[10159], 99.90th=[16057], 99.95th=[17433], | 99.99th=[32900] bw ( KiB/s): min=335872, max=1007616, per=100.00%, avg=577623.42, stdev=121668.94, samples=60 iops : min= 164, max= 492, avg=282.03, stdev=59.41, samples=60 write: IOPS=71, BW=143MiB/s (150MB/s)(4296MiB/30004msec) clat (usec): min=4673, max=62452, avg=12301.89, stdev=3607.03 lat (usec): min=4867, max=62638, avg=12506.91, stdev=3608.34 clat percentiles (usec): | 1.00th=[ 5604], 5.00th=[ 7373], 10.00th=[ 8455], 20.00th=[ 9503], | 30.00th=[10421], 40.00th=[11076], 50.00th=[11731], 60.00th=[12518], | 70.00th=[14222], 80.00th=[15533], 90.00th=[16450], 95.00th=[17171], | 99.00th=[19268], 99.50th=[23725], 99.90th=[45351], 99.95th=[46924], | 99.99th=[62653] bw ( KiB/s): min=122880, max=172032, per=99.97%, avg=146572.90, stdev=10767.28, samples=60 iops : min= 60, max= 84, avg=71.57, stdev= 5.26, samples=60 lat (usec) : 250=29.00%, 500=48.03%, 750=2.05%, 1000=0.14% lat (msec) : 2=0.03%, 4=0.02%, 10=5.25%, 20=15.30%, 50=0.18% lat (msec) : 100=0.01% cpu : usr=1.61%, sys=1.43%, ctx=10616, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8461,2148,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=564MiB/s (591MB/s), 564MiB/s-564MiB/s (591MB/s-591MB/s), io=16.5GiB (17.7GB), run=30004-30004msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4296MiB (4505MB), run=30004-30004msec Disk stats (read/write): vdg: ios=16816/4268, merge=0/0, ticks=744/42227, in_queue=49155, util=95.91% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974266: Fri Jan 7 14:34:25 2022 read: IOPS=352, BW=705MiB/s (739MB/s)(21.1GiB/30676msec) slat (usec): min=23, max=494588, avg=2080.19, stdev=10991.24 clat (msec): min=85, max=3030, avg=321.14, stdev=153.33 lat (msec): min=85, max=3030, avg=323.22, stdev=153.65 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 150], 10.00th=[ 194], 20.00th=[ 241], | 30.00th=[ 266], 40.00th=[ 284], 50.00th=[ 300], 60.00th=[ 313], | 70.00th=[ 334], 80.00th=[ 359], 90.00th=[ 456], 95.00th=[ 642], | 99.00th=[ 852], 99.50th=[ 953], 99.90th=[ 1787], 99.95th=[ 2500], | 99.99th=[ 2970] bw ( KiB/s): min= 4104, max=1186116, per=99.59%, avg=718705.41, stdev=245041.05, samples=61 iops : min= 2, max= 579, avg=350.90, stdev=119.63, samples=61 write: IOPS=89, BW=179MiB/s (188MB/s)(5506MiB/30676msec) slat (usec): min=500, max=532482, avg=2714.36, stdev=11730.28 clat (msec): min=103, max=3704, avg=951.24, stdev=335.50 lat (msec): min=163, max=3705, avg=953.95, stdev=334.97 clat percentiles (msec): | 1.00th=[ 447], 5.00th=[ 600], 10.00th=[ 651], 20.00th=[ 735], | 30.00th=[ 785], 40.00th=[ 827], 50.00th=[ 877], 60.00th=[ 927], | 70.00th=[ 1011], 80.00th=[ 1133], 90.00th=[ 1318], 95.00th=[ 1502], | 99.00th=[ 2198], 99.50th=[ 2735], 99.90th=[ 3507], 99.95th=[ 3708], | 99.99th=[ 3708] bw ( KiB/s): min=20480, max=368640, per=100.00%, avg=184828.76, stdev=67481.60, samples=59 iops : min= 10, max= 180, avg=90.24, stdev=32.94, samples=59 lat (msec) : 100=0.68%, 250=18.27%, 500=55.14%, 750=8.77%, 1000=10.60% lat (msec) : 2000=6.08%, >=2000=0.46% cpu : usr=6.13%, sys=1.69%, ctx=2727, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10810,2753,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=705MiB/s (739MB/s), 705MiB/s-705MiB/s (739MB/s-739MB/s), io=21.1GiB (22.7GB), run=30676-30676msec WRITE: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=5506MiB (5773MB), run=30676-30676msec Disk stats (read/write): vdg: ios=21669/5497, merge=0/0, ticks=144634/3579987, in_queue=3751542, util=99.00% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975110: Fri Jan 7 14:35:01 2022 read: IOPS=164, BW=659MiB/s (691MB/s)(19.3GiB/30006msec) clat (usec): min=405, max=29703, avg=826.08, stdev=1491.83 lat (usec): min=405, max=29703, avg=826.29, stdev=1491.85 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 424], 10.00th=[ 429], 20.00th=[ 437], | 30.00th=[ 445], 40.00th=[ 453], 50.00th=[ 465], 60.00th=[ 490], | 70.00th=[ 578], 80.00th=[ 1303], 90.00th=[ 1369], 95.00th=[ 1434], | 99.00th=[ 9634], 99.50th=[13829], 99.90th=[18482], 99.95th=[18744], | 99.99th=[29754] bw ( KiB/s): min=286720, max=1130496, per=100.00%, avg=675066.48, stdev=165094.03, samples=60 iops : min= 70, max= 276, avg=164.80, stdev=40.30, samples=60 write: IOPS=42, BW=170MiB/s (178MB/s)(5096MiB/30006msec) clat (usec): min=11286, max=63060, avg=19737.26, stdev=4389.79 lat (usec): min=11983, max=63473, avg=20333.97, stdev=4435.62 clat percentiles (usec): | 1.00th=[13173], 5.00th=[14353], 10.00th=[15139], 20.00th=[16188], | 30.00th=[17171], 40.00th=[17957], 50.00th=[18744], 60.00th=[20317], | 70.00th=[22152], 80.00th=[23462], 90.00th=[24773], 95.00th=[25822], | 99.00th=[32637], 99.50th=[36439], 99.90th=[58983], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min=155648, max=196608, per=99.95%, avg=173818.68, stdev=8938.18, samples=60 iops : min= 38, max= 48, avg=42.43, stdev= 2.18, samples=60 lat (usec) : 500=49.77%, 750=10.78%, 1000=0.95% lat (msec) : 2=16.90%, 4=0.08%, 10=0.29%, 20=12.74%, 50=8.44% lat (msec) : 100=0.05% cpu : usr=2.74%, sys=2.06%, ctx=6230, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4944,1274,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=659MiB/s (691MB/s), 659MiB/s-659MiB/s (691MB/s-691MB/s), io=19.3GiB (20.7GB), run=30006-30006msec WRITE: bw=170MiB/s (178MB/s), 170MiB/s-170MiB/s (178MB/s-178MB/s), io=5096MiB (5344MB), run=30006-30006msec Disk stats (read/write): vdg: ios=19664/5060, merge=0/0, ticks=5103/80824, in_queue=95942, util=96.50% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976335: Fri Jan 7 14:35:38 2022 read: IOPS=165, BW=664MiB/s (696MB/s)(19.4GiB/30009msec) slat (usec): min=72, max=248, avg=94.27, stdev=17.58 clat (usec): min=258, max=21996, avg=703.28, stdev=1450.10 lat (usec): min=402, max=22096, avg=797.82, stdev=1450.73 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 334], 10.00th=[ 338], 20.00th=[ 347], | 30.00th=[ 355], 40.00th=[ 363], 50.00th=[ 367], 60.00th=[ 379], | 70.00th=[ 424], 80.00th=[ 1012], 90.00th=[ 1221], 95.00th=[ 1270], | 99.00th=[10159], 99.50th=[13042], 99.90th=[18220], 99.95th=[18744], | 99.99th=[21890] bw ( KiB/s): min=409600, max=1130496, per=100.00%, avg=679766.05, stdev=172944.97, samples=60 iops : min= 100, max= 276, avg=165.93, stdev=42.23, samples=60 write: IOPS=43, BW=172MiB/s (180MB/s)(5164MiB/30009msec) slat (usec): min=259, max=1691, avg=634.90, stdev=302.88 clat (msec): min=10, max=189, avg=19.52, stdev= 6.46 lat (msec): min=11, max=189, avg=20.16, stdev= 6.49 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 33], 99.50th=[ 37], 99.90th=[ 73], 99.95th=[ 190], | 99.99th=[ 190] bw ( KiB/s): min=114688, max=204800, per=99.97%, avg=176156.68, stdev=13009.91, samples=60 iops : min= 28, max= 50, avg=43.00, stdev= 3.18, samples=60 lat (usec) : 500=58.86%, 750=2.46%, 1000=2.07% lat (msec) : 2=14.98%, 10=0.24%, 20=13.62%, 50=7.72%, 100=0.03% lat (msec) : 250=0.02% cpu : usr=2.64%, sys=1.96%, ctx=6283, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4978,1291,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=664MiB/s (696MB/s), 664MiB/s-664MiB/s (696MB/s-696MB/s), io=19.4GiB (20.9GB), run=30009-30009msec WRITE: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s), io=5164MiB (5415MB), run=30009-30009msec Disk stats (read/write): vdg: ios=19836/5121, merge=0/0, ticks=4716/81025, in_queue=95591, util=96.28% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977098: Fri Jan 7 14:36:15 2022 read: IOPS=167, BW=670MiB/s (702MB/s)(19.6GiB/30001msec) clat (usec): min=372, max=22401, avg=756.02, stdev=1531.03 lat (usec): min=372, max=22401, avg=756.22, stdev=1531.04 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 400], | 30.00th=[ 404], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 482], 80.00th=[ 996], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[10552], 99.50th=[14222], 99.90th=[18482], 99.95th=[18744], | 99.99th=[22414] bw ( KiB/s): min=287294, max=1151485, per=100.00%, avg=686322.93, stdev=157267.54, samples=60 iops : min= 70, max= 281, avg=167.40, stdev=38.37, samples=60 write: IOPS=43, BW=174MiB/s (182MB/s)(5212MiB/30001msec) clat (usec): min=10760, max=57360, avg=19543.89, stdev=4349.84 lat (usec): min=11156, max=57757, avg=20099.04, stdev=4407.50 clat percentiles (usec): | 1.00th=[12387], 5.00th=[14091], 10.00th=[15008], 20.00th=[16057], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18744], 60.00th=[19792], | 70.00th=[21627], 80.00th=[23462], 90.00th=[24773], 95.00th=[25822], | 99.00th=[30016], 99.50th=[36439], 99.90th=[56361], 99.95th=[57410], | 99.99th=[57410] bw ( KiB/s): min=155648, max=205210, per=100.00%, avg=177932.32, stdev=11424.17, samples=60 iops : min= 38, max= 50, avg=43.38, stdev= 2.81, samples=60 lat (usec) : 500=56.62%, 750=4.60%, 1000=2.39% lat (msec) : 2=14.77%, 10=0.17%, 20=13.53%, 50=7.89%, 100=0.03% cpu : usr=2.68%, sys=1.20%, ctx=6338, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5022,1303,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=670MiB/s (702MB/s), 670MiB/s-670MiB/s (702MB/s-702MB/s), io=19.6GiB (21.1GB), run=30001-30001msec WRITE: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=5212MiB (5465MB), run=30001-30001msec Disk stats (read/write): vdg: ios=20088/5209, merge=0/0, ticks=4511/82359, in_queue=96645, util=96.48% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977782: Fri Jan 7 14:36:52 2022 read: IOPS=184, BW=737MiB/s (773MB/s)(21.7GiB/30174msec) slat (usec): min=38, max=256, avg=60.15, stdev=16.34 clat (usec): min=256, max=2114.9k, avg=5848.70, stdev=71604.67 lat (usec): min=377, max=2114.9k, avg=5909.12, stdev=71604.60 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 338], 10.00th=[ 343], | 20.00th=[ 351], 30.00th=[ 359], 40.00th=[ 367], | 50.00th=[ 375], 60.00th=[ 392], 70.00th=[ 457], | 80.00th=[ 1221], 90.00th=[ 1287], 95.00th=[ 1336], | 99.00th=[ 68682], 99.50th=[ 291505], 99.90th=[1350566], | 99.95th=[1837106], 99.99th=[2122318] bw ( KiB/s): min=376832, max=1343488, per=100.00%, avg=758852.27, stdev=196544.16, samples=60 iops : min= 92, max= 328, avg=185.27, stdev=47.98, samples=60 write: IOPS=48, BW=194MiB/s (203MB/s)(5852MiB/30174msec) slat (usec): min=233, max=1916, avg=780.62, stdev=389.92 clat (msec): min=19, max=2545, avg=182.88, stdev=320.26 lat (msec): min=19, max=2546, avg=183.66, stdev=320.27 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 35], 10.00th=[ 38], 20.00th=[ 54], | 30.00th=[ 60], 40.00th=[ 75], 50.00th=[ 95], 60.00th=[ 123], | 70.00th=[ 159], 80.00th=[ 224], 90.00th=[ 368], 95.00th=[ 523], | 99.00th=[ 2500], 99.50th=[ 2534], 99.90th=[ 2534], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=172032, max=212992, per=99.89%, avg=198382.93, stdev=10350.25, samples=60 iops : min= 42, max= 52, avg=48.43, stdev= 2.53, samples=60 lat (usec) : 500=56.69%, 750=1.88%, 1000=0.54% lat (msec) : 2=18.60%, 4=0.14%, 10=0.16%, 20=0.16%, 50=3.80% lat (msec) : 100=7.35%, 250=6.67%, 500=2.61%, 750=0.70%, 1000=0.20% lat (msec) : 2000=0.20%, >=2000=0.31% cpu : usr=3.72%, sys=1.54%, ctx=6987, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5558,1463,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=737MiB/s (773MB/s), 737MiB/s-737MiB/s (773MB/s-773MB/s), io=21.7GiB (23.3GB), run=30174-30174msec WRITE: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=5852MiB (6136MB), run=30174-30174msec Disk stats (read/write): vdg: ios=22320/5855, merge=0/0, ticks=37357/1064719, in_queue=1117537, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978559: Fri Jan 7 14:37:29 2022 read: IOPS=165, BW=663MiB/s (695MB/s)(19.4GiB/30005msec) clat (usec): min=377, max=22137, avg=764.81, stdev=1508.20 lat (usec): min=377, max=22137, avg=765.00, stdev=1508.21 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 445], | 70.00th=[ 490], 80.00th=[ 988], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[10552], 99.50th=[14091], 99.90th=[18482], 99.95th=[19006], | 99.99th=[22152] bw ( KiB/s): min=344064, max=1105920, per=100.00%, avg=678707.20, stdev=179299.08, samples=60 iops : min= 84, max= 270, avg=165.70, stdev=43.77, samples=60 write: IOPS=42, BW=172MiB/s (180MB/s)(5152MiB/30005msec) clat (usec): min=11167, max=75338, avg=19770.23, stdev=5402.73 lat (usec): min=11666, max=75729, avg=20331.71, stdev=5444.28 clat percentiles (usec): | 1.00th=[12256], 5.00th=[14091], 10.00th=[14877], 20.00th=[16057], | 30.00th=[16909], 40.00th=[17695], 50.00th=[18744], 60.00th=[20055], | 70.00th=[21890], 80.00th=[23462], 90.00th=[24773], 95.00th=[26084], | 99.00th=[34866], 99.50th=[58459], 99.90th=[74974], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=147456, max=188416, per=99.94%, avg=175718.40, stdev=10815.99, samples=60 iops : min= 36, max= 46, avg=42.90, stdev= 2.64, samples=60 lat (usec) : 500=56.18%, 750=5.08%, 1000=2.44% lat (msec) : 2=14.65%, 4=0.02%, 10=0.18%, 20=13.28%, 50=8.04% lat (msec) : 100=0.14% cpu : usr=2.49%, sys=1.78%, ctx=6272, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4971,1288,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=663MiB/s (695MB/s), 663MiB/s-663MiB/s (695MB/s-695MB/s), io=19.4GiB (20.8GB), run=30005-30005msec WRITE: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s), io=5152MiB (5402MB), run=30005-30005msec Disk stats (read/write): vdg: ios=19868/5137, merge=0/0, ticks=4432/82086, in_queue=96352, util=96.48% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979527: Fri Jan 7 14:38:07 2022 read: IOPS=181, BW=725MiB/s (760MB/s)(21.7GiB/30630msec) slat (usec): min=45, max=470060, avg=4049.84, stdev=14850.78 clat (msec): min=17, max=1890, avg=82.87, stdev=99.87 lat (msec): min=17, max=1893, avg=86.92, stdev=101.25 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 39], | 30.00th=[ 46], 40.00th=[ 56], 50.00th=[ 66], 60.00th=[ 75], | 70.00th=[ 88], 80.00th=[ 103], 90.00th=[ 130], 95.00th=[ 155], | 99.00th=[ 567], 99.50th=[ 693], 99.90th=[ 1519], 99.95th=[ 1720], | 99.99th=[ 1888] bw ( KiB/s): min= 8192, max=1572864, per=100.00%, avg=743634.66, stdev=337480.51, samples=61 iops : min= 2, max= 384, avg=181.54, stdev=82.38, samples=61 write: IOPS=47, BW=191MiB/s (200MB/s)(5848MiB/30630msec) slat (usec): min=1122, max=445397, avg=5122.39, stdev=14982.22 clat (msec): min=103, max=3466, avg=711.03, stdev=342.76 lat (msec): min=132, max=3467, avg=716.15, stdev=342.65 clat percentiles (msec): | 1.00th=[ 275], 5.00th=[ 401], 10.00th=[ 502], 20.00th=[ 550], | 30.00th=[ 575], 40.00th=[ 600], 50.00th=[ 625], 60.00th=[ 667], | 70.00th=[ 726], 80.00th=[ 818], 90.00th=[ 1003], 95.00th=[ 1150], | 99.00th=[ 3004], 99.50th=[ 3071], 99.90th=[ 3440], 99.95th=[ 3473], | 99.99th=[ 3473] bw ( KiB/s): min= 8192, max=368640, per=99.73%, avg=194979.98, stdev=73733.90, samples=60 iops : min= 2, max= 90, avg=47.60, stdev=18.00, samples=60 lat (msec) : 20=0.20%, 50=27.19%, 100=34.79%, 250=15.40%, 500=2.41% lat (msec) : 750=14.21%, 1000=3.61%, 2000=1.92%, >=2000=0.27% cpu : usr=6.51%, sys=1.62%, ctx=2595, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5552,1462,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=725MiB/s (760MB/s), 725MiB/s-725MiB/s (760MB/s-760MB/s), io=21.7GiB (23.3GB), run=30630-30630msec WRITE: bw=191MiB/s (200MB/s), 191MiB/s-191MiB/s (200MB/s-200MB/s), io=5848MiB (6132MB), run=30630-30630msec Disk stats (read/write): vdg: ios=22208/5814, merge=0/0, ticks=135753/3591283, in_queue=3763581, util=99.40% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980492: Fri Jan 7 14:38:43 2022 read: IOPS=165, BW=664MiB/s (696MB/s)(19.5GiB/30011msec) clat (usec): min=377, max=22142, avg=752.56, stdev=1469.43 lat (usec): min=377, max=22142, avg=752.75, stdev=1469.44 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 441], | 70.00th=[ 478], 80.00th=[ 979], 90.00th=[ 1237], 95.00th=[ 1303], | 99.00th=[10552], 99.50th=[13304], 99.90th=[17957], 99.95th=[18744], | 99.99th=[22152] bw ( KiB/s): min=303104, max=1091719, per=100.00%, avg=680108.92, stdev=171968.16, samples=60 iops : min= 74, max= 266, avg=166.03, stdev=41.96, samples=60 write: IOPS=43, BW=172MiB/s (181MB/s)(5168MiB/30011msec) clat (msec): min=11, max=108, avg=19.76, stdev= 5.75 lat (msec): min=11, max=109, avg=20.32, stdev= 5.78 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 20], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 25], 95.00th=[ 26], | 99.00th=[ 37], 99.50th=[ 53], 99.90th=[ 78], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min=155648, max=204800, per=99.96%, avg=176270.00, stdev=11724.96, samples=60 iops : min= 38, max= 50, avg=43.03, stdev= 2.86, samples=60 lat (usec) : 500=57.01%, 750=4.27%, 1000=2.42% lat (msec) : 2=14.65%, 4=0.02%, 10=0.18%, 20=13.53%, 50=7.81% lat (msec) : 100=0.10%, 250=0.02% cpu : usr=2.52%, sys=1.51%, ctx=6281, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4981,1292,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=664MiB/s (696MB/s), 664MiB/s-664MiB/s (696MB/s-696MB/s), io=19.5GiB (20.9GB), run=30011-30011msec WRITE: bw=172MiB/s (181MB/s), 172MiB/s-172MiB/s (181MB/s-181MB/s), io=5168MiB (5419MB), run=30011-30011msec Disk stats (read/write): vdg: ios=19860/5129, merge=0/0, ticks=4361/81932, in_queue=96063, util=96.03% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980956: Fri Jan 7 14:39:21 2022 read: IOPS=176, BW=704MiB/s (739MB/s)(21.1GiB/30701msec) slat (usec): min=46, max=465305, avg=4219.24, stdev=14501.39 clat (msec): min=85, max=2809, avg=309.67, stdev=152.36 lat (msec): min=85, max=2818, avg=313.89, stdev=153.02 clat percentiles (msec): | 1.00th=[ 95], 5.00th=[ 125], 10.00th=[ 174], 20.00th=[ 213], | 30.00th=[ 241], 40.00th=[ 271], 50.00th=[ 292], 60.00th=[ 317], | 70.00th=[ 342], 80.00th=[ 376], 90.00th=[ 426], 95.00th=[ 550], | 99.00th=[ 776], 99.50th=[ 894], 99.90th=[ 1955], 99.95th=[ 2433], | 99.99th=[ 2802] bw ( KiB/s): min= 8192, max=1515520, per=99.66%, avg=718960.67, stdev=295263.23, samples=61 iops : min= 2, max= 370, avg=175.51, stdev=72.09, samples=61 write: IOPS=46, BW=186MiB/s (195MB/s)(5704MiB/30701msec) slat (usec): min=1155, max=457795, avg=5048.89, stdev=15164.97 clat (msec): min=139, max=3447, avg=953.29, stdev=429.39 lat (msec): min=141, max=3448, avg=958.34, stdev=429.34 clat percentiles (msec): | 1.00th=[ 468], 5.00th=[ 584], 10.00th=[ 642], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 852], 60.00th=[ 902], | 70.00th=[ 969], 80.00th=[ 1070], 90.00th=[ 1267], 95.00th=[ 1586], | 99.00th=[ 3272], 99.50th=[ 3339], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3440] bw ( KiB/s): min=16384, max=344064, per=100.00%, avg=191768.61, stdev=69686.36, samples=59 iops : min= 4, max= 84, avg=46.81, stdev=17.02, samples=59 lat (msec) : 100=1.42%, 250=25.08%, 500=47.96%, 750=8.68%, 1000=11.06% lat (msec) : 2000=5.09%, >=2000=0.70% cpu : usr=6.26%, sys=1.56%, ctx=2617, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5407,1426,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=704MiB/s (739MB/s), 704MiB/s-704MiB/s (739MB/s-739MB/s), io=21.1GiB (22.7GB), run=30701-30701msec WRITE: bw=186MiB/s (195MB/s), 186MiB/s-186MiB/s (195MB/s-195MB/s), io=5704MiB (5981MB), run=30701-30701msec Disk stats (read/write): vdg: ios=21628/5688, merge=0/0, ticks=123357/3621126, in_queue=3770901, util=98.99% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981899: Fri Jan 7 14:39:57 2022 read: IOPS=163, BW=653MiB/s (685MB/s)(19.2GiB/30011msec) clat (usec): min=366, max=28767, avg=757.96, stdev=1565.82 lat (usec): min=366, max=28768, avg=758.16, stdev=1565.83 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 478], 80.00th=[ 963], 90.00th=[ 1237], 95.00th=[ 1303], | 99.00th=[10552], 99.50th=[14353], 99.90th=[18482], 99.95th=[19006], | 99.99th=[28705] bw ( KiB/s): min=294912, max=1228800, per=100.00%, avg=669459.57, stdev=175187.69, samples=60 iops : min= 72, max= 300, avg=163.43, stdev=42.78, samples=60 write: IOPS=42, BW=168MiB/s (177MB/s)(5052MiB/30011msec) clat (msec): min=11, max=116, avg=20.26, stdev= 8.04 lat (msec): min=11, max=117, avg=20.81, stdev= 8.08 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 21], | 70.00th=[ 22], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 27], | 99.00th=[ 59], 99.50th=[ 75], 99.90th=[ 114], 99.95th=[ 116], | 99.99th=[ 116] bw ( KiB/s): min=131072, max=204800, per=99.96%, avg=172317.10, stdev=14936.43, samples=60 iops : min= 32, max= 50, avg=42.07, stdev= 3.64, samples=60 lat (usec) : 500=56.47%, 750=5.22%, 1000=2.56% lat (msec) : 2=14.21%, 4=0.02%, 10=0.18%, 20=13.19%, 50=7.88% lat (msec) : 100=0.23%, 250=0.05% cpu : usr=2.42%, sys=1.48%, ctx=6178, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4903,1263,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=653MiB/s (685MB/s), 653MiB/s-653MiB/s (685MB/s-685MB/s), io=19.2GiB (20.6GB), run=30011-30011msec WRITE: bw=168MiB/s (177MB/s), 168MiB/s-168MiB/s (177MB/s-177MB/s), io=5052MiB (5297MB), run=30011-30011msec Disk stats (read/write): vdg: ios=19479/5012, merge=0/0, ticks=4265/82236, in_queue=96101, util=95.30% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983244: Fri Jan 7 14:40:35 2022 read: IOPS=176, BW=708MiB/s (742MB/s)(21.1GiB/30596msec) slat (usec): min=45, max=591677, avg=4229.86, stdev=16613.66 clat (msec): min=329, max=2654, avg=754.67, stdev=201.53 lat (msec): min=329, max=2655, avg=758.90, stdev=202.08 clat percentiles (msec): | 1.00th=[ 405], 5.00th=[ 460], 10.00th=[ 514], 20.00th=[ 609], | 30.00th=[ 659], 40.00th=[ 701], 50.00th=[ 735], 60.00th=[ 768], | 70.00th=[ 827], 80.00th=[ 885], 90.00th=[ 1011], 95.00th=[ 1116], | 99.00th=[ 1385], 99.50th=[ 1435], 99.90th=[ 2165], 99.95th=[ 2601], | 99.99th=[ 2668] bw ( KiB/s): min= 8192, max=1425408, per=99.70%, avg=722606.60, stdev=301841.28, samples=60 iops : min= 2, max= 348, avg=176.38, stdev=73.72, samples=60 write: IOPS=46, BW=187MiB/s (196MB/s)(5712MiB/30596msec) slat (usec): min=1160, max=510458, avg=4947.88, stdev=15912.04 clat (msec): min=603, max=4220, avg=1378.32, stdev=435.78 lat (msec): min=604, max=4221, avg=1383.27, stdev=435.56 clat percentiles (msec): | 1.00th=[ 709], 5.00th=[ 919], 10.00th=[ 1020], 20.00th=[ 1116], | 30.00th=[ 1200], 40.00th=[ 1250], 50.00th=[ 1301], 60.00th=[ 1368], | 70.00th=[ 1452], 80.00th=[ 1552], 90.00th=[ 1720], 95.00th=[ 1905], | 99.00th=[ 3708], 99.50th=[ 3910], 99.90th=[ 4144], 99.95th=[ 4212], | 99.99th=[ 4212] bw ( KiB/s): min=24576, max=475136, per=99.83%, avg=190843.38, stdev=97979.44, samples=58 iops : min= 6, max= 116, avg=46.57, stdev=23.94, samples=58 lat (msec) : 500=6.69%, 750=37.84%, 1000=28.27%, 2000=26.24%, >=2000=0.96% cpu : usr=6.39%, sys=1.61%, ctx=2536, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.5%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5414,1428,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=708MiB/s (742MB/s), 708MiB/s-708MiB/s (742MB/s-742MB/s), io=21.1GiB (22.7GB), run=30596-30596msec WRITE: bw=187MiB/s (196MB/s), 187MiB/s-187MiB/s (196MB/s-196MB/s), io=5712MiB (5989MB), run=30596-30596msec Disk stats (read/write): vdg: ios=21656/5671, merge=0/0, ticks=135493/3579531, in_queue=3759169, util=98.43%