30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890676: Fri Jan 7 13:33:20 2022 read: IOPS=2442, BW=9769KiB/s (10.0MB/s)(286MiB/30001msec) clat (usec): min=25, max=22787, avg=58.08, stdev=383.18 lat (usec): min=25, max=22787, avg=58.22, stdev=383.18 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 64], | 99.00th=[ 80], 99.50th=[ 93], 99.90th=[ 545], 99.95th=[11731], | 99.99th=[16712] bw ( KiB/s): min= 7240, max=14584, per=100.00%, avg=9770.65, stdev=1262.69, samples=60 iops : min= 1810, max= 3646, avg=2442.65, stdev=315.67, samples=60 write: IOPS=610, BW=2441KiB/s (2500kB/s)(71.5MiB/30001msec) clat (usec): min=195, max=53827, avg=1396.94, stdev=1658.65 lat (usec): min=195, max=53827, avg=1397.27, stdev=1658.66 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 562], 10.00th=[ 783], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1713], 95.00th=[ 1991], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[17433], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min= 1952, max= 3784, per=100.00%, avg=2441.17, stdev=271.93, samples=60 iops : min= 488, max= 946, avg=610.28, stdev=67.98, samples=60 lat (usec) : 50=56.08%, 100=23.63%, 250=0.32%, 500=0.73%, 750=0.97% lat (usec) : 1000=2.34% lat (msec) : 2=14.88%, 4=0.67%, 10=0.28%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.43%, sys=3.85%, ctx=91587, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73273,18308,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9769KiB/s (10.0MB/s), 9769KiB/s-9769KiB/s (10.0MB/s-10.0MB/s), io=286MiB (300MB), run=30001-30001msec WRITE: bw=2441KiB/s (2500kB/s), 2441KiB/s-2441KiB/s (2500kB/s-2500kB/s), io=71.5MiB (74.0MB), run=30001-30001msec Disk stats (read/write): vdl: ios=73197/18292, merge=0/0, ticks=746/17825, in_queue=28287, util=93.91% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891554: Fri Jan 7 13:33:57 2022 read: IOPS=2463, BW=9854KiB/s (10.1MB/s)(289MiB/30001msec) slat (usec): min=5, max=155, avg= 9.47, stdev= 3.59 clat (nsec): min=725, max=24033k, avg=45894.54, stdev=399439.82 lat (usec): min=21, max=24040, avg=55.51, stdev=399.46 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 34], 60.00th=[ 35], | 70.00th=[ 37], 80.00th=[ 39], 90.00th=[ 43], 95.00th=[ 46], | 99.00th=[ 57], 99.50th=[ 65], 99.90th=[ 297], 99.95th=[12387], | 99.99th=[16188] bw ( KiB/s): min= 7944, max=14464, per=100.00%, avg=9855.40, stdev=1052.00, samples=60 iops : min= 1986, max= 3616, avg=2463.83, stdev=263.01, samples=60 write: IOPS=615, BW=2461KiB/s (2520kB/s)(72.1MiB/30001msec) slat (usec): min=6, max=206, avg=10.51, stdev= 3.87 clat (usec): min=151, max=44273, avg=1384.28, stdev=1615.87 lat (usec): min=197, max=44282, avg=1394.94, stdev=1615.86 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 570], 10.00th=[ 807], 20.00th=[ 1004], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1713], 95.00th=[ 1991], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[35914], 99.95th=[39060], | 99.99th=[43779] bw ( KiB/s): min= 2000, max= 3712, per=100.00%, avg=2461.22, stdev=260.52, samples=60 iops : min= 500, max= 928, avg=615.30, stdev=65.13, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.06%, 4=0.01%, 10=0.02%, 20=0.13%, 50=77.82% lat (usec) : 100=1.82%, 250=0.17%, 500=0.71%, 750=0.83%, 1000=2.31% lat (msec) : 2=15.07%, 4=0.68%, 10=0.22%, 20=0.10%, 50=0.02% cpu : usr=1.40%, sys=5.94%, ctx=92250, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73906,18458,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9854KiB/s (10.1MB/s), 9854KiB/s-9854KiB/s (10.1MB/s-10.1MB/s), io=289MiB (303MB), run=30001-30001msec WRITE: bw=2461KiB/s (2520kB/s), 2461KiB/s-2461KiB/s (2520kB/s-2520kB/s), io=72.1MiB (75.6MB), run=30001-30001msec Disk stats (read/write): vdl: ios=73264/18306, merge=0/0, ticks=815/17881, in_queue=28087, util=93.85% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892546: Fri Jan 7 13:34:34 2022 read: IOPS=2442, BW=9770KiB/s (10.0MB/s)(286MiB/30001msec) clat (usec): min=25, max=59272, avg=59.07, stdev=459.18 lat (usec): min=25, max=59273, avg=59.20, stdev=459.18 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 80], 99.50th=[ 93], 99.90th=[ 461], 99.95th=[12125], | 99.99th=[19268] bw ( KiB/s): min= 7424, max=14680, per=100.00%, avg=9770.95, stdev=1149.82, samples=60 iops : min= 1856, max= 3670, avg=2442.73, stdev=287.46, samples=60 write: IOPS=610, BW=2441KiB/s (2500kB/s)(71.5MiB/30001msec) clat (usec): min=201, max=46583, avg=1392.68, stdev=1598.68 lat (usec): min=202, max=46583, avg=1392.96, stdev=1598.68 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 578], 10.00th=[ 799], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1303], | 70.00th=[ 1385], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 1975], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[19530], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min= 1928, max= 3808, per=100.00%, avg=2441.27, stdev=268.97, samples=60 iops : min= 482, max= 952, avg=610.32, stdev=67.24, samples=60 lat (usec) : 50=57.02%, 100=22.71%, 250=0.28%, 500=0.74%, 750=0.90% lat (usec) : 1000=2.30% lat (msec) : 2=15.01%, 4=0.67%, 10=0.24%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.49%, sys=3.81%, ctx=91593, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73280,18310,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9770KiB/s (10.0MB/s), 9770KiB/s-9770KiB/s (10.0MB/s-10.0MB/s), io=286MiB (300MB), run=30001-30001msec WRITE: bw=2441KiB/s (2500kB/s), 2441KiB/s-2441KiB/s (2500kB/s-2500kB/s), io=71.5MiB (74.0MB), run=30001-30001msec Disk stats (read/write): vdl: ios=73197/18292, merge=0/0, ticks=850/17688, in_queue=28301, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892994: Fri Jan 7 13:35:11 2022 read: IOPS=2497, BW=9989KiB/s (10.2MB/s)(293MiB/30065msec) slat (usec): min=2, max=1113, avg= 9.63, stdev= 5.68 clat (nsec): min=689, max=2490.8M, avg=875509.62, stdev=37838262.87 lat (usec): min=23, max=2490.8k, avg=885.29, stdev=37838.28 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 33], 40.00th=[ 34], | 50.00th=[ 36], 60.00th=[ 37], 70.00th=[ 39], | 80.00th=[ 42], 90.00th=[ 47], 95.00th=[ 53], | 99.00th=[ 80], 99.50th=[ 99], 99.90th=[ 562], | 99.95th=[ 574620], 99.99th=[2499806] bw ( KiB/s): min= 7560, max=17432, per=100.00%, avg=10010.00, stdev=1398.71, samples=60 iops : min= 1890, max= 4358, avg=2502.50, stdev=349.68, samples=60 write: IOPS=622, BW=2492KiB/s (2552kB/s)(73.2MiB/30065msec) slat (nsec): min=2729, max=85726, avg=10745.63, stdev=3702.92 clat (usec): min=316, max=86476, avg=12475.35, stdev=6054.02 lat (usec): min=321, max=86490, avg=12486.25, stdev=6054.22 clat percentiles (usec): | 1.00th=[ 1844], 5.00th=[ 6390], 10.00th=[ 7963], 20.00th=[ 9503], | 30.00th=[10552], 40.00th=[11338], 50.00th=[11994], 60.00th=[12518], | 70.00th=[13173], 80.00th=[13960], 90.00th=[16581], 95.00th=[20579], | 99.00th=[30016], 99.50th=[61604], 99.90th=[71828], 99.95th=[72877], | 99.99th=[85459] bw ( KiB/s): min= 1920, max= 4400, per=100.00%, avg=2496.80, stdev=312.78, samples=60 iops : min= 480, max= 1100, avg=624.20, stdev=78.20, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.09%, 4=0.01%, 10=0.05%, 20=0.23%, 50=74.34% lat (usec) : 100=4.89%, 250=0.28%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.22%, 4=0.12%, 10=4.58%, 20=13.87%, 50=1.02% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.01% cpu : usr=1.36%, sys=6.09%, ctx=91618, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=75082,18729,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9989KiB/s (10.2MB/s), 9989KiB/s-9989KiB/s (10.2MB/s-10.2MB/s), io=293MiB (308MB), run=30065-30065msec WRITE: bw=2492KiB/s (2552kB/s), 2492KiB/s-2492KiB/s (2552kB/s-2552kB/s), io=73.2MiB (76.7MB), run=30065-30065msec Disk stats (read/write): vdl: ios=75075/18724, merge=0/0, ticks=58005/224045, in_queue=297921, util=99.44% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894410: Fri Jan 7 13:35:48 2022 read: IOPS=2445, BW=9783KiB/s (10.0MB/s)(287MiB/30009msec) clat (usec): min=25, max=58687, avg=57.59, stdev=432.04 lat (usec): min=25, max=58687, avg=57.73, stdev=432.04 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 36], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 47], | 70.00th=[ 49], 80.00th=[ 52], 90.00th=[ 57], 95.00th=[ 62], | 99.00th=[ 76], 99.50th=[ 88], 99.90th=[ 457], 99.95th=[12256], | 99.99th=[16057] bw ( KiB/s): min= 6928, max=15584, per=100.00%, avg=9785.27, stdev=1233.60, samples=60 iops : min= 1732, max= 3896, avg=2446.32, stdev=308.40, samples=60 write: IOPS=611, BW=2444KiB/s (2503kB/s)(71.6MiB/30009msec) clat (usec): min=194, max=44030, avg=1396.93, stdev=1628.00 lat (usec): min=194, max=44030, avg=1397.21, stdev=1628.00 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 578], 10.00th=[ 799], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1500], 90.00th=[ 1696], 95.00th=[ 1975], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[17957], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min= 1816, max= 4000, per=100.00%, avg=2444.58, stdev=304.54, samples=60 iops : min= 454, max= 1000, avg=611.13, stdev=76.14, samples=60 lat (usec) : 50=59.31%, 100=20.47%, 250=0.23%, 500=0.71%, 750=0.92% lat (usec) : 1000=2.43% lat (msec) : 2=14.89%, 4=0.66%, 10=0.26%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.43%, sys=3.75%, ctx=91739, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73392,18336,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9783KiB/s (10.0MB/s), 9783KiB/s-9783KiB/s (10.0MB/s-10.0MB/s), io=287MiB (301MB), run=30009-30009msec WRITE: bw=2444KiB/s (2503kB/s), 2444KiB/s-2444KiB/s (2503kB/s-2503kB/s), io=71.6MiB (75.1MB), run=30009-30009msec Disk stats (read/write): vdl: ios=73336/18322, merge=0/0, ticks=814/17811, in_queue=28348, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894982: Fri Jan 7 13:36:25 2022 read: IOPS=2483, BW=9934KiB/s (10.2MB/s)(292MiB/30114msec) slat (usec): min=2, max=245, avg= 9.60, stdev= 5.22 clat (nsec): min=688, max=2511.0M, avg=876883.97, stdev=36983446.84 lat (usec): min=21, max=2511.0k, avg=886.63, stdev=36983.43 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 41], | 80.00th=[ 45], 90.00th=[ 56], 95.00th=[ 73], | 99.00th=[ 178], 99.50th=[ 375], 99.90th=[ 1205], | 99.95th=[ 517997], 99.99th=[2499806] bw ( KiB/s): min= 7976, max=18264, per=100.00%, avg=9971.05, stdev=1386.95, samples=60 iops : min= 1994, max= 4566, avg=2492.75, stdev=346.72, samples=60 write: IOPS=619, BW=2480KiB/s (2539kB/s)(72.9MiB/30114msec) slat (usec): min=2, max=152, avg=10.77, stdev= 4.51 clat (msec): min=7, max=2391, avg=77.03, stdev=60.01 lat (msec): min=7, max=2391, avg=77.04, stdev=60.01 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 65], 40.00th=[ 68], 50.00th=[ 70], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 92], 95.00th=[ 117], | 99.00th=[ 213], 99.50th=[ 451], 99.90th=[ 936], 99.95th=[ 1083], | 99.99th=[ 1821] bw ( KiB/s): min= 2056, max= 4328, per=100.00%, avg=2483.42, stdev=295.27, samples=60 iops : min= 514, max= 1082, avg=620.85, stdev=73.81, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.16%, 50=68.77% lat (usec) : 100=9.26%, 250=1.07%, 500=0.34%, 750=0.13%, 1000=0.04% lat (msec) : 2=0.03%, 10=0.17%, 20=0.10%, 50=0.26%, 100=17.91% lat (msec) : 250=1.36%, 500=0.11%, 750=0.06%, 1000=0.03%, 2000=0.03% lat (msec) : >=2000=0.01% cpu : usr=1.35%, sys=6.03%, ctx=88811, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=74785,18670,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9934KiB/s (10.2MB/s), 9934KiB/s-9934KiB/s (10.2MB/s-10.2MB/s), io=292MiB (306MB), run=30114-30114msec WRITE: bw=2480KiB/s (2539kB/s), 2480KiB/s-2480KiB/s (2539kB/s-2539kB/s), io=72.9MiB (76.5MB), run=30114-30114msec Disk stats (read/write): vdl: ios=74675/18589, merge=0/0, ticks=58570/1417275, in_queue=1494794, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895823: Fri Jan 7 13:37:02 2022 read: IOPS=2452, BW=9808KiB/s (10.0MB/s)(287MiB/30001msec) clat (usec): min=24, max=19956, avg=57.45, stdev=375.54 lat (usec): min=25, max=19956, avg=57.58, stdev=375.54 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 79], 99.50th=[ 92], 99.90th=[ 506], 99.95th=[11469], | 99.99th=[15795] bw ( KiB/s): min= 7200, max=14520, per=100.00%, avg=9831.92, stdev=1198.65, samples=59 iops : min= 1800, max= 3630, avg=2457.97, stdev=299.66, samples=59 write: IOPS=612, BW=2451KiB/s (2510kB/s)(71.8MiB/30001msec) clat (usec): min=200, max=44571, avg=1392.65, stdev=1688.28 lat (usec): min=200, max=44571, avg=1392.92, stdev=1688.28 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 570], 10.00th=[ 783], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 1975], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[37487], 99.95th=[41681], | 99.99th=[44303] bw ( KiB/s): min= 1936, max= 3752, per=100.00%, avg=2460.00, stdev=256.48, samples=59 iops : min= 484, max= 938, avg=615.00, stdev=64.12, samples=59 lat (usec) : 50=57.93%, 100=21.80%, 250=0.24%, 500=0.75%, 750=1.02% lat (usec) : 1000=2.29% lat (msec) : 2=14.96%, 4=0.65%, 10=0.24%, 20=0.09%, 50=0.02% cpu : usr=1.41%, sys=3.85%, ctx=91955, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73563,18382,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9808KiB/s (10.0MB/s), 9808KiB/s-9808KiB/s (10.0MB/s-10.0MB/s), io=287MiB (301MB), run=30001-30001msec WRITE: bw=2451KiB/s (2510kB/s), 2451KiB/s-2451KiB/s (2510kB/s-2510kB/s), io=71.8MiB (75.3MB), run=30001-30001msec Disk stats (read/write): vdl: ios=73390/18334, merge=0/0, ticks=766/17761, in_queue=28292, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896940: Fri Jan 7 13:37:39 2022 read: IOPS=2485, BW=9943KiB/s (10.2MB/s)(293MiB/30164msec) slat (usec): min=2, max=481, avg= 9.89, stdev= 8.61 clat (nsec): min=636, max=2503.2M, avg=910365.25, stdev=34621950.43 lat (usec): min=21, max=2503.3k, avg=920.42, stdev=34621.94 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 42], | 80.00th=[ 46], 90.00th=[ 62], 95.00th=[ 139], | 99.00th=[ 1565], 99.50th=[ 2507], 99.90th=[ 5080], | 99.95th=[ 734004], 99.99th=[1870660] bw ( KiB/s): min= 7760, max=18776, per=100.00%, avg=9997.87, stdev=1498.48, samples=60 iops : min= 1940, max= 4694, avg=2499.45, stdev=374.62, samples=60 write: IOPS=620, BW=2481KiB/s (2540kB/s)(73.1MiB/30164msec) slat (usec): min=2, max=600, avg=11.11, stdev= 9.80 clat (msec): min=2, max=2425, avg=157.37, stdev=90.26 lat (msec): min=2, max=2425, avg=157.38, stdev=90.26 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 134], | 30.00th=[ 140], 40.00th=[ 144], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 157], 80.00th=[ 165], 90.00th=[ 192], 95.00th=[ 207], | 99.00th=[ 435], 99.50th=[ 760], 99.90th=[ 1452], 99.95th=[ 1921], | 99.99th=[ 2072] bw ( KiB/s): min= 1920, max= 4072, per=100.00%, avg=2481.62, stdev=279.83, samples=60 iops : min= 480, max= 1018, avg=620.40, stdev=69.96, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.10%, 4=0.01%, 10=0.03%, 20=0.16%, 50=67.45% lat (usec) : 100=6.90%, 250=2.56%, 500=0.79%, 750=0.37%, 1000=0.27% lat (msec) : 2=0.78%, 4=0.44%, 10=0.13%, 20=0.17%, 50=0.14% lat (msec) : 100=0.20%, 250=18.87%, 500=0.34%, 750=0.08%, 1000=0.06% lat (msec) : 2000=0.07%, >=2000=0.01% cpu : usr=1.60%, sys=5.88%, ctx=87407, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=74977,18707,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9943KiB/s (10.2MB/s), 9943KiB/s-9943KiB/s (10.2MB/s-10.2MB/s), io=293MiB (307MB), run=30164-30164msec WRITE: bw=2481KiB/s (2540kB/s), 2481KiB/s-2481KiB/s (2540kB/s-2540kB/s), io=73.1MiB (76.6MB), run=30164-30164msec Disk stats (read/write): vdl: ios=74975/18638, merge=0/0, ticks=61499/2912483, in_queue=2997041, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897510: Fri Jan 7 13:38:16 2022 read: IOPS=2419, BW=9679KiB/s (9911kB/s)(284MiB/30009msec) clat (usec): min=26, max=65755, avg=60.74, stdev=462.46 lat (usec): min=26, max=65755, avg=60.88, stdev=462.47 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 84], 99.50th=[ 99], 99.90th=[ 570], 99.95th=[12387], | 99.99th=[16581] bw ( KiB/s): min= 6952, max=14944, per=100.00%, avg=9681.27, stdev=1190.56, samples=60 iops : min= 1738, max= 3736, avg=2420.32, stdev=297.64, samples=60 write: IOPS=605, BW=2422KiB/s (2480kB/s)(70.0MiB/30009msec) clat (usec): min=200, max=44335, avg=1399.35, stdev=1664.31 lat (usec): min=200, max=44336, avg=1399.65, stdev=1664.31 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 562], 10.00th=[ 766], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1713], 95.00th=[ 1991], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[21627], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min= 1880, max= 3848, per=100.00%, avg=2422.18, stdev=273.62, samples=60 iops : min= 470, max= 962, avg=605.53, stdev=68.41, samples=60 lat (usec) : 50=50.65%, 100=28.96%, 250=0.37%, 500=0.76%, 750=1.09% lat (usec) : 1000=2.36% lat (msec) : 2=14.77%, 4=0.64%, 10=0.29%, 20=0.09%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.61%, sys=3.81%, ctx=90788, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72612,18168,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9679KiB/s (9911kB/s), 9679KiB/s-9679KiB/s (9911kB/s-9911kB/s), io=284MiB (297MB), run=30009-30009msec WRITE: bw=2422KiB/s (2480kB/s), 2422KiB/s-2422KiB/s (2480kB/s-2480kB/s), io=70.0MiB (74.4MB), run=30009-30009msec Disk stats (read/write): vdl: ios=72528/18146, merge=0/0, ticks=824/17614, in_queue=28111, util=93.36% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898366: Fri Jan 7 13:38:53 2022 read: IOPS=2504, BW=9.78MiB/s (10.3MB/s)(295MiB/30199msec) slat (usec): min=2, max=222401, avg=316.05, stdev=1878.42 clat (usec): min=507, max=2550.9k, avg=25785.35, stdev=41891.83 lat (usec): min=510, max=2550.9k, avg=26101.54, stdev=41938.22 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 7504], 10.00th=[ 13829], | 20.00th=[ 17171], 30.00th=[ 19530], 40.00th=[ 21365], | 50.00th=[ 23200], 60.00th=[ 25035], 70.00th=[ 27132], | 80.00th=[ 30278], 90.00th=[ 34866], 95.00th=[ 40633], | 99.00th=[ 81265], 99.50th=[ 177210], 99.90th=[ 248513], | 99.95th=[ 557843], 99.99th=[2533360] bw ( KiB/s): min= 7752, max=19336, per=100.00%, avg=10077.07, stdev=1541.04, samples=60 iops : min= 1938, max= 4834, avg=2519.27, stdev=385.26, samples=60 write: IOPS=624, BW=2500KiB/s (2560kB/s)(73.7MiB/30199msec) slat (usec): min=2, max=169992, avg=311.79, stdev=2285.05 clat (msec): min=11, max=2391, avg=214.74, stdev=118.72 lat (msec): min=11, max=2391, avg=215.05, stdev=118.71 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 144], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 188], 40.00th=[ 194], 50.00th=[ 201], 60.00th=[ 207], | 70.00th=[ 213], 80.00th=[ 228], 90.00th=[ 257], 95.00th=[ 292], | 99.00th=[ 735], 99.50th=[ 1036], 99.90th=[ 1821], 99.95th=[ 1938], | 99.99th=[ 2198] bw ( KiB/s): min= 2072, max= 4056, per=100.00%, avg=2498.93, stdev=273.51, samples=60 iops : min= 518, max= 1014, avg=624.73, stdev=68.38, samples=60 lat (usec) : 750=0.08%, 1000=0.50% lat (msec) : 2=1.65%, 4=1.20%, 10=1.17%, 20=21.40%, 50=52.22% lat (msec) : 100=1.82%, 250=17.53%, 500=2.07%, 750=0.15%, 1000=0.07% lat (msec) : 2000=0.12%, >=2000=0.02% cpu : usr=0.76%, sys=2.00%, ctx=10260, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75646,18874,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9.78MiB/s (10.3MB/s), 9.78MiB/s-9.78MiB/s (10.3MB/s-10.3MB/s), io=295MiB (310MB), run=30199-30199msec WRITE: bw=2500KiB/s (2560kB/s), 2500KiB/s-2500KiB/s (2560kB/s-2560kB/s), io=73.7MiB (77.3MB), run=30199-30199msec Disk stats (read/write): vdl: ios=75643/18874, merge=0/0, ticks=70871/3587696, in_queue=3677683, util=99.26% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898992: Fri Jan 7 13:39:30 2022 read: IOPS=2380, BW=37.2MiB/s (38.0MB/s)(1116MiB/30001msec) clat (usec): min=27, max=60519, avg=63.18, stdev=451.06 lat (usec): min=27, max=60519, avg=63.32, stdev=451.06 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 832], 99.95th=[12649], | 99.99th=[16057] bw ( KiB/s): min=28288, max=53952, per=100.00%, avg=38119.59, stdev=4183.12, samples=59 iops : min= 1768, max= 3372, avg=2382.47, stdev=261.44, samples=59 write: IOPS=595, BW=9531KiB/s (9760kB/s)(279MiB/30001msec) clat (usec): min=235, max=44308, avg=1415.95, stdev=1596.53 lat (usec): min=236, max=44309, avg=1417.60, stdev=1596.50 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 570], 10.00th=[ 799], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[17171], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min= 7488, max=14016, per=100.00%, avg=9539.80, stdev=973.32, samples=59 iops : min= 468, max= 876, avg=596.24, stdev=60.83, samples=59 lat (usec) : 50=44.41%, 100=35.11%, 250=0.35%, 500=0.78%, 750=1.00% lat (usec) : 1000=2.06% lat (msec) : 2=15.11%, 4=0.81%, 10=0.25%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.48%, sys=3.88%, ctx=89293, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71411,17871,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.2MiB/s (38.0MB/s), 37.2MiB/s-37.2MiB/s (38.0MB/s-38.0MB/s), io=1116MiB (1170MB), run=30001-30001msec WRITE: bw=9531KiB/s (9760kB/s), 9531KiB/s-9531KiB/s (9760kB/s-9760kB/s), io=279MiB (293MB), run=30001-30001msec Disk stats (read/write): vdl: ios=70942/17743, merge=0/0, ticks=788/17533, in_queue=28064, util=93.93% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899808: Fri Jan 7 13:40:07 2022 read: IOPS=2381, BW=37.2MiB/s (39.0MB/s)(1116MiB/30003msec) slat (usec): min=5, max=218, avg=10.49, stdev= 5.07 clat (nsec): min=671, max=18182k, avg=47502.42, stdev=377382.25 lat (usec): min=23, max=18189, avg=58.15, stdev=377.42 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 32], | 30.00th=[ 33], 40.00th=[ 35], 50.00th=[ 36], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 46], 95.00th=[ 50], | 99.00th=[ 63], 99.50th=[ 75], 99.90th=[ 396], 99.95th=[12256], | 99.99th=[16319] bw ( KiB/s): min=29760, max=58016, per=100.00%, avg=38107.40, stdev=4444.53, samples=60 iops : min= 1860, max= 3626, avg=2381.70, stdev=277.78, samples=60 write: IOPS=596, BW=9536KiB/s (9765kB/s)(279MiB/30003msec) slat (usec): min=8, max=191, avg=12.77, stdev= 4.94 clat (usec): min=222, max=44566, avg=1424.62, stdev=1627.92 lat (usec): min=232, max=44579, avg=1437.55, stdev=1627.94 clat percentiles (usec): | 1.00th=[ 297], 5.00th=[ 594], 10.00th=[ 824], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[36439], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min= 8096, max=14912, per=100.00%, avg=9536.83, stdev=959.86, samples=60 iops : min= 506, max= 932, avg=596.05, stdev=59.99, samples=60 lat (nsec) : 750=0.03%, 1000=0.09% lat (usec) : 2=0.13%, 4=0.01%, 10=0.06%, 20=0.22%, 50=75.60% lat (usec) : 100=3.67%, 250=0.12%, 500=0.73%, 750=0.88%, 1000=1.96% lat (msec) : 2=15.34%, 4=0.80%, 10=0.25%, 20=0.10%, 50=0.02% cpu : usr=1.43%, sys=6.14%, ctx=89024, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71449,17882,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1116MiB (1171MB), run=30003-30003msec WRITE: bw=9536KiB/s (9765kB/s), 9536KiB/s-9536KiB/s (9765kB/s-9765kB/s), io=279MiB (293MB), run=30003-30003msec Disk stats (read/write): vdl: ios=70994/17756, merge=0/0, ticks=716/17919, in_queue=27957, util=93.60% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901196: Fri Jan 7 13:40:44 2022 read: IOPS=2389, BW=37.3MiB/s (39.2MB/s)(1120MiB/30002msec) clat (usec): min=26, max=23332, avg=61.76, stdev=410.17 lat (usec): min=26, max=23332, avg=61.90, stdev=410.17 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 97], 99.90th=[ 486], 99.95th=[12911], | 99.99th=[16909] bw ( KiB/s): min=30592, max=56288, per=100.00%, avg=38239.57, stdev=4412.77, samples=60 iops : min= 1912, max= 3518, avg=2389.97, stdev=275.80, samples=60 write: IOPS=598, BW=9574KiB/s (9804kB/s)(281MiB/30002msec) clat (usec): min=233, max=44095, avg=1414.35, stdev=1608.97 lat (usec): min=235, max=44097, avg=1415.79, stdev=1608.96 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 586], 10.00th=[ 807], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2008], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[38536], | 99.99th=[44303] bw ( KiB/s): min= 7776, max=14336, per=100.00%, avg=9574.15, stdev=962.28, samples=60 iops : min= 486, max= 896, avg=598.38, stdev=60.14, samples=60 lat (usec) : 50=47.28%, 100=32.33%, 250=0.27%, 500=0.76%, 750=0.93% lat (usec) : 1000=2.05% lat (msec) : 2=15.31%, 4=0.73%, 10=0.22%, 20=0.10%, 50=0.02% cpu : usr=1.41%, sys=3.90%, ctx=89652, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71697,17952,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.3MiB/s (39.2MB/s), 37.3MiB/s-37.3MiB/s (39.2MB/s-39.2MB/s), io=1120MiB (1175MB), run=30002-30002msec WRITE: bw=9574KiB/s (9804kB/s), 9574KiB/s-9574KiB/s (9804kB/s-9804kB/s), io=281MiB (294MB), run=30002-30002msec Disk stats (read/write): vdl: ios=71219/17822, merge=0/0, ticks=801/17524, in_queue=28094, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901859: Fri Jan 7 13:41:21 2022 read: IOPS=2430, BW=37.0MiB/s (39.8MB/s)(1141MiB/30031msec) slat (usec): min=2, max=223, avg=10.97, stdev= 4.58 clat (nsec): min=669, max=2496.6M, avg=915340.26, stdev=38173794.08 lat (usec): min=23, max=2496.6k, avg=926.47, stdev=38173.79 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 43], | 80.00th=[ 46], 90.00th=[ 52], 95.00th=[ 58], | 99.00th=[ 92], 99.50th=[ 126], 99.90th=[ 717], | 99.95th=[ 608175], 99.99th=[2499806] bw ( KiB/s): min=29664, max=68736, per=100.00%, avg=38930.17, stdev=5530.51, samples=60 iops : min= 1854, max= 4296, avg=2433.13, stdev=345.66, samples=60 write: IOPS=607, BW=9722KiB/s (9956kB/s)(285MiB/30031msec) slat (usec): min=3, max=136, avg=13.75, stdev= 4.92 clat (usec): min=433, max=72391, avg=12725.84, stdev=5862.29 lat (usec): min=441, max=72408, avg=12739.76, stdev=5862.37 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 7177], 10.00th=[ 8455], 20.00th=[ 9634], | 30.00th=[10552], 40.00th=[11338], 50.00th=[12125], 60.00th=[12911], | 70.00th=[13566], 80.00th=[14353], 90.00th=[16712], 95.00th=[20841], | 99.00th=[29754], 99.50th=[61080], 99.90th=[65799], 99.95th=[69731], | 99.99th=[71828] bw ( KiB/s): min= 7904, max=17280, per=100.00%, avg=9730.20, stdev=1212.66, samples=60 iops : min= 494, max= 1080, avg=608.13, stdev=75.79, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.04%, 4=0.01%, 10=0.03%, 20=0.11%, 50=70.38% lat (usec) : 100=8.69%, 250=0.53%, 500=0.04%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.22%, 4=0.08%, 10=4.48%, 20=13.98%, 50=1.07% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.49%, sys=6.44%, ctx=89578, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72994,18248,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.0MiB/s (39.8MB/s), 37.0MiB/s-37.0MiB/s (39.8MB/s-39.8MB/s), io=1141MiB (1196MB), run=30031-30031msec WRITE: bw=9722KiB/s (9956kB/s), 9722KiB/s-9722KiB/s (9956kB/s-9956kB/s), io=285MiB (299MB), run=30031-30031msec Disk stats (read/write): vdl: ios=72442/18126, merge=0/0, ticks=59387/221762, in_queue=296010, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902763: Fri Jan 7 13:41:58 2022 read: IOPS=2393, BW=37.4MiB/s (39.2MB/s)(1122MiB/30001msec) clat (usec): min=26, max=19588, avg=60.20, stdev=375.10 lat (usec): min=26, max=19588, avg=60.34, stdev=375.10 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 67], | 99.00th=[ 83], 99.50th=[ 95], 99.90th=[ 355], 99.95th=[11207], | 99.99th=[16319] bw ( KiB/s): min=30080, max=59680, per=100.00%, avg=38450.29, stdev=4456.25, samples=59 iops : min= 1880, max= 3730, avg=2403.12, stdev=278.51, samples=59 write: IOPS=599, BW=9586KiB/s (9816kB/s)(281MiB/30001msec) clat (usec): min=237, max=44669, avg=1418.50, stdev=1629.65 lat (usec): min=239, max=44670, avg=1419.96, stdev=1629.64 clat percentiles (usec): | 1.00th=[ 314], 5.00th=[ 603], 10.00th=[ 824], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2024], | 99.00th=[ 9110], 99.50th=[ 9634], 99.90th=[37487], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min= 7872, max=15328, per=100.00%, avg=9613.78, stdev=1073.53, samples=59 iops : min= 492, max= 958, avg=600.85, stdev=67.09, samples=59 lat (usec) : 50=47.82%, 100=31.86%, 250=0.22%, 500=0.73%, 750=0.88% lat (usec) : 1000=1.98% lat (msec) : 2=15.39%, 4=0.79%, 10=0.22%, 20=0.09%, 50=0.02% cpu : usr=1.38%, sys=3.98%, ctx=89781, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71801,17975,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.4MiB/s (39.2MB/s), 37.4MiB/s-37.4MiB/s (39.2MB/s-39.2MB/s), io=1122MiB (1176MB), run=30001-30001msec WRITE: bw=9586KiB/s (9816kB/s), 9586KiB/s-9586KiB/s (9816kB/s-9816kB/s), io=281MiB (295MB), run=30001-30001msec Disk stats (read/write): vdl: ios=71398/17861, merge=0/0, ticks=737/17676, in_queue=28090, util=93.86% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903786: Fri Jan 7 13:42:35 2022 read: IOPS=2431, BW=37.0MiB/s (39.8MB/s)(1144MiB/30118msec) slat (usec): min=2, max=456, avg=10.00, stdev= 4.99 clat (nsec): min=730, max=2573.5M, avg=997126.28, stdev=40460148.15 lat (usec): min=18, max=2573.5k, avg=1007.27, stdev=40460.15 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 40], 70.00th=[ 42], | 80.00th=[ 46], 90.00th=[ 55], 95.00th=[ 71], | 99.00th=[ 227], 99.50th=[ 498], 99.90th=[ 1156], | 99.95th=[ 658506], 99.99th=[2499806] bw ( KiB/s): min=32096, max=70272, per=100.00%, avg=39058.60, stdev=5160.45, samples=60 iops : min= 2006, max= 4392, avg=2441.15, stdev=322.54, samples=60 write: IOPS=607, BW=9723KiB/s (9957kB/s)(286MiB/30118msec) slat (usec): min=3, max=117, avg=12.96, stdev= 4.82 clat (msec): min=8, max=2520, avg=78.21, stdev=68.37 lat (msec): min=8, max=2520, avg=78.22, stdev=68.37 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 55], 10.00th=[ 59], 20.00th=[ 63], | 30.00th=[ 67], 40.00th=[ 69], 50.00th=[ 71], 60.00th=[ 73], | 70.00th=[ 77], 80.00th=[ 81], 90.00th=[ 89], 95.00th=[ 118], | 99.00th=[ 234], 99.50th=[ 502], 99.90th=[ 1053], 99.95th=[ 1250], | 99.99th=[ 2433] bw ( KiB/s): min= 8384, max=16672, per=100.00%, avg=9737.47, stdev=1104.45, samples=60 iops : min= 524, max= 1042, avg=608.58, stdev=69.03, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.08%, 4=0.01%, 10=0.06%, 20=0.22%, 50=68.69% lat (usec) : 100=9.21%, 250=0.96%, 500=0.34%, 750=0.17%, 1000=0.10% lat (msec) : 2=0.06%, 10=0.08%, 20=0.24%, 50=0.22%, 100=17.98% lat (msec) : 250=1.30%, 500=0.10%, 750=0.06%, 1000=0.03%, 2000=0.04% lat (msec) : >=2000=0.02% cpu : usr=1.40%, sys=5.87%, ctx=88518, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=73230,18303,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.0MiB/s (39.8MB/s), 37.0MiB/s-37.0MiB/s (39.8MB/s-39.8MB/s), io=1144MiB (1200MB), run=30118-30118msec WRITE: bw=9723KiB/s (9957kB/s), 9723KiB/s-9723KiB/s (9957kB/s-9957kB/s), io=286MiB (300MB), run=30118-30118msec Disk stats (read/write): vdl: ios=73228/18262, merge=0/0, ticks=69429/1415250, in_queue=1498971, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904249: Fri Jan 7 13:43:12 2022 read: IOPS=2372, BW=37.1MiB/s (38.9MB/s)(1112MiB/30002msec) clat (usec): min=28, max=29060, avg=62.09, stdev=419.65 lat (usec): min=28, max=29060, avg=62.23, stdev=419.65 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 68], | 99.00th=[ 87], 99.50th=[ 102], 99.90th=[ 465], 99.95th=[12125], | 99.99th=[17171] bw ( KiB/s): min=27750, max=58100, per=100.00%, avg=38197.57, stdev=4432.89, samples=60 iops : min= 1734, max= 3631, avg=2386.92, stdev=277.06, samples=60 write: IOPS=593, BW=9497KiB/s (9725kB/s)(278MiB/30002msec) clat (usec): min=237, max=45334, avg=1426.41, stdev=1641.03 lat (usec): min=239, max=45335, avg=1427.90, stdev=1641.02 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 594], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min= 7805, max=14909, per=100.00%, avg=9556.13, stdev=989.18, samples=60 iops : min= 487, max= 931, avg=596.92, stdev=61.75, samples=60 lat (usec) : 50=46.02%, 100=33.54%, 250=0.33%, 500=0.75%, 750=0.97% lat (usec) : 1000=2.00% lat (msec) : 2=15.26%, 4=0.76%, 10=0.26%, 20=0.09%, 50=0.03% cpu : usr=1.41%, sys=3.95%, ctx=88980, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71167,17808,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1112MiB (1166MB), run=30002-30002msec WRITE: bw=9497KiB/s (9725kB/s), 9497KiB/s-9497KiB/s (9725kB/s-9725kB/s), io=278MiB (292MB), run=30002-30002msec Disk stats (read/write): vdl: ios=71156/17803, merge=0/0, ticks=809/17724, in_queue=28327, util=94.10% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905222: Fri Jan 7 13:43:49 2022 read: IOPS=2446, BW=38.2MiB/s (40.1MB/s)(1153MiB/30169msec) slat (usec): min=2, max=199, avg= 9.81, stdev= 4.71 clat (nsec): min=752, max=2537.4M, avg=961054.39, stdev=39269740.11 lat (usec): min=19, max=2537.4k, avg=971.00, stdev=39269.74 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 43], | 80.00th=[ 48], 90.00th=[ 63], 95.00th=[ 124], | 99.00th=[ 930], 99.50th=[ 1434], 99.90th=[ 3032], | 99.95th=[ 683672], 99.99th=[2499806] bw ( KiB/s): min=28416, max=68832, per=100.00%, avg=39358.83, stdev=5787.65, samples=60 iops : min= 1776, max= 4302, avg=2459.92, stdev=361.74, samples=60 write: IOPS=611, BW=9779KiB/s (10.0MB/s)(288MiB/30169msec) slat (usec): min=3, max=223, avg=12.90, stdev= 5.23 clat (msec): min=7, max=2505, avg=159.69, stdev=106.04 lat (msec): min=7, max=2505, avg=159.70, stdev=106.04 clat percentiles (msec): | 1.00th=[ 18], 5.00th=[ 100], 10.00th=[ 116], 20.00th=[ 136], | 30.00th=[ 140], 40.00th=[ 144], 50.00th=[ 148], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 192], 95.00th=[ 215], | 99.00th=[ 550], 99.50th=[ 919], 99.90th=[ 1670], 99.95th=[ 1955], | 99.99th=[ 2500] bw ( KiB/s): min= 6912, max=14432, per=100.00%, avg=9782.65, stdev=1138.03, samples=60 iops : min= 432, max= 902, avg=611.40, stdev=71.14, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.06%, 4=0.01%, 10=0.03%, 20=0.12%, 50=66.37% lat (usec) : 100=8.32%, 250=2.60%, 500=0.84%, 750=0.56%, 1000=0.33% lat (msec) : 2=0.52%, 4=0.14%, 10=0.01%, 20=0.23%, 50=0.18% lat (msec) : 100=0.65%, 250=18.33%, 500=0.41%, 750=0.09%, 1000=0.04% lat (msec) : 2000=0.10%, >=2000=0.02% cpu : usr=1.45%, sys=5.75%, ctx=86475, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73804,18438,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=38.2MiB/s (40.1MB/s), 38.2MiB/s-38.2MiB/s (40.1MB/s-40.1MB/s), io=1153MiB (1209MB), run=30169-30169msec WRITE: bw=9779KiB/s (10.0MB/s), 9779KiB/s-9779KiB/s (10.0MB/s-10.0MB/s), io=288MiB (302MB), run=30169-30169msec Disk stats (read/write): vdl: ios=73733/18326, merge=0/0, ticks=63958/2904202, in_queue=2995341, util=99.62% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905799: Fri Jan 7 13:44:26 2022 read: IOPS=2373, BW=37.1MiB/s (38.9MB/s)(1113MiB/30002msec) clat (usec): min=27, max=22217, avg=60.73, stdev=399.53 lat (usec): min=27, max=22217, avg=60.87, stdev=399.54 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 82], 99.50th=[ 94], 99.90th=[ 717], 99.95th=[12911], | 99.99th=[16581] bw ( KiB/s): min=30432, max=56832, per=100.00%, avg=37983.05, stdev=4144.11, samples=60 iops : min= 1902, max= 3552, avg=2373.93, stdev=259.01, samples=60 write: IOPS=594, BW=9504KiB/s (9733kB/s)(278MiB/30002msec) clat (usec): min=234, max=45740, avg=1430.85, stdev=1622.22 lat (usec): min=236, max=45742, avg=1432.30, stdev=1622.22 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 594], 10.00th=[ 824], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1418], 80.00th=[ 1549], 90.00th=[ 1778], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35914], 99.95th=[40109], | 99.99th=[41681] bw ( KiB/s): min= 7712, max=14464, per=100.00%, avg=9504.83, stdev=964.12, samples=60 iops : min= 482, max= 904, avg=594.05, stdev=60.26, samples=60 lat (usec) : 50=49.43%, 100=30.24%, 250=0.23%, 500=0.73%, 750=0.87% lat (usec) : 1000=1.97% lat (msec) : 2=15.35%, 4=0.81%, 10=0.24%, 20=0.10%, 50=0.02% cpu : usr=1.44%, sys=3.83%, ctx=89046, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71216,17822,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1113MiB (1167MB), run=30002-30002msec WRITE: bw=9504KiB/s (9733kB/s), 9504KiB/s-9504KiB/s (9733kB/s-9733kB/s), io=278MiB (292MB), run=30002-30002msec Disk stats (read/write): vdl: ios=70792/17699, merge=0/0, ticks=764/17732, in_queue=28024, util=93.79% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906620: Fri Jan 7 13:45:03 2022 read: IOPS=2425, BW=37.9MiB/s (39.7MB/s)(1147MiB/30255msec) slat (usec): min=2, max=208069, avg=324.77, stdev=2008.55 clat (usec): min=706, max=2636.8k, avg=26716.63, stdev=47919.33 lat (usec): min=710, max=2636.8k, avg=27041.52, stdev=47968.83 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 8029], 10.00th=[ 14353], | 20.00th=[ 17957], 30.00th=[ 20055], 40.00th=[ 21890], | 50.00th=[ 23725], 60.00th=[ 25560], 70.00th=[ 27919], | 80.00th=[ 31065], 90.00th=[ 36439], 95.00th=[ 41681], | 99.00th=[ 85459], 99.50th=[ 187696], 99.90th=[ 267387], | 99.95th=[ 683672], 99.99th=[2533360] bw ( KiB/s): min=31094, max=73690, per=100.00%, avg=39550.00, stdev=6037.87, samples=60 iops : min= 1943, max= 4605, avg=2471.42, stdev=377.35, samples=60 write: IOPS=606, BW=9696KiB/s (9929kB/s)(286MiB/30255msec) slat (usec): min=3, max=197952, avg=326.70, stdev=2108.96 clat (msec): min=6, max=2891, avg=221.38, stdev=171.61 lat (msec): min=19, max=2891, avg=221.71, stdev=171.60 clat percentiles (msec): | 1.00th=[ 28], 5.00th=[ 146], 10.00th=[ 159], 20.00th=[ 174], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 201], 60.00th=[ 209], | 70.00th=[ 215], 80.00th=[ 230], 90.00th=[ 264], 95.00th=[ 296], | 99.00th=[ 986], 99.50th=[ 1636], 99.90th=[ 2635], 99.95th=[ 2903], | 99.99th=[ 2903] bw ( KiB/s): min= 8114, max=15162, per=100.00%, avg=9816.83, stdev=1066.70, samples=60 iops : min= 507, max= 947, avg=613.05, stdev=66.66, samples=60 lat (usec) : 750=0.01%, 1000=0.26% lat (msec) : 2=1.43%, 4=1.70%, 10=1.01%, 20=19.29%, 50=54.45% lat (msec) : 100=1.78%, 250=17.30%, 500=2.38%, 750=0.08%, 1000=0.07% lat (msec) : 2000=0.17%, >=2000=0.06% cpu : usr=0.83%, sys=2.04%, ctx=10295, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73391,18335,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.9MiB/s (39.7MB/s), 37.9MiB/s-37.9MiB/s (39.7MB/s-39.7MB/s), io=1147MiB (1202MB), run=30255-30255msec WRITE: bw=9696KiB/s (9929kB/s), 9696KiB/s-9696KiB/s (9929kB/s-9929kB/s), io=286MiB (300MB), run=30255-30255msec Disk stats (read/write): vdl: ios=73389/18333, merge=0/0, ticks=81537/3577503, in_queue=3676760, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908019: Fri Jan 7 13:45:40 2022 read: IOPS=2286, BW=71.5MiB/s (74.9MB/s)(2144MiB/30001msec) clat (usec): min=30, max=26592, avg=64.61, stdev=403.92 lat (usec): min=30, max=26592, avg=64.74, stdev=403.92 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 72], | 99.00th=[ 87], 99.50th=[ 99], 99.90th=[ 478], 99.95th=[13042], | 99.99th=[16581] bw ( KiB/s): min=54912, max=109568, per=100.00%, avg=73176.65, stdev=9086.29, samples=60 iops : min= 1716, max= 3424, avg=2286.73, stdev=283.94, samples=60 write: IOPS=572, BW=17.9MiB/s (18.7MB/s)(536MiB/30001msec) clat (usec): min=275, max=46957, avg=1478.44, stdev=1649.80 lat (usec): min=277, max=46960, avg=1481.45, stdev=1649.79 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 611], 10.00th=[ 840], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1844], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[17957], 99.95th=[40633], | 99.99th=[44827] bw ( KiB/s): min=14912, max=28224, per=100.00%, avg=18310.13, stdev=1976.67, samples=60 iops : min= 466, max= 882, avg=572.18, stdev=61.77, samples=60 lat (usec) : 50=38.48%, 100=41.11%, 250=0.30%, 500=0.62%, 750=0.97% lat (usec) : 1000=1.68% lat (msec) : 2=15.42%, 4=1.04%, 10=0.25%, 20=0.11%, 50=0.02% cpu : usr=1.40%, sys=3.85%, ctx=85765, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68596,17165,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.5MiB/s (74.9MB/s), 71.5MiB/s-71.5MiB/s (74.9MB/s-74.9MB/s), io=2144MiB (2248MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=536MiB (562MB), run=30001-30001msec Disk stats (read/write): vdl: ios=68202/17070, merge=0/0, ticks=775/17629, in_queue=28142, util=93.94% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908676: Fri Jan 7 13:46:17 2022 read: IOPS=2303, BW=71.0MiB/s (75.5MB/s)(2159MiB/30002msec) slat (usec): min=5, max=269, avg=11.70, stdev= 5.77 clat (nsec): min=656, max=29781k, avg=52661.12, stdev=420355.80 lat (usec): min=26, max=29788, avg=64.51, stdev=420.40 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 31], 10.00th=[ 33], 20.00th=[ 35], | 30.00th=[ 37], 40.00th=[ 38], 50.00th=[ 39], 60.00th=[ 41], | 70.00th=[ 44], 80.00th=[ 47], 90.00th=[ 51], 95.00th=[ 56], | 99.00th=[ 70], 99.50th=[ 81], 99.90th=[ 375], 99.95th=[12649], | 99.99th=[17695] bw ( KiB/s): min=58752, max=110976, per=100.00%, avg=73707.37, stdev=8563.47, samples=60 iops : min= 1836, max= 3468, avg=2303.35, stdev=267.60, samples=60 write: IOPS=575, BW=17.0MiB/s (18.9MB/s)(540MiB/30002msec) slat (usec): min=9, max=147, avg=15.78, stdev= 6.63 clat (usec): min=260, max=45430, avg=1454.44, stdev=1593.28 lat (usec): min=274, max=45444, avg=1470.39, stdev=1593.36 clat percentiles (usec): | 1.00th=[ 338], 5.00th=[ 603], 10.00th=[ 840], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1467], 80.00th=[ 1582], 90.00th=[ 1811], 95.00th=[ 2073], | 99.00th=[ 9241], 99.50th=[ 9765], 99.90th=[17695], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=15488, max=28480, per=100.00%, avg=18430.55, stdev=1981.69, samples=60 iops : min= 484, max= 890, avg=575.95, stdev=61.92, samples=60 lat (nsec) : 750=0.01%, 1000=0.11% lat (usec) : 2=0.18%, 4=0.01%, 10=0.05%, 20=0.19%, 50=70.63% lat (usec) : 100=8.64%, 250=0.08%, 500=0.68%, 750=0.89%, 1000=1.63% lat (msec) : 2=15.67%, 4=0.90%, 10=0.21%, 20=0.11%, 50=0.02% cpu : usr=1.42%, sys=6.42%, ctx=85998, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69098,17279,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.0MiB/s (75.5MB/s), 71.0MiB/s-71.0MiB/s (75.5MB/s-75.5MB/s), io=2159MiB (2264MB), run=30002-30002msec WRITE: bw=17.0MiB/s (18.9MB/s), 17.0MiB/s-17.0MiB/s (18.9MB/s-18.9MB/s), io=540MiB (566MB), run=30002-30002msec Disk stats (read/write): vdl: ios=68634/17172, merge=0/0, ticks=790/17556, in_queue=27952, util=93.54% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909515: Fri Jan 7 13:46:54 2022 read: IOPS=2291, BW=71.6MiB/s (75.1MB/s)(2148MiB/30001msec) clat (usec): min=26, max=18938, avg=61.74, stdev=379.75 lat (usec): min=26, max=18938, avg=61.87, stdev=379.75 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 100], 99.90th=[ 486], 99.95th=[11600], | 99.99th=[16188] bw ( KiB/s): min=60741, max=117117, per=100.00%, avg=74207.95, stdev=8585.08, samples=60 iops : min= 1898, max= 3659, avg=2318.65, stdev=268.18, samples=60 write: IOPS=573, BW=17.9MiB/s (18.8MB/s)(537MiB/30001msec) clat (usec): min=269, max=46354, avg=1487.00, stdev=1635.93 lat (usec): min=272, max=46357, avg=1489.94, stdev=1635.92 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2073], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[17695], 99.95th=[38536], | 99.99th=[46400] bw ( KiB/s): min=14784, max=30056, per=100.00%, avg=18558.55, stdev=2072.43, samples=60 iops : min= 462, max= 939, avg=579.60, stdev=64.72, samples=60 lat (usec) : 50=45.73%, 100=33.87%, 250=0.29%, 500=0.60%, 750=0.87% lat (usec) : 1000=1.51% lat (msec) : 2=15.84%, 4=0.91%, 10=0.24%, 20=0.12%, 50=0.02% cpu : usr=1.48%, sys=3.72%, ctx=85948, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68739,17191,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=71.6MiB/s (75.1MB/s), 71.6MiB/s-71.6MiB/s (75.1MB/s-75.1MB/s), io=2148MiB (2252MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=537MiB (563MB), run=30001-30001msec Disk stats (read/write): vdl: ios=68688/17183, merge=0/0, ticks=726/17947, in_queue=28334, util=94.16% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910181: Fri Jan 7 13:47:31 2022 read: IOPS=2336, BW=73.0MiB/s (76.6MB/s)(2192MiB/30017msec) slat (usec): min=3, max=239, avg=11.38, stdev= 4.31 clat (nsec): min=741, max=2503.7M, avg=1132841.26, stdev=45549172.67 lat (usec): min=26, max=2503.7k, avg=1144.38, stdev=45549.16 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 33], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 37], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 48], 90.00th=[ 53], 95.00th=[ 60], | 99.00th=[ 93], 99.50th=[ 126], 99.90th=[ 603], | 99.95th=[ 826278], 99.99th=[2499806] bw ( KiB/s): min=62525, max=126080, per=100.00%, avg=74831.83, stdev=9047.27, samples=60 iops : min= 1953, max= 3940, avg=2338.45, stdev=282.77, samples=60 write: IOPS=585, BW=18.3MiB/s (19.2MB/s)(549MiB/30017msec) slat (usec): min=5, max=118, avg=15.55, stdev= 4.78 clat (usec): min=510, max=81166, avg=12493.60, stdev=6023.15 lat (usec): min=519, max=81183, avg=12509.31, stdev=6023.41 clat percentiles (usec): | 1.00th=[ 2008], 5.00th=[ 6194], 10.00th=[ 7439], 20.00th=[ 9503], | 30.00th=[10421], 40.00th=[11338], 50.00th=[11994], 60.00th=[12649], | 70.00th=[13435], 80.00th=[14484], 90.00th=[16581], 95.00th=[20317], | 99.00th=[30540], 99.50th=[62129], 99.90th=[72877], 99.95th=[72877], | 99.99th=[79168] bw ( KiB/s): min=16256, max=31680, per=100.00%, avg=18727.20, stdev=2021.29, samples=60 iops : min= 508, max= 990, avg=585.22, stdev=63.17, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.06%, 50=68.15% lat (usec) : 100=11.01%, 250=0.54%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.19%, 4=0.16%, 10=4.54%, 20=14.04%, 50=0.95% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.02% cpu : usr=1.43%, sys=6.23%, ctx=86544, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70144,17564,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=73.0MiB/s (76.6MB/s), 73.0MiB/s-73.0MiB/s (76.6MB/s-76.6MB/s), io=2192MiB (2298MB), run=30017-30017msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=549MiB (576MB), run=30017-30017msec Disk stats (read/write): vdl: ios=69816/17453, merge=0/0, ticks=76454/208655, in_queue=296291, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910979: Fri Jan 7 13:48:08 2022 read: IOPS=2294, BW=71.7MiB/s (75.2MB/s)(2151MiB/30002msec) clat (usec): min=26, max=24198, avg=63.17, stdev=410.36 lat (usec): min=26, max=24199, avg=63.30, stdev=410.36 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 64], 95.00th=[ 70], | 99.00th=[ 89], 99.50th=[ 102], 99.90th=[ 562], 99.95th=[12780], | 99.99th=[17171] bw ( KiB/s): min=59264, max=121344, per=100.00%, avg=73426.32, stdev=9358.57, samples=60 iops : min= 1852, max= 3792, avg=2294.55, stdev=292.47, samples=60 write: IOPS=573, BW=17.9MiB/s (18.8MB/s)(538MiB/30002msec) clat (usec): min=275, max=44922, avg=1478.70, stdev=1655.97 lat (usec): min=278, max=44925, avg=1481.69, stdev=1655.97 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 619], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1303], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1811], 95.00th=[ 2089], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35390], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=14592, max=31360, per=100.00%, avg=18362.80, stdev=2232.10, samples=60 iops : min= 456, max= 980, avg=573.83, stdev=69.75, samples=60 lat (usec) : 50=43.70%, 100=35.86%, 250=0.32%, 500=0.63%, 750=0.85% lat (usec) : 1000=1.60% lat (msec) : 2=15.77%, 4=0.92%, 10=0.22%, 20=0.11%, 50=0.03% cpu : usr=1.42%, sys=3.96%, ctx=86055, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68833,17215,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=71.7MiB/s (75.2MB/s), 71.7MiB/s-71.7MiB/s (75.2MB/s-75.2MB/s), io=2151MiB (2256MB), run=30002-30002msec WRITE: bw=17.9MiB/s (18.8MB/s), 17.9MiB/s-17.9MiB/s (18.8MB/s-18.8MB/s), io=538MiB (564MB), run=30002-30002msec Disk stats (read/write): vdl: ios=68259/17080, merge=0/0, ticks=776/17715, in_queue=28104, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911960: Fri Jan 7 13:48:45 2022 read: IOPS=2342, BW=73.2MiB/s (76.8MB/s)(2199MiB/30037msec) slat (usec): min=2, max=260, avg=10.84, stdev= 5.17 clat (nsec): min=701, max=2518.3M, avg=1081837.88, stdev=44123809.98 lat (usec): min=25, max=2518.3k, avg=1092.83, stdev=44123.81 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 47], 90.00th=[ 54], 95.00th=[ 63], | 99.00th=[ 202], 99.50th=[ 420], 99.90th=[ 1156], | 99.95th=[ 734004], 99.99th=[2499806] bw ( KiB/s): min=54400, max=149312, per=100.00%, avg=75059.88, stdev=12030.37, samples=60 iops : min= 1700, max= 4666, avg=2345.60, stdev=375.94, samples=60 write: IOPS=586, BW=18.3MiB/s (19.2MB/s)(550MiB/30037msec) slat (usec): min=5, max=280, avg=16.57, stdev= 7.57 clat (msec): min=8, max=2024, avg=80.90, stdev=63.87 lat (msec): min=8, max=2024, avg=80.92, stdev=63.87 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 57], 10.00th=[ 62], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 83], 90.00th=[ 92], 95.00th=[ 122], | 99.00th=[ 253], 99.50th=[ 558], 99.90th=[ 1028], 99.95th=[ 1116], | 99.99th=[ 1687] bw ( KiB/s): min=14784, max=35264, per=99.87%, avg=18731.73, stdev=2586.52, samples=60 iops : min= 462, max= 1102, avg=585.35, stdev=80.82, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.09%, 4=0.01%, 10=0.04%, 20=0.15%, 50=68.25% lat (usec) : 100=9.94%, 250=0.78%, 500=0.36%, 750=0.12%, 1000=0.09% lat (msec) : 2=0.04%, 10=0.03%, 20=0.26%, 50=0.20%, 100=18.03% lat (msec) : 250=1.31%, 500=0.10%, 750=0.06%, 1000=0.04%, 2000=0.03% lat (msec) : >=2000=0.02% cpu : usr=1.53%, sys=5.99%, ctx=85437, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=70358,17606,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=73.2MiB/s (76.8MB/s), 73.2MiB/s-73.2MiB/s (76.8MB/s-76.8MB/s), io=2199MiB (2305MB), run=30037-30037msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=550MiB (577MB), run=30037-30037msec Disk stats (read/write): vdl: ios=70357/17606, merge=0/0, ticks=72785/1414437, in_queue=1498650, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912592: Fri Jan 7 13:49:22 2022 read: IOPS=2303, BW=71.0MiB/s (75.5MB/s)(2160MiB/30001msec) clat (usec): min=27, max=19834, avg=61.44, stdev=389.63 lat (usec): min=27, max=19835, avg=61.58, stdev=389.63 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 86], 99.50th=[ 100], 99.90th=[ 594], 99.95th=[12256], | 99.99th=[16712] bw ( KiB/s): min=55104, max=111232, per=100.00%, avg=73725.45, stdev=8943.11, samples=60 iops : min= 1722, max= 3476, avg=2303.92, stdev=279.47, samples=60 write: IOPS=576, BW=18.0MiB/s (18.9MB/s)(540MiB/30001msec) clat (usec): min=266, max=56306, avg=1478.89, stdev=1683.65 lat (usec): min=273, max=56309, avg=1481.85, stdev=1683.62 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 627], 10.00th=[ 873], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2073], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[42206], | 99.99th=[43779] bw ( KiB/s): min=14144, max=28480, per=100.00%, avg=18433.72, stdev=1957.49, samples=60 iops : min= 442, max= 890, avg=576.05, stdev=61.17, samples=60 lat (usec) : 50=47.87%, 100=31.73%, 250=0.27%, 500=0.65%, 750=0.81% lat (usec) : 1000=1.50% lat (msec) : 2=15.90%, 4=0.92%, 10=0.22%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.36%, sys=3.88%, ctx=86402, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69115,17282,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=71.0MiB/s (75.5MB/s), 71.0MiB/s-71.0MiB/s (75.5MB/s-75.5MB/s), io=2160MiB (2265MB), run=30001-30001msec WRITE: bw=18.0MiB/s (18.9MB/s), 18.0MiB/s-18.0MiB/s (18.9MB/s-18.9MB/s), io=540MiB (566MB), run=30001-30001msec Disk stats (read/write): vdl: ios=68528/17147, merge=0/0, ticks=740/17861, in_queue=28132, util=94.17% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913424: Fri Jan 7 13:49:59 2022 read: IOPS=2339, BW=73.1MiB/s (76.7MB/s)(2206MiB/30183msec) slat (usec): min=2, max=208, avg=10.85, stdev= 4.84 clat (nsec): min=760, max=2581.8M, avg=1195466.79, stdev=46504036.24 lat (usec): min=26, max=2581.8k, avg=1206.48, stdev=46504.06 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 44], 70.00th=[ 47], | 80.00th=[ 52], 90.00th=[ 62], 95.00th=[ 92], | 99.00th=[ 1090], 99.50th=[ 1745], 99.90th=[ 2868], | 99.95th=[1061159], 99.99th=[2499806] bw ( KiB/s): min=60544, max=150080, per=100.00%, avg=75318.65, stdev=11936.41, samples=60 iops : min= 1892, max= 4690, avg=2353.67, stdev=373.01, samples=60 write: IOPS=584, BW=18.3MiB/s (19.2MB/s)(552MiB/30183msec) slat (usec): min=5, max=252, avg=17.30, stdev= 6.98 clat (msec): min=11, max=2004, avg=166.08, stdev=89.37 lat (msec): min=11, max=2004, avg=166.09, stdev=89.37 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 121], 10.00th=[ 136], 20.00th=[ 144], | 30.00th=[ 148], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 161], | 70.00th=[ 167], 80.00th=[ 174], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 388], 99.50th=[ 894], 99.90th=[ 1385], 99.95th=[ 1636], | 99.99th=[ 1972] bw ( KiB/s): min=16192, max=32384, per=100.00%, avg=18734.18, stdev=2121.87, samples=60 iops : min= 506, max= 1012, avg=585.43, stdev=66.31, samples=60 lat (nsec) : 1000=0.04% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.07%, 50=61.75% lat (usec) : 100=14.33%, 250=1.44%, 500=0.74%, 750=0.39%, 1000=0.27% lat (msec) : 2=0.60%, 4=0.23%, 20=0.08%, 50=0.32%, 100=0.22% lat (msec) : 250=18.90%, 500=0.33%, 750=0.06%, 1000=0.04%, 2000=0.10% lat (msec) : >=2000=0.02% cpu : usr=1.55%, sys=6.07%, ctx=83156, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70606,17656,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=73.1MiB/s (76.7MB/s), 73.1MiB/s-73.1MiB/s (76.7MB/s-76.7MB/s), io=2206MiB (2314MB), run=30183-30183msec WRITE: bw=18.3MiB/s (19.2MB/s), 18.3MiB/s-18.3MiB/s (19.2MB/s-19.2MB/s), io=552MiB (579MB), run=30183-30183msec Disk stats (read/write): vdl: ios=70602/17575, merge=0/0, ticks=75583/2901102, in_queue=2999999, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914736: Fri Jan 7 13:50:36 2022 read: IOPS=2305, BW=72.0MiB/s (75.5MB/s)(2161MiB/30001msec) clat (usec): min=28, max=61121, avg=63.85, stdev=525.95 lat (usec): min=28, max=61121, avg=63.99, stdev=525.95 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 83], 99.50th=[ 94], 99.90th=[ 441], 99.95th=[13435], | 99.99th=[17957] bw ( KiB/s): min=58048, max=120064, per=100.00%, avg=73763.20, stdev=9344.43, samples=60 iops : min= 1814, max= 3752, avg=2305.10, stdev=292.01, samples=60 write: IOPS=576, BW=18.0MiB/s (18.9MB/s)(541MiB/30001msec) clat (usec): min=272, max=43746, avg=1467.88, stdev=1558.21 lat (usec): min=274, max=43749, avg=1470.87, stdev=1558.21 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 644], 10.00th=[ 881], 20.00th=[ 1106], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1811], 95.00th=[ 2057], | 99.00th=[ 9110], 99.50th=[ 9765], 99.90th=[23987], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min=15104, max=30848, per=100.00%, avg=18449.07, stdev=2180.94, samples=60 iops : min= 472, max= 964, avg=576.53, stdev=68.15, samples=60 lat (usec) : 50=46.76%, 100=32.92%, 250=0.22%, 500=0.64%, 750=0.78% lat (usec) : 1000=1.48% lat (msec) : 2=15.98%, 4=0.89%, 10=0.21%, 20=0.10%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.33%, sys=3.96%, ctx=86458, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69153,17297,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=72.0MiB/s (75.5MB/s), 72.0MiB/s-72.0MiB/s (75.5MB/s-75.5MB/s), io=2161MiB (2266MB), run=30001-30001msec WRITE: bw=18.0MiB/s (18.9MB/s), 18.0MiB/s-18.0MiB/s (18.9MB/s-18.9MB/s), io=541MiB (567MB), run=30001-30001msec Disk stats (read/write): vdl: ios=69129/17282, merge=0/0, ticks=907/17738, in_queue=28360, util=94.29% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915356: Fri Jan 7 13:51:13 2022 read: IOPS=2330, BW=72.8MiB/s (76.4MB/s)(2202MiB/30243msec) slat (usec): min=2, max=188223, avg=333.97, stdev=1991.73 clat (usec): min=694, max=2562.2k, avg=28048.50, stdev=54737.70 lat (usec): min=698, max=2562.2k, avg=28382.62, stdev=54772.59 clat percentiles (usec): | 1.00th=[ 1385], 5.00th=[ 6783], 10.00th=[ 15139], | 20.00th=[ 18744], 30.00th=[ 21103], 40.00th=[ 23200], | 50.00th=[ 25035], 60.00th=[ 26870], 70.00th=[ 29230], | 80.00th=[ 32113], 90.00th=[ 37487], 95.00th=[ 42730], | 99.00th=[ 88605], 99.50th=[ 189793], 99.90th=[ 258999], | 99.95th=[1400898], 99.99th=[2533360] bw ( KiB/s): min=59264, max=147776, per=100.00%, avg=75110.03, stdev=11495.32, samples=60 iops : min= 1852, max= 4618, avg=2347.17, stdev=359.23, samples=60 write: IOPS=583, BW=18.2MiB/s (19.1MB/s)(551MiB/30243msec) slat (usec): min=4, max=186201, avg=356.41, stdev=2149.33 clat (msec): min=18, max=3959, avg=229.12, stdev=181.03 lat (msec): min=21, max=3959, avg=229.48, stdev=181.01 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 153], 10.00th=[ 167], 20.00th=[ 184], | 30.00th=[ 197], 40.00th=[ 205], 50.00th=[ 211], 60.00th=[ 218], | 70.00th=[ 226], 80.00th=[ 239], 90.00th=[ 264], 95.00th=[ 300], | 99.00th=[ 936], 99.50th=[ 1301], 99.90th=[ 3306], 99.95th=[ 3306], | 99.99th=[ 3977] bw ( KiB/s): min=15616, max=30464, per=100.00%, avg=18668.97, stdev=1996.71, samples=60 iops : min= 488, max= 952, avg=583.40, stdev=62.40, samples=60 lat (usec) : 750=0.01%, 1000=0.16% lat (msec) : 2=1.49%, 4=1.65%, 10=1.29%, 20=14.87%, 50=58.40% lat (msec) : 100=2.01%, 250=17.26%, 500=2.46%, 750=0.08%, 1000=0.10% lat (msec) : 2000=0.16%, >=2000=0.07% cpu : usr=1.13%, sys=2.09%, ctx=9869, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70476,17636,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=72.8MiB/s (76.4MB/s), 72.8MiB/s-72.8MiB/s (76.4MB/s-76.4MB/s), io=2202MiB (2309MB), run=30243-30243msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=551MiB (578MB), run=30243-30243msec Disk stats (read/write): vdl: ios=70474/17530, merge=0/0, ticks=96745/3538726, in_queue=3668079, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916227: Fri Jan 7 13:51:50 2022 read: IOPS=2114, BW=132MiB/s (139MB/s)(3967MiB/30009msec) clat (usec): min=33, max=66594, avg=75.48, stdev=507.35 lat (usec): min=33, max=66594, avg=75.62, stdev=507.35 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 48], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 77], 95.00th=[ 85], | 99.00th=[ 114], 99.50th=[ 137], 99.90th=[ 1020], 99.95th=[13829], | 99.99th=[17695] bw ( KiB/s): min=99072, max=209280, per=100.00%, avg=135401.43, stdev=16988.85, samples=60 iops : min= 1548, max= 3270, avg=2115.63, stdev=265.43, samples=60 write: IOPS=531, BW=33.2MiB/s (34.8MB/s)(997MiB/30009msec) clat (usec): min=360, max=44430, avg=1566.25, stdev=1594.87 lat (usec): min=366, max=44438, avg=1572.76, stdev=1594.83 clat percentiles (usec): | 1.00th=[ 441], 5.00th=[ 660], 10.00th=[ 889], 20.00th=[ 1156], | 30.00th=[ 1270], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1483], | 70.00th=[ 1582], 80.00th=[ 1696], 90.00th=[ 1942], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[18482], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min=27008, max=54016, per=100.00%, avg=34019.32, stdev=3610.23, samples=60 iops : min= 422, max= 844, avg=531.55, stdev=56.41, samples=60 lat (usec) : 50=12.00%, 100=66.41%, 250=1.33%, 500=0.51%, 750=0.96% lat (usec) : 1000=1.24% lat (msec) : 2=15.75%, 4=1.43%, 10=0.21%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.69%, sys=3.81%, ctx=79419, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63467,15947,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3967MiB (4159MB), run=30009-30009msec WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=997MiB (1045MB), run=30009-30009msec Disk stats (read/write): vdl: ios=63032/15847, merge=0/0, ticks=830/17082, in_queue=28082, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916849: Fri Jan 7 13:52:27 2022 read: IOPS=2120, BW=133MiB/s (139MB/s)(3977MiB/30010msec) slat (usec): min=6, max=123, avg=13.75, stdev= 4.54 clat (nsec): min=739, max=29352k, avg=63164.39, stdev=438613.31 lat (usec): min=34, max=29367, avg=77.08, stdev=438.68 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 85], 99.50th=[ 100], 99.90th=[ 750], 99.95th=[13566], | 99.99th=[17171] bw ( KiB/s): min=105856, max=215808, per=100.00%, avg=135754.18, stdev=17107.73, samples=60 iops : min= 1654, max= 3372, avg=2121.13, stdev=267.28, samples=60 write: IOPS=533, BW=33.3MiB/s (34.9MB/s)(1000MiB/30010msec) slat (usec): min=11, max=172, avg=20.60, stdev= 6.13 clat (usec): min=350, max=44625, avg=1540.98, stdev=1644.40 lat (usec): min=366, max=44642, avg=1561.76, stdev=1644.38 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 635], 10.00th=[ 848], 20.00th=[ 1123], | 30.00th=[ 1254], 40.00th=[ 1336], 50.00th=[ 1385], 60.00th=[ 1450], | 70.00th=[ 1549], 80.00th=[ 1680], 90.00th=[ 1909], 95.00th=[ 2180], | 99.00th=[ 9372], 99.50th=[ 9896], 99.90th=[36963], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=26496, max=56064, per=100.00%, avg=34127.28, stdev=4034.85, samples=60 iops : min= 414, max= 876, avg=533.23, stdev=63.04, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.03%, 10=0.01%, 20=0.02%, 50=47.42%, 100=32.03% lat (usec) : 250=0.27%, 500=0.49%, 750=1.10%, 1000=1.40% lat (msec) : 2=15.57%, 4=1.28%, 10=0.22%, 20=0.13%, 50=0.02% cpu : usr=1.65%, sys=6.39%, ctx=79603, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63631,15996,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=3977MiB (4170MB), run=30010-30010msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1000MiB (1048MB), run=30010-30010msec Disk stats (read/write): vdl: ios=63333/15907, merge=0/0, ticks=778/16919, in_queue=27801, util=92.96% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917795: Fri Jan 7 13:53:04 2022 read: IOPS=2120, BW=133MiB/s (139MB/s)(3977MiB/30002msec) clat (usec): min=30, max=24896, avg=69.58, stdev=437.12 lat (usec): min=30, max=24896, avg=69.73, stdev=437.12 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 78], | 99.00th=[ 102], 99.50th=[ 119], 99.90th=[ 1012], 99.95th=[13960], | 99.99th=[18220] bw ( KiB/s): min=100736, max=209408, per=100.00%, avg=135747.67, stdev=16599.89, samples=60 iops : min= 1574, max= 3272, avg=2121.05, stdev=259.38, samples=60 write: IOPS=533, BW=33.3MiB/s (34.9MB/s)(1000MiB/30002msec) clat (usec): min=357, max=44797, avg=1583.67, stdev=1626.68 lat (usec): min=363, max=44803, avg=1590.04, stdev=1626.65 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 685], 10.00th=[ 938], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1713], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[38011], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=27520, max=54144, per=100.00%, avg=34125.75, stdev=3612.88, samples=60 iops : min= 430, max= 846, avg=533.20, stdev=56.47, samples=60 lat (usec) : 50=27.05%, 100=52.01%, 250=0.72%, 500=0.48%, 750=0.80% lat (usec) : 1000=1.13% lat (msec) : 2=15.99%, 4=1.45%, 10=0.22%, 20=0.12%, 50=0.02% cpu : usr=1.70%, sys=3.81%, ctx=79633, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63631,15996,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=3977MiB (4170MB), run=30002-30002msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1000MiB (1048MB), run=30002-30002msec Disk stats (read/write): vdl: ios=63249/15890, merge=0/0, ticks=781/17413, in_queue=28002, util=93.73% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918740: Fri Jan 7 13:53:41 2022 read: IOPS=2166, BW=135MiB/s (142MB/s)(4066MiB/30029msec) slat (usec): min=3, max=201, avg=12.10, stdev= 4.45 clat (nsec): min=646, max=2492.1M, avg=1413937.39, stdev=52834331.04 lat (usec): min=28, max=2492.1k, avg=1426.20, stdev=52834.32 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 34], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 38], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 49], 90.00th=[ 55], 95.00th=[ 61], | 99.00th=[ 83], 99.50th=[ 109], 99.90th=[ 627], | 99.95th=[1333789], 99.99th=[2499806] bw ( KiB/s): min=104704, max=252800, per=100.00%, avg=138795.18, stdev=19329.38, samples=60 iops : min= 1636, max= 3950, avg=2168.67, stdev=302.02, samples=60 write: IOPS=543, BW=33.0MiB/s (35.6MB/s)(1020MiB/30029msec) slat (usec): min=8, max=181, avg=20.35, stdev= 7.14 clat (usec): min=553, max=67608, avg=12689.03, stdev=6144.74 lat (usec): min=577, max=67627, avg=12709.55, stdev=6145.50 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 4424], 10.00th=[ 6325], 20.00th=[ 8717], | 30.00th=[10552], 40.00th=[11863], 50.00th=[12780], 60.00th=[13566], | 70.00th=[14353], 80.00th=[15270], 90.00th=[16909], 95.00th=[21103], | 99.00th=[29492], 99.50th=[60556], 99.90th=[65274], 99.95th=[65799], | 99.99th=[67634] bw ( KiB/s): min=27776, max=63872, per=100.00%, avg=34789.45, stdev=4467.82, samples=60 iops : min= 434, max= 998, avg=543.58, stdev=69.81, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.04%, 50=65.56% lat (usec) : 100=13.80%, 250=0.33%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.37%, 4=0.49%, 10=4.30%, 20=13.62%, 50=1.11% lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.03% cpu : usr=1.56%, sys=6.01%, ctx=80572, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=65061,16314,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=4066MiB (4264MB), run=30029-30029msec WRITE: bw=33.0MiB/s (35.6MB/s), 33.0MiB/s-33.0MiB/s (35.6MB/s-35.6MB/s), io=1020MiB (1069MB), run=30029-30029msec Disk stats (read/write): vdl: ios=65060/16314, merge=0/0, ticks=88678/198645, in_queue=298248, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919320: Fri Jan 7 13:54:18 2022 read: IOPS=2120, BW=133MiB/s (139MB/s)(3977MiB/30006msec) clat (usec): min=32, max=25178, avg=68.89, stdev=462.13 lat (usec): min=32, max=25179, avg=69.03, stdev=462.14 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 92], 99.50th=[ 106], 99.90th=[ 1352], 99.95th=[14353], | 99.99th=[18220] bw ( KiB/s): min=104550, max=212777, per=100.00%, avg=136789.17, stdev=17135.58, samples=60 iops : min= 1633, max= 3324, avg=2136.83, stdev=267.75, samples=60 write: IOPS=533, BW=33.3MiB/s (34.9MB/s)(1000MiB/30006msec) clat (usec): min=346, max=46359, avg=1587.49, stdev=1634.25 lat (usec): min=351, max=46366, avg=1593.48, stdev=1634.22 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 685], 10.00th=[ 955], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1713], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min=27096, max=55150, per=100.00%, avg=34386.87, stdev=3711.94, samples=60 iops : min= 423, max= 861, avg=536.93, stdev=57.97, samples=60 lat (usec) : 50=31.71%, 100=47.69%, 250=0.38%, 500=0.47%, 750=0.79% lat (usec) : 1000=1.00% lat (msec) : 2=16.11%, 4=1.46%, 10=0.22%, 20=0.13%, 50=0.03% cpu : usr=1.48%, sys=3.79%, ctx=79631, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63631,15996,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=3977MiB (4170MB), run=30006-30006msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=1000MiB (1048MB), run=30006-30006msec Disk stats (read/write): vdl: ios=63630/15996, merge=0/0, ticks=836/17675, in_queue=28307, util=94.05% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920167: Fri Jan 7 13:54:55 2022 read: IOPS=2153, BW=135MiB/s (141MB/s)(4054MiB/30113msec) slat (usec): min=3, max=174, avg=11.61, stdev= 4.35 clat (nsec): min=774, max=2574.9M, avg=1380735.43, stdev=51369558.74 lat (usec): min=29, max=2574.0k, avg=1392.49, stdev=51369.55 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 33], 10.00th=[ 34], | 20.00th=[ 36], 30.00th=[ 38], 40.00th=[ 39], | 50.00th=[ 41], 60.00th=[ 45], 70.00th=[ 49], | 80.00th=[ 54], 90.00th=[ 62], 95.00th=[ 71], | 99.00th=[ 302], 99.50th=[ 619], 99.90th=[ 1647], | 99.95th=[1300235], 99.99th=[2499806] bw ( KiB/s): min=96896, max=263168, per=100.00%, avg=138367.00, stdev=21328.63, samples=60 iops : min= 1514, max= 4112, avg=2161.98, stdev=333.26, samples=60 write: IOPS=540, BW=33.8MiB/s (35.4MB/s)(1017MiB/30113msec) slat (usec): min=9, max=140, avg=21.79, stdev= 8.19 clat (msec): min=8, max=2146, avg=86.92, stdev=70.86 lat (msec): min=8, max=2146, avg=86.94, stdev=70.86 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 74], 40.00th=[ 78], 50.00th=[ 80], 60.00th=[ 82], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 100], 95.00th=[ 130], | 99.00th=[ 197], 99.50th=[ 567], 99.90th=[ 1133], 99.95th=[ 1418], | 99.99th=[ 1770] bw ( KiB/s): min=27904, max=61056, per=100.00%, avg=34623.10, stdev=4283.34, samples=60 iops : min= 436, max= 954, avg=540.98, stdev=66.93, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.04%, 50=57.96% lat (usec) : 100=20.10%, 250=0.82%, 500=0.44%, 750=0.16%, 1000=0.09% lat (msec) : 2=0.17%, 10=0.02%, 20=0.31%, 50=0.15%, 100=17.71% lat (msec) : 250=1.71%, 500=0.05%, 750=0.06%, 1000=0.04%, 2000=0.06% lat (msec) : >=2000=0.02% cpu : usr=1.68%, sys=5.64%, ctx=78788, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=64861,16275,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4054MiB (4251MB), run=30113-30113msec WRITE: bw=33.8MiB/s (35.4MB/s), 33.8MiB/s-33.8MiB/s (35.4MB/s-35.4MB/s), io=1017MiB (1067MB), run=30113-30113msec Disk stats (read/write): vdl: ios=64857/16249, merge=0/0, ticks=82924/1401101, in_queue=1499875, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921319: Fri Jan 7 13:55:32 2022 read: IOPS=2115, BW=132MiB/s (139MB/s)(3967MiB/30001msec) clat (usec): min=32, max=24670, avg=69.86, stdev=419.36 lat (usec): min=32, max=24670, avg=70.01, stdev=419.36 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 96], 99.50th=[ 111], 99.90th=[ 848], 99.95th=[12649], | 99.99th=[16909] bw ( KiB/s): min=94336, max=193408, per=100.00%, avg=135411.88, stdev=16564.88, samples=60 iops : min= 1474, max= 3022, avg=2115.80, stdev=258.80, samples=60 write: IOPS=531, BW=33.2MiB/s (34.8MB/s)(997MiB/30001msec) clat (usec): min=356, max=44991, avg=1588.23, stdev=1666.98 lat (usec): min=362, max=44997, avg=1594.33, stdev=1666.92 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 668], 10.00th=[ 922], 20.00th=[ 1172], | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2245], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[37487], 99.95th=[40109], | 99.99th=[44827] bw ( KiB/s): min=28544, max=50432, per=100.00%, avg=34023.62, stdev=3279.54, samples=60 iops : min= 446, max= 788, avg=531.62, stdev=51.24, samples=60 lat (usec) : 50=22.71%, 100=56.55%, 250=0.54%, 500=0.50%, 750=0.86% lat (usec) : 1000=1.14% lat (msec) : 2=15.85%, 4=1.48%, 10=0.23%, 20=0.12%, 50=0.03% cpu : usr=1.63%, sys=3.87%, ctx=79428, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63472,15949,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3967MiB (4160MB), run=30001-30001msec WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=997MiB (1045MB), run=30001-30001msec Disk stats (read/write): vdl: ios=62973/15834, merge=0/0, ticks=749/17405, in_queue=28076, util=94.03% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922080: Fri Jan 7 13:56:10 2022 read: IOPS=2149, BW=134MiB/s (141MB/s)(4057MiB/30190msec) slat (usec): min=2, max=230, avg=11.29, stdev= 6.40 clat (nsec): min=742, max=2558.4M, avg=1322915.72, stdev=50238689.42 lat (usec): min=28, max=2558.4k, avg=1334.37, stdev=50238.65 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 47], 70.00th=[ 52], | 80.00th=[ 58], 90.00th=[ 69], 95.00th=[ 113], | 99.00th=[ 1467], 99.50th=[ 2376], 99.90th=[ 3687], | 99.95th=[ 943719], 99.99th=[2499806] bw ( KiB/s): min=105600, max=288448, per=100.00%, avg=138482.75, stdev=23113.81, samples=60 iops : min= 1650, max= 4507, avg=2163.78, stdev=361.15, samples=60 write: IOPS=539, BW=33.7MiB/s (35.4MB/s)(1018MiB/30190msec) slat (usec): min=9, max=324, avg=22.65, stdev=11.58 clat (msec): min=6, max=3201, avg=179.96, stdev=149.66 lat (msec): min=7, max=3201, avg=179.98, stdev=149.66 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 116], 10.00th=[ 124], 20.00th=[ 142], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 169], 60.00th=[ 174], | 70.00th=[ 180], 80.00th=[ 186], 90.00th=[ 211], 95.00th=[ 239], | 99.00th=[ 550], 99.50th=[ 1183], 99.90th=[ 2433], 99.95th=[ 3138], | 99.99th=[ 3205] bw ( KiB/s): min=28288, max=61563, per=100.00%, avg=34549.37, stdev=4246.47, samples=60 iops : min= 442, max= 961, avg=539.82, stdev=66.25, samples=60 lat (nsec) : 750=0.01%, 1000=0.07% lat (usec) : 2=0.12%, 4=0.01%, 10=0.03%, 20=0.07%, 50=53.47% lat (usec) : 100=21.87%, 250=1.61%, 500=0.74%, 750=0.49%, 1000=0.32% lat (msec) : 2=0.62%, 4=0.46%, 10=0.01%, 20=0.03%, 50=0.34% lat (msec) : 100=0.23%, 250=18.69%, 500=0.58%, 750=0.08%, 1000=0.03% lat (msec) : 2000=0.08%, >=2000=0.08% cpu : usr=1.80%, sys=5.58%, ctx=76287, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64908,16290,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=134MiB/s (141MB/s), 134MiB/s-134MiB/s (141MB/s-141MB/s), io=4057MiB (4254MB), run=30190-30190msec WRITE: bw=33.7MiB/s (35.4MB/s), 33.7MiB/s-33.7MiB/s (35.4MB/s-35.4MB/s), io=1018MiB (1068MB), run=30190-30190msec Disk stats (read/write): vdl: ios=64905/16212, merge=0/0, ticks=78484/2899966, in_queue=3000912, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923040: Fri Jan 7 13:56:46 2022 read: IOPS=2116, BW=132MiB/s (139MB/s)(3973MiB/30032msec) clat (usec): min=32, max=61754, avg=67.85, stdev=505.77 lat (usec): min=32, max=61754, avg=67.99, stdev=505.77 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 87], 99.50th=[ 99], 99.90th=[ 269], 99.95th=[13698], | 99.99th=[17695] bw ( KiB/s): min=104704, max=211712, per=100.00%, avg=135615.08, stdev=16945.45, samples=60 iops : min= 1636, max= 3308, avg=2118.95, stdev=264.76, samples=60 write: IOPS=532, BW=33.3MiB/s (34.9MB/s)(999MiB/30032msec) clat (usec): min=351, max=45163, avg=1595.19, stdev=1616.15 lat (usec): min=356, max=45169, avg=1601.21, stdev=1616.11 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 701], 10.00th=[ 971], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1975], 95.00th=[ 2245], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[35914], 99.95th=[39584], | 99.99th=[44827] bw ( KiB/s): min=28544, max=54912, per=100.00%, avg=34091.93, stdev=3737.94, samples=60 iops : min= 446, max= 858, avg=532.67, stdev=58.42, samples=60 lat (usec) : 50=34.37%, 100=45.16%, 250=0.29%, 500=0.45%, 750=0.73% lat (usec) : 1000=1.00% lat (msec) : 2=16.05%, 4=1.59%, 10=0.21%, 20=0.13%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.53%, sys=3.62%, ctx=79547, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63563,15980,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3973MiB (4166MB), run=30032-30032msec WRITE: bw=33.3MiB/s (34.9MB/s), 33.3MiB/s-33.3MiB/s (34.9MB/s-34.9MB/s), io=999MiB (1047MB), run=30032-30032msec Disk stats (read/write): vdl: ios=63563/15979, merge=0/0, ticks=854/17756, in_queue=28334, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923716: Fri Jan 7 13:57:23 2022 read: IOPS=2185, BW=137MiB/s (143MB/s)(4128MiB/30217msec) slat (usec): min=2, max=235376, avg=354.31, stdev=2107.84 clat (usec): min=970, max=2580.8k, avg=29797.70, stdev=55142.53 lat (usec): min=973, max=2586.0k, avg=30152.16, stdev=55199.90 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 16], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 27], 60.00th=[ 29], | 70.00th=[ 32], 80.00th=[ 35], 90.00th=[ 40], 95.00th=[ 45], | 99.00th=[ 86], 99.50th=[ 213], 99.90th=[ 264], 99.95th=[ 1234], | 99.99th=[ 2534] bw ( KiB/s): min=103808, max=301312, per=100.00%, avg=140773.45, stdev=24710.44, samples=60 iops : min= 1622, max= 4708, avg=2199.58, stdev=386.10, samples=60 write: IOPS=547, BW=34.2MiB/s (35.9MB/s)(1034MiB/30217msec) slat (usec): min=6, max=196193, avg=388.64, stdev=2701.63 clat (msec): min=9, max=3188, avg=244.45, stdev=141.85 lat (msec): min=16, max=3188, avg=244.83, stdev=141.84 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 165], 10.00th=[ 184], 20.00th=[ 209], | 30.00th=[ 220], 40.00th=[ 226], 50.00th=[ 232], 60.00th=[ 239], | 70.00th=[ 247], 80.00th=[ 259], 90.00th=[ 288], 95.00th=[ 313], | 99.00th=[ 693], 99.50th=[ 1267], 99.90th=[ 1787], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min=29440, max=62720, per=99.95%, avg=35021.00, stdev=4317.78, samples=60 iops : min= 460, max= 980, avg=547.20, stdev=67.46, samples=60 lat (usec) : 1000=0.01% lat (msec) : 2=0.69%, 4=1.92%, 10=1.83%, 20=10.65%, 50=62.54% lat (msec) : 100=2.34%, 250=14.55%, 500=5.14%, 750=0.10%, 1000=0.05% lat (msec) : 2000=0.14%, >=2000=0.05% cpu : usr=1.61%, sys=2.03%, ctx=9246, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=66054,16544,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s), io=4128MiB (4329MB), run=30217-30217msec WRITE: bw=34.2MiB/s (35.9MB/s), 34.2MiB/s-34.2MiB/s (35.9MB/s-35.9MB/s), io=1034MiB (1084MB), run=30217-30217msec Disk stats (read/write): vdl: ios=66051/16438, merge=0/0, ticks=92300/3540294, in_queue=3672223, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924585: Fri Jan 7 13:58:00 2022 read: IOPS=2026, BW=158MiB/s (166MB/s)(4749MiB/30001msec) clat (usec): min=34, max=62633, avg=76.99, stdev=552.03 lat (usec): min=34, max=62633, avg=77.14, stdev=552.03 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 75], 95.00th=[ 83], | 99.00th=[ 114], 99.50th=[ 147], 99.90th=[ 8848], 99.95th=[14353], | 99.99th=[19792] bw ( KiB/s): min=132160, max=254720, per=100.00%, avg=162101.24, stdev=18793.50, samples=59 iops : min= 1652, max= 3184, avg=2026.25, stdev=234.93, samples=59 write: IOPS=509, BW=39.8MiB/s (41.7MB/s)(1194MiB/30001msec) clat (usec): min=398, max=100760, avg=1640.58, stdev=1814.25 lat (usec): min=402, max=100768, avg=1648.58, stdev=1814.21 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 685], 10.00th=[ 922], 20.00th=[ 1205], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1647], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2311], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36439], 99.95th=[39584], | 99.99th=[50594] bw ( KiB/s): min=34880, max=65920, per=99.94%, avg=40711.73, stdev=4399.70, samples=59 iops : min= 436, max= 824, avg=508.86, stdev=55.02, samples=59 lat (usec) : 50=13.16%, 100=65.36%, 250=1.18%, 500=0.44%, 750=0.98% lat (usec) : 1000=1.08% lat (msec) : 2=15.51%, 4=1.89%, 10=0.23%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.69%, sys=3.89%, ctx=76069, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60784,15277,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4749MiB (4979MB), run=30001-30001msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1194MiB (1251MB), run=30001-30001msec Disk stats (read/write): vdl: ios=60401/15174, merge=0/0, ticks=948/17155, in_queue=28001, util=93.65% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925585: Fri Jan 7 13:58:37 2022 read: IOPS=2025, BW=158MiB/s (166MB/s)(4747MiB/30001msec) slat (usec): min=7, max=161, avg=14.64, stdev= 4.57 clat (nsec): min=780, max=26206k, avg=68465.22, stdev=478531.79 lat (usec): min=39, max=26226, avg=83.27, stdev=478.58 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 72], | 99.00th=[ 91], 99.50th=[ 106], 99.90th=[ 7046], 99.95th=[14222], | 99.99th=[18744] bw ( KiB/s): min=128160, max=257280, per=100.00%, avg=162470.76, stdev=19438.02, samples=59 iops : min= 1602, max= 3216, avg=2030.86, stdev=242.98, samples=59 write: IOPS=509, BW=39.8MiB/s (41.7MB/s)(1193MiB/30001msec) slat (usec): min=13, max=132, avg=22.92, stdev= 5.89 clat (usec): min=383, max=48254, avg=1602.11, stdev=1662.94 lat (usec): min=404, max=48275, avg=1625.21, stdev=1663.00 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 668], 10.00th=[ 857], 20.00th=[ 1139], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1762], 90.00th=[ 1991], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[39060], | 99.99th=[43779] bw ( KiB/s): min=32480, max=66720, per=100.00%, avg=40817.64, stdev=4538.01, samples=59 iops : min= 406, max= 834, avg=510.20, stdev=56.74, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.01%, 50=36.26%, 100=43.12% lat (usec) : 250=0.37%, 500=0.36%, 750=1.14%, 1000=1.33% lat (msec) : 2=15.30%, 4=1.67%, 10=0.22%, 20=0.16%, 50=0.03% cpu : usr=1.57%, sys=6.40%, ctx=76017, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60763,15273,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4747MiB (4978MB), run=30001-30001msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1193MiB (1251MB), run=30001-30001msec Disk stats (read/write): vdl: ios=60401/15176, merge=0/0, ticks=877/16726, in_queue=27778, util=92.96% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926127: Fri Jan 7 13:59:14 2022 read: IOPS=2033, BW=159MiB/s (167MB/s)(4766MiB/30002msec) clat (usec): min=34, max=25651, avg=72.69, stdev=452.66 lat (usec): min=34, max=25652, avg=72.83, stdev=452.66 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 72], 95.00th=[ 78], | 99.00th=[ 96], 99.50th=[ 112], 99.90th=[ 2638], 99.95th=[13304], | 99.99th=[17433] bw ( KiB/s): min=118720, max=249760, per=100.00%, avg=162691.70, stdev=19573.22, samples=60 iops : min= 1484, max= 3122, avg=2033.63, stdev=244.66, samples=60 write: IOPS=510, BW=39.9MiB/s (41.9MB/s)(1198MiB/30002msec) clat (usec): min=390, max=44599, avg=1652.19, stdev=1675.49 lat (usec): min=397, max=44607, avg=1659.70, stdev=1675.48 clat percentiles (usec): | 1.00th=[ 469], 5.00th=[ 709], 10.00th=[ 971], 20.00th=[ 1237], | 30.00th=[ 1336], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[37487], 99.95th=[41157], | 99.99th=[44303] bw ( KiB/s): min=29920, max=65280, per=100.00%, avg=40873.43, stdev=4446.99, samples=60 iops : min= 374, max= 816, avg=510.90, stdev=55.58, samples=60 lat (usec) : 50=17.10%, 100=62.17%, 250=0.54%, 500=0.35%, 750=0.86% lat (usec) : 1000=0.96% lat (msec) : 2=15.68%, 4=1.96%, 10=0.21%, 20=0.16%, 50=0.02% cpu : usr=1.61%, sys=3.76%, ctx=76340, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61007,15328,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4766MiB (4998MB), run=30002-30002msec WRITE: bw=39.9MiB/s (41.9MB/s), 39.9MiB/s-39.9MiB/s (41.9MB/s-41.9MB/s), io=1198MiB (1256MB), run=30002-30002msec Disk stats (read/write): vdl: ios=61007/15327, merge=0/0, ticks=827/17490, in_queue=28315, util=94.09% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926961: Fri Jan 7 13:59:51 2022 read: IOPS=2081, BW=163MiB/s (171MB/s)(4884MiB/30033msec) slat (usec): min=4, max=139, avg=13.24, stdev= 4.49 clat (nsec): min=780, max=2497.6M, avg=1325607.01, stdev=49306242.94 lat (usec): min=30, max=2497.6k, avg=1339.01, stdev=49306.24 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 35], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 45], 60.00th=[ 48], 70.00th=[ 51], | 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 69], | 99.00th=[ 97], 99.50th=[ 121], 99.90th=[ 23725], | 99.95th=[1115685], 99.99th=[2499806] bw ( KiB/s): min=118560, max=298880, per=100.00%, avg=166688.00, stdev=23460.01, samples=60 iops : min= 1482, max= 3736, avg=2083.60, stdev=293.25, samples=60 write: IOPS=523, BW=40.9MiB/s (42.9MB/s)(1228MiB/30033msec) slat (usec): min=9, max=162, avg=22.78, stdev= 6.76 clat (usec): min=591, max=77523, avg=13751.47, stdev=6681.56 lat (usec): min=610, max=77543, avg=13774.45, stdev=6681.73 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 5342], 10.00th=[ 6783], 20.00th=[ 9372], | 30.00th=[11207], 40.00th=[12518], 50.00th=[13829], 60.00th=[15008], | 70.00th=[15926], 80.00th=[16909], 90.00th=[18220], 95.00th=[21890], | 99.00th=[31589], 99.50th=[63701], 99.90th=[70779], 99.95th=[76022], | 99.99th=[77071] bw ( KiB/s): min=35360, max=75520, per=100.00%, avg=41885.33, stdev=5135.75, samples=60 iops : min= 442, max= 944, avg=523.57, stdev=64.20, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=55.03% lat (usec) : 100=24.07%, 250=0.56%, 500=0.05%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.17%, 4=0.53%, 10=3.90%, 20=14.06%, 50=1.26% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.02% cpu : usr=1.81%, sys=5.94%, ctx=77378, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62509,15715,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4884MiB (5121MB), run=30033-30033msec WRITE: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=1228MiB (1287MB), run=30033-30033msec Disk stats (read/write): vdl: ios=62151/15613, merge=0/0, ticks=74688/206693, in_queue=296255, util=99.72% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927608: Fri Jan 7 14:00:28 2022 read: IOPS=2036, BW=159MiB/s (167MB/s)(4773MiB/30001msec) clat (usec): min=33, max=69047, avg=72.51, stdev=522.90 lat (usec): min=33, max=69047, avg=72.65, stdev=522.90 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 77], | 99.00th=[ 95], 99.50th=[ 110], 99.90th=[ 1532], 99.95th=[13698], | 99.99th=[17433] bw ( KiB/s): min=124800, max=223520, per=100.00%, avg=162975.02, stdev=17764.53, samples=60 iops : min= 1560, max= 2794, avg=2037.12, stdev=222.05, samples=60 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1199MiB/30001msec) clat (usec): min=389, max=47530, avg=1650.27, stdev=1669.51 lat (usec): min=396, max=47536, avg=1657.78, stdev=1669.45 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 709], 10.00th=[ 971], 20.00th=[ 1237], | 30.00th=[ 1352], 40.00th=[ 1401], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[22676], 99.95th=[41681], | 99.99th=[44303] bw ( KiB/s): min=32800, max=58080, per=100.00%, avg=40941.63, stdev=3730.82, samples=60 iops : min= 410, max= 726, avg=511.73, stdev=46.65, samples=60 lat (usec) : 50=20.22%, 100=59.09%, 250=0.50%, 500=0.34%, 750=0.86% lat (usec) : 1000=0.97% lat (msec) : 2=15.67%, 4=1.95%, 10=0.22%, 20=0.15%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.56%, sys=3.67%, ctx=76450, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61095,15349,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4773MiB (5005MB), run=30001-30001msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1199MiB (1257MB), run=30001-30001msec Disk stats (read/write): vdl: ios=60695/15256, merge=0/0, ticks=871/17410, in_queue=28125, util=94.31% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928860: Fri Jan 7 14:01:05 2022 read: IOPS=2082, BW=163MiB/s (171MB/s)(4898MiB/30109msec) slat (usec): min=2, max=171, avg=11.18, stdev= 4.28 clat (nsec): min=747, max=2538.8M, avg=1487956.03, stdev=54230115.62 lat (usec): min=30, max=2538.8k, avg=1499.29, stdev=54230.11 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 40], | 50.00th=[ 42], 60.00th=[ 46], 70.00th=[ 51], | 80.00th=[ 58], 90.00th=[ 65], 95.00th=[ 75], | 99.00th=[ 351], 99.50th=[ 676], 99.90th=[ 11076], | 99.95th=[1535116], 99.99th=[2499806] bw ( KiB/s): min=126720, max=326720, per=100.00%, avg=167228.30, stdev=27332.27, samples=60 iops : min= 1584, max= 4084, avg=2090.28, stdev=341.68, samples=60 write: IOPS=523, BW=40.9MiB/s (42.9MB/s)(1231MiB/30109msec) slat (usec): min=9, max=168, avg=23.73, stdev= 9.35 clat (msec): min=8, max=2536, avg=89.51, stdev=80.72 lat (msec): min=8, max=2536, avg=89.53, stdev=80.72 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 63], 10.00th=[ 67], 20.00th=[ 73], | 30.00th=[ 78], 40.00th=[ 80], 50.00th=[ 82], 60.00th=[ 84], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 101], 95.00th=[ 131], | 99.00th=[ 230], 99.50th=[ 575], 99.90th=[ 1401], 99.95th=[ 1653], | 99.99th=[ 2165] bw ( KiB/s): min=32320, max=75680, per=100.00%, avg=41909.30, stdev=5295.54, samples=60 iops : min= 404, max= 946, avg=523.82, stdev=66.23, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.03%, 50=55.33% lat (usec) : 100=22.50%, 250=0.98%, 500=0.44%, 750=0.23%, 1000=0.08% lat (msec) : 2=0.19%, 10=0.03%, 20=0.31%, 50=0.15%, 100=17.58% lat (msec) : 250=1.84%, 500=0.09%, 750=0.05%, 1000=0.03%, 2000=0.07% lat (msec) : >=2000=0.03% cpu : usr=1.82%, sys=5.29%, ctx=76069, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=62696,15761,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4898MiB (5136MB), run=30109-30109msec WRITE: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=1231MiB (1291MB), run=30109-30109msec Disk stats (read/write): vdl: ios=62695/15724, merge=0/0, ticks=87657/1395075, in_queue=1499862, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929911: Fri Jan 7 14:01:42 2022 read: IOPS=2025, BW=158MiB/s (166MB/s)(4750MiB/30010msec) clat (usec): min=31, max=28422, avg=73.79, stdev=455.65 lat (usec): min=31, max=28422, avg=73.93, stdev=455.65 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 55], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 101], 99.50th=[ 121], 99.90th=[ 6128], 99.95th=[13304], | 99.99th=[17695] bw ( KiB/s): min=130720, max=241760, per=100.00%, avg=162130.67, stdev=18421.12, samples=60 iops : min= 1634, max= 3022, avg=2026.63, stdev=230.26, samples=60 write: IOPS=509, BW=39.8MiB/s (41.7MB/s)(1194MiB/30010msec) clat (usec): min=392, max=45178, avg=1653.77, stdev=1683.98 lat (usec): min=399, max=45186, avg=1661.83, stdev=1684.03 clat percentiles (usec): | 1.00th=[ 482], 5.00th=[ 693], 10.00th=[ 955], 20.00th=[ 1221], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2057], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[37487], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min=31520, max=63040, per=100.00%, avg=40744.00, stdev=4182.97, samples=60 iops : min= 394, max= 788, avg=509.30, stdev=52.29, samples=60 lat (usec) : 50=13.44%, 100=65.64%, 250=0.70%, 500=0.34%, 750=0.97% lat (usec) : 1000=0.95% lat (msec) : 2=15.59%, 4=1.96%, 10=0.23%, 20=0.16%, 50=0.03% cpu : usr=1.67%, sys=3.84%, ctx=76084, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60799,15280,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4750MiB (4981MB), run=30010-30010msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1194MiB (1252MB), run=30010-30010msec Disk stats (read/write): vdl: ios=60732/15267, merge=0/0, ticks=818/17496, in_queue=28165, util=93.64% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930584: Fri Jan 7 14:02:20 2022 read: IOPS=2071, BW=162MiB/s (170MB/s)(4889MiB/30207msec) slat (usec): min=2, max=198, avg=11.02, stdev= 4.82 clat (nsec): min=800, max=2527.8M, avg=1483703.27, stdev=53043359.94 lat (usec): min=30, max=2527.8k, avg=1494.87, stdev=53043.34 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 44], 60.00th=[ 48], 70.00th=[ 55], | 80.00th=[ 61], 90.00th=[ 74], 95.00th=[ 141], | 99.00th=[ 1827], 99.50th=[ 3064], 99.90th=[ 15664], | 99.95th=[1249903], 99.99th=[2499806] bw ( KiB/s): min=132640, max=348800, per=100.00%, avg=166889.98, stdev=28364.81, samples=60 iops : min= 1658, max= 4360, avg=2086.10, stdev=354.57, samples=60 write: IOPS=520, BW=40.7MiB/s (42.7MB/s)(1229MiB/30207msec) slat (usec): min=10, max=221, avg=24.80, stdev=10.21 clat (msec): min=19, max=3348, avg=186.04, stdev=153.43 lat (msec): min=19, max=3349, avg=186.07, stdev=153.43 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 129], 10.00th=[ 134], 20.00th=[ 148], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 180], | 70.00th=[ 186], 80.00th=[ 194], 90.00th=[ 220], 95.00th=[ 245], | 99.00th=[ 523], 99.50th=[ 1083], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3239] bw ( KiB/s): min=31840, max=73920, per=100.00%, avg=41684.03, stdev=5255.04, samples=60 iops : min= 398, max= 924, avg=521.03, stdev=65.70, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.02%, 50=51.54% lat (usec) : 100=23.60%, 250=1.61%, 500=0.89%, 750=0.61%, 1000=0.33% lat (msec) : 2=0.52%, 4=0.63%, 10=0.04%, 20=0.01%, 50=0.42% lat (msec) : 100=0.23%, 250=18.54%, 500=0.70%, 750=0.09%, 1000=0.03% lat (msec) : 2000=0.09%, >=2000=0.07% cpu : usr=1.90%, sys=5.20%, ctx=73239, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62579,15729,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=4889MiB (5126MB), run=30207-30207msec WRITE: bw=40.7MiB/s (42.7MB/s), 40.7MiB/s-40.7MiB/s (42.7MB/s-42.7MB/s), io=1229MiB (1289MB), run=30207-30207msec Disk stats (read/write): vdl: ios=62578/15638, merge=0/0, ticks=85826/2889938, in_queue=2999618, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931412: Fri Jan 7 14:02:56 2022 read: IOPS=2037, BW=159MiB/s (167MB/s)(4775MiB/30002msec) clat (usec): min=33, max=24970, avg=70.13, stdev=459.50 lat (usec): min=33, max=24970, avg=70.27, stdev=459.50 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 94], 99.50th=[ 106], 99.90th=[ 1532], 99.95th=[14091], | 99.99th=[17957] bw ( KiB/s): min=120480, max=255680, per=100.00%, avg=162999.55, stdev=21398.57, samples=60 iops : min= 1506, max= 3196, avg=2037.48, stdev=267.49, samples=60 write: IOPS=511, BW=39.0MiB/s (41.9MB/s)(1200MiB/30002msec) clat (usec): min=388, max=49887, avg=1658.55, stdev=1678.43 lat (usec): min=394, max=49894, avg=1666.12, stdev=1678.38 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 717], 10.00th=[ 996], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1680], 80.00th=[ 1811], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[37487], 99.95th=[41681], | 99.99th=[45351] bw ( KiB/s): min=31360, max=66560, per=100.00%, avg=40947.90, stdev=4763.97, samples=60 iops : min= 392, max= 832, avg=511.83, stdev=59.56, samples=60 lat (usec) : 50=29.28%, 100=50.09%, 250=0.44%, 500=0.34%, 750=0.80% lat (usec) : 1000=0.91% lat (msec) : 2=15.79%, 4=1.97%, 10=0.20%, 20=0.16%, 50=0.02% cpu : usr=1.53%, sys=3.60%, ctx=76487, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61123,15356,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=159MiB/s (167MB/s), 159MiB/s-159MiB/s (167MB/s-167MB/s), io=4775MiB (5007MB), run=30002-30002msec WRITE: bw=39.0MiB/s (41.9MB/s), 39.0MiB/s-39.0MiB/s (41.9MB/s-41.9MB/s), io=1200MiB (1258MB), run=30002-30002msec Disk stats (read/write): vdl: ios=60648/15240, merge=0/0, ticks=838/17457, in_queue=28131, util=94.17% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932354: Fri Jan 7 14:03:34 2022 read: IOPS=2080, BW=163MiB/s (170MB/s)(4921MiB/30275msec) slat (usec): min=2, max=240633, avg=378.82, stdev=2487.33 clat (usec): min=1066, max=2589.5k, avg=31301.30, stdev=54066.29 lat (usec): min=1071, max=2589.6k, avg=31680.27, stdev=54127.39 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 31], | 70.00th=[ 33], 80.00th=[ 36], 90.00th=[ 42], 95.00th=[ 48], | 99.00th=[ 90], 99.50th=[ 220], 99.90th=[ 300], 99.95th=[ 1200], | 99.99th=[ 2534] bw ( KiB/s): min=122004, max=354720, per=100.00%, avg=167825.85, stdev=28629.02, samples=60 iops : min= 1525, max= 4434, avg=2097.80, stdev=357.86, samples=60 write: IOPS=523, BW=40.9MiB/s (42.9MB/s)(1238MiB/30275msec) slat (usec): min=7, max=59530, avg=377.64, stdev=1619.53 clat (msec): min=21, max=4329, avg=255.81, stdev=252.39 lat (msec): min=23, max=4329, avg=256.18, stdev=252.38 clat percentiles (msec): | 1.00th=[ 33], 5.00th=[ 155], 10.00th=[ 178], 20.00th=[ 194], | 30.00th=[ 209], 40.00th=[ 222], 50.00th=[ 232], 60.00th=[ 243], | 70.00th=[ 251], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 326], | 99.00th=[ 1183], 99.50th=[ 2400], 99.90th=[ 3507], 99.95th=[ 3809], | 99.99th=[ 4245] bw ( KiB/s): min=30460, max=72480, per=100.00%, avg=41886.52, stdev=5219.66, samples=60 iops : min= 380, max= 906, avg=523.55, stdev=65.28, samples=60 lat (msec) : 2=0.68%, 4=1.87%, 10=2.08%, 20=8.70%, 50=63.74% lat (msec) : 100=2.80%, 250=13.85%, 500=5.85%, 750=0.08%, 1000=0.08% lat (msec) : 2000=0.11%, >=2000=0.15% cpu : usr=1.75%, sys=2.06%, ctx=8828, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62994,15841,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=163MiB/s (170MB/s), 163MiB/s-163MiB/s (170MB/s-170MB/s), io=4921MiB (5160MB), run=30275-30275msec WRITE: bw=40.9MiB/s (42.9MB/s), 40.9MiB/s-40.9MiB/s (42.9MB/s-42.9MB/s), io=1238MiB (1298MB), run=30275-30275msec Disk stats (read/write): vdl: ios=62993/15735, merge=0/0, ticks=90776/3509343, in_queue=3672022, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932852: Fri Jan 7 14:04:10 2022 read: IOPS=1798, BW=225MiB/s (236MB/s)(6746MiB/30002msec) clat (usec): min=48, max=218470, avg=98.85, stdev=1060.10 lat (usec): min=48, max=218471, avg=99.00, stdev=1060.10 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 61], 10.00th=[ 63], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 72], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 137], 99.50th=[ 159], 99.90th=[11076], 99.95th=[15008], | 99.99th=[19268] bw ( KiB/s): min=110336, max=356352, per=100.00%, avg=230266.42, stdev=31910.03, samples=60 iops : min= 862, max= 2784, avg=1798.93, stdev=249.29, samples=60 write: IOPS=451, BW=56.5MiB/s (59.2MB/s)(1695MiB/30002msec) clat (usec): min=531, max=55796, avg=1797.54, stdev=1806.96 lat (usec): min=542, max=55807, avg=1810.35, stdev=1806.92 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 906], 10.00th=[ 1074], 20.00th=[ 1237], | 30.00th=[ 1385], 40.00th=[ 1500], 50.00th=[ 1598], 60.00th=[ 1696], | 70.00th=[ 1811], 80.00th=[ 1942], 90.00th=[ 2180], 95.00th=[ 2507], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[39060], 99.95th=[41681], | 99.99th=[46400] bw ( KiB/s): min=31744, max=92416, per=100.00%, avg=57855.82, stdev=7409.56, samples=60 iops : min= 248, max= 722, avg=451.98, stdev=57.89, samples=60 lat (usec) : 50=0.01%, 100=72.77%, 250=6.95%, 500=0.06%, 750=0.56% lat (usec) : 1000=0.94% lat (msec) : 2=15.29%, 4=2.83%, 10=0.38%, 20=0.18%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.65%, sys=3.78%, ctx=67531, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53966,13559,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6746MiB (7073MB), run=30002-30002msec WRITE: bw=56.5MiB/s (59.2MB/s), 56.5MiB/s-56.5MiB/s (59.2MB/s-59.2MB/s), io=1695MiB (1777MB), run=30002-30002msec Disk stats (read/write): vdl: ios=53631/13475, merge=0/0, ticks=1038/17031, in_queue=28071, util=93.87% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933837: Fri Jan 7 14:04:47 2022 read: IOPS=1842, BW=230MiB/s (242MB/s)(6911MiB/30001msec) slat (usec): min=7, max=132, avg=13.93, stdev= 4.22 clat (nsec): min=760, max=23966k, avg=65311.19, stdev=517659.11 lat (usec): min=36, max=23976, avg=79.40, stdev=517.71 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 60], 95.00th=[ 65], | 99.00th=[ 80], 99.50th=[ 92], 99.90th=[11469], 99.95th=[15664], | 99.99th=[19268] bw ( KiB/s): min=154880, max=385792, per=100.00%, avg=236157.83, stdev=31176.76, samples=59 iops : min= 1210, max= 3014, avg=1844.98, stdev=243.57, samples=59 write: IOPS=463, BW=57.9MiB/s (60.7MB/s)(1737MiB/30001msec) slat (usec): min=14, max=139, avg=26.91, stdev= 7.32 clat (usec): min=502, max=44288, avg=1808.28, stdev=1646.52 lat (usec): min=526, max=44313, avg=1835.37, stdev=1646.69 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 783], 10.00th=[ 1037], 20.00th=[ 1369], | 30.00th=[ 1483], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1844], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2573], | 99.00th=[ 9765], 99.50th=[10290], 99.90th=[36439], 99.95th=[41681], | 99.99th=[43254] bw ( KiB/s): min=48128, max=100352, per=100.00%, avg=59344.27, stdev=6823.23, samples=59 iops : min= 376, max= 784, avg=463.63, stdev=53.31, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=53.10% lat (usec) : 100=26.50%, 250=0.16%, 500=0.02%, 750=0.82%, 1000=1.05% lat (msec) : 2=14.32%, 4=3.56%, 10=0.20%, 20=0.22%, 50=0.02% cpu : usr=1.63%, sys=5.46%, ctx=69191, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55288,13898,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=230MiB/s (242MB/s), 230MiB/s-230MiB/s (242MB/s-242MB/s), io=6911MiB (7247MB), run=30001-30001msec WRITE: bw=57.9MiB/s (60.7MB/s), 57.9MiB/s-57.9MiB/s (60.7MB/s-60.7MB/s), io=1737MiB (1822MB), run=30001-30001msec Disk stats (read/write): vdl: ios=54886/13795, merge=0/0, ticks=913/17309, in_queue=28050, util=93.88% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934421: Fri Jan 7 14:05:24 2022 read: IOPS=1836, BW=230MiB/s (241MB/s)(6887MiB/30001msec) clat (usec): min=39, max=24998, avg=80.13, stdev=524.19 lat (usec): min=39, max=24998, avg=80.27, stdev=524.19 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 71], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 104], 99.50th=[ 122], 99.90th=[11863], 99.95th=[15795], | 99.99th=[19006] bw ( KiB/s): min=174080, max=369920, per=100.00%, avg=235127.12, stdev=30339.28, samples=59 iops : min= 1360, max= 2890, avg=1836.92, stdev=237.03, samples=59 write: IOPS=461, BW=57.7MiB/s (60.5MB/s)(1731MiB/30001msec) clat (usec): min=520, max=151113, avg=1827.46, stdev=2061.40 lat (usec): min=529, max=151125, avg=1839.57, stdev=2061.36 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 799], 10.00th=[ 1057], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1565], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9634], 99.50th=[10290], 99.90th=[38011], 99.95th=[41681], | 99.99th=[43254] bw ( KiB/s): min=47872, max=95488, per=100.00%, avg=59080.83, stdev=6303.83, samples=59 iops : min= 374, max= 746, avg=461.53, stdev=49.28, samples=59 lat (usec) : 50=3.05%, 100=75.85%, 250=0.88%, 500=0.03%, 750=0.80% lat (usec) : 1000=1.04% lat (msec) : 2=13.96%, 4=3.99%, 10=0.17%, 20=0.20%, 50=0.03% lat (msec) : 250=0.01% cpu : usr=1.57%, sys=3.54%, ctx=68950, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55095,13847,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6887MiB (7221MB), run=30001-30001msec WRITE: bw=57.7MiB/s (60.5MB/s), 57.7MiB/s-57.7MiB/s (60.5MB/s-60.5MB/s), io=1731MiB (1815MB), run=30001-30001msec Disk stats (read/write): vdl: ios=54689/13740, merge=0/0, ticks=924/17362, in_queue=28173, util=94.28% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935726: Fri Jan 7 14:06:01 2022 read: IOPS=1873, BW=234MiB/s (246MB/s)(7034MiB/30039msec) slat (usec): min=5, max=624, avg=14.16, stdev= 5.17 clat (nsec): min=816, max=2492.2M, avg=1569864.45, stdev=55849181.90 lat (usec): min=36, max=2492.2k, avg=1584.18, stdev=55849.16 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 43], 40.00th=[ 45], | 50.00th=[ 46], 60.00th=[ 49], 70.00th=[ 53], | 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 76], | 99.00th=[ 108], 99.50th=[ 174], 99.90th=[ 139461], | 99.95th=[1434452], 99.99th=[2499806] bw ( KiB/s): min=190976, max=418304, per=100.00%, avg=240133.02, stdev=30828.12, samples=60 iops : min= 1492, max= 3268, avg=1876.00, stdev=240.88, samples=60 write: IOPS=470, BW=58.9MiB/s (61.7MB/s)(1768MiB/30039msec) slat (usec): min=13, max=757, avg=29.01, stdev=11.36 clat (usec): min=1572, max=77673, avg=14891.51, stdev=6910.59 lat (usec): min=1629, max=77696, avg=14920.70, stdev=6910.89 clat percentiles (usec): | 1.00th=[ 3621], 5.00th=[ 7111], 10.00th=[ 8160], 20.00th=[10290], | 30.00th=[12125], 40.00th=[13304], 50.00th=[14484], 60.00th=[15533], | 70.00th=[16581], 80.00th=[17957], 90.00th=[20317], 95.00th=[24511], | 99.00th=[37487], 99.50th=[64226], 99.90th=[74974], 99.95th=[77071], | 99.99th=[77071] bw ( KiB/s): min=50176, max=105472, per=100.00%, avg=60336.30, stdev=7170.71, samples=60 iops : min= 392, max= 824, avg=471.32, stdev=56.04, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.03%, 50=50.88% lat (usec) : 100=28.03%, 250=0.63%, 500=0.14%, 750=0.04%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.32%, 10=3.42%, 20=14.17%, 50=2.03% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.04% cpu : usr=1.83%, sys=5.44%, ctx=69707, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=56274,14147,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=7034MiB (7376MB), run=30039-30039msec WRITE: bw=58.9MiB/s (61.7MB/s), 58.9MiB/s-58.9MiB/s (61.7MB/s-61.7MB/s), io=1768MiB (1854MB), run=30039-30039msec Disk stats (read/write): vdl: ios=55988/14060, merge=0/0, ticks=85156/201297, in_queue=296323, util=99.75% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936697: Fri Jan 7 14:06:38 2022 read: IOPS=1832, BW=229MiB/s (240MB/s)(6874MiB/30001msec) clat (usec): min=39, max=27253, avg=82.56, stdev=515.58 lat (usec): min=39, max=27253, avg=82.70, stdev=515.58 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 54], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 61], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 71], 80.00th=[ 75], 90.00th=[ 81], 95.00th=[ 87], | 99.00th=[ 105], 99.50th=[ 120], 99.90th=[11600], 99.95th=[15270], | 99.99th=[19268] bw ( KiB/s): min=193024, max=378368, per=100.00%, avg=234644.03, stdev=28330.64, samples=60 iops : min= 1508, max= 2956, avg=1833.13, stdev=221.33, samples=60 write: IOPS=460, BW=57.6MiB/s (60.4MB/s)(1727MiB/30001msec) clat (usec): min=507, max=43689, avg=1820.97, stdev=1668.50 lat (usec): min=519, max=43701, avg=1833.98, stdev=1668.51 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 799], 10.00th=[ 1057], 20.00th=[ 1369], | 30.00th=[ 1483], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2245], 95.00th=[ 2540], | 99.00th=[ 9765], 99.50th=[10159], 99.90th=[36439], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min=47104, max=98816, per=100.00%, avg=58958.37, stdev=6594.19, samples=60 iops : min= 368, max= 772, avg=460.57, stdev=51.52, samples=60 lat (usec) : 50=1.80%, 100=76.93%, 250=1.08%, 500=0.01%, 750=0.79% lat (usec) : 1000=0.99% lat (msec) : 2=14.29%, 4=3.64%, 10=0.24%, 20=0.20%, 50=0.03% cpu : usr=1.69%, sys=3.70%, ctx=68818, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54989,13818,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6874MiB (7208MB), run=30001-30001msec WRITE: bw=57.6MiB/s (60.4MB/s), 57.6MiB/s-57.6MiB/s (60.4MB/s-60.4MB/s), io=1727MiB (1811MB), run=30001-30001msec Disk stats (read/write): vdl: ios=54623/13725, merge=0/0, ticks=905/17211, in_queue=28012, util=93.68% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937337: Fri Jan 7 14:07:16 2022 read: IOPS=1870, BW=234MiB/s (245MB/s)(7045MiB/30127msec) slat (usec): min=3, max=143, avg=12.75, stdev= 4.87 clat (nsec): min=899, max=2600.3M, avg=1738293.43, stdev=59386091.35 lat (usec): min=33, max=2600.3k, avg=1751.20, stdev=59386.08 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 49], 60.00th=[ 53], 70.00th=[ 60], | 80.00th=[ 74], 90.00th=[ 85], 95.00th=[ 99], | 99.00th=[ 603], 99.50th=[ 1139], 99.90th=[ 270533], | 99.95th=[1870660], 99.99th=[2533360] bw ( KiB/s): min=193792, max=480256, per=100.00%, avg=240463.70, stdev=39342.10, samples=60 iops : min= 1514, max= 3752, avg=1878.62, stdev=307.36, samples=60 write: IOPS=470, BW=58.8MiB/s (61.7MB/s)(1772MiB/30127msec) slat (usec): min=10, max=210, avg=32.60, stdev=13.57 clat (msec): min=6, max=2564, avg=99.27, stdev=68.60 lat (msec): min=6, max=2564, avg=99.30, stdev=68.60 clat percentiles (msec): | 1.00th=[ 19], 5.00th=[ 75], 10.00th=[ 81], 20.00th=[ 85], | 30.00th=[ 88], 40.00th=[ 90], 50.00th=[ 93], 60.00th=[ 95], | 70.00th=[ 97], 80.00th=[ 103], 90.00th=[ 114], 95.00th=[ 142], | 99.00th=[ 213], 99.50th=[ 542], 99.90th=[ 1083], 99.95th=[ 1368], | 99.99th=[ 2165] bw ( KiB/s): min=48640, max=110848, per=100.00%, avg=60277.03, stdev=7694.41, samples=60 iops : min= 380, max= 866, avg=470.90, stdev=60.12, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 10=0.01%, 20=0.02%, 50=42.31%, 100=33.75% lat (usec) : 250=2.31%, 500=0.51%, 750=0.36%, 1000=0.16% lat (msec) : 2=0.25%, 4=0.11%, 10=0.01%, 20=0.34%, 50=0.20% lat (msec) : 100=14.91%, 250=4.46%, 500=0.09%, 750=0.06%, 1000=0.03% lat (msec) : 2000=0.04%, >=2000=0.04% cpu : usr=2.17%, sys=5.04%, ctx=68099, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=56362,14172,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=7045MiB (7387MB), run=30127-30127msec WRITE: bw=58.8MiB/s (61.7MB/s), 58.8MiB/s-58.8MiB/s (61.7MB/s-61.7MB/s), io=1772MiB (1858MB), run=30127-30127msec Disk stats (read/write): vdl: ios=56357/14172, merge=0/0, ticks=88957/1397212, in_queue=1501648, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938227: Fri Jan 7 14:07:52 2022 read: IOPS=1846, BW=231MiB/s (242MB/s)(6925MiB/30001msec) clat (usec): min=38, max=26638, avg=77.64, stdev=499.59 lat (usec): min=38, max=26638, avg=77.79, stdev=499.59 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 66], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 98], 99.50th=[ 111], 99.90th=[10683], 99.95th=[14222], | 99.99th=[19268] bw ( KiB/s): min=169811, max=385792, per=100.00%, avg=236565.61, stdev=29307.82, samples=59 iops : min= 1326, max= 3014, avg=1848.14, stdev=228.99, samples=59 write: IOPS=464, BW=58.0MiB/s (60.8MB/s)(1741MiB/30001msec) clat (usec): min=510, max=45138, avg=1824.53, stdev=1704.80 lat (usec): min=522, max=45151, avg=1837.02, stdev=1704.79 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 816], 10.00th=[ 1123], 20.00th=[ 1401], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 1991], 90.00th=[ 2212], 95.00th=[ 2474], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[36963], 99.95th=[43779], | 99.99th=[44827] bw ( KiB/s): min=47616, max=100352, per=100.00%, avg=59445.36, stdev=6834.47, samples=59 iops : min= 372, max= 784, avg=464.37, stdev=53.42, samples=59 lat (usec) : 50=5.34%, 100=73.88%, 250=0.59%, 500=0.01%, 750=0.76% lat (usec) : 1000=0.79% lat (msec) : 2=14.56%, 4=3.65%, 10=0.18%, 20=0.22%, 50=0.02% cpu : usr=1.73%, sys=3.39%, ctx=69331, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55398,13926,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=6925MiB (7261MB), run=30001-30001msec WRITE: bw=58.0MiB/s (60.8MB/s), 58.0MiB/s-58.0MiB/s (60.8MB/s-60.8MB/s), io=1741MiB (1825MB), run=30001-30001msec Disk stats (read/write): vdl: ios=55004/13821, merge=0/0, ticks=875/17425, in_queue=28261, util=94.59% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938830: Fri Jan 7 14:08:30 2022 read: IOPS=1877, BW=235MiB/s (246MB/s)(7098MiB/30236msec) slat (usec): min=3, max=236, avg=11.56, stdev= 5.53 clat (nsec): min=762, max=2596.6M, avg=1642094.55, stdev=55604343.23 lat (usec): min=36, max=2596.7k, avg=1653.81, stdev=55604.37 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 44], 40.00th=[ 46], | 50.00th=[ 49], 60.00th=[ 53], 70.00th=[ 62], | 80.00th=[ 75], 90.00th=[ 88], 95.00th=[ 161], | 99.00th=[ 1958], 99.50th=[ 3851], 99.90th=[ 129500], | 99.95th=[1417675], 99.99th=[2533360] bw ( KiB/s): min=185600, max=521984, per=100.00%, avg=242277.63, stdev=43016.24, samples=60 iops : min= 1450, max= 4078, avg=1892.77, stdev=336.06, samples=60 write: IOPS=472, BW=59.1MiB/s (61.0MB/s)(1787MiB/30236msec) slat (usec): min=12, max=297, avg=32.38, stdev=15.02 clat (msec): min=3, max=2483, avg=204.91, stdev=82.51 lat (msec): min=3, max=2483, avg=204.94, stdev=82.51 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 155], 10.00th=[ 176], 20.00th=[ 184], | 30.00th=[ 188], 40.00th=[ 192], 50.00th=[ 197], 60.00th=[ 201], | 70.00th=[ 207], 80.00th=[ 218], 90.00th=[ 249], 95.00th=[ 268], | 99.00th=[ 418], 99.50th=[ 785], 99.90th=[ 1334], 99.95th=[ 1519], | 99.99th=[ 2039] bw ( KiB/s): min=48640, max=108032, per=100.00%, avg=60588.38, stdev=7266.50, samples=60 iops : min= 380, max= 844, avg=473.30, stdev=56.77, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.03%, 50=43.08% lat (usec) : 100=31.09%, 250=2.31%, 500=0.85%, 750=0.55%, 1000=0.43% lat (msec) : 2=0.72%, 4=0.43%, 10=0.28%, 20=0.04%, 50=0.36% lat (msec) : 100=0.16%, 250=17.74%, 500=1.68%, 750=0.06%, 1000=0.07% lat (msec) : 2000=0.05%, >=2000=0.04% cpu : usr=2.20%, sys=4.74%, ctx=66394, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56783,14293,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=235MiB/s (246MB/s), 235MiB/s-235MiB/s (246MB/s-246MB/s), io=7098MiB (7443MB), run=30236-30236msec WRITE: bw=59.1MiB/s (61.0MB/s), 59.1MiB/s-59.1MiB/s (61.0MB/s-61.0MB/s), io=1787MiB (1873MB), run=30236-30236msec Disk stats (read/write): vdl: ios=56778/14201, merge=0/0, ticks=81090/2890603, in_queue=3000725, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939649: Fri Jan 7 14:09:06 2022 read: IOPS=1840, BW=230MiB/s (241MB/s)(6904MiB/30002msec) clat (usec): min=38, max=29525, avg=79.36, stdev=531.23 lat (usec): min=38, max=29525, avg=79.51, stdev=531.23 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 57], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 70], 90.00th=[ 77], 95.00th=[ 83], | 99.00th=[ 101], 99.50th=[ 116], 99.90th=[11600], 99.95th=[15664], | 99.99th=[19006] bw ( KiB/s): min=175872, max=386816, per=100.00%, avg=235647.85, stdev=31404.95, samples=60 iops : min= 1374, max= 3022, avg=1840.98, stdev=245.34, samples=60 write: IOPS=462, BW=57.8MiB/s (60.6MB/s)(1735MiB/30002msec) clat (usec): min=522, max=84313, avg=1825.58, stdev=1778.59 lat (usec): min=532, max=84325, avg=1837.77, stdev=1778.48 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 807], 10.00th=[ 1106], 20.00th=[ 1418], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 1991], 90.00th=[ 2245], 95.00th=[ 2474], | 99.00th=[ 9765], 99.50th=[10290], 99.90th=[36963], 99.95th=[40633], | 99.99th=[45351] bw ( KiB/s): min=46592, max=100864, per=100.00%, avg=59206.35, stdev=6827.55, samples=60 iops : min= 364, max= 788, avg=462.53, stdev=53.33, samples=60 lat (usec) : 50=4.32%, 100=74.73%, 250=0.73%, 500=0.02%, 750=0.79% lat (usec) : 1000=0.83% lat (msec) : 2=14.49%, 4=3.66%, 10=0.19%, 20=0.21%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.52%, sys=3.61%, ctx=69112, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55228,13877,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=6904MiB (7239MB), run=30002-30002msec WRITE: bw=57.8MiB/s (60.6MB/s), 57.8MiB/s-57.8MiB/s (60.6MB/s-60.6MB/s), io=1735MiB (1819MB), run=30002-30002msec Disk stats (read/write): vdl: ios=54839/13781, merge=0/0, ticks=932/17337, in_queue=28206, util=94.32% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940616: Fri Jan 7 14:09:44 2022 read: IOPS=1875, BW=234MiB/s (246MB/s)(7094MiB/30264msec) slat (usec): min=3, max=258632, avg=414.86, stdev=2733.95 clat (usec): min=1468, max=2591.4k, avg=34950.18, stdev=61267.55 lat (usec): min=1474, max=2592.5k, avg=35365.20, stdev=61342.81 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 8], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 30], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 53], | 99.00th=[ 94], 99.50th=[ 266], 99.90th=[ 384], 99.95th=[ 1636], | 99.99th=[ 2567] bw ( KiB/s): min=190464, max=537082, per=100.00%, avg=244160.18, stdev=44631.12, samples=60 iops : min= 1488, max= 4195, avg=1907.17, stdev=348.58, samples=60 write: IOPS=472, BW=59.0MiB/s (61.9MB/s)(1786MiB/30264msec) slat (usec): min=11, max=181947, avg=441.71, stdev=2351.11 clat (msec): min=6, max=3387, avg=282.68, stdev=168.14 lat (msec): min=6, max=3387, avg=283.12, stdev=168.12 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 188], 10.00th=[ 220], 20.00th=[ 239], | 30.00th=[ 249], 40.00th=[ 259], 50.00th=[ 266], 60.00th=[ 271], | 70.00th=[ 279], 80.00th=[ 296], 90.00th=[ 326], 95.00th=[ 376], | 99.00th=[ 1003], 99.50th=[ 1519], 99.90th=[ 2265], 99.95th=[ 3373], | 99.99th=[ 3373] bw ( KiB/s): min=46924, max=106537, per=100.00%, avg=60951.20, stdev=7239.86, samples=60 iops : min= 366, max= 832, avg=475.87, stdev=56.57, samples=60 lat (msec) : 2=0.21%, 4=1.22%, 10=3.13%, 20=4.89%, 50=66.04% lat (msec) : 100=4.15%, 250=5.90%, 500=13.88%, 750=0.23%, 1000=0.10% lat (msec) : 2000=0.19%, >=2000=0.06% cpu : usr=2.34%, sys=1.98%, ctx=7894, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56751,14285,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=7094MiB (7438MB), run=30264-30264msec WRITE: bw=59.0MiB/s (61.9MB/s), 59.0MiB/s-59.0MiB/s (61.9MB/s-61.9MB/s), io=1786MiB (1872MB), run=30264-30264msec Disk stats (read/write): vdl: ios=56747/14285, merge=0/0, ticks=99590/3558551, in_queue=3687371, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941234: Fri Jan 7 14:10:20 2022 read: IOPS=1263, BW=316MiB/s (331MB/s)(9475MiB/30003msec) clat (usec): min=56, max=21543, avg=103.40, stdev=420.84 lat (usec): min=57, max=21543, avg=103.56, stdev=420.84 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 75], | 30.00th=[ 77], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 111], 95.00th=[ 122], | 99.00th=[ 157], 99.50th=[ 184], 99.90th=[ 8094], 99.95th=[12256], | 99.99th=[17433] bw ( KiB/s): min=179200, max=570880, per=100.00%, avg=323387.47, stdev=57076.67, samples=60 iops : min= 700, max= 2230, avg=1263.20, stdev=222.96, samples=60 write: IOPS=319, BW=79.8MiB/s (83.7MB/s)(2395MiB/30003msec) clat (usec): min=865, max=209702, avg=2685.10, stdev=3232.69 lat (usec): min=894, max=209767, avg=2713.31, stdev=3232.68 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 1123], 10.00th=[ 1401], 20.00th=[ 1663], | 30.00th=[ 1893], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2737], 90.00th=[ 4424], 95.00th=[ 5538], | 99.00th=[ 11076], 99.50th=[ 13173], 99.90th=[ 40633], 99.95th=[ 53740], | 99.99th=[208667] bw ( KiB/s): min=44544, max=152064, per=100.00%, avg=81739.52, stdev=14257.51, samples=60 iops : min= 174, max= 594, avg=319.27, stdev=55.71, samples=60 lat (usec) : 100=63.95%, 250=15.63%, 500=0.08%, 750=0.02%, 1000=0.50% lat (msec) : 2=7.23%, 4=10.12%, 10=1.83%, 20=0.62%, 50=0.01% lat (msec) : 100=0.02%, 250=0.01% cpu : usr=1.76%, sys=2.93%, ctx=47489, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=37898,9580,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=316MiB/s (331MB/s), 316MiB/s-316MiB/s (331MB/s-331MB/s), io=9475MiB (9935MB), run=30003-30003msec WRITE: bw=79.8MiB/s (83.7MB/s), 79.8MiB/s-79.8MiB/s (83.7MB/s-83.7MB/s), io=2395MiB (2511MB), run=30003-30003msec Disk stats (read/write): vdl: ios=37694/9511, merge=0/0, ticks=538/20680, in_queue=28292, util=94.67% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942447: Fri Jan 7 14:10:57 2022 read: IOPS=1314, BW=329MiB/s (345MB/s)(9857MiB/30001msec) slat (usec): min=12, max=145, avg=19.83, stdev= 5.03 clat (nsec): min=1742, max=20959k, avg=91639.29, stdev=429448.23 lat (usec): min=63, max=20975, avg=111.63, stdev=429.50 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 64], 20.00th=[ 66], | 30.00th=[ 68], 40.00th=[ 70], 50.00th=[ 73], 60.00th=[ 77], | 70.00th=[ 81], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 121], 99.50th=[ 135], 99.90th=[ 8586], 99.95th=[12256], | 99.99th=[16581] bw ( KiB/s): min=256512, max=555520, per=100.00%, avg=336792.68, stdev=44379.89, samples=59 iops : min= 1002, max= 2170, avg=1315.53, stdev=173.33, samples=59 write: IOPS=331, BW=82.9MiB/s (86.9MB/s)(2487MiB/30001msec) slat (usec): min=25, max=163, avg=43.23, stdev= 7.84 clat (usec): min=842, max=69966, avg=2521.84, stdev=2408.21 lat (usec): min=884, max=70011, avg=2565.26, stdev=2408.15 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1045], 10.00th=[ 1123], 20.00th=[ 1532], | 30.00th=[ 1860], 40.00th=[ 1975], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2704], 90.00th=[ 4015], 95.00th=[ 4883], | 99.00th=[10683], 99.50th=[12387], 99.90th=[45876], 99.95th=[53216], | 99.99th=[69731] bw ( KiB/s): min=67072, max=147968, per=100.00%, avg=85081.58, stdev=10217.63, samples=59 iops : min= 262, max= 578, avg=332.31, stdev=39.91, samples=59 lat (usec) : 2=0.01%, 20=0.01%, 50=0.06%, 100=75.28%, 250=4.35% lat (usec) : 500=0.02%, 750=0.01%, 1000=0.59% lat (msec) : 2=7.85%, 4=9.69%, 10=1.59%, 20=0.52%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.68%, sys=4.71%, ctx=49386, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39428,9948,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=329MiB/s (345MB/s), 329MiB/s-329MiB/s (345MB/s-345MB/s), io=9857MiB (10.3GB), run=30001-30001msec WRITE: bw=82.9MiB/s (86.9MB/s), 82.9MiB/s-82.9MiB/s (86.9MB/s-86.9MB/s), io=2487MiB (2608MB), run=30001-30001msec Disk stats (read/write): vdl: ios=39166/9877, merge=0/0, ticks=571/20355, in_queue=28251, util=94.59% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943471: Fri Jan 7 14:11:35 2022 read: IOPS=1318, BW=330MiB/s (346MB/s)(9889MiB/30001msec) clat (usec): min=63, max=17786, avg=105.65, stdev=380.56 lat (usec): min=64, max=17787, avg=105.79, stdev=380.56 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 77], 10.00th=[ 78], 20.00th=[ 80], | 30.00th=[ 82], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 103], 90.00th=[ 112], 95.00th=[ 120], | 99.00th=[ 143], 99.50th=[ 161], 99.90th=[ 8455], 99.95th=[10683], | 99.99th=[14091] bw ( KiB/s): min=259584, max=555520, per=100.00%, avg=338748.07, stdev=43648.69, samples=59 iops : min= 1014, max= 2170, avg=1323.22, stdev=170.49, samples=59 write: IOPS=332, BW=83.2MiB/s (87.3MB/s)(2497MiB/30001msec) clat (usec): min=856, max=61790, avg=2552.39, stdev=2479.23 lat (usec): min=891, max=61814, avg=2576.71, stdev=2478.99 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 1074], 10.00th=[ 1156], 20.00th=[ 1631], | 30.00th=[ 1909], 40.00th=[ 2024], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2671], 90.00th=[ 3621], 95.00th=[ 4817], | 99.00th=[10683], 99.50th=[12256], 99.90th=[52691], 99.95th=[54264], | 99.99th=[61604] bw ( KiB/s): min=70656, max=147968, per=100.00%, avg=85446.07, stdev=9679.35, samples=59 iops : min= 276, max= 578, avg=333.76, stdev=37.81, samples=59 lat (usec) : 100=60.35%, 250=19.34%, 500=0.02%, 750=0.01%, 1000=0.49% lat (msec) : 2=7.15%, 4=10.69%, 10=1.36%, 20=0.58%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.74%, sys=2.65%, ctx=49550, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39556,9987,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=330MiB/s (346MB/s), 330MiB/s-330MiB/s (346MB/s-346MB/s), io=9889MiB (10.4GB), run=30001-30001msec WRITE: bw=83.2MiB/s (87.3MB/s), 83.2MiB/s-83.2MiB/s (87.3MB/s-87.3MB/s), io=2497MiB (2618MB), run=30001-30001msec Disk stats (read/write): vdl: ios=39397/9938, merge=0/0, ticks=519/20540, in_queue=28325, util=94.80% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943997: Fri Jan 7 14:12:11 2022 read: IOPS=1416, BW=354MiB/s (371MB/s)(10.4GiB/30022msec) slat (usec): min=8, max=135, avg=17.55, stdev= 5.34 clat (nsec): min=1409, max=84904k, avg=132009.60, stdev=1504311.68 lat (usec): min=47, max=84929, avg=149.72, stdev=1504.30 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 66], 60.00th=[ 72], | 70.00th=[ 81], 80.00th=[ 98], 90.00th=[ 121], 95.00th=[ 143], | 99.00th=[ 258], 99.50th=[ 396], 99.90th=[31065], 99.95th=[37487], | 99.99th=[59507] bw ( KiB/s): min=293376, max=733696, per=100.00%, avg=362851.17, stdev=58877.00, samples=60 iops : min= 1146, max= 2866, avg=1417.37, stdev=230.00, samples=60 write: IOPS=357, BW=89.4MiB/s (93.7MB/s)(2683MiB/30022msec) slat (usec): min=21, max=430, avg=50.83, stdev=20.13 clat (usec): min=1720, max=111462, avg=27326.12, stdev=11127.63 lat (usec): min=1754, max=111508, avg=27377.14, stdev=11128.68 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 9], 10.00th=[ 13], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 30], | 70.00th=[ 32], 80.00th=[ 35], 90.00th=[ 40], 95.00th=[ 44], | 99.00th=[ 66], 99.50th=[ 82], 99.90th=[ 92], 99.95th=[ 93], | 99.99th=[ 108] bw ( KiB/s): min=74752, max=184832, per=99.99%, avg=91482.95, stdev=13476.38, samples=60 iops : min= 292, max= 722, avg=357.33, stdev=52.65, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=4.50%, 100=60.04% lat (usec) : 250=14.47%, 500=0.54%, 750=0.12%, 1000=0.04% lat (msec) : 2=0.01%, 4=0.02%, 10=1.38%, 20=2.64%, 50=15.81% lat (msec) : 100=0.42%, 250=0.01% cpu : usr=2.22%, sys=4.47%, ctx=49260, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=42519,10730,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=354MiB/s (371MB/s), 354MiB/s-354MiB/s (371MB/s-371MB/s), io=10.4GiB (11.1GB), run=30022-30022msec WRITE: bw=89.4MiB/s (93.7MB/s), 89.4MiB/s-89.4MiB/s (93.7MB/s-93.7MB/s), io=2683MiB (2813MB), run=30022-30022msec Disk stats (read/write): vdl: ios=42136/10631, merge=0/0, ticks=2246/285299, in_queue=296108, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944975: Fri Jan 7 14:12:48 2022 read: IOPS=1358, BW=340MiB/s (356MB/s)(9.95GiB/30001msec) clat (usec): min=52, max=49451, avg=96.06, stdev=472.99 lat (usec): min=52, max=49451, avg=96.20, stdev=473.00 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 90], 90.00th=[ 98], 95.00th=[ 105], | 99.00th=[ 126], 99.50th=[ 145], 99.90th=[ 8225], 99.95th=[11469], | 99.99th=[15795] bw ( KiB/s): min=245248, max=568320, per=100.00%, avg=348064.54, stdev=42838.31, samples=59 iops : min= 958, max= 2220, avg=1359.63, stdev=167.34, samples=59 write: IOPS=343, BW=85.8MiB/s (89.9MB/s)(2573MiB/30001msec) clat (usec): min=851, max=62755, avg=2500.80, stdev=2247.90 lat (usec): min=874, max=62766, avg=2525.40, stdev=2247.68 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2671], 90.00th=[ 3228], 95.00th=[ 4555], | 99.00th=[10552], 99.50th=[11207], 99.90th=[15139], 99.95th=[53216], | 99.99th=[61080] bw ( KiB/s): min=73728, max=151552, per=100.00%, avg=87907.80, stdev=9813.69, samples=59 iops : min= 288, max= 592, avg=343.39, stdev=38.33, samples=59 lat (usec) : 100=73.68%, 250=5.98%, 500=0.03%, 750=0.02%, 1000=0.54% lat (msec) : 2=6.83%, 4=11.24%, 10=1.16%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.66%, sys=3.24%, ctx=51072, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40769,10293,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=340MiB/s (356MB/s), 340MiB/s-340MiB/s (356MB/s-356MB/s), io=9.95GiB (10.7GB), run=30001-30001msec WRITE: bw=85.8MiB/s (89.9MB/s), 85.8MiB/s-85.8MiB/s (89.9MB/s-89.9MB/s), io=2573MiB (2698MB), run=30001-30001msec Disk stats (read/write): vdl: ios=40758/10288, merge=0/0, ticks=612/20664, in_queue=28347, util=94.22% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945602: Fri Jan 7 14:13:26 2022 read: IOPS=1422, BW=356MiB/s (373MB/s)(10.5GiB/30147msec) slat (usec): min=4, max=249, avg=13.80, stdev= 5.80 clat (nsec): min=1073, max=2564.5M, avg=2453331.69, stdev=69719902.19 lat (usec): min=45, max=2564.5k, avg=2467.29, stdev=69719.88 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 55], 40.00th=[ 58], | 50.00th=[ 62], 60.00th=[ 68], 70.00th=[ 86], | 80.00th=[ 112], 90.00th=[ 128], 95.00th=[ 157], | 99.00th=[ 1237], 99.50th=[ 2999], 99.90th=[ 977273], | 99.95th=[2499806], 99.99th=[2533360] bw ( KiB/s): min=272896, max=791040, per=100.00%, avg=365955.63, stdev=64775.27, samples=60 iops : min= 1066, max= 3090, avg=1429.48, stdev=253.02, samples=60 write: IOPS=358, BW=89.6MiB/s (93.0MB/s)(2703MiB/30147msec) slat (usec): min=19, max=256, avg=55.18, stdev=25.32 clat (msec): min=15, max=1907, avg=129.57, stdev=63.69 lat (msec): min=15, max=1907, avg=129.62, stdev=63.69 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 93], 10.00th=[ 102], 20.00th=[ 111], | 30.00th=[ 116], 40.00th=[ 121], 50.00th=[ 126], 60.00th=[ 130], | 70.00th=[ 136], 80.00th=[ 142], 90.00th=[ 155], 95.00th=[ 171], | 99.00th=[ 253], 99.50th=[ 468], 99.90th=[ 1099], 99.95th=[ 1418], | 99.99th=[ 1888] bw ( KiB/s): min=78336, max=180736, per=100.00%, avg=91869.97, stdev=13066.37, samples=60 iops : min= 306, max= 706, avg=358.83, stdev=51.05, samples=60 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=6.29%, 100=52.28% lat (usec) : 250=19.11%, 500=0.67%, 750=0.37%, 1000=0.20% lat (msec) : 2=0.31%, 4=0.41%, 10=0.08%, 20=0.06%, 50=0.53% lat (msec) : 100=1.13%, 250=18.20%, 500=0.14%, 750=0.04%, 1000=0.04% lat (msec) : 2000=0.05%, >=2000=0.05% cpu : usr=2.55%, sys=3.87%, ctx=48729, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=42886,10810,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=356MiB/s (373MB/s), 356MiB/s-356MiB/s (373MB/s-373MB/s), io=10.5GiB (11.2GB), run=30147-30147msec WRITE: bw=89.6MiB/s (93.0MB/s), 89.6MiB/s-89.6MiB/s (93.0MB/s-93.0MB/s), io=2703MiB (2834MB), run=30147-30147msec Disk stats (read/write): vdl: ios=42881/10775, merge=0/0, ticks=96181/1386606, in_queue=1499479, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946469: Fri Jan 7 14:14:02 2022 read: IOPS=1362, BW=341MiB/s (357MB/s)(9.98GiB/30001msec) clat (usec): min=50, max=24134, avg=92.81, stdev=427.85 lat (usec): min=50, max=24134, avg=92.95, stdev=427.86 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 74], 60.00th=[ 78], | 70.00th=[ 82], 80.00th=[ 87], 90.00th=[ 94], 95.00th=[ 101], | 99.00th=[ 124], 99.50th=[ 139], 99.90th=[ 8291], 99.95th=[12780], | 99.99th=[14484] bw ( KiB/s): min=253952, max=576512, per=100.00%, avg=348828.08, stdev=43944.18, samples=59 iops : min= 992, max= 2252, avg=1362.58, stdev=171.66, samples=59 write: IOPS=343, BW=85.0MiB/s (90.1MB/s)(2579MiB/30001msec) clat (usec): min=847, max=55038, avg=2507.45, stdev=2277.84 lat (usec): min=873, max=55064, avg=2531.61, stdev=2277.48 clat percentiles (usec): | 1.00th=[ 922], 5.00th=[ 1074], 10.00th=[ 1205], 20.00th=[ 1811], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2638], 90.00th=[ 3097], 95.00th=[ 4555], | 99.00th=[10552], 99.50th=[11076], 99.90th=[28967], 99.95th=[53740], | 99.99th=[53740] bw ( KiB/s): min=69632, max=152576, per=100.00%, avg=88046.58, stdev=10176.46, samples=59 iops : min= 272, max= 596, avg=343.90, stdev=39.76, samples=59 lat (usec) : 100=75.58%, 250=4.11%, 500=0.04%, 750=0.01%, 1000=0.54% lat (msec) : 2=6.79%, 4=11.33%, 10=1.07%, 20=0.52%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.71%, sys=2.96%, ctx=51206, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40880,10317,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=341MiB/s (357MB/s), 341MiB/s-341MiB/s (357MB/s-357MB/s), io=9.98GiB (10.7GB), run=30001-30001msec WRITE: bw=85.0MiB/s (90.1MB/s), 85.0MiB/s-85.0MiB/s (90.1MB/s-90.1MB/s), io=2579MiB (2705MB), run=30001-30001msec Disk stats (read/write): vdl: ios=40609/10249, merge=0/0, ticks=587/20630, in_queue=28329, util=94.74% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947444: Fri Jan 7 14:14:40 2022 read: IOPS=1421, BW=355MiB/s (373MB/s)(10.5GiB/30287msec) slat (usec): min=4, max=286, avg=13.07, stdev= 6.87 clat (nsec): min=944, max=2700.2M, avg=2568859.05, stdev=69918950.22 lat (usec): min=46, max=2700.2k, avg=2582.08, stdev=69918.91 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 55], 40.00th=[ 58], | 50.00th=[ 63], 60.00th=[ 72], 70.00th=[ 102], | 80.00th=[ 117], 90.00th=[ 153], 95.00th=[ 553], | 99.00th=[ 5932], 99.50th=[ 7504], 99.90th=[ 994051], | 99.95th=[2499806], 99.99th=[2533360] bw ( KiB/s): min=253952, max=850944, per=100.00%, avg=367297.57, stdev=73964.46, samples=60 iops : min= 992, max= 3324, avg=1434.72, stdev=288.93, samples=60 write: IOPS=358, BW=89.6MiB/s (93.0MB/s)(2715MiB/30287msec) slat (usec): min=22, max=373, avg=52.55, stdev=25.59 clat (msec): min=18, max=2971, avg=268.57, stdev=118.77 lat (msec): min=18, max=2971, avg=268.63, stdev=118.77 clat percentiles (msec): | 1.00th=[ 53], 5.00th=[ 161], 10.00th=[ 222], 20.00th=[ 239], | 30.00th=[ 247], 40.00th=[ 255], 50.00th=[ 262], 60.00th=[ 271], | 70.00th=[ 279], 80.00th=[ 292], 90.00th=[ 313], 95.00th=[ 347], | 99.00th=[ 542], 99.50th=[ 684], 99.90th=[ 2635], 99.95th=[ 2635], | 99.99th=[ 2769] bw ( KiB/s): min=73728, max=171008, per=100.00%, avg=91848.18, stdev=12483.42, samples=60 iops : min= 288, max= 668, avg=358.73, stdev=48.76, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=6.36% lat (usec) : 100=49.21%, 250=18.68%, 500=1.42%, 750=0.63%, 1000=0.41% lat (msec) : 2=0.99%, 4=0.71%, 10=1.27%, 20=0.04%, 50=0.17% lat (msec) : 100=0.57%, 250=6.17%, 500=12.92%, 750=0.26%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.08% cpu : usr=2.39%, sys=3.73%, ctx=46645, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=43040,10859,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=355MiB/s (373MB/s), 355MiB/s-355MiB/s (373MB/s-373MB/s), io=10.5GiB (11.3GB), run=30287-30287msec WRITE: bw=89.6MiB/s (93.0MB/s), 89.6MiB/s-89.6MiB/s (93.0MB/s-93.0MB/s), io=2715MiB (2847MB), run=30287-30287msec Disk stats (read/write): vdl: ios=43039/10811, merge=0/0, ticks=102523/2884549, in_queue=3013997, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948018: Fri Jan 7 14:15:17 2022 read: IOPS=1377, BW=344MiB/s (361MB/s)(10.1GiB/30001msec) clat (usec): min=51, max=20941, avg=94.88, stdev=446.85 lat (usec): min=51, max=20941, avg=95.03, stdev=446.85 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 66], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 88], 90.00th=[ 96], 95.00th=[ 104], | 99.00th=[ 135], 99.50th=[ 159], 99.90th=[ 8291], 99.95th=[12256], | 99.99th=[18482] bw ( KiB/s): min=264308, max=575483, per=100.00%, avg=356688.32, stdev=43442.86, samples=60 iops : min= 1032, max= 2247, avg=1392.75, stdev=169.66, samples=60 write: IOPS=347, BW=86.0MiB/s (91.2MB/s)(2610MiB/30001msec) clat (usec): min=791, max=61622, avg=2463.46, stdev=2351.20 lat (usec): min=817, max=61646, avg=2489.31, stdev=2351.23 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1057], 10.00th=[ 1188], 20.00th=[ 1778], | 30.00th=[ 1909], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2409], 80.00th=[ 2606], 90.00th=[ 3097], 95.00th=[ 4359], | 99.00th=[10552], 99.50th=[11076], 99.90th=[52167], 99.95th=[54789], | 99.99th=[61080] bw ( KiB/s): min=74623, max=153290, per=100.00%, avg=90066.42, stdev=10088.26, samples=60 iops : min= 291, max= 598, avg=351.40, stdev=39.38, samples=60 lat (usec) : 100=74.28%, 250=5.38%, 500=0.04%, 750=0.01%, 1000=0.62% lat (msec) : 2=6.92%, 4=11.21%, 10=1.07%, 20=0.44%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.72%, sys=3.12%, ctx=51778, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=41334,10439,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=344MiB/s (361MB/s), 344MiB/s-344MiB/s (361MB/s-361MB/s), io=10.1GiB (10.8GB), run=30001-30001msec WRITE: bw=86.0MiB/s (91.2MB/s), 86.0MiB/s-86.0MiB/s (91.2MB/s-91.2MB/s), io=2610MiB (2737MB), run=30001-30001msec Disk stats (read/write): vdl: ios=41329/10436, merge=0/0, ticks=631/20574, in_queue=28552, util=94.78% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949371: Fri Jan 7 14:15:54 2022 read: IOPS=1415, BW=354MiB/s (371MB/s)(10.5GiB/30326msec) slat (usec): min=4, max=324455, avg=536.84, stdev=4416.87 clat (msec): min=2, max=2635, avg=46.83, stdev=77.60 lat (msec): min=2, max=2635, avg=47.36, stdev=77.72 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 10], 10.00th=[ 16], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 38], 50.00th=[ 42], 60.00th=[ 45], | 70.00th=[ 50], 80.00th=[ 55], 90.00th=[ 67], 95.00th=[ 78], | 99.00th=[ 142], 99.50th=[ 351], 99.90th=[ 1036], 99.95th=[ 2534], | 99.99th=[ 2567] bw ( KiB/s): min=230912, max=901120, per=100.00%, avg=365795.52, stdev=80351.24, samples=60 iops : min= 902, max= 3520, avg=1428.87, stdev=313.88, samples=60 write: IOPS=356, BW=89.2MiB/s (93.5MB/s)(2705MiB/30326msec) slat (usec): min=27, max=300263, avg=631.78, stdev=4272.95 clat (msec): min=54, max=2633, avg=371.82, stdev=120.14 lat (msec): min=54, max=2633, avg=372.45, stdev=120.17 clat percentiles (msec): | 1.00th=[ 72], 5.00th=[ 215], 10.00th=[ 313], 20.00th=[ 338], | 30.00th=[ 351], 40.00th=[ 359], 50.00th=[ 363], 60.00th=[ 372], | 70.00th=[ 384], 80.00th=[ 401], 90.00th=[ 435], 95.00th=[ 514], | 99.00th=[ 684], 99.50th=[ 894], 99.90th=[ 1737], 99.95th=[ 2299], | 99.99th=[ 2635] bw ( KiB/s): min=74240, max=169472, per=99.84%, avg=91199.60, stdev=12416.87, samples=60 iops : min= 290, max= 662, avg=356.20, stdev=48.49, samples=60 lat (msec) : 4=0.13%, 10=4.45%, 20=5.23%, 50=47.94%, 100=20.92% lat (msec) : 250=1.55%, 500=18.64%, 750=0.94%, 1000=0.05%, 2000=0.08% lat (msec) : >=2000=0.07% cpu : usr=3.42%, sys=1.74%, ctx=5773, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42926,10821,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=354MiB/s (371MB/s), 354MiB/s-354MiB/s (371MB/s-371MB/s), io=10.5GiB (11.3GB), run=30326-30326msec WRITE: bw=89.2MiB/s (93.5MB/s), 89.2MiB/s-89.2MiB/s (93.5MB/s-93.5MB/s), io=2705MiB (2837MB), run=30326-30326msec Disk stats (read/write): vdl: ios=42923/10795, merge=0/0, ticks=120365/3538111, in_queue=3698799, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950052: Fri Jan 7 14:16:31 2022 read: IOPS=923, BW=462MiB/s (484MB/s)(13.5GiB/30001msec) clat (usec): min=103, max=22196, avg=166.89, stdev=570.83 lat (usec): min=103, max=22196, avg=167.05, stdev=570.84 clat percentiles (usec): | 1.00th=[ 113], 5.00th=[ 117], 10.00th=[ 119], 20.00th=[ 122], | 30.00th=[ 125], 40.00th=[ 129], 50.00th=[ 135], 60.00th=[ 141], | 70.00th=[ 149], 80.00th=[ 159], 90.00th=[ 176], 95.00th=[ 192], | 99.00th=[ 241], 99.50th=[ 281], 99.90th=[11731], 99.95th=[14222], | 99.99th=[20055] bw ( KiB/s): min=342016, max=644096, per=100.00%, avg=473322.75, stdev=58254.10, samples=59 iops : min= 668, max= 1258, avg=924.44, stdev=113.80, samples=59 write: IOPS=232, BW=116MiB/s (122MB/s)(3489MiB/30001msec) clat (usec): min=1404, max=62681, avg=3578.34, stdev=2601.42 lat (usec): min=1462, max=62717, avg=3628.03, stdev=2601.05 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1696], 10.00th=[ 1827], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 2900], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3490], 80.00th=[ 3818], 90.00th=[ 6128], 95.00th=[ 7242], | 99.00th=[11731], 99.50th=[14484], 99.90th=[52691], 99.95th=[54789], | 99.99th=[62653] bw ( KiB/s): min=91318, max=180224, per=100.00%, avg=119229.63, stdev=13550.82, samples=59 iops : min= 178, max= 352, avg=232.85, stdev=26.47, samples=59 lat (usec) : 250=79.22%, 500=0.46%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.45%, 4=14.24%, 10=2.94%, 20=0.64%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.72%, sys=2.72%, ctx=34686, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27699,6977,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=462MiB/s (484MB/s), 462MiB/s-462MiB/s (484MB/s-484MB/s), io=13.5GiB (14.5GB), run=30001-30001msec WRITE: bw=116MiB/s (122MB/s), 116MiB/s-116MiB/s (122MB/s-122MB/s), io=3489MiB (3658MB), run=30001-30001msec Disk stats (read/write): vdl: ios=27537/6937, merge=0/0, ticks=668/21093, in_queue=28412, util=95.11% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950871: Fri Jan 7 14:17:08 2022 read: IOPS=978, BW=489MiB/s (513MB/s)(14.3GiB/30005msec) slat (usec): min=16, max=110, avg=25.48, stdev= 5.75 clat (nsec): min=1970, max=20012k, avg=114585.39, stdev=572474.10 lat (usec): min=81, max=20055, avg=140.23, stdev=572.58 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 74], 10.00th=[ 75], 20.00th=[ 77], | 30.00th=[ 79], 40.00th=[ 81], 50.00th=[ 84], 60.00th=[ 88], | 70.00th=[ 94], 80.00th=[ 100], 90.00th=[ 110], 95.00th=[ 120], | 99.00th=[ 159], 99.50th=[ 192], 99.90th=[12649], 99.95th=[14615], | 99.99th=[18482] bw ( KiB/s): min=359424, max=621568, per=100.00%, avg=500953.92, stdev=57544.50, samples=60 iops : min= 702, max= 1214, avg=978.40, stdev=112.41, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3706MiB/30005msec) slat (usec): min=37, max=293, avg=73.31, stdev=20.34 clat (usec): min=1410, max=62683, avg=3411.08, stdev=2400.20 lat (usec): min=1461, max=62766, avg=3484.59, stdev=2399.78 clat percentiles (usec): | 1.00th=[ 1500], 5.00th=[ 1631], 10.00th=[ 1713], 20.00th=[ 2376], | 30.00th=[ 2704], 40.00th=[ 2933], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3458], 80.00th=[ 3752], 90.00th=[ 4555], 95.00th=[ 6456], | 99.00th=[11469], 99.50th=[11994], 99.90th=[53216], 99.95th=[54789], | 99.99th=[62653] bw ( KiB/s): min=111616, max=163840, per=100.00%, avg=126489.10, stdev=8206.57, samples=60 iops : min= 218, max= 320, avg=247.03, stdev=16.04, samples=60 lat (usec) : 2=0.01%, 50=0.02%, 100=63.74%, 250=15.82%, 500=0.05% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=2.87%, 4=14.25%, 10=2.74%, 20=0.46%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=2.07%, sys=4.03%, ctx=36781, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29351,7412,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=489MiB/s (513MB/s), 489MiB/s-489MiB/s (513MB/s-513MB/s), io=14.3GiB (15.4GB), run=30005-30005msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3706MiB (3886MB), run=30005-30005msec Disk stats (read/write): vdl: ios=29153/7365, merge=0/0, ticks=712/21349, in_queue=28188, util=94.34% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951821: Fri Jan 7 14:17:45 2022 read: IOPS=977, BW=489MiB/s (512MB/s)(14.3GiB/30003msec) clat (usec): min=78, max=20865, avg=134.30, stdev=550.92 lat (usec): min=78, max=20865, avg=134.45, stdev=550.92 clat percentiles (usec): | 1.00th=[ 87], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 96], | 30.00th=[ 98], 40.00th=[ 100], 50.00th=[ 104], 60.00th=[ 110], | 70.00th=[ 117], 80.00th=[ 124], 90.00th=[ 135], 95.00th=[ 145], | 99.00th=[ 174], 99.50th=[ 198], 99.90th=[12387], 99.95th=[14222], | 99.99th=[15926] bw ( KiB/s): min=371712, max=633856, per=100.00%, avg=500429.30, stdev=56846.91, samples=60 iops : min= 726, max= 1238, avg=977.40, stdev=111.03, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3702MiB/30003msec) clat (usec): min=1473, max=61885, avg=3462.73, stdev=2419.68 lat (usec): min=1501, max=61947, avg=3511.85, stdev=2419.13 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3785], 90.00th=[ 4555], 95.00th=[ 6456], | 99.00th=[11600], 99.50th=[12125], 99.90th=[53740], 99.95th=[54264], | 99.99th=[62129] bw ( KiB/s): min=104448, max=171008, per=100.00%, avg=126331.50, stdev=9142.86, samples=60 iops : min= 204, max= 334, avg=246.73, stdev=17.86, samples=60 lat (usec) : 100=31.17%, 250=48.45%, 500=0.04%, 1000=0.01% lat (msec) : 2=2.86%, 4=14.11%, 10=2.80%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.72%, sys=3.01%, ctx=36733, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29321,7403,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=489MiB/s (512MB/s), 489MiB/s-489MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30003-30003msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3702MiB (3881MB), run=30003-30003msec Disk stats (read/write): vdl: ios=29076/7348, merge=0/0, ticks=695/21508, in_queue=28293, util=94.70% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952403: Fri Jan 7 14:18:22 2022 read: IOPS=989, BW=495MiB/s (519MB/s)(14.5GiB/30035msec) slat (usec): min=6, max=417, avg=17.00, stdev= 7.10 clat (usec): min=12, max=61440, avg=178.50, stdev=1594.56 lat (usec): min=67, max=61454, avg=195.67, stdev=1594.57 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 73], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 86], | 70.00th=[ 98], 80.00th=[ 167], 90.00th=[ 192], 95.00th=[ 208], | 99.00th=[ 306], 99.50th=[ 537], 99.90th=[32375], 99.95th=[42206], | 99.99th=[49546] bw ( KiB/s): min=346112, max=760832, per=100.00%, avg=507292.43, stdev=65515.85, samples=60 iops : min= 676, max= 1486, avg=990.78, stdev=127.96, samples=60 write: IOPS=249, BW=125MiB/s (131MB/s)(3749MiB/30035msec) slat (usec): min=36, max=346, avg=90.30, stdev=42.19 clat (msec): min=2, max=127, avg=39.18, stdev=15.89 lat (msec): min=2, max=127, avg=39.27, stdev=15.89 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 19], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 43], | 70.00th=[ 46], 80.00th=[ 52], 90.00th=[ 58], 95.00th=[ 66], | 99.00th=[ 86], 99.50th=[ 102], 99.90th=[ 115], 99.95th=[ 123], | 99.99th=[ 128] bw ( KiB/s): min=106496, max=207872, per=99.99%, avg=127807.80, stdev=13096.86, samples=60 iops : min= 208, max= 406, avg=249.60, stdev=25.58, samples=60 lat (usec) : 20=0.01%, 50=0.04%, 100=56.62%, 250=21.69%, 500=1.08% lat (usec) : 750=0.11%, 1000=0.05% lat (msec) : 2=0.08%, 4=0.01%, 10=0.61%, 20=1.55%, 50=13.79% lat (msec) : 100=4.24%, 250=0.11% cpu : usr=2.75%, sys=2.93%, ctx=35866, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29721,7498,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=495MiB/s (519MB/s), 495MiB/s-495MiB/s (519MB/s-519MB/s), io=14.5GiB (15.6GB), run=30035-30035msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3749MiB (3931MB), run=30035-30035msec Disk stats (read/write): vdl: ios=29595/7456, merge=0/0, ticks=2085/287650, in_queue=296698, util=99.76% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953232: Fri Jan 7 14:18:59 2022 read: IOPS=995, BW=498MiB/s (522MB/s)(14.6GiB/30003msec) clat (usec): min=76, max=56883, avg=132.81, stdev=614.57 lat (usec): min=77, max=56884, avg=132.96, stdev=614.57 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 97], 40.00th=[ 100], 50.00th=[ 103], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 122], 90.00th=[ 131], 95.00th=[ 139], | 99.00th=[ 163], 99.50th=[ 184], 99.90th=[11076], 99.95th=[13566], | 99.99th=[20055] bw ( KiB/s): min=372736, max=619520, per=100.00%, avg=510036.52, stdev=47073.43, samples=60 iops : min= 728, max= 1210, avg=996.13, stdev=91.95, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3772MiB/30003msec) clat (usec): min=1468, max=62836, avg=3393.94, stdev=2188.49 lat (usec): min=1520, max=62881, avg=3442.12, stdev=2187.98 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1663], 10.00th=[ 1745], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3294], | 70.00th=[ 3458], 80.00th=[ 3720], 90.00th=[ 4359], 95.00th=[ 6194], | 99.00th=[11469], 99.50th=[12125], 99.90th=[16450], 99.95th=[54789], | 99.99th=[62653] bw ( KiB/s): min=108544, max=178176, per=100.00%, avg=128746.88, stdev=9583.00, samples=60 iops : min= 212, max= 348, avg=251.43, stdev=18.71, samples=60 lat (usec) : 100=31.98%, 250=47.68%, 500=0.02%, 1000=0.01% lat (msec) : 2=2.75%, 4=14.71%, 10=2.43%, 20=0.43%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.94%, sys=2.77%, ctx=37435, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29882,7544,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=14.6GiB (15.7GB), run=30003-30003msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3772MiB (3955MB), run=30003-30003msec Disk stats (read/write): vdl: ios=29712/7491, merge=0/0, ticks=709/21386, in_queue=28213, util=94.43% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954259: Fri Jan 7 14:19:36 2022 read: IOPS=984, BW=492MiB/s (516MB/s)(14.5GiB/30190msec) slat (usec): min=5, max=221, avg=15.18, stdev= 6.88 clat (usec): min=34, max=339356, avg=391.56, stdev=4950.79 lat (usec): min=70, max=339368, avg=406.90, stdev=4950.77 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 74], 40.00th=[ 77], 50.00th=[ 80], 60.00th=[ 87], | 70.00th=[ 106], 80.00th=[ 184], 90.00th=[ 204], 95.00th=[ 281], | 99.00th=[ 4424], 99.50th=[ 6849], 99.90th=[ 84411], 99.95th=[115868], | 99.99th=[227541] bw ( KiB/s): min=356352, max=891904, per=100.00%, avg=507402.30, stdev=75368.92, samples=60 iops : min= 696, max= 1742, avg=990.98, stdev=147.21, samples=60 write: IOPS=248, BW=124MiB/s (130MB/s)(3752MiB/30190msec) slat (usec): min=35, max=394, avg=95.85, stdev=48.22 clat (msec): min=15, max=2822, avg=199.43, stdev=109.15 lat (msec): min=15, max=2822, avg=199.53, stdev=109.15 clat percentiles (msec): | 1.00th=[ 54], 5.00th=[ 72], 10.00th=[ 87], 20.00th=[ 108], | 30.00th=[ 134], 40.00th=[ 169], 50.00th=[ 203], 60.00th=[ 226], | 70.00th=[ 247], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330], | 99.00th=[ 451], 99.50th=[ 609], 99.90th=[ 1284], 99.95th=[ 1905], | 99.99th=[ 2836] bw ( KiB/s): min=110592, max=199680, per=99.98%, avg=127218.45, stdev=11714.08, samples=60 iops : min= 216, max= 390, avg=248.45, stdev=22.89, samples=60 lat (usec) : 50=0.05%, 100=54.26%, 250=21.02%, 500=1.59%, 750=0.55% lat (usec) : 1000=0.31% lat (msec) : 2=0.72%, 4=0.53%, 10=0.66%, 20=0.01%, 50=0.18% lat (msec) : 100=3.32%, 250=11.11%, 500=5.56%, 750=0.07%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.01% cpu : usr=2.74%, sys=2.80%, ctx=34693, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=29731,7503,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=492MiB/s (516MB/s), 492MiB/s-492MiB/s (516MB/s-516MB/s), io=14.5GiB (15.6GB), run=30190-30190msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3752MiB (3934MB), run=30190-30190msec Disk stats (read/write): vdl: ios=29730/7454, merge=0/0, ticks=7153/1472051, in_queue=1497981, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954783: Fri Jan 7 14:20:13 2022 read: IOPS=981, BW=491MiB/s (515MB/s)(14.4GiB/30006msec) clat (usec): min=79, max=19294, avg=135.83, stdev=562.49 lat (usec): min=79, max=19294, avg=135.98, stdev=562.49 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 102], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 118], 80.00th=[ 125], 90.00th=[ 135], 95.00th=[ 143], | 99.00th=[ 176], 99.50th=[ 202], 99.90th=[12780], 99.95th=[14353], | 99.99th=[16581] bw ( KiB/s): min=336869, max=653312, per=100.00%, avg=504323.80, stdev=62018.36, samples=60 iops : min= 657, max= 1276, avg=984.63, stdev=121.16, samples=60 write: IOPS=247, BW=124MiB/s (130MB/s)(3717MiB/30006msec) clat (usec): min=1460, max=61526, avg=3440.53, stdev=2361.14 lat (usec): min=1511, max=61579, avg=3489.35, stdev=2360.53 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3785], 90.00th=[ 4490], 95.00th=[ 6456], | 99.00th=[11338], 99.50th=[11863], 99.90th=[53216], 99.95th=[54264], | 99.99th=[61604] bw ( KiB/s): min=106108, max=186368, per=100.00%, avg=127254.67, stdev=10867.08, samples=60 iops : min= 207, max= 364, avg=248.15, stdev=21.29, samples=60 lat (usec) : 100=27.44%, 250=52.18%, 500=0.05%, 750=0.01% lat (msec) : 2=2.82%, 4=14.29%, 10=2.69%, 20=0.49%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.79%, sys=2.98%, ctx=36898, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29455,7434,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=491MiB/s (515MB/s), 491MiB/s-491MiB/s (515MB/s-515MB/s), io=14.4GiB (15.4GB), run=30006-30006msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3717MiB (3898MB), run=30006-30006msec Disk stats (read/write): vdl: ios=29453/7432, merge=0/0, ticks=711/21560, in_queue=28457, util=94.49% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956009: Fri Jan 7 14:20:50 2022 read: IOPS=996, BW=498MiB/s (522MB/s)(14.8GiB/30425msec) slat (usec): min=5, max=411, avg=14.53, stdev= 9.40 clat (nsec): min=914, max=516770k, avg=848854.91, stdev=6528067.60 lat (usec): min=69, max=516782, avg=863.53, stdev=6528.00 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 71], 20.00th=[ 72], | 30.00th=[ 74], 40.00th=[ 76], 50.00th=[ 80], 60.00th=[ 89], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 441], 95.00th=[ 4178], | 99.00th=[ 13173], 99.50th=[ 16188], 99.90th=[100140], 99.95th=[122160], | 99.99th=[179307] bw ( KiB/s): min=71680, max=1192960, per=100.00%, avg=517463.20, stdev=134097.44, samples=60 iops : min= 140, max= 2330, avg=1010.62, stdev=261.92, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3823MiB/30425msec) slat (usec): min=35, max=586, avg=94.76, stdev=48.41 clat (msec): min=16, max=2961, avg=394.29, stdev=159.01 lat (msec): min=16, max=2961, avg=394.38, stdev=159.00 clat percentiles (msec): | 1.00th=[ 80], 5.00th=[ 201], 10.00th=[ 262], 20.00th=[ 292], | 30.00th=[ 321], 40.00th=[ 355], 50.00th=[ 393], 60.00th=[ 422], | 70.00th=[ 447], 80.00th=[ 468], 90.00th=[ 514], 95.00th=[ 600], | 99.00th=[ 827], 99.50th=[ 902], 99.90th=[ 2467], 99.95th=[ 2635], | 99.99th=[ 2970] bw ( KiB/s): min=105472, max=211968, per=100.00%, avg=128815.40, stdev=14152.18, samples=60 iops : min= 206, max= 414, avg=251.52, stdev=27.64, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 50=0.03%, 100=51.15%, 250=18.60%, 500=2.46%, 750=1.05% lat (usec) : 1000=0.53% lat (msec) : 2=1.09%, 4=0.91%, 10=2.42%, 20=1.41%, 50=0.08% lat (msec) : 100=0.38%, 250=1.36%, 500=15.95%, 750=2.21%, 1000=0.32% lat (msec) : 2000=0.03%, >=2000=0.03% cpu : usr=2.77%, sys=2.58%, ctx=32648, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30318,7646,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=14.8GiB (15.9GB), run=30425-30425msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3823MiB (4009MB), run=30425-30425msec Disk stats (read/write): vdl: ios=30318/7564, merge=0/0, ticks=18679/2951474, in_queue=3016716, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956759: Fri Jan 7 14:21:27 2022 read: IOPS=977, BW=489MiB/s (512MB/s)(14.3GiB/30004msec) clat (usec): min=77, max=22395, avg=133.65, stdev=601.73 lat (usec): min=77, max=22395, avg=133.80, stdev=601.73 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 90], 10.00th=[ 92], 20.00th=[ 94], | 30.00th=[ 96], 40.00th=[ 98], 50.00th=[ 101], 60.00th=[ 105], | 70.00th=[ 113], 80.00th=[ 121], 90.00th=[ 130], 95.00th=[ 139], | 99.00th=[ 174], 99.50th=[ 200], 99.90th=[13042], 99.95th=[15008], | 99.99th=[18482] bw ( KiB/s): min=400384, max=635126, per=100.00%, avg=500450.95, stdev=51280.46, samples=60 iops : min= 782, max= 1240, avg=977.43, stdev=100.13, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3702MiB/30004msec) clat (usec): min=1448, max=55746, avg=3466.29, stdev=2368.00 lat (usec): min=1495, max=55806, avg=3514.64, stdev=2367.54 clat percentiles (usec): | 1.00th=[ 1532], 5.00th=[ 1647], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2769], 40.00th=[ 2999], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3785], 90.00th=[ 4490], 95.00th=[ 6521], | 99.00th=[11600], 99.50th=[12256], 99.90th=[47449], 99.95th=[54264], | 99.99th=[55837] bw ( KiB/s): min=101376, max=176128, per=100.00%, avg=126336.13, stdev=10764.41, samples=60 iops : min= 198, max= 344, avg=246.73, stdev=21.02, samples=60 lat (usec) : 100=38.69%, 250=40.91%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.76%, 4=14.40%, 10=2.60%, 20=0.55%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.96%, sys=2.59%, ctx=36734, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29321,7403,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=489MiB/s (512MB/s), 489MiB/s-489MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30004-30004msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3702MiB (3881MB), run=30004-30004msec Disk stats (read/write): vdl: ios=29062/7345, merge=0/0, ticks=761/21571, in_queue=28309, util=94.66% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957729: Fri Jan 7 14:22:04 2022 read: IOPS=993, BW=497MiB/s (521MB/s)(14.8GiB/30525msec) slat (usec): min=6, max=468028, avg=761.52, stdev=6494.32 clat (msec): min=8, max=698, avg=64.56, stdev=57.69 lat (msec): min=8, max=699, avg=65.32, stdev=58.12 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 17], 10.00th=[ 22], 20.00th=[ 35], | 30.00th=[ 44], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 64], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 95], 95.00th=[ 110], | 99.00th=[ 481], 99.50th=[ 506], 99.90th=[ 584], 99.95th=[ 600], | 99.99th=[ 617] bw ( KiB/s): min=44032, max=1231872, per=100.00%, avg=516710.40, stdev=189321.45, samples=60 iops : min= 86, max= 2406, avg=1009.20, stdev=369.77, samples=60 write: IOPS=250, BW=125MiB/s (131MB/s)(3828MiB/30525msec) slat (usec): min=124, max=452073, avg=889.28, stdev=6103.26 clat (msec): min=37, max=3809, avg=537.40, stdev=229.10 lat (msec): min=46, max=3809, avg=538.29, stdev=228.95 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 279], 10.00th=[ 359], 20.00th=[ 414], | 30.00th=[ 451], 40.00th=[ 489], 50.00th=[ 523], 60.00th=[ 558], | 70.00th=[ 584], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 844], | 99.00th=[ 1536], 99.50th=[ 2165], 99.90th=[ 2903], 99.95th=[ 2903], | 99.99th=[ 3809] bw ( KiB/s): min= 1024, max=224256, per=98.25%, avg=126153.44, stdev=42469.39, samples=61 iops : min= 2, max= 438, avg=246.39, stdev=82.95, samples=61 lat (msec) : 10=0.03%, 20=6.89%, 50=22.93%, 100=43.97%, 250=5.81% lat (msec) : 500=8.64%, 750=10.13%, 1000=1.17%, 2000=0.33%, >=2000=0.11% cpu : usr=4.57%, sys=1.65%, ctx=3878, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30335,7655,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=497MiB/s (521MB/s), 497MiB/s-497MiB/s (521MB/s-521MB/s), io=14.8GiB (15.9GB), run=30525-30525msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3828MiB (4013MB), run=30525-30525msec Disk stats (read/write): vdl: ios=30335/7627, merge=0/0, ticks=65000/3621923, in_queue=3720432, util=99.51% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958714: Fri Jan 7 14:22:41 2022 read: IOPS=603, BW=603MiB/s (632MB/s)(17.7GiB/30006msec) clat (usec): min=131, max=24277, avg=234.26, stdev=987.88 lat (usec): min=131, max=24277, avg=234.42, stdev=987.88 clat percentiles (usec): | 1.00th=[ 143], 5.00th=[ 147], 10.00th=[ 149], 20.00th=[ 151], | 30.00th=[ 155], 40.00th=[ 157], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 194], 90.00th=[ 210], 95.00th=[ 227], | 99.00th=[ 314], 99.50th=[ 424], 99.90th=[17695], 99.95th=[18482], | 99.99th=[22676] bw ( KiB/s): min=374784, max=860160, per=100.00%, avg=617766.40, stdev=92143.91, samples=60 iops : min= 366, max= 840, avg=603.28, stdev=89.98, samples=60 write: IOPS=152, BW=152MiB/s (160MB/s)(4571MiB/30006msec) clat (usec): min=2557, max=66059, avg=5529.49, stdev=3153.75 lat (usec): min=2639, max=66119, avg=5627.55, stdev=3152.46 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3458], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[11338], | 99.00th=[16319], 99.50th=[17695], 99.90th=[26084], 99.95th=[57410], | 99.99th=[65799] bw ( KiB/s): min=129024, max=217088, per=100.00%, avg=155994.67, stdev=13349.81, samples=60 iops : min= 126, max= 212, avg=152.33, stdev=13.04, samples=60 lat (usec) : 250=77.74%, 500=1.72%, 750=0.03% lat (msec) : 4=5.08%, 10=13.64%, 20=1.72%, 50=0.05%, 100=0.02% cpu : usr=2.03%, sys=2.38%, ctx=22681, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18098,4571,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=603MiB/s (632MB/s), 603MiB/s-603MiB/s (632MB/s-632MB/s), io=17.7GiB (18.0GB), run=30006-30006msec WRITE: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=4571MiB (4793MB), run=30006-30006msec Disk stats (read/write): vdl: ios=18085/4567, merge=0/0, ticks=1097/22662, in_queue=28574, util=95.01% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959293: Fri Jan 7 14:23:18 2022 read: IOPS=603, BW=604MiB/s (633MB/s)(17.7GiB/30005msec) slat (usec): min=28, max=126, avg=39.20, stdev= 7.82 clat (usec): min=74, max=25489, avg=205.86, stdev=1013.70 lat (usec): min=136, max=25523, avg=245.26, stdev=1013.93 clat percentiles (usec): | 1.00th=[ 115], 5.00th=[ 118], 10.00th=[ 120], 20.00th=[ 122], | 30.00th=[ 125], 40.00th=[ 127], 50.00th=[ 131], 60.00th=[ 137], | 70.00th=[ 149], 80.00th=[ 161], 90.00th=[ 176], 95.00th=[ 194], | 99.00th=[ 297], 99.50th=[ 445], 99.90th=[17957], 99.95th=[19268], | 99.99th=[22938] bw ( KiB/s): min=452608, max=790528, per=100.00%, avg=618570.95, stdev=83535.75, samples=60 iops : min= 442, max= 772, avg=604.07, stdev=81.58, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4577MiB/30005msec) slat (usec): min=68, max=427, avg=129.08, stdev=27.50 clat (usec): min=2507, max=65543, avg=5446.53, stdev=3088.70 lat (usec): min=2627, max=65661, avg=5575.84, stdev=3087.58 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2704], 20.00th=[ 3392], | 30.00th=[ 4293], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5997], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[15926], 99.50th=[16909], 99.90th=[50594], 99.95th=[55837], | 99.99th=[65799] bw ( KiB/s): min=122880, max=217088, per=100.00%, avg=156205.07, stdev=12630.84, samples=60 iops : min= 120, max= 212, avg=152.53, stdev=12.33, samples=60 lat (usec) : 100=0.03%, 250=78.58%, 500=0.84%, 750=0.04% lat (msec) : 2=0.01%, 4=5.29%, 10=13.45%, 20=1.70%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=2.06%, sys=3.41%, ctx=22709, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18121,4577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=17.7GiB (19.0GB), run=30005-30005msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4577MiB (4799MB), run=30005-30005msec Disk stats (read/write): vdl: ios=17981/4541, merge=0/0, ticks=1138/22345, in_queue=28522, util=95.45% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960094: Fri Jan 7 14:23:55 2022 read: IOPS=607, BW=607MiB/s (637MB/s)(17.8GiB/30010msec) clat (usec): min=129, max=22399, avg=230.06, stdev=978.86 lat (usec): min=129, max=22399, avg=230.22, stdev=978.86 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 151], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 206], 95.00th=[ 225], | 99.00th=[ 306], 99.50th=[ 523], 99.90th=[17695], 99.95th=[19006], | 99.99th=[22152] bw ( KiB/s): min=425984, max=802816, per=100.00%, avg=622118.22, stdev=83070.83, samples=60 iops : min= 416, max= 784, avg=607.53, stdev=81.12, samples=60 write: IOPS=153, BW=154MiB/s (161MB/s)(4608MiB/30010msec) clat (usec): min=2525, max=61304, avg=5496.43, stdev=3061.29 lat (usec): min=2630, max=61410, avg=5593.04, stdev=3060.17 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3523], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5145], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6063], 90.00th=[ 8291], 95.00th=[11076], | 99.00th=[16188], 99.50th=[17171], 99.90th=[54264], 99.95th=[55837], | 99.99th=[61080] bw ( KiB/s): min=137216, max=210944, per=100.00%, avg=157262.78, stdev=11071.91, samples=60 iops : min= 134, max= 206, avg=153.57, stdev=10.81, samples=60 lat (usec) : 250=77.85%, 500=1.55%, 750=0.07% lat (msec) : 2=0.01%, 4=4.92%, 10=13.92%, 20=1.62%, 50=0.03% lat (msec) : 100=0.02% cpu : usr=2.04%, sys=2.35%, ctx=22838, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18225,4608,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=607MiB/s (637MB/s), 607MiB/s-607MiB/s (637MB/s-637MB/s), io=17.8GiB (19.1GB), run=30010-30010msec WRITE: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4608MiB (4832MB), run=30010-30010msec Disk stats (read/write): vdl: ios=18225/4607, merge=0/0, ticks=1092/22780, in_queue=28625, util=95.13% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961060: Fri Jan 7 14:24:32 2022 read: IOPS=573, BW=573MiB/s (601MB/s)(16.8GiB/30068msec) slat (usec): min=9, max=110, avg=23.06, stdev=10.33 clat (usec): min=44, max=101372, avg=336.63, stdev=2665.21 lat (usec): min=113, max=101389, avg=359.85, stdev=2665.22 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 108], 10.00th=[ 110], 20.00th=[ 112], | 30.00th=[ 114], 40.00th=[ 117], 50.00th=[ 121], 60.00th=[ 130], | 70.00th=[ 151], 80.00th=[ 322], 90.00th=[ 343], 95.00th=[ 375], | 99.00th=[ 603], 99.50th=[ 2507], 99.90th=[45876], 99.95th=[50070], | 99.99th=[85459] bw ( KiB/s): min=372736, max=925696, per=100.00%, avg=588356.27, stdev=95724.69, samples=60 iops : min= 364, max= 904, avg=574.57, stdev=93.48, samples=60 write: IOPS=145, BW=145MiB/s (152MB/s)(4371MiB/30068msec) slat (usec): min=66, max=646, avg=169.78, stdev=87.77 clat (msec): min=3, max=198, avg=67.18, stdev=26.56 lat (msec): min=3, max=198, avg=67.35, stdev=26.56 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 19], 10.00th=[ 26], 20.00th=[ 46], | 30.00th=[ 58], 40.00th=[ 65], 50.00th=[ 70], 60.00th=[ 75], | 70.00th=[ 81], 80.00th=[ 88], 90.00th=[ 99], 95.00th=[ 108], | 99.00th=[ 133], 99.50th=[ 140], 99.90th=[ 165], 99.95th=[ 178], | 99.99th=[ 199] bw ( KiB/s): min=124928, max=231424, per=100.00%, avg=148855.47, stdev=13037.44, samples=60 iops : min= 122, max= 226, avg=145.37, stdev=12.73, samples=60 lat (usec) : 50=0.01%, 100=0.14%, 250=62.10%, 500=16.46%, 750=0.37% lat (usec) : 1000=0.07% lat (msec) : 2=0.12%, 4=0.17%, 10=0.08%, 20=1.12%, 50=3.75% lat (msec) : 100=13.82%, 250=1.78% cpu : usr=2.64%, sys=2.13%, ctx=21175, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17237,4371,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=573MiB/s (601MB/s), 573MiB/s-573MiB/s (601MB/s-601MB/s), io=16.8GiB (18.1GB), run=30068-30068msec WRITE: bw=145MiB/s (152MB/s), 145MiB/s-145MiB/s (152MB/s-152MB/s), io=4371MiB (4583MB), run=30068-30068msec Disk stats (read/write): vdl: ios=17237/4370, merge=0/0, ticks=2767/291039, in_queue=299016, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961582: Fri Jan 7 14:25:09 2022 read: IOPS=608, BW=608MiB/s (638MB/s)(17.8GiB/30003msec) clat (usec): min=114, max=25405, avg=206.48, stdev=932.54 lat (usec): min=114, max=25405, avg=206.64, stdev=932.54 clat percentiles (usec): | 1.00th=[ 122], 5.00th=[ 127], 10.00th=[ 128], 20.00th=[ 131], | 30.00th=[ 133], 40.00th=[ 137], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 182], 95.00th=[ 194], | 99.00th=[ 260], 99.50th=[ 396], 99.90th=[16712], 99.95th=[19268], | 99.99th=[22152] bw ( KiB/s): min=444416, max=845824, per=100.00%, avg=622660.27, stdev=93587.15, samples=60 iops : min= 434, max= 826, avg=608.07, stdev=91.39, samples=60 write: IOPS=153, BW=154MiB/s (161MB/s)(4616MiB/30003msec) clat (usec): min=2521, max=62384, avg=5577.52, stdev=3145.21 lat (usec): min=2581, max=62429, avg=5674.83, stdev=3143.13 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3523], | 30.00th=[ 4490], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 8717], 95.00th=[11469], | 99.00th=[16057], 99.50th=[16909], 99.90th=[50594], 99.95th=[56361], | 99.99th=[62129] bw ( KiB/s): min=131072, max=202752, per=99.99%, avg=157525.33, stdev=11476.06, samples=60 iops : min= 128, max= 198, avg=153.83, stdev=11.21, samples=60 lat (usec) : 250=78.90%, 500=0.55% lat (msec) : 2=0.02%, 4=4.82%, 10=13.82%, 20=1.83%, 50=0.03% lat (msec) : 100=0.02% cpu : usr=1.96%, sys=1.45%, ctx=22867, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18242,4616,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=608MiB/s (638MB/s), 608MiB/s-608MiB/s (638MB/s-638MB/s), io=17.8GiB (19.1GB), run=30003-30003msec WRITE: bw=154MiB/s (161MB/s), 154MiB/s-154MiB/s (161MB/s-161MB/s), io=4616MiB (4840MB), run=30003-30003msec Disk stats (read/write): vdl: ios=18229/4610, merge=0/0, ticks=1032/23164, in_queue=28896, util=96.04% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962986: Fri Jan 7 14:25:46 2022 read: IOPS=590, BW=591MiB/s (620MB/s)(17.5GiB/30365msec) slat (usec): min=9, max=447, avg=20.05, stdev= 9.53 clat (usec): min=85, max=435687, avg=1117.97, stdev=10238.80 lat (usec): min=113, max=435704, avg=1138.19, stdev=10238.77 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 124], 60.00th=[ 135], | 70.00th=[ 204], 80.00th=[ 338], 90.00th=[ 412], 95.00th=[ 2409], | 99.00th=[ 12387], 99.50th=[ 15795], 99.90th=[173016], 99.95th=[217056], | 99.99th=[379585] bw ( KiB/s): min=344064, max=1208320, per=100.00%, avg=612394.37, stdev=132804.72, samples=60 iops : min= 336, max= 1180, avg=598.03, stdev=129.68, samples=60 write: IOPS=149, BW=149MiB/s (157MB/s)(4536MiB/30365msec) slat (usec): min=61, max=659, avg=182.01, stdev=93.66 clat (msec): min=13, max=1219, avg=329.90, stdev=175.57 lat (msec): min=13, max=1219, avg=330.08, stdev=175.56 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 126], 10.00th=[ 138], 20.00th=[ 167], | 30.00th=[ 197], 40.00th=[ 236], 50.00th=[ 292], 60.00th=[ 359], | 70.00th=[ 426], 80.00th=[ 485], 90.00th=[ 575], 95.00th=[ 651], | 99.00th=[ 802], 99.50th=[ 869], 99.90th=[ 1070], 99.95th=[ 1083], | 99.99th=[ 1217] bw ( KiB/s): min=92160, max=241664, per=100.00%, avg=153151.08, stdev=24061.27, samples=60 iops : min= 90, max= 236, avg=149.55, stdev=23.50, samples=60 lat (usec) : 100=0.16%, 250=56.81%, 500=15.66%, 750=1.00%, 1000=0.54% lat (msec) : 2=1.42%, 4=1.20%, 10=1.74%, 20=0.97%, 50=0.07% lat (msec) : 100=0.40%, 250=8.41%, 500=7.98%, 750=3.29%, 1000=0.34% lat (msec) : 2000=0.03% cpu : usr=2.94%, sys=1.94%, ctx=20182, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=17942,4536,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=591MiB/s (620MB/s), 591MiB/s-591MiB/s (620MB/s-620MB/s), io=17.5GiB (18.8GB), run=30365-30365msec WRITE: bw=149MiB/s (157MB/s), 149MiB/s-149MiB/s (157MB/s-157MB/s), io=4536MiB (4756MB), run=30365-30365msec Disk stats (read/write): vdl: ios=17942/4517, merge=0/0, ticks=15430/1475666, in_queue=1504308, util=99.63% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963567: Fri Jan 7 14:26:23 2022 read: IOPS=605, BW=606MiB/s (635MB/s)(17.8GiB/30005msec) clat (usec): min=124, max=25561, avg=224.83, stdev=973.66 lat (usec): min=124, max=25561, avg=224.99, stdev=973.66 clat percentiles (usec): | 1.00th=[ 133], 5.00th=[ 139], 10.00th=[ 141], 20.00th=[ 143], | 30.00th=[ 147], 40.00th=[ 149], 50.00th=[ 153], 60.00th=[ 159], | 70.00th=[ 172], 80.00th=[ 186], 90.00th=[ 200], 95.00th=[ 215], | 99.00th=[ 281], 99.50th=[ 474], 99.90th=[17695], 99.95th=[18744], | 99.99th=[22676] bw ( KiB/s): min=391168, max=831488, per=100.00%, avg=620517.57, stdev=94222.62, samples=60 iops : min= 382, max= 812, avg=605.97, stdev=92.01, samples=60 write: IOPS=153, BW=153MiB/s (161MB/s)(4599MiB/30005msec) clat (usec): min=2509, max=65778, avg=5529.31, stdev=3145.71 lat (usec): min=2615, max=65874, avg=5626.32, stdev=3144.72 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3556], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5145], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6063], 90.00th=[ 8455], 95.00th=[11338], | 99.00th=[16188], 99.50th=[17171], 99.90th=[54789], 99.95th=[55837], | 99.99th=[65799] bw ( KiB/s): min=131072, max=219136, per=100.00%, avg=156955.23, stdev=13193.88, samples=60 iops : min= 128, max= 214, avg=153.27, stdev=12.89, samples=60 lat (usec) : 250=78.44%, 500=0.99%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.01%, 4=4.84%, 10=13.90%, 20=1.73%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.84%, sys=2.28%, ctx=22791, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18178,4599,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=606MiB/s (635MB/s), 606MiB/s-606MiB/s (635MB/s-635MB/s), io=17.8GiB (19.1GB), run=30005-30005msec WRITE: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=4599MiB (4822MB), run=30005-30005msec Disk stats (read/write): vdl: ios=18042/4561, merge=0/0, ticks=1085/22685, in_queue=28521, util=95.32% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964381: Fri Jan 7 14:27:00 2022 read: IOPS=590, BW=591MiB/s (619MB/s)(17.7GiB/30715msec) slat (usec): min=9, max=690, avg=19.40, stdev=20.23 clat (usec): min=68, max=892790, avg=2842.67, stdev=15527.35 lat (usec): min=113, max=892815, avg=2862.23, stdev=15527.63 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 110], 10.00th=[ 111], 20.00th=[ 115], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 133], 60.00th=[ 225], | 70.00th=[ 343], 80.00th=[ 523], 90.00th=[ 6194], 95.00th=[ 18220], | 99.00th=[ 36439], 99.50th=[ 42730], 99.90th=[170918], 99.95th=[191890], | 99.99th=[775947] bw ( KiB/s): min=34816, max=1460224, per=100.00%, avg=619202.40, stdev=227697.17, samples=60 iops : min= 34, max= 1426, avg=604.68, stdev=222.36, samples=60 write: IOPS=149, BW=149MiB/s (156MB/s)(4584MiB/30715msec) slat (usec): min=62, max=2011, avg=202.52, stdev=115.50 clat (msec): min=29, max=1887, avg=658.19, stdev=262.74 lat (msec): min=30, max=1887, avg=658.39, stdev=262.73 clat percentiles (msec): | 1.00th=[ 123], 5.00th=[ 305], 10.00th=[ 384], 20.00th=[ 468], | 30.00th=[ 510], 40.00th=[ 550], 50.00th=[ 609], 60.00th=[ 684], | 70.00th=[ 768], 80.00th=[ 835], 90.00th=[ 969], 95.00th=[ 1150], | 99.00th=[ 1502], 99.50th=[ 1737], 99.90th=[ 1804], 99.95th=[ 1804], | 99.99th=[ 1888] bw ( KiB/s): min= 2048, max=247808, per=98.53%, avg=150583.67, stdev=52791.13, samples=61 iops : min= 2, max= 242, avg=147.05, stdev=51.55, samples=61 lat (usec) : 100=0.08%, 250=48.42%, 500=15.18%, 750=1.80%, 1000=1.32% lat (msec) : 2=2.08%, 4=1.83%, 10=2.55%, 20=3.11%, 50=3.16% lat (msec) : 100=0.12%, 250=0.77%, 500=4.97%, 750=8.04%, 1000=4.75% lat (msec) : 2000=1.82% cpu : usr=3.28%, sys=1.73%, ctx=17561, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18140,4584,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=591MiB/s (619MB/s), 591MiB/s-591MiB/s (619MB/s-619MB/s), io=17.7GiB (19.0GB), run=30715-30715msec WRITE: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=4584MiB (4807MB), run=30715-30715msec Disk stats (read/write): vdl: ios=18140/4583, merge=0/0, ticks=42087/2975438, in_queue=3025140, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965449: Fri Jan 7 14:27:37 2022 read: IOPS=605, BW=606MiB/s (635MB/s)(17.7GiB/30003msec) clat (usec): min=131, max=22697, avg=229.25, stdev=967.72 lat (usec): min=131, max=22697, avg=229.40, stdev=967.72 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 190], 90.00th=[ 204], 95.00th=[ 219], | 99.00th=[ 285], 99.50th=[ 416], 99.90th=[17695], 99.95th=[19006], | 99.99th=[22152] bw ( KiB/s): min=417792, max=798720, per=100.00%, avg=620320.32, stdev=92393.43, samples=60 iops : min= 408, max= 780, avg=605.77, stdev=90.25, samples=60 write: IOPS=153, BW=153MiB/s (161MB/s)(4596MiB/30003msec) clat (usec): min=2527, max=59154, avg=5515.56, stdev=3092.29 lat (usec): min=2623, max=59208, avg=5612.47, stdev=3091.44 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3490], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5145], 60.00th=[ 5407], | 70.00th=[ 5669], 80.00th=[ 6128], 90.00th=[ 8717], 95.00th=[11207], | 99.00th=[15926], 99.50th=[17171], 99.90th=[54264], 99.95th=[56361], | 99.99th=[58983] bw ( KiB/s): min=126976, max=208896, per=99.99%, avg=156847.23, stdev=12363.09, samples=60 iops : min= 124, max= 204, avg=153.17, stdev=12.08, samples=60 lat (usec) : 250=78.45%, 500=1.01%, 750=0.02%, 1000=0.01% lat (msec) : 4=5.01%, 10=13.67%, 20=1.79%, 50=0.03%, 100=0.02% cpu : usr=1.96%, sys=2.38%, ctx=22781, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18173,4596,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=606MiB/s (635MB/s), 606MiB/s-606MiB/s (635MB/s-635MB/s), io=17.7GiB (19.1GB), run=30003-30003msec WRITE: bw=153MiB/s (161MB/s), 153MiB/s-153MiB/s (161MB/s-161MB/s), io=4596MiB (4819MB), run=30003-30003msec Disk stats (read/write): vdl: ios=18128/4578, merge=0/0, ticks=1075/22676, in_queue=28586, util=94.86% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966064: Fri Jan 7 14:28:15 2022 read: IOPS=592, BW=593MiB/s (622MB/s)(17.8GiB/30822msec) slat (usec): min=11, max=749211, avg=1271.14, stdev=12786.94 clat (msec): min=17, max=988, avg=110.73, stdev=112.92 lat (msec): min=17, max=988, avg=112.00, stdev=113.70 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 32], 10.00th=[ 40], 20.00th=[ 56], | 30.00th=[ 69], 40.00th=[ 82], 50.00th=[ 92], 60.00th=[ 104], | 70.00th=[ 116], 80.00th=[ 133], 90.00th=[ 159], 95.00th=[ 201], | 99.00th=[ 802], 99.50th=[ 835], 99.90th=[ 927], 99.95th=[ 953], | 99.99th=[ 986] bw ( KiB/s): min= 2048, max=1431552, per=100.00%, avg=611632.67, stdev=237689.94, samples=61 iops : min= 2, max= 1398, avg=597.30, stdev=232.12, samples=61 write: IOPS=150, BW=150MiB/s (157MB/s)(4627MiB/30822msec) slat (usec): min=275, max=737950, avg=1450.05, stdev=11794.53 clat (msec): min=65, max=2104, avg=887.27, stdev=291.28 lat (msec): min=66, max=2105, avg=888.72, stdev=291.04 clat percentiles (msec): | 1.00th=[ 167], 5.00th=[ 472], 10.00th=[ 567], 20.00th=[ 676], | 30.00th=[ 751], 40.00th=[ 802], 50.00th=[ 869], 60.00th=[ 927], | 70.00th=[ 1003], 80.00th=[ 1083], 90.00th=[ 1234], 95.00th=[ 1452], | 99.00th=[ 1720], 99.50th=[ 1838], 99.90th=[ 1905], 99.95th=[ 2005], | 99.99th=[ 2106] bw ( KiB/s): min=14336, max=305152, per=99.63%, avg=153160.63, stdev=59707.42, samples=60 iops : min= 14, max= 298, avg=149.57, stdev=58.31, samples=60 lat (msec) : 20=0.07%, 50=13.80%, 100=31.52%, 250=32.07%, 500=2.10% lat (msec) : 750=4.85%, 1000=9.44%, 2000=6.14%, >=2000=0.01% cpu : usr=5.26%, sys=1.47%, ctx=2185, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18276,4627,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=593MiB/s (622MB/s), 593MiB/s-593MiB/s (622MB/s-622MB/s), io=17.8GiB (19.2GB), run=30822-30822msec WRITE: bw=150MiB/s (157MB/s), 150MiB/s-150MiB/s (157MB/s-157MB/s), io=4627MiB (4852MB), run=30822-30822msec Disk stats (read/write): vdl: ios=18276/4598, merge=0/0, ticks=119932/3585356, in_queue=3741702, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966919: Fri Jan 7 14:28:51 2022 read: IOPS=252, BW=506MiB/s (530MB/s)(14.8GiB/30013msec) clat (usec): min=244, max=24558, avg=453.97, stdev=1008.63 lat (usec): min=244, max=24558, avg=454.16, stdev=1008.64 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 258], 10.00th=[ 262], 20.00th=[ 269], | 30.00th=[ 273], 40.00th=[ 281], 50.00th=[ 289], 60.00th=[ 306], | 70.00th=[ 330], 80.00th=[ 578], 90.00th=[ 709], 95.00th=[ 742], | 99.00th=[ 873], 99.50th=[ 9372], 99.90th=[15008], 99.95th=[16188], | 99.99th=[24511] bw ( KiB/s): min=263196, max=982709, per=100.00%, avg=518707.18, stdev=128715.55, samples=60 iops : min= 128, max= 479, avg=253.00, stdev=62.80, samples=60 write: IOPS=64, BW=129MiB/s (135MB/s)(3870MiB/30013msec) clat (usec): min=5282, max=50201, avg=13439.07, stdev=3542.97 lat (usec): min=5848, max=50424, avg=13718.17, stdev=3565.46 clat percentiles (usec): | 1.00th=[ 6783], 5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[10421], | 30.00th=[11338], 40.00th=[12387], 50.00th=[13829], 60.00th=[14746], | 70.00th=[15533], 80.00th=[16057], 90.00th=[16581], 95.00th=[17171], | 99.00th=[23200], 99.50th=[27919], 99.90th=[50070], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min=114688, max=148346, per=100.00%, avg=132233.45, stdev=7755.92, samples=60 iops : min= 56, max= 72, avg=64.42, stdev= 3.78, samples=60 lat (usec) : 250=0.48%, 500=62.22%, 750=13.76%, 1000=2.65% lat (msec) : 2=0.03%, 10=3.64%, 20=16.90%, 50=0.30%, 100=0.01% cpu : usr=1.95%, sys=1.91%, ctx=9532, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7590,1935,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=506MiB/s (530MB/s), 506MiB/s-506MiB/s (530MB/s-530MB/s), io=14.8GiB (15.9GB), run=30013-30013msec WRITE: bw=129MiB/s (135MB/s), 129MiB/s-129MiB/s (135MB/s-135MB/s), io=3870MiB (4058MB), run=30013-30013msec Disk stats (read/write): vdl: ios=15164/3867, merge=0/0, ticks=642/45963, in_queue=53107, util=95.67% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967512: Fri Jan 7 14:29:28 2022 read: IOPS=257, BW=515MiB/s (540MB/s)(15.1GiB/30007msec) slat (usec): min=54, max=479, avg=75.65, stdev=14.40 clat (usec): min=178, max=23528, avg=385.79, stdev=1014.29 lat (usec): min=270, max=23598, avg=461.68, stdev=1014.89 clat percentiles (usec): | 1.00th=[ 217], 5.00th=[ 221], 10.00th=[ 223], 20.00th=[ 227], | 30.00th=[ 233], 40.00th=[ 237], 50.00th=[ 243], 60.00th=[ 255], | 70.00th=[ 289], 80.00th=[ 351], 90.00th=[ 578], 95.00th=[ 660], | 99.00th=[ 832], 99.50th=[ 8848], 99.90th=[15270], 99.95th=[16450], | 99.99th=[23462] bw ( KiB/s): min=270336, max=974848, per=100.00%, avg=527540.82, stdev=118256.28, samples=60 iops : min= 132, max= 476, avg=257.57, stdev=57.76, samples=60 write: IOPS=65, BW=131MiB/s (137MB/s)(3930MiB/30007msec) slat (usec): min=137, max=1030, avg=297.55, stdev=108.72 clat (usec): min=4681, max=50283, avg=13146.63, stdev=3842.09 lat (usec): min=4854, max=50529, avg=13444.50, stdev=3853.26 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 6980], 10.00th=[ 8160], 20.00th=[ 9765], | 30.00th=[10945], 40.00th=[11863], 50.00th=[13698], 60.00th=[14877], | 70.00th=[15533], 80.00th=[16188], 90.00th=[16712], 95.00th=[17171], | 99.00th=[23987], 99.50th=[27919], 99.90th=[46400], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min=118784, max=147456, per=99.98%, avg=134088.70, stdev=7566.90, samples=60 iops : min= 58, max= 72, avg=65.47, stdev= 3.70, samples=60 lat (usec) : 250=44.95%, 500=24.72%, 750=8.70%, 1000=0.72% lat (msec) : 2=0.09%, 4=0.01%, 10=4.49%, 20=15.91%, 50=0.39% lat (msec) : 100=0.01% cpu : usr=1.81%, sys=2.58%, ctx=9702, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7727,1965,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=515MiB/s (540MB/s), 515MiB/s-515MiB/s (540MB/s-540MB/s), io=15.1GiB (16.2GB), run=30007-30007msec WRITE: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3930MiB (4121MB), run=30007-30007msec Disk stats (read/write): vdl: ios=15374/3906, merge=0/0, ticks=665/49370, in_queue=56767, util=96.15% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968314: Fri Jan 7 14:30:05 2022 read: IOPS=259, BW=519MiB/s (545MB/s)(15.2GiB/30002msec) clat (usec): min=231, max=20139, avg=393.59, stdev=1010.95 lat (usec): min=231, max=20139, avg=393.76, stdev=1010.95 clat percentiles (usec): | 1.00th=[ 239], 5.00th=[ 245], 10.00th=[ 247], 20.00th=[ 253], | 30.00th=[ 258], 40.00th=[ 260], 50.00th=[ 265], 60.00th=[ 273], | 70.00th=[ 293], 80.00th=[ 351], 90.00th=[ 478], 95.00th=[ 619], | 99.00th=[ 807], 99.50th=[ 9634], 99.90th=[15139], 99.95th=[16450], | 99.99th=[20055] bw ( KiB/s): min=262144, max=845466, per=100.00%, avg=531878.15, stdev=132467.83, samples=60 iops : min= 128, max= 412, avg=259.67, stdev=64.65, samples=60 write: IOPS=66, BW=132MiB/s (139MB/s)(3966MiB/30002msec) clat (usec): min=4667, max=56463, avg=13340.73, stdev=3630.50 lat (usec): min=4835, max=56680, avg=13572.84, stdev=3639.35 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 7373], 10.00th=[ 8586], 20.00th=[10028], | 30.00th=[11338], 40.00th=[12256], 50.00th=[14091], 60.00th=[15139], | 70.00th=[15795], 80.00th=[16319], 90.00th=[16909], 95.00th=[17171], | 99.00th=[19530], 99.50th=[23462], 99.90th=[44827], 99.95th=[56361], | 99.99th=[56361] bw ( KiB/s): min=118784, max=147456, per=99.97%, avg=135323.12, stdev=7398.31, samples=60 iops : min= 58, max= 72, avg=66.07, stdev= 3.61, samples=60 lat (usec) : 250=12.18%, 500=60.21%, 750=6.22%, 1000=0.53% lat (msec) : 2=0.03%, 10=4.18%, 20=16.45%, 50=0.18%, 100=0.01% cpu : usr=1.75%, sys=1.52%, ctx=9782, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7790,1983,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=15.2GiB (16.3GB), run=30002-30002msec WRITE: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3966MiB (4159MB), run=30002-30002msec Disk stats (read/write): vdl: ios=15488/3939, merge=0/0, ticks=654/42071, in_queue=48686, util=96.43% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969573: Fri Jan 7 14:30:42 2022 read: IOPS=350, BW=702MiB/s (736MB/s)(20.6GiB/30117msec) slat (usec): min=21, max=310, avg=35.14, stdev=11.54 clat (usec): min=91, max=1996.9k, avg=1773.31, stdev=32621.89 lat (usec): min=205, max=1996.9k, avg=1808.67, stdev=32621.69 clat percentiles (usec): | 1.00th=[ 184], 5.00th=[ 188], 10.00th=[ 190], | 20.00th=[ 194], 30.00th=[ 198], 40.00th=[ 202], | 50.00th=[ 206], 60.00th=[ 215], 70.00th=[ 241], | 80.00th=[ 627], 90.00th=[ 652], 95.00th=[ 676], | 99.00th=[ 979], 99.50th=[ 20841], 99.90th=[ 526386], | 99.95th=[ 759170], 99.99th=[1098908] bw ( KiB/s): min=487424, max=1032192, per=100.00%, avg=721510.40, stdev=126023.62, samples=60 iops : min= 238, max= 504, avg=352.30, stdev=61.53, samples=60 write: IOPS=89, BW=178MiB/s (187MB/s)(5374MiB/30117msec) slat (usec): min=120, max=956, avg=343.24, stdev=180.79 clat (msec): min=7, max=2502, avg=104.57, stdev=191.90 lat (msec): min=7, max=2502, avg=104.91, stdev=191.90 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 19], 10.00th=[ 21], 20.00th=[ 30], | 30.00th=[ 38], 40.00th=[ 48], 50.00th=[ 64], 60.00th=[ 82], | 70.00th=[ 104], 80.00th=[ 138], 90.00th=[ 215], 95.00th=[ 284], | 99.00th=[ 514], 99.50th=[ 1284], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min=163840, max=196608, per=100.00%, avg=182749.87, stdev=7876.71, samples=60 iops : min= 80, max= 96, avg=89.23, stdev= 3.85, samples=60 lat (usec) : 100=0.01%, 250=56.92%, 500=5.35%, 750=15.86%, 1000=0.84% lat (msec) : 2=0.13%, 4=0.05%, 10=0.17%, 20=1.86%, 50=6.61% lat (msec) : 100=5.64%, 250=5.02%, 500=1.24%, 750=0.14%, 1000=0.04% lat (msec) : 2000=0.04%, >=2000=0.10% cpu : usr=3.28%, sys=1.69%, ctx=13237, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10569,2687,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=702MiB/s (736MB/s), 702MiB/s-702MiB/s (736MB/s-736MB/s), io=20.6GiB (22.2GB), run=30117-30117msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=5374MiB (5635MB), run=30117-30117msec Disk stats (read/write): vdl: ios=21184/5364, merge=0/0, ticks=16686/577814, in_queue=612181, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970160: Fri Jan 7 14:31:19 2022 read: IOPS=258, BW=518MiB/s (543MB/s)(15.2GiB/30004msec) clat (usec): min=215, max=20327, avg=361.64, stdev=1033.77 lat (usec): min=215, max=20328, avg=361.81, stdev=1033.78 clat percentiles (usec): | 1.00th=[ 223], 5.00th=[ 229], 10.00th=[ 233], 20.00th=[ 237], | 30.00th=[ 241], 40.00th=[ 245], 50.00th=[ 251], 60.00th=[ 258], | 70.00th=[ 273], 80.00th=[ 318], 90.00th=[ 363], 95.00th=[ 420], | 99.00th=[ 725], 99.50th=[ 9896], 99.90th=[15664], 99.95th=[16450], | 99.99th=[20317] bw ( KiB/s): min=258048, max=827392, per=100.00%, avg=530291.35, stdev=125287.89, samples=60 iops : min= 126, max= 404, avg=258.90, stdev=61.18, samples=60 write: IOPS=65, BW=132MiB/s (138MB/s)(3952MiB/30004msec) clat (usec): min=4665, max=57065, avg=13547.19, stdev=3765.10 lat (usec): min=4862, max=57291, avg=13752.43, stdev=3764.79 clat percentiles (usec): | 1.00th=[ 5342], 5.00th=[ 7439], 10.00th=[ 8717], 20.00th=[10290], | 30.00th=[11469], 40.00th=[12387], 50.00th=[14484], 60.00th=[15401], | 70.00th=[15926], 80.00th=[16450], 90.00th=[16909], 95.00th=[17433], | 99.00th=[22676], 99.50th=[25560], 99.90th=[49546], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min=118784, max=151552, per=99.98%, avg=134845.38, stdev=7789.87, samples=60 iops : min= 58, max= 74, avg=65.83, stdev= 3.80, samples=60 lat (usec) : 250=39.32%, 500=38.19%, 750=1.48%, 1000=0.16% lat (msec) : 2=0.02%, 10=3.79%, 20=16.77%, 50=0.26%, 100=0.01% cpu : usr=1.46%, sys=1.24%, ctx=9749, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7767,1976,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=518MiB/s (543MB/s), 518MiB/s-518MiB/s (543MB/s-543MB/s), io=15.2GiB (16.3GB), run=30004-30004msec WRITE: bw=132MiB/s (138MB/s), 132MiB/s-132MiB/s (138MB/s-138MB/s), io=3952MiB (4144MB), run=30004-30004msec Disk stats (read/write): vdl: ios=15428/3920, merge=0/0, ticks=669/41363, in_queue=47771, util=96.94% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971117: Fri Jan 7 14:31:56 2022 read: IOPS=362, BW=726MiB/s (761MB/s)(21.6GiB/30524msec) slat (usec): min=19, max=658, avg=34.55, stdev=19.37 clat (usec): min=65, max=2557.2k, avg=4185.56, stdev=57411.50 lat (usec): min=204, max=2557.2k, avg=4220.31, stdev=57411.63 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 190], 10.00th=[ 192], | 20.00th=[ 196], 30.00th=[ 200], 40.00th=[ 206], | 50.00th=[ 215], 60.00th=[ 241], 70.00th=[ 627], | 80.00th=[ 660], 90.00th=[ 1385], 95.00th=[ 9765], | 99.00th=[ 25297], 99.50th=[ 71828], 99.90th=[1182794], | 99.95th=[1434452], 99.99th=[2021655] bw ( KiB/s): min=290816, max=1380352, per=100.00%, avg=758454.62, stdev=173960.25, samples=60 iops : min= 142, max= 674, avg=370.27, stdev=84.94, samples=60 write: IOPS=92, BW=185MiB/s (193MB/s)(5632MiB/30524msec) slat (usec): min=118, max=1273, avg=383.84, stdev=204.87 clat (msec): min=74, max=3279, avg=524.69, stdev=267.45 lat (msec): min=75, max=3279, avg=525.07, stdev=267.45 clat percentiles (msec): | 1.00th=[ 192], 5.00th=[ 300], 10.00th=[ 342], 20.00th=[ 372], | 30.00th=[ 393], 40.00th=[ 414], 50.00th=[ 439], 60.00th=[ 477], | 70.00th=[ 542], 80.00th=[ 634], 90.00th=[ 802], 95.00th=[ 1003], | 99.00th=[ 1519], 99.50th=[ 1854], 99.90th=[ 3104], 99.95th=[ 3138], | 99.99th=[ 3272] bw ( KiB/s): min= 4096, max=233580, per=98.67%, avg=186430.77, stdev=29306.52, samples=61 iops : min= 2, max= 114, avg=90.92, stdev=14.32, samples=61 lat (usec) : 100=0.01%, 250=49.19%, 500=5.61%, 750=14.74%, 1000=1.39% lat (msec) : 2=1.48%, 4=1.23%, 10=2.15%, 20=2.56%, 50=0.90% lat (msec) : 100=0.17%, 250=0.58%, 500=12.55%, 750=4.82%, 1000=1.52% lat (msec) : 2000=0.99%, >=2000=0.12% cpu : usr=3.61%, sys=1.82%, ctx=12282, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=11076,2816,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=726MiB/s (761MB/s), 726MiB/s-726MiB/s (761MB/s-761MB/s), io=21.6GiB (23.2GB), run=30524-30524msec WRITE: bw=185MiB/s (193MB/s), 185MiB/s-185MiB/s (193MB/s-193MB/s), io=5632MiB (5906MB), run=30524-30524msec Disk stats (read/write): vdl: ios=22152/5624, merge=0/0, ticks=53175/2928345, in_queue=2997346, util=99.49% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972163: Fri Jan 7 14:32:33 2022 read: IOPS=259, BW=519MiB/s (544MB/s)(15.2GiB/30010msec) clat (usec): min=234, max=25038, avg=386.94, stdev=1049.57 lat (usec): min=234, max=25039, avg=387.11, stdev=1049.57 clat percentiles (usec): | 1.00th=[ 243], 5.00th=[ 249], 10.00th=[ 251], 20.00th=[ 258], | 30.00th=[ 262], 40.00th=[ 265], 50.00th=[ 273], 60.00th=[ 281], | 70.00th=[ 302], 80.00th=[ 343], 90.00th=[ 392], 95.00th=[ 461], | 99.00th=[ 766], 99.50th=[ 9896], 99.90th=[15533], 99.95th=[16450], | 99.99th=[25035] bw ( KiB/s): min=274432, max=843776, per=100.00%, avg=531817.17, stdev=131350.73, samples=60 iops : min= 134, max= 412, avg=259.67, stdev=64.13, samples=60 write: IOPS=66, BW=132MiB/s (139MB/s)(3966MiB/30010msec) clat (usec): min=4664, max=56990, avg=13393.27, stdev=3554.88 lat (usec): min=4859, max=57208, avg=13602.79, stdev=3553.68 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 7373], 10.00th=[ 8586], 20.00th=[10290], | 30.00th=[11338], 40.00th=[12387], 50.00th=[14222], 60.00th=[15270], | 70.00th=[15926], 80.00th=[16450], 90.00th=[16909], 95.00th=[17171], | 99.00th=[19268], 99.50th=[23725], 99.90th=[36439], 99.95th=[56886], | 99.99th=[56886] bw ( KiB/s): min=122880, max=151552, per=99.99%, avg=135309.32, stdev=7393.09, samples=60 iops : min= 60, max= 74, avg=66.07, stdev= 3.61, samples=60 lat (usec) : 250=5.48%, 500=71.27%, 750=2.11%, 1000=0.29% lat (msec) : 2=0.01%, 10=3.89%, 20=16.79%, 50=0.15%, 100=0.01% cpu : usr=1.53%, sys=1.71%, ctx=9779, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7790,1983,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=519MiB/s (544MB/s), 519MiB/s-519MiB/s (544MB/s-544MB/s), io=15.2GiB (16.3GB), run=30010-30010msec WRITE: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3966MiB (4159MB), run=30010-30010msec Disk stats (read/write): vdl: ios=15544/3955, merge=0/0, ticks=705/42749, in_queue=49227, util=96.07% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972686: Fri Jan 7 14:33:11 2022 read: IOPS=365, BW=731MiB/s (766MB/s)(21.9GiB/30671msec) slat (usec): min=21, max=486734, avg=2008.20, stdev=9798.71 clat (msec): min=20, max=2661, avg=87.12, stdev=91.79 lat (msec): min=20, max=2661, avg=89.13, stdev=92.38 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 31], 10.00th=[ 37], 20.00th=[ 48], | 30.00th=[ 56], 40.00th=[ 65], 50.00th=[ 73], 60.00th=[ 83], | 70.00th=[ 93], 80.00th=[ 108], 90.00th=[ 127], 95.00th=[ 157], | 99.00th=[ 506], 99.50th=[ 535], 99.90th=[ 1452], 99.95th=[ 1720], | 99.99th=[ 2635] bw ( KiB/s): min= 4096, max=1323008, per=100.00%, avg=750465.00, stdev=266944.49, samples=61 iops : min= 2, max= 646, avg=366.43, stdev=130.35, samples=61 write: IOPS=92, BW=185MiB/s (194MB/s)(5688MiB/30671msec) slat (usec): min=453, max=456106, avg=2619.66, stdev=10272.22 clat (msec): min=167, max=3560, avg=723.63, stdev=304.94 lat (msec): min=169, max=3561, avg=726.25, stdev=304.91 clat percentiles (msec): | 1.00th=[ 279], 5.00th=[ 409], 10.00th=[ 481], 20.00th=[ 550], | 30.00th=[ 584], 40.00th=[ 609], 50.00th=[ 642], 60.00th=[ 684], | 70.00th=[ 751], 80.00th=[ 869], 90.00th=[ 1083], 95.00th=[ 1250], | 99.00th=[ 1938], 99.50th=[ 2433], 99.90th=[ 3272], 99.95th=[ 3574], | 99.99th=[ 3574] bw ( KiB/s): min= 8192, max=290816, per=99.87%, avg=189649.85, stdev=57118.50, samples=60 iops : min= 4, max= 142, avg=92.60, stdev=27.89, samples=60 lat (msec) : 50=18.38%, 100=41.76%, 250=18.11%, 500=3.11%, 750=12.47% lat (msec) : 1000=3.54%, 2000=2.43%, >=2000=0.21% cpu : usr=6.29%, sys=1.81%, ctx=2872, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11209,2844,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=731MiB/s (766MB/s), 731MiB/s-731MiB/s (766MB/s-766MB/s), io=21.9GiB (23.5GB), run=30671-30671msec WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5688MiB (5964MB), run=30671-30671msec Disk stats (read/write): vdl: ios=22418/5686, merge=0/0, ticks=115475/3624037, in_queue=3756068, util=99.37% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973648: Fri Jan 7 14:33:47 2022 read: IOPS=260, BW=520MiB/s (545MB/s)(15.2GiB/30006msec) clat (usec): min=221, max=20282, avg=376.14, stdev=1028.14 lat (usec): min=221, max=20282, avg=376.31, stdev=1028.14 clat percentiles (usec): | 1.00th=[ 229], 5.00th=[ 237], 10.00th=[ 239], 20.00th=[ 243], | 30.00th=[ 249], 40.00th=[ 253], 50.00th=[ 260], 60.00th=[ 269], | 70.00th=[ 289], 80.00th=[ 338], 90.00th=[ 388], 95.00th=[ 474], | 99.00th=[ 766], 99.50th=[ 9765], 99.90th=[15533], 99.95th=[16057], | 99.99th=[20317] bw ( KiB/s): min=262144, max=839680, per=100.00%, avg=532663.03, stdev=130771.31, samples=60 iops : min= 128, max= 410, avg=260.07, stdev=63.82, samples=60 write: IOPS=66, BW=132MiB/s (139MB/s)(3970MiB/30006msec) clat (usec): min=4662, max=50236, avg=13417.99, stdev=3701.91 lat (usec): min=4786, max=50484, avg=13627.17, stdev=3703.26 clat percentiles (usec): | 1.00th=[ 5407], 5.00th=[ 7242], 10.00th=[ 8586], 20.00th=[10159], | 30.00th=[11338], 40.00th=[12256], 50.00th=[14222], 60.00th=[15270], | 70.00th=[15926], 80.00th=[16450], 90.00th=[16909], 95.00th=[17433], | 99.00th=[19530], 99.50th=[24511], 99.90th=[48497], 99.95th=[50070], | 99.99th=[50070] bw ( KiB/s): min=110592, max=147751, per=99.98%, avg=135450.90, stdev=8236.80, samples=60 iops : min= 54, max= 72, avg=66.13, stdev= 4.01, samples=60 lat (usec) : 250=26.75%, 500=49.70%, 750=2.44%, 1000=0.20% lat (msec) : 2=0.07%, 10=3.94%, 20=16.70%, 50=0.18%, 100=0.01% cpu : usr=1.60%, sys=1.30%, ctx=9793, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7802,1985,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=15.2GiB (16.4GB), run=30006-30006msec WRITE: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3970MiB (4163MB), run=30006-30006msec Disk stats (read/write): vdl: ios=15522/3947, merge=0/0, ticks=681/41736, in_queue=48178, util=96.09% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974271: Fri Jan 7 14:34:25 2022 read: IOPS=361, BW=723MiB/s (758MB/s)(21.6GiB/30650msec) slat (usec): min=22, max=480286, avg=2115.52, stdev=11685.62 clat (msec): min=85, max=2929, avg=311.12, stdev=138.95 lat (msec): min=85, max=2929, avg=313.24, stdev=139.41 clat percentiles (msec): | 1.00th=[ 97], 5.00th=[ 122], 10.00th=[ 199], 20.00th=[ 239], | 30.00th=[ 257], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 334], 80.00th=[ 363], 90.00th=[ 418], 95.00th=[ 592], | 99.00th=[ 768], 99.50th=[ 869], 99.90th=[ 1905], 99.95th=[ 2039], | 99.99th=[ 2836] bw ( KiB/s): min= 4096, max=1503232, per=99.47%, avg=736529.20, stdev=273064.72, samples=61 iops : min= 2, max= 734, avg=359.59, stdev=133.32, samples=61 write: IOPS=91, BW=184MiB/s (193MB/s)(5634MiB/30650msec) slat (usec): min=553, max=75149, avg=2313.27, stdev=5367.91 clat (msec): min=468, max=4100, avg=937.34, stdev=320.18 lat (msec): min=468, max=4100, avg=939.65, stdev=320.11 clat percentiles (msec): | 1.00th=[ 535], 5.00th=[ 600], 10.00th=[ 651], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 869], 60.00th=[ 911], | 70.00th=[ 986], 80.00th=[ 1099], 90.00th=[ 1284], 95.00th=[ 1469], | 99.00th=[ 2123], 99.50th=[ 2702], 99.90th=[ 3608], 99.95th=[ 3608], | 99.99th=[ 4111] bw ( KiB/s): min=12288, max=352256, per=100.00%, avg=189610.68, stdev=69529.41, samples=59 iops : min= 6, max= 172, avg=92.56, stdev=33.95, samples=59 lat (msec) : 100=1.37%, 250=19.38%, 500=53.92%, 750=8.99%, 1000=10.40% lat (msec) : 2000=5.59%, >=2000=0.35% cpu : usr=6.32%, sys=1.77%, ctx=2738, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11081,2817,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=723MiB/s (758MB/s), 723MiB/s-723MiB/s (758MB/s-758MB/s), io=21.6GiB (23.2GB), run=30650-30650msec WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5634MiB (5908MB), run=30650-30650msec Disk stats (read/write): vdl: ios=22159/5620, merge=0/0, ticks=136961/3584122, in_queue=3750892, util=98.91% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975114: Fri Jan 7 14:35:01 2022 read: IOPS=157, BW=630MiB/s (660MB/s)(18.4GiB/30008msec) clat (usec): min=400, max=71009, avg=836.92, stdev=1774.76 lat (usec): min=400, max=71009, avg=837.13, stdev=1774.76 clat percentiles (usec): | 1.00th=[ 416], 5.00th=[ 424], 10.00th=[ 429], 20.00th=[ 441], | 30.00th=[ 449], 40.00th=[ 457], 50.00th=[ 469], 60.00th=[ 490], | 70.00th=[ 570], 80.00th=[ 1287], 90.00th=[ 1352], 95.00th=[ 1434], | 99.00th=[ 9372], 99.50th=[13566], 99.90th=[20579], 99.95th=[21627], | 99.99th=[70779] bw ( KiB/s): min=303104, max=1064960, per=100.00%, avg=644893.43, stdev=180668.49, samples=60 iops : min= 74, max= 260, avg=157.43, stdev=44.10, samples=60 write: IOPS=40, BW=163MiB/s (171MB/s)(4904MiB/30008msec) clat (usec): min=11020, max=62316, avg=20642.52, stdev=4953.00 lat (usec): min=11642, max=63450, avg=21239.28, stdev=4990.91 clat percentiles (usec): | 1.00th=[12518], 5.00th=[14091], 10.00th=[15008], 20.00th=[16057], | 30.00th=[17433], 40.00th=[19006], 50.00th=[21103], 60.00th=[22676], | 70.00th=[23462], 80.00th=[24249], 90.00th=[25035], 95.00th=[26084], | 99.00th=[34341], 99.50th=[40633], 99.90th=[61080], 99.95th=[62129], | 99.99th=[62129] bw ( KiB/s): min=122880, max=180585, per=99.95%, avg=167264.82, stdev=10268.03, samples=60 iops : min= 30, max= 44, avg=40.83, stdev= 2.51, samples=60 lat (usec) : 500=49.49%, 750=10.79%, 1000=0.91% lat (msec) : 2=17.11%, 4=0.08%, 10=0.25%, 20=10.04%, 50=11.23% lat (msec) : 100=0.10% cpu : usr=2.56%, sys=2.02%, ctx=5960, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4723,1226,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=630MiB/s (660MB/s), 630MiB/s-630MiB/s (660MB/s-660MB/s), io=18.4GiB (19.8GB), run=30008-30008msec WRITE: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4904MiB (5142MB), run=30008-30008msec Disk stats (read/write): vdl: ios=18784/4865, merge=0/0, ticks=5014/80689, in_queue=94977, util=96.17% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976338: Fri Jan 7 14:35:38 2022 read: IOPS=161, BW=644MiB/s (676MB/s)(18.9GiB/30009msec) slat (usec): min=74, max=287, avg=98.44, stdev=18.76 clat (usec): min=310, max=21599, avg=725.77, stdev=1440.44 lat (usec): min=427, max=21712, avg=824.46, stdev=1441.59 clat percentiles (usec): | 1.00th=[ 343], 5.00th=[ 355], 10.00th=[ 363], 20.00th=[ 367], | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 388], 60.00th=[ 400], | 70.00th=[ 457], 80.00th=[ 1090], 90.00th=[ 1254], 95.00th=[ 1287], | 99.00th=[ 9503], 99.50th=[13173], 99.90th=[19530], 99.95th=[20841], | 99.99th=[21627] bw ( KiB/s): min=270336, max=1064960, per=100.00%, avg=660162.18, stdev=169606.49, samples=60 iops : min= 66, max= 260, avg=161.17, stdev=41.41, samples=60 write: IOPS=41, BW=166MiB/s (174MB/s)(4988MiB/30009msec) slat (usec): min=254, max=1966, avg=650.17, stdev=316.44 clat (usec): min=10669, max=82911, avg=20205.67, stdev=5286.92 lat (usec): min=11299, max=84000, avg=20856.22, stdev=5342.23 clat percentiles (usec): | 1.00th=[11863], 5.00th=[13698], 10.00th=[14615], 20.00th=[15664], | 30.00th=[16909], 40.00th=[18220], 50.00th=[20317], 60.00th=[22414], | 70.00th=[23200], 80.00th=[23987], 90.00th=[24773], 95.00th=[25560], | 99.00th=[32900], 99.50th=[36963], 99.90th=[71828], 99.95th=[83362], | 99.99th=[83362] bw ( KiB/s): min=122880, max=188416, per=99.95%, avg=170126.00, stdev=10700.93, samples=60 iops : min= 30, max= 46, avg=41.53, stdev= 2.61, samples=60 lat (usec) : 500=57.22%, 750=4.01%, 1000=1.48% lat (msec) : 2=15.73%, 4=0.05%, 10=0.28%, 20=10.56%, 50=10.59% lat (msec) : 100=0.08% cpu : usr=2.51%, sys=2.08%, ctx=6094, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4835,1247,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=644MiB/s (676MB/s), 644MiB/s-644MiB/s (676MB/s-676MB/s), io=18.9GiB (20.3GB), run=30009-30009msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4988MiB (5230MB), run=30009-30009msec Disk stats (read/write): vdl: ios=19268/4956, merge=0/0, ticks=4701/82774, in_queue=97312, util=96.20% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977102: Fri Jan 7 14:36:15 2022 read: IOPS=163, BW=652MiB/s (684MB/s)(19.1GiB/30003msec) clat (usec): min=372, max=22303, avg=758.32, stdev=1520.27 lat (usec): min=372, max=22303, avg=758.52, stdev=1520.28 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 400], | 30.00th=[ 404], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 474], 80.00th=[ 971], 90.00th=[ 1254], 95.00th=[ 1319], | 99.00th=[10552], 99.50th=[14091], 99.90th=[19530], 99.95th=[21627], | 99.99th=[22414] bw ( KiB/s): min=340640, max=1198829, per=100.00%, avg=675445.10, stdev=163087.12, samples=60 iops : min= 83, max= 292, avg=164.42, stdev=39.82, samples=60 write: IOPS=41, BW=168MiB/s (176MB/s)(5028MiB/30003msec) clat (usec): min=11045, max=73638, avg=20347.71, stdev=4909.03 lat (usec): min=11438, max=74018, avg=20906.07, stdev=4951.31 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13960], 10.00th=[14746], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20579], 60.00th=[22414], | 70.00th=[23462], 80.00th=[24249], 90.00th=[25035], 95.00th=[25560], | 99.00th=[33424], 99.50th=[34866], 99.90th=[57410], 99.95th=[73925], | 99.99th=[73925] bw ( KiB/s): min=148945, max=190704, per=100.00%, avg=173443.48, stdev=7984.86, samples=60 iops : min= 36, max= 46, avg=41.87, stdev= 1.94, samples=60 lat (usec) : 500=57.11%, 750=4.36%, 1000=2.49% lat (msec) : 2=14.48%, 4=0.03%, 10=0.23%, 20=10.54%, 50=10.70% lat (msec) : 100=0.07% cpu : usr=2.48%, sys=1.35%, ctx=6158, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4891,1257,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=652MiB/s (684MB/s), 652MiB/s-652MiB/s (684MB/s-684MB/s), io=19.1GiB (20.5GB), run=30003-30003msec WRITE: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=5028MiB (5272MB), run=30003-30003msec Disk stats (read/write): vdl: ios=19564/5025, merge=0/0, ticks=4404/82213, in_queue=96225, util=96.64% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977884: Fri Jan 7 14:36:52 2022 read: IOPS=192, BW=768MiB/s (806MB/s)(22.7GiB/30186msec) slat (usec): min=41, max=308, avg=61.63, stdev=16.62 clat (usec): min=218, max=2125.2k, avg=5933.90, stdev=69817.92 lat (usec): min=376, max=2125.2k, avg=5995.79, stdev=69817.63 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 338], 10.00th=[ 343], | 20.00th=[ 351], 30.00th=[ 355], 40.00th=[ 363], | 50.00th=[ 371], 60.00th=[ 383], 70.00th=[ 449], | 80.00th=[ 1221], 90.00th=[ 1270], 95.00th=[ 1336], | 99.00th=[ 60556], 99.50th=[ 404751], 99.90th=[1149240], | 99.95th=[1568670], 99.99th=[2122318] bw ( KiB/s): min=360448, max=1376256, per=100.00%, avg=791851.98, stdev=199747.54, samples=60 iops : min= 88, max= 336, avg=193.30, stdev=48.75, samples=60 write: IOPS=50, BW=200MiB/s (210MB/s)(6048MiB/30186msec) slat (usec): min=238, max=2144, avg=784.76, stdev=390.27 clat (msec): min=18, max=2517, avg=175.72, stdev=173.35 lat (msec): min=18, max=2517, avg=176.51, stdev=173.37 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 35], 10.00th=[ 48], 20.00th=[ 58], | 30.00th=[ 77], 40.00th=[ 99], 50.00th=[ 123], 60.00th=[ 153], | 70.00th=[ 190], 80.00th=[ 247], 90.00th=[ 368], 95.00th=[ 527], | 99.00th=[ 835], 99.50th=[ 936], 99.90th=[ 1318], 99.95th=[ 2534], | 99.99th=[ 2534] bw ( KiB/s): min=180224, max=221184, per=99.96%, avg=205093.57, stdev=7832.15, samples=60 iops : min= 44, max= 54, avg=50.07, stdev= 1.91, samples=60 lat (usec) : 250=0.01%, 500=57.19%, 750=2.07%, 1000=0.36% lat (msec) : 2=18.19%, 4=0.21%, 10=0.18%, 20=0.23%, 50=2.37% lat (msec) : 100=6.29%, 250=8.36%, 500=3.13%, 750=0.98%, 1000=0.23% lat (msec) : 2000=0.16%, >=2000=0.04% cpu : usr=3.78%, sys=1.71%, ctx=7267, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5799,1512,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=768MiB/s (806MB/s), 768MiB/s-768MiB/s (806MB/s-806MB/s), io=22.7GiB (24.3GB), run=30186-30186msec WRITE: bw=200MiB/s (210MB/s), 200MiB/s-200MiB/s (210MB/s-210MB/s), io=6048MiB (6342MB), run=30186-30186msec Disk stats (read/write): vdl: ios=23196/6020, merge=0/0, ticks=39789/1030809, in_queue=1089545, util=99.60% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978562: Fri Jan 7 14:37:29 2022 read: IOPS=161, BW=647MiB/s (679MB/s)(18.0GiB/30006msec) clat (usec): min=375, max=24052, avg=759.64, stdev=1564.15 lat (usec): min=376, max=24052, avg=759.84, stdev=1564.16 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 388], 10.00th=[ 392], 20.00th=[ 400], | 30.00th=[ 404], 40.00th=[ 412], 50.00th=[ 420], 60.00th=[ 429], | 70.00th=[ 461], 80.00th=[ 1012], 90.00th=[ 1270], 95.00th=[ 1303], | 99.00th=[10421], 99.50th=[14353], 99.90th=[19792], 99.95th=[21890], | 99.99th=[23987] bw ( KiB/s): min=344064, max=1056768, per=100.00%, avg=663041.70, stdev=154456.35, samples=60 iops : min= 84, max= 258, avg=161.87, stdev=37.72, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5008MiB/30006msec) clat (usec): min=11098, max=78142, avg=20447.09, stdev=5382.97 lat (usec): min=11600, max=78576, avg=21007.71, stdev=5420.30 clat percentiles (usec): | 1.00th=[12125], 5.00th=[14091], 10.00th=[14746], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20579], 60.00th=[22676], | 70.00th=[23462], 80.00th=[24249], 90.00th=[24773], 95.00th=[25822], | 99.00th=[32900], 99.50th=[40109], 99.90th=[77071], 99.95th=[78119], | 99.99th=[78119] bw ( KiB/s): min=147456, max=188416, per=99.95%, avg=170814.40, stdev=8897.37, samples=60 iops : min= 36, max= 46, avg=41.70, stdev= 2.17, samples=60 lat (usec) : 500=57.91%, 750=3.62%, 1000=1.95% lat (msec) : 2=14.95%, 4=0.05%, 10=0.16%, 20=10.56%, 50=10.72% lat (msec) : 100=0.08% cpu : usr=2.46%, sys=1.54%, ctx=6118, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4856,1252,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=647MiB/s (679MB/s), 647MiB/s-647MiB/s (679MB/s-679MB/s), io=18.0GiB (20.4GB), run=30006-30006msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5008MiB (5251MB), run=30006-30006msec Disk stats (read/write): vdl: ios=19356/4993, merge=0/0, ticks=4400/82132, in_queue=95966, util=96.55% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979529: Fri Jan 7 14:38:07 2022 read: IOPS=189, BW=758MiB/s (795MB/s)(22.7GiB/30600msec) slat (usec): min=45, max=624153, avg=3954.48, stdev=15831.54 clat (msec): min=13, max=2441, avg=81.47, stdev=115.36 lat (msec): min=13, max=2441, avg=85.43, stdev=116.67 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 37], | 30.00th=[ 44], 40.00th=[ 52], 50.00th=[ 62], 60.00th=[ 70], | 70.00th=[ 83], 80.00th=[ 97], 90.00th=[ 123], 95.00th=[ 150], | 99.00th=[ 625], 99.50th=[ 684], 99.90th=[ 1821], 99.95th=[ 2165], | 99.99th=[ 2433] bw ( KiB/s): min=147456, max=1348982, per=100.00%, avg=789832.92, stdev=274870.14, samples=60 iops : min= 36, max= 329, avg=192.82, stdev=67.09, samples=60 write: IOPS=49, BW=198MiB/s (207MB/s)(6044MiB/30600msec) slat (usec): min=1109, max=432271, avg=4663.79, stdev=14727.72 clat (msec): min=193, max=3398, avg=679.07, stdev=260.78 lat (msec): min=194, max=3408, avg=683.73, stdev=260.46 clat percentiles (msec): | 1.00th=[ 241], 5.00th=[ 426], 10.00th=[ 477], 20.00th=[ 523], | 30.00th=[ 550], 40.00th=[ 584], 50.00th=[ 609], 60.00th=[ 651], | 70.00th=[ 709], 80.00th=[ 810], 90.00th=[ 986], 95.00th=[ 1133], | 99.00th=[ 1485], 99.50th=[ 1620], 99.90th=[ 3373], 99.95th=[ 3406], | 99.99th=[ 3406] bw ( KiB/s): min= 8192, max=368640, per=98.01%, avg=198221.43, stdev=71232.90, samples=61 iops : min= 2, max= 90, avg=48.38, stdev=17.39, samples=61 lat (msec) : 20=0.31%, 50=30.08%, 100=34.28%, 250=12.82%, 500=3.49% lat (msec) : 750=13.52%, 1000=3.43%, 2000=1.96%, >=2000=0.11% cpu : usr=6.78%, sys=1.65%, ctx=2696, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5799,1511,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=758MiB/s (795MB/s), 758MiB/s-758MiB/s (795MB/s-795MB/s), io=22.7GiB (24.3GB), run=30600-30600msec WRITE: bw=198MiB/s (207MB/s), 198MiB/s-198MiB/s (207MB/s-207MB/s), io=6044MiB (6338MB), run=30600-30600msec Disk stats (read/write): vdl: ios=23196/6012, merge=0/0, ticks=147304/3585681, in_queue=3763792, util=99.39% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980494: Fri Jan 7 14:38:43 2022 read: IOPS=163, BW=652MiB/s (684MB/s)(19.1GiB/30015msec) clat (usec): min=379, max=22149, avg=758.63, stdev=1505.19 lat (usec): min=379, max=22149, avg=758.83, stdev=1505.20 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 404], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 441], | 70.00th=[ 474], 80.00th=[ 979], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[10159], 99.50th=[13829], 99.90th=[19530], 99.95th=[21890], | 99.99th=[22152] bw ( KiB/s): min=253952, max=1064960, per=100.00%, avg=668379.63, stdev=156820.40, samples=60 iops : min= 62, max= 260, avg=163.17, stdev=38.28, samples=60 write: IOPS=41, BW=168MiB/s (176MB/s)(5040MiB/30015msec) clat (usec): min=10911, max=74541, avg=20305.60, stdev=4745.98 lat (usec): min=11303, max=74996, avg=20862.29, stdev=4793.66 clat percentiles (usec): | 1.00th=[12256], 5.00th=[13960], 10.00th=[14746], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20579], 60.00th=[22676], | 70.00th=[23462], 80.00th=[24249], 90.00th=[24773], 95.00th=[25560], | 99.00th=[31327], 99.50th=[39060], 99.90th=[60031], 99.95th=[74974], | 99.99th=[74974] bw ( KiB/s): min=139264, max=188416, per=99.98%, avg=171907.50, stdev=8740.57, samples=60 iops : min= 34, max= 46, avg=41.97, stdev= 2.13, samples=60 lat (usec) : 500=57.27%, 750=4.05%, 1000=2.55% lat (msec) : 2=14.57%, 4=0.03%, 10=0.24%, 20=10.46%, 50=10.77% lat (msec) : 100=0.05% cpu : usr=2.52%, sys=1.85%, ctx=6166, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4895,1260,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=652MiB/s (684MB/s), 652MiB/s-652MiB/s (684MB/s-684MB/s), io=19.1GiB (20.5GB), run=30015-30015msec WRITE: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=5040MiB (5285MB), run=30015-30015msec Disk stats (read/write): vdl: ios=19440/5008, merge=0/0, ticks=4257/81286, in_queue=94884, util=96.10% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981074: Fri Jan 7 14:39:21 2022 read: IOPS=189, BW=758MiB/s (795MB/s)(22.7GiB/30608msec) slat (usec): min=45, max=469834, avg=3976.38, stdev=15322.54 clat (msec): min=87, max=2623, avg=292.18, stdev=142.50 lat (msec): min=87, max=2623, avg=296.16, stdev=143.05 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 117], 10.00th=[ 163], 20.00th=[ 199], | 30.00th=[ 226], 40.00th=[ 253], 50.00th=[ 279], 60.00th=[ 300], | 70.00th=[ 321], 80.00th=[ 351], 90.00th=[ 414], 95.00th=[ 550], | 99.00th=[ 751], 99.50th=[ 877], 99.90th=[ 1670], 99.95th=[ 2089], | 99.99th=[ 2635] bw ( KiB/s): min=139264, max=1597440, per=100.00%, avg=784887.70, stdev=295324.04, samples=60 iops : min= 34, max= 390, avg=191.60, stdev=72.08, samples=60 write: IOPS=49, BW=198MiB/s (207MB/s)(6052MiB/30608msec) slat (usec): min=1114, max=147585, avg=4567.22, stdev=10065.57 clat (msec): min=233, max=3512, avg=878.66, stdev=254.30 lat (msec): min=234, max=3514, avg=883.23, stdev=254.11 clat percentiles (msec): | 1.00th=[ 443], 5.00th=[ 592], 10.00th=[ 651], 20.00th=[ 709], | 30.00th=[ 751], 40.00th=[ 793], 50.00th=[ 827], 60.00th=[ 877], | 70.00th=[ 927], 80.00th=[ 1028], 90.00th=[ 1183], 95.00th=[ 1301], | 99.00th=[ 1720], 99.50th=[ 1770], 99.90th=[ 3306], 99.95th=[ 3507], | 99.99th=[ 3507] bw ( KiB/s): min= 8192, max=352256, per=98.87%, avg=200180.28, stdev=66124.03, samples=60 iops : min= 2, max= 86, avg=48.87, stdev=16.14, samples=60 lat (msec) : 100=1.52%, 250=29.56%, 500=43.65%, 750=9.90%, 1000=10.46% lat (msec) : 2000=4.83%, >=2000=0.08% cpu : usr=6.70%, sys=1.74%, ctx=2775, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5800,1513,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=758MiB/s (795MB/s), 758MiB/s-758MiB/s (795MB/s-795MB/s), io=22.7GiB (24.3GB), run=30608-30608msec WRITE: bw=198MiB/s (207MB/s), 198MiB/s-198MiB/s (207MB/s-207MB/s), io=6052MiB (6346MB), run=30608-30608msec Disk stats (read/write): vdl: ios=23200/6043, merge=0/0, ticks=138888/3607718, in_queue=3767156, util=99.12% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981898: Fri Jan 7 14:39:57 2022 read: IOPS=162, BW=652MiB/s (684MB/s)(19.1GiB/30007msec) clat (usec): min=379, max=24424, avg=760.09, stdev=1533.17 lat (usec): min=379, max=24424, avg=760.27, stdev=1533.17 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 404], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 420], 60.00th=[ 433], | 70.00th=[ 465], 80.00th=[ 996], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[10159], 99.50th=[13960], 99.90th=[19530], 99.95th=[21890], | 99.99th=[24511] bw ( KiB/s): min=327680, max=1097728, per=100.00%, avg=667843.07, stdev=150189.98, samples=60 iops : min= 80, max= 268, avg=163.03, stdev=36.67, samples=60 write: IOPS=41, BW=168MiB/s (176MB/s)(5028MiB/30007msec) clat (usec): min=10947, max=71532, avg=20352.20, stdev=4797.49 lat (usec): min=11351, max=71977, avg=20902.79, stdev=4838.35 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13960], 10.00th=[14877], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18482], 50.00th=[20579], 60.00th=[22414], | 70.00th=[23462], 80.00th=[24249], 90.00th=[25035], 95.00th=[25822], | 99.00th=[32637], 99.50th=[34341], 99.90th=[58983], 99.95th=[71828], | 99.99th=[71828] bw ( KiB/s): min=147456, max=188416, per=99.95%, avg=171502.80, stdev=7666.60, samples=60 iops : min= 36, max= 46, avg=41.87, stdev= 1.87, samples=60 lat (usec) : 500=57.14%, 750=4.26%, 1000=2.24% lat (msec) : 2=14.82%, 4=0.03%, 10=0.26%, 20=10.54%, 50=10.67% lat (msec) : 100=0.03% cpu : usr=2.37%, sys=1.84%, ctx=6155, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4891,1257,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=652MiB/s (684MB/s), 652MiB/s-652MiB/s (684MB/s-684MB/s), io=19.1GiB (20.5GB), run=30007-30007msec WRITE: bw=168MiB/s (176MB/s), 168MiB/s-168MiB/s (176MB/s-176MB/s), io=5028MiB (5272MB), run=30007-30007msec Disk stats (read/write): vdl: ios=19424/5004, merge=0/0, ticks=4357/81721, in_queue=95481, util=95.48% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983248: Fri Jan 7 14:40:35 2022 read: IOPS=189, BW=758MiB/s (794MB/s)(22.6GiB/30588msec) slat (usec): min=45, max=455945, avg=3882.14, stdev=13279.78 clat (msec): min=280, max=2765, avg=707.75, stdev=166.18 lat (msec): min=280, max=2768, avg=711.64, stdev=166.71 clat percentiles (msec): | 1.00th=[ 418], 5.00th=[ 456], 10.00th=[ 506], 20.00th=[ 584], | 30.00th=[ 617], 40.00th=[ 659], 50.00th=[ 693], 60.00th=[ 735], | 70.00th=[ 776], 80.00th=[ 827], 90.00th=[ 919], 95.00th=[ 986], | 99.00th=[ 1099], 99.50th=[ 1150], 99.90th=[ 1871], 99.95th=[ 2702], | 99.99th=[ 2769] bw ( KiB/s): min= 8192, max=1482752, per=99.45%, avg=771501.95, stdev=272055.81, samples=60 iops : min= 2, max= 362, avg=188.33, stdev=66.41, samples=60 write: IOPS=49, BW=197MiB/s (207MB/s)(6032MiB/30588msec) slat (usec): min=1127, max=444935, avg=4957.03, stdev=17588.62 clat (msec): min=624, max=4872, avg=1290.26, stdev=372.94 lat (msec): min=625, max=4874, avg=1295.21, stdev=372.64 clat percentiles (msec): | 1.00th=[ 718], 5.00th=[ 894], 10.00th=[ 978], 20.00th=[ 1083], | 30.00th=[ 1133], 40.00th=[ 1183], 50.00th=[ 1250], 60.00th=[ 1301], | 70.00th=[ 1368], 80.00th=[ 1452], 90.00th=[ 1586], 95.00th=[ 1720], | 99.00th=[ 2265], 99.50th=[ 4245], 99.90th=[ 571], 99.95th=[ 571], | 99.99th=[ 571] bw ( KiB/s): min= 8192, max=434176, per=99.85%, avg=201624.58, stdev=90560.23, samples=59 iops : min= 2, max= 106, avg=49.22, stdev=22.11, samples=59 lat (msec) : 500=7.38%, 750=43.43%, 1000=27.91%, 2000=20.72%, >=2000=0.55% cpu : usr=6.75%, sys=1.76%, ctx=2739, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5793,1508,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=758MiB/s (794MB/s), 758MiB/s-758MiB/s (794MB/s-794MB/s), io=22.6GiB (24.3GB), run=30588-30588msec WRITE: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=6032MiB (6325MB), run=30588-30588msec Disk stats (read/write): vdl: ios=23172/5989, merge=0/0, ticks=132609/3586883, in_queue=3757438, util=98.43%