30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890673: Fri Jan 7 13:33:20 2022 read: IOPS=2425, BW=9701KiB/s (9933kB/s)(284MiB/30003msec) clat (usec): min=25, max=66899, avg=62.38, stdev=572.54 lat (usec): min=25, max=66900, avg=62.52, stdev=572.54 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 68], | 99.00th=[ 86], 99.50th=[ 98], 99.90th=[ 424], 99.95th=[12911], | 99.99th=[17171] bw ( KiB/s): min= 7360, max=14920, per=100.00%, avg=9701.30, stdev=1148.96, samples=60 iops : min= 1840, max= 3730, avg=2425.32, stdev=287.25, samples=60 write: IOPS=606, BW=2425KiB/s (2484kB/s)(71.1MiB/30003msec) clat (usec): min=197, max=47202, avg=1388.84, stdev=1614.56 lat (usec): min=197, max=47202, avg=1389.15, stdev=1614.55 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 562], 10.00th=[ 775], 20.00th=[ 971], | 30.00th=[ 1090], 40.00th=[ 1172], 50.00th=[ 1221], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1680], 95.00th=[ 1991], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[17171], 99.95th=[39584], | 99.99th=[45876] bw ( KiB/s): min= 1904, max= 3840, per=100.00%, avg=2425.52, stdev=278.26, samples=60 iops : min= 476, max= 960, avg=606.37, stdev=69.58, samples=60 lat (usec) : 50=50.68%, 100=28.95%, 250=0.38%, 500=0.72%, 750=1.02% lat (usec) : 1000=2.53% lat (msec) : 2=14.68%, 4=0.62%, 10=0.28%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.73%, sys=3.64%, ctx=90968, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72762,18193,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9701KiB/s (9933kB/s), 9701KiB/s-9701KiB/s (9933kB/s-9933kB/s), io=284MiB (298MB), run=30003-30003msec WRITE: bw=2425KiB/s (2484kB/s), 2425KiB/s-2425KiB/s (2484kB/s-2484kB/s), io=71.1MiB (74.5MB), run=30003-30003msec Disk stats (read/write): vde: ios=72464/18136, merge=0/0, ticks=942/17440, in_queue=28227, util=93.68% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891553: Fri Jan 7 13:33:57 2022 read: IOPS=2421, BW=9685KiB/s (9918kB/s)(284MiB/30002msec) slat (usec): min=5, max=425, avg= 9.56, stdev= 4.43 clat (nsec): min=583, max=23594k, avg=46768.95, stdev=389516.06 lat (usec): min=23, max=23603, avg=56.49, stdev=389.59 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 33], 50.00th=[ 35], 60.00th=[ 36], | 70.00th=[ 39], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 51], | 99.00th=[ 65], 99.50th=[ 75], 99.90th=[ 355], 99.95th=[11731], | 99.99th=[15926] bw ( KiB/s): min= 8128, max=15152, per=100.00%, avg=9687.00, stdev=1084.77, samples=60 iops : min= 2032, max= 3788, avg=2421.75, stdev=271.18, samples=60 write: IOPS=605, BW=2423KiB/s (2481kB/s)(71.0MiB/30002msec) slat (usec): min=6, max=114, avg=10.60, stdev= 3.91 clat (usec): min=181, max=50859, avg=1405.80, stdev=1680.14 lat (usec): min=190, max=50875, avg=1416.56, stdev=1680.20 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 586], 10.00th=[ 807], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1975], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[34341], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min= 1992, max= 3912, per=100.00%, avg=2423.62, stdev=256.98, samples=60 iops : min= 498, max= 978, avg=605.90, stdev=64.24, samples=60 lat (nsec) : 750=0.03%, 1000=0.09% lat (usec) : 2=0.07%, 4=0.01%, 10=0.04%, 20=0.22%, 50=75.15% lat (usec) : 100=4.24%, 250=0.21%, 500=0.64%, 750=0.89%, 1000=2.38% lat (msec) : 2=15.03%, 4=0.61%, 10=0.30%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.33%, sys=5.88%, ctx=90600, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72644,18176,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=9685KiB/s (9918kB/s), 9685KiB/s-9685KiB/s (9918kB/s-9918kB/s), io=284MiB (298MB), run=30002-30002msec WRITE: bw=2423KiB/s (2481kB/s), 2423KiB/s-2423KiB/s (2481kB/s-2481kB/s), io=71.0MiB (74.4MB), run=30002-30002msec Disk stats (read/write): vde: ios=71982/18031, merge=0/0, ticks=743/17955, in_queue=28109, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892545: Fri Jan 7 13:34:34 2022 read: IOPS=2437, BW=9751KiB/s (9985kB/s)(286MiB/30001msec) clat (usec): min=25, max=23121, avg=59.45, stdev=387.48 lat (usec): min=25, max=23121, avg=59.59, stdev=387.48 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 84], 99.50th=[ 96], 99.90th=[ 416], 99.95th=[11994], | 99.99th=[16188] bw ( KiB/s): min= 7472, max=15008, per=99.98%, avg=9749.02, stdev=1138.27, samples=59 iops : min= 1868, max= 3752, avg=2437.25, stdev=284.57, samples=59 write: IOPS=609, BW=2437KiB/s (2496kB/s)(71.4MiB/30001msec) clat (usec): min=198, max=45029, avg=1393.77, stdev=1656.94 lat (usec): min=198, max=45029, avg=1394.06, stdev=1656.94 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 562], 10.00th=[ 775], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1991], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[36963], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 1984, max= 3856, per=100.00%, avg=2441.08, stdev=264.50, samples=59 iops : min= 496, max= 964, avg=610.27, stdev=66.13, samples=59 lat (usec) : 50=51.31%, 100=28.37%, 250=0.32%, 500=0.74%, 750=1.04% lat (usec) : 1000=2.38% lat (msec) : 2=14.80%, 4=0.65%, 10=0.29%, 20=0.09%, 50=0.02% cpu : usr=1.54%, sys=3.71%, ctx=91424, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73136,18281,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9751KiB/s (9985kB/s), 9751KiB/s-9751KiB/s (9985kB/s-9985kB/s), io=286MiB (300MB), run=30001-30001msec WRITE: bw=2437KiB/s (2496kB/s), 2437KiB/s-2437KiB/s (2496kB/s-2496kB/s), io=71.4MiB (74.9MB), run=30001-30001msec Disk stats (read/write): vde: ios=73084/18268, merge=0/0, ticks=776/17733, in_queue=28271, util=93.82% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892995: Fri Jan 7 13:35:11 2022 read: IOPS=2481, BW=9925KiB/s (10.2MB/s)(291MiB/30037msec) slat (usec): min=2, max=177, avg= 9.76, stdev= 4.04 clat (nsec): min=657, max=2492.3M, avg=864850.50, stdev=37057240.89 lat (usec): min=20, max=2492.4k, avg=874.76, stdev=37057.25 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 28], 10.00th=[ 30], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 35], | 50.00th=[ 37], 60.00th=[ 39], 70.00th=[ 42], | 80.00th=[ 45], 90.00th=[ 51], 95.00th=[ 59], | 99.00th=[ 88], 99.50th=[ 112], 99.90th=[ 498], | 99.95th=[ 505414], 99.99th=[2499806] bw ( KiB/s): min= 7952, max=16064, per=100.00%, avg=9937.13, stdev=1329.27, samples=60 iops : min= 1988, max= 4016, avg=2484.28, stdev=332.32, samples=60 write: IOPS=618, BW=2475KiB/s (2535kB/s)(72.6MiB/30037msec) slat (usec): min=2, max=165, avg=10.91, stdev= 4.01 clat (usec): min=295, max=78473, avg=12624.64, stdev=6073.79 lat (usec): min=299, max=78491, avg=12635.71, stdev=6073.94 clat percentiles (usec): | 1.00th=[ 1647], 5.00th=[ 5932], 10.00th=[ 7701], 20.00th=[ 9503], | 30.00th=[10683], 40.00th=[11469], 50.00th=[12125], 60.00th=[12780], | 70.00th=[13304], 80.00th=[14091], 90.00th=[17957], 95.00th=[20841], | 99.00th=[30278], 99.50th=[61604], 99.90th=[70779], 99.95th=[72877], | 99.99th=[78119] bw ( KiB/s): min= 2104, max= 4048, per=100.00%, avg=2477.67, stdev=271.06, samples=60 iops : min= 526, max= 1012, avg=619.42, stdev=67.76, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.07%, 4=0.01%, 10=0.04%, 20=0.18%, 50=70.84% lat (usec) : 100=8.35%, 250=0.43%, 500=0.02%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.21%, 4=0.27%, 10=4.23%, 20=13.91%, 50=1.18% lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.22%, sys=6.26%, ctx=91134, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=74531,18587,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=9925KiB/s (10.2MB/s), 9925KiB/s-9925KiB/s (10.2MB/s-10.2MB/s), io=291MiB (305MB), run=30037-30037msec WRITE: bw=2475KiB/s (2535kB/s), 2475KiB/s-2475KiB/s (2535kB/s-2535kB/s), io=72.6MiB (76.1MB), run=30037-30037msec Disk stats (read/write): vde: ios=74500/18574, merge=0/0, ticks=58378/225022, in_queue=297918, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894407: Fri Jan 7 13:35:48 2022 read: IOPS=2429, BW=9720KiB/s (9953kB/s)(285MiB/30004msec) clat (usec): min=25, max=24316, avg=59.04, stdev=401.17 lat (usec): min=25, max=24316, avg=59.18, stdev=401.17 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 83], 99.50th=[ 93], 99.90th=[ 396], 99.95th=[12387], | 99.99th=[16450] bw ( KiB/s): min= 7784, max=15096, per=100.00%, avg=9721.07, stdev=1177.16, samples=60 iops : min= 1946, max= 3774, avg=2430.27, stdev=294.29, samples=60 write: IOPS=607, BW=2429KiB/s (2488kB/s)(71.2MiB/30004msec) clat (usec): min=196, max=45391, avg=1400.89, stdev=1668.30 lat (usec): min=197, max=45391, avg=1401.17, stdev=1668.30 clat percentiles (usec): | 1.00th=[ 277], 5.00th=[ 570], 10.00th=[ 783], 20.00th=[ 996], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1713], 95.00th=[ 1991], | 99.00th=[ 9241], 99.50th=[ 9634], 99.90th=[36439], 99.95th=[40109], | 99.99th=[43779] bw ( KiB/s): min= 1920, max= 3904, per=100.00%, avg=2429.47, stdev=278.41, samples=60 iops : min= 480, max= 976, avg=607.37, stdev=69.60, samples=60 lat (usec) : 50=53.66%, 100=26.06%, 250=0.29%, 500=0.73%, 750=1.01% lat (usec) : 1000=2.26% lat (msec) : 2=14.93%, 4=0.67%, 10=0.28%, 20=0.09%, 50=0.03% cpu : usr=1.40%, sys=3.76%, ctx=91138, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72908,18222,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9720KiB/s (9953kB/s), 9720KiB/s-9720KiB/s (9953kB/s-9953kB/s), io=285MiB (299MB), run=30004-30004msec WRITE: bw=2429KiB/s (2488kB/s), 2429KiB/s-2429KiB/s (2488kB/s-2488kB/s), io=71.2MiB (74.6MB), run=30004-30004msec Disk stats (read/write): vde: ios=72908/18221, merge=0/0, ticks=785/17814, in_queue=28390, util=94.18% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894981: Fri Jan 7 13:36:25 2022 read: IOPS=2482, BW=9932KiB/s (10.2MB/s)(292MiB/30100msec) slat (usec): min=2, max=200, avg= 9.38, stdev= 4.00 clat (nsec): min=819, max=2510.4M, avg=861047.04, stdev=35779738.23 lat (usec): min=17, max=2510.4k, avg=870.57, stdev=35779.72 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 32], 30.00th=[ 34], 40.00th=[ 36], | 50.00th=[ 38], 60.00th=[ 41], 70.00th=[ 44], | 80.00th=[ 49], 90.00th=[ 62], 95.00th=[ 81], | 99.00th=[ 204], 99.50th=[ 400], 99.90th=[ 1270], | 99.95th=[ 583009], 99.99th=[2332034] bw ( KiB/s): min= 7896, max=17688, per=100.00%, avg=9964.58, stdev=1277.61, samples=60 iops : min= 1974, max= 4422, avg=2491.13, stdev=319.41, samples=60 write: IOPS=619, BW=2479KiB/s (2538kB/s)(72.9MiB/30100msec) slat (usec): min=2, max=225, avg=10.68, stdev= 4.46 clat (msec): min=6, max=2407, avg=77.13, stdev=63.06 lat (msec): min=6, max=2407, avg=77.15, stdev=63.06 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 54], 10.00th=[ 58], 20.00th=[ 63], | 30.00th=[ 65], 40.00th=[ 67], 50.00th=[ 69], 60.00th=[ 72], | 70.00th=[ 75], 80.00th=[ 80], 90.00th=[ 92], 95.00th=[ 116], | 99.00th=[ 234], 99.50th=[ 451], 99.90th=[ 986], 99.95th=[ 1116], | 99.99th=[ 1921] bw ( KiB/s): min= 1984, max= 4184, per=100.00%, avg=2481.55, stdev=284.76, samples=60 iops : min= 496, max= 1046, avg=620.38, stdev=71.19, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.13%, 50=64.83% lat (usec) : 100=12.82%, 250=1.49%, 500=0.39%, 750=0.14%, 1000=0.03% lat (msec) : 2=0.04%, 4=0.01%, 10=0.18%, 20=0.12%, 50=0.27% lat (msec) : 100=17.81%, 250=1.42%, 500=0.12%, 750=0.05%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.01% cpu : usr=1.34%, sys=5.88%, ctx=88524, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=74738,18654,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=9932KiB/s (10.2MB/s), 9932KiB/s-9932KiB/s (10.2MB/s-10.2MB/s), io=292MiB (306MB), run=30100-30100msec WRITE: bw=2479KiB/s (2538kB/s), 2479KiB/s-2479KiB/s (2538kB/s-2538kB/s), io=72.9MiB (76.4MB), run=30100-30100msec Disk stats (read/write): vde: ios=74671/18587, merge=0/0, ticks=56433/1418793, in_queue=1495768, util=99.43% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895822: Fri Jan 7 13:37:02 2022 read: IOPS=2438, BW=9756KiB/s (9990kB/s)(286MiB/30001msec) clat (usec): min=24, max=23314, avg=59.71, stdev=412.01 lat (usec): min=24, max=23314, avg=59.85, stdev=412.01 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 37], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 67], | 99.00th=[ 84], 99.50th=[ 96], 99.90th=[ 502], 99.95th=[12649], | 99.99th=[16581] bw ( KiB/s): min= 7936, max=14920, per=100.00%, avg=9755.31, stdev=1173.47, samples=59 iops : min= 1984, max= 3730, avg=2438.85, stdev=293.35, samples=59 write: IOPS=609, BW=2438KiB/s (2497kB/s)(71.4MiB/30001msec) clat (usec): min=203, max=43072, avg=1392.02, stdev=1662.94 lat (usec): min=204, max=43072, avg=1392.29, stdev=1662.94 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 570], 10.00th=[ 775], 20.00th=[ 988], | 30.00th=[ 1106], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1467], 90.00th=[ 1680], 95.00th=[ 1991], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[36439], 99.95th=[38011], | 99.99th=[42730] bw ( KiB/s): min= 2000, max= 3832, per=100.00%, avg=2443.86, stdev=265.65, samples=59 iops : min= 500, max= 958, avg=610.97, stdev=66.41, samples=59 lat (usec) : 50=53.64%, 100=26.05%, 250=0.28%, 500=0.78%, 750=1.02% lat (usec) : 1000=2.36% lat (msec) : 2=14.84%, 4=0.64%, 10=0.28%, 20=0.09%, 50=0.03% cpu : usr=1.41%, sys=3.76%, ctx=91462, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=73169,18288,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9756KiB/s (9990kB/s), 9756KiB/s-9756KiB/s (9990kB/s-9990kB/s), io=286MiB (300MB), run=30001-30001msec WRITE: bw=2438KiB/s (2497kB/s), 2438KiB/s-2438KiB/s (2497kB/s-2497kB/s), io=71.4MiB (74.9MB), run=30001-30001msec Disk stats (read/write): vde: ios=73034/18257, merge=0/0, ticks=820/17742, in_queue=28261, util=93.91% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896947: Fri Jan 7 13:37:39 2022 read: IOPS=2495, BW=9982KiB/s (10.2MB/s)(294MiB/30198msec) slat (usec): min=2, max=419, avg= 9.48, stdev= 4.86 clat (nsec): min=678, max=2509.6M, avg=904117.61, stdev=37103093.47 lat (usec): min=18, max=2509.6k, avg=913.74, stdev=37103.09 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 31], | 20.00th=[ 33], 30.00th=[ 35], 40.00th=[ 36], | 50.00th=[ 39], 60.00th=[ 41], 70.00th=[ 45], | 80.00th=[ 51], 90.00th=[ 70], 95.00th=[ 127], | 99.00th=[ 750], 99.50th=[ 1270], 99.90th=[ 2999], | 99.95th=[ 566232], 99.99th=[2499806] bw ( KiB/s): min= 7896, max=18624, per=100.00%, avg=10047.33, stdev=1511.21, samples=60 iops : min= 1974, max= 4656, avg=2511.83, stdev=377.80, samples=60 write: IOPS=622, BW=2490KiB/s (2549kB/s)(73.4MiB/30198msec) slat (usec): min=2, max=703, avg=10.69, stdev= 6.86 clat (msec): min=9, max=2832, avg=156.90, stdev=148.50 lat (msec): min=9, max=2832, avg=156.91, stdev=148.50 clat percentiles (msec): | 1.00th=[ 17], 5.00th=[ 104], 10.00th=[ 111], 20.00th=[ 120], | 30.00th=[ 129], 40.00th=[ 136], 50.00th=[ 142], 60.00th=[ 148], | 70.00th=[ 155], 80.00th=[ 163], 90.00th=[ 190], 95.00th=[ 207], | 99.00th=[ 617], 99.50th=[ 1250], 99.90th=[ 2836], 99.95th=[ 2836], | 99.99th=[ 2836] bw ( KiB/s): min= 2104, max= 4000, per=100.00%, avg=2493.07, stdev=272.51, samples=60 iops : min= 526, max= 1000, avg=623.27, stdev=68.13, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.15%, 50=63.52% lat (usec) : 100=11.07%, 250=3.21%, 500=0.81%, 750=0.39%, 1000=0.22% lat (msec) : 2=0.41%, 4=0.10%, 10=0.01%, 20=0.25%, 50=0.09% lat (msec) : 100=0.45%, 250=18.63%, 500=0.33%, 750=0.08%, 1000=0.06% lat (msec) : 2000=0.09%, >=2000=0.05% cpu : usr=1.35%, sys=5.80%, ctx=87456, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=75356,18796,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=9982KiB/s (10.2MB/s), 9982KiB/s-9982KiB/s (10.2MB/s-10.2MB/s), io=294MiB (309MB), run=30198-30198msec WRITE: bw=2490KiB/s (2549kB/s), 2490KiB/s-2490KiB/s (2549kB/s-2549kB/s), io=73.4MiB (76.0MB), run=30198-30198msec Disk stats (read/write): vde: ios=75335/18693, merge=0/0, ticks=60877/2910617, in_queue=2995540, util=99.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897511: Fri Jan 7 13:38:16 2022 read: IOPS=2416, BW=9666KiB/s (9898kB/s)(283MiB/30002msec) clat (usec): min=25, max=74088, avg=62.74, stdev=508.98 lat (usec): min=26, max=74088, avg=62.88, stdev=508.98 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 89], 99.50th=[ 103], 99.90th=[ 494], 99.95th=[12256], | 99.99th=[16909] bw ( KiB/s): min= 7639, max=14328, per=100.00%, avg=9674.15, stdev=985.17, samples=60 iops : min= 1909, max= 3582, avg=2418.38, stdev=246.33, samples=60 write: IOPS=604, BW=2419KiB/s (2477kB/s)(70.9MiB/30002msec) clat (usec): min=194, max=44646, avg=1393.54, stdev=1641.40 lat (usec): min=195, max=44646, avg=1393.86, stdev=1641.41 clat percentiles (usec): | 1.00th=[ 281], 5.00th=[ 545], 10.00th=[ 758], 20.00th=[ 971], | 30.00th=[ 1090], 40.00th=[ 1172], 50.00th=[ 1237], 60.00th=[ 1287], | 70.00th=[ 1369], 80.00th=[ 1483], 90.00th=[ 1696], 95.00th=[ 1991], | 99.00th=[ 9372], 99.50th=[ 9634], 99.90th=[17433], 99.95th=[36963], | 99.99th=[43779] bw ( KiB/s): min= 2040, max= 3688, per=100.00%, avg=2420.38, stdev=241.15, samples=60 iops : min= 510, max= 922, avg=605.05, stdev=60.27, samples=60 lat (usec) : 50=45.85%, 100=33.69%, 250=0.42%, 500=0.80%, 750=1.12% lat (usec) : 1000=2.40% lat (msec) : 2=14.67%, 4=0.64%, 10=0.30%, 20=0.08%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.57%, sys=3.85%, ctx=90648, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=72498,18142,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9666KiB/s (9898kB/s), 9666KiB/s-9666KiB/s (9898kB/s-9898kB/s), io=283MiB (297MB), run=30002-30002msec WRITE: bw=2419KiB/s (2477kB/s), 2419KiB/s-2419KiB/s (2477kB/s-2477kB/s), io=70.9MiB (74.3MB), run=30002-30002msec Disk stats (read/write): vde: ios=72496/18139, merge=0/0, ticks=839/17485, in_queue=28220, util=93.57% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898371: Fri Jan 7 13:38:53 2022 read: IOPS=2479, BW=9918KiB/s (10.2MB/s)(293MiB/30238msec) slat (usec): min=2, max=238226, avg=316.35, stdev=1895.80 clat (usec): min=599, max=2597.6k, avg=25964.82, stdev=41964.87 lat (usec): min=601, max=2597.6k, avg=26281.30, stdev=42009.29 clat percentiles (usec): | 1.00th=[ 1188], 5.00th=[ 8586], 10.00th=[ 14222], | 20.00th=[ 17433], 30.00th=[ 19530], 40.00th=[ 21365], | 50.00th=[ 23200], 60.00th=[ 25297], 70.00th=[ 27395], | 80.00th=[ 30278], 90.00th=[ 35390], 95.00th=[ 41157], | 99.00th=[ 78119], 99.50th=[ 166724], 99.90th=[ 263193], | 99.95th=[ 624952], 99.99th=[2533360] bw ( KiB/s): min= 7928, max=20032, per=100.00%, avg=9988.42, stdev=1609.58, samples=60 iops : min= 1982, max= 5008, avg=2497.13, stdev=402.42, samples=60 write: IOPS=618, BW=2475KiB/s (2534kB/s)(73.1MiB/30238msec) slat (usec): min=2, max=197327, avg=325.88, stdev=2363.82 clat (msec): min=14, max=2775, avg=217.43, stdev=133.15 lat (msec): min=14, max=2775, avg=217.75, stdev=133.15 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 155], 10.00th=[ 167], 20.00th=[ 182], | 30.00th=[ 190], 40.00th=[ 197], 50.00th=[ 201], 60.00th=[ 209], | 70.00th=[ 218], 80.00th=[ 232], 90.00th=[ 255], 95.00th=[ 292], | 99.00th=[ 726], 99.50th=[ 1020], 99.90th=[ 2198], 99.95th=[ 2769], | 99.99th=[ 2769] bw ( KiB/s): min= 2040, max= 4192, per=100.00%, avg=2476.33, stdev=301.53, samples=60 iops : min= 510, max= 1048, avg=619.08, stdev=75.38, samples=60 lat (usec) : 750=0.10%, 1000=0.34% lat (msec) : 2=1.35%, 4=1.34%, 10=1.39%, 20=21.26%, 50=52.18% lat (msec) : 100=2.09%, 250=17.47%, 500=2.17%, 750=0.11%, 1000=0.09% lat (msec) : 2000=0.09%, >=2000=0.04% cpu : usr=0.69%, sys=1.93%, ctx=10281, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=74977,18707,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=9918KiB/s (10.2MB/s), 9918KiB/s-9918KiB/s (10.2MB/s-10.2MB/s), io=293MiB (307MB), run=30238-30238msec WRITE: bw=2475KiB/s (2534kB/s), 2475KiB/s-2475KiB/s (2534kB/s-2534kB/s), io=73.1MiB (76.6MB), run=30238-30238msec Disk stats (read/write): vde: ios=74976/18703, merge=0/0, ticks=68871/3585066, in_queue=3673580, util=99.45% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898988: Fri Jan 7 13:39:30 2022 read: IOPS=2372, BW=37.1MiB/s (38.9MB/s)(1112MiB/30001msec) clat (usec): min=26, max=32880, avg=64.05, stdev=402.31 lat (usec): min=26, max=32881, avg=64.19, stdev=402.31 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 51], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 74], | 99.00th=[ 94], 99.50th=[ 109], 99.90th=[ 668], 99.95th=[12256], | 99.99th=[16319] bw ( KiB/s): min=29649, max=56832, per=100.00%, avg=38081.45, stdev=4539.13, samples=60 iops : min= 1853, max= 3552, avg=2379.80, stdev=283.73, samples=60 write: IOPS=593, BW=9501KiB/s (9729kB/s)(278MiB/30001msec) clat (usec): min=246, max=44478, avg=1417.38, stdev=1641.89 lat (usec): min=250, max=44479, avg=1418.83, stdev=1641.86 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 578], 10.00th=[ 791], 20.00th=[ 1012], | 30.00th=[ 1123], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1500], 90.00th=[ 1729], 95.00th=[ 2008], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[40633], | 99.99th=[42730] bw ( KiB/s): min= 7646, max=14464, per=100.00%, avg=9528.38, stdev=987.37, samples=60 iops : min= 477, max= 904, avg=595.28, stdev=61.76, samples=60 lat (usec) : 50=39.15%, 100=40.27%, 250=0.45%, 500=0.76%, 750=1.05% lat (usec) : 1000=2.09% lat (msec) : 2=15.13%, 4=0.70%, 10=0.28%, 20=0.09%, 50=0.02% cpu : usr=1.42%, sys=3.89%, ctx=89010, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71187,17815,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.1MiB/s (38.9MB/s), 37.1MiB/s-37.1MiB/s (38.9MB/s-38.9MB/s), io=1112MiB (1166MB), run=30001-30001msec WRITE: bw=9501KiB/s (9729kB/s), 9501KiB/s-9501KiB/s (9729kB/s-9729kB/s), io=278MiB (292MB), run=30001-30001msec Disk stats (read/write): vde: ios=71187/17813, merge=0/0, ticks=776/17641, in_queue=28284, util=93.95% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899807: Fri Jan 7 13:40:07 2022 read: IOPS=2369, BW=37.0MiB/s (38.8MB/s)(1111MiB/30001msec) slat (usec): min=5, max=246, avg=10.32, stdev= 4.32 clat (nsec): min=660, max=19353k, avg=50136.11, stdev=369427.58 lat (usec): min=23, max=19367, avg=60.61, stdev=369.48 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 30], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 36], 50.00th=[ 38], 60.00th=[ 40], | 70.00th=[ 42], 80.00th=[ 46], 90.00th=[ 51], 95.00th=[ 56], | 99.00th=[ 73], 99.50th=[ 83], 99.90th=[ 453], 99.95th=[11469], | 99.99th=[15401] bw ( KiB/s): min=30880, max=55104, per=100.00%, avg=37907.33, stdev=4222.69, samples=60 iops : min= 1930, max= 3444, avg=2369.20, stdev=263.92, samples=60 write: IOPS=592, BW=9482KiB/s (9710kB/s)(278MiB/30001msec) slat (usec): min=8, max=160, avg=12.55, stdev= 4.49 clat (usec): min=212, max=45040, avg=1424.64, stdev=1683.34 lat (usec): min=230, max=45051, avg=1437.36, stdev=1683.40 clat percentiles (usec): | 1.00th=[ 310], 5.00th=[ 594], 10.00th=[ 816], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1729], 95.00th=[ 2008], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[36963], 99.95th=[41681], | 99.99th=[43779] bw ( KiB/s): min= 7680, max=14176, per=100.00%, avg=9482.43, stdev=947.42, samples=60 iops : min= 480, max= 886, avg=592.65, stdev=59.21, samples=60 lat (nsec) : 750=0.02%, 1000=0.06% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.09%, 50=71.47% lat (usec) : 100=8.07%, 250=0.13%, 500=0.74%, 750=0.90%, 1000=2.03% lat (msec) : 2=15.34%, 4=0.68%, 10=0.28%, 20=0.10%, 50=0.02% cpu : usr=1.41%, sys=5.98%, ctx=88712, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71074,17780,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.0MiB/s (38.8MB/s), 37.0MiB/s-37.0MiB/s (38.8MB/s-38.8MB/s), io=1111MiB (1164MB), run=30001-30001msec WRITE: bw=9482KiB/s (9710kB/s), 9482KiB/s-9482KiB/s (9710kB/s-9710kB/s), io=278MiB (291MB), run=30001-30001msec Disk stats (read/write): vde: ios=70792/17699, merge=0/0, ticks=708/17824, in_queue=28110, util=94.01% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901194: Fri Jan 7 13:40:44 2022 read: IOPS=2360, BW=36.9MiB/s (38.7MB/s)(1107MiB/30001msec) clat (usec): min=26, max=40365, avg=60.56, stdev=388.08 lat (usec): min=26, max=40366, avg=60.70, stdev=388.09 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 70], | 99.00th=[ 87], 99.50th=[ 98], 99.90th=[ 404], 99.95th=[11076], | 99.99th=[16319] bw ( KiB/s): min=31136, max=61024, per=100.00%, avg=37788.75, stdev=4694.00, samples=59 iops : min= 1946, max= 3814, avg=2361.80, stdev=293.38, samples=59 write: IOPS=590, BW=9444KiB/s (9671kB/s)(277MiB/30001msec) clat (usec): min=224, max=51002, avg=1442.10, stdev=1656.65 lat (usec): min=226, max=51003, avg=1443.52, stdev=1656.64 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 594], 10.00th=[ 816], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2024], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[34866], 99.95th=[38536], | 99.99th=[44827] bw ( KiB/s): min= 7808, max=15744, per=100.00%, avg=9447.59, stdev=1111.38, samples=59 iops : min= 488, max= 984, avg=590.47, stdev=69.46, samples=59 lat (usec) : 50=46.95%, 100=32.69%, 250=0.28%, 500=0.73%, 750=0.93% lat (usec) : 1000=1.93% lat (msec) : 2=15.37%, 4=0.73%, 10=0.28%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.40%, sys=3.74%, ctx=88544, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70829,17708,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=36.9MiB/s (38.7MB/s), 36.9MiB/s-36.9MiB/s (38.7MB/s-38.7MB/s), io=1107MiB (1160MB), run=30001-30001msec WRITE: bw=9444KiB/s (9671kB/s), 9444KiB/s-9444KiB/s (9671kB/s-9671kB/s), io=277MiB (290MB), run=30001-30001msec Disk stats (read/write): vde: ios=70292/17594, merge=0/0, ticks=699/17811, in_queue=28165, util=94.20% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901857: Fri Jan 7 13:41:21 2022 read: IOPS=2380, BW=37.2MiB/s (38.0MB/s)(1117MiB/30040msec) slat (usec): min=2, max=149, avg=10.88, stdev= 4.39 clat (nsec): min=694, max=2490.8M, avg=892707.59, stdev=37278392.89 lat (usec): min=19, max=2490.8k, avg=903.75, stdev=37278.38 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 49], 90.00th=[ 56], 95.00th=[ 64], | 99.00th=[ 99], 99.50th=[ 135], 99.90th=[ 865], | 99.95th=[ 557843], 99.99th=[2139096] bw ( KiB/s): min=30080, max=60032, per=100.00%, avg=38130.67, stdev=4599.95, samples=60 iops : min= 1880, max= 3752, avg=2383.17, stdev=287.50, samples=60 write: IOPS=595, BW=9533KiB/s (9762kB/s)(280MiB/30040msec) slat (usec): min=4, max=182, avg=13.71, stdev= 4.94 clat (usec): min=388, max=73215, avg=13147.17, stdev=6636.23 lat (usec): min=396, max=73230, avg=13161.04, stdev=6636.50 clat percentiles (usec): | 1.00th=[ 1729], 5.00th=[ 4359], 10.00th=[ 6783], 20.00th=[ 9634], | 30.00th=[11207], 40.00th=[12125], 50.00th=[12911], 60.00th=[13566], | 70.00th=[14222], 80.00th=[15008], 90.00th=[18482], 95.00th=[21890], | 99.00th=[35914], 99.50th=[62129], 99.90th=[69731], 99.95th=[72877], | 99.99th=[72877] bw ( KiB/s): min= 7584, max=15136, per=100.00%, avg=9541.33, stdev=1004.75, samples=60 iops : min= 474, max= 946, avg=596.33, stdev=62.80, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.07%, 4=0.01%, 10=0.02%, 20=0.12%, 50=66.42% lat (usec) : 100=12.54%, 250=0.62%, 500=0.06%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.24%, 4=0.54%, 10=3.68%, 20=13.91%, 50=1.48% lat (msec) : 100=0.17%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.01% cpu : usr=1.48%, sys=6.25%, ctx=87732, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71496,17899,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=37.2MiB/s (38.0MB/s), 37.2MiB/s-37.2MiB/s (38.0MB/s-38.0MB/s), io=1117MiB (1171MB), run=30040-30040msec WRITE: bw=9533KiB/s (9762kB/s), 9533KiB/s-9533KiB/s (9762kB/s-9762kB/s), io=280MiB (293MB), run=30040-30040msec Disk stats (read/write): vde: ios=71495/17899, merge=0/0, ticks=60528/225974, in_queue=298258, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902760: Fri Jan 7 13:41:58 2022 read: IOPS=2381, BW=37.2MiB/s (39.0MB/s)(1116MiB/30001msec) clat (usec): min=26, max=24630, avg=62.48, stdev=393.35 lat (usec): min=26, max=24631, avg=62.63, stdev=393.35 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 91], 99.50th=[ 103], 99.90th=[ 461], 99.95th=[12256], | 99.99th=[16450] bw ( KiB/s): min=27712, max=58144, per=100.00%, avg=38097.60, stdev=4337.51, samples=60 iops : min= 1732, max= 3634, avg=2381.10, stdev=271.09, samples=60 write: IOPS=595, BW=9533KiB/s (9762kB/s)(279MiB/30001msec) clat (usec): min=244, max=44342, avg=1418.08, stdev=1606.36 lat (usec): min=246, max=44343, avg=1419.53, stdev=1606.35 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 586], 10.00th=[ 799], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1401], 80.00th=[ 1532], 90.00th=[ 1745], 95.00th=[ 2024], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[35390], 99.95th=[40109], | 99.99th=[43254] bw ( KiB/s): min= 7712, max=15040, per=100.00%, avg=9532.80, stdev=1007.09, samples=60 iops : min= 482, max= 940, avg=595.80, stdev=62.94, samples=60 lat (usec) : 50=42.82%, 100=36.70%, 250=0.38%, 500=0.75%, 750=0.98% lat (usec) : 1000=2.03% lat (msec) : 2=15.21%, 4=0.78%, 10=0.24%, 20=0.10%, 50=0.02% cpu : usr=1.53%, sys=3.86%, ctx=89316, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71433,17875,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.2MiB/s (39.0MB/s), 37.2MiB/s-37.2MiB/s (39.0MB/s-39.0MB/s), io=1116MiB (1170MB), run=30001-30001msec WRITE: bw=9533KiB/s (9762kB/s), 9533KiB/s-9533KiB/s (9762kB/s-9762kB/s), io=279MiB (293MB), run=30001-30001msec Disk stats (read/write): vde: ios=71012/17766, merge=0/0, ticks=756/17542, in_queue=28152, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903783: Fri Jan 7 13:42:35 2022 read: IOPS=2422, BW=37.8MiB/s (39.7MB/s)(1139MiB/30098msec) slat (usec): min=2, max=307, avg=10.00, stdev= 5.49 clat (nsec): min=706, max=2517.4M, avg=792449.19, stdev=34881037.16 lat (usec): min=23, max=2517.4k, avg=802.59, stdev=34881.05 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 30], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 37], | 50.00th=[ 39], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 50], 90.00th=[ 61], 95.00th=[ 80], | 99.00th=[ 255], 99.50th=[ 490], 99.90th=[ 1172], | 99.95th=[ 346031], 99.99th=[2399142] bw ( KiB/s): min=31360, max=71968, per=100.00%, avg=38882.67, stdev=5236.07, samples=60 iops : min= 1960, max= 4498, avg=2430.17, stdev=327.25, samples=60 write: IOPS=605, BW=9687KiB/s (9919kB/s)(285MiB/30098msec) slat (usec): min=4, max=182, avg=13.22, stdev= 6.13 clat (msec): min=7, max=2438, avg=79.34, stdev=68.02 lat (msec): min=7, max=2438, avg=79.35, stdev=68.02 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 56], 10.00th=[ 60], 20.00th=[ 65], | 30.00th=[ 67], 40.00th=[ 69], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 83], 90.00th=[ 93], 95.00th=[ 120], | 99.00th=[ 255], 99.50th=[ 510], 99.90th=[ 1045], 99.95th=[ 1368], | 99.99th=[ 2400] bw ( KiB/s): min= 8064, max=16960, per=100.00%, avg=9693.33, stdev=1184.08, samples=60 iops : min= 504, max= 1060, avg=605.83, stdev=74.01, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.09%, 4=0.01%, 10=0.03%, 20=0.10%, 50=64.61% lat (usec) : 100=12.79%, 250=1.54%, 500=0.43%, 750=0.19%, 1000=0.08% lat (msec) : 2=0.04%, 4=0.01%, 10=0.04%, 20=0.25%, 50=0.23% lat (msec) : 100=17.93%, 250=1.36%, 500=0.12%, 750=0.05%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.01% cpu : usr=1.28%, sys=6.01%, ctx=88138, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=72908,18222,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.7MB/s), 37.8MiB/s-37.8MiB/s (39.7MB/s-39.7MB/s), io=1139MiB (1195MB), run=30098-30098msec WRITE: bw=9687KiB/s (9919kB/s), 9687KiB/s-9687KiB/s (9919kB/s-9919kB/s), io=285MiB (299MB), run=30098-30098msec Disk stats (read/write): vde: ios=72905/18186, merge=0/0, ticks=52625/1431233, in_queue=1499248, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904246: Fri Jan 7 13:43:12 2022 read: IOPS=2354, BW=36.8MiB/s (38.6MB/s)(1104MiB/30001msec) clat (usec): min=24, max=19187, avg=61.76, stdev=365.00 lat (usec): min=24, max=19187, avg=61.90, stdev=365.00 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 90], 99.50th=[ 102], 99.90th=[ 408], 99.95th=[10159], | 99.99th=[16319] bw ( KiB/s): min=29536, max=53248, per=100.00%, avg=37687.63, stdev=4275.63, samples=59 iops : min= 1846, max= 3328, avg=2355.47, stdev=267.23, samples=59 write: IOPS=588, BW=9424KiB/s (9650kB/s)(276MiB/30001msec) clat (usec): min=247, max=114637, avg=1440.98, stdev=1876.28 lat (usec): min=248, max=114638, avg=1442.41, stdev=1876.27 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 578], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1336], | 70.00th=[ 1418], 80.00th=[ 1532], 90.00th=[ 1762], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[36439], 99.95th=[42206], | 99.99th=[46400] bw ( KiB/s): min= 6720, max=13888, per=100.00%, avg=9427.03, stdev=999.75, samples=59 iops : min= 420, max= 868, avg=589.17, stdev=62.50, samples=59 lat (usec) : 50=42.10%, 100=37.45%, 250=0.36%, 500=0.77%, 750=0.91% lat (usec) : 1000=1.92% lat (msec) : 2=15.33%, 4=0.78%, 10=0.27%, 20=0.09%, 50=0.02% lat (msec) : 250=0.01% cpu : usr=1.36%, sys=3.96%, ctx=88321, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=70640,17670,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=36.8MiB/s (38.6MB/s), 36.8MiB/s-36.8MiB/s (38.6MB/s-38.6MB/s), io=1104MiB (1157MB), run=30001-30001msec WRITE: bw=9424KiB/s (9650kB/s), 9424KiB/s-9424KiB/s (9650kB/s-9650kB/s), io=276MiB (290MB), run=30001-30001msec Disk stats (read/write): vde: ios=70064/17540, merge=0/0, ticks=665/17754, in_queue=28171, util=94.33% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905220: Fri Jan 7 13:43:49 2022 read: IOPS=2427, BW=37.9MiB/s (39.8MB/s)(1145MiB/30197msec) slat (usec): min=2, max=326, avg= 9.74, stdev= 4.52 clat (nsec): min=654, max=2525.9M, avg=1020388.96, stdev=41212069.39 lat (usec): min=23, max=2525.9k, avg=1030.26, stdev=41212.05 clat percentiles (usec): | 1.00th=[ 26], 5.00th=[ 31], 10.00th=[ 32], | 20.00th=[ 34], 30.00th=[ 35], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 52], 90.00th=[ 73], 95.00th=[ 135], | 99.00th=[ 963], 99.50th=[ 1598], 99.90th=[ 4883], | 99.95th=[ 574620], 99.99th=[2499806] bw ( KiB/s): min=27447, max=76761, per=100.00%, avg=39530.78, stdev=6979.14, samples=60 iops : min= 1715, max= 4797, avg=2470.38, stdev=436.19, samples=60 write: IOPS=606, BW=9705KiB/s (9938kB/s)(286MiB/30197msec) slat (usec): min=3, max=202, avg=12.75, stdev= 4.74 clat (msec): min=12, max=2197, avg=160.66, stdev=101.33 lat (msec): min=12, max=2197, avg=160.67, stdev=101.33 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 105], 10.00th=[ 121], 20.00th=[ 136], | 30.00th=[ 142], 40.00th=[ 146], 50.00th=[ 150], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 167], 90.00th=[ 199], 95.00th=[ 218], | 99.00th=[ 472], 99.50th=[ 961], 99.90th=[ 1586], 99.95th=[ 1636], | 99.99th=[ 2072] bw ( KiB/s): min= 7200, max=16744, per=100.00%, avg=9827.48, stdev=1384.75, samples=60 iops : min= 450, max= 1046, avg=613.92, stdev=86.50, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.09%, 50=61.91% lat (usec) : 100=12.32%, 250=3.30%, 500=0.84%, 750=0.42%, 1000=0.24% lat (msec) : 2=0.55%, 4=0.08%, 10=0.07%, 20=0.15%, 50=0.23% lat (msec) : 100=0.41%, 250=18.61%, 500=0.42%, 750=0.07%, 1000=0.06% lat (msec) : 2000=0.08%, >=2000=0.02% cpu : usr=1.41%, sys=5.61%, ctx=85604, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73304,18316,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=37.9MiB/s (39.8MB/s), 37.9MiB/s-37.9MiB/s (39.8MB/s-39.8MB/s), io=1145MiB (1201MB), run=30197-30197msec WRITE: bw=9705KiB/s (9938kB/s), 9705KiB/s-9705KiB/s (9938kB/s-9938kB/s), io=286MiB (300MB), run=30197-30197msec Disk stats (read/write): vde: ios=73267/18214, merge=0/0, ticks=65144/2907314, in_queue=2996018, util=99.58% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905796: Fri Jan 7 13:44:26 2022 read: IOPS=2366, BW=36.0MiB/s (38.8MB/s)(1110MiB/30001msec) clat (usec): min=25, max=19227, avg=61.35, stdev=380.79 lat (usec): min=25, max=19227, avg=61.49, stdev=380.79 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 63], 95.00th=[ 69], | 99.00th=[ 86], 99.50th=[ 99], 99.90th=[ 285], 99.95th=[11994], | 99.99th=[16319] bw ( KiB/s): min=29408, max=53120, per=100.00%, avg=37875.03, stdev=4210.14, samples=60 iops : min= 1838, max= 3320, avg=2367.17, stdev=263.13, samples=60 write: IOPS=592, BW=9475KiB/s (9702kB/s)(278MiB/30001msec) clat (usec): min=223, max=42226, avg=1432.98, stdev=1661.29 lat (usec): min=224, max=42227, avg=1434.39, stdev=1661.29 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 586], 10.00th=[ 816], 20.00th=[ 1029], | 30.00th=[ 1139], 40.00th=[ 1205], 50.00th=[ 1270], 60.00th=[ 1319], | 70.00th=[ 1401], 80.00th=[ 1516], 90.00th=[ 1745], 95.00th=[ 2040], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[37487], 99.95th=[40109], | 99.99th=[41681] bw ( KiB/s): min= 7552, max=13792, per=100.00%, avg=9475.30, stdev=912.37, samples=60 iops : min= 472, max= 862, avg=592.20, stdev=57.02, samples=60 lat (usec) : 50=45.41%, 100=34.22%, 250=0.28%, 500=0.72%, 750=0.95% lat (usec) : 1000=1.94% lat (msec) : 2=15.34%, 4=0.76%, 10=0.26%, 20=0.11%, 50=0.02% cpu : usr=1.36%, sys=3.89%, ctx=88785, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=71011,17766,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=36.0MiB/s (38.8MB/s), 36.0MiB/s-36.0MiB/s (38.8MB/s-38.8MB/s), io=1110MiB (1163MB), run=30001-30001msec WRITE: bw=9475KiB/s (9702kB/s), 9475KiB/s-9475KiB/s (9702kB/s-9702kB/s), io=278MiB (291MB), run=30001-30001msec Disk stats (read/write): vde: ios=70486/17636, merge=0/0, ticks=739/17797, in_queue=28157, util=94.19% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906618: Fri Jan 7 13:45:03 2022 read: IOPS=2421, BW=37.8MiB/s (39.7MB/s)(1143MiB/30216msec) slat (usec): min=2, max=210557, avg=323.52, stdev=2010.88 clat (usec): min=644, max=2555.8k, avg=26595.68, stdev=43276.77 lat (usec): min=647, max=2555.8k, avg=26919.35, stdev=43332.11 clat percentiles (usec): | 1.00th=[ 1270], 5.00th=[ 7504], 10.00th=[ 14615], | 20.00th=[ 18220], 30.00th=[ 20317], 40.00th=[ 22152], | 50.00th=[ 23987], 60.00th=[ 25822], 70.00th=[ 28181], | 80.00th=[ 30802], 90.00th=[ 35390], 95.00th=[ 41157], | 99.00th=[ 79168], 99.50th=[ 170918], 99.90th=[ 238027], | 99.95th=[ 633340], 99.99th=[2533360] bw ( KiB/s): min=31936, max=75232, per=100.00%, avg=38981.45, stdev=5679.30, samples=60 iops : min= 1996, max= 4702, avg=2436.33, stdev=354.96, samples=60 write: IOPS=605, BW=9683KiB/s (9915kB/s)(286MiB/30216msec) slat (usec): min=3, max=195506, avg=336.48, stdev=2039.86 clat (msec): min=4, max=2930, avg=222.38, stdev=159.22 lat (msec): min=14, max=2930, avg=222.72, stdev=159.20 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 157], 10.00th=[ 167], 20.00th=[ 178], | 30.00th=[ 188], 40.00th=[ 199], 50.00th=[ 205], 60.00th=[ 211], | 70.00th=[ 220], 80.00th=[ 228], 90.00th=[ 253], 95.00th=[ 284], | 99.00th=[ 844], 99.50th=[ 1502], 99.90th=[ 2534], 99.95th=[ 2937], | 99.99th=[ 2937] bw ( KiB/s): min= 7552, max=15680, per=100.00%, avg=9682.43, stdev=1015.63, samples=60 iops : min= 472, max= 980, avg=605.15, stdev=63.48, samples=60 lat (usec) : 750=0.05%, 1000=0.28% lat (msec) : 2=1.53%, 4=1.49%, 10=1.06%, 20=17.93%, 50=55.59% lat (msec) : 100=2.04%, 250=17.83%, 500=1.80%, 750=0.10%, 1000=0.10% lat (msec) : 2000=0.15%, >=2000=0.05% cpu : usr=0.89%, sys=1.98%, ctx=10243, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=73155,18286,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=37.8MiB/s (39.7MB/s), 37.8MiB/s-37.8MiB/s (39.7MB/s-39.7MB/s), io=1143MiB (1199MB), run=30216-30216msec WRITE: bw=9683KiB/s (9915kB/s), 9683KiB/s-9683KiB/s (9915kB/s-9915kB/s), io=286MiB (300MB), run=30216-30216msec Disk stats (read/write): vde: ios=73154/18174, merge=0/0, ticks=70184/3563401, in_queue=3665424, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908073: Fri Jan 7 13:45:40 2022 read: IOPS=2268, BW=70.9MiB/s (74.3MB/s)(2127MiB/30002msec) clat (usec): min=29, max=22534, avg=67.55, stdev=414.61 lat (usec): min=29, max=22534, avg=67.68, stdev=414.61 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 63], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 93], 99.50th=[ 105], 99.90th=[ 490], 99.95th=[12911], | 99.99th=[16450] bw ( KiB/s): min=56000, max=115968, per=100.00%, avg=72593.07, stdev=8093.37, samples=60 iops : min= 1750, max= 3624, avg=2268.53, stdev=252.92, samples=60 write: IOPS=567, BW=17.7MiB/s (18.6MB/s)(532MiB/30002msec) clat (usec): min=287, max=45386, avg=1480.13, stdev=1710.17 lat (usec): min=290, max=45389, avg=1483.07, stdev=1710.18 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 603], 10.00th=[ 816], 20.00th=[ 1037], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[36439], 99.95th=[41157], | 99.99th=[43779] bw ( KiB/s): min=14848, max=29760, per=100.00%, avg=18163.20, stdev=1920.79, samples=60 iops : min= 464, max= 930, avg=567.60, stdev=60.02, samples=60 lat (usec) : 50=30.23%, 100=49.24%, 250=0.41%, 500=0.64%, 750=1.06% lat (usec) : 1000=1.84% lat (msec) : 2=15.25%, 4=0.92%, 10=0.26%, 20=0.13%, 50=0.02% cpu : usr=1.40%, sys=3.82%, ctx=85092, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68056,17029,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=70.9MiB/s (74.3MB/s), 70.9MiB/s-70.9MiB/s (74.3MB/s-74.3MB/s), io=2127MiB (2230MB), run=30002-30002msec WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=532MiB (558MB), run=30002-30002msec Disk stats (read/write): vde: ios=67772/16949, merge=0/0, ticks=784/17454, in_queue=28129, util=93.90% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908677: Fri Jan 7 13:46:17 2022 read: IOPS=2282, BW=71.3MiB/s (74.8MB/s)(2140MiB/30001msec) slat (usec): min=5, max=419, avg=11.41, stdev= 4.62 clat (nsec): min=741, max=64067k, avg=55504.59, stdev=486086.33 lat (usec): min=27, max=64080, avg=67.06, stdev=486.16 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 46], 80.00th=[ 49], 90.00th=[ 55], 95.00th=[ 60], | 99.00th=[ 77], 99.50th=[ 87], 99.90th=[ 523], 99.95th=[12649], | 99.99th=[16909] bw ( KiB/s): min=53888, max=109952, per=100.00%, avg=73035.30, stdev=7924.19, samples=60 iops : min= 1684, max= 3436, avg=2282.35, stdev=247.63, samples=60 write: IOPS=571, BW=17.8MiB/s (18.7MB/s)(536MiB/30001msec) slat (usec): min=9, max=201, avg=15.09, stdev= 4.92 clat (usec): min=249, max=43001, avg=1459.93, stdev=1602.39 lat (usec): min=261, max=43019, avg=1475.18, stdev=1602.39 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 603], 10.00th=[ 840], 20.00th=[ 1074], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1811], 95.00th=[ 2114], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[16909], 99.95th=[40633], | 99.99th=[42730] bw ( KiB/s): min=14656, max=28288, per=100.00%, avg=18276.77, stdev=1762.05, samples=60 iops : min= 458, max= 884, avg=571.13, stdev=55.08, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.05%, 50=66.57% lat (usec) : 100=13.04%, 250=0.11%, 500=0.65%, 750=0.96%, 1000=1.64% lat (msec) : 2=15.54%, 4=0.97%, 10=0.23%, 20=0.10%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.50%, sys=6.08%, ctx=85515, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68473,17136,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=71.3MiB/s (74.8MB/s), 71.3MiB/s-71.3MiB/s (74.8MB/s-74.8MB/s), io=2140MiB (2244MB), run=30001-30001msec WRITE: bw=17.8MiB/s (18.7MB/s), 17.8MiB/s-17.8MiB/s (18.7MB/s-18.7MB/s), io=536MiB (562MB), run=30001-30001msec Disk stats (read/write): vde: ios=68173/17063, merge=0/0, ticks=849/17550, in_queue=28023, util=93.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909513: Fri Jan 7 13:46:54 2022 read: IOPS=2281, BW=71.3MiB/s (74.8MB/s)(2139MiB/30001msec) clat (usec): min=27, max=25396, avg=64.00, stdev=416.13 lat (usec): min=27, max=25396, avg=64.14, stdev=416.14 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 72], | 99.00th=[ 90], 99.50th=[ 100], 99.90th=[ 545], 99.95th=[11994], | 99.99th=[17433] bw ( KiB/s): min=58560, max=106496, per=100.00%, avg=73219.19, stdev=7785.83, samples=59 iops : min= 1830, max= 3328, avg=2288.07, stdev=243.29, samples=59 write: IOPS=570, BW=17.8MiB/s (18.7MB/s)(535MiB/30001msec) clat (usec): min=260, max=42602, avg=1484.45, stdev=1638.06 lat (usec): min=263, max=42605, avg=1487.37, stdev=1638.04 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 619], 10.00th=[ 857], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1303], 60.00th=[ 1369], | 70.00th=[ 1450], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[17695], 99.95th=[39584], | 99.99th=[42206] bw ( KiB/s): min=15168, max=27840, per=100.00%, avg=18303.71, stdev=1782.67, samples=59 iops : min= 474, max= 870, avg=571.98, stdev=55.71, samples=59 lat (usec) : 50=40.66%, 100=38.92%, 250=0.30%, 500=0.64%, 750=0.87% lat (usec) : 1000=1.59% lat (msec) : 2=15.66%, 4=0.96%, 10=0.28%, 20=0.10%, 50=0.03% cpu : usr=1.38%, sys=3.76%, ctx=85595, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68458,17129,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=71.3MiB/s (74.8MB/s), 71.3MiB/s-71.3MiB/s (74.8MB/s-74.8MB/s), io=2139MiB (2243MB), run=30001-30001msec WRITE: bw=17.8MiB/s (18.7MB/s), 17.8MiB/s-17.8MiB/s (18.7MB/s-18.7MB/s), io=535MiB (561MB), run=30001-30001msec Disk stats (read/write): vde: ios=68061/17031, merge=0/0, ticks=760/17809, in_queue=28147, util=94.25% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910044: Fri Jan 7 13:47:31 2022 read: IOPS=2301, BW=71.9MiB/s (75.4MB/s)(2160MiB/30037msec) slat (usec): min=3, max=239, avg=11.14, stdev= 5.47 clat (nsec): min=651, max=2496.4M, avg=1284117.31, stdev=49968672.47 lat (usec): min=26, max=2496.4k, avg=1295.43, stdev=49968.68 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 35], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 42], 70.00th=[ 45], | 80.00th=[ 49], 90.00th=[ 56], 95.00th=[ 63], | 99.00th=[ 91], 99.50th=[ 113], 99.90th=[ 586], | 99.95th=[1166017], 99.99th=[2499806] bw ( KiB/s): min=59200, max=126076, per=100.00%, avg=73745.40, stdev=9600.91, samples=60 iops : min= 1850, max= 3939, avg=2304.50, stdev=299.96, samples=60 write: IOPS=575, BW=17.0MiB/s (18.9MB/s)(540MiB/30037msec) slat (usec): min=5, max=167, avg=15.94, stdev= 7.31 clat (usec): min=442, max=79218, avg=12169.18, stdev=6777.55 lat (usec): min=454, max=79231, avg=12185.28, stdev=6778.00 clat percentiles (usec): | 1.00th=[ 1172], 5.00th=[ 2114], 10.00th=[ 4359], 20.00th=[ 8160], | 30.00th=[ 9896], 40.00th=[11338], 50.00th=[12256], 60.00th=[13042], | 70.00th=[13829], 80.00th=[14746], 90.00th=[17433], 95.00th=[21627], | 99.00th=[31327], 99.50th=[60556], 99.90th=[68682], 99.95th=[73925], | 99.99th=[78119] bw ( KiB/s): min=15552, max=31743, per=100.00%, avg=18433.17, stdev=2136.09, samples=60 iops : min= 486, max= 991, avg=576.02, stdev=66.65, samples=60 lat (nsec) : 750=0.01%, 1000=0.11% lat (usec) : 2=0.09%, 4=0.01%, 10=0.03%, 20=0.10%, 50=66.49% lat (usec) : 100=12.59%, 250=0.45%, 500=0.03%, 750=0.04%, 1000=0.08% lat (msec) : 2=0.83%, 4=0.89%, 10=4.33%, 20=12.36%, 50=1.36% lat (msec) : 100=0.14%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.03% cpu : usr=1.54%, sys=6.06%, ctx=85229, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=69131,17285,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=71.9MiB/s (75.4MB/s), 71.9MiB/s-71.9MiB/s (75.4MB/s-75.4MB/s), io=2160MiB (2265MB), run=30037-30037msec WRITE: bw=17.0MiB/s (18.9MB/s), 17.0MiB/s-17.0MiB/s (18.9MB/s-18.9MB/s), io=540MiB (566MB), run=30037-30037msec Disk stats (read/write): vde: ios=68910/17220, merge=0/0, ticks=82817/200327, in_queue=296562, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910983: Fri Jan 7 13:48:08 2022 read: IOPS=2270, BW=70.0MiB/s (74.4MB/s)(2129MiB/30005msec) clat (usec): min=28, max=19990, avg=65.76, stdev=406.33 lat (usec): min=28, max=19990, avg=65.90, stdev=406.34 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 55], | 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 74], | 99.00th=[ 96], 99.50th=[ 113], 99.90th=[ 734], 99.95th=[12518], | 99.99th=[16909] bw ( KiB/s): min=53504, max=113664, per=100.00%, avg=72683.37, stdev=8695.02, samples=60 iops : min= 1672, max= 3552, avg=2271.33, stdev=271.71, samples=60 write: IOPS=568, BW=17.8MiB/s (18.6MB/s)(533MiB/30005msec) clat (usec): min=279, max=44908, avg=1485.07, stdev=1638.11 lat (usec): min=282, max=44911, avg=1488.02, stdev=1638.11 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 619], 10.00th=[ 848], 20.00th=[ 1074], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1598], 90.00th=[ 1827], 95.00th=[ 2114], | 99.00th=[ 9503], 99.50th=[ 9765], 99.90th=[21890], 99.95th=[38536], | 99.99th=[44303] bw ( KiB/s): min=12736, max=28992, per=100.00%, avg=18192.18, stdev=2057.18, samples=60 iops : min= 398, max= 906, avg=568.50, stdev=64.28, samples=60 lat (usec) : 50=36.45%, 100=42.91%, 250=0.48%, 500=0.66%, 750=0.92% lat (usec) : 1000=1.60% lat (msec) : 2=15.61%, 4=0.99%, 10=0.24%, 20=0.12%, 50=0.02% cpu : usr=1.39%, sys=3.98%, ctx=85201, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68137,17054,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=70.0MiB/s (74.4MB/s), 70.0MiB/s-70.0MiB/s (74.4MB/s-74.4MB/s), io=2129MiB (2233MB), run=30005-30005msec WRITE: bw=17.8MiB/s (18.6MB/s), 17.8MiB/s-17.8MiB/s (18.6MB/s-18.6MB/s), io=533MiB (559MB), run=30005-30005msec Disk stats (read/write): vde: ios=67790/16958, merge=0/0, ticks=767/17655, in_queue=28176, util=94.13% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911958: Fri Jan 7 13:48:45 2022 read: IOPS=2323, BW=72.6MiB/s (76.1MB/s)(2184MiB/30082msec) slat (usec): min=2, max=173, avg=10.68, stdev= 4.41 clat (nsec): min=716, max=2572.4M, avg=1135891.32, stdev=45828891.86 lat (usec): min=26, max=2572.4k, avg=1146.73, stdev=45828.93 clat percentiles (usec): | 1.00th=[ 27], 5.00th=[ 31], 10.00th=[ 33], | 20.00th=[ 34], 30.00th=[ 36], 40.00th=[ 38], | 50.00th=[ 40], 60.00th=[ 43], 70.00th=[ 46], | 80.00th=[ 51], 90.00th=[ 60], 95.00th=[ 72], | 99.00th=[ 245], 99.50th=[ 490], 99.90th=[ 1237], | 99.95th=[ 692061], 99.99th=[2499806] bw ( KiB/s): min=55168, max=139456, per=100.00%, avg=74551.77, stdev=11028.78, samples=60 iops : min= 1724, max= 4358, avg=2329.73, stdev=344.65, samples=60 write: IOPS=581, BW=18.2MiB/s (19.1MB/s)(547MiB/30082msec) slat (usec): min=5, max=133, avg=16.22, stdev= 5.76 clat (msec): min=5, max=1999, avg=81.33, stdev=62.16 lat (msec): min=5, max=1999, avg=81.35, stdev=62.16 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 56], 10.00th=[ 61], 20.00th=[ 66], | 30.00th=[ 69], 40.00th=[ 71], 50.00th=[ 74], 60.00th=[ 77], | 70.00th=[ 80], 80.00th=[ 85], 90.00th=[ 99], 95.00th=[ 121], | 99.00th=[ 243], 99.50th=[ 527], 99.90th=[ 986], 99.95th=[ 1150], | 99.99th=[ 1603] bw ( KiB/s): min=14784, max=32448, per=99.99%, avg=18607.50, stdev=2187.87, samples=60 iops : min= 462, max= 1014, avg=581.48, stdev=68.37, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.07%, 4=0.01%, 10=0.02%, 20=0.09%, 50=63.45% lat (usec) : 100=14.40%, 250=1.12%, 500=0.40%, 750=0.17%, 1000=0.09% lat (msec) : 2=0.05%, 10=0.03%, 20=0.28%, 50=0.22%, 100=17.62% lat (msec) : 250=1.69%, 500=0.10%, 750=0.06%, 1000=0.04%, 2000=0.03% lat (msec) : >=2000=0.02% cpu : usr=1.59%, sys=5.79%, ctx=84763, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=69890,17494,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=72.6MiB/s (76.1MB/s), 72.6MiB/s-72.6MiB/s (76.1MB/s-76.1MB/s), io=2184MiB (2290MB), run=30082-30082msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=547MiB (573MB), run=30082-30082msec Disk stats (read/write): vde: ios=69890/17450, merge=0/0, ticks=76021/1408376, in_queue=1499730, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912590: Fri Jan 7 13:49:22 2022 read: IOPS=2262, BW=70.7MiB/s (74.1MB/s)(2121MiB/30001msec) clat (usec): min=27, max=64654, avg=65.96, stdev=534.76 lat (usec): min=27, max=64654, avg=66.10, stdev=534.76 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 73], | 99.00th=[ 93], 99.50th=[ 106], 99.90th=[ 486], 99.95th=[13304], | 99.99th=[17957] bw ( KiB/s): min=54400, max=114368, per=100.00%, avg=72411.02, stdev=8554.38, samples=60 iops : min= 1700, max= 3574, avg=2262.83, stdev=267.32, samples=60 write: IOPS=566, BW=17.7MiB/s (18.6MB/s)(531MiB/30001msec) clat (usec): min=282, max=64395, avg=1490.36, stdev=1677.07 lat (usec): min=291, max=64398, avg=1493.35, stdev=1677.02 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 619], 10.00th=[ 865], 20.00th=[ 1090], | 30.00th=[ 1188], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1598], 90.00th=[ 1844], 95.00th=[ 2147], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[36963], 99.95th=[41157], | 99.99th=[45876] bw ( KiB/s): min=14016, max=29376, per=100.00%, avg=18121.17, stdev=2151.50, samples=60 iops : min= 438, max= 918, avg=566.28, stdev=67.23, samples=60 lat (usec) : 50=40.30%, 100=39.18%, 250=0.39%, 500=0.64%, 750=0.89% lat (usec) : 1000=1.56% lat (msec) : 2=15.57%, 4=1.11%, 10=0.24%, 20=0.11%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.48%, sys=3.69%, ctx=84878, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=67883,16989,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=70.7MiB/s (74.1MB/s), 70.7MiB/s-70.7MiB/s (74.1MB/s-74.1MB/s), io=2121MiB (2224MB), run=30001-30001msec WRITE: bw=17.7MiB/s (18.6MB/s), 17.7MiB/s-17.7MiB/s (18.6MB/s-18.6MB/s), io=531MiB (557MB), run=30001-30001msec Disk stats (read/write): vde: ios=67630/16905, merge=0/0, ticks=890/17574, in_queue=28114, util=94.08% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913425: Fri Jan 7 13:49:59 2022 read: IOPS=2323, BW=72.6MiB/s (76.1MB/s)(2191MiB/30181msec) slat (usec): min=2, max=234, avg=11.27, stdev= 4.79 clat (nsec): min=707, max=2568.9M, avg=1158768.95, stdev=45070362.24 lat (usec): min=25, max=2568.9k, avg=1170.19, stdev=45070.37 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 47], 70.00th=[ 50], | 80.00th=[ 56], 90.00th=[ 68], 95.00th=[ 99], | 99.00th=[ 1139], 99.50th=[ 1893], 99.90th=[ 3130], | 99.95th=[ 843056], 99.99th=[2499806] bw ( KiB/s): min=60672, max=152128, per=100.00%, avg=74802.63, stdev=11918.85, samples=60 iops : min= 1896, max= 4754, avg=2337.57, stdev=372.47, samples=60 write: IOPS=581, BW=18.2MiB/s (19.1MB/s)(549MiB/30181msec) slat (usec): min=6, max=287, avg=17.85, stdev= 6.81 clat (msec): min=11, max=2676, avg=167.18, stdev=106.21 lat (msec): min=11, max=2676, avg=167.20, stdev=106.21 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 114], 10.00th=[ 128], 20.00th=[ 140], | 30.00th=[ 148], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 161], | 70.00th=[ 167], 80.00th=[ 178], 90.00th=[ 203], 95.00th=[ 220], | 99.00th=[ 502], 99.50th=[ 1083], 99.90th=[ 1485], 99.95th=[ 1636], | 99.99th=[ 2668] bw ( KiB/s): min=15488, max=33152, per=100.00%, avg=18624.13, stdev=2292.80, samples=60 iops : min= 484, max= 1036, avg=582.00, stdev=71.65, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.07%, 50=55.98% lat (usec) : 100=19.90%, 250=1.61%, 500=0.69%, 750=0.40%, 1000=0.34% lat (msec) : 2=0.56%, 4=0.29%, 20=0.13%, 50=0.27%, 100=0.22% lat (msec) : 250=18.91%, 500=0.31%, 750=0.07%, 1000=0.03%, 2000=0.13% lat (msec) : >=2000=0.02% cpu : usr=1.47%, sys=6.21%, ctx=82769, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=70125,17557,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=72.6MiB/s (76.1MB/s), 72.6MiB/s-72.6MiB/s (76.1MB/s-76.1MB/s), io=2191MiB (2298MB), run=30181-30181msec WRITE: bw=18.2MiB/s (19.1MB/s), 18.2MiB/s-18.2MiB/s (19.1MB/s-19.1MB/s), io=549MiB (575MB), run=30181-30181msec Disk stats (read/write): vde: ios=70123/17484, merge=0/0, ticks=74755/2901800, in_queue=3000757, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914738: Fri Jan 7 13:50:36 2022 read: IOPS=2284, BW=71.4MiB/s (74.9MB/s)(2142MiB/30001msec) clat (usec): min=27, max=23827, avg=63.39, stdev=409.16 lat (usec): min=27, max=23827, avg=63.53, stdev=409.16 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 47], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 71], | 99.00th=[ 88], 99.50th=[ 99], 99.90th=[ 519], 99.95th=[12256], | 99.99th=[18220] bw ( KiB/s): min=54957, max=120384, per=100.00%, avg=73152.80, stdev=9253.55, samples=60 iops : min= 1717, max= 3762, avg=2285.87, stdev=289.21, samples=60 write: IOPS=571, BW=17.9MiB/s (18.7MB/s)(536MiB/30001msec) clat (usec): min=276, max=79150, avg=1484.57, stdev=1744.81 lat (usec): min=278, max=79153, avg=1487.52, stdev=1744.80 clat percentiles (usec): | 1.00th=[ 363], 5.00th=[ 627], 10.00th=[ 873], 20.00th=[ 1090], | 30.00th=[ 1205], 40.00th=[ 1254], 50.00th=[ 1319], 60.00th=[ 1385], | 70.00th=[ 1467], 80.00th=[ 1582], 90.00th=[ 1827], 95.00th=[ 2089], | 99.00th=[ 9372], 99.50th=[ 9765], 99.90th=[36439], 99.95th=[40633], | 99.99th=[43779] bw ( KiB/s): min=14714, max=30976, per=100.00%, avg=18303.78, stdev=2148.54, samples=60 iops : min= 459, max= 968, avg=571.90, stdev=67.21, samples=60 lat (usec) : 50=41.94%, 100=37.65%, 250=0.28%, 500=0.62%, 750=0.81% lat (usec) : 1000=1.52% lat (msec) : 2=15.85%, 4=0.95%, 10=0.25%, 20=0.09%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.32%, sys=3.91%, ctx=85698, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=68536,17151,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=71.4MiB/s (74.9MB/s), 71.4MiB/s-71.4MiB/s (74.9MB/s-74.9MB/s), io=2142MiB (2246MB), run=30001-30001msec WRITE: bw=17.9MiB/s (18.7MB/s), 17.9MiB/s-17.9MiB/s (18.7MB/s-18.7MB/s), io=536MiB (562MB), run=30001-30001msec Disk stats (read/write): vde: ios=68536/17149, merge=0/0, ticks=770/17945, in_queue=28310, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915354: Fri Jan 7 13:51:13 2022 read: IOPS=2311, BW=72.2MiB/s (75.7MB/s)(2183MiB/30217msec) slat (usec): min=2, max=180082, avg=348.38, stdev=2136.20 clat (usec): min=807, max=2616.8k, avg=28092.76, stdev=50905.52 lat (usec): min=811, max=2616.8k, avg=28441.29, stdev=50951.55 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 6783], 10.00th=[ 15139], | 20.00th=[ 19006], 30.00th=[ 21365], 40.00th=[ 23462], | 50.00th=[ 25297], 60.00th=[ 27395], 70.00th=[ 29754], | 80.00th=[ 32900], 90.00th=[ 37487], 95.00th=[ 43254], | 99.00th=[ 84411], 99.50th=[ 181404], 99.90th=[ 240124], | 99.95th=[ 935330], 99.99th=[2533360] bw ( KiB/s): min=59008, max=144128, per=100.00%, avg=74454.78, stdev=11259.26, samples=60 iops : min= 1844, max= 4504, avg=2326.70, stdev=351.85, samples=60 write: IOPS=578, BW=18.1MiB/s (18.0MB/s)(546MiB/30217msec) slat (usec): min=4, max=57130, avg=314.75, stdev=1473.26 clat (msec): min=23, max=3507, avg=231.70, stdev=172.56 lat (msec): min=23, max=3507, avg=232.02, stdev=172.56 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 144], 10.00th=[ 161], 20.00th=[ 188], | 30.00th=[ 203], 40.00th=[ 211], 50.00th=[ 218], 60.00th=[ 226], | 70.00th=[ 234], 80.00th=[ 247], 90.00th=[ 268], 95.00th=[ 292], | 99.00th=[ 818], 99.50th=[ 1183], 99.90th=[ 3440], 99.95th=[ 3440], | 99.99th=[ 3440] bw ( KiB/s): min=14528, max=29504, per=99.88%, avg=18489.18, stdev=1951.18, samples=60 iops : min= 454, max= 922, avg=577.78, stdev=60.97, samples=60 lat (usec) : 1000=0.14% lat (msec) : 2=1.40%, 4=1.56%, 10=1.45%, 20=14.41%, 50=58.90% lat (msec) : 100=2.12%, 250=16.41%, 500=3.25%, 750=0.09%, 1000=0.08% lat (msec) : 2000=0.13%, >=2000=0.07% cpu : usr=1.13%, sys=1.99%, ctx=9845, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=69852,17480,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=72.2MiB/s (75.7MB/s), 72.2MiB/s-72.2MiB/s (75.7MB/s-75.7MB/s), io=2183MiB (2289MB), run=30217-30217msec WRITE: bw=18.1MiB/s (18.0MB/s), 18.1MiB/s-18.1MiB/s (18.0MB/s-18.0MB/s), io=546MiB (573MB), run=30217-30217msec Disk stats (read/write): vde: ios=69851/17381, merge=0/0, ticks=86612/3547686, in_queue=3670134, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916228: Fri Jan 7 13:51:50 2022 read: IOPS=2091, BW=131MiB/s (137MB/s)(3922MiB/30001msec) clat (usec): min=37, max=26555, avg=77.45, stdev=425.37 lat (usec): min=37, max=26555, avg=77.59, stdev=425.38 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 74], 90.00th=[ 82], 95.00th=[ 90], | 99.00th=[ 121], 99.50th=[ 143], 99.90th=[ 1029], 99.95th=[13042], | 99.99th=[17433] bw ( KiB/s): min=102656, max=192768, per=100.00%, avg=134264.34, stdev=14565.04, samples=59 iops : min= 1604, max= 3012, avg=2097.85, stdev=227.57, samples=59 write: IOPS=526, BW=32.9MiB/s (34.5MB/s)(986MiB/30001msec) clat (usec): min=363, max=49737, avg=1578.50, stdev=1705.43 lat (usec): min=370, max=49751, avg=1584.67, stdev=1705.38 clat percentiles (usec): | 1.00th=[ 457], 5.00th=[ 660], 10.00th=[ 857], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1467], | 70.00th=[ 1565], 80.00th=[ 1696], 90.00th=[ 1942], 95.00th=[ 2245], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[37487], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=28544, max=50048, per=100.00%, avg=33691.31, stdev=3163.58, samples=59 iops : min= 446, max= 782, avg=526.42, stdev=49.43, samples=59 lat (usec) : 50=4.47%, 100=73.39%, 250=1.87%, 500=0.48%, 750=1.09% lat (usec) : 1000=1.34% lat (msec) : 2=15.55%, 4=1.40%, 10=0.25%, 20=0.13%, 50=0.03% cpu : usr=1.48%, sys=3.89%, ctx=78535, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62744,15782,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3922MiB (4112MB), run=30001-30001msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=986MiB (1034MB), run=30001-30001msec Disk stats (read/write): vde: ios=62428/15692, merge=0/0, ticks=736/17110, in_queue=28104, util=94.06% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916845: Fri Jan 7 13:52:27 2022 read: IOPS=2094, BW=131MiB/s (137MB/s)(3927MiB/30001msec) slat (usec): min=6, max=247, avg=13.09, stdev= 4.69 clat (nsec): min=794, max=23105k, avg=60277.48, stdev=442062.33 lat (usec): min=29, max=23115, avg=73.52, stdev=442.13 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 36], 10.00th=[ 38], 20.00th=[ 40], | 30.00th=[ 41], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 67], | 99.00th=[ 84], 99.50th=[ 97], 99.90th=[ 807], 99.95th=[13960], | 99.99th=[19006] bw ( KiB/s): min=104960, max=209152, per=100.00%, avg=134331.31, stdev=16113.35, samples=59 iops : min= 1640, max= 3268, avg=2098.90, stdev=251.78, samples=59 write: IOPS=526, BW=32.9MiB/s (34.5MB/s)(988MiB/30001msec) slat (usec): min=11, max=420, avg=20.24, stdev= 7.81 clat (usec): min=348, max=41977, avg=1577.80, stdev=1635.39 lat (usec): min=367, max=41999, avg=1598.21, stdev=1635.46 clat percentiles (usec): | 1.00th=[ 429], 5.00th=[ 660], 10.00th=[ 889], 20.00th=[ 1139], | 30.00th=[ 1270], 40.00th=[ 1336], 50.00th=[ 1401], 60.00th=[ 1483], | 70.00th=[ 1565], 80.00th=[ 1713], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35390], 99.95th=[38536], | 99.99th=[41681] bw ( KiB/s): min=28160, max=53760, per=100.00%, avg=33720.73, stdev=3641.16, samples=59 iops : min= 440, max= 840, avg=526.86, stdev=56.90, samples=59 lat (nsec) : 1000=0.02% lat (usec) : 2=0.04%, 4=0.01%, 10=0.02%, 20=0.06%, 50=55.54% lat (usec) : 100=23.87%, 250=0.22%, 500=0.52%, 750=0.93%, 1000=1.24% lat (msec) : 2=15.68%, 4=1.45%, 10=0.24%, 20=0.14%, 50=0.02% cpu : usr=1.60%, sys=6.17%, ctx=78577, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62836,15804,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3927MiB (4118MB), run=30001-30001msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=988MiB (1036MB), run=30001-30001msec Disk stats (read/write): vde: ios=62445/15695, merge=0/0, ticks=779/17319, in_queue=27964, util=93.56% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917792: Fri Jan 7 13:53:04 2022 read: IOPS=2080, BW=130MiB/s (136MB/s)(3901MiB/30002msec) clat (usec): min=31, max=65053, avg=71.57, stdev=490.66 lat (usec): min=32, max=65053, avg=71.72, stdev=490.66 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 80], | 99.00th=[ 106], 99.50th=[ 125], 99.90th=[ 1090], 99.95th=[13042], | 99.99th=[17695] bw ( KiB/s): min=102656, max=197760, per=100.00%, avg=133166.93, stdev=16249.22, samples=60 iops : min= 1604, max= 3090, avg=2080.73, stdev=253.89, samples=60 write: IOPS=523, BW=32.7MiB/s (34.3MB/s)(981MiB/30002msec) clat (usec): min=365, max=44608, avg=1611.74, stdev=1694.95 lat (usec): min=370, max=44614, avg=1618.37, stdev=1694.88 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 676], 10.00th=[ 922], 20.00th=[ 1172], | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1729], 90.00th=[ 1991], 95.00th=[ 2311], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[35390], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=27008, max=51456, per=100.00%, avg=33474.13, stdev=3359.19, samples=60 iops : min= 422, max= 804, avg=523.03, stdev=52.49, samples=60 lat (usec) : 50=22.56%, 100=56.31%, 250=0.91%, 500=0.48%, 750=0.87% lat (usec) : 1000=1.15% lat (msec) : 2=15.71%, 4=1.60%, 10=0.26%, 20=0.14%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.61%, sys=3.76%, ctx=78124, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62422,15692,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3901MiB (4091MB), run=30002-30002msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=981MiB (1028MB), run=30002-30002msec Disk stats (read/write): vde: ios=62007/15565, merge=0/0, ticks=793/17494, in_queue=27991, util=93.73% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918737: Fri Jan 7 13:53:41 2022 read: IOPS=2135, BW=133MiB/s (140MB/s)(4011MiB/30042msec) slat (usec): min=3, max=279, avg=12.56, stdev= 5.62 clat (nsec): min=725, max=2531.9M, avg=1250476.26, stdev=50707893.65 lat (usec): min=29, max=2531.9k, avg=1263.20, stdev=50707.88 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 44], 60.00th=[ 46], 70.00th=[ 49], | 80.00th=[ 53], 90.00th=[ 61], 95.00th=[ 69], | 99.00th=[ 96], 99.50th=[ 119], 99.90th=[ 709], | 99.95th=[ 717226], 99.99th=[2499806] bw ( KiB/s): min=94080, max=241536, per=100.00%, avg=136888.53, stdev=19423.81, samples=60 iops : min= 1470, max= 3774, avg=2138.87, stdev=303.49, samples=60 write: IOPS=536, BW=33.5MiB/s (35.2MB/s)(1007MiB/30042msec) slat (usec): min=7, max=213, avg=20.64, stdev= 8.11 clat (usec): min=695, max=72620, avg=13580.32, stdev=6382.19 lat (usec): min=708, max=72642, avg=13601.13, stdev=6382.71 clat percentiles (usec): | 1.00th=[ 2212], 5.00th=[ 6456], 10.00th=[ 8225], 20.00th=[ 9896], | 30.00th=[10814], 40.00th=[11863], 50.00th=[13042], 60.00th=[14091], | 70.00th=[15139], 80.00th=[16188], 90.00th=[18482], 95.00th=[22938], | 99.00th=[33424], 99.50th=[63177], 99.90th=[68682], 99.95th=[69731], | 99.99th=[71828] bw ( KiB/s): min=27264, max=61312, per=100.00%, avg=34374.53, stdev=4259.60, samples=60 iops : min= 426, max= 958, avg=537.10, stdev=66.56, samples=60 lat (nsec) : 750=0.01%, 1000=0.10% lat (usec) : 2=0.09%, 4=0.01%, 10=0.02%, 20=0.10%, 50=57.93% lat (usec) : 100=20.98%, 250=0.55%, 500=0.03%, 750=0.02%, 1000=0.01% lat (msec) : 2=0.13%, 4=0.34%, 10=3.83%, 20=14.20%, 50=1.43% lat (msec) : 100=0.15%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.54%, sys=6.07%, ctx=79166, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=64169,16115,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4011MiB (4205MB), run=30042-30042msec WRITE: bw=33.5MiB/s (35.2MB/s), 33.5MiB/s-33.5MiB/s (35.2MB/s-35.2MB/s), io=1007MiB (1056MB), run=30042-30042msec Disk stats (read/write): vde: ios=64168/16115, merge=0/0, ticks=75932/210689, in_queue=298397, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919318: Fri Jan 7 13:54:18 2022 read: IOPS=2114, BW=132MiB/s (139MB/s)(3964MiB/30001msec) clat (usec): min=31, max=25091, avg=69.32, stdev=443.25 lat (usec): min=32, max=25091, avg=69.46, stdev=443.25 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 77], | 99.00th=[ 96], 99.50th=[ 113], 99.90th=[ 840], 99.95th=[13042], | 99.99th=[17957] bw ( KiB/s): min=109825, max=215272, per=100.00%, avg=137141.98, stdev=16996.32, samples=60 iops : min= 1716, max= 3363, avg=2142.37, stdev=265.58, samples=60 write: IOPS=530, BW=33.2MiB/s (34.8MB/s)(996MiB/30001msec) clat (usec): min=343, max=47684, avg=1592.16, stdev=1647.21 lat (usec): min=349, max=47690, avg=1598.60, stdev=1647.14 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 693], 10.00th=[ 963], 20.00th=[ 1188], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1582], 80.00th=[ 1729], 90.00th=[ 1958], 95.00th=[ 2212], | 99.00th=[ 9634], 99.50th=[ 9896], 99.90th=[38011], 99.95th=[42206], | 99.99th=[43779] bw ( KiB/s): min=26742, max=56371, per=100.00%, avg=34440.30, stdev=3948.94, samples=60 iops : min= 417, max= 880, avg=537.63, stdev=61.64, samples=60 lat (usec) : 50=28.97%, 100=50.29%, 250=0.56%, 500=0.46%, 750=0.79% lat (usec) : 1000=1.03% lat (msec) : 2=16.09%, 4=1.43%, 10=0.24%, 20=0.12%, 50=0.02% cpu : usr=1.64%, sys=3.49%, ctx=79368, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63425,15928,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=132MiB/s (139MB/s), 132MiB/s-132MiB/s (139MB/s-139MB/s), io=3964MiB (4157MB), run=30001-30001msec WRITE: bw=33.2MiB/s (34.8MB/s), 33.2MiB/s-33.2MiB/s (34.8MB/s-34.8MB/s), io=996MiB (1044MB), run=30001-30001msec Disk stats (read/write): vde: ios=63404/15923, merge=0/0, ticks=799/17600, in_queue=28415, util=94.46% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920166: Fri Jan 7 13:54:55 2022 read: IOPS=2131, BW=133MiB/s (140MB/s)(4012MiB/30107msec) slat (usec): min=2, max=291, avg=10.91, stdev= 4.64 clat (nsec): min=813, max=2525.7M, avg=1263485.16, stdev=48470078.91 lat (usec): min=29, max=2525.7k, avg=1274.54, stdev=48470.11 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 32], 10.00th=[ 34], | 20.00th=[ 37], 30.00th=[ 38], 40.00th=[ 41], | 50.00th=[ 43], 60.00th=[ 47], 70.00th=[ 51], | 80.00th=[ 57], 90.00th=[ 66], 95.00th=[ 77], | 99.00th=[ 363], 99.50th=[ 685], 99.90th=[ 1926], | 99.95th=[ 994051], 99.99th=[2499806] bw ( KiB/s): min=111616, max=252800, per=100.00%, avg=136936.28, stdev=19275.64, samples=60 iops : min= 1744, max= 3950, avg=2139.62, stdev=301.21, samples=60 write: IOPS=535, BW=33.5MiB/s (35.1MB/s)(1008MiB/30107msec) slat (usec): min=7, max=294, avg=21.10, stdev= 8.69 clat (msec): min=10, max=2824, avg=88.27, stdev=77.59 lat (msec): min=10, max=2824, avg=88.29, stdev=77.59 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 62], 10.00th=[ 68], 20.00th=[ 73], | 30.00th=[ 77], 40.00th=[ 79], 50.00th=[ 81], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 90], 90.00th=[ 101], 95.00th=[ 129], | 99.00th=[ 203], 99.50th=[ 558], 99.90th=[ 1368], 99.95th=[ 1787], | 99.99th=[ 2165] bw ( KiB/s): min=28032, max=58624, per=100.00%, avg=34301.92, stdev=3928.16, samples=60 iops : min= 438, max= 916, avg=535.95, stdev=61.40, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.05%, 50=53.95% lat (usec) : 100=23.75%, 250=1.07%, 500=0.42%, 750=0.26%, 1000=0.10% lat (msec) : 2=0.18%, 20=0.27%, 50=0.19%, 100=17.51%, 250=1.93% lat (msec) : 500=0.08%, 750=0.06%, 1000=0.03%, 2000=0.05%, >=2000=0.03% cpu : usr=1.75%, sys=5.31%, ctx=77944, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=64188,16120,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4012MiB (4207MB), run=30107-30107msec WRITE: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=1008MiB (1056MB), run=30107-30107msec Disk stats (read/write): vde: ios=64182/16109, merge=0/0, ticks=73112/1411189, in_queue=1500374, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921317: Fri Jan 7 13:55:32 2022 read: IOPS=2081, BW=130MiB/s (136MB/s)(3904MiB/30012msec) clat (usec): min=31, max=31178, avg=72.31, stdev=465.39 lat (usec): min=32, max=31178, avg=72.46, stdev=465.39 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 74], 95.00th=[ 80], | 99.00th=[ 100], 99.50th=[ 114], 99.90th=[ 1188], 99.95th=[14222], | 99.99th=[17695] bw ( KiB/s): min=98432, max=211072, per=100.00%, avg=133257.03, stdev=16449.40, samples=60 iops : min= 1538, max= 3298, avg=2082.12, stdev=257.02, samples=60 write: IOPS=523, BW=32.7MiB/s (34.3MB/s)(981MiB/30012msec) clat (usec): min=351, max=44583, avg=1609.07, stdev=1711.98 lat (usec): min=356, max=44589, avg=1615.10, stdev=1711.95 clat percentiles (usec): | 1.00th=[ 445], 5.00th=[ 676], 10.00th=[ 922], 20.00th=[ 1172], | 30.00th=[ 1287], 40.00th=[ 1352], 50.00th=[ 1418], 60.00th=[ 1500], | 70.00th=[ 1598], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36963], 99.95th=[42206], | 99.99th=[43779] bw ( KiB/s): min=24832, max=54528, per=100.00%, avg=33500.70, stdev=3961.59, samples=60 iops : min= 388, max= 852, avg=523.42, stdev=61.93, samples=60 lat (usec) : 50=21.13%, 100=58.00%, 250=0.66%, 500=0.47%, 750=0.89% lat (usec) : 1000=1.14% lat (msec) : 2=15.73%, 4=1.56%, 10=0.25%, 20=0.15%, 50=0.03% cpu : usr=1.67%, sys=3.63%, ctx=78176, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62459,15702,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=130MiB/s (136MB/s), 130MiB/s-130MiB/s (136MB/s-136MB/s), io=3904MiB (4093MB), run=30012-30012msec WRITE: bw=32.7MiB/s (34.3MB/s), 32.7MiB/s-32.7MiB/s (34.3MB/s-34.3MB/s), io=981MiB (1029MB), run=30012-30012msec Disk stats (read/write): vde: ios=62031/15579, merge=0/0, ticks=825/17457, in_queue=28143, util=94.22% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922079: Fri Jan 7 13:56:10 2022 read: IOPS=2133, BW=133MiB/s (140MB/s)(4026MiB/30190msec) slat (usec): min=2, max=217, avg=10.64, stdev= 5.61 clat (nsec): min=764, max=2522.0M, avg=1189350.08, stdev=45027809.94 lat (usec): min=27, max=2522.0k, avg=1200.14, stdev=45027.79 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 33], 10.00th=[ 35], | 20.00th=[ 37], 30.00th=[ 39], 40.00th=[ 41], | 50.00th=[ 45], 60.00th=[ 49], 70.00th=[ 54], | 80.00th=[ 60], 90.00th=[ 74], 95.00th=[ 123], | 99.00th=[ 1500], 99.50th=[ 2638], 99.90th=[ 3949], | 99.95th=[ 734004], 99.99th=[2499806] bw ( KiB/s): min=109824, max=280064, per=100.00%, avg=137419.72, stdev=22085.33, samples=60 iops : min= 1716, max= 4376, avg=2147.17, stdev=345.08, samples=60 write: IOPS=535, BW=33.5MiB/s (35.1MB/s)(1011MiB/30190msec) slat (usec): min=8, max=232, avg=20.98, stdev= 9.38 clat (msec): min=19, max=3350, avg=181.77, stdev=156.41 lat (msec): min=19, max=3350, avg=181.79, stdev=156.41 clat percentiles (msec): | 1.00th=[ 26], 5.00th=[ 124], 10.00th=[ 136], 20.00th=[ 148], | 30.00th=[ 159], 40.00th=[ 165], 50.00th=[ 169], 60.00th=[ 174], | 70.00th=[ 180], 80.00th=[ 190], 90.00th=[ 215], 95.00th=[ 236], | 99.00th=[ 435], 99.50th=[ 927], 99.90th=[ 3339], 99.95th=[ 3339], | 99.99th=[ 3339] bw ( KiB/s): min=28672, max=59776, per=100.00%, avg=34308.37, stdev=4028.92, samples=60 iops : min= 448, max= 934, avg=536.07, stdev=62.95, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.08%, 4=0.01%, 10=0.02%, 20=0.06%, 50=49.80% lat (usec) : 100=25.29%, 250=1.69%, 500=0.88%, 750=0.51%, 1000=0.40% lat (msec) : 2=0.56%, 4=0.53%, 10=0.01%, 20=0.01%, 50=0.42% lat (msec) : 100=0.24%, 250=18.64%, 500=0.63%, 750=0.05%, 1000=0.03% lat (msec) : 2000=0.05%, >=2000=0.07% cpu : usr=1.88%, sys=5.14%, ctx=75463, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64417,16175,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=4026MiB (4222MB), run=30190-30190msec WRITE: bw=33.5MiB/s (35.1MB/s), 33.5MiB/s-33.5MiB/s (35.1MB/s-35.1MB/s), io=1011MiB (1060MB), run=30190-30190msec Disk stats (read/write): vde: ios=64411/16102, merge=0/0, ticks=66355/2910611, in_queue=3000079, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923042: Fri Jan 7 13:56:46 2022 read: IOPS=2091, BW=131MiB/s (137MB/s)(3921MiB/30001msec) clat (usec): min=31, max=26604, avg=67.80, stdev=432.98 lat (usec): min=31, max=26604, avg=67.93, stdev=432.98 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 69], 95.00th=[ 75], | 99.00th=[ 91], 99.50th=[ 101], 99.90th=[ 453], 99.95th=[13042], | 99.99th=[17695] bw ( KiB/s): min=102861, max=191872, per=100.00%, avg=134319.24, stdev=14474.02, samples=59 iops : min= 1607, max= 2998, avg=2098.73, stdev=226.16, samples=59 write: IOPS=526, BW=32.9MiB/s (34.5MB/s)(986MiB/30001msec) clat (usec): min=369, max=73968, avg=1617.52, stdev=1741.40 lat (usec): min=375, max=73975, avg=1623.46, stdev=1741.37 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 693], 10.00th=[ 955], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1369], 50.00th=[ 1434], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1745], 90.00th=[ 1975], 95.00th=[ 2278], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36439], 99.95th=[40109], | 99.99th=[41681] bw ( KiB/s): min=27776, max=49792, per=100.00%, avg=33716.08, stdev=3113.02, samples=59 iops : min= 434, max= 778, avg=526.80, stdev=48.66, samples=59 lat (usec) : 50=31.29%, 100=48.19%, 250=0.33%, 500=0.44%, 750=0.76% lat (usec) : 1000=1.02% lat (msec) : 2=15.99%, 4=1.58%, 10=0.23%, 20=0.14%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.58%, sys=3.43%, ctx=78529, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62738,15781,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3921MiB (4112MB), run=30001-30001msec WRITE: bw=32.9MiB/s (34.5MB/s), 32.9MiB/s-32.9MiB/s (34.5MB/s-34.5MB/s), io=986MiB (1034MB), run=30001-30001msec Disk stats (read/write): vde: ios=62327/15657, merge=0/0, ticks=751/17680, in_queue=28212, util=94.42% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923717: Fri Jan 7 13:57:24 2022 read: IOPS=2153, BW=135MiB/s (141MB/s)(4070MiB/30240msec) slat (usec): min=2, max=255823, avg=361.45, stdev=2202.44 clat (usec): min=1222, max=2553.3k, avg=30030.17, stdev=50161.63 lat (usec): min=1226, max=2553.3k, avg=30391.77, stdev=50228.45 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 17], 20.00th=[ 21], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 30], | 70.00th=[ 32], 80.00th=[ 35], 90.00th=[ 40], 95.00th=[ 46], | 99.00th=[ 89], 99.50th=[ 220], 99.90th=[ 279], 99.95th=[ 844], | 99.99th=[ 2534] bw ( KiB/s): min=108544, max=282880, per=100.00%, avg=138824.25, stdev=22790.45, samples=60 iops : min= 1696, max= 4420, avg=2169.10, stdev=356.10, samples=60 write: IOPS=540, BW=33.8MiB/s (35.4MB/s)(1021MiB/30240msec) slat (usec): min=8, max=212922, avg=385.40, stdev=2749.66 clat (msec): min=4, max=3376, avg=248.61, stdev=156.03 lat (msec): min=5, max=3376, avg=248.99, stdev=156.00 clat percentiles (msec): | 1.00th=[ 40], 5.00th=[ 155], 10.00th=[ 188], 20.00th=[ 207], | 30.00th=[ 218], 40.00th=[ 226], 50.00th=[ 234], 60.00th=[ 241], | 70.00th=[ 249], 80.00th=[ 264], 90.00th=[ 292], 95.00th=[ 321], | 99.00th=[ 894], 99.50th=[ 1653], 99.90th=[ 1921], 99.95th=[ 2198], | 99.99th=[ 3373] bw ( KiB/s): min=29184, max=57856, per=99.96%, avg=34558.32, stdev=3885.11, samples=60 iops : min= 456, max= 904, avg=539.97, stdev=60.70, samples=60 lat (msec) : 2=0.64%, 4=2.16%, 10=1.97%, 20=9.54%, 50=63.20% lat (msec) : 100=2.40%, 250=14.15%, 500=5.58%, 750=0.10%, 1000=0.05% lat (msec) : 2000=0.19%, >=2000=0.04% cpu : usr=1.63%, sys=2.02%, ctx=9069, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=65126,16336,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4070MiB (4268MB), run=30240-30240msec WRITE: bw=33.8MiB/s (35.4MB/s), 33.8MiB/s-33.8MiB/s (35.4MB/s-35.4MB/s), io=1021MiB (1071MB), run=30240-30240msec Disk stats (read/write): vde: ios=65122/16227, merge=0/0, ticks=79008/3553294, in_queue=3674660, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924583: Fri Jan 7 13:58:00 2022 read: IOPS=2002, BW=156MiB/s (164MB/s)(4694MiB/30004msec) clat (usec): min=35, max=68818, avg=79.67, stdev=552.14 lat (usec): min=35, max=68818, avg=79.82, stdev=552.14 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 52], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 87], | 99.00th=[ 119], 99.50th=[ 147], 99.90th=[ 9503], 99.95th=[14746], | 99.99th=[19268] bw ( KiB/s): min=116000, max=241120, per=100.00%, avg=160216.00, stdev=18810.28, samples=60 iops : min= 1450, max= 3014, avg=2002.70, stdev=235.13, samples=60 write: IOPS=503, BW=39.3MiB/s (41.2MB/s)(1179MiB/30004msec) clat (usec): min=404, max=109420, avg=1653.89, stdev=1855.88 lat (usec): min=412, max=109428, avg=1661.76, stdev=1855.82 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 693], 10.00th=[ 930], 20.00th=[ 1205], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1467], 60.00th=[ 1549], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2376], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[34866], 99.95th=[39584], | 99.99th=[43779] bw ( KiB/s): min=30080, max=62720, per=100.00%, avg=40253.33, stdev=4513.99, samples=60 iops : min= 376, max= 784, avg=503.17, stdev=56.38, samples=60 lat (usec) : 50=9.25%, 100=68.92%, 250=1.54%, 500=0.35%, 750=1.02% lat (usec) : 1000=1.14% lat (msec) : 2=15.46%, 4=1.91%, 10=0.23%, 20=0.16%, 50=0.02% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.44%, sys=4.08%, ctx=75184, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60081,15096,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4694MiB (4922MB), run=30004-30004msec WRITE: bw=39.3MiB/s (41.2MB/s), 39.3MiB/s-39.3MiB/s (41.2MB/s-41.2MB/s), io=1179MiB (1237MB), run=30004-30004msec Disk stats (read/write): vde: ios=59856/15040, merge=0/0, ticks=873/17100, in_queue=28035, util=93.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925583: Fri Jan 7 13:58:37 2022 read: IOPS=2024, BW=158MiB/s (166MB/s)(4745MiB/30001msec) slat (usec): min=6, max=136, avg=14.43, stdev= 4.50 clat (nsec): min=848, max=25219k, avg=68966.56, stdev=449835.61 lat (usec): min=39, max=25231, avg=83.55, stdev=449.89 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 70], 95.00th=[ 76], | 99.00th=[ 95], 99.50th=[ 109], 99.90th=[ 8848], 99.95th=[13304], | 99.99th=[17433] bw ( KiB/s): min=126880, max=248480, per=100.00%, avg=161961.38, stdev=18930.14, samples=60 iops : min= 1586, max= 3106, avg=2024.50, stdev=236.64, samples=60 write: IOPS=508, BW=39.8MiB/s (41.7MB/s)(1193MiB/30001msec) slat (usec): min=13, max=149, avg=22.78, stdev= 6.33 clat (usec): min=387, max=44537, avg=1601.97, stdev=1626.09 lat (usec): min=408, max=44562, avg=1624.92, stdev=1626.05 clat percentiles (usec): | 1.00th=[ 478], 5.00th=[ 668], 10.00th=[ 857], 20.00th=[ 1139], | 30.00th=[ 1287], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1516], | 70.00th=[ 1614], 80.00th=[ 1762], 90.00th=[ 2024], 95.00th=[ 2343], | 99.00th=[ 9503], 99.50th=[ 9896], 99.90th=[17433], 99.95th=[40633], | 99.99th=[43254] bw ( KiB/s): min=33280, max=64960, per=100.00%, avg=40714.62, stdev=4361.39, samples=60 iops : min= 416, max= 812, avg=508.92, stdev=54.53, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=31.95% lat (usec) : 100=47.34%, 250=0.47%, 500=0.34%, 750=1.21%, 1000=1.31% lat (msec) : 2=15.14%, 4=1.81%, 10=0.25%, 20=0.13%, 50=0.02% cpu : usr=1.51%, sys=6.39%, ctx=75999, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60732,15268,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4745MiB (4975MB), run=30001-30001msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1193MiB (1251MB), run=30001-30001msec Disk stats (read/write): vde: ios=60360/15163, merge=0/0, ticks=816/16712, in_queue=27808, util=93.04% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926126: Fri Jan 7 13:59:14 2022 read: IOPS=2016, BW=158MiB/s (165MB/s)(4725MiB/30001msec) clat (usec): min=34, max=25364, avg=74.49, stdev=448.71 lat (usec): min=34, max=25364, avg=74.63, stdev=448.71 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 101], 99.50th=[ 117], 99.90th=[ 8586], 99.95th=[13042], | 99.99th=[17957] bw ( KiB/s): min=126240, max=258560, per=100.00%, avg=161294.13, stdev=19418.65, samples=60 iops : min= 1578, max= 3232, avg=2016.17, stdev=242.72, samples=60 write: IOPS=506, BW=39.6MiB/s (41.5MB/s)(1187MiB/30001msec) clat (usec): min=403, max=44830, avg=1661.22, stdev=1677.01 lat (usec): min=410, max=44838, avg=1669.00, stdev=1676.98 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 701], 10.00th=[ 955], 20.00th=[ 1237], | 30.00th=[ 1336], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1795], 90.00th=[ 2057], 95.00th=[ 2376], | 99.00th=[ 9765], 99.50th=[10028], 99.90th=[17695], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=32480, max=66880, per=100.00%, avg=40526.85, stdev=4517.16, samples=60 iops : min= 406, max= 836, avg=506.58, stdev=56.46, samples=60 lat (usec) : 50=12.96%, 100=66.08%, 250=0.76%, 500=0.29%, 750=0.93% lat (usec) : 1000=1.08% lat (msec) : 2=15.50%, 4=1.96%, 10=0.24%, 20=0.16%, 50=0.02% cpu : usr=1.54%, sys=3.78%, ctx=75689, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60483,15198,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=158MiB/s (165MB/s), 158MiB/s-158MiB/s (165MB/s-165MB/s), io=4725MiB (4955MB), run=30001-30001msec WRITE: bw=39.6MiB/s (41.5MB/s), 39.6MiB/s-39.6MiB/s (41.5MB/s-41.5MB/s), io=1187MiB (1245MB), run=30001-30001msec Disk stats (read/write): vde: ios=60060/15087, merge=0/0, ticks=807/17259, in_queue=28089, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926962: Fri Jan 7 13:59:51 2022 read: IOPS=2057, BW=161MiB/s (169MB/s)(4827MiB/30033msec) slat (usec): min=4, max=280, avg=13.57, stdev= 4.78 clat (nsec): min=789, max=2494.7M, avg=1280647.19, stdev=49851439.10 lat (usec): min=30, max=2494.7k, avg=1294.38, stdev=49851.45 clat percentiles (usec): | 1.00th=[ 33], 5.00th=[ 37], 10.00th=[ 39], | 20.00th=[ 42], 30.00th=[ 44], 40.00th=[ 46], | 50.00th=[ 49], 60.00th=[ 52], 70.00th=[ 56], | 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 77], | 99.00th=[ 111], 99.50th=[ 149], 99.90th=[ 20055], | 99.95th=[1044382], 99.99th=[2499806] bw ( KiB/s): min=133440, max=298400, per=100.00%, avg=164764.73, stdev=22798.83, samples=60 iops : min= 1668, max= 3730, avg=2059.55, stdev=284.99, samples=60 write: IOPS=516, BW=40.3MiB/s (42.3MB/s)(1211MiB/30033msec) slat (usec): min=10, max=167, avg=22.67, stdev= 6.14 clat (usec): min=569, max=85455, avg=14179.33, stdev=6624.82 lat (usec): min=586, max=85485, avg=14202.18, stdev=6625.04 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 4686], 10.00th=[ 8160], 20.00th=[10945], | 30.00th=[11863], 40.00th=[12780], 50.00th=[13698], 60.00th=[14746], | 70.00th=[15926], 80.00th=[16909], 90.00th=[19268], 95.00th=[22938], | 99.00th=[34341], 99.50th=[62653], 99.90th=[68682], 99.95th=[77071], | 99.99th=[85459] bw ( KiB/s): min=33280, max=75360, per=100.00%, avg=41329.22, stdev=5257.36, samples=60 iops : min= 416, max= 942, avg=516.60, stdev=65.73, samples=60 lat (nsec) : 1000=0.02% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.02%, 50=44.34% lat (usec) : 100=34.41%, 250=0.93%, 500=0.07%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.16%, 4=0.67%, 10=2.25%, 20=15.14%, 50=1.68% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=1.64%, sys=6.05%, ctx=76445, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61789,15504,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4827MiB (5062MB), run=30033-30033msec WRITE: bw=40.3MiB/s (42.3MB/s), 40.3MiB/s-40.3MiB/s (42.3MB/s-42.3MB/s), io=1211MiB (1270MB), run=30033-30033msec Disk stats (read/write): vde: ios=61419/15429, merge=0/0, ticks=71540/210498, in_queue=296329, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927594: Fri Jan 7 14:00:28 2022 read: IOPS=2021, BW=158MiB/s (166MB/s)(4739MiB/30004msec) clat (usec): min=34, max=60214, avg=72.84, stdev=504.42 lat (usec): min=34, max=60214, avg=72.98, stdev=504.42 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 73], 95.00th=[ 79], | 99.00th=[ 99], 99.50th=[ 113], 99.90th=[ 9241], 99.95th=[13435], | 99.99th=[17957] bw ( KiB/s): min=122720, max=255040, per=100.00%, avg=161791.30, stdev=20152.36, samples=60 iops : min= 1534, max= 3188, avg=2022.33, stdev=251.93, samples=60 write: IOPS=508, BW=39.7MiB/s (41.6MB/s)(1191MiB/30004msec) clat (usec): min=402, max=46507, avg=1661.98, stdev=1637.32 lat (usec): min=410, max=46513, avg=1669.45, stdev=1637.26 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 717], 10.00th=[ 996], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1663], 80.00th=[ 1811], 90.00th=[ 2057], 95.00th=[ 2376], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[17695], 99.95th=[39584], | 99.99th=[43254] bw ( KiB/s): min=31680, max=66240, per=100.00%, avg=40673.35, stdev=4467.10, samples=60 iops : min= 396, max= 828, avg=508.38, stdev=55.85, samples=60 lat (usec) : 50=21.53%, 100=57.66%, 250=0.60%, 500=0.28%, 750=0.89% lat (usec) : 1000=0.92% lat (msec) : 2=15.69%, 4=2.03%, 10=0.22%, 20=0.16%, 50=0.02% lat (msec) : 100=0.01% cpu : usr=1.47%, sys=3.68%, ctx=75917, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60662,15251,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4739MiB (4969MB), run=30004-30004msec WRITE: bw=39.7MiB/s (41.6MB/s), 39.7MiB/s-39.7MiB/s (41.6MB/s-41.6MB/s), io=1191MiB (1249MB), run=30004-30004msec Disk stats (read/write): vde: ios=60478/15194, merge=0/0, ticks=861/17448, in_queue=28265, util=94.75% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928864: Fri Jan 7 14:01:05 2022 read: IOPS=2056, BW=161MiB/s (168MB/s)(4838MiB/30117msec) slat (usec): min=2, max=317, avg=11.71, stdev= 7.17 clat (nsec): min=734, max=2539.9M, avg=1503617.60, stdev=55375705.18 lat (usec): min=29, max=2539.9k, avg=1515.50, stdev=55375.71 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 34], 10.00th=[ 36], | 20.00th=[ 38], 30.00th=[ 40], 40.00th=[ 42], | 50.00th=[ 45], 60.00th=[ 49], 70.00th=[ 55], | 80.00th=[ 60], 90.00th=[ 70], 95.00th=[ 82], | 99.00th=[ 437], 99.50th=[ 922], 99.90th=[ 87557], | 99.95th=[1619002], 99.99th=[2499806] bw ( KiB/s): min=130240, max=328800, per=100.00%, avg=165148.73, stdev=26220.34, samples=60 iops : min= 1628, max= 4110, avg=2064.35, stdev=327.77, samples=60 write: IOPS=516, BW=40.3MiB/s (42.3MB/s)(1214MiB/30117msec) slat (usec): min=9, max=279, avg=25.41, stdev=13.75 clat (msec): min=4, max=2618, avg=90.80, stdev=79.40 lat (msec): min=4, max=2618, avg=90.82, stdev=79.40 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 61], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 78], 40.00th=[ 81], 50.00th=[ 83], 60.00th=[ 86], | 70.00th=[ 89], 80.00th=[ 94], 90.00th=[ 104], 95.00th=[ 132], | 99.00th=[ 232], 99.50th=[ 523], 99.90th=[ 1385], 99.95th=[ 1502], | 99.99th=[ 2299] bw ( KiB/s): min=32480, max=76160, per=100.00%, avg=41330.73, stdev=5321.85, samples=60 iops : min= 406, max= 952, avg=516.58, stdev=66.55, samples=60 lat (nsec) : 750=0.01%, 1000=0.09% lat (usec) : 2=0.13%, 4=0.01%, 10=0.05%, 20=0.12%, 50=50.17% lat (usec) : 100=26.98%, 250=1.19%, 500=0.50%, 750=0.22%, 1000=0.10% lat (msec) : 2=0.27%, 4=0.02%, 10=0.04%, 20=0.30%, 50=0.17% lat (msec) : 100=17.21%, 250=2.18%, 500=0.08%, 750=0.04%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.04% cpu : usr=1.85%, sys=5.41%, ctx=74883, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=61922,15543,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=161MiB/s (168MB/s), 161MiB/s-161MiB/s (168MB/s-168MB/s), io=4838MiB (5073MB), run=30117-30117msec WRITE: bw=40.3MiB/s (42.3MB/s), 40.3MiB/s-40.3MiB/s (42.3MB/s-42.3MB/s), io=1214MiB (1273MB), run=30117-30117msec Disk stats (read/write): vde: ios=61920/15517, merge=0/0, ticks=89047/1398311, in_queue=1500549, util=100.00% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929909: Fri Jan 7 14:01:42 2022 read: IOPS=2026, BW=158MiB/s (166MB/s)(4750MiB/30009msec) clat (usec): min=35, max=23676, avg=75.41, stdev=466.32 lat (usec): min=35, max=23676, avg=75.55, stdev=466.32 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 69], 90.00th=[ 76], 95.00th=[ 82], | 99.00th=[ 105], 99.50th=[ 120], 99.90th=[ 8455], 99.95th=[14222], | 99.99th=[18482] bw ( KiB/s): min=122400, max=247125, per=100.00%, avg=163912.32, stdev=20709.82, samples=60 iops : min= 1530, max= 3089, avg=2048.58, stdev=258.95, samples=60 write: IOPS=509, BW=39.8MiB/s (41.7MB/s)(1194MiB/30009msec) clat (usec): min=408, max=46874, avg=1646.34, stdev=1672.40 lat (usec): min=423, max=46882, avg=1654.21, stdev=1672.32 clat percentiles (usec): | 1.00th=[ 494], 5.00th=[ 701], 10.00th=[ 947], 20.00th=[ 1221], | 30.00th=[ 1336], 40.00th=[ 1401], 50.00th=[ 1483], 60.00th=[ 1565], | 70.00th=[ 1647], 80.00th=[ 1795], 90.00th=[ 2040], 95.00th=[ 2343], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[36963], 99.95th=[41157], | 99.99th=[45351] bw ( KiB/s): min=33727, max=64615, per=100.00%, avg=41202.52, stdev=4529.70, samples=60 iops : min= 421, max= 807, avg=514.62, stdev=56.51, samples=60 lat (usec) : 50=12.19%, 100=66.70%, 250=0.88%, 500=0.27%, 750=0.99% lat (usec) : 1000=1.07% lat (msec) : 2=15.56%, 4=1.94%, 10=0.22%, 20=0.15%, 50=0.03% cpu : usr=1.63%, sys=3.85%, ctx=76097, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60802,15284,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=158MiB/s (166MB/s), 158MiB/s-158MiB/s (166MB/s-166MB/s), io=4750MiB (4981MB), run=30009-30009msec WRITE: bw=39.8MiB/s (41.7MB/s), 39.8MiB/s-39.8MiB/s (41.7MB/s-41.7MB/s), io=1194MiB (1252MB), run=30009-30009msec Disk stats (read/write): vde: ios=60798/15278, merge=0/0, ticks=847/17330, in_queue=28157, util=93.57% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930581: Fri Jan 7 14:02:20 2022 read: IOPS=2066, BW=161MiB/s (169MB/s)(4880MiB/30223msec) slat (usec): min=2, max=226, avg=11.66, stdev= 5.31 clat (nsec): min=794, max=2527.2M, avg=1349127.46, stdev=50378764.00 lat (usec): min=30, max=2527.2k, avg=1360.94, stdev=50378.74 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 35], 10.00th=[ 37], | 20.00th=[ 39], 30.00th=[ 41], 40.00th=[ 43], | 50.00th=[ 46], 60.00th=[ 51], 70.00th=[ 58], | 80.00th=[ 65], 90.00th=[ 79], 95.00th=[ 143], | 99.00th=[ 1762], 99.50th=[ 3064], 99.90th=[ 18482], | 99.95th=[ 809501], 99.99th=[2499806] bw ( KiB/s): min=121920, max=350621, per=100.00%, avg=166577.07, stdev=27999.00, samples=60 iops : min= 1524, max= 4382, avg=2082.18, stdev=349.91, samples=60 write: IOPS=519, BW=40.6MiB/s (42.6MB/s)(1227MiB/30223msec) slat (usec): min=10, max=157, avg=25.02, stdev= 9.86 clat (msec): min=5, max=3698, avg=186.89, stdev=171.36 lat (msec): min=5, max=3698, avg=186.92, stdev=171.36 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 126], 10.00th=[ 133], 20.00th=[ 146], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 180], | 70.00th=[ 186], 80.00th=[ 194], 90.00th=[ 218], 95.00th=[ 241], | 99.00th=[ 477], 99.50th=[ 1334], 99.90th=[ 3037], 99.95th=[ 3037], | 99.99th=[ 3574] bw ( KiB/s): min=35520, max=74549, per=100.00%, avg=41629.12, stdev=5150.36, samples=60 iops : min= 444, max= 931, avg=520.33, stdev=64.30, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.03%, 50=46.50% lat (usec) : 100=28.33%, 250=1.86%, 500=0.93%, 750=0.55%, 1000=0.39% lat (msec) : 2=0.53%, 4=0.62%, 10=0.05%, 20=0.05%, 50=0.40% lat (msec) : 100=0.20%, 250=18.64%, 500=0.63%, 750=0.06%, 1000=0.03% lat (msec) : 2000=0.06%, >=2000=0.10% cpu : usr=1.84%, sys=5.38%, ctx=73183, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62460,15706,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4880MiB (5117MB), run=30223-30223msec WRITE: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=1227MiB (1287MB), run=30223-30223msec Disk stats (read/write): vde: ios=62458/15619, merge=0/0, ticks=77974/2898138, in_queue=2998518, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931410: Fri Jan 7 14:02:56 2022 read: IOPS=1999, BW=156MiB/s (164MB/s)(4687MiB/30002msec) clat (usec): min=32, max=30857, avg=72.74, stdev=497.41 lat (usec): min=32, max=30857, avg=72.88, stdev=497.41 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 72], 95.00th=[ 79], | 99.00th=[ 98], 99.50th=[ 113], 99.90th=[ 9634], 99.95th=[14615], | 99.99th=[20579] bw ( KiB/s): min=127040, max=228800, per=100.00%, avg=159992.00, stdev=18087.70, samples=60 iops : min= 1588, max= 2860, avg=1999.90, stdev=226.10, samples=60 write: IOPS=502, BW=39.2MiB/s (41.2MB/s)(1177MiB/30002msec) clat (usec): min=407, max=43821, avg=1684.35, stdev=1689.65 lat (usec): min=414, max=43829, avg=1691.91, stdev=1689.59 clat percentiles (usec): | 1.00th=[ 486], 5.00th=[ 717], 10.00th=[ 1012], 20.00th=[ 1254], | 30.00th=[ 1352], 40.00th=[ 1418], 50.00th=[ 1500], 60.00th=[ 1582], | 70.00th=[ 1680], 80.00th=[ 1827], 90.00th=[ 2073], 95.00th=[ 2376], | 99.00th=[ 9634], 99.50th=[10028], 99.90th=[35914], 99.95th=[41157], | 99.99th=[43254] bw ( KiB/s): min=33280, max=59040, per=100.00%, avg=40186.67, stdev=3551.93, samples=60 iops : min= 416, max= 738, avg=502.33, stdev=44.40, samples=60 lat (usec) : 50=24.96%, 100=54.25%, 250=0.59%, 500=0.30%, 750=0.84% lat (usec) : 1000=0.85% lat (msec) : 2=15.66%, 4=2.11%, 10=0.24%, 20=0.16%, 50=0.03% cpu : usr=1.47%, sys=3.55%, ctx=75089, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59997,15071,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4687MiB (4915MB), run=30002-30002msec WRITE: bw=39.2MiB/s (41.2MB/s), 39.2MiB/s-39.2MiB/s (41.2MB/s-41.2MB/s), io=1177MiB (1235MB), run=30002-30002msec Disk stats (read/write): vde: ios=59447/14954, merge=0/0, ticks=879/17545, in_queue=28294, util=94.71% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932353: Fri Jan 7 14:03:34 2022 read: IOPS=2066, BW=161MiB/s (169MB/s)(4882MiB/30241msec) slat (usec): min=2, max=254296, avg=383.45, stdev=2617.41 clat (usec): min=1091, max=2567.2k, avg=31381.40, stdev=53391.44 lat (usec): min=1100, max=2567.2k, avg=31764.99, stdev=53463.62 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 7], 10.00th=[ 17], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 27], 50.00th=[ 29], 60.00th=[ 31], | 70.00th=[ 34], 80.00th=[ 37], 90.00th=[ 42], 95.00th=[ 47], | 99.00th=[ 90], 99.50th=[ 243], 99.90th=[ 284], 99.95th=[ 894], | 99.99th=[ 2534] bw ( KiB/s): min=131520, max=359520, per=100.00%, avg=166482.57, stdev=28899.99, samples=60 iops : min= 1644, max= 4494, avg=2081.02, stdev=361.26, samples=60 write: IOPS=519, BW=40.6MiB/s (42.6MB/s)(1228MiB/30241msec) slat (usec): min=7, max=72753, avg=374.75, stdev=1618.55 clat (msec): min=21, max=3740, avg=258.16, stdev=178.51 lat (msec): min=21, max=3740, avg=258.54, stdev=178.50 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 180], 10.00th=[ 194], 20.00th=[ 211], | 30.00th=[ 224], 40.00th=[ 234], 50.00th=[ 241], 60.00th=[ 249], | 70.00th=[ 255], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 347], | 99.00th=[ 885], 99.50th=[ 1385], 99.90th=[ 3104], 99.95th=[ 3104], | 99.99th=[ 3608] bw ( KiB/s): min=33920, max=73440, per=99.94%, avg=41538.50, stdev=5141.72, samples=60 iops : min= 424, max= 918, avg=519.22, stdev=64.29, samples=60 lat (msec) : 2=0.71%, 4=1.70%, 10=2.45%, 20=8.39%, 50=64.06% lat (msec) : 100=2.50%, 250=12.55%, 500=7.21%, 750=0.16%, 1000=0.08% lat (msec) : 2000=0.11%, >=2000=0.09% cpu : usr=1.78%, sys=1.98%, ctx=8781, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=62491,15712,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=161MiB/s (169MB/s), 161MiB/s-161MiB/s (169MB/s-169MB/s), io=4882MiB (5119MB), run=30241-30241msec WRITE: bw=40.6MiB/s (42.6MB/s), 40.6MiB/s-40.6MiB/s (42.6MB/s-42.6MB/s), io=1228MiB (1287MB), run=30241-30241msec Disk stats (read/write): vde: ios=62489/15621, merge=0/0, ticks=87178/3549482, in_queue=3674169, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932855: Fri Jan 7 14:04:10 2022 read: IOPS=1787, BW=223MiB/s (234MB/s)(6703MiB/30002msec) clat (usec): min=45, max=23854, avg=94.38, stdev=508.57 lat (usec): min=45, max=23854, avg=94.53, stdev=508.57 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 62], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 99], 95.00th=[ 109], | 99.00th=[ 137], 99.50th=[ 157], 99.90th=[10814], 99.95th=[15926], | 99.99th=[19792] bw ( KiB/s): min=98560, max=378624, per=100.00%, avg=228795.73, stdev=35702.56, samples=60 iops : min= 770, max= 2958, avg=1787.47, stdev=278.93, samples=60 write: IOPS=448, BW=56.1MiB/s (58.8MB/s)(1684MiB/30002msec) clat (usec): min=534, max=201535, avg=1830.08, stdev=2481.24 lat (usec): min=546, max=201548, avg=1842.67, stdev=2481.22 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 914], 10.00th=[ 1090], 20.00th=[ 1237], | 30.00th=[ 1385], 40.00th=[ 1516], 50.00th=[ 1614], 60.00th=[ 1713], | 70.00th=[ 1827], 80.00th=[ 1958], 90.00th=[ 2245], 95.00th=[ 2606], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[35914], 99.95th=[40109], | 99.99th=[72877] bw ( KiB/s): min=28160, max=99072, per=100.00%, avg=57467.73, stdev=8100.85, samples=60 iops : min= 220, max= 774, avg=448.97, stdev=63.29, samples=60 lat (usec) : 50=0.05%, 100=72.73%, 250=6.97%, 500=0.06%, 750=0.51% lat (usec) : 1000=0.91% lat (msec) : 2=15.07%, 4=3.09%, 10=0.40%, 20=0.19%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.71%, sys=3.66%, ctx=67103, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53624,13470,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=6703MiB (7029MB), run=30002-30002msec WRITE: bw=56.1MiB/s (58.8MB/s), 56.1MiB/s-56.1MiB/s (58.8MB/s-58.8MB/s), io=1684MiB (1766MB), run=30002-30002msec Disk stats (read/write): vde: ios=53314/13384, merge=0/0, ticks=855/17362, in_queue=28122, util=94.02% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933840: Fri Jan 7 14:04:47 2022 read: IOPS=1816, BW=227MiB/s (238MB/s)(6812MiB/30001msec) slat (usec): min=9, max=136, avg=15.50, stdev= 4.61 clat (nsec): min=1070, max=25587k, avg=78888.48, stdev=543411.04 lat (usec): min=47, max=25599, avg=94.54, stdev=543.48 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 52], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 76], 95.00th=[ 83], | 99.00th=[ 100], 99.50th=[ 112], 99.90th=[11863], 99.95th=[16450], | 99.99th=[19792] bw ( KiB/s): min=183040, max=366592, per=100.00%, avg=232756.07, stdev=29083.38, samples=59 iops : min= 1430, max= 2864, avg=1818.41, stdev=227.21, samples=59 write: IOPS=456, BW=57.1MiB/s (59.8MB/s)(1712MiB/30001msec) slat (usec): min=15, max=133, avg=28.04, stdev= 6.43 clat (usec): min=524, max=45446, avg=1778.76, stdev=1658.75 lat (usec): min=552, max=45484, avg=1806.97, stdev=1658.77 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 775], 10.00th=[ 930], 20.00th=[ 1270], | 30.00th=[ 1434], 40.00th=[ 1532], 50.00th=[ 1614], 60.00th=[ 1696], | 70.00th=[ 1827], 80.00th=[ 1975], 90.00th=[ 2245], 95.00th=[ 2606], | 99.00th=[ 9634], 99.50th=[10159], 99.90th=[35390], 99.95th=[39060], | 99.99th=[44303] bw ( KiB/s): min=47616, max=94720, per=100.00%, avg=58485.15, stdev=6562.54, samples=59 iops : min= 372, max= 740, avg=456.92, stdev=51.27, samples=59 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=10.00%, 100=69.08% lat (usec) : 250=0.70%, 500=0.01%, 750=0.87%, 1000=1.53% lat (msec) : 2=13.93%, 4=3.39%, 10=0.25%, 20=0.20%, 50=0.03% cpu : usr=1.63%, sys=5.69%, ctx=68196, majf=0, minf=14 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54494,13695,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6812MiB (7143MB), run=30001-30001msec WRITE: bw=57.1MiB/s (59.8MB/s), 57.1MiB/s-57.1MiB/s (59.8MB/s-59.8MB/s), io=1712MiB (1795MB), run=30001-30001msec Disk stats (read/write): vde: ios=54219/13623, merge=0/0, ticks=947/16703, in_queue=28068, util=93.83% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934416: Fri Jan 7 14:05:24 2022 read: IOPS=1814, BW=227MiB/s (238MB/s)(6804MiB/30002msec) clat (usec): min=38, max=65577, avg=84.51, stdev=670.92 lat (usec): min=38, max=65577, avg=84.65, stdev=670.92 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 73], 90.00th=[ 80], 95.00th=[ 87], | 99.00th=[ 109], 99.50th=[ 126], 99.90th=[12125], 99.95th=[15664], | 99.99th=[22152] bw ( KiB/s): min=167424, max=378368, per=100.00%, avg=232257.95, stdev=30566.57, samples=60 iops : min= 1308, max= 2956, avg=1814.48, stdev=238.81, samples=60 write: IOPS=455, BW=56.0MiB/s (59.8MB/s)(1710MiB/30002msec) clat (usec): min=527, max=138069, avg=1836.24, stdev=1978.77 lat (usec): min=537, max=138092, avg=1848.43, stdev=1978.74 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 791], 10.00th=[ 1057], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1762], | 70.00th=[ 1876], 80.00th=[ 2024], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9765], 99.50th=[10290], 99.90th=[18744], 99.95th=[42206], | 99.99th=[45351] bw ( KiB/s): min=48640, max=98816, per=100.00%, avg=58363.23, stdev=6775.00, samples=60 iops : min= 380, max= 772, avg=455.92, stdev=52.95, samples=60 lat (usec) : 50=2.65%, 100=75.92%, 250=1.22%, 500=0.01%, 750=0.82% lat (usec) : 1000=1.01% lat (msec) : 2=14.00%, 4=3.91%, 10=0.20%, 20=0.22%, 50=0.03% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=1.66%, sys=3.41%, ctx=68126, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54432,13679,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6804MiB (7135MB), run=30002-30002msec WRITE: bw=56.0MiB/s (59.8MB/s), 56.0MiB/s-56.0MiB/s (59.8MB/s-59.8MB/s), io=1710MiB (1793MB), run=30002-30002msec Disk stats (read/write): vde: ios=54152/13608, merge=0/0, ticks=1056/17246, in_queue=28246, util=94.45% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935721: Fri Jan 7 14:06:01 2022 read: IOPS=1864, BW=233MiB/s (244MB/s)(6999MiB/30030msec) slat (usec): min=5, max=164, avg=13.71, stdev= 4.56 clat (nsec): min=890, max=2531.7M, avg=1495128.42, stdev=52750918.60 lat (usec): min=36, max=2531.7k, avg=1508.99, stdev=52750.92 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], | 20.00th=[ 42], 30.00th=[ 43], 40.00th=[ 45], | 50.00th=[ 48], 60.00th=[ 51], 70.00th=[ 56], | 80.00th=[ 62], 90.00th=[ 71], 95.00th=[ 79], | 99.00th=[ 103], 99.50th=[ 122], 99.90th=[ 112722], | 99.95th=[1300235], 99.99th=[2499806] bw ( KiB/s): min=188160, max=444928, per=100.00%, avg=238936.63, stdev=34868.89, samples=60 iops : min= 1470, max= 3476, avg=1866.67, stdev=272.41, samples=60 write: IOPS=468, BW=58.6MiB/s (61.4MB/s)(1759MiB/30030msec) slat (usec): min=12, max=253, avg=28.63, stdev= 9.32 clat (usec): min=1839, max=76364, avg=15296.96, stdev=6249.39 lat (usec): min=1863, max=76389, avg=15325.76, stdev=6249.75 clat percentiles (usec): | 1.00th=[ 3851], 5.00th=[ 8094], 10.00th=[10159], 20.00th=[11863], | 30.00th=[12780], 40.00th=[13829], 50.00th=[14615], 60.00th=[15664], | 70.00th=[16581], 80.00th=[17695], 90.00th=[20055], 95.00th=[23462], | 99.00th=[34341], 99.50th=[63701], 99.90th=[68682], 99.95th=[72877], | 99.99th=[73925] bw ( KiB/s): min=50176, max=111872, per=100.00%, avg=60016.73, stdev=7751.16, samples=60 iops : min= 392, max= 874, avg=468.80, stdev=60.53, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.03%, 50=46.30% lat (usec) : 100=32.61%, 250=0.75%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.26%, 10=1.63%, 20=16.19%, 50=1.84% lat (msec) : 100=0.16%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.02%, >=2000=0.03% cpu : usr=1.64%, sys=5.47%, ctx=69508, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=55993,14072,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=233MiB/s (244MB/s), 233MiB/s-233MiB/s (244MB/s-244MB/s), io=6999MiB (7339MB), run=30030-30030msec WRITE: bw=58.6MiB/s (61.4MB/s), 58.6MiB/s-58.6MiB/s (61.4MB/s-61.4MB/s), io=1759MiB (1844MB), run=30030-30030msec Disk stats (read/write): vde: ios=55662/13984, merge=0/0, ticks=77737/206504, in_queue=296601, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936715: Fri Jan 7 14:06:38 2022 read: IOPS=1818, BW=227MiB/s (238MB/s)(6820MiB/30001msec) clat (usec): min=39, max=28723, avg=84.63, stdev=551.58 lat (usec): min=39, max=28723, avg=84.77, stdev=551.58 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 52], 10.00th=[ 54], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 61], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 76], 90.00th=[ 83], 95.00th=[ 90], | 99.00th=[ 111], 99.50th=[ 124], 99.90th=[11600], 99.95th=[16188], | 99.99th=[20579] bw ( KiB/s): min=166144, max=382464, per=100.00%, avg=233562.86, stdev=30149.19, samples=59 iops : min= 1298, max= 2988, avg=1824.68, stdev=235.52, samples=59 write: IOPS=456, BW=57.1MiB/s (59.9MB/s)(1714MiB/30001msec) clat (usec): min=526, max=45749, avg=1828.09, stdev=1679.20 lat (usec): min=539, max=45761, avg=1843.03, stdev=1679.23 clat percentiles (usec): | 1.00th=[ 611], 5.00th=[ 799], 10.00th=[ 1074], 20.00th=[ 1369], | 30.00th=[ 1483], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1844], 80.00th=[ 2008], 90.00th=[ 2278], 95.00th=[ 2606], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[36439], 99.95th=[39584], | 99.99th=[44303] bw ( KiB/s): min=42752, max=99328, per=100.00%, avg=58645.00, stdev=7329.00, samples=59 iops : min= 334, max= 776, avg=458.10, stdev=57.25, samples=59 lat (usec) : 50=2.69%, 100=75.63%, 250=1.48%, 500=0.02%, 750=0.77% lat (usec) : 1000=1.01% lat (msec) : 2=14.21%, 4=3.74%, 10=0.21%, 20=0.22%, 50=0.03% cpu : usr=1.82%, sys=3.52%, ctx=68277, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54560,13708,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6820MiB (7151MB), run=30001-30001msec WRITE: bw=57.1MiB/s (59.9MB/s), 57.1MiB/s-57.1MiB/s (59.9MB/s-59.9MB/s), io=1714MiB (1797MB), run=30001-30001msec Disk stats (read/write): vde: ios=54253/13634, merge=0/0, ticks=946/17225, in_queue=28111, util=94.04% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937334: Fri Jan 7 14:07:16 2022 read: IOPS=1851, BW=231MiB/s (243MB/s)(6971MiB/30117msec) slat (usec): min=3, max=546, avg=12.83, stdev= 5.32 clat (nsec): min=757, max=2533.5M, avg=1573197.26, stdev=56530786.27 lat (usec): min=34, max=2533.5k, avg=1586.18, stdev=56530.76 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 44], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 50], 60.00th=[ 55], 70.00th=[ 63], | 80.00th=[ 75], 90.00th=[ 86], 95.00th=[ 101], | 99.00th=[ 457], 99.50th=[ 996], 99.90th=[ 109577], | 99.95th=[1753220], 99.99th=[2499806] bw ( KiB/s): min=186368, max=455936, per=100.00%, avg=237960.03, stdev=35595.84, samples=60 iops : min= 1456, max= 3562, avg=1859.05, stdev=278.09, samples=60 write: IOPS=465, BW=58.2MiB/s (61.0MB/s)(1753MiB/30117msec) slat (usec): min=12, max=308, avg=33.30, stdev=13.99 clat (msec): min=4, max=2362, avg=101.01, stdev=66.44 lat (msec): min=4, max=2362, avg=101.04, stdev=66.44 clat percentiles (msec): | 1.00th=[ 20], 5.00th=[ 75], 10.00th=[ 82], 20.00th=[ 86], | 30.00th=[ 90], 40.00th=[ 92], 50.00th=[ 94], 60.00th=[ 96], | 70.00th=[ 100], 80.00th=[ 105], 90.00th=[ 116], 95.00th=[ 142], | 99.00th=[ 215], 99.50th=[ 567], 99.90th=[ 995], 99.95th=[ 1183], | 99.99th=[ 1989] bw ( KiB/s): min=49664, max=103936, per=100.00%, avg=59624.28, stdev=6850.99, samples=60 iops : min= 388, max= 812, avg=465.80, stdev=53.53, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.02%, 50=39.56%, 100=36.24% lat (usec) : 250=2.87%, 500=0.45%, 750=0.22%, 1000=0.13% lat (msec) : 2=0.23%, 4=0.07%, 10=0.01%, 20=0.25%, 50=0.24% lat (msec) : 100=13.84%, 250=5.60%, 500=0.08%, 750=0.08%, 1000=0.04% lat (msec) : 2000=0.03%, >=2000=0.04% cpu : usr=2.09%, sys=5.15%, ctx=67622, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=55771,14022,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=231MiB/s (243MB/s), 231MiB/s-231MiB/s (243MB/s-243MB/s), io=6971MiB (7310MB), run=30117-30117msec WRITE: bw=58.2MiB/s (61.0MB/s), 58.2MiB/s-58.2MiB/s (61.0MB/s-61.0MB/s), io=1753MiB (1838MB), run=30117-30117msec Disk stats (read/write): vde: ios=55770/13988, merge=0/0, ticks=82978/1399387, in_queue=1498983, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938224: Fri Jan 7 14:07:52 2022 read: IOPS=1822, BW=228MiB/s (239MB/s)(6835MiB/30001msec) clat (usec): min=38, max=25802, avg=80.70, stdev=548.71 lat (usec): min=38, max=25802, avg=80.83, stdev=548.71 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 78], 95.00th=[ 84], | 99.00th=[ 100], 99.50th=[ 111], 99.90th=[11863], 99.95th=[16319], | 99.99th=[20579] bw ( KiB/s): min=170752, max=392192, per=100.00%, avg=233720.69, stdev=31621.01, samples=59 iops : min= 1334, max= 3064, avg=1825.93, stdev=247.04, samples=59 write: IOPS=457, BW=57.2MiB/s (60.0MB/s)(1717MiB/30001msec) clat (usec): min=526, max=44019, avg=1842.21, stdev=1660.64 lat (usec): min=538, max=44024, avg=1854.45, stdev=1660.55 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 807], 10.00th=[ 1123], 20.00th=[ 1401], | 30.00th=[ 1500], 40.00th=[ 1582], 50.00th=[ 1663], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2278], 95.00th=[ 2573], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[35390], 99.95th=[39060], | 99.99th=[43254] bw ( KiB/s): min=46336, max=102400, per=100.00%, avg=58666.66, stdev=7065.73, samples=59 iops : min= 362, max= 800, avg=458.31, stdev=55.22, samples=59 lat (usec) : 50=3.57%, 100=75.56%, 250=0.69%, 500=0.01%, 750=0.78% lat (usec) : 1000=0.85% lat (msec) : 2=14.32%, 4=3.75%, 10=0.22%, 20=0.22%, 50=0.03% cpu : usr=1.69%, sys=3.24%, ctx=68425, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54680,13737,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6835MiB (7167MB), run=30001-30001msec WRITE: bw=57.2MiB/s (60.0MB/s), 57.2MiB/s-57.2MiB/s (60.0MB/s-60.0MB/s), io=1717MiB (1801MB), run=30001-30001msec Disk stats (read/write): vde: ios=54293/13645, merge=0/0, ticks=946/17432, in_queue=28249, util=94.48% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938827: Fri Jan 7 14:08:30 2022 read: IOPS=1853, BW=232MiB/s (243MB/s)(7005MiB/30228msec) slat (usec): min=3, max=301, avg=11.42, stdev= 5.68 clat (nsec): min=817, max=2531.5M, avg=1561574.48, stdev=54490553.76 lat (usec): min=36, max=2531.5k, avg=1573.15, stdev=54490.53 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], | 20.00th=[ 43], 30.00th=[ 45], 40.00th=[ 47], | 50.00th=[ 51], 60.00th=[ 57], 70.00th=[ 69], | 80.00th=[ 78], 90.00th=[ 95], 95.00th=[ 229], | 99.00th=[ 2507], 99.50th=[ 3720], 99.90th=[ 130548], | 99.95th=[1434452], 99.99th=[2533360] bw ( KiB/s): min=188928, max=515328, per=100.00%, avg=239090.12, stdev=42044.88, samples=60 iops : min= 1476, max= 4026, avg=1867.88, stdev=328.48, samples=60 write: IOPS=466, BW=58.3MiB/s (61.1MB/s)(1761MiB/30228msec) slat (usec): min=10, max=277, avg=32.59, stdev=14.69 clat (msec): min=15, max=2473, avg=208.25, stdev=90.06 lat (msec): min=15, max=2473, avg=208.29, stdev=90.06 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 186], | 30.00th=[ 192], 40.00th=[ 197], 50.00th=[ 201], 60.00th=[ 205], | 70.00th=[ 213], 80.00th=[ 222], 90.00th=[ 243], 95.00th=[ 259], | 99.00th=[ 558], 99.50th=[ 869], 99.90th=[ 1334], 99.95th=[ 1334], | 99.99th=[ 2056] bw ( KiB/s): min=53248, max=107264, per=100.00%, avg=59692.65, stdev=7134.56, samples=60 iops : min= 416, max= 838, avg=466.33, stdev=55.74, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.01%, 50=39.03%, 100=33.87% lat (usec) : 250=3.12%, 500=1.10%, 750=0.67%, 1000=0.42% lat (msec) : 2=0.75%, 4=0.62%, 10=0.22%, 20=0.04%, 50=0.41% lat (msec) : 100=0.22%, 250=18.00%, 500=1.23%, 750=0.09%, 1000=0.08% lat (msec) : 2000=0.07%, >=2000=0.04% cpu : usr=1.98%, sys=4.82%, ctx=65169, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56038,14087,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=232MiB/s (243MB/s), 232MiB/s-232MiB/s (243MB/s-243MB/s), io=7005MiB (7345MB), run=30228-30228msec WRITE: bw=58.3MiB/s (61.1MB/s), 58.3MiB/s-58.3MiB/s (61.1MB/s-61.1MB/s), io=1761MiB (1846MB), run=30228-30228msec Disk stats (read/write): vde: ios=56035/14004, merge=0/0, ticks=77164/2893764, in_queue=3000303, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939646: Fri Jan 7 14:09:06 2022 read: IOPS=1819, BW=227MiB/s (238MB/s)(6823MiB/30003msec) clat (usec): min=36, max=24561, avg=80.56, stdev=519.39 lat (usec): min=36, max=24561, avg=80.70, stdev=519.40 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 72], 90.00th=[ 79], 95.00th=[ 86], | 99.00th=[ 109], 99.50th=[ 122], 99.90th=[11994], 99.95th=[15533], | 99.99th=[18482] bw ( KiB/s): min=179456, max=375808, per=100.00%, avg=232919.65, stdev=28631.18, samples=60 iops : min= 1402, max= 2936, avg=1819.67, stdev=223.68, samples=60 write: IOPS=457, BW=57.2MiB/s (59.9MB/s)(1715MiB/30003msec) clat (usec): min=523, max=44178, avg=1845.56, stdev=1692.02 lat (usec): min=536, max=44191, avg=1857.64, stdev=1691.91 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 807], 10.00th=[ 1106], 20.00th=[ 1385], | 30.00th=[ 1500], 40.00th=[ 1565], 50.00th=[ 1647], 60.00th=[ 1745], | 70.00th=[ 1860], 80.00th=[ 2008], 90.00th=[ 2278], 95.00th=[ 2606], | 99.00th=[ 9896], 99.50th=[10290], 99.90th=[35390], 99.95th=[40633], | 99.99th=[44303] bw ( KiB/s): min=45568, max=97792, per=100.00%, avg=58538.12, stdev=6674.89, samples=60 iops : min= 356, max= 764, avg=457.30, stdev=52.16, samples=60 lat (usec) : 50=3.59%, 100=75.06%, 250=1.14%, 500=0.02%, 750=0.76% lat (usec) : 1000=0.89% lat (msec) : 2=14.30%, 4=3.76%, 10=0.21%, 20=0.24%, 50=0.03% cpu : usr=1.65%, sys=3.39%, ctx=68314, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54587,13720,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6823MiB (7155MB), run=30003-30003msec WRITE: bw=57.2MiB/s (59.9MB/s), 57.2MiB/s-57.2MiB/s (59.9MB/s-59.9MB/s), io=1715MiB (1798MB), run=30003-30003msec Disk stats (read/write): vde: ios=54283/13642, merge=0/0, ticks=893/17444, in_queue=28193, util=94.25% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940614: Fri Jan 7 14:09:44 2022 read: IOPS=1866, BW=233MiB/s (245MB/s)(7066MiB/30287msec) slat (usec): min=3, max=261058, avg=421.48, stdev=2902.21 clat (usec): min=1353, max=2610.3k, avg=34948.97, stdev=60953.40 lat (usec): min=1357, max=2610.3k, avg=35370.61, stdev=61034.99 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 8], 10.00th=[ 19], 20.00th=[ 24], | 30.00th=[ 27], 40.00th=[ 29], 50.00th=[ 32], 60.00th=[ 34], | 70.00th=[ 37], 80.00th=[ 41], 90.00th=[ 46], 95.00th=[ 53], | 99.00th=[ 91], 99.50th=[ 262], 99.90th=[ 321], 99.95th=[ 1737], | 99.99th=[ 2567] bw ( KiB/s): min=190464, max=542464, per=100.00%, avg=241035.02, stdev=45283.92, samples=60 iops : min= 1488, max= 4238, avg=1882.95, stdev=353.80, samples=60 write: IOPS=469, BW=58.7MiB/s (61.5MB/s)(1777MiB/30287msec) slat (usec): min=10, max=55804, avg=423.68, stdev=1606.07 clat (msec): min=25, max=2161, avg=284.96, stdev=131.02 lat (msec): min=30, max=2161, avg=285.38, stdev=131.03 clat percentiles (msec): | 1.00th=[ 44], 5.00th=[ 203], 10.00th=[ 230], 20.00th=[ 247], | 30.00th=[ 257], 40.00th=[ 266], 50.00th=[ 275], 60.00th=[ 284], | 70.00th=[ 292], 80.00th=[ 305], 90.00th=[ 326], 95.00th=[ 363], | 99.00th=[ 743], 99.50th=[ 1183], 99.90th=[ 2089], 99.95th=[ 2089], | 99.99th=[ 2089] bw ( KiB/s): min=49664, max=107520, per=100.00%, avg=60124.25, stdev=7444.27, samples=60 iops : min= 388, max= 840, avg=469.50, stdev=58.22, samples=60 lat (msec) : 2=0.15%, 4=1.41%, 10=3.38%, 20=4.81%, 50=65.54% lat (msec) : 100=4.39%, 250=4.51%, 500=15.27%, 750=0.27%, 1000=0.08% lat (msec) : 2000=0.11%, >=2000=0.07% cpu : usr=2.38%, sys=1.97%, ctx=7853, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=56531,14215,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=233MiB/s (245MB/s), 233MiB/s-233MiB/s (245MB/s-245MB/s), io=7066MiB (7410MB), run=30287-30287msec WRITE: bw=58.7MiB/s (61.5MB/s), 58.7MiB/s-58.7MiB/s (61.5MB/s-61.5MB/s), io=1777MiB (1863MB), run=30287-30287msec Disk stats (read/write): vde: ios=56527/14208, merge=0/0, ticks=97727/3565601, in_queue=3698249, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941233: Fri Jan 7 14:10:20 2022 read: IOPS=1244, BW=311MiB/s (326MB/s)(9336MiB/30003msec) clat (usec): min=69, max=33717, avg=126.70, stdev=513.91 lat (usec): min=70, max=33717, avg=126.86, stdev=513.91 clat percentiles (usec): | 1.00th=[ 83], 5.00th=[ 87], 10.00th=[ 88], 20.00th=[ 92], | 30.00th=[ 95], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 111], | 70.00th=[ 117], 80.00th=[ 125], 90.00th=[ 137], 95.00th=[ 149], | 99.00th=[ 190], 99.50th=[ 225], 99.90th=[ 7570], 99.95th=[12780], | 99.99th=[21890] bw ( KiB/s): min=204800, max=555520, per=100.00%, avg=318693.55, stdev=62779.79, samples=60 iops : min= 800, max= 2170, avg=1244.88, stdev=245.23, samples=60 write: IOPS=313, BW=78.5MiB/s (82.3MB/s)(2355MiB/30003msec) clat (usec): min=887, max=83423, avg=2648.51, stdev=2665.19 lat (usec): min=911, max=83452, avg=2673.67, stdev=2665.00 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1139], 10.00th=[ 1336], 20.00th=[ 1598], | 30.00th=[ 1811], 40.00th=[ 1958], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2442], 80.00th=[ 2704], 90.00th=[ 4555], 95.00th=[ 5538], | 99.00th=[11076], 99.50th=[13042], 99.90th=[43254], 99.95th=[54789], | 99.99th=[83362] bw ( KiB/s): min=52736, max=147968, per=100.00%, avg=80365.22, stdev=15184.45, samples=60 iops : min= 206, max= 578, avg=313.93, stdev=59.26, samples=60 lat (usec) : 100=33.32%, 250=46.24%, 500=0.12%, 750=0.04%, 1000=0.41% lat (msec) : 2=8.29%, 4=9.02%, 10=2.00%, 20=0.52%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=1.67%, sys=2.90%, ctx=46769, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=37344,9419,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=311MiB/s (326MB/s), 311MiB/s-311MiB/s (326MB/s-326MB/s), io=9336MiB (9790MB), run=30003-30003msec WRITE: bw=78.5MiB/s (82.3MB/s), 78.5MiB/s-78.5MiB/s (82.3MB/s-82.3MB/s), io=2355MiB (2469MB), run=30003-30003msec Disk stats (read/write): vde: ios=37195/9364, merge=0/0, ticks=616/19857, in_queue=28370, util=94.92% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942445: Fri Jan 7 14:10:57 2022 read: IOPS=1312, BW=328MiB/s (344MB/s)(9843MiB/30001msec) slat (usec): min=12, max=117, avg=19.73, stdev= 5.30 clat (nsec): min=945, max=20282k, avg=90010.16, stdev=409005.82 lat (usec): min=63, max=20303, avg=109.91, stdev=409.13 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 60], 10.00th=[ 62], 20.00th=[ 64], | 30.00th=[ 66], 40.00th=[ 69], 50.00th=[ 73], 60.00th=[ 76], | 70.00th=[ 80], 80.00th=[ 86], 90.00th=[ 94], 95.00th=[ 102], | 99.00th=[ 124], 99.50th=[ 141], 99.90th=[ 8356], 99.95th=[11469], | 99.99th=[15795] bw ( KiB/s): min=246784, max=558080, per=100.00%, avg=335979.13, stdev=44996.43, samples=60 iops : min= 964, max= 2180, avg=1312.40, stdev=175.77, samples=60 write: IOPS=330, BW=82.7MiB/s (86.8MB/s)(2483MiB/30001msec) slat (usec): min=24, max=262, avg=43.48, stdev= 9.15 clat (usec): min=797, max=61763, avg=2533.37, stdev=2432.45 lat (usec): min=847, max=61802, avg=2577.05, stdev=2432.38 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1057], 10.00th=[ 1123], 20.00th=[ 1549], | 30.00th=[ 1860], 40.00th=[ 1991], 50.00th=[ 2114], 60.00th=[ 2245], | 70.00th=[ 2409], 80.00th=[ 2671], 90.00th=[ 3982], 95.00th=[ 5014], | 99.00th=[10683], 99.50th=[11863], 99.90th=[52691], 99.95th=[53740], | 99.99th=[61604] bw ( KiB/s): min=68608, max=149504, per=100.00%, avg=84732.65, stdev=10248.07, samples=60 iops : min= 268, max= 584, avg=330.97, stdev=40.04, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 50=0.10%, 100=75.18%, 250=4.43%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.62% lat (msec) : 2=7.61%, 4=9.94%, 10=1.54%, 20=0.54%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.74%, sys=4.65%, ctx=49310, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=39370,9930,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=9843MiB (10.3GB), run=30001-30001msec WRITE: bw=82.7MiB/s (86.8MB/s), 82.7MiB/s-82.7MiB/s (86.8MB/s-86.8MB/s), io=2483MiB (2603MB), run=30001-30001msec Disk stats (read/write): vde: ios=39138/9871, merge=0/0, ticks=536/20441, in_queue=28256, util=94.56% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943468: Fri Jan 7 14:11:35 2022 read: IOPS=1351, BW=338MiB/s (354MB/s)(9.90GiB/30001msec) clat (usec): min=52, max=20216, avg=93.64, stdev=407.29 lat (usec): min=52, max=20216, avg=93.78, stdev=407.30 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 89], 90.00th=[ 97], 95.00th=[ 104], | 99.00th=[ 127], 99.50th=[ 141], 99.90th=[ 8291], 99.95th=[10945], | 99.99th=[15008] bw ( KiB/s): min=237056, max=598016, per=100.00%, avg=346018.73, stdev=49959.43, samples=59 iops : min= 926, max= 2336, avg=1351.63, stdev=195.16, samples=59 write: IOPS=341, BW=85.3MiB/s (89.4MB/s)(2559MiB/30001msec) clat (usec): min=863, max=62393, avg=2527.61, stdev=2354.28 lat (usec): min=887, max=62418, avg=2551.65, stdev=2354.05 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2671], 90.00th=[ 3326], 95.00th=[ 4621], | 99.00th=[10683], 99.50th=[11863], 99.90th=[46924], 99.95th=[53216], | 99.99th=[54264] bw ( KiB/s): min=71168, max=157696, per=100.00%, avg=87433.44, stdev=11013.15, samples=59 iops : min= 278, max= 616, avg=341.53, stdev=43.02, samples=59 lat (usec) : 100=74.11%, 250=5.58%, 500=0.02%, 750=0.01%, 1000=0.54% lat (msec) : 2=6.84%, 4=11.16%, 10=1.21%, 20=0.51%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.60%, sys=3.01%, ctx=50786, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40541,10235,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=338MiB/s (354MB/s), 338MiB/s-338MiB/s (354MB/s-354MB/s), io=9.90GiB (10.6GB), run=30001-30001msec WRITE: bw=85.3MiB/s (89.4MB/s), 85.3MiB/s-85.3MiB/s (89.4MB/s-89.4MB/s), io=2559MiB (2683MB), run=30001-30001msec Disk stats (read/write): vde: ios=40263/10165, merge=0/0, ticks=572/20674, in_queue=28320, util=94.76% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943995: Fri Jan 7 14:12:11 2022 read: IOPS=1399, BW=350MiB/s (367MB/s)(10.3GiB/30023msec) slat (usec): min=7, max=581, avg=17.48, stdev= 6.54 clat (nsec): min=901, max=96922k, avg=134389.43, stdev=1572976.18 lat (usec): min=49, max=96946, avg=152.02, stdev=1572.98 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 64], 60.00th=[ 70], | 70.00th=[ 80], 80.00th=[ 97], 90.00th=[ 119], 95.00th=[ 139], | 99.00th=[ 229], 99.50th=[ 375], 99.90th=[31589], 99.95th=[43779], | 99.99th=[55313] bw ( KiB/s): min=250368, max=738816, per=100.00%, avg=358540.62, stdev=61151.61, samples=60 iops : min= 978, max= 2886, avg=1400.53, stdev=238.87, samples=60 write: IOPS=353, BW=88.4MiB/s (92.7MB/s)(2653MiB/30023msec) slat (usec): min=18, max=342, avg=50.10, stdev=20.49 clat (msec): min=2, max=106, avg=27.63, stdev=11.86 lat (msec): min=2, max=106, avg=27.68, stdev=11.86 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 12], 20.00th=[ 19], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 30], | 70.00th=[ 33], 80.00th=[ 36], 90.00th=[ 41], 95.00th=[ 45], | 99.00th=[ 65], 99.50th=[ 82], 99.90th=[ 95], 99.95th=[ 100], | 99.99th=[ 105] bw ( KiB/s): min=76288, max=185856, per=99.99%, avg=90473.62, stdev=14047.34, samples=60 iops : min= 298, max= 726, avg=353.40, stdev=54.87, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=5.08%, 100=59.92% lat (usec) : 250=14.11%, 500=0.41%, 750=0.10%, 1000=0.05% lat (msec) : 2=0.02%, 4=0.02%, 10=1.47%, 20=3.02%, 50=15.22% lat (msec) : 100=0.55%, 250=0.01% cpu : usr=1.96%, sys=4.58%, ctx=48740, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=42015,10612,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=350MiB/s (367MB/s), 350MiB/s-350MiB/s (367MB/s-367MB/s), io=10.3GiB (11.0GB), run=30023-30023msec WRITE: bw=88.4MiB/s (92.7MB/s), 88.4MiB/s-88.4MiB/s (92.7MB/s-92.7MB/s), io=2653MiB (2782MB), run=30023-30023msec Disk stats (read/write): vde: ios=41690/10524, merge=0/0, ticks=2339/285491, in_queue=296293, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944972: Fri Jan 7 14:12:48 2022 read: IOPS=1345, BW=336MiB/s (353MB/s)(9.86GiB/30004msec) clat (usec): min=51, max=18279, avg=95.89, stdev=384.42 lat (usec): min=51, max=18279, avg=96.03, stdev=384.43 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 69], 20.00th=[ 71], | 30.00th=[ 74], 40.00th=[ 76], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 92], 90.00th=[ 99], 95.00th=[ 108], | 99.00th=[ 129], 99.50th=[ 147], 99.90th=[ 8160], 99.95th=[10683], | 99.99th=[13960] bw ( KiB/s): min=285419, max=596480, per=100.00%, avg=345809.70, stdev=45207.61, samples=60 iops : min= 1114, max= 2330, avg=1350.42, stdev=176.65, samples=60 write: IOPS=339, BW=84.0MiB/s (89.1MB/s)(2549MiB/30004msec) clat (usec): min=846, max=56877, avg=2529.22, stdev=2383.15 lat (usec): min=870, max=56896, avg=2553.94, stdev=2383.06 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1057], 10.00th=[ 1172], 20.00th=[ 1778], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2180], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2671], 90.00th=[ 3359], 95.00th=[ 4686], | 99.00th=[10683], 99.50th=[12256], 99.90th=[52691], 99.95th=[53740], | 99.99th=[55313] bw ( KiB/s): min=75870, max=156672, per=100.00%, avg=87317.85, stdev=10512.30, samples=60 iops : min= 296, max= 612, avg=340.73, stdev=41.09, samples=60 lat (usec) : 100=72.47%, 250=7.21%, 500=0.04%, 750=0.01%, 1000=0.59% lat (msec) : 2=6.79%, 4=11.09%, 10=1.26%, 20=0.53%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.71%, sys=3.17%, ctx=50592, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40380,10197,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=336MiB/s (353MB/s), 336MiB/s-336MiB/s (353MB/s-353MB/s), io=9.86GiB (10.6GB), run=30004-30004msec WRITE: bw=84.0MiB/s (89.1MB/s), 84.0MiB/s-84.0MiB/s (89.1MB/s-89.1MB/s), io=2549MiB (2673MB), run=30004-30004msec Disk stats (read/write): vde: ios=40376/10195, merge=0/0, ticks=544/20839, in_queue=28426, util=94.45% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945600: Fri Jan 7 14:13:26 2022 read: IOPS=1399, BW=350MiB/s (367MB/s)(10.3GiB/30175msec) slat (usec): min=4, max=225, avg=13.78, stdev= 5.66 clat (nsec): min=899, max=2576.5M, avg=2059591.78, stdev=60549206.67 lat (usec): min=46, max=2576.5k, avg=2073.53, stdev=60549.18 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 52], | 20.00th=[ 54], 30.00th=[ 57], 40.00th=[ 60], | 50.00th=[ 64], 60.00th=[ 71], 70.00th=[ 91], | 80.00th=[ 114], 90.00th=[ 133], 95.00th=[ 165], | 99.00th=[ 1336], 99.50th=[ 3032], 99.90th=[ 725615], | 99.95th=[1518339], 99.99th=[2533360] bw ( KiB/s): min=265728, max=793088, per=100.00%, avg=360420.45, stdev=67884.89, samples=60 iops : min= 1038, max= 3098, avg=1407.87, stdev=265.18, samples=60 write: IOPS=353, BW=88.3MiB/s (92.6MB/s)(2665MiB/30175msec) slat (usec): min=20, max=249, avg=55.90, stdev=25.72 clat (msec): min=15, max=1930, avg=133.20, stdev=64.15 lat (msec): min=16, max=1930, avg=133.26, stdev=64.15 clat percentiles (msec): | 1.00th=[ 35], 5.00th=[ 93], 10.00th=[ 103], 20.00th=[ 112], | 30.00th=[ 118], 40.00th=[ 123], 50.00th=[ 128], 60.00th=[ 134], | 70.00th=[ 140], 80.00th=[ 148], 90.00th=[ 163], 95.00th=[ 178], | 99.00th=[ 271], 99.50th=[ 498], 99.90th=[ 995], 99.95th=[ 1435], | 99.99th=[ 1888] bw ( KiB/s): min=74752, max=181760, per=100.00%, avg=90553.07, stdev=13649.72, samples=60 iops : min= 292, max= 710, avg=353.70, stdev=53.32, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=4.52%, 100=53.35% lat (usec) : 250=19.67%, 500=0.70%, 750=0.34%, 1000=0.21% lat (msec) : 2=0.44%, 4=0.43%, 10=0.06%, 20=0.05%, 50=0.51% lat (msec) : 100=1.03%, 250=18.35%, 500=0.13%, 750=0.07%, 1000=0.04% lat (msec) : 2000=0.05%, >=2000=0.04% cpu : usr=2.49%, sys=3.85%, ctx=48009, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=42237,10658,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=350MiB/s (367MB/s), 350MiB/s-350MiB/s (367MB/s-367MB/s), io=10.3GiB (11.1GB), run=30175-30175msec WRITE: bw=88.3MiB/s (92.6MB/s), 88.3MiB/s-88.3MiB/s (92.6MB/s-92.6MB/s), io=2665MiB (2794MB), run=30175-30175msec Disk stats (read/write): vde: ios=42234/10625, merge=0/0, ticks=81050/1404933, in_queue=1499696, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946484: Fri Jan 7 14:14:02 2022 read: IOPS=1357, BW=339MiB/s (356MB/s)(9.94GiB/30004msec) clat (usec): min=49, max=16193, avg=92.62, stdev=386.22 lat (usec): min=49, max=16193, avg=92.76, stdev=386.22 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 70], 40.00th=[ 72], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 89], 90.00th=[ 96], 95.00th=[ 103], | 99.00th=[ 125], 99.50th=[ 141], 99.90th=[ 7898], 99.95th=[11076], | 99.99th=[14484] bw ( KiB/s): min=252416, max=578560, per=100.00%, avg=347522.63, stdev=44251.13, samples=60 iops : min= 986, max= 2260, avg=1357.50, stdev=172.86, samples=60 write: IOPS=342, BW=85.6MiB/s (89.8MB/s)(2569MiB/30004msec) clat (usec): min=819, max=62810, avg=2518.98, stdev=2319.73 lat (usec): min=853, max=62834, avg=2545.12, stdev=2319.52 clat percentiles (usec): | 1.00th=[ 906], 5.00th=[ 1074], 10.00th=[ 1188], 20.00th=[ 1795], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2671], 90.00th=[ 3261], 95.00th=[ 4555], | 99.00th=[10814], 99.50th=[12387], 99.90th=[21890], 99.95th=[53740], | 99.99th=[54789] bw ( KiB/s): min=73728, max=154112, per=100.00%, avg=87665.87, stdev=9955.50, samples=60 iops : min= 288, max= 602, avg=342.43, stdev=38.89, samples=60 lat (usec) : 50=0.01%, 100=74.43%, 250=5.27%, 500=0.02%, 750=0.01% lat (usec) : 1000=0.55% lat (msec) : 2=6.78%, 4=11.30%, 10=1.11%, 20=0.51%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.78%, sys=2.91%, ctx=51023, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40724,10274,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=339MiB/s (356MB/s), 339MiB/s-339MiB/s (356MB/s-356MB/s), io=9.94GiB (10.7GB), run=30004-30004msec WRITE: bw=85.6MiB/s (89.8MB/s), 85.6MiB/s-85.6MiB/s (89.8MB/s-89.8MB/s), io=2569MiB (2693MB), run=30004-30004msec Disk stats (read/write): vde: ios=40461/10220, merge=0/0, ticks=544/20754, in_queue=28294, util=94.47% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947443: Fri Jan 7 14:14:40 2022 read: IOPS=1394, BW=349MiB/s (366MB/s)(10.3GiB/30342msec) slat (usec): min=4, max=284, avg=12.97, stdev= 7.43 clat (nsec): min=859, max=2784.1M, avg=2005936.82, stdev=57853817.92 lat (usec): min=46, max=2784.1k, avg=2019.06, stdev=57853.81 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 49], 10.00th=[ 51], | 20.00th=[ 53], 30.00th=[ 56], 40.00th=[ 60], | 50.00th=[ 65], 60.00th=[ 75], 70.00th=[ 105], | 80.00th=[ 120], 90.00th=[ 161], 95.00th=[ 750], | 99.00th=[ 6128], 99.50th=[ 7832], 99.90th=[ 522191], | 99.95th=[1702888], 99.99th=[2533360] bw ( KiB/s): min=259591, max=924672, per=100.00%, avg=361086.85, stdev=83791.48, samples=60 iops : min= 1014, max= 3612, avg=1410.47, stdev=327.32, samples=60 write: IOPS=352, BW=88.0MiB/s (92.3MB/s)(2671MiB/30342msec) slat (usec): min=20, max=805, avg=55.87, stdev=29.44 clat (msec): min=23, max=2330, avg=275.88, stdev=90.09 lat (msec): min=23, max=2330, avg=275.94, stdev=90.09 clat percentiles (msec): | 1.00th=[ 64], 5.00th=[ 157], 10.00th=[ 232], 20.00th=[ 247], | 30.00th=[ 255], 40.00th=[ 264], 50.00th=[ 271], 60.00th=[ 279], | 70.00th=[ 288], 80.00th=[ 300], 90.00th=[ 330], 95.00th=[ 372], | 99.00th=[ 542], 99.50th=[ 584], 99.90th=[ 1385], 99.95th=[ 1502], | 99.99th=[ 2165] bw ( KiB/s): min=69120, max=186880, per=100.00%, avg=90362.43, stdev=14605.32, samples=60 iops : min= 270, max= 730, avg=352.93, stdev=57.06, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=6.79%, 100=47.76% lat (usec) : 250=19.21%, 500=1.31%, 750=0.74%, 1000=0.60% lat (msec) : 2=1.27%, 4=0.84%, 10=1.07%, 20=0.07%, 50=0.13% lat (msec) : 100=0.52%, 250=4.01%, 500=15.13%, 750=0.35%, 1000=0.04% lat (msec) : 2000=0.05%, >=2000=0.04% cpu : usr=2.46%, sys=3.57%, ctx=45536, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42314,10684,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=349MiB/s (366MB/s), 349MiB/s-349MiB/s (366MB/s-366MB/s), io=10.3GiB (11.1GB), run=30342-30342msec WRITE: bw=88.0MiB/s (92.3MB/s), 88.0MiB/s-88.0MiB/s (92.3MB/s-92.3MB/s), io=2671MiB (2801MB), run=30342-30342msec Disk stats (read/write): vde: ios=42310/10658, merge=0/0, ticks=73977/2915798, in_queue=3011671, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948017: Fri Jan 7 14:15:17 2022 read: IOPS=1345, BW=336MiB/s (353MB/s)(9.86GiB/30007msec) clat (usec): min=52, max=20262, avg=94.67, stdev=416.85 lat (usec): min=52, max=20262, avg=94.82, stdev=416.85 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 97], 95.00th=[ 105], | 99.00th=[ 133], 99.50th=[ 153], 99.90th=[ 9110], 99.95th=[11469], | 99.99th=[15401] bw ( KiB/s): min=264192, max=555520, per=100.00%, avg=344468.67, stdev=41949.54, samples=60 iops : min= 1032, max= 2170, avg=1345.57, stdev=163.87, samples=60 write: IOPS=339, BW=84.9MiB/s (89.1MB/s)(2549MiB/30007msec) clat (usec): min=850, max=53377, avg=2535.57, stdev=2315.79 lat (usec): min=875, max=53401, avg=2559.86, stdev=2315.61 clat percentiles (usec): | 1.00th=[ 914], 5.00th=[ 1057], 10.00th=[ 1156], 20.00th=[ 1778], | 30.00th=[ 1926], 40.00th=[ 2040], 50.00th=[ 2147], 60.00th=[ 2278], | 70.00th=[ 2442], 80.00th=[ 2704], 90.00th=[ 3392], 95.00th=[ 4686], | 99.00th=[10683], 99.50th=[11863], 99.90th=[20841], 99.95th=[52691], | 99.99th=[53216] bw ( KiB/s): min=67072, max=148480, per=100.00%, avg=86974.87, stdev=10275.31, samples=60 iops : min= 262, max= 580, avg=339.73, stdev=40.15, samples=60 lat (usec) : 100=73.68%, 250=5.98%, 500=0.03%, 750=0.03%, 1000=0.57% lat (msec) : 2=6.77%, 4=11.15%, 10=1.21%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.75%, sys=2.89%, ctx=50574, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=40370,10194,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=336MiB/s (353MB/s), 336MiB/s-336MiB/s (353MB/s-353MB/s), io=9.86GiB (10.6GB), run=30007-30007msec WRITE: bw=84.9MiB/s (89.1MB/s), 84.9MiB/s-84.9MiB/s (89.1MB/s-89.1MB/s), io=2549MiB (2672MB), run=30007-30007msec Disk stats (read/write): vde: ios=40370/10193, merge=0/0, ticks=571/20879, in_queue=28557, util=94.80% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949370: Fri Jan 7 14:15:54 2022 read: IOPS=1388, BW=347MiB/s (364MB/s)(10.3GiB/30349msec) slat (usec): min=4, max=354858, avg=557.27, stdev=4924.19 clat (msec): min=4, max=2863, avg=47.32, stdev=69.01 lat (msec): min=4, max=2863, avg=47.87, stdev=69.15 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 12], 20.00th=[ 29], | 30.00th=[ 35], 40.00th=[ 39], 50.00th=[ 43], 60.00th=[ 47], | 70.00th=[ 52], 80.00th=[ 58], 90.00th=[ 69], 95.00th=[ 85], | 99.00th=[ 249], 99.50th=[ 355], 99.90th=[ 447], 99.95th=[ 1921], | 99.99th=[ 2601] bw ( KiB/s): min=225280, max=908288, per=100.00%, avg=359065.08, stdev=84294.92, samples=60 iops : min= 880, max= 3548, avg=1402.58, stdev=329.28, samples=60 write: IOPS=350, BW=87.7MiB/s (91.9MB/s)(2661MiB/30349msec) slat (usec): min=19, max=97971, avg=601.61, stdev=3448.39 clat (msec): min=33, max=3083, avg=380.01, stdev=210.17 lat (msec): min=33, max=3083, avg=380.61, stdev=210.16 clat percentiles (msec): | 1.00th=[ 62], 5.00th=[ 215], 10.00th=[ 271], 20.00th=[ 317], | 30.00th=[ 338], 40.00th=[ 351], 50.00th=[ 363], 60.00th=[ 376], | 70.00th=[ 388], 80.00th=[ 409], 90.00th=[ 447], 95.00th=[ 575], | 99.00th=[ 936], 99.50th=[ 1955], 99.90th=[ 3071], 99.95th=[ 3071], | 99.99th=[ 3071] bw ( KiB/s): min=73216, max=171008, per=99.89%, avg=89677.15, stdev=13431.05, samples=60 iops : min= 286, max= 668, avg=350.27, stdev=52.46, samples=60 lat (msec) : 10=5.80%, 20=6.64%, 50=41.28%, 100=24.58%, 250=2.16% lat (msec) : 500=18.13%, 750=0.99%, 1000=0.19%, 2000=0.12%, >=2000=0.11% cpu : usr=3.25%, sys=1.84%, ctx=5584, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=42138,10643,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=347MiB/s (364MB/s), 347MiB/s-347MiB/s (364MB/s-364MB/s), io=10.3GiB (11.0GB), run=30349-30349msec WRITE: bw=87.7MiB/s (91.9MB/s), 87.7MiB/s-87.7MiB/s (91.9MB/s-91.9MB/s), io=2661MiB (2790MB), run=30349-30349msec Disk stats (read/write): vde: ios=42135/10632, merge=0/0, ticks=97641/3567135, in_queue=3698069, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950053: Fri Jan 7 14:16:31 2022 read: IOPS=911, BW=456MiB/s (478MB/s)(13.4GiB/30001msec) clat (usec): min=103, max=18795, avg=165.82, stdev=489.06 lat (usec): min=103, max=18795, avg=165.97, stdev=489.06 clat percentiles (usec): | 1.00th=[ 114], 5.00th=[ 118], 10.00th=[ 120], 20.00th=[ 123], | 30.00th=[ 127], 40.00th=[ 131], 50.00th=[ 137], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 161], 90.00th=[ 178], 95.00th=[ 194], | 99.00th=[ 249], 99.50th=[ 281], 99.90th=[ 9896], 99.95th=[12780], | 99.99th=[16057] bw ( KiB/s): min=336896, max=630784, per=100.00%, avg=466755.75, stdev=61394.46, samples=60 iops : min= 658, max= 1232, avg=911.62, stdev=119.90, samples=60 write: IOPS=229, BW=115MiB/s (120MB/s)(3446MiB/30001msec) clat (usec): min=1447, max=60656, avg=3636.43, stdev=2755.36 lat (usec): min=1506, max=60710, avg=3686.28, stdev=2754.93 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1696], 10.00th=[ 1811], 20.00th=[ 2409], | 30.00th=[ 2671], 40.00th=[ 2900], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3458], 80.00th=[ 3851], 90.00th=[ 6128], 95.00th=[ 7504], | 99.00th=[13435], 99.50th=[15533], 99.90th=[54264], 99.95th=[54264], | 99.99th=[60556] bw ( KiB/s): min=89088, max=163840, per=99.99%, avg=117593.80, stdev=12885.88, samples=60 iops : min= 174, max= 320, avg=229.67, stdev=25.16, samples=60 lat (usec) : 250=79.10%, 500=0.57%, 750=0.02%, 1000=0.01% lat (msec) : 2=2.50%, 4=14.10%, 10=2.97%, 20=0.70%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.78%, sys=2.63%, ctx=34247, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=27348,6891,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=456MiB/s (478MB/s), 456MiB/s-456MiB/s (478MB/s-478MB/s), io=13.4GiB (14.3GB), run=30001-30001msec WRITE: bw=115MiB/s (120MB/s), 115MiB/s-115MiB/s (120MB/s-120MB/s), io=3446MiB (3613MB), run=30001-30001msec Disk stats (read/write): vde: ios=27192/6855, merge=0/0, ticks=570/21238, in_queue=28358, util=94.92% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950867: Fri Jan 7 14:17:08 2022 read: IOPS=968, BW=484MiB/s (508MB/s)(14.2GiB/30001msec) slat (usec): min=16, max=244, avg=26.09, stdev= 6.25 clat (nsec): min=1396, max=56392k, avg=119204.95, stdev=643591.86 lat (usec): min=83, max=56415, avg=145.47, stdev=643.61 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 76], 10.00th=[ 77], 20.00th=[ 80], | 30.00th=[ 82], 40.00th=[ 84], 50.00th=[ 88], 60.00th=[ 92], | 70.00th=[ 98], 80.00th=[ 104], 90.00th=[ 115], 95.00th=[ 124], | 99.00th=[ 159], 99.50th=[ 188], 99.90th=[11863], 99.95th=[14484], | 99.99th=[16909] bw ( KiB/s): min=366592, max=619520, per=100.00%, avg=497073.90, stdev=53995.21, samples=59 iops : min= 716, max= 1210, avg=970.85, stdev=105.46, samples=59 write: IOPS=244, BW=122MiB/s (128MB/s)(3672MiB/30001msec) slat (usec): min=38, max=292, avg=72.31, stdev=15.63 clat (usec): min=1356, max=55706, avg=3428.76, stdev=2293.05 lat (usec): min=1423, max=55777, avg=3501.28, stdev=2292.66 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1647], 10.00th=[ 1696], 20.00th=[ 2343], | 30.00th=[ 2671], 40.00th=[ 2933], 50.00th=[ 3097], 60.00th=[ 3261], | 70.00th=[ 3490], 80.00th=[ 3785], 90.00th=[ 5014], 95.00th=[ 6718], | 99.00th=[11469], 99.50th=[11994], 99.90th=[18744], 99.95th=[53740], | 99.99th=[55837] bw ( KiB/s): min=103424, max=181248, per=100.00%, avg=125400.71, stdev=10048.05, samples=59 iops : min= 202, max= 354, avg=244.92, stdev=19.63, samples=59 lat (usec) : 2=0.01%, 10=0.01%, 50=0.02%, 100=58.47%, 250=21.08% lat (usec) : 500=0.05%, 750=0.01% lat (msec) : 2=3.18%, 4=13.76%, 10=2.86%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.88%, sys=4.18%, ctx=36421, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29059,7344,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=484MiB/s (508MB/s), 484MiB/s-484MiB/s (508MB/s-508MB/s), io=14.2GiB (15.2GB), run=30001-30001msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3672MiB (3850MB), run=30001-30001msec Disk stats (read/write): vde: ios=28867/7292, merge=0/0, ticks=750/21303, in_queue=28300, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951819: Fri Jan 7 14:17:45 2022 read: IOPS=969, BW=485MiB/s (509MB/s)(14.2GiB/30006msec) clat (usec): min=77, max=20718, avg=135.94, stdev=543.80 lat (usec): min=78, max=20718, avg=136.09, stdev=543.80 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 94], 20.00th=[ 96], | 30.00th=[ 99], 40.00th=[ 102], 50.00th=[ 106], 60.00th=[ 111], | 70.00th=[ 118], 80.00th=[ 126], 90.00th=[ 137], 95.00th=[ 149], | 99.00th=[ 186], 99.50th=[ 208], 99.90th=[11338], 99.95th=[13566], | 99.99th=[17433] bw ( KiB/s): min=369664, max=626688, per=100.00%, avg=496691.20, stdev=55951.46, samples=60 iops : min= 722, max= 1224, avg=970.10, stdev=109.28, samples=60 write: IOPS=245, BW=123MiB/s (128MB/s)(3677MiB/30006msec) clat (usec): min=1373, max=55882, avg=3479.62, stdev=2451.86 lat (usec): min=1427, max=55921, avg=3532.68, stdev=2451.53 clat percentiles (usec): | 1.00th=[ 1483], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3818], 90.00th=[ 4621], 95.00th=[ 6783], | 99.00th=[11600], 99.50th=[11994], 99.90th=[53216], 99.95th=[54264], | 99.99th=[55837] bw ( KiB/s): min=107520, max=181248, per=100.00%, avg=125491.20, stdev=10301.30, samples=60 iops : min= 210, max= 354, avg=245.10, stdev=20.12, samples=60 lat (usec) : 100=27.54%, 250=52.06%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.91%, 4=14.04%, 10=2.77%, 20=0.60%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.86%, sys=2.90%, ctx=36469, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29103,7354,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=485MiB/s (509MB/s), 485MiB/s-485MiB/s (509MB/s-509MB/s), io=14.2GiB (15.3GB), run=30006-30006msec WRITE: bw=123MiB/s (128MB/s), 123MiB/s-123MiB/s (128MB/s-128MB/s), io=3677MiB (3856MB), run=30006-30006msec Disk stats (read/write): vde: ios=28885/7299, merge=0/0, ticks=686/21523, in_queue=28377, util=94.93% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952401: Fri Jan 7 14:18:22 2022 read: IOPS=990, BW=495MiB/s (519MB/s)(14.5GiB/30033msec) slat (usec): min=5, max=135, avg=17.62, stdev= 6.82 clat (nsec): min=1018, max=92578k, avg=189269.30, stdev=1740043.97 lat (usec): min=70, max=92596, avg=207.05, stdev=1740.11 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 70], 10.00th=[ 72], 20.00th=[ 74], | 30.00th=[ 77], 40.00th=[ 82], 50.00th=[ 88], 60.00th=[ 95], | 70.00th=[ 111], 80.00th=[ 174], 90.00th=[ 198], 95.00th=[ 221], | 99.00th=[ 318], 99.50th=[ 515], 99.90th=[33817], 99.95th=[41157], | 99.99th=[54789] bw ( KiB/s): min=359424, max=700416, per=100.00%, avg=507801.60, stdev=60785.58, samples=60 iops : min= 702, max= 1368, avg=991.80, stdev=118.72, samples=60 write: IOPS=249, BW=125MiB/s (131MB/s)(3754MiB/30033msec) slat (usec): min=35, max=324, avg=89.21, stdev=41.43 clat (msec): min=2, max=131, avg=39.08, stdev=16.10 lat (msec): min=2, max=132, avg=39.17, stdev=16.10 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 13], 10.00th=[ 18], 20.00th=[ 28], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 43], | 70.00th=[ 47], 80.00th=[ 52], 90.00th=[ 59], 95.00th=[ 65], | 99.00th=[ 84], 99.50th=[ 100], 99.90th=[ 121], 99.95th=[ 125], | 99.99th=[ 132] bw ( KiB/s): min=107520, max=195584, per=99.98%, avg=127965.87, stdev=10926.77, samples=60 iops : min= 210, max= 382, avg=249.93, stdev=21.34, samples=60 lat (usec) : 2=0.01%, 50=0.03%, 100=51.79%, 250=26.07%, 500=1.55% lat (usec) : 750=0.10%, 1000=0.04% lat (msec) : 2=0.10%, 4=0.01%, 10=0.66%, 20=1.68%, 50=13.41% lat (msec) : 100=4.47%, 250=0.10% cpu : usr=2.61%, sys=3.08%, ctx=35721, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29754,7508,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=495MiB/s (519MB/s), 495MiB/s-495MiB/s (519MB/s-519MB/s), io=14.5GiB (15.6GB), run=30033-30033msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3754MiB (3936MB), run=30033-30033msec Disk stats (read/write): vde: ios=29586/7452, merge=0/0, ticks=2194/287234, in_queue=296530, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953229: Fri Jan 7 14:18:59 2022 read: IOPS=975, BW=488MiB/s (511MB/s)(14.3GiB/30008msec) clat (usec): min=79, max=22409, avg=135.12, stdev=550.07 lat (usec): min=79, max=22409, avg=135.27, stdev=550.07 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 93], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 102], 50.00th=[ 105], 60.00th=[ 111], | 70.00th=[ 118], 80.00th=[ 125], 90.00th=[ 135], 95.00th=[ 143], | 99.00th=[ 172], 99.50th=[ 194], 99.90th=[11863], 99.95th=[13960], | 99.99th=[17695] bw ( KiB/s): min=388096, max=625664, per=100.00%, avg=499370.77, stdev=51234.17, samples=60 iops : min= 758, max= 1222, avg=975.32, stdev=100.08, samples=60 write: IOPS=246, BW=123MiB/s (129MB/s)(3696MiB/30008msec) clat (usec): min=1462, max=61570, avg=3466.61, stdev=2389.96 lat (usec): min=1507, max=61631, avg=3514.43, stdev=2389.55 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2409], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3163], 60.00th=[ 3326], | 70.00th=[ 3523], 80.00th=[ 3818], 90.00th=[ 4621], 95.00th=[ 6652], | 99.00th=[11600], 99.50th=[12125], 99.90th=[45876], 99.95th=[53740], | 99.99th=[61604] bw ( KiB/s): min=102605, max=169984, per=100.00%, avg=126147.82, stdev=9073.91, samples=60 iops : min= 200, max= 332, avg=246.35, stdev=17.74, samples=60 lat (usec) : 100=27.91%, 250=51.68%, 500=0.04%, 750=0.01%, 1000=0.01% lat (msec) : 2=3.02%, 4=13.87%, 10=2.92%, 20=0.52%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=1.73%, sys=2.86%, ctx=36663, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29258,7392,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=488MiB/s (511MB/s), 488MiB/s-488MiB/s (511MB/s-511MB/s), io=14.3GiB (15.3GB), run=30008-30008msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3696MiB (3876MB), run=30008-30008msec Disk stats (read/write): vde: ios=29043/7336, merge=0/0, ticks=688/21499, in_queue=28311, util=94.73% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954258: Fri Jan 7 14:19:36 2022 read: IOPS=1001, BW=501MiB/s (525MB/s)(14.8GiB/30188msec) slat (usec): min=5, max=225, avg=15.15, stdev= 6.58 clat (nsec): min=1113, max=297655k, avg=408436.87, stdev=5039171.89 lat (usec): min=68, max=297667, avg=423.75, stdev=5039.10 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 69], 10.00th=[ 70], 20.00th=[ 72], | 30.00th=[ 75], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 88], | 70.00th=[ 105], 80.00th=[ 184], 90.00th=[ 204], 95.00th=[ 273], | 99.00th=[ 4686], 99.50th=[ 6849], 99.90th=[ 89654], 99.95th=[127402], | 99.99th=[179307] bw ( KiB/s): min=363520, max=872448, per=100.00%, avg=516232.67, stdev=70934.65, samples=60 iops : min= 710, max= 1704, avg=1008.27, stdev=138.54, samples=60 write: IOPS=252, BW=126MiB/s (132MB/s)(3813MiB/30188msec) slat (usec): min=35, max=384, avg=93.58, stdev=47.58 clat (msec): min=9, max=2611, avg=196.14, stdev=115.33 lat (msec): min=9, max=2611, avg=196.23, stdev=115.33 clat percentiles (msec): | 1.00th=[ 51], 5.00th=[ 70], 10.00th=[ 82], 20.00th=[ 103], | 30.00th=[ 126], 40.00th=[ 155], 50.00th=[ 192], 60.00th=[ 222], | 70.00th=[ 245], 80.00th=[ 271], 90.00th=[ 309], 95.00th=[ 347], | 99.00th=[ 435], 99.50th=[ 542], 99.90th=[ 1536], 99.95th=[ 2039], | 99.99th=[ 2601] bw ( KiB/s): min=112640, max=193536, per=99.98%, avg=129301.50, stdev=10512.12, samples=60 iops : min= 220, max= 378, avg=252.53, stdev=20.53, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=0.05%, 100=54.57%, 250=20.90% lat (usec) : 500=1.48%, 750=0.45%, 1000=0.31% lat (msec) : 2=0.67%, 4=0.51%, 10=0.76%, 20=0.02%, 50=0.20% lat (msec) : 100=3.76%, 250=10.67%, 500=5.53%, 750=0.07%, 1000=0.01% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.76%, sys=2.78%, ctx=35472, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=30248,7625,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=501MiB/s (525MB/s), 501MiB/s-501MiB/s (525MB/s-525MB/s), io=14.8GiB (15.9GB), run=30188-30188msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3813MiB (3998MB), run=30188-30188msec Disk stats (read/write): vde: ios=30247/7583, merge=0/0, ticks=8032/1478292, in_queue=1499245, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954781: Fri Jan 7 14:20:13 2022 read: IOPS=980, BW=490MiB/s (514MB/s)(14.4GiB/30001msec) clat (usec): min=79, max=20247, avg=139.46, stdev=585.05 lat (usec): min=79, max=20247, avg=139.61, stdev=585.05 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 94], 10.00th=[ 96], 20.00th=[ 99], | 30.00th=[ 101], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 114], | 70.00th=[ 120], 80.00th=[ 128], 90.00th=[ 139], 95.00th=[ 149], | 99.00th=[ 182], 99.50th=[ 206], 99.90th=[13042], 99.95th=[14353], | 99.99th=[18482] bw ( KiB/s): min=372736, max=619520, per=99.78%, avg=500666.58, stdev=59060.46, samples=59 iops : min= 728, max= 1210, avg=977.86, stdev=115.35, samples=59 write: IOPS=247, BW=124MiB/s (130MB/s)(3712MiB/30001msec) clat (usec): min=1408, max=56209, avg=3430.79, stdev=2386.38 lat (usec): min=1459, max=56273, avg=3480.22, stdev=2385.65 clat percentiles (usec): | 1.00th=[ 1549], 5.00th=[ 1663], 10.00th=[ 1745], 20.00th=[ 2442], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3720], 90.00th=[ 4424], 95.00th=[ 6456], | 99.00th=[11469], 99.50th=[12125], 99.90th=[52691], 99.95th=[53740], | 99.99th=[56361] bw ( KiB/s): min=106496, max=173056, per=99.82%, avg=126455.32, stdev=10187.56, samples=59 iops : min= 208, max= 338, avg=246.98, stdev=19.90, samples=59 lat (usec) : 100=20.97%, 250=58.63%, 500=0.05%, 750=0.01% lat (msec) : 2=2.87%, 4=14.38%, 10=2.51%, 20=0.55%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.88%, sys=2.90%, ctx=36840, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29401,7423,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=490MiB/s (514MB/s), 490MiB/s-490MiB/s (514MB/s-514MB/s), io=14.4GiB (15.4GB), run=30001-30001msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3712MiB (3892MB), run=30001-30001msec Disk stats (read/write): vde: ios=29378/7418, merge=0/0, ticks=743/21447, in_queue=28491, util=94.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956006: Fri Jan 7 14:20:50 2022 read: IOPS=996, BW=498MiB/s (523MB/s)(14.8GiB/30435msec) slat (usec): min=5, max=423, avg=14.48, stdev= 9.31 clat (nsec): min=1165, max=491794k, avg=797211.07, stdev=5691195.89 lat (usec): min=69, max=491805, avg=811.84, stdev=5691.05 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 68], 10.00th=[ 70], 20.00th=[ 73], | 30.00th=[ 75], 40.00th=[ 78], 50.00th=[ 83], 60.00th=[ 95], | 70.00th=[ 180], 80.00th=[ 196], 90.00th=[ 725], 95.00th=[ 3130], | 99.00th=[ 13042], 99.50th=[ 15270], 99.90th=[ 91751], 99.95th=[119014], | 99.99th=[162530] bw ( KiB/s): min=350208, max=1057792, per=100.00%, avg=517733.70, stdev=94206.21, samples=60 iops : min= 684, max= 2066, avg=1011.15, stdev=184.01, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3826MiB/30435msec) slat (usec): min=32, max=1841, avg=98.60, stdev=53.82 clat (msec): min=13, max=2629, avg=394.30, stdev=152.21 lat (msec): min=13, max=2629, avg=394.40, stdev=152.21 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 192], 10.00th=[ 259], 20.00th=[ 288], | 30.00th=[ 313], 40.00th=[ 342], 50.00th=[ 384], 60.00th=[ 418], | 70.00th=[ 447], 80.00th=[ 477], 90.00th=[ 550], 95.00th=[ 642], | 99.00th=[ 852], 99.50th=[ 911], 99.90th=[ 1854], 99.95th=[ 2123], | 99.99th=[ 2635] bw ( KiB/s): min=98304, max=183296, per=100.00%, avg=128905.72, stdev=12044.17, samples=60 iops : min= 192, max= 358, avg=251.73, stdev=23.50, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.06%, 100=49.16% lat (usec) : 250=19.14%, 500=2.35%, 750=1.28%, 1000=0.81% lat (msec) : 2=1.90%, 4=1.89%, 10=1.62%, 20=1.46%, 50=0.05% lat (msec) : 100=0.39%, 250=1.43%, 500=15.31%, 750=2.71%, 1000=0.36% lat (msec) : 2000=0.04%, >=2000=0.01% cpu : usr=2.88%, sys=2.53%, ctx=32011, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30332,7652,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=498MiB/s (523MB/s), 498MiB/s-498MiB/s (523MB/s-523MB/s), io=14.8GiB (15.9GB), run=30435-30435msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3826MiB (4012MB), run=30435-30435msec Disk stats (read/write): vde: ios=30332/7614, merge=0/0, ticks=16634/2974215, in_queue=3012976, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956751: Fri Jan 7 14:21:27 2022 read: IOPS=988, BW=494MiB/s (518MB/s)(14.5GiB/30005msec) clat (usec): min=76, max=20487, avg=132.73, stdev=552.25 lat (usec): min=76, max=20487, avg=132.88, stdev=552.25 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 97], 40.00th=[ 99], 50.00th=[ 102], 60.00th=[ 108], | 70.00th=[ 115], 80.00th=[ 122], 90.00th=[ 133], 95.00th=[ 143], | 99.00th=[ 182], 99.50th=[ 210], 99.90th=[11338], 99.95th=[14615], | 99.99th=[18744] bw ( KiB/s): min=362496, max=653312, per=100.00%, avg=506110.65, stdev=58006.80, samples=60 iops : min= 708, max= 1276, avg=988.47, stdev=113.31, samples=60 write: IOPS=249, BW=125MiB/s (131MB/s)(3742MiB/30005msec) clat (usec): min=1409, max=63655, avg=3424.80, stdev=2392.13 lat (usec): min=1458, max=63703, avg=3474.38, stdev=2391.42 clat percentiles (usec): | 1.00th=[ 1516], 5.00th=[ 1663], 10.00th=[ 1729], 20.00th=[ 2442], | 30.00th=[ 2737], 40.00th=[ 2966], 50.00th=[ 3130], 60.00th=[ 3294], | 70.00th=[ 3490], 80.00th=[ 3752], 90.00th=[ 4490], 95.00th=[ 6390], | 99.00th=[11600], 99.50th=[12256], 99.90th=[47449], 99.95th=[54264], | 99.99th=[63701] bw ( KiB/s): min=106496, max=186368, per=100.00%, avg=127722.25, stdev=10182.58, samples=60 iops : min= 208, max= 364, avg=249.43, stdev=19.89, samples=60 lat (usec) : 100=35.37%, 250=44.23%, 500=0.05%, 750=0.01%, 1000=0.01% lat (msec) : 2=2.87%, 4=14.38%, 10=2.57%, 20=0.48%, 50=0.01% lat (msec) : 100=0.02% cpu : usr=1.91%, sys=2.72%, ctx=37145, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29652,7484,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=14.5GiB (15.5GB), run=30005-30005msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3742MiB (3924MB), run=30005-30005msec Disk stats (read/write): vde: ios=29434/7425, merge=0/0, ticks=683/21436, in_queue=28271, util=94.50% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957727: Fri Jan 7 14:22:04 2022 read: IOPS=995, BW=498MiB/s (522MB/s)(14.8GiB/30479msec) slat (usec): min=6, max=514537, avg=756.55, stdev=6361.36 clat (msec): min=9, max=636, avg=64.58, stdev=56.23 lat (msec): min=9, max=636, avg=65.34, stdev=56.60 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 18], 10.00th=[ 26], 20.00th=[ 37], | 30.00th=[ 45], 40.00th=[ 52], 50.00th=[ 58], 60.00th=[ 65], | 70.00th=[ 72], 80.00th=[ 81], 90.00th=[ 94], 95.00th=[ 109], | 99.00th=[ 468], 99.50th=[ 518], 99.90th=[ 558], 99.95th=[ 575], | 99.99th=[ 584] bw ( KiB/s): min=22528, max=1150175, per=100.00%, avg=518235.47, stdev=190636.88, samples=60 iops : min= 44, max= 2246, avg=1012.10, stdev=372.33, samples=60 write: IOPS=251, BW=126MiB/s (132MB/s)(3828MiB/30479msec) slat (usec): min=134, max=459750, avg=908.73, stdev=6304.40 clat (msec): min=21, max=3886, avg=536.04, stdev=239.60 lat (msec): min=21, max=3886, avg=536.95, stdev=239.47 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 279], 10.00th=[ 351], 20.00th=[ 409], | 30.00th=[ 447], 40.00th=[ 485], 50.00th=[ 523], 60.00th=[ 550], | 70.00th=[ 584], 80.00th=[ 617], 90.00th=[ 693], 95.00th=[ 860], | 99.00th=[ 1502], 99.50th=[ 1921], 99.90th=[ 3876], 99.95th=[ 3876], | 99.99th=[ 3876] bw ( KiB/s): min=10240, max=244736, per=100.00%, avg=128665.57, stdev=34222.24, samples=60 iops : min= 20, max= 478, avg=251.22, stdev=66.82, samples=60 lat (msec) : 10=0.02%, 20=5.38%, 50=24.78%, 100=44.06%, 250=5.50% lat (msec) : 500=8.56%, 750=10.18%, 1000=1.06%, 2000=0.40%, >=2000=0.06% cpu : usr=4.52%, sys=1.75%, ctx=3889, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30334,7655,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=498MiB/s (522MB/s), 498MiB/s-498MiB/s (522MB/s-522MB/s), io=14.8GiB (15.9GB), run=30479-30479msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3828MiB (4013MB), run=30479-30479msec Disk stats (read/write): vde: ios=30334/7590, merge=0/0, ticks=69745/3597595, in_queue=3708922, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958712: Fri Jan 7 14:22:41 2022 read: IOPS=603, BW=604MiB/s (633MB/s)(17.7GiB/30009msec) clat (usec): min=127, max=27632, avg=230.96, stdev=988.81 lat (usec): min=127, max=27633, avg=231.13, stdev=988.81 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 159], 60.00th=[ 165], | 70.00th=[ 176], 80.00th=[ 192], 90.00th=[ 208], 95.00th=[ 223], | 99.00th=[ 302], 99.50th=[ 416], 99.90th=[16909], 99.95th=[20055], | 99.99th=[25822] bw ( KiB/s): min=504700, max=770064, per=100.00%, avg=626053.62, stdev=74425.50, samples=60 iops : min= 492, max= 752, avg=610.88, stdev=72.65, samples=60 write: IOPS=152, BW=152MiB/s (160MB/s)(4575MiB/30009msec) clat (usec): min=2546, max=56915, avg=5537.12, stdev=3140.56 lat (usec): min=2618, max=57037, avg=5635.29, stdev=3140.11 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3458], | 30.00th=[ 4359], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 6063], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[16188], 99.50th=[17433], 99.90th=[50594], 99.95th=[54264], | 99.99th=[56886] bw ( KiB/s): min=130327, max=214709, per=100.00%, avg=158086.85, stdev=12953.33, samples=60 iops : min= 127, max= 209, avg=153.90, stdev=12.63, samples=60 lat (usec) : 250=78.05%, 500=1.42%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.05%, 10=13.54%, 20=1.83%, 50=0.06% lat (msec) : 100=0.02% cpu : usr=1.98%, sys=2.38%, ctx=22706, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18115,4575,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=17.7GiB (18.0GB), run=30009-30009msec WRITE: bw=152MiB/s (160MB/s), 152MiB/s-152MiB/s (160MB/s-160MB/s), io=4575MiB (4797MB), run=30009-30009msec Disk stats (read/write): vde: ios=18105/4572, merge=0/0, ticks=1085/22767, in_queue=28642, util=95.20% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959291: Fri Jan 7 14:23:18 2022 read: IOPS=610, BW=611MiB/s (640MB/s)(17.9GiB/30005msec) slat (usec): min=28, max=196, avg=37.98, stdev= 8.11 clat (usec): min=48, max=24781, avg=194.18, stdev=948.39 lat (usec): min=131, max=24817, avg=232.34, stdev=948.61 clat percentiles (usec): | 1.00th=[ 108], 5.00th=[ 112], 10.00th=[ 113], 20.00th=[ 116], | 30.00th=[ 119], 40.00th=[ 121], 50.00th=[ 125], 60.00th=[ 130], | 70.00th=[ 141], 80.00th=[ 155], 90.00th=[ 169], 95.00th=[ 188], | 99.00th=[ 285], 99.50th=[ 482], 99.90th=[16909], 99.95th=[19006], | 99.99th=[23725] bw ( KiB/s): min=442368, max=880640, per=100.00%, avg=625550.37, stdev=97509.23, samples=60 iops : min= 432, max= 860, avg=610.88, stdev=95.22, samples=60 write: IOPS=154, BW=155MiB/s (162MB/s)(4644MiB/30005msec) slat (usec): min=68, max=480, avg=128.68, stdev=26.24 clat (usec): min=2532, max=65063, avg=5406.30, stdev=2977.70 lat (usec): min=2647, max=65182, avg=5535.21, stdev=2976.50 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2638], 10.00th=[ 2704], 20.00th=[ 3523], | 30.00th=[ 4359], 40.00th=[ 4752], 50.00th=[ 5014], 60.00th=[ 5276], | 70.00th=[ 5538], 80.00th=[ 5932], 90.00th=[ 8586], 95.00th=[10945], | 99.00th=[15533], 99.50th=[16909], 99.90th=[25035], 99.95th=[55837], | 99.99th=[65274] bw ( KiB/s): min=135168, max=221184, per=100.00%, avg=158486.33, stdev=12636.23, samples=60 iops : min= 132, max= 216, avg=154.77, stdev=12.34, samples=60 lat (usec) : 50=0.01%, 100=0.12%, 250=78.57%, 500=0.69%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=5.02%, 10=13.85%, 20=1.62%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=2.00%, sys=3.44%, ctx=22987, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18326,4644,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=611MiB/s (640MB/s), 611MiB/s-611MiB/s (640MB/s-640MB/s), io=17.9GiB (19.2GB), run=30005-30005msec WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=4644MiB (4870MB), run=30005-30005msec Disk stats (read/write): vde: ios=18229/4610, merge=0/0, ticks=1050/22499, in_queue=28538, util=95.50% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960089: Fri Jan 7 14:23:55 2022 read: IOPS=604, BW=605MiB/s (634MB/s)(17.7GiB/30003msec) clat (usec): min=131, max=27282, avg=236.41, stdev=1013.78 lat (usec): min=131, max=27282, avg=236.56, stdev=1013.78 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 147], 10.00th=[ 149], 20.00th=[ 151], | 30.00th=[ 155], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 169], | 70.00th=[ 182], 80.00th=[ 196], 90.00th=[ 212], 95.00th=[ 233], | 99.00th=[ 318], 99.50th=[ 486], 99.90th=[18220], 99.95th=[19268], | 99.99th=[25297] bw ( KiB/s): min=460800, max=786432, per=100.00%, avg=619191.37, stdev=78391.76, samples=60 iops : min= 450, max= 768, avg=604.67, stdev=76.54, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4583MiB/30003msec) clat (usec): min=2554, max=63354, avg=5503.47, stdev=3158.06 lat (usec): min=2663, max=63455, avg=5601.44, stdev=3156.83 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3458], | 30.00th=[ 4359], 40.00th=[ 4752], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5997], 90.00th=[ 8979], 95.00th=[11469], | 99.00th=[16188], 99.50th=[17695], 99.90th=[50594], 99.95th=[55837], | 99.99th=[63177] bw ( KiB/s): min=133120, max=202752, per=100.00%, avg=156409.45, stdev=11294.34, samples=60 iops : min= 130, max= 198, avg=152.73, stdev=11.03, samples=60 lat (usec) : 250=77.38%, 500=2.06%, 750=0.05% lat (msec) : 2=0.01%, 4=5.14%, 10=13.50%, 20=1.81%, 50=0.04% lat (msec) : 100=0.02% cpu : usr=1.89%, sys=2.53%, ctx=22736, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18139,4583,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=605MiB/s (634MB/s), 605MiB/s-605MiB/s (634MB/s-634MB/s), io=17.7GiB (19.0GB), run=30003-30003msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4583MiB (4806MB), run=30003-30003msec Disk stats (read/write): vde: ios=18013/4548, merge=0/0, ticks=1112/22464, in_queue=28398, util=95.08% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961056: Fri Jan 7 14:24:32 2022 read: IOPS=567, BW=567MiB/s (595MB/s)(16.7GiB/30074msec) slat (usec): min=9, max=130, avg=23.41, stdev= 9.64 clat (usec): min=43, max=89014, avg=338.63, stdev=2640.22 lat (usec): min=110, max=89034, avg=362.21, stdev=2640.39 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 110], 20.00th=[ 113], | 30.00th=[ 116], 40.00th=[ 119], 50.00th=[ 124], 60.00th=[ 131], | 70.00th=[ 145], 80.00th=[ 326], 90.00th=[ 347], 95.00th=[ 367], | 99.00th=[ 750], 99.50th=[ 2638], 99.90th=[50070], 99.95th=[59507], | 99.99th=[67634] bw ( KiB/s): min=417792, max=942080, per=100.00%, avg=582350.83, stdev=99150.28, samples=60 iops : min= 408, max= 920, avg=568.70, stdev=96.83, samples=60 write: IOPS=143, BW=144MiB/s (151MB/s)(4330MiB/30074msec) slat (usec): min=60, max=572, avg=168.05, stdev=85.92 clat (msec): min=5, max=174, avg=67.83, stdev=28.51 lat (msec): min=5, max=174, avg=68.00, stdev=28.51 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 18], 10.00th=[ 25], 20.00th=[ 41], | 30.00th=[ 56], 40.00th=[ 65], 50.00th=[ 71], 60.00th=[ 77], | 70.00th=[ 84], 80.00th=[ 90], 90.00th=[ 102], 95.00th=[ 112], | 99.00th=[ 140], 99.50th=[ 148], 99.90th=[ 165], 99.95th=[ 171], | 99.99th=[ 176] bw ( KiB/s): min=125178, max=235520, per=100.00%, avg=147464.88, stdev=13241.77, samples=60 iops : min= 122, max= 230, avg=144.00, stdev=12.94, samples=60 lat (usec) : 50=0.01%, 100=0.15%, 250=61.95%, 500=16.55%, 750=0.30% lat (usec) : 1000=0.15% lat (msec) : 2=0.11%, 4=0.20%, 10=0.04%, 20=1.25%, 50=4.13% lat (msec) : 100=12.98%, 250=2.19% cpu : usr=2.67%, sys=2.05%, ctx=20975, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17060,4330,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=567MiB/s (595MB/s), 567MiB/s-567MiB/s (595MB/s-595MB/s), io=16.7GiB (17.9GB), run=30074-30074msec WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=4330MiB (4540MB), run=30074-30074msec Disk stats (read/write): vde: ios=17060/4327, merge=0/0, ticks=2757/291055, in_queue=299038, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961579: Fri Jan 7 14:25:09 2022 read: IOPS=608, BW=609MiB/s (638MB/s)(17.8GiB/30001msec) clat (usec): min=129, max=21287, avg=224.46, stdev=945.38 lat (usec): min=129, max=21288, avg=224.61, stdev=945.39 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 153], 50.00th=[ 157], 60.00th=[ 161], | 70.00th=[ 172], 80.00th=[ 186], 90.00th=[ 200], 95.00th=[ 212], | 99.00th=[ 273], 99.50th=[ 420], 99.90th=[17171], 99.95th=[18744], | 99.99th=[21365] bw ( KiB/s): min=432128, max=776192, per=100.00%, avg=623567.57, stdev=79796.96, samples=60 iops : min= 422, max= 758, avg=608.95, stdev=77.93, samples=60 write: IOPS=154, BW=154MiB/s (162MB/s)(4626MiB/30001msec) clat (usec): min=2527, max=59381, avg=5491.75, stdev=2984.30 lat (usec): min=2623, max=59479, avg=5589.91, stdev=2982.75 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3523], | 30.00th=[ 4424], 40.00th=[ 4817], 50.00th=[ 5145], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5997], 90.00th=[ 8848], 95.00th=[11207], | 99.00th=[15533], 99.50th=[16909], 99.90th=[21627], 99.95th=[55837], | 99.99th=[59507] bw ( KiB/s): min=141312, max=210944, per=99.99%, avg=157871.72, stdev=10842.55, samples=60 iops : min= 138, max= 206, avg=154.17, stdev=10.59, samples=60 lat (usec) : 250=78.71%, 500=0.74%, 1000=0.01% lat (msec) : 2=0.01%, 4=5.01%, 10=13.76%, 20=1.72%, 50=0.03% lat (msec) : 100=0.02% cpu : usr=1.80%, sys=2.51%, ctx=22905, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18268,4626,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=609MiB/s (638MB/s), 609MiB/s-609MiB/s (638MB/s-638MB/s), io=17.8GiB (19.2GB), run=30001-30001msec WRITE: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=4626MiB (4851MB), run=30001-30001msec Disk stats (read/write): vde: ios=18264/4623, merge=0/0, ticks=1051/22791, in_queue=28630, util=95.11% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962984: Fri Jan 7 14:25:46 2022 read: IOPS=597, BW=597MiB/s (626MB/s)(17.7GiB/30311msec) slat (usec): min=9, max=321, avg=20.69, stdev=11.43 clat (usec): min=62, max=579567, avg=1263.65, stdev=12722.73 lat (usec): min=115, max=579584, avg=1284.52, stdev=12722.75 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 120], 50.00th=[ 126], 60.00th=[ 141], | 70.00th=[ 235], 80.00th=[ 343], 90.00th=[ 465], 95.00th=[ 3392], | 99.00th=[ 12911], 99.50th=[ 16450], 99.90th=[179307], 99.95th=[242222], | 99.99th=[566232] bw ( KiB/s): min=309248, max=1130496, per=100.00%, avg=617676.80, stdev=144934.81, samples=60 iops : min= 302, max= 1104, avg=603.20, stdev=141.54, samples=60 write: IOPS=150, BW=151MiB/s (158MB/s)(4569MiB/30311msec) slat (usec): min=62, max=535, avg=177.67, stdev=91.10 clat (msec): min=11, max=2152, avg=326.33, stdev=181.09 lat (msec): min=11, max=2152, avg=326.51, stdev=181.08 clat percentiles (msec): | 1.00th=[ 79], 5.00th=[ 121], 10.00th=[ 140], 20.00th=[ 167], | 30.00th=[ 203], 40.00th=[ 245], 50.00th=[ 296], 60.00th=[ 355], | 70.00th=[ 409], 80.00th=[ 460], 90.00th=[ 550], 95.00th=[ 634], | 99.00th=[ 852], 99.50th=[ 944], 99.90th=[ 2072], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min=90112, max=225280, per=99.93%, avg=154248.53, stdev=26128.05, samples=60 iops : min= 88, max= 220, avg=150.63, stdev=25.52, samples=60 lat (usec) : 100=0.11%, 250=56.17%, 500=15.90%, 750=1.22%, 1000=0.63% lat (msec) : 2=1.20%, 4=0.92%, 10=2.05%, 20=1.32%, 50=0.04% lat (msec) : 100=0.55%, 250=7.95%, 500=9.01%, 750=2.54%, 1000=0.31% lat (msec) : 2000=0.05%, >=2000=0.03% cpu : usr=2.86%, sys=2.04%, ctx=20222, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=18096,4569,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=597MiB/s (626MB/s), 597MiB/s-597MiB/s (626MB/s-626MB/s), io=17.7GiB (18.0GB), run=30311-30311msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4569MiB (4791MB), run=30311-30311msec Disk stats (read/write): vde: ios=18096/4555, merge=0/0, ticks=18098/1476379, in_queue=1504604, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963563: Fri Jan 7 14:26:23 2022 read: IOPS=604, BW=604MiB/s (633MB/s)(17.7GiB/30001msec) clat (usec): min=132, max=26213, avg=230.76, stdev=967.32 lat (usec): min=132, max=26213, avg=230.92, stdev=967.32 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 161], 60.00th=[ 167], | 70.00th=[ 178], 80.00th=[ 192], 90.00th=[ 208], 95.00th=[ 223], | 99.00th=[ 302], 99.50th=[ 537], 99.90th=[17171], 99.95th=[18482], | 99.99th=[25560] bw ( KiB/s): min=432128, max=792576, per=100.00%, avg=618496.00, stdev=80429.45, samples=60 iops : min= 422, max= 774, avg=604.03, stdev=78.59, samples=60 write: IOPS=152, BW=153MiB/s (160MB/s)(4577MiB/30001msec) clat (usec): min=2535, max=64805, avg=5534.60, stdev=3244.20 lat (usec): min=2639, max=64880, avg=5631.93, stdev=3243.25 clat percentiles (usec): | 1.00th=[ 2638], 5.00th=[ 2671], 10.00th=[ 2704], 20.00th=[ 3425], | 30.00th=[ 4293], 40.00th=[ 4817], 50.00th=[ 5080], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5997], 90.00th=[ 9110], 95.00th=[11469], | 99.00th=[16319], 99.50th=[17957], 99.90th=[55313], 99.95th=[55837], | 99.99th=[64750] bw ( KiB/s): min=135168, max=221184, per=99.98%, avg=156194.13, stdev=12327.66, samples=60 iops : min= 132, max= 216, avg=152.53, stdev=12.11, samples=60 lat (usec) : 250=78.03%, 500=1.39%, 750=0.03%, 1000=0.01% lat (msec) : 2=0.03%, 4=5.18%, 10=13.40%, 20=1.85%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.96%, sys=2.35%, ctx=22713, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18121,4577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=604MiB/s (633MB/s), 604MiB/s-604MiB/s (633MB/s-633MB/s), io=17.7GiB (19.0GB), run=30001-30001msec WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=4577MiB (4799MB), run=30001-30001msec Disk stats (read/write): vde: ios=18111/4574, merge=0/0, ticks=1063/22776, in_queue=28631, util=94.99% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964382: Fri Jan 7 14:27:00 2022 read: IOPS=595, BW=595MiB/s (624MB/s)(17.9GiB/30698msec) slat (usec): min=9, max=790, avg=19.22, stdev=19.22 clat (usec): min=58, max=874089, avg=3159.11, stdev=18188.47 lat (usec): min=114, max=874107, avg=3178.48, stdev=18188.87 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 117], 40.00th=[ 121], 50.00th=[ 131], 60.00th=[ 212], | 70.00th=[ 338], 80.00th=[ 519], 90.00th=[ 7898], 95.00th=[ 20317], | 99.00th=[ 35390], 99.50th=[ 42206], 99.90th=[210764], 99.95th=[354419], | 99.99th=[859833] bw ( KiB/s): min=28672, max=1347584, per=100.00%, avg=624029.95, stdev=225500.74, samples=60 iops : min= 28, max= 1316, avg=609.37, stdev=220.21, samples=60 write: IOPS=150, BW=151MiB/s (158MB/s)(4630MiB/30698msec) slat (usec): min=62, max=1042, avg=193.99, stdev=104.61 clat (msec): min=29, max=2715, avg=649.99, stdev=274.43 lat (msec): min=29, max=2715, avg=650.19, stdev=274.42 clat percentiles (msec): | 1.00th=[ 132], 5.00th=[ 300], 10.00th=[ 372], 20.00th=[ 464], | 30.00th=[ 506], 40.00th=[ 550], 50.00th=[ 609], 60.00th=[ 667], | 70.00th=[ 743], 80.00th=[ 810], 90.00th=[ 961], 95.00th=[ 1133], | 99.00th=[ 1418], 99.50th=[ 1871], 99.90th=[ 2702], 99.95th=[ 2702], | 99.99th=[ 2702] bw ( KiB/s): min=10240, max=260096, per=100.00%, avg=154688.00, stdev=48494.14, samples=60 iops : min= 10, max= 254, avg=151.03, stdev=47.34, samples=60 lat (usec) : 100=0.07%, 250=48.91%, 500=14.70%, 750=1.73%, 1000=1.04% lat (msec) : 2=1.88%, 4=1.74%, 10=2.51%, 20=3.15%, 50=3.75% lat (msec) : 100=0.09%, 250=0.86%, 500=5.16%, 750=8.47%, 1000=4.18% lat (msec) : 2000=1.66%, >=2000=0.10% cpu : usr=3.21%, sys=1.70%, ctx=17801, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18279,4630,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=595MiB/s (624MB/s), 595MiB/s-595MiB/s (624MB/s-624MB/s), io=17.9GiB (19.2GB), run=30698-30698msec WRITE: bw=151MiB/s (158MB/s), 151MiB/s-151MiB/s (158MB/s-158MB/s), io=4630MiB (4855MB), run=30698-30698msec Disk stats (read/write): vde: ios=18279/4606, merge=0/0, ticks=48696/2951065, in_queue=3022399, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965448: Fri Jan 7 14:27:37 2022 read: IOPS=615, BW=616MiB/s (645MB/s)(18.0GiB/30001msec) clat (usec): min=113, max=27790, avg=207.85, stdev=966.88 lat (usec): min=113, max=27790, avg=208.00, stdev=966.88 clat percentiles (usec): | 1.00th=[ 120], 5.00th=[ 126], 10.00th=[ 128], 20.00th=[ 130], | 30.00th=[ 133], 40.00th=[ 137], 50.00th=[ 139], 60.00th=[ 145], | 70.00th=[ 153], 80.00th=[ 169], 90.00th=[ 184], 95.00th=[ 198], | 99.00th=[ 277], 99.50th=[ 400], 99.90th=[17957], 99.95th=[19530], | 99.99th=[26084] bw ( KiB/s): min=466944, max=845824, per=100.00%, avg=630943.02, stdev=87279.19, samples=59 iops : min= 456, max= 826, avg=616.15, stdev=85.23, samples=59 write: IOPS=155, BW=156MiB/s (163MB/s)(4673MiB/30001msec) clat (usec): min=2503, max=64294, avg=5492.63, stdev=3024.10 lat (usec): min=2591, max=64418, avg=5589.46, stdev=3022.46 clat percentiles (usec): | 1.00th=[ 2606], 5.00th=[ 2671], 10.00th=[ 2737], 20.00th=[ 3752], | 30.00th=[ 4555], 40.00th=[ 4883], 50.00th=[ 5145], 60.00th=[ 5342], | 70.00th=[ 5604], 80.00th=[ 5932], 90.00th=[ 8029], 95.00th=[10945], | 99.00th=[15926], 99.50th=[16909], 99.90th=[55313], 99.95th=[56886], | 99.99th=[64226] bw ( KiB/s): min=141312, max=215040, per=100.00%, avg=159576.34, stdev=11865.89, samples=59 iops : min= 138, max= 210, avg=155.83, stdev=11.58, samples=59 lat (usec) : 250=78.71%, 500=0.73%, 750=0.01% lat (msec) : 2=0.01%, 4=4.44%, 10=14.42%, 20=1.60%, 50=0.05% lat (msec) : 100=0.02% cpu : usr=1.97%, sys=1.40%, ctx=23150, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18467,4673,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=616MiB/s (645MB/s), 616MiB/s-616MiB/s (645MB/s-645MB/s), io=18.0GiB (19.4GB), run=30001-30001msec WRITE: bw=156MiB/s (163MB/s), 156MiB/s-156MiB/s (163MB/s-163MB/s), io=4673MiB (4900MB), run=30001-30001msec Disk stats (read/write): vde: ios=18421/4662, merge=0/0, ticks=1053/23046, in_queue=28881, util=95.83% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966067: Fri Jan 7 14:28:15 2022 read: IOPS=601, BW=602MiB/s (631MB/s)(18.1GiB/30782msec) slat (usec): min=11, max=727868, avg=1215.15, stdev=11925.42 clat (msec): min=17, max=1081, avg=109.94, stdev=112.02 lat (msec): min=17, max=1081, avg=111.16, stdev=112.76 clat percentiles (msec): | 1.00th=[ 25], 5.00th=[ 30], 10.00th=[ 38], 20.00th=[ 56], | 30.00th=[ 69], 40.00th=[ 81], 50.00th=[ 92], 60.00th=[ 103], | 70.00th=[ 116], 80.00th=[ 132], 90.00th=[ 157], 95.00th=[ 194], | 99.00th=[ 793], 99.50th=[ 844], 99.90th=[ 894], 99.95th=[ 902], | 99.99th=[ 1003] bw ( KiB/s): min= 2048, max=1470464, per=100.00%, avg=619440.15, stdev=247781.21, samples=61 iops : min= 2, max= 1436, avg=604.90, stdev=241.98, samples=61 write: IOPS=152, BW=152MiB/s (159MB/s)(4682MiB/30782msec) slat (usec): min=272, max=722641, avg=1592.47, stdev=12930.76 clat (msec): min=42, max=2822, avg=872.59, stdev=321.83 lat (msec): min=42, max=2826, avg=874.19, stdev=321.36 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 418], 10.00th=[ 535], 20.00th=[ 642], | 30.00th=[ 701], 40.00th=[ 776], 50.00th=[ 827], 60.00th=[ 902], | 70.00th=[ 978], 80.00th=[ 1062], 90.00th=[ 1267], 95.00th=[ 1469], | 99.00th=[ 1888], 99.50th=[ 2140], 99.90th=[ 2366], 99.95th=[ 2366], | 99.99th=[ 2836] bw ( KiB/s): min=10240, max=294912, per=99.72%, avg=155313.90, stdev=60307.35, samples=60 iops : min= 10, max= 288, avg=151.67, stdev=58.90, samples=60 lat (msec) : 20=0.03%, 50=13.47%, 100=33.13%, 250=30.64%, 500=2.44% lat (msec) : 750=5.98%, 1000=9.04%, 2000=5.12%, >=2000=0.16% cpu : usr=5.25%, sys=1.47%, ctx=2235, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.7% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=18516,4682,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=602MiB/s (631MB/s), 602MiB/s-602MiB/s (631MB/s-631MB/s), io=18.1GiB (19.4GB), run=30782-30782msec WRITE: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=4682MiB (4909MB), run=30782-30782msec Disk stats (read/write): vde: ios=18516/4663, merge=0/0, ticks=123699/3594596, in_queue=3744928, util=99.30% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966918: Fri Jan 7 14:28:51 2022 read: IOPS=257, BW=515MiB/s (540MB/s)(15.1GiB/30009msec) clat (usec): min=261, max=25415, avg=481.66, stdev=1070.69 lat (usec): min=261, max=25415, avg=481.84, stdev=1070.69 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 277], 10.00th=[ 281], 20.00th=[ 289], | 30.00th=[ 293], 40.00th=[ 302], 50.00th=[ 310], 60.00th=[ 326], | 70.00th=[ 355], 80.00th=[ 619], 90.00th=[ 734], 95.00th=[ 766], | 99.00th=[ 906], 99.50th=[10552], 99.90th=[15926], 99.95th=[17695], | 99.99th=[25297] bw ( KiB/s): min=331776, max=847872, per=100.00%, avg=527463.40, stdev=109074.10, samples=60 iops : min= 162, max= 414, avg=257.53, stdev=53.27, samples=60 write: IOPS=65, BW=131MiB/s (137MB/s)(3930MiB/30009msec) clat (usec): min=6073, max=54154, avg=13088.23, stdev=3506.98 lat (usec): min=6288, max=54361, avg=13365.49, stdev=3523.94 clat percentiles (usec): | 1.00th=[ 6849], 5.00th=[ 8586], 10.00th=[ 9110], 20.00th=[10028], | 30.00th=[10945], 40.00th=[11731], 50.00th=[13042], 60.00th=[14222], | 70.00th=[15008], 80.00th=[15926], 90.00th=[16581], 95.00th=[17171], | 99.00th=[23200], 99.50th=[24773], 99.90th=[49546], 99.95th=[54264], | 99.99th=[54264] bw ( KiB/s): min=110592, max=155648, per=99.97%, avg=134067.00, stdev=10311.23, samples=60 iops : min= 54, max= 76, avg=65.43, stdev= 5.05, samples=60 lat (usec) : 500=62.33%, 750=11.76%, 1000=4.98% lat (msec) : 2=0.09%, 10=4.08%, 20=16.33%, 50=0.41%, 100=0.01% cpu : usr=2.02%, sys=1.90%, ctx=9706, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7727,1965,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=515MiB/s (540MB/s), 515MiB/s-515MiB/s (540MB/s-540MB/s), io=15.1GiB (16.2GB), run=30009-30009msec WRITE: bw=131MiB/s (137MB/s), 131MiB/s-131MiB/s (137MB/s-137MB/s), io=3930MiB (4121MB), run=30009-30009msec Disk stats (read/write): vde: ios=15454/3928, merge=0/0, ticks=700/49082, in_queue=56739, util=95.61% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967509: Fri Jan 7 14:29:28 2022 read: IOPS=265, BW=531MiB/s (557MB/s)(15.6GiB/30006msec) slat (usec): min=54, max=248, avg=76.71, stdev=15.67 clat (usec): min=189, max=17889, avg=388.55, stdev=989.61 lat (usec): min=271, max=17956, avg=465.52, stdev=990.65 clat percentiles (usec): | 1.00th=[ 219], 5.00th=[ 225], 10.00th=[ 227], 20.00th=[ 233], | 30.00th=[ 237], 40.00th=[ 243], 50.00th=[ 249], 60.00th=[ 262], | 70.00th=[ 293], 80.00th=[ 355], 90.00th=[ 570], 95.00th=[ 652], | 99.00th=[ 832], 99.50th=[10028], 99.90th=[15401], 99.95th=[16319], | 99.99th=[17957] bw ( KiB/s): min=320128, max=950272, per=100.00%, avg=543915.82, stdev=109686.12, samples=60 iops : min= 156, max= 464, avg=265.57, stdev=53.55, samples=60 write: IOPS=67, BW=135MiB/s (141MB/s)(4048MiB/30006msec) slat (usec): min=141, max=962, avg=301.14, stdev=110.77 clat (usec): min=4657, max=55502, avg=12680.13, stdev=3720.19 lat (usec): min=4868, max=55833, avg=12981.61, stdev=3736.72 clat percentiles (usec): | 1.00th=[ 5014], 5.00th=[ 6980], 10.00th=[ 8094], 20.00th=[ 9503], | 30.00th=[10552], 40.00th=[11338], 50.00th=[12387], 60.00th=[14222], | 70.00th=[15139], 80.00th=[15926], 90.00th=[16581], 95.00th=[17171], | 99.00th=[21365], 99.50th=[24511], 99.90th=[31589], 99.95th=[45351], | 99.99th=[55313] bw ( KiB/s): min=118784, max=159744, per=99.98%, avg=138113.17, stdev=9060.64, samples=60 iops : min= 58, max= 78, avg=67.43, stdev= 4.42, samples=60 lat (usec) : 250=40.56%, 500=29.23%, 750=8.65%, 1000=0.73% lat (msec) : 2=0.04%, 10=5.12%, 20=15.40%, 50=0.26%, 100=0.01% cpu : usr=1.90%, sys=2.68%, ctx=10004, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7967,2024,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=531MiB/s (557MB/s), 531MiB/s-531MiB/s (557MB/s-557MB/s), io=15.6GiB (16.7GB), run=30006-30006msec WRITE: bw=135MiB/s (141MB/s), 135MiB/s-135MiB/s (141MB/s-141MB/s), io=4048MiB (4245MB), run=30006-30006msec Disk stats (read/write): vde: ios=15846/4018, merge=0/0, ticks=674/48942, in_queue=56606, util=95.96% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968311: Fri Jan 7 14:30:05 2022 read: IOPS=268, BW=537MiB/s (564MB/s)(15.7GiB/30001msec) clat (usec): min=223, max=18547, avg=381.67, stdev=1012.06 lat (usec): min=223, max=18547, avg=381.84, stdev=1012.06 clat percentiles (usec): | 1.00th=[ 231], 5.00th=[ 237], 10.00th=[ 241], 20.00th=[ 245], | 30.00th=[ 249], 40.00th=[ 255], 50.00th=[ 260], 60.00th=[ 269], | 70.00th=[ 289], 80.00th=[ 338], 90.00th=[ 433], 95.00th=[ 578], | 99.00th=[ 799], 99.50th=[ 9896], 99.90th=[15533], 99.95th=[16712], | 99.99th=[18482] bw ( KiB/s): min=339968, max=806912, per=99.95%, avg=550084.12, stdev=115041.55, samples=59 iops : min= 166, max= 394, avg=268.58, stdev=56.16, samples=59 write: IOPS=68, BW=136MiB/s (143MB/s)(4086MiB/30001msec) clat (usec): min=4672, max=48117, avg=12947.50, stdev=3773.24 lat (usec): min=4862, max=48355, avg=13167.34, stdev=3777.19 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 7308], 10.00th=[ 8455], 20.00th=[ 9765], | 30.00th=[10814], 40.00th=[11469], 50.00th=[12649], 60.00th=[14353], | 70.00th=[15401], 80.00th=[16188], 90.00th=[16909], 95.00th=[17171], | 99.00th=[22152], 99.50th=[26608], 99.90th=[45351], 99.95th=[46924], | 99.99th=[47973] bw ( KiB/s): min=122880, max=159744, per=99.96%, avg=139411.88, stdev=9308.67, samples=59 iops : min= 60, max= 78, avg=68.07, stdev= 4.55, samples=59 lat (usec) : 250=24.39%, 500=49.33%, 750=5.06%, 1000=0.44% lat (msec) : 2=0.03%, 4=0.01%, 10=4.62%, 20=15.85%, 50=0.27% cpu : usr=1.64%, sys=1.45%, ctx=10113, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8062,2043,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=537MiB/s (564MB/s), 537MiB/s-537MiB/s (564MB/s-564MB/s), io=15.7GiB (16.9GB), run=30001-30001msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4086MiB (4284MB), run=30001-30001msec Disk stats (read/write): vde: ios=15962/4056, merge=0/0, ticks=691/41416, in_queue=48166, util=96.53% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969571: Fri Jan 7 14:30:42 2022 read: IOPS=363, BW=726MiB/s (761MB/s)(21.3GiB/30095msec) slat (usec): min=22, max=221, avg=38.10, stdev=14.98 clat (usec): min=91, max=1329.6k, avg=1837.32, stdev=32103.70 lat (usec): min=204, max=1329.6k, avg=1875.62, stdev=32103.64 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 188], 10.00th=[ 192], | 20.00th=[ 196], 30.00th=[ 200], 40.00th=[ 206], | 50.00th=[ 212], 60.00th=[ 225], 70.00th=[ 247], | 80.00th=[ 519], 90.00th=[ 660], 95.00th=[ 693], | 99.00th=[ 889], 99.50th=[ 4621], 99.90th=[ 541066], | 99.95th=[ 868221], 99.99th=[1052771] bw ( KiB/s): min=368226, max=1148372, per=100.00%, avg=753597.05, stdev=131621.41, samples=60 iops : min= 179, max= 560, avg=367.52, stdev=64.32, samples=60 write: IOPS=92, BW=184MiB/s (193MB/s)(5540MiB/30095msec) slat (usec): min=120, max=998, avg=334.21, stdev=173.38 clat (msec): min=8, max=2509, avg=100.87, stdev=128.19 lat (msec): min=8, max=2509, avg=101.20, stdev=128.20 clat percentiles (msec): | 1.00th=[ 16], 5.00th=[ 20], 10.00th=[ 22], 20.00th=[ 32], | 30.00th=[ 42], 40.00th=[ 53], 50.00th=[ 69], 60.00th=[ 88], | 70.00th=[ 113], 80.00th=[ 146], 90.00th=[ 213], 95.00th=[ 279], | 99.00th=[ 451], 99.50th=[ 542], 99.90th=[ 2500], 99.95th=[ 2500], | 99.99th=[ 2500] bw ( KiB/s): min=164828, max=206451, per=100.00%, avg=190546.80, stdev=7530.14, samples=60 iops : min= 80, max= 100, avg=92.50, stdev= 3.70, samples=60 lat (usec) : 100=0.01%, 250=56.37%, 500=7.37%, 750=14.42%, 1000=0.96% lat (msec) : 2=0.13%, 4=0.05%, 10=0.15%, 20=1.37%, 50=6.30% lat (msec) : 100=5.62%, 250=5.69%, 500=1.29%, 750=0.15%, 1000=0.07% lat (msec) : 2000=0.01%, >=2000=0.03% cpu : usr=3.04%, sys=2.09%, ctx=13680, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10926,2770,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=726MiB/s (761MB/s), 726MiB/s-726MiB/s (761MB/s-761MB/s), io=21.3GiB (22.9GB), run=30095-30095msec WRITE: bw=184MiB/s (193MB/s), 184MiB/s-184MiB/s (193MB/s-193MB/s), io=5540MiB (5809MB), run=30095-30095msec Disk stats (read/write): vde: ios=21848/5524, merge=0/0, ticks=16667/552719, in_queue=581423, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970158: Fri Jan 7 14:31:19 2022 read: IOPS=268, BW=538MiB/s (564MB/s)(15.8GiB/30017msec) clat (usec): min=235, max=18422, avg=380.64, stdev=1018.96 lat (usec): min=235, max=18422, avg=380.80, stdev=1018.96 clat percentiles (usec): | 1.00th=[ 243], 5.00th=[ 249], 10.00th=[ 251], 20.00th=[ 255], | 30.00th=[ 260], 40.00th=[ 265], 50.00th=[ 269], 60.00th=[ 277], | 70.00th=[ 293], 80.00th=[ 343], 90.00th=[ 388], 95.00th=[ 457], | 99.00th=[ 758], 99.50th=[10814], 99.90th=[15795], 99.95th=[16450], | 99.99th=[18482] bw ( KiB/s): min=339968, max=847872, per=100.00%, avg=550817.45, stdev=116184.20, samples=60 iops : min= 166, max= 414, avg=268.93, stdev=56.72, samples=60 write: IOPS=68, BW=136MiB/s (143MB/s)(4092MiB/30017msec) clat (usec): min=4665, max=49116, avg=12950.12, stdev=3682.86 lat (usec): min=4872, max=49295, avg=13159.60, stdev=3681.95 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 7308], 10.00th=[ 8356], 20.00th=[ 9896], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12649], 60.00th=[14484], | 70.00th=[15533], 80.00th=[16319], 90.00th=[16909], 95.00th=[17171], | 99.00th=[20317], 99.50th=[24773], 99.90th=[41681], 99.95th=[43254], | 99.99th=[49021] bw ( KiB/s): min=118784, max=163840, per=100.00%, avg=139614.35, stdev=10234.28, samples=60 iops : min= 58, max= 80, avg=68.17, stdev= 5.00, samples=60 lat (usec) : 250=6.17%, 500=70.59%, 750=2.19%, 1000=0.28% lat (msec) : 2=0.01%, 4=0.01%, 10=4.43%, 20=16.11%, 50=0.22% cpu : usr=1.61%, sys=1.65%, ctx=10123, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8068,2046,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=538MiB/s (564MB/s), 538MiB/s-538MiB/s (564MB/s-564MB/s), io=15.8GiB (16.9GB), run=30017-30017msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4092MiB (4291MB), run=30017-30017msec Disk stats (read/write): vde: ios=15976/4058, merge=0/0, ticks=695/42173, in_queue=48860, util=96.35% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971116: Fri Jan 7 14:31:56 2022 read: IOPS=365, BW=730MiB/s (766MB/s)(21.7GiB/30487msec) slat (usec): min=19, max=942, avg=33.73, stdev=18.88 clat (usec): min=154, max=2747.1k, avg=7965.95, stdev=108507.58 lat (usec): min=205, max=2747.1k, avg=7999.87, stdev=108507.70 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 188], 10.00th=[ 192], | 20.00th=[ 196], 30.00th=[ 200], 40.00th=[ 206], | 50.00th=[ 217], 60.00th=[ 255], 70.00th=[ 627], | 80.00th=[ 660], 90.00th=[ 1876], 95.00th=[ 12125], | 99.00th=[ 29492], 99.50th=[ 261096], 99.90th=[2499806], | 99.95th=[2566915], 99.99th=[2600469] bw ( KiB/s): min=434176, max=1200128, per=100.00%, avg=762421.15, stdev=157015.83, samples=60 iops : min= 212, max= 586, avg=372.13, stdev=76.66, samples=60 write: IOPS=92, BW=186MiB/s (195MB/s)(5656MiB/30487msec) slat (usec): min=119, max=1590, avg=379.43, stdev=201.07 clat (msec): min=66, max=3322, avg=506.88, stdev=230.67 lat (msec): min=67, max=3323, avg=507.26, stdev=230.68 clat percentiles (msec): | 1.00th=[ 203], 5.00th=[ 309], 10.00th=[ 347], 20.00th=[ 372], | 30.00th=[ 388], 40.00th=[ 414], 50.00th=[ 443], 60.00th=[ 481], | 70.00th=[ 531], 80.00th=[ 625], 90.00th=[ 760], 95.00th=[ 877], | 99.00th=[ 1284], 99.50th=[ 1586], 99.90th=[ 3171], 99.95th=[ 3272], | 99.99th=[ 3339] bw ( KiB/s): min=104276, max=237568, per=100.00%, avg=190610.33, stdev=16929.27, samples=60 iops : min= 50, max= 116, avg=92.95, stdev= 8.35, samples=60 lat (usec) : 250=47.48%, 500=7.28%, 750=14.14%, 1000=1.74% lat (msec) : 2=1.20%, 4=1.02%, 10=2.21%, 20=2.84%, 50=1.29% lat (msec) : 100=0.09%, 250=0.41%, 500=12.73%, 750=5.26%, 1000=1.55% lat (msec) : 2000=0.60%, >=2000=0.17% cpu : usr=3.62%, sys=1.74%, ctx=12294, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=11135,2828,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=730MiB/s (766MB/s), 730MiB/s-730MiB/s (766MB/s-766MB/s), io=21.7GiB (23.4GB), run=30487-30487msec WRITE: bw=186MiB/s (195MB/s), 186MiB/s-186MiB/s (195MB/s-195MB/s), io=5656MiB (5931MB), run=30487-30487msec Disk stats (read/write): vde: ios=22270/5654, merge=0/0, ticks=95488/2846733, in_queue=2953600, util=99.51% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972161: Fri Jan 7 14:32:33 2022 read: IOPS=268, BW=537MiB/s (563MB/s)(15.7GiB/30015msec) clat (usec): min=245, max=34303, avg=397.62, stdev=1093.95 lat (usec): min=245, max=34303, avg=397.79, stdev=1093.95 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 258], 10.00th=[ 262], 20.00th=[ 269], | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 293], | 70.00th=[ 314], 80.00th=[ 355], 90.00th=[ 404], 95.00th=[ 482], | 99.00th=[ 775], 99.50th=[10683], 99.90th=[15795], 99.95th=[17171], | 99.99th=[34341] bw ( KiB/s): min=319488, max=864256, per=100.00%, avg=549751.47, stdev=117744.30, samples=60 iops : min= 156, max= 422, avg=268.43, stdev=57.49, samples=60 write: IOPS=68, BW=136MiB/s (143MB/s)(4086MiB/30015msec) clat (usec): min=4678, max=51848, avg=12906.63, stdev=3688.06 lat (usec): min=4865, max=51987, avg=13113.61, stdev=3691.52 clat percentiles (usec): | 1.00th=[ 5276], 5.00th=[ 7242], 10.00th=[ 8356], 20.00th=[ 9765], | 30.00th=[10814], 40.00th=[11600], 50.00th=[12649], 60.00th=[14353], | 70.00th=[15401], 80.00th=[16188], 90.00th=[16909], 95.00th=[17171], | 99.00th=[20317], 99.50th=[25035], 99.90th=[43254], 99.95th=[46924], | 99.99th=[51643] bw ( KiB/s): min=114688, max=163840, per=100.00%, avg=139400.53, stdev=11031.18, samples=60 iops : min= 56, max= 80, avg=68.07, stdev= 5.39, samples=60 lat (usec) : 250=0.55%, 500=75.81%, 750=2.46%, 1000=0.39% lat (msec) : 2=0.02%, 4=0.01%, 10=4.47%, 20=16.06%, 50=0.23% lat (msec) : 100=0.01% cpu : usr=1.51%, sys=1.87%, ctx=10108, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8053,2043,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=537MiB/s (563MB/s), 537MiB/s-537MiB/s (563MB/s-563MB/s), io=15.7GiB (16.9GB), run=30015-30015msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4086MiB (4284MB), run=30015-30015msec Disk stats (read/write): vde: ios=16046/4072, merge=0/0, ticks=722/48885, in_queue=55736, util=96.03% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972683: Fri Jan 7 14:33:11 2022 read: IOPS=364, BW=728MiB/s (763MB/s)(21.8GiB/30653msec) slat (usec): min=22, max=482114, avg=1977.55, stdev=9969.02 clat (msec): min=17, max=2845, avg=89.37, stdev=116.71 lat (msec): min=17, max=2845, avg=91.35, stdev=117.25 clat percentiles (msec): | 1.00th=[ 24], 5.00th=[ 29], 10.00th=[ 34], 20.00th=[ 46], | 30.00th=[ 57], 40.00th=[ 65], 50.00th=[ 73], 60.00th=[ 82], | 70.00th=[ 92], 80.00th=[ 105], 90.00th=[ 127], 95.00th=[ 155], | 99.00th=[ 514], 99.50th=[ 693], 99.90th=[ 1804], 99.95th=[ 2601], | 99.99th=[ 2635] bw ( KiB/s): min= 4096, max=1572864, per=100.00%, avg=747240.97, stdev=290315.52, samples=61 iops : min= 2, max= 768, avg=364.85, stdev=141.76, samples=61 write: IOPS=92, BW=185MiB/s (194MB/s)(5672MiB/30653msec) slat (usec): min=558, max=471430, avg=2781.52, stdev=13572.73 clat (msec): min=89, max=3519, avg=717.78, stdev=331.26 lat (msec): min=90, max=3521, avg=720.56, stdev=331.31 clat percentiles (msec): | 1.00th=[ 288], 5.00th=[ 405], 10.00th=[ 451], 20.00th=[ 523], | 30.00th=[ 567], 40.00th=[ 600], 50.00th=[ 642], 60.00th=[ 684], | 70.00th=[ 751], 80.00th=[ 860], 90.00th=[ 1036], 95.00th=[ 1217], | 99.00th=[ 2500], 99.50th=[ 2970], 99.90th=[ 3272], 99.95th=[ 3440], | 99.99th=[ 3507] bw ( KiB/s): min=12288, max=319488, per=100.00%, avg=192101.71, stdev=63022.83, samples=59 iops : min= 6, max= 156, avg=93.80, stdev=30.77, samples=59 lat (msec) : 20=0.05%, 50=18.87%, 100=42.51%, 250=16.28%, 500=4.27% lat (msec) : 750=11.73%, 1000=3.79%, 2000=2.19%, >=2000=0.30% cpu : usr=6.32%, sys=1.68%, ctx=2760, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11158,2836,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=728MiB/s (763MB/s), 728MiB/s-728MiB/s (763MB/s-763MB/s), io=21.8GiB (23.4GB), run=30653-30653msec WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5672MiB (5948MB), run=30653-30653msec Disk stats (read/write): vde: ios=22314/5636, merge=0/0, ticks=158198/3546628, in_queue=3746784, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973647: Fri Jan 7 14:33:47 2022 read: IOPS=267, BW=536MiB/s (562MB/s)(15.7GiB/30001msec) clat (usec): min=237, max=24412, avg=397.16, stdev=1039.22 lat (usec): min=237, max=24412, avg=397.34, stdev=1039.23 clat percentiles (usec): | 1.00th=[ 247], 5.00th=[ 253], 10.00th=[ 258], 20.00th=[ 265], | 30.00th=[ 269], 40.00th=[ 273], 50.00th=[ 281], 60.00th=[ 293], | 70.00th=[ 318], 80.00th=[ 363], 90.00th=[ 420], 95.00th=[ 502], | 99.00th=[ 758], 99.50th=[10683], 99.90th=[15664], 99.95th=[17433], | 99.99th=[24511] bw ( KiB/s): min=344064, max=913408, per=99.99%, avg=548303.15, stdev=120410.33, samples=59 iops : min= 168, max= 446, avg=267.69, stdev=58.77, samples=59 write: IOPS=67, BW=136MiB/s (143MB/s)(4078MiB/30001msec) clat (usec): min=4682, max=54307, avg=12928.66, stdev=3831.97 lat (usec): min=4890, max=54484, avg=13137.82, stdev=3834.23 clat percentiles (usec): | 1.00th=[ 5080], 5.00th=[ 7242], 10.00th=[ 8455], 20.00th=[ 9765], | 30.00th=[10814], 40.00th=[11469], 50.00th=[12518], 60.00th=[14353], | 70.00th=[15401], 80.00th=[16188], 90.00th=[16909], 95.00th=[17171], | 99.00th=[22414], 99.50th=[25822], 99.90th=[41681], 99.95th=[43254], | 99.99th=[54264] bw ( KiB/s): min=118784, max=163840, per=99.86%, avg=139001.31, stdev=10603.06, samples=59 iops : min= 58, max= 80, avg=67.86, stdev= 5.17, samples=59 lat (usec) : 250=1.58%, 500=74.11%, 750=3.25%, 1000=0.26% lat (msec) : 2=0.03%, 10=4.49%, 20=16.01%, 50=0.27%, 100=0.01% cpu : usr=1.63%, sys=1.73%, ctx=10080, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8033,2039,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=536MiB/s (562MB/s), 536MiB/s-536MiB/s (562MB/s-562MB/s), io=15.7GiB (16.8GB), run=30001-30001msec WRITE: bw=136MiB/s (143MB/s), 136MiB/s-136MiB/s (143MB/s-143MB/s), io=4078MiB (4276MB), run=30001-30001msec Disk stats (read/write): vde: ios=15952/4050, merge=0/0, ticks=705/42449, in_queue=49266, util=95.64% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974270: Fri Jan 7 14:34:25 2022 read: IOPS=363, BW=727MiB/s (762MB/s)(21.7GiB/30639msec) slat (usec): min=22, max=486739, avg=2064.69, stdev=10955.54 clat (msec): min=87, max=2937, avg=311.74, stdev=156.39 lat (msec): min=87, max=2937, avg=313.81, stdev=156.73 clat percentiles (msec): | 1.00th=[ 102], 5.00th=[ 127], 10.00th=[ 178], 20.00th=[ 236], | 30.00th=[ 259], 40.00th=[ 275], 50.00th=[ 292], 60.00th=[ 309], | 70.00th=[ 330], 80.00th=[ 359], 90.00th=[ 418], 95.00th=[ 600], | 99.00th=[ 802], 99.50th=[ 877], 99.90th=[ 2366], 99.95th=[ 2869], | 99.99th=[ 2937] bw ( KiB/s): min=73728, max=1380352, per=100.00%, avg=752216.17, stdev=254238.88, samples=60 iops : min= 36, max= 674, avg=367.27, stdev=124.13, samples=60 write: IOPS=92, BW=185MiB/s (194MB/s)(5654MiB/30639msec) slat (usec): min=499, max=483833, avg=2458.47, stdev=10458.78 clat (msec): min=142, max=4113, avg=924.39, stdev=314.54 lat (msec): min=142, max=4114, avg=926.85, stdev=314.32 clat percentiles (msec): | 1.00th=[ 485], 5.00th=[ 592], 10.00th=[ 659], 20.00th=[ 726], | 30.00th=[ 776], 40.00th=[ 818], 50.00th=[ 860], 60.00th=[ 911], | 70.00th=[ 969], 80.00th=[ 1083], 90.00th=[ 1250], 95.00th=[ 1418], | 99.00th=[ 2333], 99.50th=[ 2534], 99.90th=[ 3540], 99.95th=[ 3540], | 99.99th=[ 4111] bw ( KiB/s): min=28672, max=368640, per=100.00%, avg=190582.73, stdev=65605.46, samples=59 iops : min= 14, max= 180, avg=93.05, stdev=32.03, samples=59 lat (msec) : 100=0.65%, 250=20.23%, 500=53.81%, 750=9.05%, 1000=10.74% lat (msec) : 2000=5.09%, >=2000=0.44% cpu : usr=6.40%, sys=1.77%, ctx=2755, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=11135,2827,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=727MiB/s (762MB/s), 727MiB/s-727MiB/s (762MB/s-762MB/s), io=21.7GiB (23.4GB), run=30639-30639msec WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5654MiB (5929MB), run=30639-30639msec Disk stats (read/write): vde: ios=22270/5646, merge=0/0, ticks=164317/3563410, in_queue=3749708, util=98.99% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975112: Fri Jan 7 14:35:01 2022 read: IOPS=160, BW=644MiB/s (675MB/s)(18.9GiB/30009msec) clat (usec): min=401, max=23136, avg=822.59, stdev=1458.57 lat (usec): min=401, max=23136, avg=822.80, stdev=1458.58 clat percentiles (usec): | 1.00th=[ 412], 5.00th=[ 420], 10.00th=[ 424], 20.00th=[ 437], | 30.00th=[ 445], 40.00th=[ 453], 50.00th=[ 465], 60.00th=[ 486], | 70.00th=[ 570], 80.00th=[ 1287], 90.00th=[ 1352], 95.00th=[ 1434], | 99.00th=[ 9896], 99.50th=[13304], 99.90th=[17695], 99.95th=[18744], | 99.99th=[23200] bw ( KiB/s): min=212992, max=1097728, per=100.00%, avg=659496.48, stdev=176323.28, samples=60 iops : min= 52, max= 268, avg=161.00, stdev=43.05, samples=60 write: IOPS=41, BW=166MiB/s (174MB/s)(4980MiB/30009msec) clat (msec): min=10, max=117, avg=20.30, stdev= 6.08 lat (msec): min=11, max=118, avg=20.90, stdev= 6.11 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 24], 90.00th=[ 26], 95.00th=[ 26], | 99.00th=[ 37], 99.50th=[ 43], 99.90th=[ 110], 99.95th=[ 117], | 99.99th=[ 117] bw ( KiB/s): min=122880, max=188416, per=99.96%, avg=169859.22, stdev=12613.27, samples=60 iops : min= 30, max= 46, avg=41.47, stdev= 3.08, samples=60 lat (usec) : 500=50.47%, 750=10.06%, 1000=0.89% lat (msec) : 2=16.97%, 4=0.08%, 10=0.26%, 20=11.09%, 50=10.07% lat (msec) : 100=0.07%, 250=0.03% cpu : usr=2.56%, sys=2.10%, ctx=6084, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4830,1245,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=644MiB/s (675MB/s), 644MiB/s-644MiB/s (675MB/s-675MB/s), io=18.9GiB (20.3GB), run=30009-30009msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4980MiB (5222MB), run=30009-30009msec Disk stats (read/write): vde: ios=19220/4952, merge=0/0, ticks=4963/81247, in_queue=95945, util=96.70% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976339: Fri Jan 7 14:35:38 2022 read: IOPS=161, BW=644MiB/s (676MB/s)(18.9GiB/30008msec) slat (usec): min=73, max=267, avg=96.94, stdev=18.07 clat (usec): min=200, max=22904, avg=715.98, stdev=1452.55 lat (usec): min=410, max=22994, avg=813.21, stdev=1453.67 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 347], 10.00th=[ 351], 20.00th=[ 359], | 30.00th=[ 367], 40.00th=[ 371], 50.00th=[ 379], 60.00th=[ 392], | 70.00th=[ 420], 80.00th=[ 1074], 90.00th=[ 1237], 95.00th=[ 1270], | 99.00th=[10290], 99.50th=[13304], 99.90th=[17433], 99.95th=[18482], | 99.99th=[22938] bw ( KiB/s): min=303104, max=1024000, per=100.00%, avg=660138.67, stdev=172841.97, samples=60 iops : min= 74, max= 250, avg=161.17, stdev=42.20, samples=60 write: IOPS=41, BW=166MiB/s (174MB/s)(4988MiB/30008msec) slat (usec): min=261, max=1602, avg=640.33, stdev=309.76 clat (msec): min=10, max=107, avg=20.26, stdev= 5.72 lat (msec): min=11, max=108, avg=20.90, stdev= 5.76 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 25], 90.00th=[ 26], 95.00th=[ 27], | 99.00th=[ 34], 99.50th=[ 47], 99.90th=[ 73], 99.95th=[ 108], | 99.99th=[ 108] bw ( KiB/s): min=147456, max=188416, per=99.95%, avg=170120.53, stdev=10380.43, samples=60 iops : min= 36, max= 46, avg=41.53, stdev= 2.53, samples=60 lat (usec) : 250=0.02%, 500=59.34%, 750=1.92%, 1000=1.46% lat (msec) : 2=15.70%, 10=0.23%, 20=11.13%, 50=10.11%, 100=0.07% lat (msec) : 250=0.02% cpu : usr=2.45%, sys=2.07%, ctx=6094, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4835,1247,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=644MiB/s (676MB/s), 644MiB/s-644MiB/s (676MB/s-676MB/s), io=18.9GiB (20.3GB), run=30008-30008msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4988MiB (5230MB), run=30008-30008msec Disk stats (read/write): vde: ios=19282/4964, merge=0/0, ticks=4706/81401, in_queue=95600, util=96.21% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977099: Fri Jan 7 14:36:15 2022 read: IOPS=160, BW=641MiB/s (672MB/s)(18.8GiB/30007msec) clat (usec): min=400, max=23291, avg=788.16, stdev=1461.87 lat (usec): min=400, max=23292, avg=788.36, stdev=1461.88 clat percentiles (usec): | 1.00th=[ 408], 5.00th=[ 416], 10.00th=[ 424], 20.00th=[ 433], | 30.00th=[ 441], 40.00th=[ 449], 50.00th=[ 457], 60.00th=[ 474], | 70.00th=[ 523], 80.00th=[ 1057], 90.00th=[ 1319], 95.00th=[ 1369], | 99.00th=[ 9765], 99.50th=[13566], 99.90th=[18482], 99.95th=[19530], | 99.99th=[23200] bw ( KiB/s): min=216455, max=1053020, per=100.00%, avg=664821.00, stdev=159200.97, samples=60 iops : min= 52, max= 257, avg=161.75, stdev=38.90, samples=60 write: IOPS=41, BW=165MiB/s (173MB/s)(4956MiB/30007msec) clat (usec): min=10768, max=79678, avg=20572.41, stdev=6023.70 lat (usec): min=11237, max=80041, avg=21149.74, stdev=6046.10 clat percentiles (usec): | 1.00th=[11994], 5.00th=[13698], 10.00th=[14615], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18220], 50.00th=[20317], 60.00th=[22414], | 70.00th=[23462], 80.00th=[24249], 90.00th=[25297], 95.00th=[26608], | 99.00th=[39060], 99.50th=[57410], 99.90th=[76022], 99.95th=[79168], | 99.99th=[79168] bw ( KiB/s): min=141241, max=191869, per=100.00%, avg=171283.18, stdev=10911.05, samples=60 iops : min= 34, max= 46, avg=41.27, stdev= 2.63, samples=60 lat (usec) : 500=53.92%, 750=7.51%, 1000=1.62% lat (msec) : 2=15.40%, 4=0.02%, 10=0.25%, 20=10.82%, 50=10.32% lat (msec) : 100=0.13% cpu : usr=2.57%, sys=1.91%, ctx=6055, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4805,1239,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=641MiB/s (672MB/s), 641MiB/s-641MiB/s (672MB/s-672MB/s), io=18.8GiB (20.2GB), run=30007-30007msec WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=4956MiB (5197MB), run=30007-30007msec Disk stats (read/write): vde: ios=19220/4952, merge=0/0, ticks=4436/81764, in_queue=95811, util=96.60% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977964: Fri Jan 7 14:36:52 2022 read: IOPS=193, BW=773MiB/s (810MB/s)(22.8GiB/30186msec) slat (usec): min=40, max=316, avg=61.43, stdev=18.09 clat (usec): min=242, max=2523.3k, avg=6661.14, stdev=80701.36 lat (usec): min=377, max=2523.3k, avg=6722.84, stdev=80701.39 clat percentiles (usec): | 1.00th=[ 330], 5.00th=[ 343], 10.00th=[ 347], | 20.00th=[ 355], 30.00th=[ 363], 40.00th=[ 371], | 50.00th=[ 379], 60.00th=[ 396], 70.00th=[ 465], | 80.00th=[ 1237], 90.00th=[ 1287], 95.00th=[ 1352], | 99.00th=[ 41681], 99.50th=[ 375391], 99.90th=[1317012], | 99.95th=[2055209], 99.99th=[2533360] bw ( KiB/s): min=376832, max=1359872, per=100.00%, avg=795852.80, stdev=191513.55, samples=60 iops : min= 92, max= 332, avg=194.30, stdev=46.76, samples=60 write: IOPS=50, BW=202MiB/s (212MB/s)(6092MiB/30186msec) slat (usec): min=247, max=2170, avg=800.62, stdev=395.36 clat (msec): min=17, max=2175, avg=171.53, stdev=155.88 lat (msec): min=18, max=2176, avg=172.33, stdev=155.88 clat percentiles (msec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 48], 20.00th=[ 57], | 30.00th=[ 74], 40.00th=[ 94], 50.00th=[ 124], 60.00th=[ 157], | 70.00th=[ 201], 80.00th=[ 259], 90.00th=[ 363], 95.00th=[ 456], | 99.00th=[ 751], 99.50th=[ 869], 99.90th=[ 1150], 99.95th=[ 2165], | 99.99th=[ 2165] bw ( KiB/s): min=180224, max=221184, per=100.00%, avg=206711.47, stdev=8841.94, samples=60 iops : min= 44, max= 54, avg=50.47, stdev= 2.16, samples=60 lat (usec) : 250=0.01%, 500=56.79%, 750=2.22%, 1000=0.39% lat (msec) : 2=18.31%, 4=0.22%, 10=0.14%, 20=0.33%, 50=2.37% lat (msec) : 100=6.62%, 250=7.74%, 500=3.77%, 750=0.61%, 1000=0.29% lat (msec) : 2000=0.15%, >=2000=0.05% cpu : usr=3.88%, sys=1.72%, ctx=7302, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=5830,1523,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=773MiB/s (810MB/s), 773MiB/s-773MiB/s (810MB/s-810MB/s), io=22.8GiB (24.5GB), run=30186-30186msec WRITE: bw=202MiB/s (212MB/s), 202MiB/s-202MiB/s (212MB/s-212MB/s), io=6092MiB (6388MB), run=30186-30186msec Disk stats (read/write): vde: ios=23319/6073, merge=0/0, ticks=46463/1019370, in_queue=1081299, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978560: Fri Jan 7 14:37:29 2022 read: IOPS=161, BW=647MiB/s (678MB/s)(18.9GiB/30006msec) clat (usec): min=378, max=23389, avg=768.48, stdev=1529.51 lat (usec): min=378, max=23389, avg=768.69, stdev=1529.51 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 412], 40.00th=[ 420], 50.00th=[ 429], 60.00th=[ 441], | 70.00th=[ 474], 80.00th=[ 1020], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[10552], 99.50th=[14353], 99.90th=[18744], 99.95th=[19530], | 99.99th=[23462] bw ( KiB/s): min=294912, max=1040384, per=100.00%, avg=662186.67, stdev=165938.20, samples=60 iops : min= 72, max= 254, avg=161.67, stdev=40.51, samples=60 write: IOPS=41, BW=167MiB/s (175MB/s)(5004MiB/30006msec) clat (usec): min=11007, max=84387, avg=20429.05, stdev=5267.64 lat (usec): min=11443, max=85587, avg=20993.74, stdev=5308.24 clat percentiles (usec): | 1.00th=[12125], 5.00th=[13829], 10.00th=[14615], 20.00th=[15795], | 30.00th=[16909], 40.00th=[18220], 50.00th=[20317], 60.00th=[22676], | 70.00th=[23725], 80.00th=[24249], 90.00th=[25035], 95.00th=[26346], | 99.00th=[34866], 99.50th=[36963], 99.90th=[62653], 99.95th=[84411], | 99.99th=[84411] bw ( KiB/s): min=147456, max=196608, per=99.94%, avg=170666.67, stdev=10893.70, samples=60 iops : min= 36, max= 48, avg=41.67, stdev= 2.66, samples=60 lat (usec) : 500=57.30%, 750=4.25%, 1000=1.80% lat (msec) : 2=15.05%, 4=0.03%, 10=0.18%, 20=10.83%, 50=10.49% lat (msec) : 100=0.07% cpu : usr=2.58%, sys=1.75%, ctx=6113, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4850,1251,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=647MiB/s (678MB/s), 647MiB/s-647MiB/s (678MB/s-678MB/s), io=18.9GiB (20.3GB), run=30006-30006msec WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=5004MiB (5247MB), run=30006-30006msec Disk stats (read/write): vde: ios=19356/4993, merge=0/0, ticks=4396/81747, in_queue=95692, util=96.47% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979523: Fri Jan 7 14:38:06 2022 read: IOPS=184, BW=739MiB/s (775MB/s)(22.1GiB/30585msec) slat (usec): min=45, max=472370, avg=3867.98, stdev=12232.17 clat (msec): min=16, max=2718, avg=83.05, stdev=120.65 lat (msec): min=16, max=2723, avg=86.92, stdev=121.52 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 25], 10.00th=[ 29], 20.00th=[ 37], | 30.00th=[ 44], 40.00th=[ 53], 50.00th=[ 62], 60.00th=[ 72], | 70.00th=[ 84], 80.00th=[ 97], 90.00th=[ 125], 95.00th=[ 157], | 99.00th=[ 625], 99.50th=[ 768], 99.90th=[ 1418], 99.95th=[ 2567], | 99.99th=[ 2735] bw ( KiB/s): min=114917, max=1548288, per=100.00%, avg=769629.58, stdev=314100.27, samples=60 iops : min= 28, max= 378, avg=187.87, stdev=76.67, samples=60 write: IOPS=48, BW=194MiB/s (203MB/s)(5924MiB/30585msec) slat (usec): min=1106, max=665684, avg=5470.70, stdev=27339.79 clat (msec): min=89, max=3206, avg=694.31, stdev=312.17 lat (msec): min=137, max=3229, avg=699.79, stdev=311.47 clat percentiles (msec): | 1.00th=[ 284], 5.00th=[ 384], 10.00th=[ 460], 20.00th=[ 514], | 30.00th=[ 550], 40.00th=[ 584], 50.00th=[ 625], 60.00th=[ 667], | 70.00th=[ 735], 80.00th=[ 827], 90.00th=[ 1011], 95.00th=[ 1200], | 99.00th=[ 1703], 99.50th=[ 3071], 99.90th=[ 3205], 99.95th=[ 3205], | 99.99th=[ 3205] bw ( KiB/s): min=16384, max=335872, per=99.76%, avg=197866.58, stdev=64427.63, samples=60 iops : min= 4, max= 82, avg=48.30, stdev=15.73, samples=60 lat (msec) : 20=0.49%, 50=29.24%, 100=34.61%, 250=13.15%, 500=3.84% lat (msec) : 750=12.52%, 1000=3.73%, 2000=2.23%, >=2000=0.20% cpu : usr=6.44%, sys=1.73%, ctx=2595, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.6%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=5653,1481,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=739MiB/s (775MB/s), 739MiB/s-739MiB/s (775MB/s-775MB/s), io=22.1GiB (23.7GB), run=30585-30585msec WRITE: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=5924MiB (6212MB), run=30585-30585msec Disk stats (read/write): vde: ios=22612/5905, merge=0/0, ticks=158335/3571401, in_queue=3760380, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980493: Fri Jan 7 14:38:43 2022 read: IOPS=157, BW=629MiB/s (660MB/s)(18.4GiB/30013msec) clat (usec): min=372, max=92068, avg=781.83, stdev=1969.50 lat (usec): min=373, max=92068, avg=782.02, stdev=1969.51 clat percentiles (usec): | 1.00th=[ 388], 5.00th=[ 396], 10.00th=[ 400], 20.00th=[ 408], | 30.00th=[ 416], 40.00th=[ 424], 50.00th=[ 433], 60.00th=[ 449], | 70.00th=[ 498], 80.00th=[ 1045], 90.00th=[ 1270], 95.00th=[ 1319], | 99.00th=[10028], 99.50th=[13960], 99.90th=[18482], 99.95th=[21627], | 99.99th=[91751] bw ( KiB/s): min=294912, max=1056768, per=100.00%, avg=644869.92, stdev=165806.74, samples=60 iops : min= 72, max= 258, avg=157.43, stdev=40.48, samples=60 write: IOPS=40, BW=163MiB/s (171MB/s)(4904MiB/30013msec) clat (msec): min=10, max=108, avg=20.90, stdev= 7.76 lat (msec): min=11, max=109, avg=21.46, stdev= 7.78 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 19], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 25], 90.00th=[ 26], 95.00th=[ 27], | 99.00th=[ 50], 99.50th=[ 73], 99.90th=[ 107], 99.95th=[ 109], | 99.99th=[ 109] bw ( KiB/s): min=114688, max=188416, per=99.97%, avg=167258.80, stdev=12998.78, samples=60 iops : min= 28, max= 46, avg=40.83, stdev= 3.17, samples=60 lat (usec) : 500=55.67%, 750=5.13%, 1000=2.19% lat (msec) : 2=15.35%, 4=0.02%, 10=0.24%, 20=10.74%, 50=10.49% lat (msec) : 100=0.12%, 250=0.07% cpu : usr=2.46%, sys=1.77%, ctx=5959, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4723,1226,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=629MiB/s (660MB/s), 629MiB/s-629MiB/s (660MB/s-660MB/s), io=18.4GiB (19.8GB), run=30013-30013msec WRITE: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=4904MiB (5142MB), run=30013-30013msec Disk stats (read/write): vde: ios=18776/4864, merge=0/0, ticks=4357/81535, in_queue=94915, util=95.99% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981078: Fri Jan 7 14:39:21 2022 read: IOPS=188, BW=752MiB/s (789MB/s)(22.5GiB/30604msec) slat (usec): min=46, max=488167, avg=3834.19, stdev=13568.11 clat (msec): min=78, max=2848, avg=294.67, stdev=154.35 lat (msec): min=78, max=2848, avg=298.51, stdev=154.77 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 107], 10.00th=[ 131], 20.00th=[ 197], | 30.00th=[ 228], 40.00th=[ 253], 50.00th=[ 279], 60.00th=[ 300], | 70.00th=[ 326], 80.00th=[ 359], 90.00th=[ 422], 95.00th=[ 617], | 99.00th=[ 743], 99.50th=[ 768], 99.90th=[ 1770], 99.95th=[ 2702], | 99.99th=[ 2836] bw ( KiB/s): min=106496, max=1417216, per=100.00%, avg=777600.90, stdev=290074.38, samples=60 iops : min= 26, max= 346, avg=189.82, stdev=70.82, samples=60 write: IOPS=49, BW=196MiB/s (206MB/s)(6000MiB/30604msec) slat (usec): min=1136, max=595892, avg=5266.51, stdev=21301.17 clat (msec): min=156, max=4155, avg=886.16, stdev=368.76 lat (msec): min=252, max=4156, avg=891.42, stdev=370.80 clat percentiles (msec): | 1.00th=[ 489], 5.00th=[ 575], 10.00th=[ 625], 20.00th=[ 693], | 30.00th=[ 726], 40.00th=[ 768], 50.00th=[ 818], 60.00th=[ 869], | 70.00th=[ 927], 80.00th=[ 1011], 90.00th=[ 1150], 95.00th=[ 1301], | 99.00th=[ 2970], 99.50th=[ 3272], 99.90th=[ 4077], 99.95th=[ 4144], | 99.99th=[ 4144] bw ( KiB/s): min=24576, max=344064, per=100.00%, avg=202870.32, stdev=73247.34, samples=59 iops : min= 6, max= 84, avg=49.51, stdev=17.89, samples=59 lat (msec) : 100=2.48%, 250=28.31%, 500=43.56%, 750=11.83%, 1000=9.12% lat (msec) : 2000=4.30%, >=2000=0.40% cpu : usr=6.74%, sys=1.63%, ctx=2640, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5755,1500,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=752MiB/s (789MB/s), 752MiB/s-752MiB/s (789MB/s-789MB/s), io=22.5GiB (24.1GB), run=30604-30604msec WRITE: bw=196MiB/s (206MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s), io=6000MiB (6291MB), run=30604-30604msec Disk stats (read/write): vde: ios=23020/5996, merge=0/0, ticks=151688/3594442, in_queue=3762816, util=99.10% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981900: Fri Jan 7 14:39:57 2022 read: IOPS=160, BW=641MiB/s (673MB/s)(18.8GiB/30001msec) clat (usec): min=372, max=86130, avg=762.80, stdev=1930.16 lat (usec): min=372, max=86131, avg=763.00, stdev=1930.17 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 400], | 30.00th=[ 408], 40.00th=[ 416], 50.00th=[ 424], 60.00th=[ 437], | 70.00th=[ 469], 80.00th=[ 963], 90.00th=[ 1254], 95.00th=[ 1303], | 99.00th=[10290], 99.50th=[13698], 99.90th=[18744], 99.95th=[23200], | 99.99th=[86508] bw ( KiB/s): min=335872, max=1105920, per=99.56%, avg=653971.53, stdev=170791.01, samples=59 iops : min= 82, max= 270, avg=159.66, stdev=41.70, samples=59 write: IOPS=41, BW=165MiB/s (173MB/s)(4956MiB/30001msec) clat (msec): min=11, max=114, avg=20.68, stdev= 6.80 lat (msec): min=11, max=115, avg=21.24, stdev= 6.84 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 14], 10.00th=[ 15], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 21], 60.00th=[ 23], | 70.00th=[ 24], 80.00th=[ 25], 90.00th=[ 26], 95.00th=[ 27], | 99.00th=[ 43], 99.50th=[ 61], 99.90th=[ 99], 99.95th=[ 115], | 99.99th=[ 115] bw ( KiB/s): min=139264, max=188416, per=100.00%, avg=169671.59, stdev=10654.04, samples=59 iops : min= 34, max= 46, avg=41.42, stdev= 2.60, samples=59 lat (usec) : 500=57.52%, 750=4.30%, 1000=2.28% lat (msec) : 2=14.40%, 10=0.21%, 20=10.74%, 50=10.38%, 100=0.15% lat (msec) : 250=0.02% cpu : usr=2.49%, sys=1.46%, ctx=6056, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4811,1239,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=641MiB/s (673MB/s), 641MiB/s-641MiB/s (673MB/s-673MB/s), io=18.8GiB (20.2GB), run=30001-30001msec WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=4956MiB (5197MB), run=30001-30001msec Disk stats (read/write): vde: ios=19076/4933, merge=0/0, ticks=4244/81874, in_queue=95523, util=95.25% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983247: Fri Jan 7 14:40:35 2022 read: IOPS=184, BW=739MiB/s (775MB/s)(22.1GiB/30629msec) slat (usec): min=45, max=519848, avg=4135.14, stdev=16462.59 clat (msec): min=315, max=3291, avg=726.78, stdev=179.30 lat (msec): min=316, max=3291, avg=730.91, stdev=179.92 clat percentiles (msec): | 1.00th=[ 422], 5.00th=[ 481], 10.00th=[ 542], 20.00th=[ 600], | 30.00th=[ 642], 40.00th=[ 676], 50.00th=[ 709], 60.00th=[ 743], | 70.00th=[ 785], 80.00th=[ 835], 90.00th=[ 927], 95.00th=[ 1028], | 99.00th=[ 1217], 99.50th=[ 1318], 99.90th=[ 2567], 99.95th=[ 3071], | 99.99th=[ 3306] bw ( KiB/s): min= 8192, max=1368064, per=99.65%, avg=754099.03, stdev=286061.18, samples=60 iops : min= 2, max= 334, avg=184.10, stdev=69.84, samples=60 write: IOPS=48, BW=194MiB/s (203MB/s)(5944MiB/30629msec) slat (usec): min=1113, max=123167, avg=4430.91, stdev=8807.69 clat (msec): min=639, max=4034, avg=1309.04, stdev=323.36 lat (msec): min=643, max=4054, avg=1313.47, stdev=323.50 clat percentiles (msec): | 1.00th=[ 735], 5.00th=[ 877], 10.00th=[ 1011], 20.00th=[ 1099], | 30.00th=[ 1167], 40.00th=[ 1217], 50.00th=[ 1267], 60.00th=[ 1318], | 70.00th=[ 1385], 80.00th=[ 1485], 90.00th=[ 1620], 95.00th=[ 1770], | 99.00th=[ 2433], 99.50th=[ 3373], 99.90th=[ 3943], 99.95th=[ 4044], | 99.99th=[ 4044] bw ( KiB/s): min=24576, max=532480, per=99.43%, avg=197589.66, stdev=95484.04, samples=59 iops : min= 6, max= 130, avg=48.24, stdev=23.31, samples=59 lat (msec) : 500=4.84%, 750=44.69%, 1000=27.07%, 2000=22.86%, >=2000=0.55% cpu : usr=6.66%, sys=1.59%, ctx=2630, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.4%, >=64=99.1% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=5659,1486,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=739MiB/s (775MB/s), 739MiB/s-739MiB/s (775MB/s-775MB/s), io=22.1GiB (23.7GB), run=30629-30629msec WRITE: bw=194MiB/s (203MB/s), 194MiB/s-194MiB/s (203MB/s-203MB/s), io=5944MiB (6233MB), run=30629-30629msec Disk stats (read/write): vde: ios=22636/5937, merge=0/0, ticks=154349/3590757, in_queue=3763644, util=98.28%