30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890675: Fri Jan 7 13:33:20 2022 read: IOPS=12.9k, BW=50.3MiB/s (52.7MB/s)(1508MiB/30001msec) clat (usec): min=24, max=2322, avg=49.18, stdev=20.17 lat (usec): min=24, max=2322, avg=49.30, stdev=20.18 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 67], | 99.00th=[ 157], 99.50th=[ 169], 99.90th=[ 196], 99.95th=[ 229], | 99.99th=[ 578] bw ( KiB/s): min=46480, max=55648, per=100.00%, avg=51488.39, stdev=1971.60, samples=59 iops : min=11620, max=13912, avg=12872.08, stdev=492.91, samples=59 write: IOPS=3232, BW=12.6MiB/s (13.2MB/s)(379MiB/30001msec) clat (usec): min=76, max=2000, avg=106.57, stdev=22.00 lat (usec): min=77, max=2001, avg=106.79, stdev=22.02 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 102], 50.00th=[ 104], 60.00th=[ 106], | 70.00th=[ 111], 80.00th=[ 115], 90.00th=[ 121], 95.00th=[ 129], | 99.00th=[ 151], 99.50th=[ 163], 99.90th=[ 243], 99.95th=[ 412], | 99.99th=[ 1270] bw ( KiB/s): min=11648, max=13824, per=99.98%, avg=12925.19, stdev=496.12, samples=59 iops : min= 2912, max= 3456, avg=3231.27, stdev=124.04, samples=59 lat (usec) : 50=55.41%, 100=29.89%, 250=14.65%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=6.60%, sys=20.33%, ctx=483111, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=386128,96969,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=50.3MiB/s (52.7MB/s), 50.3MiB/s-50.3MiB/s (52.7MB/s-52.7MB/s), io=1508MiB (1582MB), run=30001-30001msec WRITE: bw=12.6MiB/s (13.2MB/s), 12.6MiB/s-12.6MiB/s (13.2MB/s-13.2MB/s), io=379MiB (397MB), run=30001-30001msec Disk stats (read/write): vdc: ios=383195/96182, merge=0/0, ticks=9/11, in_queue=21910, util=73.33% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891549: Fri Jan 7 13:33:57 2022 read: IOPS=13.5k, BW=52.7MiB/s (55.2MB/s)(1580MiB/30001msec) slat (usec): min=3, max=260, avg= 8.82, stdev= 4.31 clat (nsec): min=604, max=2381.6k, avg=37854.17, stdev=19247.58 lat (usec): min=21, max=2396, avg=46.81, stdev=19.58 clat percentiles (usec): | 1.00th=[ 18], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 48], 95.00th=[ 55], | 99.00th=[ 145], 99.50th=[ 157], 99.90th=[ 180], 99.95th=[ 210], | 99.99th=[ 404] bw ( KiB/s): min=50584, max=56480, per=99.94%, avg=53899.92, stdev=1308.28, samples=59 iops : min=12646, max=14120, avg=13474.97, stdev=327.07, samples=59 write: IOPS=3384, BW=13.2MiB/s (13.9MB/s)(397MiB/30001msec) slat (usec): min=5, max=373, avg= 9.54, stdev= 4.71 clat (nsec): min=1040, max=2745.4k, avg=93019.74, stdev=20063.92 lat (usec): min=73, max=2760, avg=102.70, stdev=20.68 clat percentiles (usec): | 1.00th=[ 74], 5.00th=[ 79], 10.00th=[ 82], 20.00th=[ 85], | 30.00th=[ 87], 40.00th=[ 89], 50.00th=[ 91], 60.00th=[ 94], | 70.00th=[ 96], 80.00th=[ 100], 90.00th=[ 108], 95.00th=[ 114], | 99.00th=[ 133], 99.50th=[ 143], 99.90th=[ 188], 99.95th=[ 269], | 99.99th=[ 848] bw ( KiB/s): min=12736, max=14320, per=99.98%, avg=13535.56, stdev=344.90, samples=59 iops : min= 3184, max= 3580, avg=3383.88, stdev=86.23, samples=59 lat (nsec) : 750=0.05%, 1000=0.22% lat (usec) : 2=0.16%, 4=0.01%, 10=0.08%, 20=0.69%, 50=72.90% lat (usec) : 100=20.58%, 250=5.27%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=6.33%, sys=32.31%, ctx=503031, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=404524,101543,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=52.7MiB/s (55.2MB/s), 52.7MiB/s-52.7MiB/s (55.2MB/s-55.2MB/s), io=1580MiB (1657MB), run=30001-30001msec WRITE: bw=13.2MiB/s (13.9MB/s), 13.2MiB/s-13.2MiB/s (13.9MB/s-13.9MB/s), io=397MiB (416MB), run=30001-30001msec Disk stats (read/write): vdc: ios=401926/100875, merge=0/0, ticks=10/11, in_queue=21584, util=72.16% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892543: Fri Jan 7 13:34:34 2022 read: IOPS=12.9k, BW=50.3MiB/s (52.8MB/s)(1511MiB/30001msec) clat (usec): min=24, max=2189, avg=49.10, stdev=20.75 lat (usec): min=24, max=2189, avg=49.22, stdev=20.76 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 67], | 99.00th=[ 157], 99.50th=[ 169], 99.90th=[ 194], 99.95th=[ 223], | 99.99th=[ 510] bw ( KiB/s): min=46688, max=54584, per=100.00%, avg=51565.78, stdev=1987.13, samples=59 iops : min=11672, max=13646, avg=12891.36, stdev=496.73, samples=59 write: IOPS=3236, BW=12.6MiB/s (13.3MB/s)(379MiB/30001msec) clat (usec): min=78, max=6359, avg=106.20, stdev=28.58 lat (usec): min=78, max=6359, avg=106.43, stdev=28.60 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 106], | 70.00th=[ 110], 80.00th=[ 114], 90.00th=[ 121], 95.00th=[ 127], | 99.00th=[ 147], 99.50th=[ 159], 99.90th=[ 212], 99.95th=[ 302], | 99.99th=[ 1270] bw ( KiB/s): min=11544, max=13672, per=100.00%, avg=12944.36, stdev=520.48, samples=59 iops : min= 2886, max= 3418, avg=3236.07, stdev=130.11, samples=59 lat (usec) : 50=55.63%, 100=29.86%, 250=14.47%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=6.34%, sys=20.72%, ctx=483801, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=386694,97093,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=50.3MiB/s (52.8MB/s), 50.3MiB/s-50.3MiB/s (52.8MB/s-52.8MB/s), io=1511MiB (1584MB), run=30001-30001msec WRITE: bw=12.6MiB/s (13.3MB/s), 12.6MiB/s-12.6MiB/s (13.3MB/s-13.3MB/s), io=379MiB (398MB), run=30001-30001msec Disk stats (read/write): vdc: ios=383953/96409, merge=0/0, ticks=14/16, in_queue=22063, util=73.81% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892997: Fri Jan 7 13:35:11 2022 read: IOPS=19.9k, BW=77.8MiB/s (81.5MB/s)(2335MiB/30033msec) slat (nsec): min=1993, max=1017.2k, avg=5376.37, stdev=7759.93 clat (nsec): min=633, max=44671k, avg=349088.90, stdev=1351021.58 lat (usec): min=22, max=44677, avg=354.57, stdev=1351.01 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 61], 10.00th=[ 71], 20.00th=[ 85], | 30.00th=[ 96], 40.00th=[ 106], 50.00th=[ 118], 60.00th=[ 130], | 70.00th=[ 145], 80.00th=[ 165], 90.00th=[ 212], 95.00th=[ 474], | 99.00th=[ 8094], 99.50th=[ 9896], 99.90th=[14746], 99.95th=[16581], | 99.99th=[26346] bw ( KiB/s): min= 5400, max=245536, per=100.00%, avg=79709.20, stdev=100132.64, samples=60 iops : min= 1350, max=61384, avg=19927.30, stdev=25033.16, samples=60 write: IOPS=4997, BW=19.5MiB/s (20.5MB/s)(586MiB/30033msec) slat (usec): min=2, max=715, avg= 6.54, stdev= 8.80 clat (usec): min=46, max=47706, avg=577.18, stdev=1898.88 lat (usec): min=63, max=47731, avg=583.84, stdev=1898.82 clat percentiles (usec): | 1.00th=[ 94], 5.00th=[ 118], 10.00th=[ 131], 20.00th=[ 151], | 30.00th=[ 172], 40.00th=[ 190], 50.00th=[ 206], 60.00th=[ 221], | 70.00th=[ 235], 80.00th=[ 258], 90.00th=[ 306], 95.00th=[ 1991], | 99.00th=[10421], 99.50th=[13435], 99.90th=[18482], 99.95th=[21627], | 99.99th=[36963] bw ( KiB/s): min= 1360, max=60848, per=100.00%, avg=20009.47, stdev=25098.64, samples=60 iops : min= 340, max=15212, avg=5002.37, stdev=6274.66, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.02%, 50=1.51%, 100=25.99% lat (usec) : 250=62.00%, 500=5.22%, 750=0.35%, 1000=0.25% lat (msec) : 2=1.15%, 4=1.18%, 10=1.71%, 20=0.59%, 50=0.03% cpu : usr=5.11%, sys=17.08%, ctx=73430, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=597825,150075,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=77.8MiB/s (81.5MB/s), 77.8MiB/s-77.8MiB/s (81.5MB/s-81.5MB/s), io=2335MiB (2449MB), run=30033-30033msec WRITE: bw=19.5MiB/s (20.5MB/s), 19.5MiB/s-19.5MiB/s (20.5MB/s-20.5MB/s), io=586MiB (615MB), run=30033-30033msec Disk stats (read/write): vdc: ios=597379/149970, merge=0/0, ticks=120470/52323, in_queue=260204, util=98.28% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894405: Fri Jan 7 13:35:48 2022 read: IOPS=12.0k, BW=50.7MiB/s (53.2MB/s)(1522MiB/30001msec) clat (usec): min=25, max=4084, avg=48.57, stdev=19.82 lat (usec): min=25, max=4085, avg=48.68, stdev=19.83 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 53], 90.00th=[ 59], 95.00th=[ 66], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 194], 99.95th=[ 219], | 99.99th=[ 392] bw ( KiB/s): min=48304, max=54800, per=100.00%, avg=51962.56, stdev=1289.02, samples=59 iops : min=12076, max=13700, avg=12990.63, stdev=322.26, samples=59 write: IOPS=3261, BW=12.7MiB/s (13.4MB/s)(382MiB/30001msec) clat (usec): min=80, max=1614, avg=106.07, stdev=18.47 lat (usec): min=80, max=1614, avg=106.28, stdev=18.49 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 106], | 70.00th=[ 110], 80.00th=[ 114], 90.00th=[ 121], 95.00th=[ 128], | 99.00th=[ 147], 99.50th=[ 157], 99.90th=[ 221], 99.95th=[ 388], | 99.99th=[ 840] bw ( KiB/s): min=12064, max=13528, per=99.99%, avg=13042.47, stdev=312.65, samples=59 iops : min= 3016, max= 3382, avg=3260.61, stdev=78.16, samples=59 lat (usec) : 50=57.45%, 100=28.15%, 250=14.37%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=6.51%, sys=20.26%, ctx=487598, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=389752,97840,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=50.7MiB/s (53.2MB/s), 50.7MiB/s-50.7MiB/s (53.2MB/s-53.2MB/s), io=1522MiB (1596MB), run=30001-30001msec WRITE: bw=12.7MiB/s (13.4MB/s), 12.7MiB/s-12.7MiB/s (13.4MB/s-13.4MB/s), io=382MiB (401MB), run=30001-30001msec Disk stats (read/write): vdc: ios=387346/97247, merge=0/0, ticks=12/6, in_queue=22296, util=74.54% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894980: Fri Jan 7 13:36:25 2022 read: IOPS=21.4k, BW=83.7MiB/s (87.7MB/s)(2511MiB/30008msec) slat (usec): min=2, max=1152, avg= 4.80, stdev= 6.57 clat (usec): min=68, max=90154, avg=1797.23, stdev=5387.26 lat (usec): min=80, max=90203, avg=1802.10, stdev=5387.18 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 231], 10.00th=[ 273], 20.00th=[ 334], | 30.00th=[ 388], 40.00th=[ 441], 50.00th=[ 498], 60.00th=[ 562], | 70.00th=[ 627], 80.00th=[ 709], 90.00th=[ 889], 95.00th=[13042], | 99.00th=[28181], 99.50th=[32900], 99.90th=[45876], 99.95th=[50594], | 99.99th=[88605] bw ( KiB/s): min= 4553, max=342600, per=100.00%, avg=85705.72, stdev=127350.65, samples=60 iops : min= 1138, max=85650, avg=21426.38, stdev=31837.61, samples=60 write: IOPS=5376, BW=21.0MiB/s (22.0MB/s)(630MiB/30008msec) slat (usec): min=2, max=221, avg= 5.75, stdev= 6.81 clat (usec): min=113, max=89876, avg=2109.46, stdev=5840.78 lat (usec): min=118, max=89881, avg=2115.28, stdev=5840.77 clat percentiles (usec): | 1.00th=[ 322], 5.00th=[ 441], 10.00th=[ 498], 20.00th=[ 553], | 30.00th=[ 594], 40.00th=[ 635], 50.00th=[ 676], 60.00th=[ 717], | 70.00th=[ 775], 80.00th=[ 873], 90.00th=[ 1074], 95.00th=[14877], | 99.00th=[30802], 99.50th=[34866], 99.90th=[49021], 99.95th=[54789], | 99.99th=[88605] bw ( KiB/s): min= 1090, max=85896, per=100.00%, avg=21510.82, stdev=31950.30, samples=60 iops : min= 272, max=21474, avg=5377.68, stdev=7987.59, samples=60 lat (usec) : 100=0.01%, 250=5.62%, 500=36.78%, 750=37.64%, 1000=10.89% lat (msec) : 2=2.10%, 4=0.10%, 10=0.90%, 20=2.87%, 50=3.03% lat (msec) : 100=0.06% cpu : usr=3.92%, sys=13.98%, ctx=14867, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=642830,161339,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=83.7MiB/s (87.7MB/s), 83.7MiB/s-83.7MiB/s (87.7MB/s-87.7MB/s), io=2511MiB (2633MB), run=30008-30008msec WRITE: bw=21.0MiB/s (22.0MB/s), 21.0MiB/s-21.0MiB/s (22.0MB/s-22.0MB/s), io=630MiB (661MB), run=30008-30008msec Disk stats (read/write): vdc: ios=642421/161236, merge=0/0, ticks=817067/229166, in_queue=1353663, util=99.53% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895824: Fri Jan 7 13:37:02 2022 read: IOPS=12.9k, BW=50.5MiB/s (52.9MB/s)(1514MiB/30001msec) clat (usec): min=24, max=1816, avg=48.95, stdev=20.24 lat (usec): min=24, max=1816, avg=49.06, stdev=20.25 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 50], 80.00th=[ 54], 90.00th=[ 60], 95.00th=[ 67], | 99.00th=[ 157], 99.50th=[ 169], 99.90th=[ 194], 99.95th=[ 227], | 99.99th=[ 498] bw ( KiB/s): min=48696, max=54192, per=100.00%, avg=51749.69, stdev=1304.02, samples=59 iops : min=12174, max=13548, avg=12937.42, stdev=326.01, samples=59 write: IOPS=3243, BW=12.7MiB/s (13.3MB/s)(380MiB/30001msec) clat (usec): min=78, max=2110, avg=106.29, stdev=21.67 lat (usec): min=79, max=2110, avg=106.51, stdev=21.69 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 92], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 101], 50.00th=[ 104], 60.00th=[ 106], | 70.00th=[ 110], 80.00th=[ 114], 90.00th=[ 121], 95.00th=[ 128], | 99.00th=[ 149], 99.50th=[ 161], 99.90th=[ 237], 99.95th=[ 424], | 99.99th=[ 1037] bw ( KiB/s): min=12008, max=13680, per=100.00%, avg=12987.15, stdev=335.86, samples=59 iops : min= 3002, max= 3420, avg=3246.78, stdev=83.96, samples=59 lat (usec) : 50=56.41%, 100=29.20%, 250=14.34%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=6.18%, sys=20.30%, ctx=484877, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=387563,97302,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=50.5MiB/s (52.9MB/s), 50.5MiB/s-50.5MiB/s (52.9MB/s-52.9MB/s), io=1514MiB (1587MB), run=30001-30001msec WRITE: bw=12.7MiB/s (13.3MB/s), 12.7MiB/s-12.7MiB/s (13.3MB/s-13.3MB/s), io=380MiB (399MB), run=30001-30001msec Disk stats (read/write): vdc: ios=384387/96522, merge=0/0, ticks=14/12, in_queue=22237, util=74.48% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896948: Fri Jan 7 13:37:39 2022 read: IOPS=22.9k, BW=89.4MiB/s (93.8MB/s)(2687MiB/30048msec) slat (usec): min=2, max=329, avg= 4.71, stdev= 9.01 clat (usec): min=155, max=185867, avg=3401.33, stdev=9715.05 lat (usec): min=159, max=185976, avg=3406.11, stdev=9715.02 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 494], 10.00th=[ 586], 20.00th=[ 709], | 30.00th=[ 816], 40.00th=[ 922], 50.00th=[ 1037], 60.00th=[ 1156], | 70.00th=[ 1287], 80.00th=[ 1434], 90.00th=[ 1713], 95.00th=[ 27919], | 99.00th=[ 47449], 99.50th=[ 53740], 99.90th=[ 72877], 99.95th=[ 81265], | 99.99th=[185598] bw ( KiB/s): min= 3152, max=321096, per=100.00%, avg=91724.73, stdev=129175.19, samples=60 iops : min= 788, max=80274, avg=22931.20, stdev=32293.85, samples=60 write: IOPS=5748, BW=22.5MiB/s (23.5MB/s)(675MiB/30048msec) slat (usec): min=2, max=284, avg= 5.60, stdev= 9.01 clat (usec): min=259, max=185780, avg=3817.68, stdev=10268.06 lat (usec): min=263, max=185789, avg=3823.35, stdev=10268.15 clat percentiles (usec): | 1.00th=[ 619], 5.00th=[ 832], 10.00th=[ 963], 20.00th=[ 1090], | 30.00th=[ 1172], 40.00th=[ 1237], 50.00th=[ 1303], 60.00th=[ 1385], | 70.00th=[ 1483], 80.00th=[ 1631], 90.00th=[ 1975], 95.00th=[ 29754], | 99.00th=[ 49546], 99.50th=[ 58459], 99.90th=[ 78119], 99.95th=[117965], | 99.99th=[185598] bw ( KiB/s): min= 848, max=79840, per=100.00%, avg=23031.75, stdev=32421.73, samples=60 iops : min= 212, max=19960, avg=5757.92, stdev=8105.43, samples=60 lat (usec) : 250=0.04%, 500=4.22%, 750=15.43%, 1000=20.12% lat (msec) : 2=52.46%, 4=1.18%, 10=0.01%, 20=0.32%, 50=5.42% lat (msec) : 100=0.76%, 250=0.04% cpu : usr=3.60%, sys=14.40%, ctx=8146, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=687971,172742,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=89.4MiB/s (93.8MB/s), 89.4MiB/s-89.4MiB/s (93.8MB/s-93.8MB/s), io=2687MiB (2818MB), run=30048-30048msec WRITE: bw=22.5MiB/s (23.5MB/s), 22.5MiB/s-22.5MiB/s (23.5MB/s-23.5MB/s), io=675MiB (708MB), run=30048-30048msec Disk stats (read/write): vdc: ios=687887/172726, merge=0/0, ticks=1757973/495741, in_queue=2735609, util=99.58% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897507: Fri Jan 7 13:38:16 2022 read: IOPS=12.5k, BW=48.9MiB/s (51.2MB/s)(1466MiB/30001msec) clat (usec): min=25, max=15827, avg=51.00, stdev=33.65 lat (usec): min=25, max=15827, avg=51.12, stdev=33.66 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 40], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 70], | 99.00th=[ 159], 99.50th=[ 172], 99.90th=[ 198], 99.95th=[ 235], | 99.99th=[ 469] bw ( KiB/s): min=41008, max=52584, per=99.94%, avg=50008.00, stdev=1777.93, samples=59 iops : min=10252, max=13146, avg=12502.00, stdev=444.48, samples=59 write: IOPS=3138, BW=12.3MiB/s (12.9MB/s)(368MiB/30001msec) clat (usec): min=81, max=21500, avg=108.16, stdev=74.46 lat (usec): min=81, max=21501, avg=108.41, stdev=74.46 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 98], | 30.00th=[ 100], 40.00th=[ 103], 50.00th=[ 105], 60.00th=[ 109], | 70.00th=[ 112], 80.00th=[ 116], 90.00th=[ 123], 95.00th=[ 130], | 99.00th=[ 155], 99.50th=[ 169], 99.90th=[ 251], 99.95th=[ 457], | 99.99th=[ 1532] bw ( KiB/s): min=10288, max=13272, per=99.97%, avg=12548.61, stdev=446.44, samples=59 iops : min= 2572, max= 3318, avg=3137.15, stdev=111.61, samples=59 lat (usec) : 50=49.69%, 100=34.60%, 250=15.66%, 500=0.04%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.86%, sys=21.34%, ctx=469458, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=375290,94155,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=48.9MiB/s (51.2MB/s), 48.9MiB/s-48.9MiB/s (51.2MB/s-51.2MB/s), io=1466MiB (1537MB), run=30001-30001msec WRITE: bw=12.3MiB/s (12.9MB/s), 12.3MiB/s-12.3MiB/s (12.9MB/s-12.9MB/s), io=368MiB (386MB), run=30001-30001msec Disk stats (read/write): vdc: ios=372323/93386, merge=0/0, ticks=33/36, in_queue=21940, util=73.46% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898372: Fri Jan 7 13:38:53 2022 read: IOPS=24.4k, BW=95.4MiB/s (100MB/s)(2868MiB/30069msec) slat (usec): min=2, max=158847, avg=31.18, stdev=1063.93 clat (usec): min=37, max=294420, avg=6751.02, stdev=19695.06 lat (usec): min=586, max=396107, avg=6782.26, stdev=19811.62 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1123], 10.00th=[ 1221], 20.00th=[ 1385], | 30.00th=[ 1516], 40.00th=[ 1680], 50.00th=[ 1958], 60.00th=[ 2409], | 70.00th=[ 2671], 80.00th=[ 2966], 90.00th=[ 3425], 95.00th=[ 46924], | 99.00th=[103285], 99.50th=[113771], 99.90th=[141558], 99.95th=[160433], | 99.99th=[263193] bw ( KiB/s): min= 3280, max=310856, per=100.00%, avg=97878.15, stdev=131920.26, samples=60 iops : min= 820, max=77714, avg=24469.55, stdev=32980.10, samples=60 write: IOPS=6123, BW=23.9MiB/s (25.1MB/s)(719MiB/30069msec) slat (usec): min=2, max=78841, avg=32.83, stdev=1050.18 clat (usec): min=766, max=262381, avg=5577.51, stdev=12603.58 lat (usec): min=771, max=262389, avg=5610.41, stdev=12782.88 clat percentiles (usec): | 1.00th=[ 1303], 5.00th=[ 1582], 10.00th=[ 1778], 20.00th=[ 2180], | 30.00th=[ 2409], 40.00th=[ 2573], 50.00th=[ 2671], 60.00th=[ 2802], | 70.00th=[ 2966], 80.00th=[ 3163], 90.00th=[ 3556], 95.00th=[ 36439], | 99.00th=[ 64750], 99.50th=[ 76022], 99.90th=[114820], 99.95th=[141558], | 99.99th=[160433] bw ( KiB/s): min= 872, max=78056, per=100.00%, avg=24550.03, stdev=33137.75, samples=60 iops : min= 218, max=19514, avg=6137.50, stdev=8284.43, samples=60 lat (usec) : 50=0.01%, 750=0.01%, 1000=1.08% lat (msec) : 2=42.89%, 4=49.00%, 10=0.75%, 20=0.04%, 50=1.86% lat (msec) : 100=3.30%, 250=1.08%, 500=0.02% cpu : usr=3.55%, sys=13.85%, ctx=7014, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=734177,184138,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=95.4MiB/s (100MB/s), 95.4MiB/s-95.4MiB/s (100MB/s-100MB/s), io=2868MiB (3007MB), run=30069-30069msec WRITE: bw=23.9MiB/s (25.1MB/s), 23.9MiB/s-23.9MiB/s (25.1MB/s-25.1MB/s), io=719MiB (754MB), run=30069-30069msec Disk stats (read/write): vdc: ios=734079/184107, merge=0/0, ticks=2400643/648893, in_queue=3492243, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898990: Fri Jan 7 13:39:30 2022 read: IOPS=11.1k, BW=174MiB/s (182MB/s)(5220MiB/30001msec) clat (usec): min=25, max=5792, avg=53.64, stdev=29.36 lat (usec): min=25, max=5792, avg=53.76, stdev=29.37 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 65], 95.00th=[ 73], | 99.00th=[ 198], 99.50th=[ 217], 99.90th=[ 281], 99.95th=[ 330], | 99.99th=[ 693] bw ( KiB/s): min=153651, max=189995, per=100.00%, avg=179284.00, stdev=7255.21, samples=60 iops : min= 9603, max=11874, avg=11204.85, stdev=453.42, samples=60 write: IOPS=2797, BW=43.7MiB/s (45.8MB/s)(1311MiB/30001msec) clat (usec): min=104, max=34244, avg=135.78, stdev=131.49 lat (usec): min=105, max=34245, avg=137.13, stdev=131.50 clat percentiles (usec): | 1.00th=[ 116], 5.00th=[ 120], 10.00th=[ 122], 20.00th=[ 125], | 30.00th=[ 128], 40.00th=[ 130], 50.00th=[ 133], 60.00th=[ 135], | 70.00th=[ 139], 80.00th=[ 143], 90.00th=[ 151], 95.00th=[ 159], | 99.00th=[ 184], 99.50th=[ 198], 99.90th=[ 343], 99.95th=[ 553], | 99.99th=[ 1319] bw ( KiB/s): min=38316, max=49454, per=100.00%, avg=45035.73, stdev=1914.95, samples=60 iops : min= 2394, max= 3090, avg=2814.30, stdev=119.68, samples=60 lat (usec) : 50=43.55%, 100=34.82%, 250=21.47%, 500=0.14%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=6.05%, sys=18.12%, ctx=418003, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=334081,83915,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=5220MiB (5474MB), run=30001-30001msec WRITE: bw=43.7MiB/s (45.8MB/s), 43.7MiB/s-43.7MiB/s (45.8MB/s-45.8MB/s), io=1311MiB (1375MB), run=30001-30001msec Disk stats (read/write): vdc: ios=334007/83901, merge=0/0, ticks=27/73, in_queue=23016, util=76.48% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899806: Fri Jan 7 13:40:07 2022 read: IOPS=11.5k, BW=180MiB/s (189MB/s)(5401MiB/30001msec) slat (usec): min=5, max=237, avg= 9.26, stdev= 2.85 clat (nsec): min=662, max=5941.3k, avg=42386.36, stdev=27234.99 lat (usec): min=24, max=5961, avg=51.77, stdev=27.60 clat percentiles (usec): | 1.00th=[ 28], 5.00th=[ 31], 10.00th=[ 32], 20.00th=[ 34], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 38], 60.00th=[ 40], | 70.00th=[ 43], 80.00th=[ 46], 90.00th=[ 52], 95.00th=[ 59], | 99.00th=[ 188], 99.50th=[ 206], 99.90th=[ 269], 99.95th=[ 318], | 99.99th=[ 537] bw ( KiB/s): min=160864, max=194912, per=100.00%, avg=184451.92, stdev=7219.97, samples=59 iops : min=10054, max=12182, avg=11528.20, stdev=451.28, samples=59 write: IOPS=2892, BW=45.2MiB/s (47.4MB/s)(1356MiB/30001msec) slat (nsec): min=7201, max=99365, avg=11084.54, stdev=2971.62 clat (usec): min=69, max=22291, avg=122.45, stdev=78.11 lat (usec): min=103, max=22309, avg=133.67, stdev=78.29 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 109], 10.00th=[ 111], 20.00th=[ 114], | 30.00th=[ 116], 40.00th=[ 118], 50.00th=[ 120], 60.00th=[ 123], | 70.00th=[ 126], 80.00th=[ 130], 90.00th=[ 137], 95.00th=[ 143], | 99.00th=[ 165], 99.50th=[ 178], 99.90th=[ 277], 99.95th=[ 478], | 99.99th=[ 1156] bw ( KiB/s): min=39648, max=48928, per=100.00%, avg=46312.22, stdev=1814.18, samples=59 iops : min= 2478, max= 3058, avg=2894.47, stdev=113.41, samples=59 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.04%, 50=70.51% lat (usec) : 100=7.99%, 250=21.27%, 500=0.11%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=5.77%, sys=28.61%, ctx=432148, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=345651,86769,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=5401MiB (5663MB), run=30001-30001msec WRITE: bw=45.2MiB/s (47.4MB/s), 45.2MiB/s-45.2MiB/s (47.4MB/s-47.4MB/s), io=1356MiB (1422MB), run=30001-30001msec Disk stats (read/write): vdc: ios=343219/86151, merge=0/0, ticks=22/31, in_queue=22468, util=75.21% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901195: Fri Jan 7 13:40:44 2022 read: IOPS=11.5k, BW=180MiB/s (189MB/s)(5412MiB/30001msec) clat (usec): min=25, max=1684, avg=51.34, stdev=24.36 lat (usec): min=25, max=1685, avg=51.45, stdev=24.37 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 70], | 99.00th=[ 196], 99.50th=[ 215], 99.90th=[ 273], 99.95th=[ 318], | 99.99th=[ 486] bw ( KiB/s): min=167599, max=194720, per=100.00%, avg=184722.34, stdev=4996.04, samples=59 iops : min=10474, max=12170, avg=11545.12, stdev=312.32, samples=59 write: IOPS=2897, BW=45.3MiB/s (47.5MB/s)(1358MiB/30001msec) clat (usec): min=103, max=2844, avg=132.57, stdev=24.58 lat (usec): min=105, max=2846, avg=133.90, stdev=24.60 clat percentiles (usec): | 1.00th=[ 115], 5.00th=[ 118], 10.00th=[ 120], 20.00th=[ 123], | 30.00th=[ 126], 40.00th=[ 128], 50.00th=[ 130], 60.00th=[ 133], | 70.00th=[ 137], 80.00th=[ 141], 90.00th=[ 147], 95.00th=[ 155], | 99.00th=[ 178], 99.50th=[ 190], 99.90th=[ 289], 99.95th=[ 502], | 99.99th=[ 1336] bw ( KiB/s): min=42292, max=49088, per=100.00%, avg=46381.76, stdev=1262.51, samples=59 iops : min= 2643, max= 3068, avg=2898.83, stdev=78.91, samples=59 lat (usec) : 50=51.47%, 100=26.99%, 250=21.39%, 500=0.13%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=5.80%, sys=18.68%, ctx=433292, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=346352,86928,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=5412MiB (5675MB), run=30001-30001msec WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s-47.5MB/s), io=1358MiB (1424MB), run=30001-30001msec Disk stats (read/write): vdc: ios=343807/86288, merge=0/0, ticks=13/15, in_queue=22846, util=76.47% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901858: Fri Jan 7 13:41:21 2022 read: IOPS=15.8k, BW=247MiB/s (259MB/s)(7420MiB/30089msec) slat (usec): min=2, max=860, avg= 6.21, stdev= 6.84 clat (nsec): min=676, max=89709k, avg=405707.38, stdev=1094416.07 lat (usec): min=29, max=89715, avg=412.01, stdev=1094.36 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 68], 10.00th=[ 79], 20.00th=[ 95], | 30.00th=[ 109], 40.00th=[ 121], 50.00th=[ 135], 60.00th=[ 149], | 70.00th=[ 165], 80.00th=[ 194], 90.00th=[ 453], 95.00th=[ 2311], | 99.00th=[ 5604], 99.50th=[ 6718], 99.90th=[ 9765], 99.95th=[11731], | 99.99th=[17957] bw ( KiB/s): min=31808, max=806912, per=100.00%, avg=253265.42, stdev=323100.32, samples=60 iops : min= 1988, max=50432, avg=15829.08, stdev=20193.77, samples=60 write: IOPS=3965, BW=61.0MiB/s (64.0MB/s)(1864MiB/30089msec) slat (usec): min=3, max=588, avg=10.12, stdev= 9.74 clat (usec): min=74, max=95941, avg=866.78, stdev=1997.42 lat (usec): min=98, max=95948, avg=877.01, stdev=1996.93 clat percentiles (usec): | 1.00th=[ 126], 5.00th=[ 161], 10.00th=[ 188], 20.00th=[ 223], | 30.00th=[ 245], 40.00th=[ 265], 50.00th=[ 281], 60.00th=[ 297], | 70.00th=[ 322], 80.00th=[ 359], 90.00th=[ 2606], 95.00th=[ 5342], | 99.00th=[ 8979], 99.50th=[10421], 99.90th=[14484], 99.95th=[17695], | 99.99th=[34866] bw ( KiB/s): min= 7616, max=206272, per=100.00%, avg=63633.98, stdev=81050.04, samples=60 iops : min= 476, max=12892, avg=3977.12, stdev=5065.63, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 10=0.01%, 20=0.01%, 50=0.75%, 100=18.38%, 250=56.84% lat (usec) : 500=13.69%, 750=0.68%, 1000=0.75% lat (msec) : 2=2.14%, 4=3.44%, 10=3.12%, 20=0.18%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=5.04%, sys=15.58%, ctx=65457, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=474874,119319,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=7420MiB (7780MB), run=30089-30089msec WRITE: bw=61.0MiB/s (64.0MB/s), 61.0MiB/s-61.0MiB/s (64.0MB/s-64.0MB/s), io=1864MiB (1955MB), run=30089-30089msec Disk stats (read/write): vdc: ios=474874/119316, merge=0/0, ticks=108240/66612, in_queue=268502, util=99.17% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902764: Fri Jan 7 13:41:58 2022 read: IOPS=11.4k, BW=179MiB/s (187MB/s)(5357MiB/30001msec) clat (usec): min=26, max=2495, avg=52.13, stdev=24.69 lat (usec): min=26, max=2495, avg=52.26, stdev=24.69 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 71], | 99.00th=[ 196], 99.50th=[ 215], 99.90th=[ 269], 99.95th=[ 306], | 99.99th=[ 449] bw ( KiB/s): min=168480, max=192224, per=100.00%, avg=182840.14, stdev=4815.94, samples=59 iops : min=10530, max=12014, avg=11427.51, stdev=301.00, samples=59 write: IOPS=2868, BW=44.8MiB/s (46.0MB/s)(1345MiB/30001msec) clat (usec): min=102, max=3407, avg=132.67, stdev=22.99 lat (usec): min=104, max=3408, avg=134.02, stdev=23.02 clat percentiles (usec): | 1.00th=[ 114], 5.00th=[ 118], 10.00th=[ 120], 20.00th=[ 123], | 30.00th=[ 126], 40.00th=[ 128], 50.00th=[ 130], 60.00th=[ 133], | 70.00th=[ 137], 80.00th=[ 141], 90.00th=[ 149], 95.00th=[ 155], | 99.00th=[ 180], 99.50th=[ 192], 99.90th=[ 258], 99.95th=[ 396], | 99.99th=[ 1020] bw ( KiB/s): min=43392, max=48128, per=100.00%, avg=45919.41, stdev=1088.63, samples=59 iops : min= 2712, max= 3008, avg=2869.95, stdev=68.03, samples=59 lat (usec) : 50=48.03%, 100=30.43%, 250=21.41%, 500=0.12%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=6.20%, sys=18.53%, ctx=428901, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=342838,86051,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=179MiB/s (187MB/s), 179MiB/s-179MiB/s (187MB/s-187MB/s), io=5357MiB (5617MB), run=30001-30001msec WRITE: bw=44.8MiB/s (46.0MB/s), 44.8MiB/s-44.8MiB/s (46.0MB/s-46.0MB/s), io=1345MiB (1410MB), run=30001-30001msec Disk stats (read/write): vdc: ios=341006/85608, merge=0/0, ticks=12/10, in_queue=22743, util=75.96% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903785: Fri Jan 7 13:42:35 2022 read: IOPS=18.1k, BW=282MiB/s (296MB/s)(8476MiB/30027msec) slat (usec): min=2, max=340, avg= 5.21, stdev= 7.74 clat (usec): min=56, max=56326, avg=2079.11, stdev=4459.20 lat (usec): min=81, max=56393, avg=2084.40, stdev=4459.11 clat percentiles (usec): | 1.00th=[ 217], 5.00th=[ 285], 10.00th=[ 326], 20.00th=[ 392], | 30.00th=[ 445], 40.00th=[ 502], 50.00th=[ 562], 60.00th=[ 619], | 70.00th=[ 693], 80.00th=[ 816], 90.00th=[ 8717], 95.00th=[13698], | 99.00th=[19530], 99.50th=[22152], 99.90th=[28705], 99.95th=[37487], | 99.99th=[41157] bw ( KiB/s): min=31232, max=1014016, per=100.00%, avg=289344.68, stdev=403124.51, samples=60 iops : min= 1952, max=63374, avg=18084.00, stdev=25195.25, samples=60 write: IOPS=4532, BW=70.8MiB/s (74.3MB/s)(2127MiB/30027msec) slat (usec): min=2, max=427, avg= 8.32, stdev= 9.82 clat (usec): min=228, max=58571, avg=2710.91, stdev=4684.38 lat (usec): min=236, max=58590, avg=2719.33, stdev=4684.19 clat percentiles (usec): | 1.00th=[ 490], 5.00th=[ 660], 10.00th=[ 775], 20.00th=[ 947], | 30.00th=[ 1029], 40.00th=[ 1090], 50.00th=[ 1156], 60.00th=[ 1221], | 70.00th=[ 1303], 80.00th=[ 1418], 90.00th=[ 9896], 95.00th=[14615], | 99.00th=[20841], 99.50th=[23725], 99.90th=[32375], 99.95th=[41157], | 99.99th=[55837] bw ( KiB/s): min= 7168, max=256288, per=100.00%, avg=72599.93, stdev=101326.41, samples=60 iops : min= 448, max=16018, avg=4537.47, stdev=6332.90, samples=60 lat (usec) : 100=0.01%, 250=1.95%, 500=30.07%, 750=30.32%, 1000=11.22% lat (msec) : 2=14.02%, 4=0.18%, 10=3.21%, 20=8.06%, 50=0.96% lat (msec) : 100=0.01% cpu : usr=4.18%, sys=13.48%, ctx=14151, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=542444,136104,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=282MiB/s (296MB/s), 282MiB/s-282MiB/s (296MB/s-296MB/s), io=8476MiB (8887MB), run=30027-30027msec WRITE: bw=70.8MiB/s (74.3MB/s), 70.8MiB/s-70.8MiB/s (74.3MB/s-74.3MB/s), io=2127MiB (2230MB), run=30027-30027msec Disk stats (read/write): vdc: ios=541819/135924, merge=0/0, ticks=815214/254036, in_queue=1357904, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904247: Fri Jan 7 13:43:12 2022 read: IOPS=11.4k, BW=178MiB/s (186MB/s)(5328MiB/30001msec) clat (usec): min=27, max=26786, avg=52.56, stdev=51.98 lat (usec): min=27, max=26786, avg=52.68, stdev=51.99 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 57], 90.00th=[ 63], 95.00th=[ 71], | 99.00th=[ 196], 99.50th=[ 215], 99.90th=[ 273], 99.95th=[ 314], | 99.99th=[ 506] bw ( KiB/s): min=158924, max=191808, per=100.00%, avg=183079.83, stdev=6204.93, samples=59 iops : min= 9932, max=11988, avg=11442.31, stdev=387.79, samples=59 write: IOPS=2853, BW=44.6MiB/s (46.7MB/s)(1337MiB/30001msec) clat (usec): min=101, max=2594, avg=132.91, stdev=21.18 lat (usec): min=105, max=2596, avg=134.28, stdev=21.21 clat percentiles (usec): | 1.00th=[ 114], 5.00th=[ 118], 10.00th=[ 120], 20.00th=[ 124], | 30.00th=[ 126], 40.00th=[ 128], 50.00th=[ 131], 60.00th=[ 133], | 70.00th=[ 137], 80.00th=[ 141], 90.00th=[ 149], 95.00th=[ 155], | 99.00th=[ 178], 99.50th=[ 192], 99.90th=[ 251], 99.95th=[ 379], | 99.99th=[ 1020] bw ( KiB/s): min=39296, max=49088, per=100.00%, avg=45981.44, stdev=1848.93, samples=59 iops : min= 2456, max= 3068, avg=2873.66, stdev=115.59, samples=59 lat (usec) : 50=46.98%, 100=31.45%, 250=21.43%, 500=0.12%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 50=0.01% cpu : usr=6.02%, sys=18.63%, ctx=426574, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=340965,85597,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=178MiB/s (186MB/s), 178MiB/s-178MiB/s (186MB/s-186MB/s), io=5328MiB (5586MB), run=30001-30001msec WRITE: bw=44.6MiB/s (46.7MB/s), 44.6MiB/s-44.6MiB/s (46.7MB/s-46.7MB/s), io=1337MiB (1402MB), run=30001-30001msec Disk stats (read/write): vdc: ios=340827/85560, merge=0/0, ticks=38/9, in_queue=22843, util=75.93% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905221: Fri Jan 7 13:43:49 2022 read: IOPS=20.2k, BW=316MiB/s (332MB/s)(9503MiB/30045msec) slat (usec): min=2, max=283, avg= 4.94, stdev= 8.65 clat (usec): min=163, max=66853, avg=3710.83, stdev=8102.30 lat (usec): min=168, max=66943, avg=3715.83, stdev=8102.32 clat percentiles (usec): | 1.00th=[ 420], 5.00th=[ 545], 10.00th=[ 627], 20.00th=[ 750], | 30.00th=[ 857], 40.00th=[ 963], 50.00th=[ 1074], 60.00th=[ 1172], | 70.00th=[ 1303], 80.00th=[ 1467], 90.00th=[15926], 95.00th=[25560], | 99.00th=[35914], 99.50th=[39584], 99.90th=[52167], 99.95th=[57410], | 99.99th=[65799] bw ( KiB/s): min=39004, max=1119424, per=100.00%, avg=324388.37, stdev=435350.95, samples=60 iops : min= 2437, max=69964, avg=20274.12, stdev=27209.54, samples=60 write: IOPS=5079, BW=79.4MiB/s (83.2MB/s)(2385MiB/30045msec) slat (usec): min=3, max=269, avg= 7.94, stdev= 8.66 clat (usec): min=497, max=90943, avg=4866.82, stdev=8207.36 lat (usec): min=506, max=91049, avg=4874.83, stdev=8207.60 clat percentiles (usec): | 1.00th=[ 1029], 5.00th=[ 1319], 10.00th=[ 1549], 20.00th=[ 1893], | 30.00th=[ 2057], 40.00th=[ 2180], 50.00th=[ 2311], 60.00th=[ 2409], | 70.00th=[ 2507], 80.00th=[ 2671], 90.00th=[17171], 95.00th=[26608], | 99.00th=[36963], 99.50th=[41157], 99.90th=[57410], 99.95th=[60556], | 99.99th=[90702] bw ( KiB/s): min=10538, max=278624, per=100.00%, avg=81401.53, stdev=109163.59, samples=60 iops : min= 658, max=17414, avg=5087.38, stdev=6822.86, samples=60 lat (usec) : 250=0.02%, 500=2.32%, 750=13.72%, 1000=18.90% lat (msec) : 2=41.27%, 4=12.91%, 10=0.07%, 20=2.18%, 50=8.48% lat (msec) : 100=0.12% cpu : usr=3.94%, sys=13.54%, ctx=8562, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=608200,152617,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=316MiB/s (332MB/s), 316MiB/s-316MiB/s (332MB/s-332MB/s), io=9503MiB (9965MB), run=30045-30045msec WRITE: bw=79.4MiB/s (83.2MB/s), 79.4MiB/s-79.4MiB/s (83.2MB/s-83.2MB/s), io=2385MiB (2500MB), run=30045-30045msec Disk stats (read/write): vdc: ios=608117/152600, merge=0/0, ticks=1730550/598900, in_queue=2764566, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905797: Fri Jan 7 13:44:26 2022 read: IOPS=11.5k, BW=179MiB/s (188MB/s)(5377MiB/30001msec) clat (usec): min=25, max=17851, avg=51.91, stdev=49.07 lat (usec): min=25, max=17851, avg=52.03, stdev=49.08 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 62], 95.00th=[ 69], | 99.00th=[ 196], 99.50th=[ 215], 99.90th=[ 269], 99.95th=[ 306], | 99.99th=[ 478] bw ( KiB/s): min=161152, max=195360, per=99.99%, avg=183495.97, stdev=6104.96, samples=59 iops : min=10072, max=12210, avg=11468.49, stdev=381.56, samples=59 write: IOPS=2878, BW=44.0MiB/s (47.2MB/s)(1349MiB/30001msec) clat (usec): min=103, max=2229, avg=132.37, stdev=22.83 lat (usec): min=105, max=2230, avg=133.74, stdev=22.85 clat percentiles (usec): | 1.00th=[ 115], 5.00th=[ 118], 10.00th=[ 120], 20.00th=[ 123], | 30.00th=[ 126], 40.00th=[ 128], 50.00th=[ 130], 60.00th=[ 133], | 70.00th=[ 137], 80.00th=[ 141], 90.00th=[ 147], 95.00th=[ 153], | 99.00th=[ 176], 99.50th=[ 190], 99.90th=[ 258], 99.95th=[ 490], | 99.99th=[ 1139] bw ( KiB/s): min=39840, max=49344, per=100.00%, avg=46070.68, stdev=1675.58, samples=59 iops : min= 2490, max= 3084, avg=2879.41, stdev=104.73, samples=59 lat (usec) : 50=49.46%, 100=29.03%, 250=21.38%, 500=0.11%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01% cpu : usr=6.06%, sys=18.58%, ctx=430486, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=344111,86366,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=5377MiB (5638MB), run=30001-30001msec WRITE: bw=44.0MiB/s (47.2MB/s), 44.0MiB/s-44.0MiB/s (47.2MB/s-47.2MB/s), io=1349MiB (1415MB), run=30001-30001msec Disk stats (read/write): vdc: ios=341421/85726, merge=0/0, ticks=26/12, in_queue=22774, util=76.24% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906619: Fri Jan 7 13:45:03 2022 read: IOPS=22.4k, BW=351MiB/s (368MB/s)(10.3GiB/30044msec) slat (usec): min=2, max=114212, avg=33.53, stdev=918.45 clat (usec): min=56, max=165950, avg=7169.94, stdev=17025.99 lat (usec): min=615, max=201060, avg=7203.53, stdev=17129.13 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 1074], 10.00th=[ 1139], 20.00th=[ 1254], | 30.00th=[ 1385], 40.00th=[ 1893], 50.00th=[ 2212], 60.00th=[ 2343], | 70.00th=[ 2474], 80.00th=[ 2606], 90.00th=[ 3425], 95.00th=[ 58983], | 99.00th=[ 77071], 99.50th=[ 82314], 99.90th=[ 95945], 99.95th=[ 99091], | 99.99th=[164627] bw ( KiB/s): min=39840, max=1215936, per=100.00%, avg=359536.68, stdev=486343.19, samples=60 iops : min= 2490, max=75998, avg=22471.05, stdev=30396.52, samples=60 write: IOPS=5634, BW=88.0MiB/s (92.3MB/s)(2645MiB/30044msec) slat (usec): min=3, max=56549, avg=37.97, stdev=957.68 clat (usec): min=1197, max=115425, avg=6760.43, stdev=9967.69 lat (usec): min=1205, max=129890, avg=6798.47, stdev=10158.47 clat percentiles (usec): | 1.00th=[ 1811], 5.00th=[ 3163], 10.00th=[ 3326], 20.00th=[ 3458], | 30.00th=[ 3556], 40.00th=[ 3621], 50.00th=[ 3687], 60.00th=[ 3785], | 70.00th=[ 3884], 80.00th=[ 4047], 90.00th=[ 5276], 95.00th=[ 33424], | 99.00th=[ 45876], 99.50th=[ 53740], 99.90th=[ 73925], 99.95th=[ 79168], | 99.99th=[114820] bw ( KiB/s): min= 9312, max=309568, per=100.00%, avg=90287.80, stdev=122043.67, samples=60 iops : min= 582, max=19348, avg=5642.97, stdev=7627.73, samples=60 lat (usec) : 100=0.01%, 750=0.02%, 1000=1.48% lat (msec) : 2=31.90%, 4=54.43%, 10=2.34%, 20=0.12%, 50=4.45% lat (msec) : 100=5.20%, 250=0.04% cpu : usr=4.12%, sys=12.28%, ctx=7738, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=674238,169296,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=10.3GiB (11.0GB), run=30044-30044msec WRITE: bw=88.0MiB/s (92.3MB/s), 88.0MiB/s-88.0MiB/s (92.3MB/s-92.3MB/s), io=2645MiB (2774MB), run=30044-30044msec Disk stats (read/write): vdc: ios=674133/169272, merge=0/0, ticks=2302353/810965, in_queue=3553309, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908079: Fri Jan 7 13:45:40 2022 read: IOPS=8828, BW=276MiB/s (289MB/s)(8279MiB/30009msec) clat (usec): min=30, max=1742, avg=58.19, stdev=29.77 lat (usec): min=30, max=1742, avg=58.31, stdev=29.77 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 70], 95.00th=[ 77], | 99.00th=[ 233], 99.50th=[ 253], 99.90th=[ 338], 99.95th=[ 392], | 99.99th=[ 619] bw ( KiB/s): min=171712, max=324544, per=100.00%, avg=282603.73, stdev=35187.54, samples=60 iops : min= 5366, max=10142, avg=8831.33, stdev=1099.60, samples=60 write: IOPS=2210, BW=69.1MiB/s (72.4MB/s)(2073MiB/30009msec) clat (usec): min=142, max=9888, avg=210.21, stdev=342.18 lat (usec): min=145, max=9909, avg=213.10, stdev=342.20 clat percentiles (usec): | 1.00th=[ 155], 5.00th=[ 159], 10.00th=[ 161], 20.00th=[ 163], | 30.00th=[ 167], 40.00th=[ 169], 50.00th=[ 172], 60.00th=[ 176], | 70.00th=[ 178], 80.00th=[ 184], 90.00th=[ 192], 95.00th=[ 202], | 99.00th=[ 1336], 99.50th=[ 2474], 99.90th=[ 5604], 99.95th=[ 6718], | 99.99th=[ 8979] bw ( KiB/s): min=40064, max=80000, per=100.00%, avg=70745.60, stdev=9133.80, samples=60 iops : min= 1252, max= 2500, avg=2210.80, stdev=285.43, samples=60 lat (usec) : 50=30.87%, 100=47.30%, 250=20.96%, 500=0.45%, 750=0.02% lat (usec) : 1000=0.12% lat (msec) : 2=0.15%, 4=0.08%, 10=0.05% cpu : usr=4.97%, sys=15.02%, ctx=331274, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=264941,66325,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=8279MiB (8682MB), run=30009-30009msec WRITE: bw=69.1MiB/s (72.4MB/s), 69.1MiB/s-69.1MiB/s (72.4MB/s-72.4MB/s), io=2073MiB (2173MB), run=30009-30009msec Disk stats (read/write): vdc: ios=263965/66094, merge=0/0, ticks=11/1809, in_queue=24082, util=80.50% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908592: Fri Jan 7 13:46:17 2022 read: IOPS=9470, BW=296MiB/s (310MB/s)(8881MiB/30010msec) slat (usec): min=5, max=1627, avg=10.33, stdev= 4.58 clat (nsec): min=692, max=2184.8k, avg=45838.27, stdev=29834.51 lat (usec): min=25, max=2193, avg=56.30, stdev=30.35 clat percentiles (usec): | 1.00th=[ 30], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 37], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 43], | 70.00th=[ 46], 80.00th=[ 49], 90.00th=[ 56], 95.00th=[ 63], | 99.00th=[ 221], 99.50th=[ 241], 99.90th=[ 334], 99.95th=[ 400], | 99.99th=[ 545] bw ( KiB/s): min=175488, max=339520, per=100.00%, avg=303177.32, stdev=39365.13, samples=60 iops : min= 5482, max=10610, avg=9474.22, stdev=1230.30, samples=60 write: IOPS=2375, BW=74.2MiB/s (77.8MB/s)(2228MiB/30010msec) slat (usec): min=8, max=245, avg=13.71, stdev= 4.01 clat (usec): min=103, max=14418, avg=176.40, stdev=168.79 lat (usec): min=143, max=14431, avg=190.24, stdev=168.98 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 155], 50.00th=[ 157], 60.00th=[ 159], | 70.00th=[ 163], 80.00th=[ 167], 90.00th=[ 176], 95.00th=[ 184], | 99.00th=[ 881], 99.50th=[ 1303], 99.90th=[ 2311], 99.95th=[ 2999], | 99.99th=[ 4047] bw ( KiB/s): min=46464, max=84288, per=100.00%, avg=76056.92, stdev=9507.67, samples=60 iops : min= 1452, max= 2634, avg=2376.75, stdev=297.11, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.05%, 50=66.10% lat (usec) : 100=12.08%, 250=21.02%, 500=0.32%, 750=0.02%, 1000=0.17% lat (msec) : 2=0.12%, 4=0.04%, 10=0.01%, 20=0.01% cpu : usr=5.28%, sys=25.04%, ctx=355188, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=284203,71298,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=296MiB/s (310MB/s), 296MiB/s-296MiB/s (310MB/s-310MB/s), io=8881MiB (9313MB), run=30010-30010msec WRITE: bw=74.2MiB/s (77.8MB/s), 74.2MiB/s-74.2MiB/s (77.8MB/s-77.8MB/s), io=2228MiB (2336MB), run=30010-30010msec Disk stats (read/write): vdc: ios=283364/71080, merge=0/0, ticks=9/669, in_queue=23238, util=77.58% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909516: Fri Jan 7 13:46:54 2022 read: IOPS=9793, BW=306MiB/s (321MB/s)(9183MiB/30005msec) clat (usec): min=27, max=2186, avg=55.08, stdev=30.62 lat (usec): min=27, max=2186, avg=55.20, stdev=30.63 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 65], 95.00th=[ 73], | 99.00th=[ 231], 99.50th=[ 249], 99.90th=[ 347], 99.95th=[ 420], | 99.99th=[ 570] bw ( KiB/s): min=186375, max=340096, per=100.00%, avg=315067.68, stdev=29826.72, samples=60 iops : min= 5824, max=10628, avg=9845.43, stdev=932.06, samples=60 write: IOPS=2459, BW=76.9MiB/s (80.6MB/s)(2306MiB/30005msec) clat (usec): min=141, max=8869, avg=177.81, stdev=115.93 lat (usec): min=144, max=8872, avg=180.68, stdev=115.94 clat percentiles (usec): | 1.00th=[ 151], 5.00th=[ 155], 10.00th=[ 157], 20.00th=[ 161], | 30.00th=[ 163], 40.00th=[ 165], 50.00th=[ 167], 60.00th=[ 169], | 70.00th=[ 174], 80.00th=[ 178], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 241], 99.50th=[ 857], 99.90th=[ 1778], 99.95th=[ 2212], | 99.99th=[ 3949] bw ( KiB/s): min=47709, max=85054, per=100.00%, avg=79127.30, stdev=7109.98, samples=60 iops : min= 1490, max= 2657, avg=2472.38, stdev=222.27, samples=60 lat (usec) : 50=41.46%, 100=36.73%, 250=21.22%, 500=0.41%, 750=0.02% lat (usec) : 1000=0.08% lat (msec) : 2=0.07%, 4=0.01%, 10=0.01% cpu : usr=5.43%, sys=16.32%, ctx=367667, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=293853,73800,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=306MiB/s (321MB/s), 306MiB/s-306MiB/s (321MB/s-321MB/s), io=9183MiB (9629MB), run=30005-30005msec WRITE: bw=76.9MiB/s (80.6MB/s), 76.9MiB/s-76.9MiB/s (80.6MB/s-80.6MB/s), io=2306MiB (2418MB), run=30005-30005msec Disk stats (read/write): vdc: ios=293853/73799, merge=0/0, ticks=13/349, in_queue=23684, util=78.71% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910180: Fri Jan 7 13:47:31 2022 read: IOPS=13.0k, BW=406MiB/s (426MB/s)(11.9GiB/30032msec) slat (usec): min=2, max=1061, avg= 6.87, stdev= 7.24 clat (nsec): min=709, max=33452k, avg=466792.22, stdev=907146.74 lat (usec): min=25, max=33457, avg=473.77, stdev=907.02 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 67], 10.00th=[ 80], 20.00th=[ 97], | 30.00th=[ 114], 40.00th=[ 129], 50.00th=[ 145], 60.00th=[ 165], | 70.00th=[ 192], 80.00th=[ 302], 90.00th=[ 1500], 95.00th=[ 2442], | 99.00th=[ 4228], 99.50th=[ 4883], 99.90th=[ 6259], 99.95th=[ 7111], | 99.99th=[15533] bw ( KiB/s): min=103488, max=1314688, per=100.00%, avg=416609.72, stdev=490621.42, samples=60 iops : min= 3234, max=41084, avg=13019.03, stdev=15331.91, samples=60 write: IOPS=3265, BW=102MiB/s (107MB/s)(3065MiB/30032msec) slat (usec): min=3, max=568, avg=12.31, stdev=10.53 clat (usec): min=116, max=34163, avg=1158.30, stdev=1696.89 lat (usec): min=146, max=34170, avg=1170.74, stdev=1696.27 clat percentiles (usec): | 1.00th=[ 180], 5.00th=[ 241], 10.00th=[ 277], 20.00th=[ 322], | 30.00th=[ 363], 40.00th=[ 396], 50.00th=[ 437], 60.00th=[ 482], | 70.00th=[ 562], 80.00th=[ 1598], 90.00th=[ 3720], 95.00th=[ 5080], | 99.00th=[ 7177], 99.50th=[ 8160], 99.90th=[10421], 99.95th=[12518], | 99.99th=[28967] bw ( KiB/s): min=26304, max=337920, per=100.00%, avg=104611.68, stdev=123216.09, samples=60 iops : min= 822, max=10560, avg=3269.10, stdev=3850.51, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.02%, 50=0.94%, 100=16.46% lat (usec) : 250=46.37%, 500=14.04%, 750=5.31%, 1000=1.75% lat (msec) : 2=5.78%, 4=6.61%, 10=2.69%, 20=0.03%, 50=0.01% cpu : usr=4.97%, sys=13.90%, ctx=63805, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=390537,98066,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=406MiB/s (426MB/s), 406MiB/s-406MiB/s (426MB/s-426MB/s), io=11.9GiB (12.8GB), run=30032-30032msec WRITE: bw=102MiB/s (107MB/s), 102MiB/s-102MiB/s (107MB/s-107MB/s), io=3065MiB (3213MB), run=30032-30032msec Disk stats (read/write): vdc: ios=389967/97901, merge=0/0, ticks=96199/69983, in_queue=270643, util=99.31% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910984: Fri Jan 7 13:48:08 2022 read: IOPS=9884, BW=309MiB/s (324MB/s)(9267MiB/30001msec) clat (usec): min=28, max=2279, avg=55.83, stdev=32.53 lat (usec): min=28, max=2279, avg=55.95, stdev=32.54 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 67], 95.00th=[ 75], | 99.00th=[ 231], 99.50th=[ 253], 99.90th=[ 367], 99.95th=[ 457], | 99.99th=[ 873] bw ( KiB/s): min=278080, max=335360, per=99.98%, avg=316251.24, stdev=10855.84, samples=59 iops : min= 8690, max=10480, avg=9882.85, stdev=339.24, samples=59 write: IOPS=2481, BW=77.6MiB/s (81.3MB/s)(2327MiB/30001msec) clat (usec): min=140, max=1640, avg=170.52, stdev=23.80 lat (usec): min=143, max=1643, avg=173.38, stdev=23.84 clat percentiles (usec): | 1.00th=[ 151], 5.00th=[ 155], 10.00th=[ 157], 20.00th=[ 161], | 30.00th=[ 163], 40.00th=[ 165], 50.00th=[ 167], 60.00th=[ 172], | 70.00th=[ 174], 80.00th=[ 178], 90.00th=[ 186], 95.00th=[ 194], | 99.00th=[ 221], 99.50th=[ 237], 99.90th=[ 461], 99.95th=[ 586], | 99.99th=[ 1221] bw ( KiB/s): min=70208, max=85952, per=99.99%, avg=79415.86, stdev=3073.77, samples=59 iops : min= 2194, max= 2686, avg=2481.75, stdev=96.06, samples=59 lat (usec) : 50=39.88%, 100=38.19%, 250=21.43%, 500=0.45%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=5.84%, sys=16.86%, ctx=371032, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=296558,74459,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=309MiB/s (324MB/s), 309MiB/s-309MiB/s (324MB/s-324MB/s), io=9267MiB (9718MB), run=30001-30001msec WRITE: bw=77.6MiB/s (81.3MB/s), 77.6MiB/s-77.6MiB/s (81.3MB/s-81.3MB/s), io=2327MiB (2440MB), run=30001-30001msec Disk stats (read/write): vdc: ios=294931/74055, merge=0/0, ticks=20/10, in_queue=23375, util=78.09% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911956: Fri Jan 7 13:48:45 2022 read: IOPS=15.8k, BW=494MiB/s (518MB/s)(14.5GiB/30033msec) slat (usec): min=2, max=356, avg= 6.74, stdev= 8.52 clat (nsec): min=631, max=52569k, avg=2004122.54, stdev=4202333.62 lat (usec): min=20, max=52639, avg=2010.98, stdev=4201.85 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 58], 10.00th=[ 72], 20.00th=[ 95], | 30.00th=[ 118], 40.00th=[ 145], 50.00th=[ 180], 60.00th=[ 237], | 70.00th=[ 347], 80.00th=[ 1057], 90.00th=[ 8979], 95.00th=[11600], | 99.00th=[16319], 99.50th=[18220], 99.90th=[30802], 99.95th=[40109], | 99.99th=[49546] bw ( KiB/s): min=61504, max=1563776, per=100.00%, avg=506177.07, stdev=621100.20, samples=60 iops : min= 1922, max=48868, avg=15818.03, stdev=19409.38, samples=60 write: IOPS=3970, BW=124MiB/s (130MB/s)(3726MiB/30033msec) slat (usec): min=4, max=561, avg=14.22, stdev=14.05 clat (usec): min=501, max=58760, avg=4569.55, stdev=3759.22 lat (usec): min=509, max=58768, avg=4583.92, stdev=3757.81 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 2057], 10.00th=[ 2442], 20.00th=[ 2802], | 30.00th=[ 2933], 40.00th=[ 3097], 50.00th=[ 3228], 60.00th=[ 3392], | 70.00th=[ 3621], 80.00th=[ 4555], 90.00th=[10159], 95.00th=[12780], | 99.00th=[17433], 99.50th=[20055], 99.90th=[34341], 99.95th=[45351], | 99.99th=[55837] bw ( KiB/s): min=15424, max=392704, per=100.00%, avg=127186.13, stdev=155772.99, samples=60 iops : min= 482, max=12272, avg=3974.57, stdev=4867.91, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=2.33% lat (usec) : 100=15.57%, 250=31.42%, 500=11.24%, 750=2.48%, 1000=0.82% lat (msec) : 2=1.63%, 4=15.98%, 10=10.05%, 20=8.06%, 50=0.37% lat (msec) : 100=0.01% cpu : usr=6.79%, sys=15.93%, ctx=46875, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=474583,119245,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=494MiB/s (518MB/s), 494MiB/s-494MiB/s (518MB/s-518MB/s), io=14.5GiB (15.6GB), run=30033-30033msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3726MiB (3907MB), run=30033-30033msec Disk stats (read/write): vdc: ios=473908/119074, merge=0/0, ticks=801249/464014, in_queue=1407344, util=99.60% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912591: Fri Jan 7 13:49:22 2022 read: IOPS=10.1k, BW=315MiB/s (330MB/s)(9448MiB/30001msec) clat (usec): min=29, max=1858, avg=54.44, stdev=30.07 lat (usec): min=29, max=1858, avg=54.56, stdev=30.08 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 65], 95.00th=[ 73], | 99.00th=[ 229], 99.50th=[ 251], 99.90th=[ 351], 99.95th=[ 416], | 99.99th=[ 562] bw ( KiB/s): min=297536, max=338560, per=100.00%, avg=322479.68, stdev=9220.58, samples=59 iops : min= 9298, max=10580, avg=10077.47, stdev=288.15, samples=59 write: IOPS=2528, BW=79.0MiB/s (82.8MB/s)(2370MiB/30001msec) clat (usec): min=140, max=1796, avg=169.10, stdev=22.03 lat (usec): min=143, max=1799, avg=171.96, stdev=22.08 clat percentiles (usec): | 1.00th=[ 151], 5.00th=[ 153], 10.00th=[ 157], 20.00th=[ 159], | 30.00th=[ 161], 40.00th=[ 163], 50.00th=[ 167], 60.00th=[ 169], | 70.00th=[ 174], 80.00th=[ 178], 90.00th=[ 184], 95.00th=[ 192], | 99.00th=[ 219], 99.50th=[ 235], 99.90th=[ 400], 99.95th=[ 545], | 99.99th=[ 979] bw ( KiB/s): min=74560, max=86016, per=100.00%, avg=80952.90, stdev=1961.72, samples=59 iops : min= 2330, max= 2688, avg=2529.76, stdev=61.32, samples=59 lat (usec) : 50=44.39%, 100=33.73%, 250=21.41%, 500=0.44%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01% cpu : usr=5.91%, sys=16.38%, ctx=378178, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=302320,75845,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=315MiB/s (330MB/s), 315MiB/s-315MiB/s (330MB/s-330MB/s), io=9448MiB (9906MB), run=30001-30001msec WRITE: bw=79.0MiB/s (82.8MB/s), 79.0MiB/s-79.0MiB/s (82.8MB/s-82.8MB/s), io=2370MiB (2485MB), run=30001-30001msec Disk stats (read/write): vdc: ios=300004/75334, merge=0/0, ticks=7/7, in_queue=23537, util=78.79% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913426: Fri Jan 7 13:49:59 2022 read: IOPS=18.6k, BW=582MiB/s (611MB/s)(17.1GiB/30039msec) slat (usec): min=2, max=341, avg= 6.72, stdev= 6.73 clat (nsec): min=690, max=57067k, avg=3065234.63, stdev=6960777.54 lat (usec): min=24, max=57072, avg=3072.04, stdev=6960.46 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 57], 10.00th=[ 70], 20.00th=[ 91], | 30.00th=[ 112], 40.00th=[ 135], 50.00th=[ 161], 60.00th=[ 200], | 70.00th=[ 262], 80.00th=[ 478], 90.00th=[16450], 95.00th=[20317], | 99.00th=[26346], 99.50th=[28443], 99.90th=[36439], 99.95th=[50070], | 99.99th=[56886] bw ( KiB/s): min=102144, max=1564608, per=100.00%, avg=597070.27, stdev=657665.80, samples=60 iops : min= 3192, max=48894, avg=18658.43, stdev=20552.07, samples=60 write: IOPS=4673, BW=146MiB/s (153MB/s)(4387MiB/30039msec) slat (usec): min=4, max=416, avg=11.83, stdev= 8.68 clat (usec): min=934, max=77828, avg=9130.11, stdev=5371.31 lat (usec): min=940, max=77849, avg=9142.04, stdev=5371.69 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3458], 10.00th=[ 4228], 20.00th=[ 5538], | 30.00th=[ 6521], 40.00th=[ 7111], 50.00th=[ 7570], 60.00th=[ 8160], | 70.00th=[ 9241], 80.00th=[11469], 90.00th=[17171], 95.00th=[20841], | 99.00th=[26608], 99.50th=[29230], 99.90th=[47973], 99.95th=[54789], | 99.99th=[77071] bw ( KiB/s): min=25920, max=386752, per=100.00%, avg=149731.25, stdev=165402.61, samples=60 iops : min= 810, max=12086, avg=4679.10, stdev=5168.83, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=2.50%, 100=17.03% lat (usec) : 250=35.30%, 500=9.39%, 750=1.25%, 1000=0.40% lat (msec) : 2=0.45%, 4=2.87%, 10=13.72%, 20=11.58%, 50=5.46% lat (msec) : 100=0.05% cpu : usr=6.00%, sys=18.74%, ctx=65452, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=559832,140389,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=582MiB/s (611MB/s), 582MiB/s-582MiB/s (611MB/s-611MB/s), io=17.1GiB (18.3GB), run=30039-30039msec WRITE: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=4387MiB (4600MB), run=30039-30039msec Disk stats (read/write): vdc: ios=558867/140154, merge=0/0, ticks=1522061/1192146, in_queue=2881482, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914735: Fri Jan 7 13:50:36 2022 read: IOPS=10.0k, BW=313MiB/s (328MB/s)(9391MiB/30001msec) clat (usec): min=28, max=2529, avg=54.79, stdev=30.58 lat (usec): min=28, max=2529, avg=54.90, stdev=30.58 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 65], 95.00th=[ 73], | 99.00th=[ 229], 99.50th=[ 251], 99.90th=[ 347], 99.95th=[ 420], | 99.99th=[ 619] bw ( KiB/s): min=294656, max=339512, per=100.00%, avg=322473.29, stdev=10160.71, samples=59 iops : min= 9208, max=10609, avg=10077.10, stdev=317.39, samples=59 write: IOPS=2514, BW=78.6MiB/s (82.4MB/s)(2357MiB/30001msec) clat (usec): min=139, max=41079, avg=169.67, stdev=152.09 lat (usec): min=142, max=41082, avg=172.54, stdev=152.10 clat percentiles (usec): | 1.00th=[ 151], 5.00th=[ 155], 10.00th=[ 157], 20.00th=[ 159], | 30.00th=[ 161], 40.00th=[ 165], 50.00th=[ 167], 60.00th=[ 169], | 70.00th=[ 174], 80.00th=[ 178], 90.00th=[ 184], 95.00th=[ 190], | 99.00th=[ 212], 99.50th=[ 227], 99.90th=[ 359], 99.95th=[ 529], | 99.99th=[ 1090] bw ( KiB/s): min=70656, max=85333, per=100.00%, avg=80962.19, stdev=2645.76, samples=59 iops : min= 2208, max= 2666, avg=2529.92, stdev=82.59, samples=59 lat (usec) : 50=42.61%, 100=35.58%, 250=21.36%, 500=0.42%, 750=0.02% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=5.87%, sys=16.77%, ctx=375963, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=300515,75436,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=313MiB/s (328MB/s), 313MiB/s-313MiB/s (328MB/s-328MB/s), io=9391MiB (9847MB), run=30001-30001msec WRITE: bw=78.6MiB/s (82.4MB/s), 78.6MiB/s-78.6MiB/s (82.4MB/s-82.4MB/s), io=2357MiB (2472MB), run=30001-30001msec Disk stats (read/write): vdc: ios=300366/75406, merge=0/0, ticks=13/55, in_queue=23272, util=77.35% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915355: Fri Jan 7 13:51:13 2022 read: IOPS=21.1k, BW=660MiB/s (692MB/s)(19.3GiB/30041msec) slat (usec): min=2, max=92226, avg=36.65, stdev=791.47 clat (usec): min=56, max=136172, avg=6311.89, stdev=14154.33 lat (usec): min=718, max=176261, avg=6348.62, stdev=14242.37 clat percentiles (usec): | 1.00th=[ 930], 5.00th=[ 1020], 10.00th=[ 1074], 20.00th=[ 1156], | 30.00th=[ 1237], 40.00th=[ 1319], 50.00th=[ 1418], 60.00th=[ 1549], | 70.00th=[ 1745], 80.00th=[ 2180], 90.00th=[ 23200], 95.00th=[ 45876], | 99.00th=[ 60556], 99.50th=[ 65799], 99.90th=[ 93848], 99.95th=[100140], | 99.99th=[135267] bw ( KiB/s): min=99200, max=1563968, per=100.00%, avg=676225.53, stdev=681012.78, samples=60 iops : min= 3100, max=48874, avg=21132.05, stdev=21281.68, samples=60 write: IOPS=5298, BW=166MiB/s (174MB/s)(4974MiB/30041msec) slat (usec): min=3, max=40243, avg=35.44, stdev=664.22 clat (msec): min=2, max=120, avg=12.42, stdev= 6.70 lat (msec): min=2, max=120, avg=12.45, stdev= 6.83 clat percentiles (usec): | 1.00th=[ 4424], 5.00th=[ 5538], 10.00th=[ 6652], 20.00th=[ 8455], | 30.00th=[ 9634], 40.00th=[10290], 50.00th=[10814], 60.00th=[11469], | 70.00th=[12387], 80.00th=[14353], 90.00th=[20579], 95.00th=[26608], | 99.00th=[35914], 99.50th=[43779], 99.90th=[65799], 99.95th=[69731], | 99.99th=[93848] bw ( KiB/s): min=23744, max=389056, per=100.00%, avg=169792.60, stdev=170525.93, samples=60 iops : min= 742, max=12158, avg=5305.98, stdev=5328.91, samples=60 lat (usec) : 100=0.01%, 750=0.01%, 1000=3.02% lat (msec) : 2=58.54%, 4=7.59%, 10=8.38%, 20=11.66%, 50=7.95% lat (msec) : 100=2.83%, 250=0.04% cpu : usr=6.09%, sys=15.06%, ctx=48823, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=634037,159183,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=660MiB/s (692MB/s), 660MiB/s-660MiB/s (692MB/s-692MB/s), io=19.3GiB (20.8GB), run=30041-30041msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4974MiB (5216MB), run=30041-30041msec Disk stats (read/write): vdc: ios=633422/159027, merge=0/0, ticks=1792722/1676325, in_queue=3687359, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916229: Fri Jan 7 13:51:50 2022 read: IOPS=7283, BW=455MiB/s (477MB/s)(13.3GiB/30001msec) clat (usec): min=36, max=2574, avg=70.14, stdev=45.41 lat (usec): min=36, max=2574, avg=70.27, stdev=45.42 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 82], 95.00th=[ 96], | 99.00th=[ 306], 99.50th=[ 343], 99.90th=[ 498], 99.95th=[ 594], | 99.99th=[ 1254] bw ( KiB/s): min=423168, max=499328, per=99.97%, avg=465998.10, stdev=19542.44, samples=59 iops : min= 6612, max= 7802, avg=7281.22, stdev=305.35, samples=59 write: IOPS=1821, BW=114MiB/s (119MB/s)(3416MiB/30001msec) clat (usec): min=219, max=2694, avg=254.98, stdev=35.68 lat (usec): min=224, max=2700, avg=260.89, stdev=35.78 clat percentiles (usec): | 1.00th=[ 233], 5.00th=[ 237], 10.00th=[ 239], 20.00th=[ 243], | 30.00th=[ 245], 40.00th=[ 247], 50.00th=[ 251], 60.00th=[ 253], | 70.00th=[ 258], 80.00th=[ 265], 90.00th=[ 273], 95.00th=[ 281], | 99.00th=[ 322], 99.50th=[ 351], 99.90th=[ 660], 99.95th=[ 840], | 99.99th=[ 1713] bw ( KiB/s): min=103680, max=125568, per=100.00%, avg=116714.31, stdev=4905.47, samples=59 iops : min= 1620, max= 1962, avg=1823.66, stdev=76.65, samples=59 lat (usec) : 50=3.38%, 100=73.05%, 250=11.72%, 500=11.73%, 750=0.09% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01% cpu : usr=5.20%, sys=13.21%, ctx=273181, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=218514,54660,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=13.3GiB (14.3GB), run=30001-30001msec WRITE: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=3416MiB (3582MB), run=30001-30001msec Disk stats (read/write): vdc: ios=216836/54287, merge=0/0, ticks=27/23, in_queue=24275, util=81.27% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916847: Fri Jan 7 13:52:27 2022 read: IOPS=7524, BW=470MiB/s (493MB/s)(13.8GiB/30001msec) slat (usec): min=6, max=427, avg=12.39, stdev= 3.55 clat (nsec): min=738, max=4896.9k, avg=54872.84, stdev=44054.25 lat (usec): min=32, max=4922, avg=67.40, stdev=44.48 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 40], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 50], | 70.00th=[ 52], 80.00th=[ 57], 90.00th=[ 64], 95.00th=[ 74], | 99.00th=[ 289], 99.50th=[ 318], 99.90th=[ 474], 99.95th=[ 545], | 99.99th=[ 963] bw ( KiB/s): min=436352, max=535296, per=99.93%, avg=481249.29, stdev=19167.88, samples=59 iops : min= 6818, max= 8364, avg=7519.49, stdev=299.46, samples=59 write: IOPS=1885, BW=118MiB/s (124MB/s)(3536MiB/30001msec) slat (usec): min=10, max=505, avg=18.76, stdev= 5.11 clat (nsec): min=1748, max=5023.7k, avg=235889.09, stdev=32844.26 lat (usec): min=220, max=5048, avg=254.80, stdev=33.60 clat percentiles (usec): | 1.00th=[ 217], 5.00th=[ 221], 10.00th=[ 223], 20.00th=[ 227], | 30.00th=[ 229], 40.00th=[ 231], 50.00th=[ 233], 60.00th=[ 235], | 70.00th=[ 239], 80.00th=[ 243], 90.00th=[ 251], 95.00th=[ 260], | 99.00th=[ 285], 99.50th=[ 302], 99.90th=[ 537], 99.95th=[ 742], | 99.99th=[ 1434] bw ( KiB/s): min=113024, max=132096, per=99.89%, avg=120557.90, stdev=4469.19, samples=59 iops : min= 1766, max= 2064, avg=1883.69, stdev=69.85, samples=59 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=49.77% lat (usec) : 100=27.79%, 250=19.21%, 500=3.12%, 750=0.06%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=5.30%, sys=22.02%, ctx=282308, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=225750,56574,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=470MiB/s (493MB/s), 470MiB/s-470MiB/s (493MB/s-493MB/s), io=13.8GiB (14.8GB), run=30001-30001msec WRITE: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=3536MiB (3708MB), run=30001-30001msec Disk stats (read/write): vdc: ios=224252/56177, merge=0/0, ticks=26/19, in_queue=23517, util=78.66% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917793: Fri Jan 7 13:53:04 2022 read: IOPS=7782, BW=486MiB/s (510MB/s)(14.2GiB/30001msec) clat (usec): min=31, max=2006, avg=63.33, stdev=42.50 lat (usec): min=31, max=2006, avg=63.45, stdev=42.51 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 73], 95.00th=[ 84], | 99.00th=[ 297], 99.50th=[ 326], 99.90th=[ 478], 99.95th=[ 562], | 99.99th=[ 971] bw ( KiB/s): min=438272, max=536704, per=99.97%, avg=497938.07, stdev=20218.48, samples=59 iops : min= 6848, max= 8386, avg=7780.27, stdev=315.93, samples=59 write: IOPS=1949, BW=122MiB/s (128MB/s)(3656MiB/30001msec) clat (usec): min=212, max=21917, avg=246.90, stdev=107.49 lat (usec): min=217, max=21923, avg=252.84, stdev=107.53 clat percentiles (usec): | 1.00th=[ 225], 5.00th=[ 229], 10.00th=[ 231], 20.00th=[ 235], | 30.00th=[ 237], 40.00th=[ 241], 50.00th=[ 243], 60.00th=[ 245], | 70.00th=[ 249], 80.00th=[ 253], 90.00th=[ 262], 95.00th=[ 269], | 99.00th=[ 306], 99.50th=[ 330], 99.90th=[ 685], 99.95th=[ 1139], | 99.99th=[ 2573] bw ( KiB/s): min=112224, max=133504, per=100.00%, avg=124827.34, stdev=5107.85, samples=59 iops : min= 1753, max= 2086, avg=1950.41, stdev=79.85, samples=59 lat (usec) : 50=21.00%, 100=56.24%, 250=16.25%, 500=6.41%, 750=0.07% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=5.67%, sys=14.00%, ctx=291990, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=233482,58499,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=486MiB/s (510MB/s), 486MiB/s-486MiB/s (510MB/s-510MB/s), io=14.2GiB (15.3GB), run=30001-30001msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3656MiB (3834MB), run=30001-30001msec Disk stats (read/write): vdc: ios=231629/58038, merge=0/0, ticks=16/72, in_queue=23867, util=79.94% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918739: Fri Jan 7 13:53:41 2022 read: IOPS=11.4k, BW=713MiB/s (748MB/s)(20.9GiB/30033msec) slat (usec): min=3, max=1303, avg= 8.69, stdev= 8.33 clat (nsec): min=637, max=35514k, avg=464996.95, stdev=890165.16 lat (usec): min=23, max=35530, avg=473.80, stdev=889.97 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 54], 10.00th=[ 65], 20.00th=[ 83], | 30.00th=[ 100], 40.00th=[ 119], 50.00th=[ 141], 60.00th=[ 169], | 70.00th=[ 212], 80.00th=[ 437], 90.00th=[ 1434], 95.00th=[ 2245], | 99.00th=[ 4047], 99.50th=[ 4817], 99.90th=[ 6915], 99.95th=[ 8455], | 99.99th=[19006] bw ( KiB/s): min=98816, max=1626368, per=100.00%, avg=731161.80, stdev=602232.40, samples=60 iops : min= 1544, max=25412, avg=11424.38, stdev=9409.90, samples=60 write: IOPS=2864, BW=179MiB/s (188MB/s)(5377MiB/30033msec) slat (usec): min=6, max=882, avg=19.91, stdev=16.65 clat (usec): min=184, max=36955, avg=1578.18, stdev=1551.32 lat (usec): min=236, max=36977, avg=1598.23, stdev=1549.03 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 529], 10.00th=[ 644], 20.00th=[ 791], | 30.00th=[ 898], 40.00th=[ 988], 50.00th=[ 1074], 60.00th=[ 1172], | 70.00th=[ 1287], 80.00th=[ 1893], 90.00th=[ 3458], 95.00th=[ 4686], | 99.00th=[ 7308], 99.50th=[ 8291], 99.90th=[14877], 99.95th=[19530], | 99.99th=[32113] bw ( KiB/s): min=25216, max=402944, per=100.00%, avg=183519.57, stdev=150866.26, samples=60 iops : min= 394, max= 6296, avg=2867.47, stdev=2357.31, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.03%, 50=2.82% lat (usec) : 100=21.32%, 250=35.82%, 500=4.94%, 750=3.60%, 1000=8.63% lat (msec) : 2=14.01%, 4=6.48%, 10=2.27%, 20=0.06%, 50=0.02% cpu : usr=6.71%, sys=15.96%, ctx=97270, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=342718,86025,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=713MiB/s (748MB/s), 713MiB/s-713MiB/s (748MB/s-748MB/s), io=20.9GiB (22.5GB), run=30033-30033msec WRITE: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=5377MiB (5638MB), run=30033-30033msec Disk stats (read/write): vdc: ios=342009/85850, merge=0/0, ticks=80163/88454, in_queue=274339, util=99.55% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919319: Fri Jan 7 13:54:18 2022 read: IOPS=7932, BW=496MiB/s (520MB/s)(14.5GiB/30001msec) clat (usec): min=31, max=22075, avg=61.69, stdev=61.29 lat (usec): min=31, max=22075, avg=61.81, stdev=61.29 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 71], 95.00th=[ 81], | 99.00th=[ 293], 99.50th=[ 326], 99.90th=[ 474], 99.95th=[ 537], | 99.99th=[ 758] bw ( KiB/s): min=460774, max=555271, per=100.00%, avg=511688.95, stdev=16074.26, samples=60 iops : min= 7199, max= 8676, avg=7994.65, stdev=251.22, samples=60 write: IOPS=1987, BW=124MiB/s (130MB/s)(3727MiB/30001msec) clat (usec): min=211, max=2651, avg=243.62, stdev=28.56 lat (usec): min=220, max=2657, avg=249.46, stdev=28.63 clat percentiles (usec): | 1.00th=[ 225], 5.00th=[ 229], 10.00th=[ 231], 20.00th=[ 233], | 30.00th=[ 237], 40.00th=[ 239], 50.00th=[ 241], 60.00th=[ 243], | 70.00th=[ 247], 80.00th=[ 251], 90.00th=[ 258], 95.00th=[ 265], | 99.00th=[ 289], 99.50th=[ 306], 99.90th=[ 506], 99.95th=[ 701], | 99.99th=[ 1549] bw ( KiB/s): min=110064, max=135255, per=100.00%, avg=128233.15, stdev=3908.37, samples=60 iops : min= 1719, max= 2113, avg=2003.20, stdev=61.14, samples=60 lat (usec) : 50=26.55%, 100=50.94%, 250=16.99%, 500=5.43%, 750=0.06% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 50=0.01% cpu : usr=5.73%, sys=13.74%, ctx=297650, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=237994,59637,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=496MiB/s (520MB/s), 496MiB/s-496MiB/s (520MB/s-520MB/s), io=14.5GiB (15.6GB), run=30001-30001msec WRITE: bw=124MiB/s (130MB/s), 124MiB/s-124MiB/s (130MB/s-130MB/s), io=3727MiB (3908MB), run=30001-30001msec Disk stats (read/write): vdc: ios=237943/59630, merge=0/0, ticks=37/18, in_queue=24164, util=80.30% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920163: Fri Jan 7 13:54:55 2022 read: IOPS=12.0k, BW=810MiB/s (849MB/s)(23.8GiB/30034msec) slat (usec): min=3, max=488, avg= 9.37, stdev= 8.15 clat (nsec): min=638, max=39095k, avg=1341708.20, stdev=3382386.85 lat (usec): min=22, max=39153, avg=1351.21, stdev=3381.53 clat percentiles (usec): | 1.00th=[ 24], 5.00th=[ 34], 10.00th=[ 38], 20.00th=[ 43], | 30.00th=[ 49], 40.00th=[ 57], 50.00th=[ 65], 60.00th=[ 75], | 70.00th=[ 90], 80.00th=[ 124], 90.00th=[ 7177], 95.00th=[ 9765], | 99.00th=[13960], 99.50th=[15533], 99.90th=[20055], 99.95th=[22676], | 99.99th=[38536] bw ( KiB/s): min=182912, max=1648865, per=100.00%, avg=830174.60, stdev=500036.26, samples=60 iops : min= 2858, max=25763, avg=12971.47, stdev=7813.05, samples=60 write: IOPS=3252, BW=203MiB/s (213MB/s)(6106MiB/30034msec) slat (usec): min=6, max=766, avg=18.93, stdev=13.10 clat (usec): min=1223, max=50046, avg=9963.90, stdev=3430.14 lat (usec): min=1249, max=50090, avg=9982.99, stdev=3429.45 clat percentiles (usec): | 1.00th=[ 5211], 5.00th=[ 6259], 10.00th=[ 6652], 20.00th=[ 7111], | 30.00th=[ 7898], 40.00th=[ 8717], 50.00th=[ 9372], 60.00th=[10028], | 70.00th=[10683], 80.00th=[11731], 90.00th=[14353], 95.00th=[17171], | 99.00th=[21890], 99.50th=[23725], 99.90th=[27657], 99.95th=[30278], | 99.99th=[41681] bw ( KiB/s): min=46208, max=412729, per=100.00%, avg=208408.75, stdev=125238.84, samples=60 iops : min= 722, max= 6448, avg=3256.37, stdev=1956.83, samples=60 lat (nsec) : 750=0.02%, 1000=0.06% lat (usec) : 2=0.05%, 4=0.01%, 10=0.06%, 20=0.33%, 50=25.47% lat (usec) : 100=33.41%, 250=8.53%, 500=0.49%, 750=0.11%, 1000=0.05% lat (msec) : 2=0.06%, 4=0.45%, 10=19.31%, 20=11.15%, 50=0.47% lat (msec) : 100=0.01% cpu : usr=7.90%, sys=22.77%, ctx=217065, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=389142,97691,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=810MiB/s (849MB/s), 810MiB/s-810MiB/s (849MB/s-849MB/s), io=23.8GiB (25.5GB), run=30034-30034msec WRITE: bw=203MiB/s (213MB/s), 203MiB/s-203MiB/s (213MB/s-213MB/s), io=6106MiB (6402MB), run=30034-30034msec Disk stats (read/write): vdc: ios=388496/97536, merge=0/0, ticks=452757/916456, in_queue=1461757, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921318: Fri Jan 7 13:55:32 2022 read: IOPS=7815, BW=488MiB/s (512MB/s)(14.3GiB/30001msec) clat (usec): min=32, max=13971, avg=63.09, stdev=50.65 lat (usec): min=32, max=13971, avg=63.22, stdev=50.66 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 66], 90.00th=[ 73], 95.00th=[ 83], | 99.00th=[ 297], 99.50th=[ 326], 99.90th=[ 478], 99.95th=[ 553], | 99.99th=[ 742] bw ( KiB/s): min=438784, max=525312, per=99.93%, avg=499850.31, stdev=14392.35, samples=59 iops : min= 6856, max= 8208, avg=7810.14, stdev=224.87, samples=59 write: IOPS=1958, BW=122MiB/s (128MB/s)(3671MiB/30001msec) clat (usec): min=214, max=17136, avg=245.52, stdev=75.82 lat (usec): min=219, max=17144, avg=251.44, stdev=75.86 clat percentiles (usec): | 1.00th=[ 225], 5.00th=[ 229], 10.00th=[ 231], 20.00th=[ 235], | 30.00th=[ 237], 40.00th=[ 239], 50.00th=[ 243], 60.00th=[ 245], | 70.00th=[ 249], 80.00th=[ 253], 90.00th=[ 260], 95.00th=[ 269], | 99.00th=[ 297], 99.50th=[ 314], 99.90th=[ 570], 99.95th=[ 750], | 99.99th=[ 1516] bw ( KiB/s): min=115200, max=134912, per=99.99%, avg=125300.83, stdev=3894.39, samples=59 iops : min= 1800, max= 2108, avg=1957.80, stdev=60.86, samples=59 lat (usec) : 50=22.34%, 100=55.07%, 250=16.29%, 500=6.22%, 750=0.07% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01% cpu : usr=5.73%, sys=13.89%, ctx=293229, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=234477,58743,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=488MiB/s (512MB/s), 488MiB/s-488MiB/s (512MB/s-512MB/s), io=14.3GiB (15.4GB), run=30001-30001msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3671MiB (3850MB), run=30001-30001msec Disk stats (read/write): vdc: ios=232603/58270, merge=0/0, ticks=27/41, in_queue=24072, util=80.61% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922077: Fri Jan 7 13:56:09 2022 read: IOPS=14.1k, BW=882MiB/s (925MB/s)(25.9GiB/30040msec) slat (usec): min=2, max=551, avg= 8.06, stdev= 7.51 clat (nsec): min=617, max=101433k, avg=2086990.15, stdev=6622546.34 lat (usec): min=22, max=101623, avg=2095.17, stdev=6622.08 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 45], | 30.00th=[ 51], 40.00th=[ 58], 50.00th=[ 66], 60.00th=[ 76], | 70.00th=[ 88], 80.00th=[ 110], 90.00th=[ 6849], 95.00th=[ 19268], | 99.00th=[ 27657], 99.50th=[ 30802], 99.90th=[ 58459], 99.95th=[ 71828], | 99.99th=[100140] bw ( KiB/s): min=91136, max=1670016, per=100.00%, avg=904319.05, stdev=514706.87, samples=60 iops : min= 1424, max=26094, avg=14129.97, stdev=8042.28, samples=60 write: IOPS=3541, BW=221MiB/s (232MB/s)(6649MiB/30040msec) slat (usec): min=5, max=441, avg=17.55, stdev=10.73 clat (msec): min=3, max=100, avg=19.86, stdev= 8.44 lat (msec): min=3, max=100, avg=19.88, stdev= 8.44 clat percentiles (msec): | 1.00th=[ 6], 5.00th=[ 8], 10.00th=[ 10], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 17], 50.00th=[ 20], 60.00th=[ 22], | 70.00th=[ 24], 80.00th=[ 27], 90.00th=[ 33], 95.00th=[ 35], | 99.00th=[ 41], 99.50th=[ 46], 99.90th=[ 68], 99.95th=[ 72], | 99.99th=[ 101] bw ( KiB/s): min=24064, max=411776, per=100.00%, avg=226948.65, stdev=129515.91, samples=60 iops : min= 376, max= 6434, avg=3546.07, stdev=2023.68, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.02%, 4=0.01%, 10=0.03%, 20=0.16%, 50=22.96% lat (usec) : 100=38.15%, 250=9.46%, 500=0.32%, 750=0.10%, 1000=0.07% lat (msec) : 2=0.10%, 4=0.06%, 10=4.04%, 20=11.69%, 50=12.61% lat (msec) : 100=0.19%, 250=0.01% cpu : usr=8.53%, sys=22.70%, ctx=246561, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=423915,106391,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=882MiB/s (925MB/s), 882MiB/s-882MiB/s (925MB/s-925MB/s), io=25.9GiB (27.8GB), run=30040-30040msec WRITE: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=6649MiB (6972MB), run=30040-30040msec Disk stats (read/write): vdc: ios=423124/106182, merge=0/0, ticks=796947/2048219, in_queue=2940956, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923045: Fri Jan 7 13:56:46 2022 read: IOPS=8039, BW=502MiB/s (527MB/s)(14.7GiB/30001msec) clat (usec): min=31, max=2509, avg=60.41, stdev=40.92 lat (usec): min=31, max=2509, avg=60.53, stdev=40.92 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 70], 95.00th=[ 79], | 99.00th=[ 293], 99.50th=[ 322], 99.90th=[ 461], 99.95th=[ 529], | 99.99th=[ 693] bw ( KiB/s): min=484608, max=544256, per=100.00%, avg=514694.34, stdev=13304.10, samples=59 iops : min= 7572, max= 8504, avg=8042.08, stdev=207.88, samples=59 write: IOPS=2015, BW=126MiB/s (132MB/s)(3779MiB/30001msec) clat (usec): min=211, max=12669, avg=242.37, stdev=57.14 lat (usec): min=217, max=12679, avg=248.26, stdev=57.19 clat percentiles (usec): | 1.00th=[ 223], 5.00th=[ 227], 10.00th=[ 229], 20.00th=[ 233], | 30.00th=[ 235], 40.00th=[ 237], 50.00th=[ 239], 60.00th=[ 243], | 70.00th=[ 245], 80.00th=[ 249], 90.00th=[ 258], 95.00th=[ 265], | 99.00th=[ 285], 99.50th=[ 293], 99.90th=[ 400], 99.95th=[ 603], | 99.99th=[ 1926] bw ( KiB/s): min=120704, max=135936, per=100.00%, avg=128998.17, stdev=3062.38, samples=59 iops : min= 1886, max= 2124, avg=2015.59, stdev=47.84, samples=59 lat (usec) : 50=31.62%, 100=45.97%, 250=17.46%, 500=4.88%, 750=0.06% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01% cpu : usr=5.50%, sys=13.44%, ctx=301681, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=241202,60468,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=502MiB/s (527MB/s), 502MiB/s-502MiB/s (527MB/s-527MB/s), io=14.7GiB (15.8GB), run=30001-30001msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3779MiB (3963MB), run=30001-30001msec Disk stats (read/write): vdc: ios=239364/59999, merge=0/0, ticks=11/27, in_queue=24385, util=81.65% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923718: Fri Jan 7 13:57:23 2022 read: IOPS=14.9k, BW=933MiB/s (979MB/s)(27.4GiB/30034msec) slat (usec): min=2, max=60960, avg=50.59, stdev=647.72 clat (usec): min=63, max=102355, avg=6151.57, stdev=11472.45 lat (usec): min=976, max=129693, avg=6202.26, stdev=11543.54 clat percentiles (usec): | 1.00th=[ 1565], 5.00th=[ 1860], 10.00th=[ 2089], 20.00th=[ 2409], | 30.00th=[ 2704], 40.00th=[ 2966], 50.00th=[ 3261], 60.00th=[ 3589], | 70.00th=[ 4047], 80.00th=[ 4686], 90.00th=[ 6456], 95.00th=[ 24511], | 99.00th=[ 66323], 99.50th=[ 71828], 99.90th=[ 81265], 99.95th=[ 87557], | 99.99th=[101188] bw ( KiB/s): min=166528, max=1660416, per=100.00%, avg=956523.75, stdev=455689.47, samples=60 iops : min= 2602, max=25944, avg=14945.68, stdev=7120.15, samples=60 write: IOPS=3747, BW=234MiB/s (246MB/s)(7035MiB/30034msec) slat (usec): min=5, max=45888, avg=57.22, stdev=551.41 clat (msec): min=5, max=119, avg=28.58, stdev=10.50 lat (msec): min=6, max=119, avg=28.64, stdev=10.54 clat percentiles (msec): | 1.00th=[ 11], 5.00th=[ 14], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 23], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 30], | 70.00th=[ 32], 80.00th=[ 36], 90.00th=[ 42], 95.00th=[ 48], | 99.00th=[ 64], 99.50th=[ 70], 99.90th=[ 84], 99.95th=[ 88], | 99.99th=[ 102] bw ( KiB/s): min=46464, max=406528, per=100.00%, avg=240084.23, stdev=114001.09, samples=60 iops : min= 726, max= 6352, avg=3751.27, stdev=1781.26, samples=60 lat (usec) : 100=0.01%, 1000=0.01% lat (msec) : 2=6.21%, 4=49.10%, 10=18.77%, 20=4.64%, 50=18.08% lat (msec) : 100=3.18%, 250=0.02% cpu : usr=6.93%, sys=12.40%, ctx=58034, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=448534,112565,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=933MiB/s (979MB/s), 933MiB/s-933MiB/s (979MB/s-979MB/s), io=27.4GiB (29.4GB), run=30034-30034msec WRITE: bw=234MiB/s (246MB/s), 234MiB/s-234MiB/s (246MB/s-246MB/s), io=7035MiB (7377MB), run=30034-30034msec Disk stats (read/write): vdc: ios=447703/112369, merge=0/0, ticks=768890/2785492, in_queue=3684948, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924555: Fri Jan 7 13:58:00 2022 read: IOPS=6521, BW=510MiB/s (534MB/s)(14.9GiB/30001msec) clat (usec): min=39, max=21994, avg=76.12, stdev=82.21 lat (usec): min=40, max=21995, avg=76.24, stdev=82.21 clat percentiles (usec): | 1.00th=[ 51], 5.00th=[ 54], 10.00th=[ 56], 20.00th=[ 58], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 78], 90.00th=[ 88], 95.00th=[ 106], | 99.00th=[ 338], 99.50th=[ 383], 99.90th=[ 562], 99.95th=[ 635], | 99.99th=[ 1319] bw ( KiB/s): min=484960, max=559200, per=100.00%, avg=522443.78, stdev=15294.11, samples=59 iops : min= 6062, max= 6990, avg=6530.51, stdev=191.21, samples=59 write: IOPS=1633, BW=128MiB/s (134MB/s)(3829MiB/30001msec) clat (usec): min=254, max=2814, avg=293.13, stdev=39.38 lat (usec): min=263, max=2830, avg=300.69, stdev=39.48 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 273], 10.00th=[ 277], 20.00th=[ 281], | 30.00th=[ 285], 40.00th=[ 285], 50.00th=[ 289], 60.00th=[ 293], | 70.00th=[ 297], 80.00th=[ 302], 90.00th=[ 310], 95.00th=[ 322], | 99.00th=[ 359], 99.50th=[ 396], 99.90th=[ 750], 99.95th=[ 1090], | 99.99th=[ 1844] bw ( KiB/s): min=117120, max=138880, per=100.00%, avg=130749.75, stdev=4656.52, samples=59 iops : min= 1464, max= 1736, avg=1634.36, stdev=58.20, samples=59 lat (usec) : 50=0.59%, 100=74.81%, 250=2.83%, 500=21.58%, 750=0.14% lat (usec) : 1000=0.02% lat (msec) : 2=0.02%, 4=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.18%, sys=12.36%, ctx=244678, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=195665,49006,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=510MiB/s (534MB/s), 510MiB/s-510MiB/s (534MB/s-534MB/s), io=14.9GiB (16.0GB), run=30001-30001msec WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=3829MiB (4015MB), run=30001-30001msec Disk stats (read/write): vdc: ios=194847/48784, merge=0/0, ticks=69/24, in_queue=24611, util=82.17% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925584: Fri Jan 7 13:58:37 2022 read: IOPS=6530, BW=510MiB/s (535MB/s)(14.9GiB/30001msec) slat (usec): min=7, max=215, avg=13.37, stdev= 3.65 clat (nsec): min=879, max=2713.1k, avg=62948.14, stdev=47446.81 lat (usec): min=38, max=2728, avg=76.45, stdev=47.85 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 65], 90.00th=[ 73], 95.00th=[ 86], | 99.00th=[ 322], 99.50th=[ 355], 99.90th=[ 515], 99.95th=[ 586], | 99.99th=[ 816] bw ( KiB/s): min=492320, max=549280, per=100.00%, avg=522691.29, stdev=15635.05, samples=59 iops : min= 6154, max= 6866, avg=6533.63, stdev=195.45, samples=59 write: IOPS=1636, BW=128MiB/s (134MB/s)(3835MiB/30001msec) slat (usec): min=11, max=1262, avg=21.23, stdev= 7.46 clat (usec): min=189, max=5725, avg=277.99, stdev=41.90 lat (usec): min=266, max=5749, avg=299.37, stdev=42.88 clat percentiles (usec): | 1.00th=[ 258], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269], | 30.00th=[ 269], 40.00th=[ 273], 50.00th=[ 273], 60.00th=[ 277], | 70.00th=[ 281], 80.00th=[ 285], 90.00th=[ 293], 95.00th=[ 302], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 619], 99.95th=[ 955], | 99.99th=[ 1795] bw ( KiB/s): min=120480, max=138400, per=99.95%, avg=130819.46, stdev=3863.38, samples=59 iops : min= 1506, max= 1730, avg=1635.24, stdev=48.29, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=29.75%, 100=47.17% lat (usec) : 250=1.57%, 500=21.38%, 750=0.10%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=4.85%, sys=19.80%, ctx=245013, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=195927,49084,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=510MiB/s (535MB/s), 510MiB/s-510MiB/s (535MB/s-535MB/s), io=14.9GiB (16.1GB), run=30001-30001msec WRITE: bw=128MiB/s (134MB/s), 128MiB/s-128MiB/s (134MB/s-134MB/s), io=3835MiB (4021MB), run=30001-30001msec Disk stats (read/write): vdc: ios=194652/48729, merge=0/0, ticks=17/31, in_queue=24237, util=81.10% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926128: Fri Jan 7 13:59:14 2022 read: IOPS=7072, BW=553MiB/s (579MB/s)(16.2GiB/30001msec) clat (usec): min=33, max=7097, avg=66.92, stdev=51.43 lat (usec): min=33, max=7097, avg=67.04, stdev=51.44 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 68], 90.00th=[ 76], 95.00th=[ 89], | 99.00th=[ 326], 99.50th=[ 367], 99.90th=[ 545], 99.95th=[ 619], | 99.99th=[ 889] bw ( KiB/s): min=507040, max=596000, per=100.00%, avg=566264.41, stdev=16775.10, samples=59 iops : min= 6338, max= 7450, avg=7078.31, stdev=209.69, samples=59 write: IOPS=1773, BW=139MiB/s (145MB/s)(4157MiB/30001msec) clat (usec): min=250, max=12238, avg=282.28, stdev=64.21 lat (usec): min=257, max=12246, avg=289.73, stdev=64.26 clat percentiles (usec): | 1.00th=[ 262], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 273], | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 297], 95.00th=[ 306], | 99.00th=[ 334], 99.50th=[ 351], 99.90th=[ 652], 99.95th=[ 1029], | 99.99th=[ 2114] bw ( KiB/s): min=134560, max=150080, per=99.97%, avg=141835.93, stdev=3417.74, samples=59 iops : min= 1682, max= 1876, avg=1772.95, stdev=42.72, samples=59 lat (usec) : 50=13.63%, 100=63.14%, 250=1.65%, 500=21.43%, 750=0.12% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=5.06%, sys=13.32%, ctx=265403, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=212186,53204,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=553MiB/s (579MB/s), 553MiB/s-553MiB/s (579MB/s-579MB/s), io=16.2GiB (17.4GB), run=30001-30001msec WRITE: bw=139MiB/s (145MB/s), 139MiB/s-139MiB/s (145MB/s-145MB/s), io=4157MiB (4358MB), run=30001-30001msec Disk stats (read/write): vdc: ios=210725/52828, merge=0/0, ticks=27/40, in_queue=24280, util=81.25% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926936: Fri Jan 7 13:59:51 2022 read: IOPS=11.0k, BW=936MiB/s (981MB/s)(27.4GiB/30029msec) slat (usec): min=4, max=574, avg= 9.97, stdev= 7.17 clat (nsec): min=673, max=31126k, avg=397490.26, stdev=850998.80 lat (usec): min=31, max=31132, avg=407.58, stdev=850.78 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 53], 10.00th=[ 63], 20.00th=[ 80], | 30.00th=[ 95], 40.00th=[ 112], 50.00th=[ 130], 60.00th=[ 151], | 70.00th=[ 184], 80.00th=[ 245], 90.00th=[ 1237], 95.00th=[ 2040], | 99.00th=[ 4080], 99.50th=[ 4948], 99.90th=[ 7177], 99.95th=[ 9503], | 99.99th=[15270] bw ( KiB/s): min=222240, max=1659680, per=100.00%, avg=959033.73, stdev=671515.24, samples=60 iops : min= 2778, max=20748, avg=11987.92, stdev=8393.94, samples=60 write: IOPS=3005, BW=235MiB/s (246MB/s)(7050MiB/30029msec) slat (usec): min=7, max=1008, avg=23.19, stdev=15.47 clat (usec): min=270, max=37378, avg=1673.93, stdev=1455.92 lat (usec): min=285, max=37396, avg=1697.27, stdev=1454.65 clat percentiles (usec): | 1.00th=[ 523], 5.00th=[ 725], 10.00th=[ 857], 20.00th=[ 1020], | 30.00th=[ 1139], 40.00th=[ 1237], 50.00th=[ 1319], 60.00th=[ 1418], | 70.00th=[ 1500], 80.00th=[ 1647], 90.00th=[ 2900], 95.00th=[ 4752], | 99.00th=[ 7570], 99.50th=[ 8586], 99.90th=[12387], 99.95th=[16581], | 99.99th=[33817] bw ( KiB/s): min=57760, max=410400, per=100.00%, avg=240650.47, stdev=168598.77, samples=60 iops : min= 722, max= 5130, avg=3008.10, stdev=2107.48, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.05%, 50=2.92% lat (usec) : 100=23.49%, 250=37.96%, 500=4.61%, 750=1.60%, 1000=4.13% lat (msec) : 2=18.63%, 4=4.32%, 10=2.19%, 20=0.07%, 50=0.01% cpu : usr=8.03%, sys=19.92%, ctx=130015, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=359598,90239,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=936MiB/s (981MB/s), 936MiB/s-936MiB/s (981MB/s-981MB/s), io=27.4GiB (29.5GB), run=30029-30029msec WRITE: bw=235MiB/s (246MB/s), 235MiB/s-235MiB/s (246MB/s-246MB/s), io=7050MiB (7392MB), run=30029-30029msec Disk stats (read/write): vdc: ios=358985/90062, merge=0/0, ticks=71661/105993, in_queue=275127, util=99.57% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927535: Fri Jan 7 14:00:28 2022 read: IOPS=7132, BW=557MiB/s (584MB/s)(16.3GiB/30001msec) clat (usec): min=33, max=3064, avg=65.74, stdev=47.80 lat (usec): min=34, max=3064, avg=65.86, stdev=47.81 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 67], 90.00th=[ 75], 95.00th=[ 87], | 99.00th=[ 326], 99.50th=[ 359], 99.90th=[ 537], 99.95th=[ 603], | 99.99th=[ 799] bw ( KiB/s): min=504320, max=612480, per=99.99%, avg=570553.98, stdev=18595.34, samples=59 iops : min= 6304, max= 7656, avg=7131.85, stdev=232.44, samples=59 write: IOPS=1787, BW=140MiB/s (146MB/s)(4190MiB/30001msec) clat (usec): min=246, max=10025, avg=282.21, stdev=62.78 lat (usec): min=253, max=10034, avg=289.82, stdev=62.83 clat percentiles (usec): | 1.00th=[ 262], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 273], | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 297], 95.00th=[ 306], | 99.00th=[ 330], 99.50th=[ 351], 99.90th=[ 603], 99.95th=[ 955], | 99.99th=[ 2606] bw ( KiB/s): min=125280, max=150880, per=99.97%, avg=142971.68, stdev=4096.38, samples=59 iops : min= 1566, max= 1886, avg=1787.08, stdev=51.18, samples=59 lat (usec) : 50=18.35%, 100=58.53%, 250=1.55%, 500=21.45%, 750=0.10% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=5.17%, sys=12.99%, ctx=267629, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=213987,53630,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=557MiB/s (584MB/s), 557MiB/s-557MiB/s (584MB/s-584MB/s), io=16.3GiB (17.5GB), run=30001-30001msec WRITE: bw=140MiB/s (146MB/s), 140MiB/s-140MiB/s (146MB/s-146MB/s), io=4190MiB (4393MB), run=30001-30001msec Disk stats (read/write): vdc: ios=212576/53296, merge=0/0, ticks=18/44, in_queue=24581, util=82.35% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928861: Fri Jan 7 14:01:05 2022 read: IOPS=10.1k, BW=793MiB/s (832MB/s)(23.2GiB/30017msec) slat (usec): min=2, max=697, avg=10.11, stdev= 6.81 clat (nsec): min=668, max=33166k, avg=525526.66, stdev=2664821.85 lat (usec): min=25, max=33173, avg=535.78, stdev=2664.84 clat percentiles (usec): | 1.00th=[ 25], 5.00th=[ 34], 10.00th=[ 37], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 46], 50.00th=[ 49], 60.00th=[ 55], | 70.00th=[ 63], 80.00th=[ 74], 90.00th=[ 96], 95.00th=[ 135], | 99.00th=[16909], 99.50th=[18482], 99.90th=[20317], 99.95th=[21103], | 99.99th=[24511] bw ( KiB/s): min=685920, max=1630400, per=100.00%, avg=812675.73, stdev=205000.53, samples=60 iops : min= 8574, max=20380, avg=10158.40, stdev=2562.51, samples=60 write: IOPS=2546, BW=199MiB/s (209MB/s)(5972MiB/30017msec) slat (usec): min=6, max=334, avg=20.56, stdev=10.81 clat (usec): min=2961, max=64261, avg=17470.81, stdev=5551.87 lat (usec): min=2997, max=64274, avg=17491.53, stdev=5552.71 clat percentiles (usec): | 1.00th=[ 8094], 5.00th=[ 9765], 10.00th=[10421], 20.00th=[11731], | 30.00th=[14746], 40.00th=[16319], 50.00th=[17695], 60.00th=[18482], | 70.00th=[19530], 80.00th=[20841], 90.00th=[23987], 95.00th=[26870], | 99.00th=[34866], 99.50th=[38536], 99.90th=[45876], 99.95th=[48497], | 99.99th=[56886] bw ( KiB/s): min=182880, max=407040, per=100.00%, avg=203773.92, stdev=50645.92, samples=60 iops : min= 2286, max= 5088, avg=2547.10, stdev=633.10, samples=60 lat (nsec) : 750=0.01%, 1000=0.09% lat (usec) : 2=0.11%, 4=0.01%, 10=0.05%, 20=0.21%, 50=41.10% lat (usec) : 100=31.29%, 250=4.22%, 500=0.18%, 750=0.06%, 1000=0.04% lat (msec) : 2=0.03%, 4=0.01%, 10=1.57%, 20=15.66%, 50=5.34% lat (msec) : 100=0.01% cpu : usr=7.36%, sys=22.34%, ctx=289335, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=304680,76440,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=793MiB/s (832MB/s), 793MiB/s-793MiB/s (832MB/s-832MB/s), io=23.2GiB (24.0GB), run=30017-30017msec WRITE: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=5972MiB (6262MB), run=30017-30017msec Disk stats (read/write): vdc: ios=302599/75875, merge=0/0, ticks=137067/1286340, in_queue=1480080, util=99.99% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929910: Fri Jan 7 14:01:42 2022 read: IOPS=7035, BW=550MiB/s (576MB/s)(16.1GiB/30001msec) clat (usec): min=33, max=2529, avg=67.32, stdev=47.17 lat (usec): min=33, max=2529, avg=67.45, stdev=47.18 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 77], 95.00th=[ 90], | 99.00th=[ 330], 99.50th=[ 363], 99.90th=[ 523], 99.95th=[ 594], | 99.99th=[ 758] bw ( KiB/s): min=505280, max=605280, per=100.00%, avg=564768.49, stdev=18150.57, samples=59 iops : min= 6316, max= 7566, avg=7059.54, stdev=226.88, samples=59 write: IOPS=1764, BW=138MiB/s (145MB/s)(4135MiB/30001msec) clat (usec): min=248, max=10467, avg=282.92, stdev=61.33 lat (usec): min=255, max=10474, avg=290.82, stdev=61.39 clat percentiles (usec): | 1.00th=[ 262], 5.00th=[ 265], 10.00th=[ 269], 20.00th=[ 273], | 30.00th=[ 273], 40.00th=[ 277], 50.00th=[ 281], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 297], 95.00th=[ 306], | 99.00th=[ 334], 99.50th=[ 355], 99.90th=[ 652], 99.95th=[ 971], | 99.99th=[ 2073] bw ( KiB/s): min=131840, max=149572, per=100.00%, avg=141494.46, stdev=3755.32, samples=59 iops : min= 1648, max= 1869, avg=1768.61, stdev=46.90, samples=59 lat (usec) : 50=11.89%, 100=64.81%, 250=1.69%, 500=21.48%, 750=0.10% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=5.57%, sys=13.18%, ctx=264024, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=211085,52930,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=550MiB/s (576MB/s), 550MiB/s-550MiB/s (576MB/s-576MB/s), io=16.1GiB (17.3GB), run=30001-30001msec WRITE: bw=138MiB/s (145MB/s), 138MiB/s-138MiB/s (145MB/s-145MB/s), io=4135MiB (4336MB), run=30001-30001msec Disk stats (read/write): vdc: ios=210992/52907, merge=0/0, ticks=11/41, in_queue=24350, util=80.96% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930583: Fri Jan 7 14:02:19 2022 read: IOPS=12.6k, BW=981MiB/s (1028MB/s)(28.8GiB/30034msec) slat (usec): min=2, max=317, avg= 9.10, stdev= 7.42 clat (nsec): min=691, max=58819k, avg=1522774.09, stdev=5608304.10 lat (usec): min=24, max=58827, avg=1532.00, stdev=5607.76 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 36], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 63], 60.00th=[ 72], | 70.00th=[ 83], 80.00th=[ 101], 90.00th=[ 161], 95.00th=[12780], | 99.00th=[28705], 99.50th=[31851], 99.90th=[44827], 99.95th=[51643], | 99.99th=[57934] bw ( KiB/s): min=228960, max=1683840, per=100.00%, avg=1005185.08, stdev=378429.79, samples=60 iops : min= 2862, max=21048, avg=12564.77, stdev=4730.36, samples=60 write: IOPS=3149, BW=246MiB/s (258MB/s)(7390MiB/30034msec) slat (usec): min=6, max=479, avg=22.59, stdev=14.08 clat (usec): min=1365, max=94382, avg=25617.22, stdev=9791.33 lat (usec): min=1385, max=94435, avg=25639.95, stdev=9792.19 clat percentiles (usec): | 1.00th=[ 8225], 5.00th=[10683], 10.00th=[13304], 20.00th=[18220], | 30.00th=[21365], 40.00th=[23725], 50.00th=[24773], 60.00th=[25822], | 70.00th=[27919], 80.00th=[31851], 90.00th=[38011], 95.00th=[43779], | 99.00th=[56886], 99.50th=[62129], 99.90th=[72877], 99.95th=[74974], | 99.99th=[84411] bw ( KiB/s): min=59520, max=413120, per=100.00%, avg=252210.90, stdev=93692.73, samples=60 iops : min= 744, max= 5164, avg=3152.58, stdev=1171.16, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.05%, 4=0.01%, 10=0.04%, 20=0.16%, 50=24.40% lat (usec) : 100=39.01%, 250=9.49%, 500=0.32%, 750=0.11%, 1000=0.07% lat (msec) : 2=0.11%, 4=0.06%, 10=1.69%, 20=6.63%, 50=17.27% lat (msec) : 100=0.53% cpu : usr=9.74%, sys=23.56%, ctx=272160, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=376958,94587,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=981MiB/s (1028MB/s), 981MiB/s-981MiB/s (1028MB/s-1028MB/s), io=28.8GiB (30.9GB), run=30034-30034msec WRITE: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=7390MiB (7749MB), run=30034-30034msec Disk stats (read/write): vdc: ios=376265/94380, merge=0/0, ticks=504875/2365399, in_queue=2951659, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931411: Fri Jan 7 14:02:56 2022 read: IOPS=7228, BW=565MiB/s (592MB/s)(16.5GiB/30001msec) clat (usec): min=33, max=2368, avg=64.42, stdev=46.75 lat (usec): min=33, max=2368, avg=64.55, stdev=46.75 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 48], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 56], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 74], 95.00th=[ 86], | 99.00th=[ 326], 99.50th=[ 359], 99.90th=[ 519], 99.95th=[ 578], | 99.99th=[ 734] bw ( KiB/s): min=547520, max=616000, per=99.96%, avg=578015.41, stdev=14532.14, samples=59 iops : min= 6844, max= 7700, avg=7225.15, stdev=181.66, samples=59 write: IOPS=1809, BW=141MiB/s (148MB/s)(4241MiB/30001msec) clat (usec): min=250, max=6270, avg=280.19, stdev=37.36 lat (usec): min=258, max=6274, avg=287.66, stdev=37.42 clat percentiles (usec): | 1.00th=[ 260], 5.00th=[ 265], 10.00th=[ 265], 20.00th=[ 269], | 30.00th=[ 273], 40.00th=[ 273], 50.00th=[ 277], 60.00th=[ 281], | 70.00th=[ 285], 80.00th=[ 289], 90.00th=[ 297], 95.00th=[ 302], | 99.00th=[ 330], 99.50th=[ 347], 99.90th=[ 578], 99.95th=[ 676], | 99.99th=[ 1434] bw ( KiB/s): min=136160, max=151840, per=100.00%, avg=144892.20, stdev=3550.68, samples=59 iops : min= 1702, max= 1898, avg=1811.10, stdev=44.39, samples=59 lat (usec) : 50=24.06%, 100=52.87%, 250=1.53%, 500=21.41%, 750=0.11% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=5.43%, sys=12.46%, ctx=271155, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=216855,54291,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=565MiB/s (592MB/s), 565MiB/s-565MiB/s (592MB/s-592MB/s), io=16.5GiB (17.8GB), run=30001-30001msec WRITE: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=4241MiB (4448MB), run=30001-30001msec Disk stats (read/write): vdc: ios=215127/53875, merge=0/0, ticks=9/18, in_queue=24575, util=82.28% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932352: Fri Jan 7 14:03:33 2022 read: IOPS=12.6k, BW=986MiB/s (1034MB/s)(28.9GiB/30024msec) slat (usec): min=2, max=47521, avg=58.66, stdev=317.52 clat (usec): min=107, max=88006, avg=5516.37, stdev=5027.89 lat (usec): min=1135, max=117404, avg=5575.14, stdev=5057.77 clat percentiles (usec): | 1.00th=[ 1958], 5.00th=[ 2474], 10.00th=[ 2802], 20.00th=[ 3261], | 30.00th=[ 3654], 40.00th=[ 4015], 50.00th=[ 4424], 60.00th=[ 4948], | 70.00th=[ 5669], 80.00th=[ 6718], 90.00th=[ 8455], 95.00th=[10683], | 99.00th=[23725], 99.50th=[32113], 99.90th=[74974], 99.95th=[82314], | 99.99th=[87557] bw ( KiB/s): min=241440, max=1497760, per=100.00%, avg=1009840.83, stdev=297600.80, samples=60 iops : min= 3018, max=18722, avg=12622.95, stdev=3720.04, samples=60 write: IOPS=3167, BW=247MiB/s (259MB/s)(7430MiB/30024msec) slat (usec): min=6, max=48609, avg=73.35, stdev=354.49 clat (msec): min=5, max=145, avg=40.85, stdev=17.23 lat (msec): min=6, max=145, avg=40.92, stdev=17.24 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 19], 10.00th=[ 25], 20.00th=[ 30], | 30.00th=[ 31], 40.00th=[ 33], 50.00th=[ 36], 60.00th=[ 39], | 70.00th=[ 45], 80.00th=[ 55], 90.00th=[ 67], 95.00th=[ 75], | 99.00th=[ 94], 99.50th=[ 101], 99.90th=[ 113], 99.95th=[ 118], | 99.99th=[ 131] bw ( KiB/s): min=65920, max=366400, per=100.00%, avg=253538.83, stdev=72656.47, samples=60 iops : min= 824, max= 4580, avg=3169.22, stdev=908.21, samples=60 lat (usec) : 250=0.01% lat (msec) : 2=0.94%, 4=30.58%, 10=43.72%, 20=4.74%, 50=14.84% lat (msec) : 100=5.07%, 250=0.10% cpu : usr=7.48%, sys=11.01%, ctx=51873, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=378839,95098,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=986MiB/s (1034MB/s), 986MiB/s-986MiB/s (1034MB/s-1034MB/s), io=28.9GiB (31.0GB), run=30024-30024msec WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=7430MiB (7790MB), run=30024-30024msec Disk stats (read/write): vdc: ios=378040/94872, merge=0/0, ticks=178849/3375422, in_queue=3673257, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932853: Fri Jan 7 14:04:10 2022 read: IOPS=1691, BW=211MiB/s (222MB/s)(6343MiB/30002msec) clat (usec): min=51, max=6370, avg=92.22, stdev=69.24 lat (usec): min=51, max=6370, avg=92.36, stdev=69.25 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 95], 90.00th=[ 110], 95.00th=[ 145], | 99.00th=[ 416], 99.50th=[ 465], 99.90th=[ 611], 99.95th=[ 758], | 99.99th=[ 1532] bw ( KiB/s): min=77824, max=641024, per=100.00%, avg=216510.90, stdev=185480.61, samples=60 iops : min= 608, max= 5008, avg=1691.52, stdev=1449.14, samples=60 write: IOPS=425, BW=53.2MiB/s (55.7MB/s)(1595MiB/30002msec) clat (usec): min=376, max=17730, avg=1963.01, stdev=2033.00 lat (usec): min=385, max=17743, avg=1976.31, stdev=2033.99 clat percentiles (usec): | 1.00th=[ 383], 5.00th=[ 392], 10.00th=[ 396], 20.00th=[ 400], | 30.00th=[ 404], 40.00th=[ 412], 50.00th=[ 429], 60.00th=[ 2073], | 70.00th=[ 2540], 80.00th=[ 3949], 90.00th=[ 5211], 95.00th=[ 5932], | 99.00th=[ 7635], 99.50th=[ 8291], 99.90th=[ 9110], 99.95th=[ 9896], | 99.99th=[15664] bw ( KiB/s): min=18688, max=161536, per=100.00%, avg=54430.62, stdev=46924.59, samples=60 iops : min= 146, max= 1262, avg=425.23, stdev=366.60, samples=60 lat (usec) : 100=67.67%, 250=9.48%, 500=13.37%, 750=0.22%, 1000=0.02% lat (msec) : 2=0.75%, 4=4.56%, 10=3.91%, 20=0.01% cpu : usr=1.68%, sys=3.26%, ctx=63506, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=50745,12758,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=211MiB/s (222MB/s), 211MiB/s-211MiB/s (222MB/s-222MB/s), io=6343MiB (6651MB), run=30002-30002msec WRITE: bw=53.2MiB/s (55.7MB/s), 53.2MiB/s-53.2MiB/s (55.7MB/s-55.7MB/s), io=1595MiB (1672MB), run=30002-30002msec Disk stats (read/write): vdc: ios=50469/12681, merge=0/0, ticks=22/19222, in_queue=28315, util=94.69% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933836: Fri Jan 7 14:04:47 2022 read: IOPS=2917, BW=365MiB/s (382MB/s)(10.7GiB/30001msec) slat (usec): min=8, max=185, avg=13.88, stdev= 4.42 clat (nsec): min=771, max=3493.8k, avg=66056.43, stdev=62668.82 lat (usec): min=41, max=3517, avg=80.10, stdev=62.99 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 43], 10.00th=[ 45], 20.00th=[ 46], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 74], 95.00th=[ 96], | 99.00th=[ 404], 99.50th=[ 445], 99.90th=[ 635], 99.95th=[ 750], | 99.99th=[ 979] bw ( KiB/s): min=189696, max=741888, per=100.00%, avg=377814.08, stdev=220258.82, samples=59 iops : min= 1482, max= 5796, avg=2951.66, stdev=1720.76, samples=59 write: IOPS=729, BW=91.2MiB/s (95.7MB/s)(2737MiB/30001msec) slat (usec): min=15, max=213, avg=26.55, stdev= 7.02 clat (usec): min=315, max=20366, avg=1016.00, stdev=913.67 lat (usec): min=370, max=20408, avg=1042.73, stdev=915.45 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 367], 10.00th=[ 367], 20.00th=[ 371], | 30.00th=[ 375], 40.00th=[ 379], 50.00th=[ 388], 60.00th=[ 412], | 70.00th=[ 1762], 80.00th=[ 1860], 90.00th=[ 2008], 95.00th=[ 2245], | 99.00th=[ 3228], 99.50th=[ 4686], 99.90th=[ 7242], 99.95th=[ 8455], | 99.99th=[13698] bw ( KiB/s): min=46592, max=182016, per=100.00%, avg=94561.07, stdev=55495.20, samples=59 iops : min= 364, max= 1422, avg=738.75, stdev=433.54, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.02%, 10=0.01%, 20=0.02%, 50=32.41%, 100=43.67% lat (usec) : 250=1.56%, 500=14.32%, 750=0.18%, 1000=0.04% lat (msec) : 2=5.72%, 4=1.93%, 10=0.11%, 20=0.01%, 50=0.01% cpu : usr=2.74%, sys=8.36%, ctx=109396, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=87519,21898,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=365MiB/s (382MB/s), 365MiB/s-365MiB/s (382MB/s-382MB/s), io=10.7GiB (11.5GB), run=30001-30001msec WRITE: bw=91.2MiB/s (95.7MB/s), 91.2MiB/s-91.2MiB/s (95.7MB/s-95.7MB/s), io=2737MiB (2870MB), run=30001-30001msec Disk stats (read/write): vdc: ios=87200/21832, merge=0/0, ticks=12/11223, in_queue=27335, util=91.48% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934420: Fri Jan 7 14:05:24 2022 read: IOPS=3834, BW=479MiB/s (503MB/s)(14.0GiB/30014msec) clat (usec): min=40, max=5161, avg=76.35, stdev=66.87 lat (usec): min=40, max=5161, avg=76.48, stdev=66.88 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 85], 95.00th=[ 113], | 99.00th=[ 420], 99.50th=[ 474], 99.90th=[ 693], 99.95th=[ 791], | 99.99th=[ 1270] bw ( KiB/s): min=82432, max=754688, per=100.00%, avg=491137.80, stdev=233097.55, samples=60 iops : min= 644, max= 5896, avg=3836.98, stdev=1821.08, samples=60 write: IOPS=955, BW=119MiB/s (125MB/s)(3585MiB/30014msec) clat (usec): min=359, max=16237, avg=719.86, stdev=692.75 lat (usec): min=371, max=16244, avg=732.51, stdev=692.63 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 379], 20.00th=[ 383], | 30.00th=[ 383], 40.00th=[ 388], 50.00th=[ 392], 60.00th=[ 400], | 70.00th=[ 408], 80.00th=[ 1565], 90.00th=[ 1713], 95.00th=[ 1893], | 99.00th=[ 2769], 99.50th=[ 3458], 99.90th=[ 6325], 99.95th=[ 6849], | 99.99th=[10814] bw ( KiB/s): min=26624, max=189440, per=100.00%, avg=122385.15, stdev=58785.64, samples=60 iops : min= 208, max= 1480, avg=956.07, stdev=459.26, samples=60 lat (usec) : 50=1.64%, 100=73.80%, 250=2.07%, 500=17.70%, 750=0.31% lat (usec) : 1000=0.05% lat (msec) : 2=3.64%, 4=0.72%, 10=0.07%, 20=0.01% cpu : usr=3.49%, sys=7.12%, ctx=143780, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=115090,28680,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=479MiB/s (503MB/s), 479MiB/s-479MiB/s (503MB/s-503MB/s), io=14.0GiB (15.1GB), run=30014-30014msec WRITE: bw=119MiB/s (125MB/s), 119MiB/s-119MiB/s (125MB/s-125MB/s), io=3585MiB (3759MB), run=30014-30014msec Disk stats (read/write): vdc: ios=114643/28580, merge=0/0, ticks=19/7808, in_queue=26569, util=88.93% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935722: Fri Jan 7 14:06:01 2022 read: IOPS=5731, BW=716MiB/s (751MB/s)(21.0GiB/30019msec) slat (usec): min=4, max=375, avg=11.57, stdev= 8.04 clat (nsec): min=754, max=31549k, avg=868961.45, stdev=1548834.47 lat (usec): min=29, max=31561, avg=880.69, stdev=1547.79 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 41], 10.00th=[ 45], 20.00th=[ 52], | 30.00th=[ 61], 40.00th=[ 74], 50.00th=[ 92], 60.00th=[ 128], | 70.00th=[ 314], 80.00th=[ 1762], 90.00th=[ 2900], 95.00th=[ 3916], | 99.00th=[ 6194], 99.50th=[ 7177], 99.90th=[13566], 99.95th=[17171], | 99.99th=[24249] bw ( KiB/s): min=89856, max=1699840, per=100.00%, avg=734122.95, stdev=598469.92, samples=60 iops : min= 702, max=13280, avg=5735.37, stdev=4675.67, samples=60 write: IOPS=1432, BW=179MiB/s (188MB/s)(5377MiB/30019msec) slat (usec): min=9, max=419, avg=28.13, stdev=14.37 clat (usec): min=492, max=46714, avg=3420.89, stdev=2372.85 lat (usec): min=526, max=46766, avg=3449.24, stdev=2372.29 clat percentiles (usec): | 1.00th=[ 1418], 5.00th=[ 1778], 10.00th=[ 1958], 20.00th=[ 2147], | 30.00th=[ 2278], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2671], | 70.00th=[ 2900], 80.00th=[ 4621], 90.00th=[ 6456], 95.00th=[ 7767], | 99.00th=[10945], 99.50th=[13566], 99.90th=[26870], 99.95th=[30540], | 99.99th=[44303] bw ( KiB/s): min=25088, max=418816, per=100.00%, avg=183548.75, stdev=148473.73, samples=60 iops : min= 196, max= 3272, avg=1433.95, stdev=1159.95, samples=60 lat (nsec) : 1000=0.03% lat (usec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.13%, 50=14.06% lat (usec) : 100=28.42%, 250=11.72%, 500=2.60%, 750=0.07%, 1000=0.01% lat (msec) : 2=9.98%, 4=24.49%, 10=7.96%, 20=0.38%, 50=0.07% cpu : usr=4.52%, sys=11.67%, ctx=98203, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=172048,43017,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=716MiB/s (751MB/s), 716MiB/s-716MiB/s (751MB/s-751MB/s), io=21.0GiB (22.6GB), run=30019-30019msec WRITE: bw=179MiB/s (188MB/s), 179MiB/s-179MiB/s (188MB/s-188MB/s), io=5377MiB (5638MB), run=30019-30019msec Disk stats (read/write): vdc: ios=171509/42880, merge=0/0, ticks=113010/124325, in_queue=287514, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936696: Fri Jan 7 14:06:38 2022 read: IOPS=5517, BW=690MiB/s (723MB/s)(20.2GiB/30001msec) clat (usec): min=39, max=2515, avg=78.07, stdev=65.39 lat (usec): min=39, max=2515, avg=78.19, stdev=65.40 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 57], | 30.00th=[ 59], 40.00th=[ 62], 50.00th=[ 65], 60.00th=[ 68], | 70.00th=[ 72], 80.00th=[ 76], 90.00th=[ 86], 95.00th=[ 109], | 99.00th=[ 424], 99.50th=[ 482], 99.90th=[ 709], 99.95th=[ 816], | 99.99th=[ 1254] bw ( KiB/s): min=642304, max=770048, per=100.00%, avg=706551.97, stdev=23957.58, samples=59 iops : min= 5018, max= 6016, avg=5519.97, stdev=187.17, samples=59 write: IOPS=1380, BW=173MiB/s (181MB/s)(5177MiB/30001msec) clat (usec): min=357, max=4960, avg=392.38, stdev=36.04 lat (usec): min=368, max=4974, avg=404.89, stdev=36.39 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 379], 20.00th=[ 383], | 30.00th=[ 383], 40.00th=[ 388], 50.00th=[ 392], 60.00th=[ 392], | 70.00th=[ 396], 80.00th=[ 400], 90.00th=[ 408], 95.00th=[ 416], | 99.00th=[ 445], 99.50th=[ 461], 99.90th=[ 685], 99.95th=[ 914], | 99.99th=[ 1713] bw ( KiB/s): min=165376, max=194048, per=99.99%, avg=176682.36, stdev=4909.50, samples=59 iops : min= 1292, max= 1516, avg=1380.31, stdev=38.35, samples=59 lat (usec) : 50=0.93%, 100=74.58%, 250=1.88%, 500=22.22%, 750=0.31% lat (usec) : 1000=0.06% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01% cpu : usr=5.12%, sys=10.65%, ctx=206951, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=165517,41416,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=690MiB/s (723MB/s), 690MiB/s-690MiB/s (723MB/s-723MB/s), io=20.2GiB (21.7GB), run=30001-30001msec WRITE: bw=173MiB/s (181MB/s), 173MiB/s-173MiB/s (181MB/s-181MB/s), io=5177MiB (5428MB), run=30001-30001msec Disk stats (read/write): vdc: ios=164430/41134, merge=0/0, ticks=23/20, in_queue=25029, util=83.73% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937336: Fri Jan 7 14:07:15 2022 read: IOPS=6929, BW=866MiB/s (908MB/s)(25.4GiB/30026msec) slat (usec): min=3, max=171, avg=10.18, stdev= 6.37 clat (nsec): min=801, max=40587k, avg=2620911.80, stdev=5316223.76 lat (usec): min=30, max=40675, avg=2631.20, stdev=5315.57 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 41], 10.00th=[ 44], 20.00th=[ 48], | 30.00th=[ 53], 40.00th=[ 60], 50.00th=[ 70], 60.00th=[ 82], | 70.00th=[ 104], 80.00th=[ 5800], 90.00th=[12387], 95.00th=[14877], | 99.00th=[19006], 99.50th=[20841], 99.90th=[26084], 99.95th=[32900], | 99.99th=[39584] bw ( KiB/s): min=278016, max=1362688, per=100.00%, avg=887596.10, stdev=457120.61, samples=60 iops : min= 2172, max=10646, avg=6934.33, stdev=3571.26, samples=60 write: IOPS=1736, BW=217MiB/s (228MB/s)(6518MiB/30026msec) slat (usec): min=8, max=239, avg=27.66, stdev=10.25 clat (usec): min=1538, max=58005, avg=18260.40, stdev=4675.37 lat (usec): min=1556, max=58059, avg=18288.20, stdev=4677.73 clat percentiles (usec): | 1.00th=[11207], 5.00th=[13042], 10.00th=[14222], 20.00th=[15401], | 30.00th=[16057], 40.00th=[16581], 50.00th=[17171], 60.00th=[17695], | 70.00th=[18482], 80.00th=[20055], 90.00th=[24773], 95.00th=[28181], | 99.00th=[34866], 99.50th=[38011], 99.90th=[48497], 99.95th=[51119], | 99.99th=[57410] bw ( KiB/s): min=74752, max=326656, per=100.00%, avg=222413.87, stdev=112975.20, samples=60 iops : min= 584, max= 2552, avg=1737.60, stdev=882.63, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=19.44% lat (usec) : 100=35.47%, 250=7.95%, 500=0.22%, 750=0.08%, 1000=0.04% lat (msec) : 2=0.06%, 4=0.06%, 10=5.13%, 20=26.99%, 50=4.51% lat (msec) : 100=0.01% cpu : usr=6.35%, sys=13.89%, ctx=159570, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=208053,52140,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=866MiB/s (908MB/s), 866MiB/s-866MiB/s (908MB/s-908MB/s), io=25.4GiB (27.3GB), run=30026-30026msec WRITE: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=6518MiB (6834MB), run=30026-30026msec Disk stats (read/write): vdc: ios=208018/52125, merge=0/0, ticks=500381/920877, in_queue=1479466, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938229: Fri Jan 7 14:07:52 2022 read: IOPS=5622, BW=703MiB/s (737MB/s)(20.6GiB/30001msec) clat (usec): min=39, max=2597, avg=75.48, stdev=64.61 lat (usec): min=40, max=2597, avg=75.60, stdev=64.62 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 83], 95.00th=[ 102], | 99.00th=[ 420], 99.50th=[ 474], 99.90th=[ 693], 99.95th=[ 807], | 99.99th=[ 1004] bw ( KiB/s): min=675840, max=780288, per=99.94%, avg=719186.44, stdev=23013.52, samples=59 iops : min= 5280, max= 6096, avg=5618.64, stdev=179.79, samples=59 write: IOPS=1405, BW=176MiB/s (184MB/s)(5270MiB/30001msec) clat (usec): min=356, max=2297, avg=390.31, stdev=27.28 lat (usec): min=370, max=2308, avg=402.44, stdev=27.44 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 379], | 30.00th=[ 383], 40.00th=[ 383], 50.00th=[ 388], 60.00th=[ 392], | 70.00th=[ 396], 80.00th=[ 400], 90.00th=[ 404], 95.00th=[ 412], | 99.00th=[ 437], 99.50th=[ 449], 99.90th=[ 627], 99.95th=[ 766], | 99.99th=[ 1532] bw ( KiB/s): min=170240, max=187136, per=100.00%, avg=179976.68, stdev=4015.76, samples=59 iops : min= 1330, max= 1462, avg=1406.07, stdev=31.37, samples=59 lat (usec) : 50=1.33%, 100=74.55%, 250=1.59%, 500=22.18%, 750=0.29% lat (usec) : 1000=0.05% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=4.81%, sys=10.21%, ctx=210847, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=168673,42162,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=20.6GiB (22.1GB), run=30001-30001msec WRITE: bw=176MiB/s (184MB/s), 176MiB/s-176MiB/s (184MB/s-184MB/s), io=5270MiB (5526MB), run=30001-30001msec Disk stats (read/write): vdc: ios=167663/41924, merge=0/0, ticks=17/12, in_queue=25371, util=84.79% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938828: Fri Jan 7 14:08:30 2022 read: IOPS=7600, BW=950MiB/s (996MB/s)(27.9GiB/30047msec) slat (usec): min=3, max=370, avg=10.30, stdev= 7.87 clat (nsec): min=728, max=61618k, avg=3226628.64, stdev=8643470.67 lat (usec): min=31, max=61630, avg=3237.07, stdev=8642.47 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 40], 10.00th=[ 43], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 62], 50.00th=[ 72], 60.00th=[ 82], | 70.00th=[ 97], 80.00th=[ 130], 90.00th=[17171], 95.00th=[27395], | 99.00th=[34866], 99.50th=[37487], 99.90th=[42206], 99.95th=[49021], | 99.99th=[59507] bw ( KiB/s): min=302848, max=1458432, per=100.00%, avg=974002.18, stdev=372610.06, samples=60 iops : min= 2366, max=11394, avg=7609.37, stdev=2911.00, samples=60 write: IOPS=1905, BW=238MiB/s (250MB/s)(7157MiB/30047msec) slat (usec): min=8, max=533, avg=35.04, stdev=20.18 clat (msec): min=4, max=117, avg=39.52, stdev=11.73 lat (msec): min=4, max=117, avg=39.56, stdev=11.73 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 18], 10.00th=[ 24], 20.00th=[ 31], | 30.00th=[ 35], 40.00th=[ 37], 50.00th=[ 40], 60.00th=[ 44], | 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 53], 95.00th=[ 57], | 99.00th=[ 66], 99.50th=[ 73], 99.90th=[ 91], 99.95th=[ 100], | 99.99th=[ 109] bw ( KiB/s): min=81664, max=362240, per=100.00%, avg=244130.60, stdev=92452.33, samples=60 iops : min= 638, max= 2830, avg=1907.23, stdev=722.26, samples=60 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.05%, 4=0.01%, 10=0.02%, 20=0.09%, 50=20.45% lat (usec) : 100=36.57%, 250=11.50%, 500=0.46%, 750=0.15%, 1000=0.09% lat (msec) : 2=0.17%, 4=0.10%, 10=0.12%, 20=4.46%, 50=22.06% lat (msec) : 100=3.68%, 250=0.01% cpu : usr=8.40%, sys=16.01%, ctx=189223, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=228381,57254,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=950MiB/s (996MB/s), 950MiB/s-950MiB/s (996MB/s-996MB/s), io=27.9GiB (29.9GB), run=30047-30047msec WRITE: bw=238MiB/s (250MB/s), 238MiB/s-238MiB/s (250MB/s-250MB/s), io=7157MiB (7504MB), run=30047-30047msec Disk stats (read/write): vdc: ios=228310/57225, merge=0/0, ticks=684456/2224276, in_queue=2965330, util=99.60% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939648: Fri Jan 7 14:09:06 2022 read: IOPS=5633, BW=704MiB/s (738MB/s)(20.6GiB/30001msec) clat (usec): min=39, max=11916, avg=75.00, stdev=71.25 lat (usec): min=39, max=11916, avg=75.12, stdev=71.26 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 65], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 83], 95.00th=[ 108], | 99.00th=[ 420], 99.50th=[ 482], 99.90th=[ 693], 99.95th=[ 816], | 99.99th=[ 1057] bw ( KiB/s): min=621568, max=770821, per=99.95%, avg=720758.41, stdev=23712.23, samples=59 iops : min= 4856, max= 6022, avg=5630.92, stdev=185.25, samples=59 write: IOPS=1407, BW=176MiB/s (185MB/s)(5280MiB/30001msec) clat (usec): min=354, max=4774, avg=390.65, stdev=44.27 lat (usec): min=367, max=4785, avg=402.82, stdev=44.44 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 375], 20.00th=[ 379], | 30.00th=[ 383], 40.00th=[ 383], 50.00th=[ 388], 60.00th=[ 392], | 70.00th=[ 396], 80.00th=[ 400], 90.00th=[ 408], 95.00th=[ 416], | 99.00th=[ 441], 99.50th=[ 457], 99.90th=[ 717], 99.95th=[ 955], | 99.99th=[ 2278] bw ( KiB/s): min=165888, max=188928, per=100.00%, avg=180363.78, stdev=4442.32, samples=59 iops : min= 1296, max= 1476, avg=1409.05, stdev=34.71, samples=59 lat (usec) : 50=2.31%, 100=73.32%, 250=1.86%, 500=22.14%, 750=0.31% lat (usec) : 1000=0.06% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=4.92%, sys=10.35%, ctx=211271, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=169020,42239,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=704MiB/s (738MB/s), 704MiB/s-704MiB/s (738MB/s-738MB/s), io=20.6GiB (22.2GB), run=30001-30001msec WRITE: bw=176MiB/s (185MB/s), 176MiB/s-176MiB/s (185MB/s-185MB/s), io=5280MiB (5536MB), run=30001-30001msec Disk stats (read/write): vdc: ios=167865/41967, merge=0/0, ticks=31/30, in_queue=25271, util=84.49% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940609: Fri Jan 7 14:09:43 2022 read: IOPS=7839, BW=980MiB/s (1027MB/s)(28.8GiB/30048msec) slat (usec): min=3, max=70490, avg=94.58, stdev=654.91 clat (usec): min=1287, max=116195, avg=9761.68, stdev=10591.35 lat (usec): min=1292, max=153933, avg=9856.37, stdev=10650.12 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 8], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 11], 90.00th=[ 14], 95.00th=[ 24], | 99.00th=[ 71], 99.50th=[ 85], 99.90th=[ 102], 99.95th=[ 104], | 99.99th=[ 109] bw ( KiB/s): min=315904, max=1447168, per=100.00%, avg=1004521.20, stdev=294631.81, samples=60 iops : min= 2468, max=11306, avg=7847.78, stdev=2301.79, samples=60 write: IOPS=1963, BW=245MiB/s (257MB/s)(7375MiB/30048msec) slat (usec): min=9, max=53949, avg=122.92, stdev=826.39 clat (usec): min=1142, max=168287, avg=62370.01, stdev=19393.47 lat (msec): min=12, max=168, avg=62.49, stdev=19.42 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 31], 10.00th=[ 43], 20.00th=[ 50], | 30.00th=[ 54], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 62], | 70.00th=[ 67], 80.00th=[ 82], 90.00th=[ 93], 95.00th=[ 99], | 99.00th=[ 107], 99.50th=[ 111], 99.90th=[ 128], 99.95th=[ 140], | 99.99th=[ 150] bw ( KiB/s): min=71168, max=354816, per=100.00%, avg=251534.62, stdev=73857.88, samples=60 iops : min= 556, max= 2772, avg=1965.07, stdev=577.01, samples=60 lat (msec) : 2=0.08%, 4=4.06%, 10=58.19%, 20=12.62%, 50=7.78% lat (msec) : 100=16.56%, 250=0.73% cpu : usr=7.34%, sys=7.36%, ctx=31856, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=235548,58998,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=980MiB/s (1027MB/s), 980MiB/s-980MiB/s (1027MB/s-1027MB/s), io=28.8GiB (30.9GB), run=30048-30048msec WRITE: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=7375MiB (7733MB), run=30048-30048msec Disk stats (read/write): vdc: ios=234922/58853, merge=0/0, ticks=415915/3170546, in_queue=3682927, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941235: Fri Jan 7 14:10:20 2022 read: IOPS=1463, BW=366MiB/s (384MB/s)(10.7GiB/30001msec) clat (usec): min=62, max=2167, avg=119.62, stdev=90.64 lat (usec): min=62, max=2168, avg=119.77, stdev=90.65 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 78], 10.00th=[ 80], 20.00th=[ 84], | 30.00th=[ 87], 40.00th=[ 92], 50.00th=[ 96], 60.00th=[ 101], | 70.00th=[ 108], 80.00th=[ 118], 90.00th=[ 145], 95.00th=[ 293], | 99.00th=[ 545], 99.50th=[ 701], 99.90th=[ 889], 99.95th=[ 1057], | 99.99th=[ 1450] bw ( KiB/s): min=135680, max=880640, per=100.00%, avg=374596.75, stdev=270414.02, samples=60 iops : min= 530, max= 3440, avg=1463.27, stdev=1056.31, samples=60 write: IOPS=369, BW=92.3MiB/s (96.8MB/s)(2769MiB/30001msec) clat (usec): min=662, max=23163, avg=2200.18, stdev=2157.37 lat (usec): min=679, max=23187, avg=2225.90, stdev=2158.75 clat percentiles (usec): | 1.00th=[ 676], 5.00th=[ 685], 10.00th=[ 685], 20.00th=[ 693], | 30.00th=[ 701], 40.00th=[ 701], 50.00th=[ 709], 60.00th=[ 2180], | 70.00th=[ 2737], 80.00th=[ 4293], 90.00th=[ 5604], 95.00th=[ 6456], | 99.00th=[ 8291], 99.50th=[ 8979], 99.90th=[10945], 99.95th=[14877], | 99.99th=[21627] bw ( KiB/s): min=32768, max=216064, per=100.00%, avg=94508.93, stdev=68473.21, samples=60 iops : min= 128, max= 844, avg=369.17, stdev=267.48, samples=60 lat (usec) : 100=46.67%, 250=28.02%, 500=4.17%, 750=12.64%, 1000=0.26% lat (msec) : 2=0.09%, 4=3.73%, 10=4.39%, 20=0.02%, 50=0.01% cpu : usr=1.91%, sys=3.39%, ctx=54985, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=43897,11076,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=366MiB/s (384MB/s), 366MiB/s-366MiB/s (384MB/s-384MB/s), io=10.7GiB (11.5GB), run=30001-30001msec WRITE: bw=92.3MiB/s (96.8MB/s), 92.3MiB/s-92.3MiB/s (96.8MB/s-96.8MB/s), io=2769MiB (2904MB), run=30001-30001msec Disk stats (read/write): vdc: ios=43699/11031, merge=0/0, ticks=28/17349, in_queue=28204, util=94.35% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942446: Fri Jan 7 14:10:57 2022 read: IOPS=2227, BW=557MiB/s (584MB/s)(16.3GiB/30001msec) slat (usec): min=11, max=121, avg=17.68, stdev= 5.10 clat (nsec): min=1379, max=8636.1k, avg=92250.30, stdev=100970.22 lat (usec): min=57, max=8653, avg=110.08, stdev=101.28 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 56], 10.00th=[ 58], 20.00th=[ 60], | 30.00th=[ 62], 40.00th=[ 65], 50.00th=[ 69], 60.00th=[ 73], | 70.00th=[ 78], 80.00th=[ 85], 90.00th=[ 104], 95.00th=[ 269], | 99.00th=[ 529], 99.50th=[ 676], 99.90th=[ 881], 99.95th=[ 1045], | 99.99th=[ 1385] bw ( KiB/s): min=270336, max=947200, per=100.00%, avg=575699.75, stdev=270147.44, samples=59 iops : min= 1056, max= 3700, avg=2248.80, stdev=1055.26, samples=59 write: IOPS=557, BW=139MiB/s (146MB/s)(4184MiB/30001msec) slat (usec): min=22, max=186, avg=42.05, stdev= 9.73 clat (usec): min=603, max=15210, avg=1303.28, stdev=983.48 lat (usec): min=673, max=15240, avg=1345.51, stdev=985.07 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 660], 20.00th=[ 660], | 30.00th=[ 668], 40.00th=[ 668], 50.00th=[ 676], 60.00th=[ 685], | 70.00th=[ 2114], 80.00th=[ 2278], 90.00th=[ 2573], 95.00th=[ 3032], | 99.00th=[ 4047], 99.50th=[ 4555], 99.90th=[ 7898], 99.95th=[ 8979], | 99.99th=[15008] bw ( KiB/s): min=58880, max=234496, per=100.00%, avg=144102.24, stdev=69116.28, samples=59 iops : min= 230, max= 916, avg=562.86, stdev=269.99, samples=59 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.76%, 100=70.59% lat (usec) : 250=4.02%, 500=3.71%, 750=14.06%, 1000=0.14% lat (msec) : 2=0.10%, 4=6.40%, 10=0.21%, 20=0.01% cpu : usr=2.63%, sys=7.47%, ctx=83590, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=66840,16737,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=557MiB/s (584MB/s), 557MiB/s-557MiB/s (584MB/s-584MB/s), io=16.3GiB (17.5GB), run=30001-30001msec WRITE: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s), io=4184MiB (4388MB), run=30001-30001msec Disk stats (read/write): vdc: ios=66644/16683, merge=0/0, ticks=54/12132, in_queue=27372, util=91.56% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943469: Fri Jan 7 14:11:35 2022 read: IOPS=2741, BW=685MiB/s (719MB/s)(20.1GiB/30001msec) clat (usec): min=52, max=2754, avg=102.98, stdev=95.68 lat (usec): min=52, max=2754, avg=103.12, stdev=95.69 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 67], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 83], | 70.00th=[ 88], 80.00th=[ 94], 90.00th=[ 114], 95.00th=[ 281], | 99.00th=[ 586], 99.50th=[ 693], 99.90th=[ 947], 99.95th=[ 1188], | 99.99th=[ 1565] bw ( KiB/s): min=201728, max=978432, per=100.00%, avg=708932.75, stdev=267398.35, samples=59 iops : min= 788, max= 3822, avg=2769.25, stdev=1044.52, samples=59 write: IOPS=686, BW=172MiB/s (180MB/s)(5152MiB/30001msec) clat (usec): min=647, max=12618, avg=1012.45, stdev=786.20 lat (usec): min=668, max=12644, avg=1037.00, stdev=786.61 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 668], 10.00th=[ 668], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 685], 60.00th=[ 685], | 70.00th=[ 693], 80.00th=[ 725], 90.00th=[ 2212], 95.00th=[ 2442], | 99.00th=[ 3458], 99.50th=[ 4228], 99.90th=[ 7504], 99.95th=[ 8586], | 99.99th=[12125] bw ( KiB/s): min=38400, max=238080, per=100.00%, avg=177411.32, stdev=66757.66, samples=59 iops : min= 150, max= 930, avg=693.00, stdev=260.76, samples=59 lat (usec) : 100=68.11%, 250=6.89%, 500=3.91%, 750=17.07%, 1000=0.22% lat (msec) : 2=0.26%, 4=3.42%, 10=0.11%, 20=0.01% cpu : usr=3.25%, sys=5.92%, ctx=102866, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=82248,20607,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=685MiB/s (719MB/s), 685MiB/s-685MiB/s (719MB/s-719MB/s), io=20.1GiB (21.6GB), run=30001-30001msec WRITE: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s), io=5152MiB (5402MB), run=30001-30001msec Disk stats (read/write): vdc: ios=82006/20530, merge=0/0, ticks=70/7918, in_queue=26890, util=90.00% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943998: Fri Jan 7 14:12:11 2022 read: IOPS=3547, BW=887MiB/s (930MB/s)(25.0GiB/30016msec) slat (usec): min=3, max=288, avg=11.24, stdev= 4.93 clat (nsec): min=1007, max=35178k, avg=1117995.98, stdev=2243539.33 lat (usec): min=47, max=35187, avg=1129.36, stdev=2243.17 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 69], 50.00th=[ 78], 60.00th=[ 97], | 70.00th=[ 159], 80.00th=[ 1893], 90.00th=[ 4293], 95.00th=[ 6194], | 99.00th=[ 9503], 99.50th=[10814], 99.90th=[16909], 99.95th=[19006], | 99.99th=[31851] bw ( KiB/s): min=288256, max=1532416, per=100.00%, avg=908534.13, stdev=347731.53, samples=60 iops : min= 1126, max= 5986, avg=3548.93, stdev=1358.33, samples=60 write: IOPS=886, BW=222MiB/s (232MB/s)(6652MiB/30016msec) slat (usec): min=16, max=215, avg=40.59, stdev=13.64 clat (usec): min=1030, max=51625, avg=6714.44, stdev=2574.23 lat (usec): min=1061, max=51681, avg=6755.18, stdev=2574.99 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3654], 10.00th=[ 4146], 20.00th=[ 4752], | 30.00th=[ 5211], 40.00th=[ 5800], 50.00th=[ 6325], 60.00th=[ 6915], | 70.00th=[ 7570], 80.00th=[ 8356], 90.00th=[ 9765], 95.00th=[10814], | 99.00th=[13960], 99.50th=[17433], 99.90th=[29754], 99.95th=[34341], | 99.99th=[40633] bw ( KiB/s): min=63488, max=386560, per=100.00%, avg=227060.63, stdev=86517.72, samples=60 iops : min= 248, max= 1510, avg=886.95, stdev=337.96, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=2.67%, 100=46.25%, 250=8.51% lat (usec) : 500=2.88%, 750=0.52%, 1000=0.09% lat (msec) : 2=3.72%, 4=8.11%, 10=24.96%, 20=2.19%, 50=0.10% lat (msec) : 100=0.01% cpu : usr=4.35%, sys=7.63%, ctx=92633, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=106470,26609,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=887MiB/s (930MB/s), 887MiB/s-887MiB/s (930MB/s-930MB/s), io=25.0GiB (27.9GB), run=30016-30016msec WRITE: bw=222MiB/s (232MB/s), 222MiB/s-222MiB/s (232MB/s-232MB/s), io=6652MiB (6975MB), run=30016-30016msec Disk stats (read/write): vdc: ios=105931/26508, merge=0/0, ticks=94957/164290, in_queue=292221, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944974: Fri Jan 7 14:12:48 2022 read: IOPS=3566, BW=892MiB/s (935MB/s)(26.1GiB/30001msec) clat (usec): min=52, max=2351, avg=102.88, stdev=98.42 lat (usec): min=52, max=2351, avg=103.01, stdev=98.42 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 68], 20.00th=[ 71], | 30.00th=[ 73], 40.00th=[ 76], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 86], 80.00th=[ 92], 90.00th=[ 110], 95.00th=[ 277], | 99.00th=[ 619], 99.50th=[ 701], 99.90th=[ 1090], 99.95th=[ 1336], | 99.99th=[ 1598] bw ( KiB/s): min=585216, max=1014093, per=100.00%, avg=917548.35, stdev=58469.59, samples=60 iops : min= 2286, max= 3961, avg=3584.03, stdev=228.39, samples=60 write: IOPS=891, BW=223MiB/s (234MB/s)(6686MiB/30001msec) clat (usec): min=639, max=9071, avg=677.91, stdev=124.73 lat (usec): min=654, max=9096, avg=702.65, stdev=125.01 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 668], 50.00th=[ 668], 60.00th=[ 668], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 685], 95.00th=[ 693], | 99.00th=[ 734], 99.50th=[ 1795], 99.90th=[ 2474], 99.95th=[ 2933], | 99.99th=[ 4047] bw ( KiB/s): min=132608, max=249756, per=100.00%, avg=229343.55, stdev=14395.27, samples=60 iops : min= 518, max= 975, avg=895.75, stdev=56.19, samples=60 lat (usec) : 100=69.51%, 250=5.59%, 500=3.80%, 750=20.67%, 1000=0.22% lat (msec) : 2=0.17%, 4=0.04%, 10=0.01% cpu : usr=4.47%, sys=7.84%, ctx=133758, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=106996,26743,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=892MiB/s (935MB/s), 892MiB/s-892MiB/s (935MB/s-935MB/s), io=26.1GiB (28.0GB), run=30001-30001msec WRITE: bw=223MiB/s (234MB/s), 223MiB/s-223MiB/s (234MB/s-234MB/s), io=6686MiB (7011MB), run=30001-30001msec Disk stats (read/write): vdc: ios=106980/26737, merge=0/0, ticks=124/210, in_queue=25965, util=86.30% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945601: Fri Jan 7 14:13:26 2022 read: IOPS=4291, BW=1073MiB/s (1125MB/s)(31.4GiB/30011msec) slat (usec): min=3, max=303, avg=11.19, stdev= 7.91 clat (nsec): min=800, max=64107k, avg=4066352.59, stdev=8114696.58 lat (usec): min=46, max=64119, avg=4077.69, stdev=8113.93 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 59], | 30.00th=[ 67], 40.00th=[ 83], 50.00th=[ 109], 60.00th=[ 128], | 70.00th=[ 190], 80.00th=[ 9503], 90.00th=[17171], 95.00th=[23462], | 99.00th=[31851], 99.50th=[34866], 99.90th=[42206], 99.95th=[46400], | 99.99th=[50594] bw ( MiB/s): min= 384, max= 1501, per=100.00%, avg=1073.07, stdev=330.17, samples=60 iops : min= 1538, max= 6004, avg=4292.27, stdev=1320.68, samples=60 write: IOPS=1072, BW=268MiB/s (281MB/s)(8046MiB/30011msec) slat (usec): min=16, max=596, avg=65.47, stdev=32.11 clat (usec): min=3832, max=99935, avg=30231.38, stdev=6606.84 lat (usec): min=3926, max=99982, avg=30297.03, stdev=6603.17 clat percentiles (usec): | 1.00th=[12911], 5.00th=[21103], 10.00th=[23462], 20.00th=[25822], | 30.00th=[27395], 40.00th=[28443], 50.00th=[29754], 60.00th=[31065], | 70.00th=[32637], 80.00th=[34341], 90.00th=[36963], 95.00th=[40633], | 99.00th=[52167], 99.50th=[57410], 99.90th=[67634], 99.95th=[71828], | 99.99th=[84411] bw ( KiB/s): min=104960, max=361472, per=99.96%, avg=274439.95, stdev=83653.42, samples=60 iops : min= 410, max= 1412, avg=1072.02, stdev=326.78, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=3.63%, 100=33.03% lat (usec) : 250=22.07%, 500=2.24%, 750=0.19%, 1000=0.11% lat (msec) : 2=0.22%, 4=0.15%, 10=2.95%, 20=10.15%, 50=24.95% lat (msec) : 100=0.27% cpu : usr=8.26%, sys=8.92%, ctx=106029, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=128784,32185,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1073MiB/s (1125MB/s), 1073MiB/s-1073MiB/s (1125MB/s-1125MB/s), io=31.4GiB (33.8GB), run=30011-30011msec WRITE: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=8046MiB (8437MB), run=30011-30011msec Disk stats (read/write): vdc: ios=128348/32068, merge=0/0, ticks=480243/948745, in_queue=1468250, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946467: Fri Jan 7 14:14:02 2022 read: IOPS=3638, BW=910MiB/s (954MB/s)(26.7GiB/30001msec) clat (usec): min=52, max=2372, avg=100.03, stdev=97.01 lat (usec): min=52, max=2372, avg=100.16, stdev=97.02 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 65], 10.00th=[ 67], 20.00th=[ 69], | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 90], 90.00th=[ 110], 95.00th=[ 273], | 99.00th=[ 603], 99.50th=[ 693], 99.90th=[ 1057], 99.95th=[ 1270], | 99.99th=[ 1565] bw ( KiB/s): min=867840, max=1013248, per=99.96%, avg=931143.80, stdev=31679.37, samples=59 iops : min= 3390, max= 3958, avg=3637.25, stdev=123.73, samples=59 write: IOPS=909, BW=227MiB/s (238MB/s)(6818MiB/30001msec) clat (usec): min=637, max=2610, avg=666.93, stdev=31.32 lat (usec): min=651, max=2633, avg=692.33, stdev=32.52 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[ 668], 60.00th=[ 668], | 70.00th=[ 668], 80.00th=[ 676], 90.00th=[ 685], 95.00th=[ 693], | 99.00th=[ 717], 99.50th=[ 725], 99.90th=[ 881], 99.95th=[ 1106], | 99.99th=[ 2442] bw ( KiB/s): min=212992, max=245248, per=100.00%, avg=232816.53, stdev=5910.49, samples=59 iops : min= 832, max= 958, avg=909.36, stdev=23.07, samples=59 lat (usec) : 100=69.95%, 250=5.28%, 500=3.70%, 750=20.77%, 1000=0.19% lat (msec) : 2=0.11%, 4=0.01% cpu : usr=4.50%, sys=7.63%, ctx=136453, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=109163,27272,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=910MiB/s (954MB/s), 910MiB/s-910MiB/s (954MB/s-954MB/s), io=26.7GiB (28.6GB), run=30001-30001msec WRITE: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=6818MiB (7149MB), run=30001-30001msec Disk stats (read/write): vdc: ios=108412/27091, merge=0/0, ticks=127/20, in_queue=25993, util=86.92% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947445: Fri Jan 7 14:14:40 2022 read: IOPS=4381, BW=1095MiB/s (1149MB/s)(32.1GiB/30005msec) slat (usec): min=4, max=428, avg=11.35, stdev= 6.72 clat (nsec): min=1143, max=108665k, avg=3037965.40, stdev=9424803.61 lat (usec): min=46, max=108674, avg=3049.45, stdev=9424.66 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 60], 40.00th=[ 68], 50.00th=[ 85], 60.00th=[ 110], | 70.00th=[ 124], 80.00th=[ 163], 90.00th=[ 1958], 95.00th=[29754], | 99.00th=[40633], 99.50th=[44303], 99.90th=[51119], 99.95th=[60031], | 99.99th=[91751] bw ( MiB/s): min= 496, max= 1474, per=99.96%, avg=1094.91, stdev=197.00, samples=60 iops : min= 1984, max= 5896, avg=4379.63, stdev=788.00, samples=60 write: IOPS=1092, BW=273MiB/s (287MB/s)(8199MiB/30005msec) slat (usec): min=17, max=416, avg=51.18, stdev=24.00 clat (msec): min=8, max=229, avg=79.20, stdev=20.39 lat (msec): min=8, max=229, avg=79.26, stdev=20.39 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 36], 10.00th=[ 51], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 78], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 94], 90.00th=[ 100], 95.00th=[ 109], | 99.00th=[ 127], 99.50th=[ 134], 99.90th=[ 163], 99.95th=[ 174], | 99.99th=[ 184] bw ( KiB/s): min=108032, max=364544, per=99.98%, avg=279758.33, stdev=45916.05, samples=60 iops : min= 422, max= 1424, avg=1092.80, stdev=179.36, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=3.47%, 100=40.40% lat (usec) : 250=25.21%, 500=2.25%, 750=0.24%, 1000=0.13% lat (msec) : 2=0.33%, 4=0.24%, 10=0.34%, 20=0.52%, 50=8.75% lat (msec) : 100=16.23%, 250=1.87% cpu : usr=7.01%, sys=9.89%, ctx=130029, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=131468,32795,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1095MiB/s (1149MB/s), 1095MiB/s-1095MiB/s (1149MB/s-1149MB/s), io=32.1GiB (34.5GB), run=30005-30005msec WRITE: bw=273MiB/s (287MB/s), 273MiB/s-273MiB/s (287MB/s-287MB/s), io=8199MiB (8597MB), run=30005-30005msec Disk stats (read/write): vdc: ios=130991/32672, merge=0/0, ticks=358786/2574548, in_queue=2966896, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947986: Fri Jan 7 14:15:16 2022 read: IOPS=3606, BW=902MiB/s (945MB/s)(26.4GiB/30001msec) clat (usec): min=52, max=3129, avg=101.88, stdev=99.67 lat (usec): min=52, max=3129, avg=102.01, stdev=99.68 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 66], 10.00th=[ 67], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 77], 60.00th=[ 80], | 70.00th=[ 85], 80.00th=[ 91], 90.00th=[ 112], 95.00th=[ 277], | 99.00th=[ 627], 99.50th=[ 693], 99.90th=[ 1106], 99.95th=[ 1303], | 99.99th=[ 1549] bw ( KiB/s): min=852480, max=1003520, per=99.96%, avg=922728.14, stdev=31453.90, samples=59 iops : min= 3330, max= 3920, avg=3604.41, stdev=122.87, samples=59 write: IOPS=901, BW=225MiB/s (236MB/s)(6758MiB/30001msec) clat (usec): min=638, max=9051, avg=670.21, stdev=72.55 lat (usec): min=655, max=9130, avg=694.47, stdev=73.04 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[ 668], 60.00th=[ 668], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 685], 95.00th=[ 693], | 99.00th=[ 725], 99.50th=[ 750], 99.90th=[ 1090], 99.95th=[ 1319], | 99.99th=[ 3523] bw ( KiB/s): min=209920, max=244224, per=100.00%, avg=230755.80, stdev=6638.49, samples=59 iops : min= 820, max= 954, avg=901.39, stdev=25.93, samples=59 lat (usec) : 100=69.71%, 250=5.40%, 500=3.75%, 750=20.77%, 1000=0.23% lat (msec) : 2=0.13%, 4=0.01%, 10=0.01% cpu : usr=4.34%, sys=7.84%, ctx=135232, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=108184,27031,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=902MiB/s (945MB/s), 902MiB/s-902MiB/s (945MB/s-945MB/s), io=26.4GiB (28.4GB), run=30001-30001msec WRITE: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=6758MiB (7086MB), run=30001-30001msec Disk stats (read/write): vdc: ios=107636/26892, merge=0/0, ticks=135/39, in_queue=26046, util=86.93% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949344: Fri Jan 7 14:15:53 2022 read: IOPS=4470, BW=1118MiB/s (1172MB/s)(32.7GiB/30007msec) slat (usec): min=4, max=91735, avg=163.58, stdev=981.79 clat (usec): min=131, max=143545, avg=17123.09, stdev=13330.02 lat (msec): min=3, max=186, avg=17.29, stdev=13.41 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 8], 10.00th=[ 9], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 13], 50.00th=[ 14], 60.00th=[ 15], | 70.00th=[ 17], 80.00th=[ 19], 90.00th=[ 26], 95.00th=[ 45], | 99.00th=[ 81], 99.50th=[ 106], 99.90th=[ 124], 99.95th=[ 127], | 99.99th=[ 131] bw ( MiB/s): min= 452, max= 1472, per=99.94%, avg=1116.89, stdev=210.84, samples=60 iops : min= 1810, max= 5888, avg=4467.58, stdev=843.40, samples=60 write: IOPS=1115, BW=279MiB/s (292MB/s)(8366MiB/30007msec) slat (usec): min=17, max=71403, avg=230.86, stdev=794.14 clat (msec): min=17, max=219, avg=109.78, stdev=24.12 lat (msec): min=17, max=219, avg=110.01, stdev=24.13 clat percentiles (msec): | 1.00th=[ 34], 5.00th=[ 61], 10.00th=[ 85], 20.00th=[ 94], | 30.00th=[ 101], 40.00th=[ 106], 50.00th=[ 111], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 130], 90.00th=[ 138], 95.00th=[ 144], | 99.00th=[ 157], 99.50th=[ 165], 99.90th=[ 201], 99.95th=[ 211], | 99.99th=[ 215] bw ( KiB/s): min=111616, max=363008, per=99.87%, avg=285124.73, stdev=49639.33, samples=60 iops : min= 436, max= 1418, avg=1113.73, stdev=193.87, samples=60 lat (usec) : 250=0.01% lat (msec) : 4=0.02%, 10=13.23%, 20=54.75%, 50=10.02%, 100=7.36% lat (msec) : 250=14.61% cpu : usr=10.02%, sys=5.34%, ctx=17631, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=134143,33463,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1118MiB/s (1172MB/s), 1118MiB/s-1118MiB/s (1172MB/s-1172MB/s), io=32.7GiB (35.2GB), run=30007-30007msec WRITE: bw=279MiB/s (292MB/s), 279MiB/s-279MiB/s (292MB/s-292MB/s), io=8366MiB (8772MB), run=30007-30007msec Disk stats (read/write): vdc: ios=133845/33376, merge=0/0, ticks=427611/3176877, in_queue=3697061, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950054: Fri Jan 7 14:16:31 2022 read: IOPS=1183, BW=592MiB/s (620MB/s)(17.3GiB/30001msec) clat (usec): min=112, max=2136, avg=200.02, stdev=145.26 lat (usec): min=112, max=2136, avg=200.17, stdev=145.27 clat percentiles (usec): | 1.00th=[ 123], 5.00th=[ 127], 10.00th=[ 129], 20.00th=[ 135], | 30.00th=[ 139], 40.00th=[ 145], 50.00th=[ 153], 60.00th=[ 163], | 70.00th=[ 176], 80.00th=[ 202], 90.00th=[ 338], 95.00th=[ 469], | 99.00th=[ 848], 99.50th=[ 1172], 99.90th=[ 1418], 99.95th=[ 1532], | 99.99th=[ 1958] bw ( KiB/s): min=192512, max=993280, per=100.00%, avg=612369.36, stdev=307711.90, samples=59 iops : min= 376, max= 1940, avg=1196.03, stdev=601.00, samples=59 write: IOPS=297, BW=149MiB/s (156MB/s)(4466MiB/30001msec) clat (usec): min=1254, max=25216, avg=2503.96, stdev=2357.66 lat (usec): min=1284, max=25267, avg=2555.39, stdev=2361.21 clat percentiles (usec): | 1.00th=[ 1270], 5.00th=[ 1270], 10.00th=[ 1270], 20.00th=[ 1287], | 30.00th=[ 1287], 40.00th=[ 1287], 50.00th=[ 1303], 60.00th=[ 1319], | 70.00th=[ 1352], 80.00th=[ 3490], 90.00th=[ 6849], 95.00th=[ 7635], | 99.00th=[ 9372], 99.50th=[10683], 99.90th=[17171], 99.95th=[19792], | 99.99th=[25297] bw ( KiB/s): min=51200, max=257024, per=100.00%, avg=153877.69, stdev=77932.14, samples=59 iops : min= 100, max= 502, avg=300.54, stdev=152.21, samples=59 lat (usec) : 250=68.60%, 500=7.87%, 750=2.12%, 1000=0.80% lat (msec) : 2=15.25%, 4=1.80%, 10=3.45%, 20=0.10%, 50=0.01% cpu : usr=2.15%, sys=3.43%, ctx=44440, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=35503,8931,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=592MiB/s (620MB/s), 592MiB/s-592MiB/s (620MB/s-620MB/s), io=17.3GiB (18.6GB), run=30001-30001msec WRITE: bw=149MiB/s (156MB/s), 149MiB/s-149MiB/s (156MB/s-156MB/s), io=4466MiB (4682MB), run=30001-30001msec Disk stats (read/write): vdc: ios=35404/8902, merge=0/0, ticks=216/18933, in_queue=28032, util=93.82% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950870: Fri Jan 7 14:17:08 2022 read: IOPS=1521, BW=761MiB/s (798MB/s)(22.3GiB/30003msec) slat (usec): min=17, max=115, avg=28.18, stdev= 6.12 clat (usec): min=37, max=2123, avg=153.38, stdev=146.31 lat (usec): min=96, max=2146, avg=181.73, stdev=146.66 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 90], 10.00th=[ 91], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 102], 50.00th=[ 108], 60.00th=[ 114], | 70.00th=[ 123], 80.00th=[ 137], 90.00th=[ 293], 95.00th=[ 424], | 99.00th=[ 807], 99.50th=[ 1156], 99.90th=[ 1418], 99.95th=[ 1680], | 99.99th=[ 2008] bw ( KiB/s): min=214016, max=1100800, per=100.00%, avg=779229.87, stdev=277007.56, samples=60 iops : min= 418, max= 2150, avg=1521.93, stdev=541.03, samples=60 write: IOPS=384, BW=192MiB/s (202MB/s)(5767MiB/30003msec) slat (usec): min=39, max=272, avg=74.56, stdev=16.62 clat (usec): min=1190, max=20320, avg=1799.84, stdev=1234.56 lat (usec): min=1259, max=20393, avg=1874.60, stdev=1236.72 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1254], 50.00th=[ 1254], 60.00th=[ 1254], | 70.00th=[ 1270], 80.00th=[ 2769], 90.00th=[ 3326], 95.00th=[ 3982], | 99.00th=[ 6718], 99.50th=[ 8717], 99.90th=[ 9896], 99.95th=[10945], | 99.99th=[17957] bw ( KiB/s): min=60416, max=272384, per=100.00%, avg=196812.80, stdev=67907.68, samples=60 iops : min= 118, max= 532, avg=384.40, stdev=132.63, samples=60 lat (usec) : 50=0.01%, 100=28.57%, 250=41.80%, 500=6.74%, 750=1.70% lat (usec) : 1000=0.53% lat (msec) : 2=16.19%, 4=3.48%, 10=0.97%, 20=0.02%, 50=0.01% cpu : usr=2.76%, sys=7.03%, ctx=57217, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=45658,11533,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=761MiB/s (798MB/s), 761MiB/s-761MiB/s (798MB/s-798MB/s), io=22.3GiB (23.9GB), run=30003-30003msec WRITE: bw=192MiB/s (202MB/s), 192MiB/s-192MiB/s (202MB/s-202MB/s), io=5767MiB (6047MB), run=30003-30003msec Disk stats (read/write): vdc: ios=45536/11497, merge=0/0, ticks=285/16935, in_queue=27392, util=91.69% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951820: Fri Jan 7 14:17:45 2022 read: IOPS=1772, BW=886MiB/s (929MB/s)(25.0GiB/30001msec) clat (usec): min=77, max=2829, avg=157.61, stdev=152.13 lat (usec): min=77, max=2829, avg=157.74, stdev=152.13 clat percentiles (usec): | 1.00th=[ 88], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 99], | 30.00th=[ 103], 40.00th=[ 108], 50.00th=[ 112], 60.00th=[ 117], | 70.00th=[ 124], 80.00th=[ 135], 90.00th=[ 297], 95.00th=[ 429], | 99.00th=[ 865], 99.50th=[ 1156], 99.90th=[ 1549], 99.95th=[ 1811], | 99.99th=[ 2057] bw ( KiB/s): min=327680, max=1140736, per=100.00%, avg=907538.78, stdev=245874.46, samples=60 iops : min= 640, max= 2228, avg=1772.52, stdev=480.25, samples=60 write: IOPS=444, BW=222MiB/s (233MB/s)(6669MiB/30001msec) clat (usec): min=1215, max=16475, avg=1563.47, stdev=903.30 lat (usec): min=1247, max=16525, avg=1612.72, stdev=903.59 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1254], 40.00th=[ 1254], 50.00th=[ 1254], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1270], 90.00th=[ 3032], 95.00th=[ 3425], | 99.00th=[ 4948], 99.50th=[ 5735], 99.90th=[ 8586], 99.95th=[11469], | 99.99th=[16188] bw ( KiB/s): min=76800, max=285696, per=100.00%, avg=227635.10, stdev=63780.58, samples=60 iops : min= 150, max= 558, avg=444.57, stdev=124.59, samples=60 lat (usec) : 100=19.09%, 250=51.20%, 500=6.85%, 750=1.62%, 1000=0.63% lat (msec) : 2=17.93%, 4=2.17%, 10=0.49%, 20=0.01% cpu : usr=3.34%, sys=4.90%, ctx=66528, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=53172,13338,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=886MiB/s (929MB/s), 886MiB/s-886MiB/s (929MB/s-929MB/s), io=25.0GiB (27.9GB), run=30001-30001msec WRITE: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=6669MiB (6993MB), run=30001-30001msec Disk stats (read/write): vdc: ios=52972/13305, merge=0/0, ticks=352/16889, in_queue=27189, util=91.00% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952402: Fri Jan 7 14:18:22 2022 read: IOPS=2060, BW=1030MiB/s (1080MB/s)(30.2GiB/30004msec) slat (usec): min=5, max=310, avg=19.20, stdev= 8.15 clat (nsec): min=1095, max=35699k, avg=2020317.93, stdev=4119122.83 lat (usec): min=70, max=35719, avg=2039.68, stdev=4118.82 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 81], | 30.00th=[ 90], 40.00th=[ 103], 50.00th=[ 128], 60.00th=[ 163], | 70.00th=[ 233], 80.00th=[ 2802], 90.00th=[ 8848], 95.00th=[11994], | 99.00th=[16909], 99.50th=[19006], 99.90th=[23725], 99.95th=[24773], | 99.99th=[26870] bw ( MiB/s): min= 483, max= 1442, per=100.00%, avg=1030.21, stdev=244.25, samples=60 iops : min= 966, max= 2884, avg=2060.45, stdev=488.53, samples=60 write: IOPS=516, BW=258MiB/s (271MB/s)(7755MiB/30004msec) slat (usec): min=30, max=440, avg=89.77, stdev=38.54 clat (usec): min=1871, max=39961, avg=11118.40, stdev=4083.99 lat (usec): min=1908, max=40056, avg=11208.37, stdev=4083.41 clat percentiles (usec): | 1.00th=[ 3785], 5.00th=[ 5538], 10.00th=[ 6521], 20.00th=[ 7832], | 30.00th=[ 8848], 40.00th=[ 9634], 50.00th=[10552], 60.00th=[11469], | 70.00th=[12649], 80.00th=[13960], 90.00th=[16188], 95.00th=[18482], | 99.00th=[23725], 99.50th=[26346], 99.90th=[32375], 99.95th=[35914], | 99.99th=[40109] bw ( KiB/s): min=131072, max=373760, per=100.00%, avg=264652.28, stdev=62577.43, samples=60 iops : min= 256, max= 730, avg=516.88, stdev=122.23, samples=60 lat (usec) : 2=0.01%, 50=0.03%, 100=30.55%, 250=26.17%, 500=3.21% lat (usec) : 750=0.78%, 1000=0.54% lat (msec) : 2=0.99%, 4=3.82%, 10=16.20%, 20=16.77%, 50=0.93% cpu : usr=4.97%, sys=6.28%, ctx=56031, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61816,15509,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1030MiB/s (1080MB/s), 1030MiB/s-1030MiB/s (1080MB/s-1080MB/s), io=30.2GiB (32.4GB), run=30004-30004msec WRITE: bw=258MiB/s (271MB/s), 258MiB/s-258MiB/s (271MB/s-271MB/s), io=7755MiB (8131MB), run=30004-30004msec Disk stats (read/write): vdc: ios=61516/15450, merge=0/0, ticks=108428/163768, in_queue=292913, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953230: Fri Jan 7 14:18:59 2022 read: IOPS=2062, BW=1031MiB/s (1081MB/s)(30.2GiB/30001msec) clat (usec): min=78, max=2456, avg=156.50, stdev=160.11 lat (usec): min=78, max=2457, avg=156.63, stdev=160.12 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 93], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 100], 40.00th=[ 104], 50.00th=[ 109], 60.00th=[ 114], | 70.00th=[ 120], 80.00th=[ 133], 90.00th=[ 297], 95.00th=[ 437], | 99.00th=[ 889], 99.50th=[ 1172], 99.90th=[ 1762], 99.95th=[ 2008], | 99.99th=[ 2089] bw ( MiB/s): min= 917, max= 1153, per=100.00%, avg=1031.75, stdev=49.78, samples=59 iops : min= 1834, max= 2306, avg=2063.49, stdev=99.55, samples=59 write: IOPS=517, BW=259MiB/s (271MB/s)(7763MiB/30001msec) clat (usec): min=1217, max=8398, avg=1252.52, stdev=98.43 lat (usec): min=1243, max=8423, avg=1300.55, stdev=99.05 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1237], 50.00th=[ 1254], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1254], 90.00th=[ 1270], 95.00th=[ 1270], | 99.00th=[ 1303], 99.50th=[ 1352], 99.90th=[ 2638], 99.95th=[ 3130], | 99.99th=[ 5735] bw ( KiB/s): min=249856, max=285696, per=100.00%, avg=265467.75, stdev=7277.66, samples=59 iops : min= 488, max= 558, avg=518.49, stdev=14.21, samples=59 lat (usec) : 100=24.36%, 250=45.89%, 500=6.80%, 750=1.60%, 1000=0.67% lat (msec) : 2=20.62%, 4=0.07%, 10=0.01% cpu : usr=3.87%, sys=5.49%, ctx=77419, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=61878,15526,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1031MiB/s (1081MB/s), 1031MiB/s-1031MiB/s (1081MB/s-1081MB/s), io=30.2GiB (32.4GB), run=30001-30001msec WRITE: bw=259MiB/s (271MB/s), 259MiB/s-259MiB/s (271MB/s-271MB/s), io=7763MiB (8140MB), run=30001-30001msec Disk stats (read/write): vdc: ios=61454/15439, merge=0/0, ticks=490/15446, in_queue=26785, util=89.63% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954256: Fri Jan 7 14:19:36 2022 read: IOPS=2188, BW=1094MiB/s (1148MB/s)(32.1GiB/30011msec) slat (usec): min=5, max=292, avg=14.40, stdev= 8.93 clat (nsec): min=1200, max=134375k, avg=8158671.70, stdev=18351642.94 lat (usec): min=70, max=134390, avg=8173.20, stdev=18351.51 clat percentiles (usec): | 1.00th=[ 68], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 78], | 30.00th=[ 88], 40.00th=[ 133], 50.00th=[ 186], 60.00th=[ 212], | 70.00th=[ 343], 80.00th=[ 848], 90.00th=[ 40633], 95.00th=[ 53216], | 99.00th=[ 72877], 99.50th=[ 83362], 99.90th=[102237], 99.95th=[107480], | 99.99th=[119014] bw ( MiB/s): min= 523, max= 1517, per=99.98%, avg=1094.16, stdev=224.75, samples=60 iops : min= 1046, max= 3034, avg=2188.30, stdev=449.49, samples=60 write: IOPS=548, BW=274MiB/s (287MB/s)(8227MiB/30011msec) slat (usec): min=33, max=1329, avg=101.25, stdev=50.49 clat (msec): min=8, max=166, avg=58.46, stdev=14.53 lat (msec): min=8, max=166, avg=58.56, stdev=14.53 clat percentiles (msec): | 1.00th=[ 29], 5.00th=[ 42], 10.00th=[ 45], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 54], 50.00th=[ 57], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 68], 90.00th=[ 75], 95.00th=[ 83], | 99.00th=[ 109], 99.50th=[ 125], 99.90th=[ 159], 99.95th=[ 163], | 99.99th=[ 167] bw ( KiB/s): min=126976, max=354304, per=99.93%, avg=280503.28, stdev=55938.20, samples=60 iops : min= 248, max= 692, avg=547.82, stdev=109.23, samples=60 lat (usec) : 2=0.01%, 50=0.01%, 100=27.81%, 250=24.03%, 500=8.57% lat (usec) : 750=3.00%, 1000=1.01% lat (msec) : 2=0.52%, 4=0.21%, 10=0.28%, 20=0.71%, 50=14.38% lat (msec) : 100=19.06%, 250=0.40% cpu : usr=6.06%, sys=5.32%, ctx=57729, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=65689,16453,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1094MiB/s (1148MB/s), 1094MiB/s-1094MiB/s (1148MB/s-1148MB/s), io=32.1GiB (34.4GB), run=30011-30011msec WRITE: bw=274MiB/s (287MB/s), 274MiB/s-274MiB/s (287MB/s-287MB/s), io=8227MiB (8626MB), run=30011-30011msec Disk stats (read/write): vdc: ios=65655/16437, merge=0/0, ticks=515112/951581, in_queue=1491121, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954782: Fri Jan 7 14:20:13 2022 read: IOPS=2088, BW=1044MiB/s (1095MB/s)(30.6GiB/30002msec) clat (usec): min=72, max=2903, avg=150.38, stdev=155.43 lat (usec): min=72, max=2903, avg=150.52, stdev=155.43 clat percentiles (usec): | 1.00th=[ 81], 5.00th=[ 85], 10.00th=[ 87], 20.00th=[ 91], | 30.00th=[ 95], 40.00th=[ 99], 50.00th=[ 104], 60.00th=[ 109], | 70.00th=[ 116], 80.00th=[ 128], 90.00th=[ 293], 95.00th=[ 424], | 99.00th=[ 865], 99.50th=[ 1156], 99.90th=[ 1598], 99.95th=[ 1844], | 99.99th=[ 2114] bw ( MiB/s): min= 916, max= 1158, per=100.00%, avg=1044.27, stdev=49.79, samples=60 iops : min= 1832, max= 2316, avg=2088.53, stdev=99.57, samples=60 write: IOPS=524, BW=262MiB/s (275MB/s)(7874MiB/30002msec) clat (usec): min=1211, max=2848, avg=1247.71, stdev=35.26 lat (usec): min=1241, max=2898, avg=1298.83, stdev=40.31 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1221], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1237], 50.00th=[ 1237], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1254], 90.00th=[ 1270], 95.00th=[ 1270], | 99.00th=[ 1303], 99.50th=[ 1319], 99.90th=[ 1582], 99.95th=[ 2180], | 99.99th=[ 2704] bw ( KiB/s): min=248832, max=287744, per=100.00%, avg=268757.82, stdev=7872.00, samples=60 iops : min= 486, max= 562, avg=524.92, stdev=15.37, samples=60 lat (usec) : 100=33.09%, 250=37.29%, 500=6.78%, 750=1.60%, 1000=0.57% lat (msec) : 2=20.64%, 4=0.03% cpu : usr=4.19%, sys=4.16%, ctx=78414, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62654,15748,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1044MiB/s (1095MB/s), 1044MiB/s-1044MiB/s (1095MB/s-1095MB/s), io=30.6GiB (32.8GB), run=30002-30002msec WRITE: bw=262MiB/s (275MB/s), 262MiB/s-262MiB/s (275MB/s-275MB/s), io=7874MiB (8256MB), run=30002-30002msec Disk stats (read/write): vdc: ios=62610/15735, merge=0/0, ticks=459/15741, in_queue=27284, util=90.67% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956008: Fri Jan 7 14:20:50 2022 read: IOPS=2245, BW=1123MiB/s (1178MB/s)(32.9GiB/30033msec) slat (usec): min=5, max=504, avg=13.34, stdev= 9.43 clat (nsec): min=1765, max=245350k, avg=8319759.91, stdev=20038196.96 lat (usec): min=72, max=245361, avg=8333.24, stdev=20038.16 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 73], 10.00th=[ 75], 20.00th=[ 79], | 30.00th=[ 89], 40.00th=[ 133], 50.00th=[ 188], 60.00th=[ 212], | 70.00th=[ 334], 80.00th=[ 750], 90.00th=[ 44827], 95.00th=[ 55837], | 99.00th=[ 77071], 99.50th=[ 86508], 99.90th=[135267], 99.95th=[179307], | 99.99th=[221250] bw ( MiB/s): min= 685, max= 1377, per=100.00%, avg=1123.32, stdev=148.09, samples=60 iops : min= 1370, max= 2754, avg=2246.62, stdev=296.17, samples=60 write: IOPS=561, BW=281MiB/s (294MB/s)(8433MiB/30033msec) slat (usec): min=29, max=377, avg=99.03, stdev=47.89 clat (msec): min=15, max=353, avg=144.62, stdev=33.84 lat (msec): min=15, max=353, avg=144.72, stdev=33.83 clat percentiles (msec): | 1.00th=[ 43], 5.00th=[ 85], 10.00th=[ 113], 20.00th=[ 124], | 30.00th=[ 132], 40.00th=[ 140], 50.00th=[ 146], 60.00th=[ 153], | 70.00th=[ 159], 80.00th=[ 169], 90.00th=[ 180], 95.00th=[ 190], | 99.00th=[ 243], 99.50th=[ 268], 99.90th=[ 317], 99.95th=[ 326], | 99.99th=[ 351] bw ( KiB/s): min=151552, max=353280, per=99.90%, avg=287212.13, stdev=37235.99, samples=60 iops : min= 296, max= 690, avg=560.95, stdev=72.72, samples=60 lat (usec) : 2=0.01%, 50=0.01%, 100=27.94%, 250=23.98%, 500=8.90% lat (usec) : 750=3.21%, 1000=1.30% lat (msec) : 2=0.85%, 4=0.46%, 10=0.63%, 20=0.48%, 50=6.46% lat (msec) : 100=7.02%, 250=18.63%, 500=0.15% cpu : usr=6.29%, sys=4.97%, ctx=59466, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=67454,16865,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1123MiB/s (1178MB/s), 1123MiB/s-1123MiB/s (1178MB/s-1178MB/s), io=32.9GiB (35.4GB), run=30033-30033msec WRITE: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=8433MiB (8842MB), run=30033-30033msec Disk stats (read/write): vdc: ios=67392/16827, merge=0/0, ticks=537200/2426564, in_queue=2991986, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956766: Fri Jan 7 14:21:27 2022 read: IOPS=2067, BW=1034MiB/s (1084MB/s)(30.3GiB/30001msec) clat (usec): min=76, max=2343, avg=155.57, stdev=157.33 lat (usec): min=76, max=2343, avg=155.70, stdev=157.33 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 92], 10.00th=[ 93], 20.00th=[ 95], | 30.00th=[ 98], 40.00th=[ 102], 50.00th=[ 106], 60.00th=[ 113], | 70.00th=[ 120], 80.00th=[ 135], 90.00th=[ 297], 95.00th=[ 433], | 99.00th=[ 889], 99.50th=[ 1172], 99.90th=[ 1598], 99.95th=[ 1876], | 99.99th=[ 2089] bw ( MiB/s): min= 939, max= 1154, per=100.00%, avg=1033.82, stdev=47.02, samples=60 iops : min= 1878, max= 2308, avg=2067.62, stdev=94.06, samples=60 write: IOPS=519, BW=260MiB/s (272MB/s)(7789MiB/30001msec) clat (usec): min=1214, max=3530, avg=1248.41, stdev=37.06 lat (usec): min=1245, max=3558, avg=1298.38, stdev=39.57 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1237], 50.00th=[ 1237], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1254], 90.00th=[ 1270], 95.00th=[ 1270], | 99.00th=[ 1303], 99.50th=[ 1336], 99.90th=[ 1647], 99.95th=[ 1762], | 99.99th=[ 2769] bw ( KiB/s): min=247808, max=288768, per=100.00%, avg=265882.43, stdev=6862.08, samples=60 iops : min= 484, max= 564, avg=519.30, stdev=13.40, samples=60 lat (usec) : 100=29.03%, 250=41.24%, 500=6.74%, 750=1.66%, 1000=0.66% lat (msec) : 2=20.65%, 4=0.03% cpu : usr=3.90%, sys=5.52%, ctx=77614, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=62026,15578,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1034MiB/s (1084MB/s), 1034MiB/s-1034MiB/s (1084MB/s-1084MB/s), io=30.3GiB (32.5GB), run=30001-30001msec WRITE: bw=260MiB/s (272MB/s), 260MiB/s-260MiB/s (272MB/s-272MB/s), io=7789MiB (8167MB), run=30001-30001msec Disk stats (read/write): vdc: ios=61548/15456, merge=0/0, ticks=462/15464, in_queue=26784, util=89.54% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957728: Fri Jan 7 14:22:04 2022 read: IOPS=2283, BW=1142MiB/s (1197MB/s)(33.5GiB/30020msec) slat (usec): min=6, max=183009, avg=320.25, stdev=1920.96 clat (msec): min=8, max=269, avg=35.31, stdev=23.91 lat (msec): min=8, max=269, avg=35.63, stdev=24.01 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 19], 20.00th=[ 22], | 30.00th=[ 24], 40.00th=[ 26], 50.00th=[ 28], 60.00th=[ 31], | 70.00th=[ 34], 80.00th=[ 41], 90.00th=[ 69], 95.00th=[ 86], | 99.00th=[ 128], 99.50th=[ 159], 99.90th=[ 207], 99.95th=[ 222], | 99.99th=[ 255] bw ( MiB/s): min= 673, max= 1375, per=99.85%, avg=1140.01, stdev=147.53, samples=60 iops : min= 1346, max= 2750, avg=2280.02, stdev=295.06, samples=60 write: IOPS=571, BW=286MiB/s (300MB/s)(8580MiB/30020msec) slat (usec): min=58, max=54880, avg=459.14, stdev=1302.07 clat (msec): min=2, max=420, avg=206.99, stdev=42.99 lat (msec): min=34, max=421, avg=207.45, stdev=42.98 clat percentiles (msec): | 1.00th=[ 70], 5.00th=[ 138], 10.00th=[ 165], 20.00th=[ 180], | 30.00th=[ 190], 40.00th=[ 199], 50.00th=[ 207], 60.00th=[ 215], | 70.00th=[ 226], 80.00th=[ 236], 90.00th=[ 255], 95.00th=[ 271], | 99.00th=[ 321], 99.50th=[ 359], 99.90th=[ 401], 99.95th=[ 409], | 99.99th=[ 422] bw ( KiB/s): min=198656, max=363520, per=99.77%, avg=291986.62, stdev=32254.40, samples=60 iops : min= 388, max= 710, avg=570.28, stdev=63.00, samples=60 lat (msec) : 4=0.01%, 10=0.06%, 20=11.84%, 50=55.95%, 100=10.61% lat (msec) : 250=19.16%, 500=2.38% cpu : usr=9.87%, sys=3.73%, ctx=8573, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=68547,17159,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1142MiB/s (1197MB/s), 1142MiB/s-1142MiB/s (1197MB/s-1197MB/s), io=33.5GiB (35.9GB), run=30020-30020msec WRITE: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=8580MiB (8996MB), run=30020-30020msec Disk stats (read/write): vdc: ios=68469/17108, merge=0/0, ticks=592331/3077445, in_queue=3710591, util=99.47% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958713: Fri Jan 7 14:22:41 2022 read: IOPS=995, BW=995MiB/s (1044MB/s)(29.2GiB/30004msec) clat (usec): min=132, max=3067, avg=273.26, stdev=238.15 lat (usec): min=132, max=3067, avg=273.42, stdev=238.15 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 149], 10.00th=[ 151], 20.00th=[ 157], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 178], 60.00th=[ 192], | 70.00th=[ 217], 80.00th=[ 351], 90.00th=[ 523], 95.00th=[ 758], | 99.00th=[ 1336], 99.50th=[ 1549], 99.90th=[ 2114], 99.95th=[ 2212], | 99.99th=[ 3032] bw ( KiB/s): min=296960, max=1316864, per=100.00%, avg=1019221.33, stdev=226596.99, samples=60 iops : min= 290, max= 1286, avg=995.37, stdev=221.28, samples=60 write: IOPS=251, BW=251MiB/s (263MB/s)(7538MiB/30004msec) clat (usec): min=2373, max=26643, avg=2789.98, stdev=1236.98 lat (usec): min=2427, max=26765, avg=2889.10, stdev=1241.20 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2409], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2442], 90.00th=[ 4228], 95.00th=[ 5080], | 99.00th=[ 8717], 99.50th=[10552], 99.90th=[12911], 99.95th=[15139], | 99.99th=[26608] bw ( KiB/s): min=79872, max=309248, per=100.00%, avg=257262.93, stdev=55300.62, samples=60 iops : min= 78, max= 302, avg=251.23, stdev=54.00, samples=60 lat (usec) : 250=59.39%, 500=11.72%, 750=4.61%, 1000=2.29% lat (msec) : 2=1.73%, 4=17.84%, 10=2.28%, 20=0.14%, 50=0.01% cpu : usr=3.15%, sys=3.97%, ctx=37411, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=29861,7538,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=995MiB/s (1044MB/s), 995MiB/s-995MiB/s (1044MB/s-1044MB/s), io=29.2GiB (31.3GB), run=30004-30004msec WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=7538MiB (7904MB), run=30004-30004msec Disk stats (read/write): vdc: ios=29854/7533, merge=0/0, ticks=670/17868, in_queue=27671, util=92.01% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959292: Fri Jan 7 14:23:18 2022 read: IOPS=1015, BW=1016MiB/s (1065MB/s)(29.8GiB/30006msec) slat (usec): min=29, max=424, avg=43.53, stdev= 8.86 clat (usec): min=117, max=13580, avg=270.85, stdev=249.83 lat (usec): min=164, max=13655, avg=314.56, stdev=250.49 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 147], 20.00th=[ 151], | 30.00th=[ 157], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 192], | 70.00th=[ 219], 80.00th=[ 347], 90.00th=[ 519], 95.00th=[ 750], | 99.00th=[ 1336], 99.50th=[ 1532], 99.90th=[ 2073], 99.95th=[ 2180], | 99.99th=[ 2835] bw ( KiB/s): min=440320, max=1245184, per=100.00%, avg=1040250.65, stdev=136656.56, samples=60 iops : min= 430, max= 1216, avg=1015.83, stdev=133.46, samples=60 write: IOPS=256, BW=256MiB/s (269MB/s)(7696MiB/30006msec) slat (usec): min=69, max=475, avg=133.72, stdev=28.85 clat (usec): min=2334, max=18248, avg=2510.92, stdev=710.78 lat (usec): min=2456, max=18385, avg=2644.86, stdev=713.17 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2409], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2442], 90.00th=[ 2442], 95.00th=[ 2474], | 99.00th=[ 6980], 99.50th=[ 8094], 99.90th=[10552], 99.95th=[12125], | 99.99th=[18220] bw ( KiB/s): min=108544, max=288768, per=100.00%, avg=262682.32, stdev=30203.18, samples=60 iops : min= 106, max= 282, avg=256.50, stdev=29.49, samples=60 lat (usec) : 250=59.19%, 500=11.97%, 750=4.67%, 1000=2.20% lat (msec) : 2=1.69%, 4=19.87%, 10=0.39%, 20=0.03% cpu : usr=3.24%, sys=6.38%, ctx=38186, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=30473,7696,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1016MiB/s (1065MB/s), 1016MiB/s-1016MiB/s (1065MB/s-1065MB/s), io=29.8GiB (31.0GB), run=30006-30006msec WRITE: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=7696MiB (8070MB), run=30006-30006msec Disk stats (read/write): vdc: ios=30376/7667, merge=0/0, ticks=752/15946, in_queue=27537, util=92.14% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960092: Fri Jan 7 14:23:55 2022 read: IOPS=1072, BW=1072MiB/s (1124MB/s)(31.4GiB/30003msec) clat (usec): min=122, max=3007, avg=258.82, stdev=242.70 lat (usec): min=122, max=3008, avg=258.99, stdev=242.70 clat percentiles (usec): | 1.00th=[ 131], 5.00th=[ 135], 10.00th=[ 137], 20.00th=[ 143], | 30.00th=[ 147], 40.00th=[ 153], 50.00th=[ 163], 60.00th=[ 176], | 70.00th=[ 200], 80.00th=[ 334], 90.00th=[ 506], 95.00th=[ 750], | 99.00th=[ 1352], 99.50th=[ 1598], 99.90th=[ 2114], 99.95th=[ 2278], | 99.99th=[ 2606] bw ( MiB/s): min= 466, max= 1288, per=100.00%, avg=1072.27, stdev=168.37, samples=60 iops : min= 466, max= 1288, avg=1072.25, stdev=168.37, samples=60 write: IOPS=271, BW=271MiB/s (284MB/s)(8138MiB/30003msec) clat (usec): min=2363, max=11751, avg=2556.98, stdev=714.27 lat (usec): min=2421, max=11864, avg=2654.86, stdev=715.63 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2376], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2442], 95.00th=[ 4015], | 99.00th=[ 6194], 99.50th=[ 6980], 99.90th=[ 8979], 99.95th=[10028], | 99.99th=[11731] bw ( KiB/s): min=137216, max=315392, per=100.00%, avg=277742.55, stdev=38000.49, samples=60 iops : min= 134, max= 308, avg=271.22, stdev=37.11, samples=60 lat (usec) : 250=60.21%, 500=11.44%, 750=4.26%, 1000=2.02% lat (msec) : 2=1.76%, 4=19.28%, 10=1.02%, 20=0.01% cpu : usr=3.50%, sys=2.93%, ctx=40311, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32168,8138,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1072MiB/s (1124MB/s), 1072MiB/s-1072MiB/s (1124MB/s-1124MB/s), io=31.4GiB (33.7GB), run=30003-30003msec WRITE: bw=271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=8138MiB (8533MB), run=30003-30003msec Disk stats (read/write): vdc: ios=32168/8137, merge=0/0, ticks=766/17499, in_queue=27989, util=93.00% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961057: Fri Jan 7 14:24:32 2022 read: IOPS=1123, BW=1124MiB/s (1178MB/s)(32.9GiB/30012msec) slat (usec): min=9, max=181, avg=22.09, stdev=10.82 clat (usec): min=71, max=52858, avg=4403.61, stdev=7810.69 lat (usec): min=117, max=52874, avg=4425.85, stdev=7810.80 clat percentiles (usec): | 1.00th=[ 110], 5.00th=[ 114], 10.00th=[ 118], 20.00th=[ 133], | 30.00th=[ 186], 40.00th=[ 273], 50.00th=[ 338], 60.00th=[ 429], | 70.00th=[ 750], 80.00th=[10421], 90.00th=[17171], 95.00th=[21890], | 99.00th=[31065], 99.50th=[33817], 99.90th=[41681], 99.95th=[43254], | 99.99th=[47973] bw ( MiB/s): min= 650, max= 1574, per=100.00%, avg=1124.07, stdev=219.53, samples=60 iops : min= 650, max= 1574, avg=1124.03, stdev=219.50, samples=60 write: IOPS=283, BW=283MiB/s (297MB/s)(8497MiB/30012msec) slat (usec): min=61, max=578, avg=185.54, stdev=85.99 clat (usec): min=2486, max=52418, avg=17561.34, stdev=7021.66 lat (usec): min=2605, max=52736, avg=17747.09, stdev=7028.49 clat percentiles (usec): | 1.00th=[ 5538], 5.00th=[ 8094], 10.00th=[ 9634], 20.00th=[11731], | 30.00th=[13435], 40.00th=[14877], 50.00th=[16450], 60.00th=[18220], | 70.00th=[20055], 80.00th=[22676], 90.00th=[26870], 95.00th=[31065], | 99.00th=[38011], 99.50th=[42730], 99.90th=[45876], 99.95th=[47449], | 99.99th=[52167] bw ( KiB/s): min=155648, max=362496, per=100.00%, avg=289979.45, stdev=46292.25, samples=60 iops : min= 152, max= 354, avg=283.13, stdev=45.19, samples=60 lat (usec) : 100=0.02%, 250=30.11%, 500=20.30%, 750=5.51%, 1000=1.74% lat (msec) : 2=1.05%, 4=0.53%, 10=6.53%, 20=22.81%, 50=11.39% lat (msec) : 100=0.01% cpu : usr=5.41%, sys=3.70%, ctx=29773, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=33723,8497,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1124MiB/s (1178MB/s), 1124MiB/s-1124MiB/s (1178MB/s-1178MB/s), io=32.9GiB (35.4GB), run=30012-30012msec WRITE: bw=283MiB/s (297MB/s), 283MiB/s-283MiB/s (297MB/s-297MB/s), io=8497MiB (8910MB), run=30012-30012msec Disk stats (read/write): vdc: ios=33580/8467, merge=0/0, ticks=134738/144278, in_queue=294340, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961581: Fri Jan 7 14:25:09 2022 read: IOPS=1122, BW=1123MiB/s (1177MB/s)(32.9GiB/30002msec) clat (usec): min=125, max=3084, avg=258.60, stdev=234.83 lat (usec): min=125, max=3084, avg=258.75, stdev=234.84 clat percentiles (usec): | 1.00th=[ 135], 5.00th=[ 139], 10.00th=[ 141], 20.00th=[ 145], | 30.00th=[ 149], 40.00th=[ 157], 50.00th=[ 165], 60.00th=[ 176], | 70.00th=[ 196], 80.00th=[ 338], 90.00th=[ 506], 95.00th=[ 725], | 99.00th=[ 1319], 99.50th=[ 1565], 99.90th=[ 2073], 99.95th=[ 2147], | 99.99th=[ 2474] bw ( MiB/s): min= 944, max= 1320, per=100.00%, avg=1122.83, stdev=81.18, samples=60 iops : min= 944, max= 1320, avg=1122.83, stdev=81.18, samples=60 write: IOPS=283, BW=283MiB/s (297MB/s)(8492MiB/30002msec) clat (usec): min=2364, max=4411, avg=2399.62, stdev=48.62 lat (usec): min=2415, max=4512, avg=2498.70, stdev=54.46 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2376], 40.00th=[ 2376], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2409], 95.00th=[ 2442], | 99.00th=[ 2474], 99.50th=[ 2540], 99.90th=[ 3097], 99.95th=[ 3392], | 99.99th=[ 4424] bw ( KiB/s): min=270336, max=313344, per=100.00%, avg=289826.13, stdev=8974.61, samples=60 iops : min= 264, max= 306, avg=283.03, stdev= 8.76, samples=60 lat (usec) : 250=60.47%, 500=11.26%, 750=4.46%, 1000=1.88% lat (msec) : 2=1.69%, 4=20.23%, 10=0.01% cpu : usr=3.56%, sys=3.51%, ctx=42195, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=33685,8492,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1123MiB/s (1177MB/s), 1123MiB/s-1123MiB/s (1177MB/s-1177MB/s), io=32.9GiB (35.3GB), run=30002-30002msec WRITE: bw=283MiB/s (297MB/s), 283MiB/s-283MiB/s (297MB/s-297MB/s), io=8492MiB (8905MB), run=30002-30002msec Disk stats (read/write): vdc: ios=33648/8484, merge=0/0, ticks=737/16980, in_queue=27751, util=92.23% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962985: Fri Jan 7 14:25:46 2022 read: IOPS=1013, BW=1013MiB/s (1062MB/s)(29.8GiB/30100msec) slat (usec): min=9, max=406, avg=18.83, stdev=10.03 clat (usec): min=59, max=268905, avg=22885.39, stdev=43856.31 lat (usec): min=114, max=268918, avg=22904.37, stdev=43856.24 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 113], 10.00th=[ 117], 20.00th=[ 137], | 30.00th=[ 314], 40.00th=[ 351], 50.00th=[ 529], 60.00th=[ 963], | 70.00th=[ 1795], 80.00th=[ 57934], 90.00th=[ 99091], 95.00th=[122160], | 99.00th=[166724], 99.50th=[179307], 99.90th=[217056], 99.95th=[233833], | 99.99th=[256902] bw ( KiB/s): min=608256, max=1398784, per=100.00%, avg=1040283.98, stdev=132032.99, samples=60 iops : min= 594, max= 1366, avg=1015.90, stdev=128.94, samples=60 write: IOPS=255, BW=256MiB/s (268MB/s)(7704MiB/30100msec) slat (usec): min=58, max=739, avg=202.41, stdev=99.76 clat (msec): min=14, max=306, avg=104.44, stdev=30.58 lat (msec): min=14, max=306, avg=104.65, stdev=30.58 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 63], 10.00th=[ 71], 20.00th=[ 82], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 101], 60.00th=[ 107], | 70.00th=[ 114], 80.00th=[ 125], 90.00th=[ 144], 95.00th=[ 163], | 99.00th=[ 203], 99.50th=[ 218], 99.90th=[ 264], 99.95th=[ 284], | 99.99th=[ 309] bw ( KiB/s): min=212992, max=294912, per=99.96%, avg=261982.15, stdev=19313.82, samples=60 iops : min= 208, max= 288, avg=255.83, stdev=18.86, samples=60 lat (usec) : 100=0.02%, 250=21.92%, 500=17.38%, 750=5.33%, 1000=3.82% lat (msec) : 2=8.47%, 4=2.82%, 10=0.80%, 20=0.85%, 50=1.60% lat (msec) : 100=18.75%, 250=18.17%, 500=0.06% cpu : usr=5.51%, sys=2.66%, ctx=22517, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=30497,7704,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1013MiB/s (1062MB/s), 1013MiB/s-1013MiB/s (1062MB/s-1062MB/s), io=29.8GiB (31.0GB), run=30100-30100msec WRITE: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=7704MiB (8078MB), run=30100-30100msec Disk stats (read/write): vdc: ios=30445/7667, merge=0/0, ticks=679093/795002, in_queue=1492860, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963562: Fri Jan 7 14:26:23 2022 read: IOPS=1113, BW=1113MiB/s (1167MB/s)(32.6GiB/30002msec) clat (usec): min=125, max=3179, avg=263.60, stdev=245.17 lat (usec): min=125, max=3179, avg=263.76, stdev=245.18 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 141], 10.00th=[ 143], 20.00th=[ 149], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 169], 60.00th=[ 180], | 70.00th=[ 202], 80.00th=[ 343], 90.00th=[ 502], 95.00th=[ 734], | 99.00th=[ 1336], 99.50th=[ 1663], 99.90th=[ 2147], 99.95th=[ 2311], | 99.99th=[ 2999] bw ( MiB/s): min= 918, max= 1274, per=100.00%, avg=1113.37, stdev=79.13, samples=60 iops : min= 918, max= 1274, avg=1113.37, stdev=79.13, samples=60 write: IOPS=280, BW=281MiB/s (295MB/s)(8430MiB/30002msec) clat (usec): min=2370, max=10756, avg=2407.79, stdev=104.81 lat (usec): min=2418, max=10853, avg=2505.99, stdev=107.71 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2442], 95.00th=[ 2442], | 99.00th=[ 2507], 99.50th=[ 2704], 99.90th=[ 3359], 99.95th=[ 3458], | 99.99th=[10814] bw ( KiB/s): min=270336, max=313344, per=100.00%, avg=287709.87, stdev=9278.40, samples=60 iops : min= 264, max= 306, avg=280.97, stdev= 9.06, samples=60 lat (usec) : 250=60.16%, 500=11.58%, 750=4.33%, 1000=1.93% lat (msec) : 2=1.61%, 4=20.38%, 10=0.01%, 20=0.01% cpu : usr=3.57%, sys=3.66%, ctx=41849, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=33401,8430,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1113MiB/s (1167MB/s), 1113MiB/s-1113MiB/s (1167MB/s-1167MB/s), io=32.6GiB (35.0GB), run=30002-30002msec WRITE: bw=281MiB/s (295MB/s), 281MiB/s-281MiB/s (295MB/s-295MB/s), io=8430MiB (8839MB), run=30002-30002msec Disk stats (read/write): vdc: ios=33376/8426, merge=0/0, ticks=827/16875, in_queue=27648, util=91.73% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964383: Fri Jan 7 14:27:00 2022 read: IOPS=1013, BW=1014MiB/s (1063MB/s)(29.0GiB/30297msec) slat (usec): min=9, max=1003, avg=18.23, stdev=13.31 clat (usec): min=36, max=504576, avg=25202.99, stdev=50296.65 lat (usec): min=115, max=504591, avg=25221.38, stdev=50296.62 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 112], 10.00th=[ 116], 20.00th=[ 149], | 30.00th=[ 330], 40.00th=[ 379], 50.00th=[ 676], 60.00th=[ 1237], | 70.00th=[ 2474], 80.00th=[ 59507], 90.00th=[103285], 95.00th=[127402], | 99.00th=[189793], 99.50th=[248513], 99.90th=[404751], 99.95th=[429917], | 99.99th=[476054] bw ( KiB/s): min=278528, max=1427456, per=100.00%, avg=1047762.13, stdev=187889.23, samples=60 iops : min= 272, max= 1394, avg=1023.15, stdev=183.51, samples=60 write: IOPS=256, BW=256MiB/s (269MB/s)(7770MiB/30297msec) slat (usec): min=62, max=1146, avg=220.29, stdev=112.18 clat (msec): min=26, max=695, avg=289.86, stdev=71.69 lat (msec): min=26, max=695, avg=290.08, stdev=71.68 clat percentiles (msec): | 1.00th=[ 89], 5.00th=[ 176], 10.00th=[ 226], 20.00th=[ 251], | 30.00th=[ 266], 40.00th=[ 275], 50.00th=[ 288], 60.00th=[ 300], | 70.00th=[ 313], 80.00th=[ 330], 90.00th=[ 355], 95.00th=[ 384], | 99.00th=[ 567], 99.50th=[ 592], 99.90th=[ 684], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=192512, max=323584, per=99.97%, avg=262547.20, stdev=25734.81, samples=60 iops : min= 188, max= 316, avg=256.35, stdev=25.15, samples=60 lat (usec) : 50=0.01%, 100=0.06%, 250=19.85%, 500=16.34%, 750=5.19% lat (usec) : 1000=3.52% lat (msec) : 2=9.11%, 4=4.02%, 10=1.44%, 20=1.10%, 50=1.85% lat (msec) : 100=9.14%, 250=11.77%, 500=16.18%, 750=0.42% cpu : usr=5.76%, sys=2.79%, ctx=21210, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30719,7770,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1014MiB/s (1063MB/s), 1014MiB/s-1014MiB/s (1063MB/s-1063MB/s), io=29.0GiB (32.2GB), run=30297-30297msec WRITE: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=7770MiB (8147MB), run=30297-30297msec Disk stats (read/write): vdc: ios=30719/7761, merge=0/0, ticks=751659/2231353, in_queue=3001084, util=99.55% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965447: Fri Jan 7 14:27:37 2022 read: IOPS=1111, BW=1111MiB/s (1165MB/s)(32.6GiB/30001msec) clat (usec): min=130, max=3075, avg=265.81, stdev=241.34 lat (usec): min=130, max=3075, avg=265.96, stdev=241.35 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 151], | 30.00th=[ 155], 40.00th=[ 161], 50.00th=[ 172], 60.00th=[ 182], | 70.00th=[ 202], 80.00th=[ 343], 90.00th=[ 510], 95.00th=[ 750], | 99.00th=[ 1352], 99.50th=[ 1598], 99.90th=[ 2114], 99.95th=[ 2376], | 99.99th=[ 2835] bw ( MiB/s): min= 912, max= 1262, per=100.00%, avg=1111.53, stdev=77.31, samples=60 iops : min= 912, max= 1262, avg=1111.53, stdev=77.31, samples=60 write: IOPS=280, BW=281MiB/s (294MB/s)(8417MiB/30001msec) clat (usec): min=2372, max=4944, avg=2405.04, stdev=50.82 lat (usec): min=2421, max=5028, avg=2502.93, stdev=54.87 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2376], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2409], 95.00th=[ 2442], | 99.00th=[ 2474], 99.50th=[ 2606], 99.90th=[ 3097], 99.95th=[ 3261], | 99.99th=[ 4948] bw ( KiB/s): min=262144, max=313344, per=99.99%, avg=287266.13, stdev=9134.58, samples=60 iops : min= 256, max= 306, avg=280.53, stdev= 8.92, samples=60 lat (usec) : 250=60.17%, 500=11.33%, 750=4.32%, 1000=2.20% lat (msec) : 2=1.68%, 4=20.30%, 10=0.01% cpu : usr=3.52%, sys=3.97%, ctx=41772, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=33346,8417,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1111MiB/s (1165MB/s), 1111MiB/s-1111MiB/s (1165MB/s-1165MB/s), io=32.6GiB (34.0GB), run=30001-30001msec WRITE: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=8417MiB (8826MB), run=30001-30001msec Disk stats (read/write): vdc: ios=33249/8396, merge=0/0, ticks=774/16804, in_queue=27668, util=91.82% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966066: Fri Jan 7 14:28:14 2022 read: IOPS=1026, BW=1027MiB/s (1077MB/s)(30.5GiB/30401msec) slat (usec): min=10, max=422997, avg=701.64, stdev=4431.73 clat (msec): min=14, max=601, avg=86.29, stdev=62.45 lat (msec): min=14, max=606, avg=86.99, stdev=62.61 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 29], 10.00th=[ 37], 20.00th=[ 47], | 30.00th=[ 54], 40.00th=[ 61], 50.00th=[ 67], 60.00th=[ 75], | 70.00th=[ 88], 80.00th=[ 122], 90.00th=[ 165], 95.00th=[ 194], | 99.00th=[ 388], 99.50th=[ 435], 99.90th=[ 550], 99.95th=[ 584], | 99.99th=[ 600] bw ( MiB/s): min= 142, max= 1694, per=100.00%, avg=1037.62, stdev=231.93, samples=60 iops : min= 142, max= 1694, avg=1037.60, stdev=231.94, samples=60 write: IOPS=259, BW=260MiB/s (272MB/s)(7895MiB/30401msec) slat (usec): min=133, max=375519, avg=1014.78, stdev=5085.63 clat (msec): min=96, max=859, avg=424.59, stdev=92.94 lat (msec): min=97, max=860, avg=425.60, stdev=92.92 clat percentiles (msec): | 1.00th=[ 144], 5.00th=[ 266], 10.00th=[ 334], 20.00th=[ 372], | 30.00th=[ 393], 40.00th=[ 409], 50.00th=[ 426], 60.00th=[ 443], | 70.00th=[ 456], 80.00th=[ 481], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 751], 99.50th=[ 793], 99.90th=[ 835], 99.95th=[ 844], | 99.99th=[ 860] bw ( KiB/s): min=165888, max=346112, per=99.94%, avg=265777.58, stdev=34061.29, samples=60 iops : min= 162, max= 338, avg=259.53, stdev=33.28, samples=60 lat (msec) : 20=0.27%, 50=19.65%, 100=39.75%, 250=19.67%, 500=17.92% lat (msec) : 750=2.54%, 1000=0.20% cpu : usr=8.92%, sys=2.35%, ctx=3585, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=31218,7895,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1027MiB/s (1077MB/s), 1027MiB/s-1027MiB/s (1077MB/s-1077MB/s), io=30.5GiB (32.7GB), run=30401-30401msec WRITE: bw=260MiB/s (272MB/s), 260MiB/s-260MiB/s (272MB/s-272MB/s), io=7895MiB (8279MB), run=30401-30401msec Disk stats (read/write): vdc: ios=31218/7840, merge=0/0, ticks=847941/2847755, in_queue=3729265, util=99.38% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966916: Fri Jan 7 14:28:51 2022 read: IOPS=201, BW=404MiB/s (424MB/s)(11.8GiB/30005msec) clat (usec): min=281, max=3802, avg=584.94, stdev=363.37 lat (usec): min=281, max=3802, avg=585.13, stdev=363.39 clat percentiles (usec): | 1.00th=[ 293], 5.00th=[ 302], 10.00th=[ 310], 20.00th=[ 322], | 30.00th=[ 338], 40.00th=[ 379], 50.00th=[ 453], 60.00th=[ 570], | 70.00th=[ 709], 80.00th=[ 783], 90.00th=[ 947], 95.00th=[ 1237], | 99.00th=[ 2278], 99.50th=[ 2409], 99.90th=[ 3064], 99.95th=[ 3195], | 99.99th=[ 3818] bw ( KiB/s): min=212992, max=630784, per=100.00%, avg=413834.82, stdev=88370.71, samples=60 iops : min= 104, max= 308, avg=202.03, stdev=43.17, samples=60 write: IOPS=52, BW=106MiB/s (111MB/s)(3170MiB/30005msec) clat (usec): min=4722, max=33747, avg=16397.77, stdev=4309.87 lat (usec): min=4907, max=33953, avg=16681.99, stdev=4310.69 clat percentiles (usec): | 1.00th=[ 4817], 5.00th=[11338], 10.00th=[12780], 20.00th=[13304], | 30.00th=[13435], 40.00th=[13698], 50.00th=[16188], 60.00th=[17957], | 70.00th=[18744], 80.00th=[19268], 90.00th=[20841], 95.00th=[24249], | 99.00th=[28705], 99.50th=[30802], 99.90th=[32637], 99.95th=[33817], | 99.99th=[33817] bw ( KiB/s): min=94208, max=147456, per=99.97%, avg=108155.98, stdev=8898.44, samples=60 iops : min= 46, max= 72, avg=52.80, stdev= 4.34, samples=60 lat (usec) : 500=43.43%, 750=16.43%, 1000=12.71% lat (msec) : 2=5.62%, 4=1.07%, 10=0.84%, 20=17.13%, 50=2.76% cpu : usr=1.58%, sys=1.61%, ctx=7653, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=6061,1585,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=404MiB/s (424MB/s), 404MiB/s-404MiB/s (424MB/s-424MB/s), io=11.8GiB (12.7GB), run=30005-30005msec WRITE: bw=106MiB/s (111MB/s), 106MiB/s-106MiB/s (111MB/s-111MB/s), io=3170MiB (3324MB), run=30005-30005msec Disk stats (read/write): vdc: ios=11964/3142, merge=0/0, ticks=589/49837, in_queue=56628, util=96.42% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967495: Fri Jan 7 14:29:28 2022 read: IOPS=364, BW=729MiB/s (764MB/s)(21.3GiB/30001msec) slat (usec): min=53, max=179, avg=74.10, stdev=12.71 clat (usec): min=166, max=4027, avg=444.07, stdev=363.09 lat (usec): min=251, max=4082, avg=518.43, stdev=363.84 clat percentiles (usec): | 1.00th=[ 200], 5.00th=[ 206], 10.00th=[ 210], 20.00th=[ 219], | 30.00th=[ 231], 40.00th=[ 258], 50.00th=[ 297], 60.00th=[ 383], | 70.00th=[ 482], 80.00th=[ 594], 90.00th=[ 816], 95.00th=[ 1139], | 99.00th=[ 2180], 99.50th=[ 2376], 99.90th=[ 3064], 99.95th=[ 3130], | 99.99th=[ 4015] bw ( KiB/s): min=360448, max=1380352, per=100.00%, avg=748688.00, stdev=318228.16, samples=59 iops : min= 176, max= 674, avg=365.54, stdev=155.40, samples=59 write: IOPS=92, BW=185MiB/s (194MB/s)(5548MiB/30001msec) slat (usec): min=137, max=952, avg=279.62, stdev=101.72 clat (usec): min=4624, max=31165, avg=8482.19, stdev=5071.18 lat (usec): min=4780, max=31402, avg=8762.13, stdev=5106.93 clat percentiles (usec): | 1.00th=[ 4621], 5.00th=[ 4621], 10.00th=[ 4621], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 7963], | 70.00th=[11600], 80.00th=[13042], 90.00th=[15926], 95.00th=[18482], | 99.00th=[22938], 99.50th=[25822], 99.90th=[28705], 99.95th=[29754], | 99.99th=[31065] bw ( KiB/s): min=106709, max=323584, per=100.00%, avg=190661.05, stdev=83316.40, samples=59 iops : min= 52, max= 158, avg=93.08, stdev=40.68, samples=59 lat (usec) : 250=30.17%, 500=27.31%, 750=12.68%, 1000=4.60% lat (msec) : 2=3.90%, 4=1.08%, 10=12.98%, 20=6.74%, 50=0.54% cpu : usr=2.30%, sys=3.64%, ctx=13711, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10930,2774,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=729MiB/s (764MB/s), 729MiB/s-729MiB/s (764MB/s-764MB/s), io=21.3GiB (22.9GB), run=30001-30001msec WRITE: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=5548MiB (5817MB), run=30001-30001msec Disk stats (read/write): vdc: ios=21766/5526, merge=0/0, ticks=1005/43452, in_queue=54966, util=94.93% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968313: Fri Jan 7 14:30:05 2022 read: IOPS=471, BW=944MiB/s (989MB/s)(27.7GiB/30009msec) clat (usec): min=231, max=4720, avg=481.54, stdev=371.27 lat (usec): min=231, max=4720, avg=481.71, stdev=371.27 clat percentiles (usec): | 1.00th=[ 241], 5.00th=[ 249], 10.00th=[ 253], 20.00th=[ 265], | 30.00th=[ 277], 40.00th=[ 297], 50.00th=[ 334], 60.00th=[ 408], | 70.00th=[ 498], 80.00th=[ 619], 90.00th=[ 848], 95.00th=[ 1188], | 99.00th=[ 2212], 99.50th=[ 2442], 99.90th=[ 3294], 99.95th=[ 3654], | 99.99th=[ 4555] bw ( KiB/s): min=417792, max=1441792, per=100.00%, avg=966642.17, stdev=300464.39, samples=60 iops : min= 204, max= 704, avg=471.98, stdev=146.72, samples=60 write: IOPS=118, BW=236MiB/s (248MB/s)(7094MiB/30009msec) clat (usec): min=4647, max=29525, avg=6309.90, stdev=3453.61 lat (usec): min=4740, max=29736, avg=6528.14, stdev=3476.18 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4752], 80.00th=[ 7570], 90.00th=[11863], 95.00th=[13829], | 99.00th=[18744], 99.50th=[21890], 99.90th=[26346], 99.95th=[28443], | 99.99th=[29492] bw ( KiB/s): min=102400, max=319488, per=100.00%, avg=242088.37, stdev=77393.15, samples=60 iops : min= 50, max= 156, avg=118.20, stdev=37.79, samples=60 lat (usec) : 250=5.35%, 500=50.93%, 750=13.28%, 1000=5.13% lat (msec) : 2=4.12%, 4=1.12%, 10=17.04%, 20=2.86%, 50=0.16% cpu : usr=2.81%, sys=2.63%, ctx=17723, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=14159,3547,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=944MiB/s (989MB/s), 944MiB/s-944MiB/s (989MB/s-989MB/s), io=27.7GiB (29.7GB), run=30009-30009msec WRITE: bw=236MiB/s (248MB/s), 236MiB/s-236MiB/s (248MB/s-248MB/s), io=7094MiB (7439MB), run=30009-30009msec Disk stats (read/write): vdc: ios=28226/7066, merge=0/0, ticks=1241/33958, in_queue=48884, util=94.08% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969572: Fri Jan 7 14:30:42 2022 read: IOPS=604, BW=1210MiB/s (1269MB/s)(35.5GiB/30027msec) slat (usec): min=19, max=322, avg=35.72, stdev=16.46 clat (usec): min=72, max=73074, avg=10179.29, stdev=11489.25 lat (usec): min=209, max=73102, avg=10215.21, stdev=11489.46 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 206], 10.00th=[ 233], 20.00th=[ 635], | 30.00th=[ 734], 40.00th=[ 1336], 50.00th=[ 5145], 60.00th=[11863], | 70.00th=[15795], 80.00th=[20055], 90.00th=[25560], 95.00th=[31065], | 99.00th=[47449], 99.50th=[53740], 99.90th=[66323], 99.95th=[69731], | 99.99th=[72877] bw ( MiB/s): min= 372, max= 1936, per=100.00%, avg=1213.28, stdev=468.69, samples=60 iops : min= 186, max= 968, avg=606.57, stdev=234.30, samples=60 write: IOPS=152, BW=306MiB/s (321MB/s)(9180MiB/30027msec) slat (usec): min=120, max=2521, avg=480.30, stdev=197.26 clat (msec): min=4, max=115, avg=24.50, stdev=14.33 lat (msec): min=5, max=116, avg=24.98, stdev=14.36 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 29], 90.00th=[ 40], 95.00th=[ 57], | 99.00th=[ 82], 99.50th=[ 91], 99.90th=[ 115], 99.95th=[ 115], | 99.99th=[ 116] bw ( KiB/s): min=118784, max=417792, per=100.00%, avg=313782.43, stdev=114897.24, samples=60 iops : min= 58, max= 204, avg=153.15, stdev=56.05, samples=60 lat (usec) : 100=0.01%, 250=8.75%, 500=3.69%, 750=12.09%, 1000=3.94% lat (msec) : 2=7.04%, 4=2.98%, 10=7.66%, 20=26.78%, 50=25.09% lat (msec) : 100=1.93%, 250=0.04% cpu : usr=7.26%, sys=2.88%, ctx=13228, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=18165,4590,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1210MiB/s (1269MB/s), 1210MiB/s-1210MiB/s (1269MB/s-1269MB/s), io=35.5GiB (38.1GB), run=30027-30027msec WRITE: bw=306MiB/s (321MB/s), 306MiB/s-306MiB/s (321MB/s-321MB/s), io=9180MiB (9626MB), run=30027-30027msec Disk stats (read/write): vdc: ios=36302/9168, merge=0/0, ticks=257880/218292, in_queue=498273, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970159: Fri Jan 7 14:31:19 2022 read: IOPS=577, BW=1155MiB/s (1211MB/s)(33.8GiB/30001msec) clat (usec): min=237, max=4675, avg=476.50, stdev=363.51 lat (usec): min=237, max=4676, avg=476.67, stdev=363.52 clat percentiles (usec): | 1.00th=[ 249], 5.00th=[ 255], 10.00th=[ 260], 20.00th=[ 269], | 30.00th=[ 281], 40.00th=[ 306], 50.00th=[ 334], 60.00th=[ 392], | 70.00th=[ 474], 80.00th=[ 594], 90.00th=[ 840], 95.00th=[ 1139], | 99.00th=[ 2212], 99.50th=[ 2409], 99.90th=[ 3163], 99.95th=[ 3589], | 99.99th=[ 4293] bw ( MiB/s): min= 888, max= 1396, per=100.00%, avg=1156.54, stdev=118.92, samples=59 iops : min= 444, max= 698, avg=578.27, stdev=59.46, samples=59 write: IOPS=146, BW=293MiB/s (308MB/s)(8804MiB/30001msec) clat (usec): min=4656, max=7068, avg=4728.42, stdev=173.06 lat (usec): min=4751, max=7510, avg=4929.61, stdev=179.61 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4686], 80.00th=[ 4686], 90.00th=[ 4752], 95.00th=[ 5014], | 99.00th=[ 5604], 99.50th=[ 5866], 99.90th=[ 6521], 99.95th=[ 6587], | 99.99th=[ 7046] bw ( KiB/s): min=274432, max=323584, per=99.99%, avg=300465.90, stdev=11428.44, samples=59 iops : min= 134, max= 158, avg=146.71, stdev= 5.58, samples=59 lat (usec) : 250=1.24%, 500=56.55%, 750=11.79%, 1000=5.03% lat (msec) : 2=4.04%, 4=1.06%, 10=20.29% cpu : usr=3.32%, sys=3.48%, ctx=21743, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17330,4402,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1155MiB/s (1211MB/s), 1155MiB/s-1155MiB/s (1211MB/s-1211MB/s), io=33.8GiB (36.3GB), run=30001-30001msec WRITE: bw=293MiB/s (308MB/s), 293MiB/s-293MiB/s (308MB/s-308MB/s), io=8804MiB (9232MB), run=30001-30001msec Disk stats (read/write): vdc: ios=34448/8732, merge=0/0, ticks=1444/29034, in_queue=47322, util=92.53% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971114: Fri Jan 7 14:31:56 2022 read: IOPS=724, BW=1448MiB/s (1518MB/s)(42.5GiB/30048msec) slat (usec): min=19, max=797, avg=34.04, stdev=37.27 clat (usec): min=145, max=239831, avg=35140.26, stdev=36398.13 lat (usec): min=212, max=239853, avg=35174.46, stdev=36399.13 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 212], 10.00th=[ 570], 20.00th=[ 881], | 30.00th=[ 1991], 40.00th=[ 4015], 50.00th=[ 32375], 60.00th=[ 47973], | 70.00th=[ 57410], 80.00th=[ 70779], 90.00th=[ 80217], 95.00th=[ 91751], | 99.00th=[135267], 99.50th=[152044], 99.90th=[198181], 99.95th=[214959], | 99.99th=[231736] bw ( MiB/s): min= 720, max= 2000, per=100.00%, avg=1448.04, stdev=256.66, samples=60 iops : min= 360, max= 1000, avg=724.02, stdev=128.34, samples=60 write: IOPS=182, BW=364MiB/s (382MB/s)(10.7GiB/30048msec) slat (usec): min=117, max=2280, avg=514.54, stdev=214.01 clat (msec): min=22, max=291, avg=134.12, stdev=28.43 lat (msec): min=23, max=291, avg=134.63, stdev=28.41 clat percentiles (msec): | 1.00th=[ 65], 5.00th=[ 92], 10.00th=[ 102], 20.00th=[ 113], | 30.00th=[ 122], 40.00th=[ 129], 50.00th=[ 136], 60.00th=[ 140], | 70.00th=[ 146], 80.00th=[ 153], 90.00th=[ 165], 95.00th=[ 178], | 99.00th=[ 224], 99.50th=[ 247], 99.90th=[ 284], 99.95th=[ 288], | 99.99th=[ 292] bw ( KiB/s): min=151552, max=417792, per=99.91%, avg=372677.58, stdev=57807.08, samples=60 iops : min= 74, max= 204, avg=181.97, stdev=28.23, samples=60 lat (usec) : 250=5.53%, 500=2.09%, 750=6.90%, 1000=2.40% lat (msec) : 2=7.09%, 4=7.90%, 10=6.27%, 20=0.67%, 50=11.27% lat (msec) : 100=28.44%, 250=21.35%, 500=0.10% cpu : usr=9.17%, sys=3.24%, ctx=10303, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=21756,5473,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1448MiB/s (1518MB/s), 1448MiB/s-1448MiB/s (1518MB/s-1518MB/s), io=42.5GiB (45.6GB), run=30048-30048msec WRITE: bw=364MiB/s (382MB/s), 364MiB/s-364MiB/s (382MB/s-382MB/s), io=10.7GiB (11.5GB), run=30048-30048msec Disk stats (read/write): vdc: ios=43440/10918, merge=0/0, ticks=1061076/1448995, in_queue=2537309, util=99.54% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972162: Fri Jan 7 14:32:33 2022 read: IOPS=575, BW=1151MiB/s (1207MB/s)(33.7GiB/30002msec) clat (usec): min=237, max=5010, avg=481.77, stdev=369.75 lat (usec): min=237, max=5010, avg=481.95, stdev=369.75 clat percentiles (usec): | 1.00th=[ 249], 5.00th=[ 258], 10.00th=[ 262], 20.00th=[ 273], | 30.00th=[ 285], 40.00th=[ 306], 50.00th=[ 334], 60.00th=[ 392], | 70.00th=[ 482], 80.00th=[ 603], 90.00th=[ 865], 95.00th=[ 1188], | 99.00th=[ 2212], 99.50th=[ 2507], 99.90th=[ 3195], 99.95th=[ 3621], | 99.99th=[ 4293] bw ( MiB/s): min= 900, max= 1432, per=100.00%, avg=1151.01, stdev=110.65, samples=60 iops : min= 450, max= 716, avg=575.50, stdev=55.32, samples=60 write: IOPS=145, BW=292MiB/s (306MB/s)(8758MiB/30002msec) clat (usec): min=4654, max=7608, avg=4744.01, stdev=209.30 lat (usec): min=4751, max=7790, avg=4942.09, stdev=213.11 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4686], 80.00th=[ 4686], 90.00th=[ 4752], 95.00th=[ 5080], | 99.00th=[ 5800], 99.50th=[ 5997], 99.90th=[ 6587], 99.95th=[ 6783], | 99.99th=[ 7635] bw ( KiB/s): min=266240, max=319488, per=99.99%, avg=298890.88, stdev=10623.16, samples=60 iops : min= 130, max= 156, avg=145.93, stdev= 5.19, samples=60 lat (usec) : 250=0.99%, 500=56.40%, 750=11.81%, 1000=4.87% lat (msec) : 2=4.66%, 4=1.01%, 10=20.27% cpu : usr=3.08%, sys=3.84%, ctx=21661, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17264,4379,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1151MiB/s (1207MB/s), 1151MiB/s-1151MiB/s (1207MB/s-1207MB/s), io=33.7GiB (36.2GB), run=30002-30002msec WRITE: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=8758MiB (9183MB), run=30002-30002msec Disk stats (read/write): vdc: ios=34436/8726, merge=0/0, ticks=1555/28909, in_queue=47186, util=92.35% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972685: Fri Jan 7 14:33:10 2022 read: IOPS=747, BW=1496MiB/s (1568MB/s)(43.9GiB/30064msec) slat (usec): min=20, max=189837, avg=937.06, stdev=3910.08 clat (msec): min=9, max=442, avg=63.80, stdev=47.99 lat (msec): min=9, max=464, avg=64.74, stdev=48.26 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 19], 10.00th=[ 22], 20.00th=[ 28], | 30.00th=[ 33], 40.00th=[ 37], 50.00th=[ 43], 60.00th=[ 67], | 70.00th=[ 85], 80.00th=[ 99], 90.00th=[ 114], 95.00th=[ 133], | 99.00th=[ 275], 99.50th=[ 300], 99.90th=[ 359], 99.95th=[ 359], | 99.99th=[ 439] bw ( MiB/s): min= 580, max= 1956, per=99.92%, avg=1494.53, stdev=281.43, samples=60 iops : min= 290, max= 978, avg=747.27, stdev=140.71, samples=60 write: IOPS=188, BW=377MiB/s (395MB/s)(11.1GiB/30064msec) slat (usec): min=278, max=229536, avg=1567.84, stdev=6252.22 clat (msec): min=56, max=599, avg=271.59, stdev=52.14 lat (msec): min=57, max=600, avg=273.16, stdev=52.06 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 194], 10.00th=[ 232], 20.00th=[ 251], | 30.00th=[ 259], 40.00th=[ 266], 50.00th=[ 271], 60.00th=[ 279], | 70.00th=[ 284], 80.00th=[ 292], 90.00th=[ 309], 95.00th=[ 347], | 99.00th=[ 468], 99.50th=[ 485], 99.90th=[ 535], 99.95th=[ 575], | 99.99th=[ 600] bw ( KiB/s): min=135168, max=438272, per=99.61%, avg=384614.40, stdev=58125.64, samples=60 iops : min= 66, max= 214, avg=187.80, stdev=28.38, samples=60 lat (msec) : 10=0.01%, 20=5.43%, 50=37.82%, 100=22.19%, 250=16.93% lat (msec) : 500=17.56%, 750=0.06% cpu : usr=12.59%, sys=3.51%, ctx=4722, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=22484,5668,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1496MiB/s (1568MB/s), 1496MiB/s-1496MiB/s (1568MB/s-1568MB/s), io=43.9GiB (47.2GB), run=30064-30064msec WRITE: bw=377MiB/s (395MB/s), 377MiB/s-377MiB/s (395MB/s-395MB/s), io=11.1GiB (11.9GB), run=30064-30064msec Disk stats (read/write): vdc: ios=44800/11268, merge=0/0, ticks=999967/2690114, in_queue=3723392, util=99.36% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973646: Fri Jan 7 14:33:47 2022 read: IOPS=575, BW=1152MiB/s (1208MB/s)(33.7GiB/30001msec) clat (usec): min=237, max=4745, avg=478.89, stdev=366.92 lat (usec): min=237, max=4745, avg=479.08, stdev=366.93 clat percentiles (usec): | 1.00th=[ 249], 5.00th=[ 255], 10.00th=[ 260], 20.00th=[ 269], | 30.00th=[ 281], 40.00th=[ 302], 50.00th=[ 334], 60.00th=[ 392], | 70.00th=[ 478], 80.00th=[ 603], 90.00th=[ 865], 95.00th=[ 1188], | 99.00th=[ 2245], 99.50th=[ 2474], 99.90th=[ 3130], 99.95th=[ 3556], | 99.99th=[ 4080] bw ( MiB/s): min= 877, max= 1384, per=100.00%, avg=1151.73, stdev=119.89, samples=60 iops : min= 438, max= 692, avg=575.85, stdev=59.98, samples=60 write: IOPS=146, BW=292MiB/s (307MB/s)(8770MiB/30001msec) clat (usec): min=4658, max=6944, avg=4752.29, stdev=217.68 lat (usec): min=4750, max=7156, avg=4945.12, stdev=220.86 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4686], 80.00th=[ 4752], 90.00th=[ 4752], 95.00th=[ 5145], | 99.00th=[ 5800], 99.50th=[ 6128], 99.90th=[ 6783], 99.95th=[ 6849], | 99.99th=[ 6915] bw ( KiB/s): min=270336, max=327680, per=99.99%, avg=299302.12, stdev=12394.42, samples=60 iops : min= 132, max= 160, avg=146.13, stdev= 6.04, samples=60 lat (usec) : 250=1.14%, 500=56.49%, 750=11.61%, 1000=4.91% lat (msec) : 2=4.57%, 4=1.00%, 10=20.27% cpu : usr=3.17%, sys=3.53%, ctx=21675, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17275,4385,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1152MiB/s (1208MB/s), 1152MiB/s-1152MiB/s (1208MB/s-1208MB/s), io=33.7GiB (36.2GB), run=30001-30001msec WRITE: bw=292MiB/s (307MB/s), 292MiB/s-292MiB/s (307MB/s-307MB/s), io=8770MiB (9196MB), run=30001-30001msec Disk stats (read/write): vdc: ios=34354/8710, merge=0/0, ticks=1514/29555, in_queue=47850, util=92.43% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974255: Fri Jan 7 14:34:24 2022 read: IOPS=761, BW=1522MiB/s (1596MB/s)(44.7GiB/30092msec) slat (usec): min=22, max=232298, avg=945.62, stdev=4086.13 clat (msec): min=79, max=659, avg=166.27, stdev=58.60 lat (msec): min=79, max=710, avg=167.21, stdev=58.83 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 106], 10.00th=[ 114], 20.00th=[ 124], | 30.00th=[ 132], 40.00th=[ 142], 50.00th=[ 155], 60.00th=[ 167], | 70.00th=[ 184], 80.00th=[ 201], 90.00th=[ 222], 95.00th=[ 251], | 99.00th=[ 426], 99.50th=[ 472], 99.90th=[ 592], 99.95th=[ 592], | 99.99th=[ 659] bw ( MiB/s): min= 408, max= 1948, per=99.68%, avg=1517.37, stdev=243.88, samples=60 iops : min= 204, max= 974, avg=758.67, stdev=121.93, samples=60 write: IOPS=191, BW=384MiB/s (402MB/s)(11.3GiB/30092msec) slat (usec): min=482, max=187556, avg=1441.97, stdev=3737.98 clat (msec): min=21, max=782, avg=375.98, stdev=63.80 lat (msec): min=118, max=782, avg=377.42, stdev=63.75 clat percentiles (msec): | 1.00th=[ 197], 5.00th=[ 284], 10.00th=[ 330], 20.00th=[ 347], | 30.00th=[ 359], 40.00th=[ 368], 50.00th=[ 376], 60.00th=[ 384], | 70.00th=[ 393], 80.00th=[ 401], 90.00th=[ 422], 95.00th=[ 464], | 99.00th=[ 609], 99.50th=[ 701], 99.90th=[ 776], 99.95th=[ 785], | 99.99th=[ 785] bw ( KiB/s): min=192512, max=479232, per=99.36%, avg=390238.65, stdev=47351.18, samples=60 iops : min= 94, max= 234, avg=190.53, stdev=23.12, samples=60 lat (msec) : 50=0.01%, 100=2.05%, 250=74.47%, 500=22.37%, 750=1.08% lat (msec) : 1000=0.03% cpu : usr=12.89%, sys=3.63%, ctx=4874, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=22904,5771,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1522MiB/s (1596MB/s), 1522MiB/s-1522MiB/s (1596MB/s-1596MB/s), io=44.7GiB (48.0GB), run=30092-30092msec WRITE: bw=384MiB/s (402MB/s), 384MiB/s-384MiB/s (402MB/s-402MB/s), io=11.3GiB (12.1GB), run=30092-30092msec Disk stats (read/write): vdc: ios=45550/11450, merge=0/0, ticks=939941/2732884, in_queue=3720440, util=98.98% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975113: Fri Jan 7 14:35:01 2022 read: IOPS=139, BW=558MiB/s (586MB/s)(16.4GiB/30005msec) clat (usec): min=408, max=6177, avg=962.59, stdev=681.22 lat (usec): min=408, max=6177, avg=962.80, stdev=681.26 clat percentiles (usec): | 1.00th=[ 433], 5.00th=[ 445], 10.00th=[ 461], 20.00th=[ 486], | 30.00th=[ 529], 40.00th=[ 603], 50.00th=[ 717], 60.00th=[ 865], | 70.00th=[ 1172], 80.00th=[ 1369], 90.00th=[ 1565], 95.00th=[ 1991], | 99.00th=[ 4293], 99.50th=[ 4555], 99.90th=[ 5145], 99.95th=[ 5211], | 99.99th=[ 6194] bw ( KiB/s): min=286720, max=1024000, per=100.00%, avg=572002.13, stdev=158690.08, samples=60 iops : min= 70, max= 250, avg=139.63, stdev=38.74, samples=60 write: IOPS=37, BW=148MiB/s (155MB/s)(4448MiB/30005msec) clat (usec): min=9217, max=65323, avg=22742.91, stdev=6792.61 lat (usec): min=9595, max=66556, avg=23344.67, stdev=6790.95 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[11207], 10.00th=[14484], 20.00th=[18482], | 30.00th=[19268], 40.00th=[20317], 50.00th=[22676], 60.00th=[23987], | 70.00th=[24773], 80.00th=[27919], 90.00th=[31851], 95.00th=[34341], | 99.00th=[41681], 99.50th=[42730], 99.90th=[49021], 99.95th=[65274], | 99.99th=[65274] bw ( KiB/s): min=122880, max=221184, per=99.94%, avg=151703.55, stdev=18378.54, samples=60 iops : min= 30, max= 54, avg=37.03, stdev= 4.49, samples=60 lat (usec) : 500=19.07%, 750=22.34%, 1000=10.24% lat (msec) : 2=23.45%, 4=2.58%, 10=2.11%, 20=7.15%, 50=13.04% lat (msec) : 100=0.02% cpu : usr=2.40%, sys=1.77%, ctx=5308, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4189,1112,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=558MiB/s (586MB/s), 558MiB/s-558MiB/s (586MB/s-586MB/s), io=16.4GiB (17.6GB), run=30005-30005msec WRITE: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=4448MiB (4664MB), run=30005-30005msec Disk stats (read/write): vdc: ios=16655/4416, merge=0/0, ticks=4660/93125, in_queue=106962, util=96.75% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976247: Fri Jan 7 14:35:38 2022 read: IOPS=209, BW=840MiB/s (881MB/s)(24.6GiB/30005msec) slat (usec): min=77, max=338, avg=100.07, stdev=17.52 clat (usec): min=323, max=6994, avg=857.51, stdev=659.84 lat (usec): min=452, max=7089, avg=957.85, stdev=661.78 clat percentiles (usec): | 1.00th=[ 379], 5.00th=[ 396], 10.00th=[ 408], 20.00th=[ 433], | 30.00th=[ 478], 40.00th=[ 553], 50.00th=[ 644], 60.00th=[ 758], | 70.00th=[ 922], 80.00th=[ 1156], 90.00th=[ 1434], 95.00th=[ 1876], | 99.00th=[ 4228], 99.50th=[ 4555], 99.90th=[ 5407], 99.95th=[ 5604], | 99.99th=[ 6980] bw ( KiB/s): min=286720, max=1510348, per=100.00%, avg=860122.50, stdev=302071.66, samples=60 iops : min= 70, max= 368, avg=209.97, stdev=73.72, samples=60 write: IOPS=54, BW=218MiB/s (228MB/s)(6536MiB/30005msec) slat (usec): min=266, max=1760, avg=596.43, stdev=247.75 clat (usec): min=9141, max=49484, avg=14062.53, stdev=7470.75 lat (usec): min=9476, max=50036, avg=14659.32, stdev=7503.65 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9372], 40.00th=[ 9634], 50.00th=[10028], 60.00th=[10683], | 70.00th=[13042], 80.00th=[19268], 90.00th=[27395], 95.00th=[29754], | 99.00th=[37487], 99.50th=[38536], 99.90th=[43779], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min=122880, max=319488, per=99.97%, avg=222982.98, stdev=74532.80, samples=60 iops : min= 30, max= 78, avg=54.43, stdev=18.19, samples=60 lat (usec) : 500=26.37%, 750=20.95%, 1000=11.23% lat (msec) : 2=17.16%, 4=2.48%, 10=11.11%, 20=6.82%, 50=3.88% cpu : usr=2.83%, sys=2.86%, ctx=7942, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=6299,1634,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=840MiB/s (881MB/s), 840MiB/s-840MiB/s (881MB/s-881MB/s), io=24.6GiB (26.4GB), run=30005-30005msec WRITE: bw=218MiB/s (228MB/s), 218MiB/s-218MiB/s (228MB/s-228MB/s), io=6536MiB (6853MB), run=30005-30005msec Disk stats (read/write): vdc: ios=31350/8145, merge=0/0, ticks=5840/104073, in_queue=128838, util=95.77% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977100: Fri Jan 7 14:36:15 2022 read: IOPS=257, BW=1031MiB/s (1081MB/s)(30.2GiB/30003msec) clat (usec): min=377, max=16715, avg=863.58, stdev=685.68 lat (usec): min=377, max=16715, avg=863.78, stdev=685.70 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 445], | 30.00th=[ 490], 40.00th=[ 562], 50.00th=[ 652], 60.00th=[ 758], | 70.00th=[ 906], 80.00th=[ 1123], 90.00th=[ 1467], 95.00th=[ 1909], | 99.00th=[ 4293], 99.50th=[ 4752], 99.90th=[ 5538], 99.95th=[ 6063], | 99.99th=[16712] bw ( MiB/s): min= 392, max= 1497, per=100.00%, avg=1044.30, stdev=258.83, samples=60 iops : min= 98, max= 374, avg=260.65, stdev=64.56, samples=60 write: IOPS=65, BW=262MiB/s (275MB/s)(7872MiB/30003msec) clat (usec): min=9189, max=63027, avg=11351.57, stdev=4468.36 lat (usec): min=9412, max=64342, avg=11841.09, stdev=4506.78 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9634], 60.00th=[10028], | 70.00th=[10421], 80.00th=[11076], 90.00th=[17695], 95.00th=[21627], | 99.00th=[31065], 99.50th=[32375], 99.90th=[39584], 99.95th=[63177], | 99.99th=[63177] bw ( KiB/s): min=139264, max=350370, per=100.00%, avg=272057.37, stdev=61107.38, samples=60 iops : min= 34, max= 85, avg=66.07, stdev=14.78, samples=60 lat (usec) : 500=24.96%, 750=22.44%, 1000=12.46% lat (msec) : 2=16.17%, 4=2.45%, 10=13.36%, 20=6.82%, 50=1.32% lat (msec) : 100=0.01% cpu : usr=3.41%, sys=2.26%, ctx=9713, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7733,1968,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1031MiB/s (1081MB/s), 1031MiB/s-1031MiB/s (1081MB/s-1081MB/s), io=30.2GiB (32.4GB), run=30003-30003msec WRITE: bw=262MiB/s (275MB/s), 262MiB/s-262MiB/s (275MB/s-275MB/s), io=7872MiB (8254MB), run=30003-30003msec Disk stats (read/write): vdc: ios=30932/7868, merge=0/0, ticks=4782/72540, in_queue=95135, util=95.60% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977949: Fri Jan 7 14:36:52 2022 read: IOPS=326, BW=1307MiB/s (1370MB/s)(38.3GiB/30030msec) slat (usec): min=38, max=369, avg=63.12, stdev=28.67 clat (usec): min=303, max=93497, avg=20146.01, stdev=15797.06 lat (usec): min=385, max=93744, avg=20209.34, stdev=15797.40 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 1057], 10.00th=[ 1254], 20.00th=[ 2704], | 30.00th=[ 8094], 40.00th=[14222], 50.00th=[19530], 60.00th=[23462], | 70.00th=[28443], 80.00th=[33424], 90.00th=[41157], 95.00th=[47973], | 99.00th=[62653], 99.50th=[69731], 99.90th=[86508], 99.95th=[90702], | 99.99th=[93848] bw ( MiB/s): min= 528, max= 2048, per=100.00%, avg=1307.37, stdev=317.29, samples=60 iops : min= 132, max= 512, avg=326.80, stdev=79.33, samples=60 write: IOPS=83, BW=333MiB/s (349MB/s)(9988MiB/30030msec) slat (usec): min=256, max=2278, avg=1142.94, stdev=310.28 clat (msec): min=9, max=149, avg=39.71, stdev=17.38 lat (msec): min=10, max=150, avg=40.86, stdev=17.40 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 19], 10.00th=[ 22], 20.00th=[ 27], | 30.00th=[ 30], 40.00th=[ 34], 50.00th=[ 38], 60.00th=[ 41], | 70.00th=[ 46], 80.00th=[ 50], 90.00th=[ 60], 95.00th=[ 72], | 99.00th=[ 104], 99.50th=[ 117], 99.90th=[ 148], 99.95th=[ 148], | 99.99th=[ 150] bw ( KiB/s): min=155648, max=409600, per=99.91%, avg=340286.72, stdev=62316.60, samples=60 iops : min= 38, max= 100, avg=83.07, stdev=15.21, samples=60 lat (usec) : 500=2.73%, 750=0.59%, 1000=0.52% lat (msec) : 2=9.40%, 4=5.69%, 10=7.26%, 20=16.03%, 50=50.67% lat (msec) : 100=6.91%, 250=0.21% cpu : usr=9.21%, sys=2.70%, ctx=6194, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=9809,2497,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1307MiB/s (1370MB/s), 1307MiB/s-1307MiB/s (1370MB/s-1370MB/s), io=38.3GiB (41.1GB), run=30030-30030msec WRITE: bw=333MiB/s (349MB/s), 333MiB/s-333MiB/s (349MB/s-349MB/s), io=9988MiB (10.5GB), run=30030-30030msec Disk stats (read/write): vdc: ios=39104/9940, merge=0/0, ticks=421068/370086, in_queue=813600, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978561: Fri Jan 7 14:37:29 2022 read: IOPS=290, BW=1160MiB/s (1217MB/s)(34.0GiB/30009msec) clat (usec): min=380, max=12832, avg=858.86, stdev=678.98 lat (usec): min=380, max=12832, avg=859.06, stdev=678.99 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 449], | 30.00th=[ 490], 40.00th=[ 562], 50.00th=[ 652], 60.00th=[ 750], | 70.00th=[ 889], 80.00th=[ 1090], 90.00th=[ 1450], 95.00th=[ 1926], | 99.00th=[ 4293], 99.50th=[ 4686], 99.90th=[ 5604], 99.95th=[ 6128], | 99.99th=[12780] bw ( MiB/s): min= 712, max= 1648, per=100.00%, avg=1160.70, stdev=187.81, samples=60 iops : min= 178, max= 412, avg=290.17, stdev=46.96, samples=60 write: IOPS=73, BW=294MiB/s (309MB/s)(8832MiB/30009msec) clat (usec): min=9189, max=13429, avg=9716.19, stdev=669.56 lat (usec): min=9387, max=13846, avg=10193.95, stdev=703.66 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[ 9896], 80.00th=[10159], 90.00th=[10683], 95.00th=[11076], | 99.00th=[12125], 99.50th=[12387], 99.90th=[13435], 99.95th=[13435], | 99.99th=[13435] bw ( KiB/s): min=262144, max=335872, per=99.99%, avg=301340.00, stdev=15481.08, samples=60 iops : min= 64, max= 82, avg=73.57, stdev= 3.77, samples=60 lat (usec) : 500=24.99%, 750=22.77%, 1000=12.86% lat (msec) : 2=15.37%, 4=2.54%, 10=16.31%, 20=5.17% cpu : usr=3.70%, sys=2.99%, ctx=10933, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8705,2208,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1160MiB/s (1217MB/s), 1160MiB/s-1160MiB/s (1217MB/s-1217MB/s), io=34.0GiB (36.5GB), run=30009-30009msec WRITE: bw=294MiB/s (309MB/s), 294MiB/s-294MiB/s (309MB/s-309MB/s), io=8832MiB (9261MB), run=30009-30009msec Disk stats (read/write): vdc: ios=34707/8812, merge=0/0, ticks=4792/65907, in_queue=90742, util=95.07% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979530: Fri Jan 7 14:38:06 2022 read: IOPS=350, BW=1401MiB/s (1469MB/s)(41.2GiB/30084msec) slat (usec): min=38, max=258308, avg=1947.27, stdev=6337.49 clat (usec): min=254, max=381798, avg=71835.72, stdev=50937.79 lat (usec): min=398, max=451507, avg=73783.53, stdev=51465.98 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 8], 10.00th=[ 13], 20.00th=[ 22], | 30.00th=[ 39], 40.00th=[ 60], 50.00th=[ 72], 60.00th=[ 84], | 70.00th=[ 92], 80.00th=[ 104], 90.00th=[ 125], 95.00th=[ 157], | 99.00th=[ 241], 99.50th=[ 309], 99.90th=[ 372], 99.95th=[ 380], | 99.99th=[ 380] bw ( MiB/s): min= 640, max= 1960, per=99.98%, avg=1400.92, stdev=275.67, samples=60 iops : min= 160, max= 490, avg=350.20, stdev=68.94, samples=60 write: IOPS=89, BW=356MiB/s (374MB/s)(10.5GiB/30084msec) slat (usec): min=421, max=238295, avg=2984.52, stdev=7472.47 clat (msec): min=51, max=575, avg=267.69, stdev=56.91 lat (msec): min=62, max=576, avg=270.68, stdev=57.07 clat percentiles (msec): | 1.00th=[ 96], 5.00th=[ 161], 10.00th=[ 205], 20.00th=[ 243], | 30.00th=[ 257], 40.00th=[ 264], 50.00th=[ 271], 60.00th=[ 279], | 70.00th=[ 288], 80.00th=[ 300], 90.00th=[ 313], 95.00th=[ 334], | 99.00th=[ 468], 99.50th=[ 518], 99.90th=[ 558], 99.95th=[ 567], | 99.99th=[ 575] bw ( KiB/s): min=180224, max=409600, per=99.62%, avg=363625.45, stdev=45856.64, samples=60 iops : min= 44, max= 100, avg=88.77, stdev=11.19, samples=60 lat (usec) : 500=0.13%, 750=0.04%, 1000=0.06% lat (msec) : 2=0.48%, 4=1.04%, 10=4.08%, 20=9.22%, 50=12.47% lat (msec) : 100=34.78%, 250=21.74%, 500=15.82%, 750=0.15% cpu : usr=11.98%, sys=2.86%, ctx=4201, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=10538,2681,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1401MiB/s (1469MB/s), 1401MiB/s-1401MiB/s (1469MB/s-1469MB/s), io=41.2GiB (44.2GB), run=30084-30084msec WRITE: bw=356MiB/s (374MB/s), 356MiB/s-356MiB/s (374MB/s-374MB/s), io=10.5GiB (11.2GB), run=30084-30084msec Disk stats (read/write): vdc: ios=42060/10684, merge=0/0, ticks=1084780/2599878, in_queue=3710775, util=99.41% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980495: Fri Jan 7 14:38:43 2022 read: IOPS=291, BW=1165MiB/s (1222MB/s)(34.1GiB/30010msec) clat (usec): min=381, max=7872, avg=845.64, stdev=667.37 lat (usec): min=381, max=7872, avg=845.84, stdev=667.38 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 441], | 30.00th=[ 482], 40.00th=[ 562], 50.00th=[ 644], 60.00th=[ 742], | 70.00th=[ 865], 80.00th=[ 1057], 90.00th=[ 1434], 95.00th=[ 1991], | 99.00th=[ 4228], 99.50th=[ 4621], 99.90th=[ 5604], 99.95th=[ 6063], | 99.99th=[ 7898] bw ( MiB/s): min= 704, max= 1776, per=100.00%, avg=1165.40, stdev=194.99, samples=60 iops : min= 176, max= 444, avg=291.33, stdev=48.76, samples=60 write: IOPS=73, BW=296MiB/s (310MB/s)(8868MiB/30010msec) clat (usec): min=9181, max=14051, avg=9720.31, stdev=684.09 lat (usec): min=9414, max=14674, avg=10191.40, stdev=705.44 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[ 9896], 80.00th=[10159], 90.00th=[10683], 95.00th=[11207], | 99.00th=[12125], 99.50th=[12387], 99.90th=[13435], 99.95th=[13566], | 99.99th=[14091] bw ( KiB/s): min=270336, max=335872, per=100.00%, avg=302579.20, stdev=14403.42, samples=60 iops : min= 66, max= 82, avg=73.87, stdev= 3.51, samples=60 lat (usec) : 500=25.90%, 750=22.53%, 1000=13.62% lat (msec) : 2=13.76%, 4=2.68%, 10=16.41%, 20=5.09% cpu : usr=3.84%, sys=2.83%, ctx=10968, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8740,2217,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1165MiB/s (1222MB/s), 1165MiB/s-1165MiB/s (1222MB/s-1222MB/s), io=34.1GiB (36.7GB), run=30010-30010msec WRITE: bw=296MiB/s (310MB/s), 296MiB/s-296MiB/s (310MB/s-310MB/s), io=8868MiB (9299MB), run=30010-30010msec Disk stats (read/write): vdc: ios=34716/8817, merge=0/0, ticks=4321/65860, in_queue=90401, util=94.75% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981077: Fri Jan 7 14:39:20 2022 read: IOPS=358, BW=1434MiB/s (1504MB/s)(42.1GiB/30063msec) slat (usec): min=45, max=246703, avg=2021.22, stdev=5446.90 clat (msec): min=39, max=509, avg=178.05, stdev=59.35 lat (msec): min=51, max=580, avg=180.07, stdev=59.75 clat percentiles (msec): | 1.00th=[ 78], 5.00th=[ 97], 10.00th=[ 110], 20.00th=[ 129], | 30.00th=[ 144], 40.00th=[ 161], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 203], 80.00th=[ 218], 90.00th=[ 241], 95.00th=[ 271], | 99.00th=[ 397], 99.50th=[ 430], 99.90th=[ 502], 99.95th=[ 506], | 99.99th=[ 510] bw ( MiB/s): min= 720, max= 2384, per=99.59%, avg=1428.55, stdev=258.51, samples=60 iops : min= 180, max= 596, avg=357.10, stdev=64.62, samples=60 write: IOPS=91, BW=365MiB/s (383MB/s)(10.7GiB/30063msec) slat (usec): min=1098, max=249962, avg=2978.22, stdev=6628.61 clat (msec): min=98, max=692, avg=383.15, stdev=67.78 lat (msec): min=99, max=693, avg=386.13, stdev=67.80 clat percentiles (msec): | 1.00th=[ 157], 5.00th=[ 271], 10.00th=[ 321], 20.00th=[ 351], | 30.00th=[ 363], 40.00th=[ 376], 50.00th=[ 384], 60.00th=[ 393], | 70.00th=[ 405], 80.00th=[ 422], 90.00th=[ 447], 95.00th=[ 477], | 99.00th=[ 600], 99.50th=[ 634], 99.90th=[ 676], 99.95th=[ 693], | 99.99th=[ 693] bw ( KiB/s): min=188416, max=491520, per=99.05%, avg=370454.03, stdev=45847.02, samples=60 iops : min= 46, max= 120, avg=90.43, stdev=11.19, samples=60 lat (msec) : 50=0.01%, 100=4.84%, 250=69.28%, 500=25.00%, 750=0.88% cpu : usr=12.32%, sys=3.04%, ctx=4383, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10781,2745,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1434MiB/s (1504MB/s), 1434MiB/s-1434MiB/s (1504MB/s-1504MB/s), io=42.1GiB (45.2GB), run=30063-30063msec WRITE: bw=365MiB/s (383MB/s), 365MiB/s-365MiB/s (383MB/s-383MB/s), io=10.7GiB (11.5GB), run=30063-30063msec Disk stats (read/write): vdc: ios=42864/10888, merge=0/0, ticks=947747/2730167, in_queue=3713406, util=99.05% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981901: Fri Jan 7 14:39:57 2022 read: IOPS=290, BW=1163MiB/s (1219MB/s)(34.1GiB/30003msec) clat (usec): min=376, max=7960, avg=854.77, stdev=672.05 lat (usec): min=376, max=7960, avg=854.96, stdev=672.06 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 445], | 30.00th=[ 494], 40.00th=[ 570], 50.00th=[ 652], 60.00th=[ 750], | 70.00th=[ 873], 80.00th=[ 1074], 90.00th=[ 1434], 95.00th=[ 1975], | 99.00th=[ 4293], 99.50th=[ 4752], 99.90th=[ 5473], 99.95th=[ 6063], | 99.99th=[ 7963] bw ( MiB/s): min= 784, max= 1800, per=100.00%, avg=1163.07, stdev=193.81, samples=60 iops : min= 196, max= 450, avg=290.77, stdev=48.45, samples=60 write: IOPS=73, BW=295MiB/s (309MB/s)(8848MiB/30003msec) clat (usec): min=9183, max=13398, avg=9718.73, stdev=689.60 lat (usec): min=9396, max=14036, avg=10182.43, stdev=706.06 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[ 9896], 80.00th=[10159], 90.00th=[10683], 95.00th=[11207], | 99.00th=[12256], 99.50th=[12387], 99.90th=[12911], 99.95th=[13173], | 99.99th=[13435] bw ( KiB/s): min=270336, max=344064, per=99.96%, avg=301875.20, stdev=14764.49, samples=60 iops : min= 66, max= 84, avg=73.70, stdev= 3.60, samples=60 lat (usec) : 500=24.73%, 750=23.38%, 1000=13.19% lat (msec) : 2=14.61%, 4=2.62%, 10=16.31%, 20=5.16% cpu : usr=3.59%, sys=2.52%, ctx=10952, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8723,2212,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1163MiB/s (1219MB/s), 1163MiB/s-1163MiB/s (1219MB/s-1219MB/s), io=34.1GiB (36.6GB), run=30003-30003msec WRITE: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=8848MiB (9278MB), run=30003-30003msec Disk stats (read/write): vdc: ios=34832/8836, merge=0/0, ticks=4587/66364, in_queue=91322, util=93.62% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983246: Fri Jan 7 14:40:34 2022 read: IOPS=363, BW=1454MiB/s (1524MB/s)(42.6GiB/30043msec) slat (usec): min=45, max=260864, avg=2022.21, stdev=5004.11 clat (msec): min=39, max=719, avg=393.75, stdev=71.24 lat (msec): min=73, max=726, avg=395.77, stdev=71.42 clat percentiles (msec): | 1.00th=[ 230], 5.00th=[ 288], 10.00th=[ 309], 20.00th=[ 338], | 30.00th=[ 359], 40.00th=[ 376], 50.00th=[ 393], 60.00th=[ 409], | 70.00th=[ 426], 80.00th=[ 447], 90.00th=[ 481], 95.00th=[ 514], | 99.00th=[ 584], 99.50th=[ 617], 99.90th=[ 659], 99.95th=[ 676], | 99.99th=[ 693] bw ( MiB/s): min= 720, max= 2000, per=98.76%, avg=1435.43, stdev=214.26, samples=60 iops : min= 180, max= 500, avg=358.83, stdev=53.57, samples=60 write: IOPS=92, BW=368MiB/s (386MB/s)(10.8GiB/30043msec) slat (usec): min=1073, max=103056, avg=2842.06, stdev=3742.01 clat (msec): min=74, max=867, avg=597.66, stdev=80.22 lat (msec): min=75, max=877, avg=600.51, stdev=80.36 clat percentiles (msec): | 1.00th=[ 342], 5.00th=[ 464], 10.00th=[ 510], 20.00th=[ 550], | 30.00th=[ 567], 40.00th=[ 584], 50.00th=[ 600], 60.00th=[ 617], | 70.00th=[ 634], 80.00th=[ 651], 90.00th=[ 684], 95.00th=[ 718], | 99.00th=[ 802], 99.50th=[ 818], 99.90th=[ 835], 99.95th=[ 860], | 99.99th=[ 869] bw ( KiB/s): min=73728, max=524288, per=98.41%, avg=371257.65, stdev=72274.29, samples=60 iops : min= 18, max= 128, avg=90.63, stdev=17.65, samples=60 lat (msec) : 50=0.01%, 100=0.16%, 250=1.48%, 500=74.64%, 750=23.32% lat (msec) : 1000=0.39% cpu : usr=12.69%, sys=3.20%, ctx=4530, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10917,2767,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1454MiB/s (1524MB/s), 1454MiB/s-1454MiB/s (1524MB/s-1524MB/s), io=42.6GiB (45.8GB), run=30043-30043msec WRITE: bw=368MiB/s (386MB/s), 368MiB/s-368MiB/s (386MB/s-386MB/s), io=10.8GiB (11.6GB), run=30043-30043msec Disk stats (read/write): vdc: ios=43446/10998, merge=0/0, ticks=939959/2746589, in_queue=3725056, util=98.39%