30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=890669: Fri Jan 7 13:33:20 2022 read: IOPS=13.1k, BW=51.0MiB/s (53.5MB/s)(1531MiB/30001msec) clat (usec): min=23, max=9411, avg=51.41, stdev=28.92 lat (usec): min=23, max=9411, avg=51.52, stdev=28.93 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 63], 95.00th=[ 74], | 99.00th=[ 165], 99.50th=[ 176], 99.90th=[ 210], 99.95th=[ 245], | 99.99th=[ 486] bw ( KiB/s): min=45960, max=56120, per=99.98%, avg=52233.81, stdev=2130.26, samples=59 iops : min=11490, max=14030, avg=13058.44, stdev=532.56, samples=59 write: IOPS=3279, BW=12.8MiB/s (13.4MB/s)(384MiB/30001msec) clat (usec): min=66, max=2318, avg=93.28, stdev=22.73 lat (usec): min=66, max=2319, avg=93.50, stdev=22.76 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 80], 10.00th=[ 81], 20.00th=[ 83], | 30.00th=[ 86], 40.00th=[ 88], 50.00th=[ 91], 60.00th=[ 94], | 70.00th=[ 97], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 135], 99.50th=[ 145], 99.90th=[ 194], 99.95th=[ 262], | 99.99th=[ 1336] bw ( KiB/s): min=11536, max=14072, per=100.00%, avg=13117.07, stdev=526.43, samples=59 iops : min= 2884, max= 3518, avg=3279.25, stdev=131.59, samples=59 lat (usec) : 50=52.68%, 100=40.24%, 250=7.03%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=6.34%, sys=21.01%, ctx=490265, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=391861,98385,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=51.0MiB/s (53.5MB/s), 51.0MiB/s-51.0MiB/s (53.5MB/s-53.5MB/s), io=1531MiB (1605MB), run=30001-30001msec WRITE: bw=12.8MiB/s (13.4MB/s), 12.8MiB/s-12.8MiB/s (13.4MB/s-13.4MB/s), io=384MiB (403MB), run=30001-30001msec Disk stats (read/write): vdb: ios=389676/97818, merge=0/0, ticks=24/14, in_queue=22234, util=74.28% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=891550: Fri Jan 7 13:33:57 2022 read: IOPS=13.8k, BW=53.8MiB/s (56.4MB/s)(1615MiB/30001msec) slat (usec): min=4, max=514, avg= 8.55, stdev= 3.80 clat (nsec): min=641, max=2466.6k, avg=40038.56, stdev=23778.66 lat (usec): min=20, max=2479, avg=48.72, stdev=24.05 clat percentiles (usec): | 1.00th=[ 20], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 38], | 70.00th=[ 40], 80.00th=[ 44], 90.00th=[ 51], 95.00th=[ 62], | 99.00th=[ 153], 99.50th=[ 163], 99.90th=[ 196], 99.95th=[ 227], | 99.99th=[ 429] bw ( KiB/s): min=51496, max=58576, per=99.95%, avg=55085.83, stdev=1456.94, samples=59 iops : min=12874, max=14644, avg=13771.42, stdev=364.27, samples=59 write: IOPS=3457, BW=13.5MiB/s (14.2MB/s)(405MiB/30001msec) slat (usec): min=5, max=224, avg= 9.28, stdev= 3.72 clat (nsec): min=1246, max=1838.5k, avg=79718.10, stdev=16917.18 lat (usec): min=63, max=1847, avg=89.13, stdev=17.50 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 67], 10.00th=[ 69], 20.00th=[ 71], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 80], | 70.00th=[ 84], 80.00th=[ 88], 90.00th=[ 95], 95.00th=[ 102], | 99.00th=[ 120], 99.50th=[ 129], 99.90th=[ 167], 99.95th=[ 206], | 99.99th=[ 570] bw ( KiB/s): min=12888, max=14952, per=99.97%, avg=13825.36, stdev=371.63, samples=59 iops : min= 3222, max= 3738, avg=3456.34, stdev=92.91, samples=59 lat (nsec) : 750=0.05%, 1000=0.16% lat (usec) : 2=0.10%, 4=0.01%, 10=0.05%, 20=0.59%, 50=70.60% lat (usec) : 100=24.86%, 250=3.55%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=6.37%, sys=31.77%, ctx=514921, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=413364,103725,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=53.8MiB/s (56.4MB/s), 53.8MiB/s-53.8MiB/s (56.4MB/s-56.4MB/s), io=1615MiB (1693MB), run=30001-30001msec WRITE: bw=13.5MiB/s (14.2MB/s), 13.5MiB/s-13.5MiB/s (14.2MB/s-14.2MB/s), io=405MiB (425MB), run=30001-30001msec Disk stats (read/write): vdb: ios=411286/103212, merge=0/0, ticks=12/5, in_queue=22047, util=73.60% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892540: Fri Jan 7 13:34:34 2022 read: IOPS=13.1k, BW=51.2MiB/s (53.7MB/s)(1538MiB/30001msec) clat (usec): min=25, max=6289, avg=51.14, stdev=26.07 lat (usec): min=25, max=6289, avg=51.25, stdev=26.07 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 63], 95.00th=[ 74], | 99.00th=[ 165], 99.50th=[ 176], 99.90th=[ 206], 99.95th=[ 243], | 99.99th=[ 461] bw ( KiB/s): min=48640, max=55888, per=100.00%, avg=52484.47, stdev=1848.29, samples=59 iops : min=12160, max=13972, avg=13121.12, stdev=462.07, samples=59 write: IOPS=3294, BW=12.9MiB/s (13.5MB/s)(386MiB/30001msec) clat (usec): min=65, max=1513, avg=92.79, stdev=17.90 lat (usec): min=65, max=1514, avg=93.01, stdev=17.93 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 79], 10.00th=[ 81], 20.00th=[ 83], | 30.00th=[ 85], 40.00th=[ 88], 50.00th=[ 90], 60.00th=[ 93], | 70.00th=[ 97], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 135], 99.50th=[ 143], 99.90th=[ 188], 99.95th=[ 235], | 99.99th=[ 676] bw ( KiB/s): min=11856, max=13872, per=99.99%, avg=13176.42, stdev=489.99, samples=59 iops : min= 2964, max= 3468, avg=3294.10, stdev=122.50, samples=59 lat (usec) : 50=53.70%, 100=39.45%, 250=6.81%, 500=0.04%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=6.51%, sys=20.92%, ctx=492455, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=393601,98843,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=51.2MiB/s (53.7MB/s), 51.2MiB/s-51.2MiB/s (53.7MB/s-53.7MB/s), io=1538MiB (1612MB), run=30001-30001msec WRITE: bw=12.9MiB/s (13.5MB/s), 12.9MiB/s-12.9MiB/s (13.5MB/s-13.5MB/s), io=386MiB (405MB), run=30001-30001msec Disk stats (read/write): vdb: ios=391425/98279, merge=0/0, ticks=17/6, in_queue=22287, util=74.45% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=892991: Fri Jan 7 13:35:11 2022 read: IOPS=20.5k, BW=80.1MiB/s (84.0MB/s)(2407MiB/30035msec) slat (usec): min=2, max=339, avg= 4.46, stdev= 3.15 clat (nsec): min=672, max=56171k, avg=340590.70, stdev=1373432.02 lat (usec): min=21, max=56180, avg=345.13, stdev=1373.51 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 49], 10.00th=[ 58], 20.00th=[ 70], | 30.00th=[ 80], 40.00th=[ 90], 50.00th=[ 100], 60.00th=[ 112], | 70.00th=[ 124], 80.00th=[ 141], 90.00th=[ 180], 95.00th=[ 725], | 99.00th=[ 8225], 99.50th=[10028], 99.90th=[15139], 99.95th=[17433], | 99.99th=[21890] bw ( KiB/s): min= 6056, max=289096, per=100.00%, avg=82159.93, stdev=118035.98, samples=60 iops : min= 1514, max=72274, avg=20539.98, stdev=29508.99, samples=60 write: IOPS=5150, BW=20.1MiB/s (21.1MB/s)(604MiB/30035msec) slat (usec): min=2, max=144, avg= 5.21, stdev= 3.40 clat (usec): min=43, max=56174, avg=557.48, stdev=1958.09 lat (usec): min=55, max=56184, avg=562.78, stdev=1958.24 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 94], 10.00th=[ 105], 20.00th=[ 119], | 30.00th=[ 129], 40.00th=[ 141], 50.00th=[ 155], 60.00th=[ 172], | 70.00th=[ 188], 80.00th=[ 210], 90.00th=[ 255], 95.00th=[ 2245], | 99.00th=[10814], 99.50th=[13566], 99.90th=[18744], 99.95th=[21627], | 99.99th=[36439] bw ( KiB/s): min= 1563, max=71936, per=100.00%, avg=20625.52, stdev=29581.51, samples=60 iops : min= 390, max=17984, avg=5156.37, stdev=7395.39, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=4.58%, 100=36.82% lat (usec) : 250=51.42%, 500=1.64%, 750=0.25%, 1000=0.26% lat (msec) : 2=1.32%, 4=1.32%, 10=1.73%, 20=0.62%, 50=0.03% lat (msec) : 100=0.01% cpu : usr=4.13%, sys=14.84%, ctx=80219, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=616199,154700,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=80.1MiB/s (84.0MB/s), 80.1MiB/s-80.1MiB/s (84.0MB/s-84.0MB/s), io=2407MiB (2524MB), run=30035-30035msec WRITE: bw=20.1MiB/s (21.1MB/s), 20.1MiB/s-20.1MiB/s (21.1MB/s-21.1MB/s), io=604MiB (634MB), run=30035-30035msec Disk stats (read/write): vdb: ios=615864/154605, merge=0/0, ticks=130681/57084, in_queue=269828, util=98.92% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894406: Fri Jan 7 13:35:48 2022 read: IOPS=13.3k, BW=51.8MiB/s (54.3MB/s)(1554MiB/30001msec) clat (usec): min=25, max=1597, avg=50.50, stdev=23.20 lat (usec): min=25, max=1597, avg=50.62, stdev=23.21 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 40], | 30.00th=[ 42], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 62], 95.00th=[ 74], | 99.00th=[ 165], 99.50th=[ 176], 99.90th=[ 210], 99.95th=[ 239], | 99.99th=[ 445] bw ( KiB/s): min=48592, max=55736, per=100.00%, avg=53035.68, stdev=1428.68, samples=60 iops : min=12148, max=13934, avg=13258.92, stdev=357.16, samples=60 write: IOPS=3329, BW=13.0MiB/s (13.6MB/s)(390MiB/30001msec) clat (usec): min=66, max=5986, avg=92.34, stdev=31.01 lat (usec): min=66, max=5987, avg=92.55, stdev=31.02 clat percentiles (usec): | 1.00th=[ 76], 5.00th=[ 79], 10.00th=[ 81], 20.00th=[ 83], | 30.00th=[ 85], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 93], | 70.00th=[ 96], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 133], 99.50th=[ 143], 99.90th=[ 182], 99.95th=[ 231], | 99.99th=[ 725] bw ( KiB/s): min=12504, max=13912, per=100.00%, avg=13319.15, stdev=318.39, samples=60 iops : min= 3126, max= 3478, avg=3329.77, stdev=79.58, samples=60 lat (usec) : 50=55.54%, 100=37.80%, 250=6.62%, 500=0.03%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=6.47%, sys=20.55%, ctx=497643, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=397741,99888,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=51.8MiB/s (54.3MB/s), 51.8MiB/s-51.8MiB/s (54.3MB/s-54.3MB/s), io=1554MiB (1629MB), run=30001-30001msec WRITE: bw=13.0MiB/s (13.6MB/s), 13.0MiB/s-13.0MiB/s (13.6MB/s-13.6MB/s), io=390MiB (409MB), run=30001-30001msec Disk stats (read/write): vdb: ios=395563/99355, merge=0/0, ticks=6/17, in_queue=22462, util=75.02% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=894976: Fri Jan 7 13:36:25 2022 read: IOPS=22.1k, BW=86.3MiB/s (90.5MB/s)(2590MiB/30007msec) slat (usec): min=2, max=718, avg= 5.01, stdev= 5.87 clat (usec): min=32, max=100794, avg=1733.96, stdev=5496.83 lat (usec): min=63, max=100798, avg=1739.05, stdev=5496.69 clat percentiles (usec): | 1.00th=[ 147], 5.00th=[ 196], 10.00th=[ 231], 20.00th=[ 281], | 30.00th=[ 322], 40.00th=[ 363], 50.00th=[ 404], 60.00th=[ 445], | 70.00th=[ 506], 80.00th=[ 594], 90.00th=[ 734], 95.00th=[13435], | 99.00th=[28443], 99.50th=[33817], 99.90th=[44827], 99.95th=[52167], | 99.99th=[98042] bw ( KiB/s): min= 4144, max=369520, per=100.00%, avg=88404.28, stdev=145055.79, samples=60 iops : min= 1036, max=92380, avg=22101.03, stdev=36263.89, samples=60 write: IOPS=5550, BW=21.7MiB/s (22.7MB/s)(651MiB/30007msec) slat (usec): min=2, max=1255, avg= 6.03, stdev= 6.66 clat (usec): min=119, max=100811, avg=2074.57, stdev=5997.94 lat (usec): min=124, max=100865, avg=2080.68, stdev=5997.82 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 367], 10.00th=[ 420], 20.00th=[ 478], | 30.00th=[ 523], 40.00th=[ 562], 50.00th=[ 603], 60.00th=[ 660], | 70.00th=[ 717], 80.00th=[ 799], 90.00th=[ 988], 95.00th=[15270], | 99.00th=[30540], 99.50th=[36439], 99.90th=[49546], 99.95th=[58459], | 99.99th=[98042] bw ( KiB/s): min= 1104, max=92352, per=100.00%, avg=22205.13, stdev=36385.49, samples=60 iops : min= 276, max=23088, avg=5551.28, stdev=9096.37, samples=60 lat (usec) : 50=0.01%, 100=0.04%, 250=11.00%, 500=49.20%, 750=27.10% lat (usec) : 1000=4.86% lat (msec) : 2=0.75%, 4=0.11%, 10=0.99%, 20=2.91%, 50=2.99% lat (msec) : 100=0.06%, 250=0.01% cpu : usr=4.03%, sys=14.96%, ctx=12615, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=663070,166548,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=86.3MiB/s (90.5MB/s), 86.3MiB/s-86.3MiB/s (90.5MB/s-90.5MB/s), io=2590MiB (2716MB), run=30007-30007msec WRITE: bw=21.7MiB/s (22.7MB/s), 21.7MiB/s-21.7MiB/s (22.7MB/s-22.7MB/s), io=651MiB (682MB), run=30007-30007msec Disk stats (read/write): vdb: ios=662666/166452, merge=0/0, ticks=858165/242257, in_queue=1341147, util=99.60% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=895818: Fri Jan 7 13:37:02 2022 read: IOPS=13.2k, BW=51.5MiB/s (53.0MB/s)(1545MiB/30001msec) clat (usec): min=25, max=3185, avg=50.77, stdev=25.14 lat (usec): min=25, max=3185, avg=50.89, stdev=25.15 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 38], 10.00th=[ 39], 20.00th=[ 41], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 48], | 70.00th=[ 51], 80.00th=[ 55], 90.00th=[ 63], 95.00th=[ 74], | 99.00th=[ 165], 99.50th=[ 176], 99.90th=[ 215], 99.95th=[ 253], | 99.99th=[ 545] bw ( KiB/s): min=46592, max=55392, per=100.00%, avg=52830.69, stdev=1521.23, samples=59 iops : min=11648, max=13848, avg=13207.68, stdev=380.27, samples=59 write: IOPS=3311, BW=12.9MiB/s (13.6MB/s)(388MiB/30001msec) clat (usec): min=67, max=22724, avg=92.95, stdev=85.92 lat (usec): min=67, max=22724, avg=93.16, stdev=85.92 clat percentiles (usec): | 1.00th=[ 77], 5.00th=[ 79], 10.00th=[ 81], 20.00th=[ 83], | 30.00th=[ 85], 40.00th=[ 87], 50.00th=[ 90], 60.00th=[ 93], | 70.00th=[ 96], 80.00th=[ 101], 90.00th=[ 109], 95.00th=[ 116], | 99.00th=[ 135], 99.50th=[ 147], 99.90th=[ 202], 99.95th=[ 343], | 99.99th=[ 1172] bw ( KiB/s): min=12120, max=13952, per=100.00%, avg=13261.58, stdev=362.03, samples=59 iops : min= 3030, max= 3488, avg=3315.36, stdev=90.50, samples=59 lat (usec) : 50=55.24%, 100=38.04%, 250=6.67%, 500=0.04%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=6.05%, sys=20.98%, ctx=494848, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=395497,99340,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=51.5MiB/s (53.0MB/s), 51.5MiB/s-51.5MiB/s (53.0MB/s-53.0MB/s), io=1545MiB (1620MB), run=30001-30001msec WRITE: bw=12.9MiB/s (13.6MB/s), 12.9MiB/s-12.9MiB/s (13.6MB/s-13.6MB/s), io=388MiB (407MB), run=30001-30001msec Disk stats (read/write): vdb: ios=393045/98732, merge=0/0, ticks=16/47, in_queue=22498, util=75.20% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=896943: Fri Jan 7 13:37:39 2022 read: IOPS=23.7k, BW=92.4MiB/s (96.9MB/s)(2774MiB/30007msec) slat (usec): min=2, max=721, avg= 5.07, stdev=10.36 clat (usec): min=144, max=149723, avg=3280.23, stdev=9598.19 lat (usec): min=148, max=149824, avg=3285.38, stdev=9598.06 clat percentiles (usec): | 1.00th=[ 326], 5.00th=[ 437], 10.00th=[ 506], 20.00th=[ 611], | 30.00th=[ 701], 40.00th=[ 799], 50.00th=[ 898], 60.00th=[ 1012], | 70.00th=[ 1156], 80.00th=[ 1319], 90.00th=[ 1631], 95.00th=[ 28181], | 99.00th=[ 47973], 99.50th=[ 54789], 99.90th=[ 70779], 99.95th=[105382], | 99.99th=[147850] bw ( KiB/s): min= 3864, max=360728, per=100.00%, avg=94668.68, stdev=139047.21, samples=60 iops : min= 966, max=90182, avg=23667.17, stdev=34761.80, samples=60 write: IOPS=5939, BW=23.2MiB/s (24.3MB/s)(696MiB/30007msec) slat (usec): min=2, max=658, avg= 6.18, stdev=11.11 clat (usec): min=237, max=148989, avg=3735.17, stdev=9900.87 lat (usec): min=242, max=148994, avg=3741.44, stdev=9900.80 clat percentiles (usec): | 1.00th=[ 537], 5.00th=[ 725], 10.00th=[ 865], 20.00th=[ 1020], | 30.00th=[ 1123], 40.00th=[ 1221], 50.00th=[ 1303], 60.00th=[ 1401], | 70.00th=[ 1516], 80.00th=[ 1696], 90.00th=[ 2089], 95.00th=[ 28967], | 99.00th=[ 50594], 99.50th=[ 56361], 99.90th=[ 74974], 99.95th=[115868], | 99.99th=[147850] bw ( KiB/s): min= 936, max=90152, per=100.00%, avg=23761.67, stdev=34926.26, samples=60 iops : min= 234, max=22538, avg=5940.42, stdev=8731.56, samples=60 lat (usec) : 250=0.10%, 500=7.65%, 750=21.50%, 1000=21.58% lat (msec) : 2=40.67%, 4=1.88%, 10=0.06%, 20=0.48%, 50=5.19% lat (msec) : 100=0.84%, 250=0.06% cpu : usr=4.66%, sys=15.97%, ctx=7036, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=710099,178226,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=92.4MiB/s (96.9MB/s), 92.4MiB/s-92.4MiB/s (96.9MB/s-96.9MB/s), io=2774MiB (2909MB), run=30007-30007msec WRITE: bw=23.2MiB/s (24.3MB/s), 23.2MiB/s-23.2MiB/s (24.3MB/s-24.3MB/s), io=696MiB (730MB), run=30007-30007msec Disk stats (read/write): vdb: ios=709609/178108, merge=0/0, ticks=1708437/481809, in_queue=2638989, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=897506: Fri Jan 7 13:38:16 2022 read: IOPS=12.6k, BW=49.4MiB/s (51.8MB/s)(1482MiB/30001msec) clat (usec): min=25, max=20994, avg=53.29, stdev=51.58 lat (usec): min=25, max=20994, avg=53.40, stdev=51.60 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 39], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 53], 80.00th=[ 58], 90.00th=[ 65], 95.00th=[ 77], | 99.00th=[ 167], 99.50th=[ 178], 99.90th=[ 215], 99.95th=[ 255], | 99.99th=[ 553] bw ( KiB/s): min=40544, max=53968, per=99.94%, avg=50536.34, stdev=2152.45, samples=59 iops : min=10136, max=13492, avg=12634.08, stdev=538.11, samples=59 write: IOPS=3173, BW=12.4MiB/s (12.0MB/s)(372MiB/30001msec) clat (usec): min=66, max=25497, avg=95.60, stdev=104.62 lat (usec): min=66, max=25497, avg=95.82, stdev=104.62 clat percentiles (usec): | 1.00th=[ 78], 5.00th=[ 81], 10.00th=[ 82], 20.00th=[ 85], | 30.00th=[ 87], 40.00th=[ 90], 50.00th=[ 93], 60.00th=[ 95], | 70.00th=[ 99], 80.00th=[ 103], 90.00th=[ 112], 95.00th=[ 119], | 99.00th=[ 141], 99.50th=[ 153], 99.90th=[ 204], 99.95th=[ 277], | 99.99th=[ 832] bw ( KiB/s): min= 9968, max=13480, per=99.86%, avg=12675.68, stdev=588.87, samples=59 iops : min= 2492, max= 3370, avg=3168.92, stdev=147.22, samples=59 lat (usec) : 50=47.37%, 100=44.62%, 250=7.95%, 500=0.04%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01% cpu : usr=6.64%, sys=20.83%, ctx=474489, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=379271,95210,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=49.4MiB/s (51.8MB/s), 49.4MiB/s-49.4MiB/s (51.8MB/s-51.8MB/s), io=1482MiB (1553MB), run=30001-30001msec WRITE: bw=12.4MiB/s (12.0MB/s), 12.4MiB/s-12.4MiB/s (12.0MB/s-12.0MB/s), io=372MiB (390MB), run=30001-30001msec Disk stats (read/write): vdb: ios=376965/94587, merge=0/0, ticks=46/52, in_queue=22112, util=73.91% 30s: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898290: Fri Jan 7 13:38:53 2022 read: IOPS=25.3k, BW=98.7MiB/s (103MB/s)(2964MiB/30029msec) slat (nsec): min=1996, max=203582k, avg=30210.80, stdev=1042281.48 clat (usec): min=58, max=404057, avg=6482.48, stdev=19508.10 lat (usec): min=545, max=443572, avg=6512.77, stdev=19620.17 clat percentiles (usec): | 1.00th=[ 865], 5.00th=[ 1012], 10.00th=[ 1123], 20.00th=[ 1287], | 30.00th=[ 1418], 40.00th=[ 1565], 50.00th=[ 1729], 60.00th=[ 1958], | 70.00th=[ 2278], 80.00th=[ 2606], 90.00th=[ 3064], 95.00th=[ 47449], | 99.00th=[ 99091], 99.50th=[109577], 99.90th=[141558], 99.95th=[183501], | 99.99th=[240124] bw ( KiB/s): min= 3272, max=337656, per=100.00%, avg=101143.58, stdev=142081.12, samples=60 iops : min= 818, max=84414, avg=25285.88, stdev=35520.29, samples=60 write: IOPS=6334, BW=24.7MiB/s (25.9MB/s)(743MiB/30029msec) slat (usec): min=2, max=198772, avg=30.72, stdev=1099.29 clat (usec): min=740, max=236563, avg=5557.84, stdev=12450.87 lat (usec): min=744, max=435337, avg=5588.63, stdev=12645.58 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 1647], 10.00th=[ 1893], 20.00th=[ 2180], | 30.00th=[ 2376], 40.00th=[ 2507], 50.00th=[ 2638], 60.00th=[ 2802], | 70.00th=[ 2966], 80.00th=[ 3195], 90.00th=[ 3687], 95.00th=[ 35390], | 99.00th=[ 62129], 99.50th=[ 73925], 99.90th=[112722], 99.95th=[125305], | 99.99th=[204473] bw ( KiB/s): min= 824, max=84904, per=100.00%, avg=25358.82, stdev=35688.15, samples=60 iops : min= 206, max=21226, avg=6339.70, stdev=8922.04, samples=60 lat (usec) : 100=0.01%, 750=0.09%, 1000=3.52% lat (msec) : 2=47.91%, 4=41.76%, 10=0.30%, 20=0.08%, 50=1.99% lat (msec) : 100=3.52%, 250=0.81%, 500=0.01% cpu : usr=4.21%, sys=14.16%, ctx=6892, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=758749,190213,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=98.7MiB/s (103MB/s), 98.7MiB/s-98.7MiB/s (103MB/s-103MB/s), io=2964MiB (3108MB), run=30029-30029msec WRITE: bw=24.7MiB/s (25.9MB/s), 24.7MiB/s-24.7MiB/s (25.9MB/s-25.9MB/s), io=743MiB (779MB), run=30029-30029msec Disk stats (read/write): vdb: ios=758332/190117, merge=0/0, ticks=2314165/654343, in_queue=3448336, util=99.52% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=898985: Fri Jan 7 13:39:30 2022 read: IOPS=11.2k, BW=175MiB/s (183MB/s)(5235MiB/30001msec) clat (usec): min=27, max=22194, avg=56.69, stdev=51.03 lat (usec): min=27, max=22195, avg=56.81, stdev=51.04 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 69], 95.00th=[ 81], | 99.00th=[ 210], 99.50th=[ 227], 99.90th=[ 302], 99.95th=[ 359], | 99.99th=[ 635] bw ( KiB/s): min=147200, max=192000, per=99.99%, avg=178676.07, stdev=7581.30, samples=59 iops : min= 9200, max=12000, avg=11167.25, stdev=473.83, samples=59 write: IOPS=2804, BW=43.8MiB/s (45.0MB/s)(1315MiB/30001msec) clat (usec): min=92, max=54024, avg=122.57, stdev=194.56 lat (usec): min=93, max=54025, avg=124.01, stdev=194.57 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 106], 10.00th=[ 109], 20.00th=[ 112], | 30.00th=[ 114], 40.00th=[ 117], 50.00th=[ 119], 60.00th=[ 122], | 70.00th=[ 126], 80.00th=[ 131], 90.00th=[ 139], 95.00th=[ 147], | 99.00th=[ 165], 99.50th=[ 178], 99.90th=[ 251], 99.95th=[ 420], | 99.99th=[ 1336] bw ( KiB/s): min=36416, max=49248, per=100.00%, avg=44894.92, stdev=1954.51, samples=59 iops : min= 2276, max= 3078, avg=2805.93, stdev=122.16, samples=59 lat (usec) : 50=40.87%, 100=36.35%, 250=22.56%, 500=0.21%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=5.73%, sys=18.69%, ctx=419230, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=335065,84152,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=175MiB/s (183MB/s), 175MiB/s-175MiB/s (183MB/s-183MB/s), io=5235MiB (5490MB), run=30001-30001msec WRITE: bw=43.8MiB/s (45.0MB/s), 43.8MiB/s-43.8MiB/s (45.0MB/s-45.0MB/s), io=1315MiB (1379MB), run=30001-30001msec Disk stats (read/write): vdb: ios=333060/83677, merge=0/0, ticks=42/81, in_queue=22927, util=76.65% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=899801: Fri Jan 7 13:40:07 2022 read: IOPS=11.6k, BW=181MiB/s (190MB/s)(5432MiB/30001msec) slat (usec): min=4, max=274, avg= 9.16, stdev= 2.98 clat (nsec): min=612, max=6896.9k, avg=45418.08, stdev=33689.30 lat (usec): min=23, max=6924, avg=54.70, stdev=34.00 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 31], 10.00th=[ 33], 20.00th=[ 34], | 30.00th=[ 36], 40.00th=[ 37], 50.00th=[ 39], 60.00th=[ 41], | 70.00th=[ 44], 80.00th=[ 48], 90.00th=[ 56], 95.00th=[ 68], | 99.00th=[ 200], 99.50th=[ 217], 99.90th=[ 297], 99.95th=[ 375], | 99.99th=[ 611] bw ( KiB/s): min=167904, max=196512, per=100.00%, avg=185514.10, stdev=6614.01, samples=59 iops : min=10494, max=12282, avg=11594.63, stdev=413.38, samples=59 write: IOPS=2908, BW=45.5MiB/s (47.7MB/s)(1364MiB/30001msec) slat (usec): min=6, max=475, avg=11.07, stdev= 3.54 clat (nsec): min=885, max=1861.7k, avg=108863.69, stdev=19522.60 lat (usec): min=90, max=1881, avg=120.06, stdev=20.29 clat percentiles (usec): | 1.00th=[ 92], 5.00th=[ 95], 10.00th=[ 97], 20.00th=[ 99], | 30.00th=[ 101], 40.00th=[ 103], 50.00th=[ 106], 60.00th=[ 110], | 70.00th=[ 113], 80.00th=[ 118], 90.00th=[ 125], 95.00th=[ 133], | 99.00th=[ 151], 99.50th=[ 161], 99.90th=[ 215], 99.95th=[ 330], | 99.99th=[ 922] bw ( KiB/s): min=42592, max=49952, per=100.00%, avg=46569.10, stdev=1653.50, samples=59 iops : min= 2662, max= 3122, avg=2910.56, stdev=103.35, samples=59 lat (nsec) : 750=0.01%, 1000=0.02% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.04%, 50=67.28% lat (usec) : 100=15.29%, 250=17.17%, 500=0.17%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=5.82%, sys=28.42%, ctx=434736, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=347676,87269,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=5432MiB (5696MB), run=30001-30001msec WRITE: bw=45.5MiB/s (47.7MB/s), 45.5MiB/s-45.5MiB/s (47.7MB/s-47.7MB/s), io=1364MiB (1430MB), run=30001-30001msec Disk stats (read/write): vdb: ios=345972/86835, merge=0/0, ticks=20/7, in_queue=22817, util=76.20% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901190: Fri Jan 7 13:40:44 2022 read: IOPS=11.6k, BW=181MiB/s (189MB/s)(5416MiB/30001msec) clat (usec): min=25, max=2839, avg=54.55, stdev=31.35 lat (usec): min=25, max=2839, avg=54.66, stdev=31.36 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 50], | 70.00th=[ 53], 80.00th=[ 58], 90.00th=[ 66], 95.00th=[ 78], | 99.00th=[ 208], 99.50th=[ 225], 99.90th=[ 302], 99.95th=[ 355], | 99.99th=[ 519] bw ( KiB/s): min=169472, max=193314, per=100.00%, avg=184868.37, stdev=4687.04, samples=59 iops : min=10592, max=12082, avg=11554.27, stdev=292.94, samples=59 write: IOPS=2899, BW=45.3MiB/s (47.5MB/s)(1359MiB/30001msec) clat (usec): min=92, max=1482, avg=119.35, stdev=18.64 lat (usec): min=93, max=1484, avg=120.75, stdev=18.69 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 106], 20.00th=[ 110], | 30.00th=[ 112], 40.00th=[ 114], 50.00th=[ 117], 60.00th=[ 120], | 70.00th=[ 124], 80.00th=[ 128], 90.00th=[ 137], 95.00th=[ 143], | 99.00th=[ 163], 99.50th=[ 176], 99.90th=[ 219], 99.95th=[ 269], | 99.99th=[ 1029] bw ( KiB/s): min=42784, max=48704, per=100.00%, avg=46408.61, stdev=1251.03, samples=59 iops : min= 2674, max= 3044, avg=2900.53, stdev=78.19, samples=59 lat (usec) : 50=48.07%, 100=29.23%, 250=22.48%, 500=0.20%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=6.34%, sys=18.28%, ctx=433620, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=346612,86998,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=181MiB/s (189MB/s), 181MiB/s-181MiB/s (189MB/s-189MB/s), io=5416MiB (5679MB), run=30001-30001msec WRITE: bw=45.3MiB/s (47.5MB/s), 45.3MiB/s-45.3MiB/s (47.5MB/s-47.5MB/s), io=1359MiB (1425MB), run=30001-30001msec Disk stats (read/write): vdb: ios=344771/86519, merge=0/0, ticks=15/9, in_queue=23054, util=77.02% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=901853: Fri Jan 7 13:41:21 2022 read: IOPS=16.2k, BW=253MiB/s (265MB/s)(7593MiB/30001msec) slat (usec): min=2, max=617, avg= 5.43, stdev= 3.08 clat (usec): min=9, max=34961, avg=397.15, stdev=1104.46 lat (usec): min=24, max=34966, avg=402.65, stdev=1104.51 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 52], 10.00th=[ 62], 20.00th=[ 76], | 30.00th=[ 87], 40.00th=[ 98], 50.00th=[ 110], 60.00th=[ 123], | 70.00th=[ 139], 80.00th=[ 161], 90.00th=[ 562], 95.00th=[ 2442], | 99.00th=[ 5604], 99.50th=[ 6849], 99.90th=[ 9896], 99.95th=[11600], | 99.99th=[21365] bw ( KiB/s): min=32256, max=945280, per=100.00%, avg=259159.67, stdev=373612.50, samples=60 iops : min= 2016, max=59080, avg=16197.47, stdev=23350.79, samples=60 write: IOPS=4066, BW=63.5MiB/s (66.6MB/s)(1906MiB/30001msec) slat (usec): min=3, max=174, avg= 7.43, stdev= 3.29 clat (usec): min=79, max=34774, avg=844.33, stdev=1899.15 lat (usec): min=90, max=34788, avg=851.84, stdev=1899.45 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 133], 10.00th=[ 151], 20.00th=[ 184], | 30.00th=[ 206], 40.00th=[ 223], 50.00th=[ 239], 60.00th=[ 258], | 70.00th=[ 281], 80.00th=[ 318], 90.00th=[ 2835], 95.00th=[ 5342], | 99.00th=[ 8979], 99.50th=[10290], 99.90th=[14353], 99.95th=[16319], | 99.99th=[32375] bw ( KiB/s): min= 8096, max=233792, per=100.00%, avg=65065.97, stdev=93715.92, samples=60 iops : min= 506, max=14612, avg=4066.62, stdev=5857.25, samples=60 lat (usec) : 10=0.01%, 20=0.01%, 50=3.25%, 100=30.30%, 250=47.45% lat (usec) : 500=8.18%, 750=0.68%, 1000=0.82% lat (msec) : 2=2.28%, 4=3.58%, 10=3.26%, 20=0.18%, 50=0.02% cpu : usr=3.78%, sys=14.00%, ctx=74002, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=485928,122001,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=253MiB/s (265MB/s), 253MiB/s-253MiB/s (265MB/s-265MB/s), io=7593MiB (7961MB), run=30001-30001msec WRITE: bw=63.5MiB/s (66.6MB/s), 63.5MiB/s-63.5MiB/s (66.6MB/s-66.6MB/s), io=1906MiB (1999MB), run=30001-30001msec Disk stats (read/write): vdb: ios=485460/121874, merge=0/0, ticks=114844/69321, in_queue=273924, util=99.50% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=902671: Fri Jan 7 13:41:58 2022 read: IOPS=11.2k, BW=175MiB/s (184MB/s)(5259MiB/30001msec) clat (usec): min=27, max=2045, avg=56.50, stdev=30.79 lat (usec): min=27, max=2045, avg=56.61, stdev=30.80 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 69], 95.00th=[ 81], | 99.00th=[ 210], 99.50th=[ 227], 99.90th=[ 302], 99.95th=[ 347], | 99.99th=[ 502] bw ( KiB/s): min=165600, max=188320, per=100.00%, avg=179547.66, stdev=5086.91, samples=59 iops : min=10350, max=11770, avg=11221.73, stdev=317.93, samples=59 write: IOPS=2818, BW=44.0MiB/s (46.2MB/s)(1321MiB/30001msec) clat (usec): min=92, max=1824, avg=121.84, stdev=20.16 lat (usec): min=93, max=1825, avg=123.20, stdev=20.20 clat percentiles (usec): | 1.00th=[ 103], 5.00th=[ 106], 10.00th=[ 109], 20.00th=[ 112], | 30.00th=[ 114], 40.00th=[ 117], 50.00th=[ 120], 60.00th=[ 123], | 70.00th=[ 126], 80.00th=[ 131], 90.00th=[ 139], 95.00th=[ 147], | 99.00th=[ 167], 99.50th=[ 178], 99.90th=[ 215], 99.95th=[ 262], | 99.99th=[ 898] bw ( KiB/s): min=41088, max=47552, per=100.00%, avg=45098.85, stdev=1289.57, samples=59 iops : min= 2568, max= 2972, avg=2818.68, stdev=80.60, samples=59 lat (usec) : 50=41.12%, 100=36.10%, 250=22.57%, 500=0.20%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=5.72%, sys=18.75%, ctx=421162, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=336602,84549,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=175MiB/s (184MB/s), 175MiB/s-175MiB/s (184MB/s-184MB/s), io=5259MiB (5515MB), run=30001-30001msec WRITE: bw=44.0MiB/s (46.2MB/s), 44.0MiB/s-44.0MiB/s (46.2MB/s-46.2MB/s), io=1321MiB (1385MB), run=30001-30001msec Disk stats (read/write): vdb: ios=334947/84127, merge=0/0, ticks=10/6, in_queue=23147, util=77.20% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=903762: Fri Jan 7 13:42:35 2022 read: IOPS=18.5k, BW=289MiB/s (303MB/s)(8662MiB/30002msec) slat (usec): min=2, max=143, avg= 5.12, stdev= 5.63 clat (usec): min=31, max=75450, avg=2030.60, stdev=4589.80 lat (usec): min=39, max=75456, avg=2035.79, stdev=4589.78 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 237], 10.00th=[ 277], 20.00th=[ 334], | 30.00th=[ 388], 40.00th=[ 441], 50.00th=[ 498], 60.00th=[ 553], | 70.00th=[ 627], 80.00th=[ 725], 90.00th=[ 8586], 95.00th=[13566], | 99.00th=[19792], 99.50th=[22152], 99.90th=[33817], 99.95th=[43254], | 99.99th=[64750] bw ( KiB/s): min=24192, max=1117216, per=100.00%, avg=295659.15, stdev=436200.93, samples=60 iops : min= 1512, max=69826, avg=18478.68, stdev=27262.57, samples=60 write: IOPS=4634, BW=72.4MiB/s (75.9MB/s)(2173MiB/30002msec) slat (usec): min=3, max=133, avg= 7.91, stdev= 5.74 clat (usec): min=189, max=75427, avg=2659.88, stdev=4800.75 lat (usec): min=197, max=75493, avg=2667.87, stdev=4800.89 clat percentiles (usec): | 1.00th=[ 474], 5.00th=[ 627], 10.00th=[ 734], 20.00th=[ 889], | 30.00th=[ 971], 40.00th=[ 1029], 50.00th=[ 1090], 60.00th=[ 1139], | 70.00th=[ 1221], 80.00th=[ 1319], 90.00th=[ 9634], 95.00th=[14877], | 99.00th=[20841], 99.50th=[23462], 99.90th=[36439], 99.95th=[44303], | 99.99th=[65799] bw ( KiB/s): min= 6208, max=277856, per=100.00%, avg=74150.67, stdev=109532.13, samples=60 iops : min= 388, max=17366, avg=4634.40, stdev=6845.77, samples=60 lat (usec) : 50=0.01%, 100=0.01%, 250=5.13%, 500=35.62%, 750=26.46% lat (usec) : 1000=9.19% lat (msec) : 2=11.28%, 4=0.16%, 10=3.24%, 20=7.87%, 50=1.01% lat (msec) : 100=0.02% cpu : usr=4.00%, sys=12.84%, ctx=15697, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=554391,139045,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=289MiB/s (303MB/s), 289MiB/s-289MiB/s (303MB/s-303MB/s), io=8662MiB (9083MB), run=30002-30002msec WRITE: bw=72.4MiB/s (75.9MB/s), 72.4MiB/s-72.4MiB/s (75.9MB/s-75.9MB/s), io=2173MiB (2278MB), run=30002-30002msec Disk stats (read/write): vdb: ios=553902/138924, merge=0/0, ticks=841601/254593, in_queue=1384145, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=904243: Fri Jan 7 13:43:12 2022 read: IOPS=11.4k, BW=179MiB/s (187MB/s)(5364MiB/30001msec) clat (usec): min=26, max=4055, avg=55.26, stdev=32.10 lat (usec): min=26, max=4055, avg=55.37, stdev=32.10 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 51], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 67], 95.00th=[ 79], | 99.00th=[ 210], 99.50th=[ 225], 99.90th=[ 302], 99.95th=[ 355], | 99.99th=[ 545] bw ( KiB/s): min=159744, max=194080, per=100.00%, avg=183229.25, stdev=5735.28, samples=59 iops : min= 9984, max=12130, avg=11451.81, stdev=358.46, samples=59 write: IOPS=2872, BW=44.9MiB/s (47.1MB/s)(1346MiB/30001msec) clat (usec): min=90, max=2103, avg=119.98, stdev=19.20 lat (usec): min=92, max=2105, avg=121.36, stdev=19.25 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 106], 20.00th=[ 110], | 30.00th=[ 113], 40.00th=[ 115], 50.00th=[ 118], 60.00th=[ 121], | 70.00th=[ 124], 80.00th=[ 129], 90.00th=[ 137], 95.00th=[ 145], | 99.00th=[ 163], 99.50th=[ 176], 99.90th=[ 210], 99.95th=[ 260], | 99.99th=[ 1057] bw ( KiB/s): min=39328, max=49568, per=100.00%, avg=46015.25, stdev=1873.54, samples=59 iops : min= 2458, max= 3098, avg=2875.93, stdev=117.10, samples=59 lat (usec) : 50=45.36%, 100=31.92%, 250=22.49%, 500=0.21%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=5.91%, sys=19.08%, ctx=429489, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=343306,86173,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=179MiB/s (187MB/s), 179MiB/s-179MiB/s (187MB/s-187MB/s), io=5364MiB (5625MB), run=30001-30001msec WRITE: bw=44.9MiB/s (47.1MB/s), 44.9MiB/s-44.9MiB/s (47.1MB/s-47.1MB/s), io=1346MiB (1412MB), run=30001-30001msec Disk stats (read/write): vdb: ios=341473/85737, merge=0/0, ticks=10/9, in_queue=22908, util=76.58% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905216: Fri Jan 7 13:43:49 2022 read: IOPS=20.7k, BW=324MiB/s (339MB/s)(9712MiB/30020msec) slat (usec): min=2, max=644, avg= 5.16, stdev= 9.78 clat (usec): min=74, max=76737, avg=3641.55, stdev=8167.20 lat (usec): min=80, max=76842, avg=3646.80, stdev=8167.09 clat percentiles (usec): | 1.00th=[ 334], 5.00th=[ 449], 10.00th=[ 529], 20.00th=[ 635], | 30.00th=[ 742], 40.00th=[ 840], 50.00th=[ 955], 60.00th=[ 1074], | 70.00th=[ 1221], 80.00th=[ 1401], 90.00th=[16188], 95.00th=[26608], | 99.00th=[35914], 99.50th=[38536], 99.90th=[46924], 99.95th=[57934], | 99.99th=[76022] bw ( KiB/s): min=43008, max=1172000, per=100.00%, avg=331510.98, stdev=473472.17, samples=60 iops : min= 2688, max=73250, avg=20719.43, stdev=29592.01, samples=60 write: IOPS=5200, BW=81.3MiB/s (85.2MB/s)(2439MiB/30020msec) slat (usec): min=2, max=345, avg= 8.59, stdev=11.60 clat (usec): min=466, max=76075, avg=4695.58, stdev=8218.86 lat (usec): min=472, max=76087, avg=4704.26, stdev=8219.02 clat percentiles (usec): | 1.00th=[ 816], 5.00th=[ 1106], 10.00th=[ 1303], 20.00th=[ 1582], | 30.00th=[ 1778], 40.00th=[ 1942], 50.00th=[ 2073], 60.00th=[ 2212], | 70.00th=[ 2343], 80.00th=[ 2540], 90.00th=[17433], 95.00th=[27395], | 99.00th=[36439], 99.50th=[39060], 99.90th=[50594], 99.95th=[60556], | 99.99th=[76022] bw ( KiB/s): min=10752, max=294144, per=100.00%, avg=83265.82, stdev=118663.60, samples=60 iops : min= 672, max=18384, avg=5204.10, stdev=7416.45, samples=60 lat (usec) : 100=0.01%, 250=0.13%, 500=6.40%, 750=18.48%, 1000=18.72% lat (msec) : 2=35.89%, 4=9.54%, 10=0.07%, 20=2.21%, 50=8.48% lat (msec) : 100=0.09% cpu : usr=5.06%, sys=14.55%, ctx=7583, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=621599,156121,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=324MiB/s (339MB/s), 324MiB/s-324MiB/s (339MB/s-339MB/s), io=9712MiB (10.2GB), run=30020-30020msec WRITE: bw=81.3MiB/s (85.2MB/s), 81.3MiB/s-81.3MiB/s (85.2MB/s-85.2MB/s), io=2439MiB (2558MB), run=30020-30020msec Disk stats (read/write): vdb: ios=621044/155976, merge=0/0, ticks=1706736/573784, in_queue=2692410, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=905794: Fri Jan 7 13:44:26 2022 read: IOPS=11.5k, BW=180MiB/s (189MB/s)(5395MiB/30001msec) clat (usec): min=25, max=6613, avg=54.89, stdev=32.87 lat (usec): min=25, max=6613, avg=55.01, stdev=32.88 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 41], 20.00th=[ 43], | 30.00th=[ 45], 40.00th=[ 46], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 54], 80.00th=[ 58], 90.00th=[ 66], 95.00th=[ 78], | 99.00th=[ 208], 99.50th=[ 225], 99.90th=[ 302], 99.95th=[ 359], | 99.99th=[ 537] bw ( KiB/s): min=168480, max=194848, per=100.00%, avg=184160.56, stdev=6109.26, samples=59 iops : min=10530, max=12178, avg=11510.03, stdev=381.83, samples=59 write: IOPS=2888, BW=45.1MiB/s (47.3MB/s)(1354MiB/30001msec) clat (usec): min=91, max=1967, avg=119.53, stdev=20.26 lat (usec): min=92, max=1969, avg=120.93, stdev=20.30 clat percentiles (usec): | 1.00th=[ 102], 5.00th=[ 105], 10.00th=[ 108], 20.00th=[ 110], | 30.00th=[ 112], 40.00th=[ 115], 50.00th=[ 117], 60.00th=[ 120], | 70.00th=[ 124], 80.00th=[ 128], 90.00th=[ 135], 95.00th=[ 143], | 99.00th=[ 161], 99.50th=[ 174], 99.90th=[ 210], 99.95th=[ 281], | 99.99th=[ 1205] bw ( KiB/s): min=41376, max=49184, per=100.00%, avg=46229.17, stdev=1605.17, samples=59 iops : min= 2586, max= 3074, avg=2889.32, stdev=100.32, samples=59 lat (usec) : 50=46.78%, 100=30.56%, 250=22.44%, 500=0.20%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=5.82%, sys=18.98%, ctx=431935, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=345265,86659,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=5395MiB (5657MB), run=30001-30001msec WRITE: bw=45.1MiB/s (47.3MB/s), 45.1MiB/s-45.1MiB/s (47.3MB/s-47.3MB/s), io=1354MiB (1420MB), run=30001-30001msec Disk stats (read/write): vdb: ios=343307/86173, merge=0/0, ticks=14/9, in_queue=22810, util=76.20% 30s: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=906614: Fri Jan 7 13:45:03 2022 read: IOPS=22.0k, BW=359MiB/s (377MB/s)(10.5GiB/30035msec) slat (usec): min=2, max=82116, avg=33.48, stdev=932.39 clat (usec): min=59, max=124440, avg=6963.62, stdev=17148.79 lat (usec): min=558, max=159609, avg=6997.16, stdev=17254.96 clat percentiles (usec): | 1.00th=[ 791], 5.00th=[ 906], 10.00th=[ 996], 20.00th=[ 1139], | 30.00th=[ 1287], 40.00th=[ 1450], 50.00th=[ 1647], 60.00th=[ 1860], | 70.00th=[ 2040], 80.00th=[ 2278], 90.00th=[ 4359], 95.00th=[ 58459], | 99.00th=[ 77071], 99.50th=[ 81265], 99.90th=[100140], 99.95th=[111674], | 99.99th=[122160] bw ( KiB/s): min=42848, max=1301760, per=100.00%, avg=368202.37, stdev=533853.03, samples=60 iops : min= 2678, max=81360, avg=23012.63, stdev=33365.80, samples=60 write: IOPS=5771, BW=90.2MiB/s (94.6MB/s)(2708MiB/30035msec) slat (usec): min=3, max=64148, avg=33.76, stdev=908.76 clat (usec): min=1101, max=124334, avg=6742.95, stdev=10020.85 lat (usec): min=1107, max=156222, avg=6776.78, stdev=10190.43 clat percentiles (usec): | 1.00th=[ 1614], 5.00th=[ 2671], 10.00th=[ 2966], 20.00th=[ 3163], | 30.00th=[ 3294], 40.00th=[ 3458], 50.00th=[ 3621], 60.00th=[ 3785], | 70.00th=[ 3982], 80.00th=[ 4359], 90.00th=[ 8717], 95.00th=[33424], | 99.00th=[46400], 99.50th=[55313], 99.90th=[73925], 99.95th=[81265], | 99.99th=[96994] bw ( KiB/s): min= 9600, max=326336, per=100.00%, avg=92442.27, stdev=134060.41, samples=60 iops : min= 600, max=20396, avg=5777.62, stdev=8378.77, samples=60 lat (usec) : 100=0.01%, 750=0.38%, 1000=7.90% lat (msec) : 2=46.16%, 4=31.65%, 10=3.93%, 20=0.13%, 50=4.57% lat (msec) : 100=5.20%, 250=0.09% cpu : usr=4.25%, sys=13.11%, ctx=9271, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=690467,173335,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=359MiB/s (377MB/s), 359MiB/s-359MiB/s (377MB/s-377MB/s), io=10.5GiB (11.3GB), run=30035-30035msec WRITE: bw=90.2MiB/s (94.6MB/s), 90.2MiB/s-90.2MiB/s (94.6MB/s-94.6MB/s), io=2708MiB (2840MB), run=30035-30035msec Disk stats (read/write): vdb: ios=689857/173176, merge=0/0, ticks=2232409/843947, in_queue=3555815, util=99.67% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908075: Fri Jan 7 13:45:40 2022 read: IOPS=8890, BW=278MiB/s (291MB/s)(8335MiB/30001msec) clat (usec): min=30, max=2483, avg=60.94, stdev=38.81 lat (usec): min=30, max=2483, avg=61.05, stdev=38.81 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 72], 95.00th=[ 85], | 99.00th=[ 249], 99.50th=[ 277], 99.90th=[ 396], 99.95th=[ 478], | 99.99th=[ 676] bw ( KiB/s): min=168640, max=321344, per=100.00%, avg=284507.73, stdev=35717.60, samples=60 iops : min= 5270, max=10042, avg=8890.87, stdev=1116.17, samples=60 write: IOPS=2226, BW=69.6MiB/s (72.0MB/s)(2087MiB/30001msec) clat (usec): min=129, max=14695, avg=195.86, stdev=354.13 lat (usec): min=131, max=14698, avg=198.87, stdev=354.13 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 149], | 30.00th=[ 151], 40.00th=[ 155], 50.00th=[ 157], 60.00th=[ 159], | 70.00th=[ 163], 80.00th=[ 169], 90.00th=[ 178], 95.00th=[ 188], | 99.00th=[ 1287], 99.50th=[ 2474], 99.90th=[ 5866], 99.95th=[ 7111], | 99.99th=[ 8586] bw ( KiB/s): min=39680, max=79168, per=100.00%, avg=71249.07, stdev=9172.57, samples=60 iops : min= 1240, max= 2474, avg=2226.53, stdev=286.64, samples=60 lat (usec) : 50=31.15%, 100=45.83%, 250=21.88%, 500=0.73%, 750=0.03% lat (usec) : 1000=0.12% lat (msec) : 2=0.14%, 4=0.08%, 10=0.05%, 20=0.01% cpu : usr=5.03%, sys=15.02%, ctx=333532, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=266726,66797,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=278MiB/s (291MB/s), 278MiB/s-278MiB/s (291MB/s-291MB/s), io=8335MiB (8740MB), run=30001-30001msec WRITE: bw=69.6MiB/s (72.0MB/s), 69.6MiB/s-69.6MiB/s (72.0MB/s-72.0MB/s), io=2087MiB (2189MB), run=30001-30001msec Disk stats (read/write): vdb: ios=265918/66590, merge=0/0, ticks=16/1865, in_queue=24093, util=80.46% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=908673: Fri Jan 7 13:46:17 2022 read: IOPS=9436, BW=295MiB/s (309MB/s)(8847MiB/30001msec) slat (usec): min=5, max=377, avg=10.63, stdev= 5.34 clat (nsec): min=635, max=2506.2k, avg=49430.30, stdev=37932.51 lat (usec): min=26, max=2515, avg=60.20, stdev=38.34 clat percentiles (usec): | 1.00th=[ 23], 5.00th=[ 32], 10.00th=[ 34], 20.00th=[ 36], | 30.00th=[ 38], 40.00th=[ 40], 50.00th=[ 42], 60.00th=[ 44], | 70.00th=[ 47], 80.00th=[ 52], 90.00th=[ 60], 95.00th=[ 73], | 99.00th=[ 237], 99.50th=[ 265], 99.90th=[ 375], 99.95th=[ 453], | 99.99th=[ 570] bw ( KiB/s): min=168704, max=330880, per=100.00%, avg=303895.86, stdev=33626.93, samples=59 iops : min= 5272, max=10340, avg=9496.75, stdev=1050.84, samples=59 write: IOPS=2367, BW=73.0MiB/s (77.6MB/s)(2219MiB/30001msec) slat (usec): min=8, max=203, avg=14.78, stdev= 6.81 clat (usec): min=58, max=7679, avg=161.05, stdev=153.03 lat (usec): min=128, max=7691, avg=175.98, stdev=153.05 clat percentiles (usec): | 1.00th=[ 123], 5.00th=[ 130], 10.00th=[ 133], 20.00th=[ 137], | 30.00th=[ 139], 40.00th=[ 141], 50.00th=[ 143], 60.00th=[ 147], | 70.00th=[ 151], 80.00th=[ 155], 90.00th=[ 165], 95.00th=[ 174], | 99.00th=[ 848], 99.50th=[ 1188], 99.90th=[ 2245], 99.95th=[ 2868], | 99.99th=[ 4080] bw ( KiB/s): min=44864, max=83904, per=100.00%, avg=76178.44, stdev=8162.47, samples=59 iops : min= 1402, max= 2622, avg=2380.58, stdev=255.08, samples=59 lat (nsec) : 750=0.01%, 1000=0.13% lat (usec) : 2=0.13%, 4=0.02%, 10=0.06%, 20=0.18%, 50=61.47% lat (usec) : 100=15.13%, 250=22.05%, 500=0.51%, 750=0.02%, 1000=0.15% lat (msec) : 2=0.09%, 4=0.03%, 10=0.01% cpu : usr=5.44%, sys=24.21%, ctx=352682, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=283091,71015,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=8847MiB (9276MB), run=30001-30001msec WRITE: bw=73.0MiB/s (77.6MB/s), 73.0MiB/s-73.0MiB/s (77.6MB/s-77.6MB/s), io=2219MiB (2327MB), run=30001-30001msec Disk stats (read/write): vdb: ios=282110/70758, merge=0/0, ticks=8/562, in_queue=23480, util=78.42% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=909509: Fri Jan 7 13:46:54 2022 read: IOPS=9777, BW=306MiB/s (320MB/s)(9167MiB/30002msec) clat (usec): min=29, max=3369, avg=58.33, stdev=38.97 lat (usec): min=29, max=3369, avg=58.44, stdev=38.98 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 56], 80.00th=[ 60], 90.00th=[ 69], 95.00th=[ 82], | 99.00th=[ 245], 99.50th=[ 273], 99.90th=[ 383], 99.95th=[ 474], | 99.99th=[ 627] bw ( KiB/s): min=192576, max=338112, per=100.00%, avg=312900.27, stdev=29243.57, samples=60 iops : min= 6018, max=10566, avg=9778.13, stdev=913.86, samples=60 write: IOPS=2454, BW=76.7MiB/s (80.4MB/s)(2302MiB/30002msec) clat (usec): min=129, max=12956, avg=165.13, stdev=122.14 lat (usec): min=132, max=12958, avg=168.08, stdev=122.15 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 145], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 151], 50.00th=[ 153], 60.00th=[ 157], | 70.00th=[ 161], 80.00th=[ 165], 90.00th=[ 174], 95.00th=[ 182], | 99.00th=[ 229], 99.50th=[ 881], 99.90th=[ 1795], 99.95th=[ 2024], | 99.99th=[ 4047] bw ( KiB/s): min=47808, max=84544, per=100.00%, avg=78556.80, stdev=6947.33, samples=60 iops : min= 1494, max= 2642, avg=2454.90, stdev=217.10, samples=60 lat (usec) : 50=40.60%, 100=36.38%, 250=22.16%, 500=0.65%, 750=0.03% lat (usec) : 1000=0.08% lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=5.76%, sys=15.90%, ctx=367003, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=293344,73648,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=306MiB/s (320MB/s), 306MiB/s-306MiB/s (320MB/s-320MB/s), io=9167MiB (9612MB), run=30002-30002msec WRITE: bw=76.7MiB/s (80.4MB/s), 76.7MiB/s-76.7MiB/s (80.4MB/s-80.4MB/s), io=2302MiB (2413MB), run=30002-30002msec Disk stats (read/write): vdb: ios=292277/73400, merge=0/0, ticks=18/332, in_queue=23668, util=79.08% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910037: Fri Jan 7 13:47:31 2022 read: IOPS=12.8k, BW=399MiB/s (418MB/s)(11.7GiB/30001msec) slat (usec): min=2, max=695, avg= 6.21, stdev= 5.08 clat (nsec): min=631, max=30981k, avg=467645.84, stdev=923533.17 lat (usec): min=18, max=30994, avg=473.95, stdev=923.47 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 51], 10.00th=[ 61], 20.00th=[ 76], | 30.00th=[ 89], 40.00th=[ 102], 50.00th=[ 118], 60.00th=[ 137], | 70.00th=[ 165], 80.00th=[ 523], 90.00th=[ 1582], 95.00th=[ 2507], | 99.00th=[ 4293], 99.50th=[ 4948], 99.90th=[ 6390], 99.95th=[ 7046], | 99.99th=[10028] bw ( KiB/s): min=98432, max=1409024, per=100.00%, avg=413294.64, stdev=518615.61, samples=59 iops : min= 3076, max=44032, avg=12915.42, stdev=16206.76, samples=59 write: IOPS=3201, BW=100MiB/s (105MB/s)(3002MiB/30001msec) slat (usec): min=4, max=265, avg=10.84, stdev= 7.63 clat (usec): min=120, max=36500, avg=1219.87, stdev=1738.56 lat (usec): min=130, max=36508, avg=1230.82, stdev=1738.04 clat percentiles (usec): | 1.00th=[ 184], 5.00th=[ 243], 10.00th=[ 281], 20.00th=[ 334], | 30.00th=[ 379], 40.00th=[ 424], 50.00th=[ 469], 60.00th=[ 523], | 70.00th=[ 603], 80.00th=[ 1876], 90.00th=[ 3916], 95.00th=[ 5211], | 99.00th=[ 7439], 99.50th=[ 8225], 99.90th=[10028], 99.95th=[11469], | 99.99th=[28443] bw ( KiB/s): min=25920, max=353472, per=100.00%, avg=103719.05, stdev=130230.80, samples=59 iops : min= 810, max=11046, avg=3241.22, stdev=4069.71, samples=59 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=3.65%, 100=27.30% lat (usec) : 250=32.76%, 500=11.29%, 750=7.27%, 1000=1.97% lat (msec) : 2=5.88%, 4=6.89%, 10=2.94%, 20=0.02%, 50=0.01% cpu : usr=4.27%, sys=12.92%, ctx=76954, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=382690,96056,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=399MiB/s (418MB/s), 399MiB/s-399MiB/s (418MB/s-418MB/s), io=11.7GiB (12.5GB), run=30001-30001msec WRITE: bw=100MiB/s (105MB/s), 100MiB/s-100MiB/s (105MB/s-105MB/s), io=3002MiB (3148MB), run=30001-30001msec Disk stats (read/write): vdb: ios=382229/95917, merge=0/0, ticks=100178/73084, in_queue=277454, util=99.54% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=910980: Fri Jan 7 13:48:08 2022 read: IOPS=9902, BW=309MiB/s (324MB/s)(9284MiB/30001msec) clat (usec): min=27, max=2345, avg=58.83, stdev=39.19 lat (usec): min=27, max=2345, avg=58.95, stdev=39.19 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 70], 95.00th=[ 84], | 99.00th=[ 247], 99.50th=[ 281], 99.90th=[ 408], 99.95th=[ 486], | 99.99th=[ 791] bw ( KiB/s): min=276672, max=336256, per=99.96%, avg=316752.27, stdev=12290.60, samples=59 iops : min= 8646, max=10508, avg=9898.51, stdev=384.08, samples=59 write: IOPS=2486, BW=77.7MiB/s (81.5MB/s)(2331MiB/30001msec) clat (usec): min=127, max=32040, avg=157.77, stdev=128.44 lat (usec): min=130, max=32043, avg=160.73, stdev=128.46 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 143], 10.00th=[ 143], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 151], 50.00th=[ 153], 60.00th=[ 157], | 70.00th=[ 161], 80.00th=[ 165], 90.00th=[ 174], 95.00th=[ 182], | 99.00th=[ 206], 99.50th=[ 221], 99.90th=[ 351], 99.95th=[ 570], | 99.99th=[ 1336] bw ( KiB/s): min=70464, max=86656, per=100.00%, avg=79559.59, stdev=3438.12, samples=59 iops : min= 2202, max= 2708, avg=2486.24, stdev=107.44, samples=59 lat (usec) : 50=39.38%, 100=37.48%, 250=22.39%, 500=0.71%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.84%, sys=17.04%, ctx=371694, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=297089,74591,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=309MiB/s (324MB/s), 309MiB/s-309MiB/s (324MB/s-324MB/s), io=9284MiB (9735MB), run=30001-30001msec WRITE: bw=77.7MiB/s (81.5MB/s), 77.7MiB/s-77.7MiB/s (81.5MB/s-81.5MB/s), io=2331MiB (2444MB), run=30001-30001msec Disk stats (read/write): vdb: ios=295615/74234, merge=0/0, ticks=15/57, in_queue=23365, util=78.03% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=911952: Fri Jan 7 13:48:45 2022 read: IOPS=15.5k, BW=485MiB/s (509MB/s)(14.2GiB/30018msec) slat (usec): min=2, max=162, avg= 5.63, stdev= 4.03 clat (nsec): min=1833, max=57554k, avg=2015532.10, stdev=4213399.50 lat (usec): min=20, max=57617, avg=2021.24, stdev=4213.37 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 53], 10.00th=[ 64], 20.00th=[ 81], | 30.00th=[ 97], 40.00th=[ 115], 50.00th=[ 135], 60.00th=[ 165], | 70.00th=[ 221], 80.00th=[ 3064], 90.00th=[ 9110], 95.00th=[11731], | 99.00th=[16319], 99.50th=[18744], 99.90th=[27395], 99.95th=[36963], | 99.99th=[47973] bw ( KiB/s): min=62400, max=1497856, per=100.00%, avg=497381.18, stdev=600708.45, samples=60 iops : min= 1950, max=46808, avg=15543.12, stdev=18772.11, samples=60 write: IOPS=3898, BW=122MiB/s (128MB/s)(3657MiB/30018msec) slat (usec): min=4, max=213, avg= 9.58, stdev= 4.48 clat (usec): min=669, max=57073, avg=4758.58, stdev=3578.89 lat (usec): min=686, max=57083, avg=4768.25, stdev=3579.50 clat percentiles (usec): | 1.00th=[ 2278], 5.00th=[ 2704], 10.00th=[ 2835], 20.00th=[ 2966], | 30.00th=[ 3097], 40.00th=[ 3228], 50.00th=[ 3392], 60.00th=[ 3556], | 70.00th=[ 3818], 80.00th=[ 4817], 90.00th=[10159], 95.00th=[12911], | 99.00th=[17695], 99.50th=[19530], 99.90th=[31327], 99.95th=[40109], | 99.99th=[49021] bw ( KiB/s): min=14464, max=373504, per=100.00%, avg=124839.95, stdev=150732.37, samples=60 iops : min= 452, max=11672, avg=3901.22, stdev=4710.35, samples=60 lat (usec) : 2=0.01%, 20=0.01%, 50=3.23%, 100=22.25%, 250=32.63% lat (usec) : 500=4.81%, 750=0.40%, 1000=0.05% lat (msec) : 2=0.12%, 4=17.01%, 10=11.04%, 20=8.12%, 50=0.32% lat (msec) : 100=0.01% cpu : usr=4.45%, sys=14.08%, ctx=73396, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=466234,117026,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=485MiB/s (509MB/s), 485MiB/s-485MiB/s (509MB/s-509MB/s), io=14.2GiB (15.3GB), run=30018-30018msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3657MiB (3835MB), run=30018-30018msec Disk stats (read/write): vdb: ios=465750/116910, merge=0/0, ticks=820124/486844, in_queue=1449017, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=912586: Fri Jan 7 13:49:22 2022 read: IOPS=10.1k, BW=315MiB/s (330MB/s)(9455MiB/30001msec) clat (usec): min=27, max=6689, avg=57.53, stdev=40.29 lat (usec): min=27, max=6689, avg=57.65, stdev=40.30 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 45], 40.00th=[ 47], 50.00th=[ 49], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 82], | 99.00th=[ 245], 99.50th=[ 277], 99.90th=[ 408], 99.95th=[ 469], | 99.99th=[ 594] bw ( KiB/s): min=298176, max=339904, per=100.00%, avg=322727.47, stdev=9575.69, samples=60 iops : min= 9318, max=10622, avg=10085.23, stdev=299.24, samples=60 write: IOPS=2530, BW=79.1MiB/s (82.9MB/s)(2372MiB/30001msec) clat (usec): min=127, max=37525, avg=155.98, stdev=146.04 lat (usec): min=129, max=37528, avg=158.94, stdev=146.05 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 141], 10.00th=[ 143], 20.00th=[ 145], | 30.00th=[ 147], 40.00th=[ 149], 50.00th=[ 153], 60.00th=[ 155], | 70.00th=[ 159], 80.00th=[ 163], 90.00th=[ 172], 95.00th=[ 180], | 99.00th=[ 202], 99.50th=[ 217], 99.90th=[ 289], 99.95th=[ 515], | 99.99th=[ 824] bw ( KiB/s): min=74944, max=86912, per=100.00%, avg=80969.60, stdev=2277.11, samples=60 iops : min= 2342, max= 2716, avg=2530.30, stdev=71.09, samples=60 lat (usec) : 50=43.85%, 100=33.06%, 250=22.38%, 500=0.68%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=6.09%, sys=16.52%, ctx=378482, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=302557,75910,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=315MiB/s (330MB/s), 315MiB/s-315MiB/s (330MB/s-330MB/s), io=9455MiB (9914MB), run=30001-30001msec WRITE: bw=79.1MiB/s (82.9MB/s), 79.1MiB/s-79.1MiB/s (82.9MB/s-82.9MB/s), io=2372MiB (2487MB), run=30001-30001msec Disk stats (read/write): vdb: ios=301001/75556, merge=0/0, ticks=18/55, in_queue=23490, util=78.45% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=913420: Fri Jan 7 13:49:59 2022 read: IOPS=18.3k, BW=570MiB/s (598MB/s)(16.7GiB/30054msec) slat (usec): min=2, max=502, avg= 6.93, stdev= 8.68 clat (nsec): min=703, max=60852k, avg=3198662.19, stdev=6997112.74 lat (usec): min=21, max=60858, avg=3205.69, stdev=6996.82 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 68], 10.00th=[ 85], 20.00th=[ 115], | 30.00th=[ 145], 40.00th=[ 180], 50.00th=[ 227], 60.00th=[ 297], | 70.00th=[ 424], 80.00th=[ 988], 90.00th=[16909], 95.00th=[20579], | 99.00th=[26608], 99.50th=[28443], 99.90th=[33817], 99.95th=[42206], | 99.99th=[60556] bw ( KiB/s): min=108288, max=1512128, per=100.00%, avg=585105.95, stdev=625708.56, samples=60 iops : min= 3384, max=47254, avg=18284.55, stdev=19553.38, samples=60 write: IOPS=4578, BW=143MiB/s (150MB/s)(4300MiB/30054msec) slat (usec): min=4, max=376, avg=13.38, stdev=12.51 clat (usec): min=1203, max=62715, avg=9043.36, stdev=5390.68 lat (usec): min=1210, max=62729, avg=9056.85, stdev=5390.19 clat percentiles (usec): | 1.00th=[ 2835], 5.00th=[ 3523], 10.00th=[ 4228], 20.00th=[ 5473], | 30.00th=[ 6390], 40.00th=[ 7046], 50.00th=[ 7570], 60.00th=[ 8094], | 70.00th=[ 8979], 80.00th=[10945], 90.00th=[17433], 95.00th=[21103], | 99.00th=[27132], 99.50th=[28705], 99.90th=[38536], 99.95th=[43779], | 99.99th=[62653] bw ( KiB/s): min=26112, max=375872, per=100.00%, avg=146754.53, stdev=157051.67, samples=60 iops : min= 816, max=11746, avg=4586.07, stdev=4907.85, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 10=0.01%, 20=0.01%, 50=1.21%, 100=10.89%, 250=31.14% lat (usec) : 500=15.28%, 750=4.04%, 1000=1.44% lat (msec) : 2=1.24%, 4=3.53%, 10=14.61%, 20=10.83%, 50=5.74% lat (msec) : 100=0.03% cpu : usr=6.76%, sys=18.61%, ctx=52810, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=548569,137593,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=570MiB/s (598MB/s), 570MiB/s-570MiB/s (598MB/s-598MB/s), io=16.7GiB (17.0GB), run=30054-30054msec WRITE: bw=143MiB/s (150MB/s), 143MiB/s-143MiB/s (150MB/s-150MB/s), io=4300MiB (4509MB), run=30054-30054msec Disk stats (read/write): vdb: ios=547766/137396, merge=0/0, ticks=1526026/1150027, in_queue=2861576, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=914732: Fri Jan 7 13:50:36 2022 read: IOPS=10.1k, BW=315MiB/s (330MB/s)(9451MiB/30001msec) clat (usec): min=26, max=2610, avg=57.68, stdev=38.67 lat (usec): min=26, max=2610, avg=57.80, stdev=38.68 clat percentiles (usec): | 1.00th=[ 39], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 59], 90.00th=[ 68], 95.00th=[ 81], | 99.00th=[ 245], 99.50th=[ 273], 99.90th=[ 396], 99.95th=[ 469], | 99.99th=[ 603] bw ( KiB/s): min=292224, max=340928, per=99.99%, avg=322556.75, stdev=9442.48, samples=59 iops : min= 9132, max=10654, avg=10079.90, stdev=295.08, samples=59 write: IOPS=2529, BW=79.0MiB/s (82.9MB/s)(2371MiB/30001msec) clat (usec): min=127, max=25887, avg=155.88, stdev=117.01 lat (usec): min=130, max=25891, avg=158.83, stdev=117.03 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 141], 10.00th=[ 143], 20.00th=[ 145], | 30.00th=[ 147], 40.00th=[ 151], 50.00th=[ 153], 60.00th=[ 155], | 70.00th=[ 159], 80.00th=[ 163], 90.00th=[ 172], 95.00th=[ 178], | 99.00th=[ 198], 99.50th=[ 208], 99.90th=[ 262], 99.95th=[ 383], | 99.99th=[ 1352] bw ( KiB/s): min=70272, max=85824, per=100.00%, avg=80976.27, stdev=2504.57, samples=59 iops : min= 2196, max= 2682, avg=2530.51, stdev=78.27, samples=59 lat (usec) : 50=42.70%, 100=34.30%, 250=22.31%, 500=0.66%, 750=0.03% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.82%, sys=16.80%, ctx=378325, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=302435,75878,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=315MiB/s (330MB/s), 315MiB/s-315MiB/s (330MB/s-330MB/s), io=9451MiB (9910MB), run=30001-30001msec WRITE: bw=79.0MiB/s (82.9MB/s), 79.0MiB/s-79.0MiB/s (82.9MB/s-82.9MB/s), io=2371MiB (2486MB), run=30001-30001msec Disk stats (read/write): vdb: ios=300542/75443, merge=0/0, ticks=16/56, in_queue=23515, util=78.62% 30s: (g=0): rw=randrw, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=915351: Fri Jan 7 13:51:13 2022 read: IOPS=20.7k, BW=647MiB/s (678MB/s)(18.0GiB/30050msec) slat (usec): min=2, max=98877, avg=36.07, stdev=763.25 clat (usec): min=83, max=141906, avg=6401.23, stdev=14070.66 lat (usec): min=715, max=154107, avg=6437.39, stdev=14153.32 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1074], 10.00th=[ 1123], 20.00th=[ 1221], | 30.00th=[ 1303], 40.00th=[ 1385], 50.00th=[ 1483], 60.00th=[ 1614], | 70.00th=[ 1811], 80.00th=[ 2212], 90.00th=[ 23725], 95.00th=[ 46400], | 99.00th=[ 60031], 99.50th=[ 64750], 99.90th=[ 91751], 99.95th=[101188], | 99.99th=[141558] bw ( KiB/s): min=109952, max=1507648, per=100.00%, avg=663493.23, stdev=654518.99, samples=60 iops : min= 3436, max=47114, avg=20734.17, stdev=20453.79, samples=60 write: IOPS=5199, BW=162MiB/s (170MB/s)(4883MiB/30050msec) slat (usec): min=3, max=77493, avg=41.18, stdev=774.02 clat (msec): min=2, max=127, avg=12.78, stdev= 6.58 lat (msec): min=2, max=140, avg=12.82, stdev= 6.74 clat percentiles (msec): | 1.00th=[ 5], 5.00th=[ 6], 10.00th=[ 7], 20.00th=[ 9], | 30.00th=[ 11], 40.00th=[ 11], 50.00th=[ 12], 60.00th=[ 12], | 70.00th=[ 13], 80.00th=[ 15], 90.00th=[ 21], 95.00th=[ 27], | 99.00th=[ 36], 99.50th=[ 42], 99.90th=[ 64], 99.95th=[ 75], | 99.99th=[ 104] bw ( KiB/s): min=28224, max=372672, per=100.00%, avg=166650.85, stdev=163825.16, samples=60 iops : min= 882, max=11646, avg=5207.82, stdev=5119.55, samples=60 lat (usec) : 100=0.01%, 750=0.01%, 1000=1.28% lat (msec) : 2=59.56%, 4=7.24%, 10=8.04%, 20=12.99%, 50=8.03% lat (msec) : 100=2.80%, 250=0.05% cpu : usr=6.35%, sys=15.22%, ctx=50591, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=622183,156258,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=647MiB/s (678MB/s), 647MiB/s-647MiB/s (678MB/s-678MB/s), io=18.0GiB (20.4GB), run=30050-30050msec WRITE: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=4883MiB (5120MB), run=30050-30050msec Disk stats (read/write): vdb: ios=621664/156135, merge=0/0, ticks=1788864/1698096, in_queue=3698942, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916108: Fri Jan 7 13:51:50 2022 read: IOPS=7283, BW=455MiB/s (477MB/s)(13.3GiB/30001msec) clat (usec): min=36, max=2535, avg=73.69, stdev=55.42 lat (usec): min=36, max=2535, avg=73.81, stdev=55.43 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 50], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 85], 95.00th=[ 113], | 99.00th=[ 330], 99.50th=[ 392], 99.90th=[ 545], 99.95th=[ 627], | 99.99th=[ 1254] bw ( KiB/s): min=409088, max=505472, per=99.96%, avg=465953.05, stdev=20902.16, samples=59 iops : min= 6392, max= 7898, avg=7280.51, stdev=326.60, samples=59 write: IOPS=1821, BW=114MiB/s (119MB/s)(3416MiB/30001msec) clat (usec): min=209, max=48133, avg=240.84, stdev=208.68 lat (usec): min=213, max=48140, avg=246.92, stdev=208.71 clat percentiles (usec): | 1.00th=[ 219], 5.00th=[ 223], 10.00th=[ 225], 20.00th=[ 227], | 30.00th=[ 231], 40.00th=[ 233], 50.00th=[ 235], 60.00th=[ 239], | 70.00th=[ 243], 80.00th=[ 249], 90.00th=[ 258], 95.00th=[ 269], | 99.00th=[ 302], 99.50th=[ 322], 99.90th=[ 611], 99.95th=[ 807], | 99.99th=[ 1516] bw ( KiB/s): min=102272, max=126720, per=100.00%, avg=116705.36, stdev=5290.62, samples=59 iops : min= 1598, max= 1980, avg=1823.51, stdev=82.66, samples=59 lat (usec) : 50=3.71%, 100=71.49%, 250=18.75%, 500=5.88%, 750=0.13% lat (usec) : 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=5.15%, sys=13.19%, ctx=273194, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=218513,54656,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=455MiB/s (477MB/s), 455MiB/s-455MiB/s (477MB/s-477MB/s), io=13.3GiB (14.3GB), run=30001-30001msec WRITE: bw=114MiB/s (119MB/s), 114MiB/s-114MiB/s (119MB/s-119MB/s), io=3416MiB (3582MB), run=30001-30001msec Disk stats (read/write): vdb: ios=217620/54451, merge=0/0, ticks=21/63, in_queue=24430, util=81.51% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=916843: Fri Jan 7 13:52:27 2022 read: IOPS=7252, BW=453MiB/s (475MB/s)(13.3GiB/30001msec) slat (usec): min=6, max=166, avg=12.80, stdev= 3.62 clat (nsec): min=698, max=4617.7k, avg=61809.68, stdev=54426.06 lat (usec): min=35, max=4640, avg=74.75, stdev=54.79 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 40], 10.00th=[ 42], 20.00th=[ 44], | 30.00th=[ 46], 40.00th=[ 48], 50.00th=[ 50], 60.00th=[ 53], | 70.00th=[ 57], 80.00th=[ 61], 90.00th=[ 71], 95.00th=[ 92], | 99.00th=[ 318], 99.50th=[ 371], 99.90th=[ 529], 99.95th=[ 594], | 99.99th=[ 947] bw ( KiB/s): min=422784, max=500992, per=99.82%, avg=463334.15, stdev=17533.32, samples=59 iops : min= 6606, max= 7828, avg=7239.59, stdev=273.96, samples=59 write: IOPS=1814, BW=113MiB/s (119MB/s)(3403MiB/30001msec) slat (usec): min=10, max=128, avg=19.21, stdev= 4.72 clat (usec): min=170, max=5785, avg=226.29, stdev=33.90 lat (usec): min=210, max=5812, avg=245.65, stdev=34.71 clat percentiles (usec): | 1.00th=[ 206], 5.00th=[ 210], 10.00th=[ 212], 20.00th=[ 215], | 30.00th=[ 219], 40.00th=[ 221], 50.00th=[ 223], 60.00th=[ 227], | 70.00th=[ 231], 80.00th=[ 235], 90.00th=[ 243], 95.00th=[ 251], | 99.00th=[ 273], 99.50th=[ 285], 99.90th=[ 420], 99.95th=[ 644], | 99.99th=[ 1336] bw ( KiB/s): min=105344, max=125696, per=100.00%, avg=116169.34, stdev=4576.76, samples=59 iops : min= 1646, max= 1964, avg=1815.14, stdev=71.51, samples=59 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=40.63% lat (usec) : 100=35.67%, 250=20.26%, 500=3.30%, 750=0.11%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=5.13%, sys=21.39%, ctx=272030, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=217597,54450,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=453MiB/s (475MB/s), 453MiB/s-453MiB/s (475MB/s-475MB/s), io=13.3GiB (14.3GB), run=30001-30001msec WRITE: bw=113MiB/s (119MB/s), 113MiB/s-113MiB/s (119MB/s-119MB/s), io=3403MiB (3568MB), run=30001-30001msec Disk stats (read/write): vdb: ios=216355/54177, merge=0/0, ticks=26/11, in_queue=23939, util=79.96% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=917788: Fri Jan 7 13:53:04 2022 read: IOPS=7767, BW=485MiB/s (509MB/s)(14.2GiB/30001msec) clat (usec): min=32, max=4402, avg=67.18, stdev=55.99 lat (usec): min=32, max=4403, avg=67.29, stdev=55.99 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 58], | 70.00th=[ 62], 80.00th=[ 67], 90.00th=[ 76], 95.00th=[ 98], | 99.00th=[ 322], 99.50th=[ 388], 99.90th=[ 553], 99.95th=[ 619], | 99.99th=[ 1221] bw ( KiB/s): min=446080, max=538368, per=100.00%, avg=497180.46, stdev=19849.31, samples=59 iops : min= 6970, max= 8412, avg=7768.42, stdev=310.16, samples=59 write: IOPS=1946, BW=122MiB/s (128MB/s)(3649MiB/30001msec) clat (usec): min=201, max=8591, avg=232.31, stdev=46.34 lat (usec): min=206, max=8598, avg=238.54, stdev=46.44 clat percentiles (usec): | 1.00th=[ 212], 5.00th=[ 217], 10.00th=[ 219], 20.00th=[ 221], | 30.00th=[ 223], 40.00th=[ 227], 50.00th=[ 229], 60.00th=[ 233], | 70.00th=[ 235], 80.00th=[ 241], 90.00th=[ 249], 95.00th=[ 258], | 99.00th=[ 285], 99.50th=[ 306], 99.90th=[ 603], 99.95th=[ 791], | 99.99th=[ 1467] bw ( KiB/s): min=111872, max=134784, per=100.00%, avg=124634.81, stdev=5211.85, samples=59 iops : min= 1748, max= 2106, avg=1947.39, stdev=81.44, samples=59 lat (usec) : 50=21.38%, 100=54.67%, 250=19.85%, 500=3.94%, 750=0.13% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=5.39%, sys=14.07%, ctx=291428, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=233033,58385,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=485MiB/s (509MB/s), 485MiB/s-485MiB/s (509MB/s-509MB/s), io=14.2GiB (15.3GB), run=30001-30001msec WRITE: bw=122MiB/s (128MB/s), 122MiB/s-122MiB/s (128MB/s-128MB/s), io=3649MiB (3826MB), run=30001-30001msec Disk stats (read/write): vdb: ios=231805/58086, merge=0/0, ticks=33/26, in_queue=24005, util=80.21% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=918734: Fri Jan 7 13:53:41 2022 read: IOPS=11.3k, BW=704MiB/s (738MB/s)(20.6GiB/30002msec) slat (usec): min=3, max=1427, avg= 8.01, stdev= 5.33 clat (nsec): min=622, max=24408k, avg=455300.71, stdev=918209.83 lat (usec): min=25, max=24413, avg=463.41, stdev=917.89 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 42], 10.00th=[ 48], 20.00th=[ 60], | 30.00th=[ 71], 40.00th=[ 83], 50.00th=[ 97], 60.00th=[ 117], | 70.00th=[ 151], 80.00th=[ 652], 90.00th=[ 1532], 95.00th=[ 2343], | 99.00th=[ 4228], 99.50th=[ 5014], 99.90th=[ 6915], 99.95th=[ 8356], | 99.99th=[16712] bw ( KiB/s): min=100096, max=1558656, per=100.00%, avg=720617.32, stdev=597901.47, samples=60 iops : min= 1564, max=24354, avg=11259.63, stdev=9342.19, samples=60 write: IOPS=2827, BW=177MiB/s (185MB/s)(5302MiB/30002msec) slat (usec): min=5, max=920, avg=16.54, stdev=10.09 clat (usec): min=220, max=35309, avg=1669.44, stdev=1515.95 lat (usec): min=237, max=35321, avg=1686.10, stdev=1515.76 clat percentiles (usec): | 1.00th=[ 449], 5.00th=[ 701], 10.00th=[ 816], 20.00th=[ 938], | 30.00th=[ 1029], 40.00th=[ 1106], 50.00th=[ 1172], 60.00th=[ 1237], | 70.00th=[ 1336], 80.00th=[ 1778], 90.00th=[ 3556], 95.00th=[ 4883], | 99.00th=[ 7439], 99.50th=[ 8455], 99.90th=[13304], 99.95th=[19268], | 99.99th=[25297] bw ( KiB/s): min=26112, max=388352, per=100.00%, avg=180974.45, stdev=149681.75, samples=60 iops : min= 408, max= 6068, avg=2827.70, stdev=2338.77, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.05%, 50=9.57% lat (usec) : 100=32.00%, 250=21.35%, 500=1.17%, 750=1.84%, 1000=7.30% lat (msec) : 2=17.69%, 4=6.43%, 10=2.51%, 20=0.06%, 50=0.01% cpu : usr=5.68%, sys=16.01%, ctx=132572, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=337767,84830,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=704MiB/s (738MB/s), 704MiB/s-704MiB/s (738MB/s-738MB/s), io=20.6GiB (22.1GB), run=30002-30002msec WRITE: bw=177MiB/s (185MB/s), 177MiB/s-177MiB/s (185MB/s-185MB/s), io=5302MiB (5559MB), run=30002-30002msec Disk stats (read/write): vdb: ios=337297/84707, merge=0/0, ticks=87136/101167, in_queue=281324, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=919314: Fri Jan 7 13:54:18 2022 read: IOPS=7836, BW=490MiB/s (514MB/s)(14.3GiB/30001msec) clat (usec): min=31, max=6862, avg=66.23, stdev=55.62 lat (usec): min=31, max=6862, avg=66.35, stdev=55.63 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 65], 90.00th=[ 75], 95.00th=[ 95], | 99.00th=[ 322], 99.50th=[ 383], 99.90th=[ 529], 99.95th=[ 594], | 99.99th=[ 898] bw ( KiB/s): min=450432, max=541056, per=99.96%, avg=501319.14, stdev=16852.32, samples=59 iops : min= 7038, max= 8454, avg=7833.10, stdev=263.32, samples=59 write: IOPS=1963, BW=123MiB/s (129MB/s)(3681MiB/30001msec) clat (usec): min=200, max=40259, avg=231.72, stdev=166.94 lat (usec): min=206, max=40265, avg=237.80, stdev=166.97 clat percentiles (usec): | 1.00th=[ 212], 5.00th=[ 217], 10.00th=[ 219], 20.00th=[ 221], | 30.00th=[ 223], 40.00th=[ 227], 50.00th=[ 229], 60.00th=[ 231], | 70.00th=[ 235], 80.00th=[ 239], 90.00th=[ 247], 95.00th=[ 255], | 99.00th=[ 277], 99.50th=[ 289], 99.90th=[ 343], 99.95th=[ 537], | 99.99th=[ 1385] bw ( KiB/s): min=110080, max=134912, per=100.00%, avg=125643.80, stdev=4209.07, samples=59 iops : min= 1720, max= 2108, avg=1963.17, stdev=65.77, samples=59 lat (usec) : 50=24.49%, 100=51.72%, 250=19.94%, 500=3.73%, 750=0.11% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=5.52%, sys=13.61%, ctx=294018, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=235104,58899,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=490MiB/s (514MB/s), 490MiB/s-490MiB/s (514MB/s-514MB/s), io=14.3GiB (15.4GB), run=30001-30001msec WRITE: bw=123MiB/s (129MB/s), 123MiB/s-123MiB/s (129MB/s-129MB/s), io=3681MiB (3860MB), run=30001-30001msec Disk stats (read/write): vdb: ios=233706/58539, merge=0/0, ticks=18/53, in_queue=24315, util=81.27% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=920160: Fri Jan 7 13:54:55 2022 read: IOPS=11.7k, BW=731MiB/s (767MB/s)(21.4GiB/30004msec) slat (usec): min=2, max=242, avg= 8.42, stdev= 4.39 clat (nsec): min=699, max=35160k, avg=928722.17, stdev=3136956.53 lat (usec): min=24, max=35169, avg=937.25, stdev=3136.73 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 35], 10.00th=[ 38], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 51], 50.00th=[ 58], 60.00th=[ 65], | 70.00th=[ 77], 80.00th=[ 94], 90.00th=[ 157], 95.00th=[10159], | 99.00th=[15008], 99.50th=[16057], 99.90th=[18744], 99.95th=[20055], | 99.99th=[34341] bw ( KiB/s): min=195584, max=1538048, per=100.00%, avg=749043.90, stdev=277438.59, samples=60 iops : min= 3056, max=24032, avg=11703.77, stdev=4334.99, samples=60 write: IOPS=2937, BW=184MiB/s (192MB/s)(5508MiB/30004msec) slat (usec): min=5, max=202, avg=16.08, stdev= 6.00 clat (usec): min=783, max=49286, avg=13266.32, stdev=4315.65 lat (usec): min=821, max=49302, avg=13282.54, stdev=4316.48 clat percentiles (usec): | 1.00th=[ 5932], 5.00th=[ 6980], 10.00th=[ 8291], 20.00th=[ 9241], | 30.00th=[10683], 40.00th=[11994], 50.00th=[13173], 60.00th=[14222], | 70.00th=[15139], 80.00th=[16188], 90.00th=[18482], 95.00th=[20841], | 99.00th=[26608], 99.50th=[28705], 99.90th=[34341], 99.95th=[35914], | 99.99th=[42206] bw ( KiB/s): min=48128, max=386176, per=100.00%, avg=187990.95, stdev=69086.68, samples=60 iops : min= 752, max= 6034, avg=2937.33, stdev=1079.51, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.05%, 50=31.21% lat (usec) : 100=34.32%, 250=7.71%, 500=0.33%, 750=0.05%, 1000=0.03% lat (msec) : 2=0.03%, 4=0.12%, 10=7.32%, 20=17.46%, 50=1.31% cpu : usr=7.12%, sys=21.74%, ctx=285202, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=351138,88129,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=731MiB/s (767MB/s), 731MiB/s-731MiB/s (767MB/s-767MB/s), io=21.4GiB (23.0GB), run=30004-30004msec WRITE: bw=184MiB/s (192MB/s), 184MiB/s-184MiB/s (192MB/s-192MB/s), io=5508MiB (5776MB), run=30004-30004msec Disk stats (read/write): vdb: ios=350577/87993, merge=0/0, ticks=282496/1121751, in_queue=1478829, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=921272: Fri Jan 7 13:55:32 2022 read: IOPS=7350, BW=459MiB/s (482MB/s)(13.5GiB/30001msec) clat (usec): min=35, max=17101, avg=72.67, stdev=67.57 lat (usec): min=35, max=17102, avg=72.79, stdev=67.57 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 59], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 83], 95.00th=[ 103], | 99.00th=[ 326], 99.50th=[ 388], 99.90th=[ 545], 99.95th=[ 619], | 99.99th=[ 1106] bw ( KiB/s): min=430976, max=503424, per=99.96%, avg=470239.46, stdev=14738.08, samples=59 iops : min= 6734, max= 7866, avg=7347.49, stdev=230.28, samples=59 write: IOPS=1840, BW=115MiB/s (121MB/s)(3451MiB/30001msec) clat (usec): min=204, max=20616, avg=239.90, stdev=118.15 lat (usec): min=209, max=20622, avg=245.85, stdev=118.20 clat percentiles (usec): | 1.00th=[ 219], 5.00th=[ 223], 10.00th=[ 225], 20.00th=[ 227], | 30.00th=[ 231], 40.00th=[ 233], 50.00th=[ 237], 60.00th=[ 239], | 70.00th=[ 243], 80.00th=[ 249], 90.00th=[ 258], 95.00th=[ 265], | 99.00th=[ 285], 99.50th=[ 302], 99.90th=[ 400], 99.95th=[ 668], | 99.99th=[ 1516] bw ( KiB/s): min=103808, max=123520, per=99.97%, avg=117733.97, stdev=4136.47, samples=59 iops : min= 1622, max= 1930, avg=1839.59, stdev=64.63, samples=59 lat (usec) : 50=5.83%, 100=69.94%, 250=18.38%, 500=5.71%, 750=0.12% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.18%, sys=13.48%, ctx=275748, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=220522,55208,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=459MiB/s (482MB/s), 459MiB/s-459MiB/s (482MB/s-482MB/s), io=13.5GiB (14.5GB), run=30001-30001msec WRITE: bw=115MiB/s (121MB/s), 115MiB/s-115MiB/s (121MB/s-121MB/s), io=3451MiB (3618MB), run=30001-30001msec Disk stats (read/write): vdb: ios=219625/54955, merge=0/0, ticks=45/54, in_queue=24361, util=81.29% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922074: Fri Jan 7 13:56:09 2022 read: IOPS=12.4k, BW=775MiB/s (813MB/s)(22.7GiB/30001msec) slat (usec): min=2, max=240, avg= 7.67, stdev= 5.42 clat (nsec): min=736, max=95993k, avg=1823688.20, stdev=6240840.02 lat (usec): min=26, max=96173, avg=1831.46, stdev=6240.44 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 37], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 51], 40.00th=[ 58], 50.00th=[ 65], 60.00th=[ 75], | 70.00th=[ 86], 80.00th=[ 106], 90.00th=[ 404], 95.00th=[17171], | 99.00th=[28443], 99.50th=[31327], 99.90th=[65274], 99.95th=[72877], | 99.99th=[94897] bw ( KiB/s): min=114661, max=1367808, per=100.00%, avg=793922.88, stdev=377242.29, samples=60 iops : min= 1791, max=21372, avg=12405.00, stdev=5894.40, samples=60 write: IOPS=3110, BW=194MiB/s (204MB/s)(5833MiB/30001msec) slat (usec): min=5, max=270, avg=16.51, stdev= 6.79 clat (msec): min=3, max=111, avg=24.82, stdev=11.46 lat (msec): min=3, max=111, avg=24.84, stdev=11.46 clat percentiles (usec): | 1.00th=[ 6718], 5.00th=[10159], 10.00th=[12911], 20.00th=[16712], | 30.00th=[19006], 40.00th=[20579], 50.00th=[22414], 60.00th=[24249], | 70.00th=[25822], 80.00th=[32113], 90.00th=[41681], 95.00th=[48497], | 99.00th=[62129], 99.50th=[66847], 99.90th=[78119], 99.95th=[82314], | 99.99th=[94897] bw ( KiB/s): min=26420, max=346240, per=100.00%, avg=199120.53, stdev=93984.39, samples=60 iops : min= 412, max= 5410, avg=3111.22, stdev=1468.51, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=22.39% lat (usec) : 100=39.78%, 250=9.53%, 500=0.29%, 750=0.11%, 1000=0.07% lat (msec) : 2=0.14%, 4=0.05%, 10=2.60%, 20=8.82%, 50=15.25% lat (msec) : 100=0.92%, 250=0.01% cpu : usr=7.27%, sys=21.42%, ctx=264860, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=372100,93325,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=775MiB/s (813MB/s), 775MiB/s-775MiB/s (813MB/s-813MB/s), io=22.7GiB (24.4GB), run=30001-30001msec WRITE: bw=194MiB/s (204MB/s), 194MiB/s-194MiB/s (204MB/s-204MB/s), io=5833MiB (6116MB), run=30001-30001msec Disk stats (read/write): vdb: ios=371540/93185, merge=0/0, ticks=611762/2262927, in_queue=2958737, util=99.75% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=922828: Fri Jan 7 13:56:46 2022 read: IOPS=7978, BW=499MiB/s (523MB/s)(14.6GiB/30001msec) clat (usec): min=31, max=2596, avg=64.56, stdev=52.94 lat (usec): min=31, max=2596, avg=64.67, stdev=52.94 clat percentiles (usec): | 1.00th=[ 42], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 63], 90.00th=[ 73], 95.00th=[ 90], | 99.00th=[ 318], 99.50th=[ 379], 99.90th=[ 529], 99.95th=[ 578], | 99.99th=[ 717] bw ( KiB/s): min=474112, max=540416, per=100.00%, avg=510915.19, stdev=14130.58, samples=59 iops : min= 7408, max= 8444, avg=7983.02, stdev=220.77, samples=59 write: IOPS=1999, BW=125MiB/s (131MB/s)(3750MiB/30001msec) clat (usec): min=200, max=21288, avg=229.23, stdev=119.68 lat (usec): min=205, max=21293, avg=235.38, stdev=119.71 clat percentiles (usec): | 1.00th=[ 212], 5.00th=[ 215], 10.00th=[ 217], 20.00th=[ 219], | 30.00th=[ 221], 40.00th=[ 223], 50.00th=[ 227], 60.00th=[ 229], | 70.00th=[ 233], 80.00th=[ 237], 90.00th=[ 245], 95.00th=[ 251], | 99.00th=[ 269], 99.50th=[ 281], 99.90th=[ 318], 99.95th=[ 375], | 99.99th=[ 1352] bw ( KiB/s): min=113536, max=135424, per=99.98%, avg=127971.80, stdev=3536.64, samples=59 iops : min= 1774, max= 2116, avg=1999.53, stdev=55.25, samples=59 lat (usec) : 50=30.55%, 100=45.83%, 250=20.17%, 500=3.32%, 750=0.11% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 50=0.01% cpu : usr=5.72%, sys=13.22%, ctx=299388, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=239369,59999,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=499MiB/s (523MB/s), 499MiB/s-499MiB/s (523MB/s-523MB/s), io=14.6GiB (15.7GB), run=30001-30001msec WRITE: bw=125MiB/s (131MB/s), 125MiB/s-125MiB/s (131MB/s-131MB/s), io=3750MiB (3932MB), run=30001-30001msec Disk stats (read/write): vdb: ios=238384/59735, merge=0/0, ticks=17/47, in_queue=24314, util=81.12% 30s: (g=0): rw=randrw, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=923570: Fri Jan 7 13:57:23 2022 read: IOPS=13.1k, BW=822MiB/s (862MB/s)(24.1GiB/30004msec) slat (usec): min=2, max=54814, avg=56.89, stdev=677.02 clat (usec): min=63, max=95063, avg=7001.95, stdev=12138.32 lat (usec): min=922, max=126582, avg=7058.94, stdev=12206.63 clat percentiles (usec): | 1.00th=[ 1713], 5.00th=[ 2147], 10.00th=[ 2409], 20.00th=[ 2868], | 30.00th=[ 3195], 40.00th=[ 3523], 50.00th=[ 3851], 60.00th=[ 4228], | 70.00th=[ 4686], 80.00th=[ 5407], 90.00th=[ 8586], 95.00th=[28705], | 99.00th=[69731], 99.50th=[73925], 99.90th=[89654], 99.95th=[92799], | 99.99th=[93848] bw ( KiB/s): min=176640, max=1582336, per=99.98%, avg=841478.70, stdev=379305.02, samples=60 iops : min= 2760, max=24724, avg=13148.07, stdev=5926.66, samples=60 write: IOPS=3303, BW=206MiB/s (216MB/s)(6194MiB/30004msec) slat (usec): min=5, max=41106, avg=67.82, stdev=669.59 clat (msec): min=5, max=107, avg=32.37, stdev=10.63 lat (msec): min=6, max=121, avg=32.44, stdev=10.68 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 20], 20.00th=[ 25], | 30.00th=[ 29], 40.00th=[ 31], 50.00th=[ 32], 60.00th=[ 33], | 70.00th=[ 35], 80.00th=[ 40], 90.00th=[ 47], 95.00th=[ 52], | 99.00th=[ 63], 99.50th=[ 69], 99.90th=[ 89], 99.95th=[ 93], | 99.99th=[ 103] bw ( KiB/s): min=47488, max=386944, per=100.00%, avg=211403.62, stdev=93527.49, samples=60 iops : min= 742, max= 6046, avg=3303.15, stdev=1461.38, samples=60 lat (usec) : 100=0.01%, 1000=0.01% lat (msec) : 2=2.68%, 4=40.78%, 10=29.07%, 20=5.03%, 50=18.45% lat (msec) : 100=3.99%, 250=0.01% cpu : usr=6.67%, sys=10.73%, ctx=51486, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=394566,99111,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=822MiB/s (862MB/s), 822MiB/s-822MiB/s (862MB/s-862MB/s), io=24.1GiB (25.9GB), run=30004-30004msec WRITE: bw=206MiB/s (216MB/s), 206MiB/s-206MiB/s (216MB/s-216MB/s), io=6194MiB (6495MB), run=30004-30004msec Disk stats (read/write): vdb: ios=394060/98976, merge=0/0, ticks=798249/2775594, in_queue=3695081, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=924581: Fri Jan 7 13:58:00 2022 read: IOPS=6831, BW=534MiB/s (560MB/s)(15.6GiB/30001msec) clat (usec): min=33, max=21755, avg=74.08, stdev=87.79 lat (usec): min=33, max=21755, avg=74.20, stdev=87.80 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 55], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 67], 80.00th=[ 72], 90.00th=[ 83], 95.00th=[ 135], | 99.00th=[ 371], 99.50th=[ 441], 99.90th=[ 627], 99.95th=[ 701], | 99.99th=[ 1106] bw ( KiB/s): min=500320, max=583680, per=100.00%, avg=546926.10, stdev=19364.26, samples=59 iops : min= 6254, max= 7296, avg=6836.58, stdev=242.05, samples=59 write: IOPS=1711, BW=134MiB/s (140MB/s)(4011MiB/30001msec) clat (usec): min=238, max=41908, avg=273.54, stdev=187.08 lat (usec): min=248, max=41915, avg=281.09, stdev=187.11 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 255], 10.00th=[ 258], 20.00th=[ 260], | 30.00th=[ 265], 40.00th=[ 265], 50.00th=[ 269], 60.00th=[ 273], | 70.00th=[ 277], 80.00th=[ 281], 90.00th=[ 289], 95.00th=[ 297], | 99.00th=[ 334], 99.50th=[ 363], 99.90th=[ 709], 99.95th=[ 947], | 99.99th=[ 1795] bw ( KiB/s): min=121760, max=147520, per=100.00%, avg=137016.95, stdev=4719.49, samples=59 iops : min= 1522, max= 1844, avg=1712.71, stdev=58.99, samples=59 lat (usec) : 50=8.08%, 100=67.01%, 250=2.35%, 500=22.27%, 750=0.25% lat (usec) : 1000=0.03% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.18%, sys=13.24%, ctx=256303, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=204950,51337,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=534MiB/s (560MB/s), 534MiB/s-534MiB/s (560MB/s-560MB/s), io=15.6GiB (16.8GB), run=30001-30001msec WRITE: bw=134MiB/s (140MB/s), 134MiB/s-134MiB/s (140MB/s-140MB/s), io=4011MiB (4206MB), run=30001-30001msec Disk stats (read/write): vdb: ios=203916/51078, merge=0/0, ticks=54/59, in_queue=24391, util=81.47% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=925577: Fri Jan 7 13:58:37 2022 read: IOPS=6460, BW=505MiB/s (529MB/s)(14.8GiB/30001msec) slat (usec): min=6, max=216, avg=13.66, stdev= 3.67 clat (nsec): min=936, max=2393.1k, avg=67613.36, stdev=60404.75 lat (usec): min=41, max=2405, avg=81.41, stdev=60.74 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 44], 10.00th=[ 45], 20.00th=[ 47], | 30.00th=[ 49], 40.00th=[ 51], 50.00th=[ 54], 60.00th=[ 57], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 77], 95.00th=[ 112], | 99.00th=[ 355], 99.50th=[ 416], 99.90th=[ 594], 99.95th=[ 668], | 99.99th=[ 1037] bw ( KiB/s): min=483040, max=556800, per=100.00%, avg=517086.63, stdev=17667.93, samples=59 iops : min= 6038, max= 6960, avg=6463.56, stdev=220.86, samples=59 write: IOPS=1617, BW=126MiB/s (132MB/s)(3790MiB/30001msec) slat (nsec): min=11867, max=93159, avg=21560.99, stdev=4802.32 clat (usec): min=204, max=1864, avg=264.80, stdev=24.55 lat (usec): min=252, max=1939, avg=286.51, stdev=25.72 clat percentiles (usec): | 1.00th=[ 245], 5.00th=[ 249], 10.00th=[ 251], 20.00th=[ 253], | 30.00th=[ 255], 40.00th=[ 260], 50.00th=[ 262], 60.00th=[ 265], | 70.00th=[ 269], 80.00th=[ 273], 90.00th=[ 281], 95.00th=[ 289], | 99.00th=[ 314], 99.50th=[ 326], 99.90th=[ 482], 99.95th=[ 693], | 99.99th=[ 1401] bw ( KiB/s): min=120480, max=137440, per=99.95%, avg=129293.86, stdev=4102.50, samples=59 iops : min= 1506, max= 1718, avg=1616.17, stdev=51.28, samples=59 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=29.29%, 100=46.41%, 250=3.52% lat (usec) : 500=20.56%, 750=0.19%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=4.67%, sys=20.02%, ctx=242367, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=193833,48513,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=505MiB/s (529MB/s), 505MiB/s-505MiB/s (529MB/s-529MB/s), io=14.8GiB (15.9GB), run=30001-30001msec WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=3790MiB (3974MB), run=30001-30001msec Disk stats (read/write): vdb: ios=192998/48285, merge=0/0, ticks=19/11, in_queue=24276, util=81.04% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926008: Fri Jan 7 13:59:14 2022 read: IOPS=6772, BW=529MiB/s (555MB/s)(15.5GiB/30001msec) clat (usec): min=38, max=21492, avg=75.26, stdev=78.80 lat (usec): min=38, max=21492, avg=75.38, stdev=78.80 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 83], 95.00th=[ 120], | 99.00th=[ 367], 99.50th=[ 433], 99.90th=[ 619], 99.95th=[ 693], | 99.99th=[ 1254] bw ( KiB/s): min=475840, max=587360, per=100.00%, avg=542284.29, stdev=19303.15, samples=59 iops : min= 5948, max= 7342, avg=6778.54, stdev=241.31, samples=59 write: IOPS=1697, BW=133MiB/s (139MB/s)(3978MiB/30001msec) clat (usec): min=241, max=2443, avg=273.97, stdev=28.26 lat (usec): min=250, max=2451, avg=281.64, stdev=28.39 clat percentiles (usec): | 1.00th=[ 253], 5.00th=[ 258], 10.00th=[ 260], 20.00th=[ 265], | 30.00th=[ 265], 40.00th=[ 269], 50.00th=[ 273], 60.00th=[ 273], | 70.00th=[ 277], 80.00th=[ 281], 90.00th=[ 289], 95.00th=[ 297], | 99.00th=[ 322], 99.50th=[ 338], 99.90th=[ 478], 99.95th=[ 783], | 99.99th=[ 1385] bw ( KiB/s): min=117760, max=142240, per=100.00%, avg=135792.98, stdev=3973.25, samples=59 iops : min= 1472, max= 1778, avg=1697.41, stdev=49.67, samples=59 lat (usec) : 50=3.44%, 100=71.86%, 250=2.07%, 500=22.37%, 750=0.22% lat (usec) : 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 50=0.01% cpu : usr=4.97%, sys=12.74%, ctx=254136, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=203196,50921,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=529MiB/s (555MB/s), 529MiB/s-529MiB/s (555MB/s-555MB/s), io=15.5GiB (16.6GB), run=30001-30001msec WRITE: bw=133MiB/s (139MB/s), 133MiB/s-133MiB/s (139MB/s-139MB/s), io=3978MiB (4171MB), run=30001-30001msec Disk stats (read/write): vdb: ios=202504/50712, merge=0/0, ticks=51/13, in_queue=24589, util=82.03% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=926959: Fri Jan 7 13:59:51 2022 read: IOPS=11.5k, BW=901MiB/s (945MB/s)(26.4GiB/30001msec) slat (usec): min=3, max=899, avg= 9.17, stdev= 4.22 clat (nsec): min=792, max=21751k, avg=398647.84, stdev=882303.56 lat (usec): min=29, max=21757, avg=407.92, stdev=881.98 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 42], 10.00th=[ 47], 20.00th=[ 57], | 30.00th=[ 68], 40.00th=[ 78], 50.00th=[ 91], 60.00th=[ 105], | 70.00th=[ 128], 80.00th=[ 184], 90.00th=[ 1483], 95.00th=[ 2147], | 99.00th=[ 4146], 99.50th=[ 4948], 99.90th=[ 7635], 99.95th=[ 9503], | 99.99th=[13829] bw ( KiB/s): min=235680, max=1591360, per=100.00%, avg=922538.67, stdev=632770.68, samples=60 iops : min= 2946, max=19892, avg=11531.73, stdev=7909.63, samples=60 write: IOPS=2894, BW=226MiB/s (237MB/s)(6784MiB/30001msec) slat (usec): min=7, max=309, avg=16.94, stdev= 4.90 clat (usec): min=278, max=33831, avg=1807.77, stdev=1389.82 lat (usec): min=291, max=33846, avg=1824.84, stdev=1389.75 clat percentiles (usec): | 1.00th=[ 693], 5.00th=[ 930], 10.00th=[ 1057], 20.00th=[ 1205], | 30.00th=[ 1303], 40.00th=[ 1385], 50.00th=[ 1450], 60.00th=[ 1532], | 70.00th=[ 1614], 80.00th=[ 1713], 90.00th=[ 3032], 95.00th=[ 4752], | 99.00th=[ 7767], 99.50th=[ 8979], 99.90th=[13042], 99.95th=[14615], | 99.99th=[24773] bw ( KiB/s): min=60160, max=395360, per=100.00%, avg=231552.00, stdev=158891.53, samples=60 iops : min= 752, max= 4942, avg=2894.40, stdev=1986.14, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 20=0.01%, 50=10.49%, 100=34.90%, 250=21.10% lat (usec) : 500=0.96%, 750=0.78%, 1000=2.85% lat (msec) : 2=21.60%, 4=4.96%, 10=2.26%, 20=0.09%, 50=0.01% cpu : usr=5.95%, sys=19.56%, ctx=175928, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=345959,86834,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=901MiB/s (945MB/s), 901MiB/s-901MiB/s (945MB/s-945MB/s), io=26.4GiB (28.3GB), run=30001-30001msec WRITE: bw=226MiB/s (237MB/s), 226MiB/s-226MiB/s (237MB/s-237MB/s), io=6784MiB (7113MB), run=30001-30001msec Disk stats (read/write): vdb: ios=345356/86688, merge=0/0, ticks=78923/114919, in_queue=283324, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=927536: Fri Jan 7 14:00:28 2022 read: IOPS=7097, BW=555MiB/s (581MB/s)(16.2GiB/30001msec) clat (usec): min=32, max=2938, avg=69.83, stdev=61.44 lat (usec): min=32, max=2938, avg=69.95, stdev=61.44 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 47], 10.00th=[ 48], 20.00th=[ 50], | 30.00th=[ 52], 40.00th=[ 54], 50.00th=[ 56], 60.00th=[ 59], | 70.00th=[ 62], 80.00th=[ 68], 90.00th=[ 78], 95.00th=[ 111], | 99.00th=[ 359], 99.50th=[ 433], 99.90th=[ 603], 99.95th=[ 668], | 99.99th=[ 848] bw ( KiB/s): min=507040, max=608800, per=100.00%, avg=567851.97, stdev=18865.92, samples=59 iops : min= 6338, max= 7610, avg=7098.12, stdev=235.83, samples=59 write: IOPS=1779, BW=139MiB/s (146MB/s)(4170MiB/30001msec) clat (usec): min=237, max=36163, avg=268.33, stdev=169.28 lat (usec): min=243, max=36170, avg=276.09, stdev=169.31 clat percentiles (usec): | 1.00th=[ 249], 5.00th=[ 251], 10.00th=[ 253], 20.00th=[ 258], | 30.00th=[ 260], 40.00th=[ 262], 50.00th=[ 265], 60.00th=[ 269], | 70.00th=[ 273], 80.00th=[ 277], 90.00th=[ 285], 95.00th=[ 293], | 99.00th=[ 314], 99.50th=[ 326], 99.90th=[ 478], 99.95th=[ 783], | 99.99th=[ 2024] bw ( KiB/s): min=125280, max=148960, per=99.93%, avg=142231.15, stdev=4639.47, samples=59 iops : min= 1566, max= 1862, avg=1777.83, stdev=57.97, samples=59 lat (usec) : 50=18.98%, 100=56.71%, 250=2.17%, 500=21.90%, 750=0.21% lat (usec) : 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.07%, sys=13.02%, ctx=266337, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=212946,53376,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=555MiB/s (581MB/s), 555MiB/s-555MiB/s (581MB/s-581MB/s), io=16.2GiB (17.4GB), run=30001-30001msec WRITE: bw=139MiB/s (146MB/s), 139MiB/s-139MiB/s (146MB/s-146MB/s), io=4170MiB (4373MB), run=30001-30001msec Disk stats (read/write): vdb: ios=211567/53036, merge=0/0, ticks=19/75, in_queue=24660, util=82.61% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=928848: Fri Jan 7 14:01:05 2022 read: IOPS=11.3k, BW=879MiB/s (922MB/s)(25.8GiB/30018msec) slat (usec): min=2, max=3086, avg= 9.67, stdev= 8.52 clat (nsec): min=675, max=33912k, avg=654319.83, stdev=2792115.40 lat (usec): min=25, max=33919, avg=664.12, stdev=2792.22 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 35], 10.00th=[ 38], 20.00th=[ 41], | 30.00th=[ 44], 40.00th=[ 48], 50.00th=[ 54], 60.00th=[ 61], | 70.00th=[ 70], 80.00th=[ 83], 90.00th=[ 113], 95.00th=[ 506], | 99.00th=[16188], 99.50th=[17695], 99.90th=[20055], 99.95th=[21103], | 99.99th=[23987] bw ( KiB/s): min=672320, max=1395520, per=100.00%, avg=901149.42, stdev=240669.66, samples=60 iops : min= 8404, max=17444, avg=11264.32, stdev=3008.39, samples=60 write: IOPS=2826, BW=221MiB/s (232MB/s)(6628MiB/30018msec) slat (usec): min=6, max=1016, avg=19.96, stdev=14.46 clat (usec): min=1110, max=53551, avg=15018.97, stdev=5438.01 lat (usec): min=1150, max=53565, avg=15039.08, stdev=5437.19 clat percentiles (usec): | 1.00th=[ 7177], 5.00th=[ 8848], 10.00th=[ 9634], 20.00th=[10552], | 30.00th=[11207], 40.00th=[11994], 50.00th=[13698], 60.00th=[15401], | 70.00th=[17171], 80.00th=[19268], 90.00th=[22414], 95.00th=[25297], | 99.00th=[31589], 99.50th=[34341], 99.90th=[40109], 99.95th=[42206], | 99.99th=[46924] bw ( KiB/s): min=181600, max=351040, per=100.00%, avg=226199.45, stdev=59085.50, samples=60 iops : min= 2270, max= 4388, avg=2827.42, stdev=738.59, samples=60 lat (nsec) : 750=0.01%, 1000=0.05% lat (usec) : 2=0.06%, 4=0.01%, 10=0.04%, 20=0.13%, 50=34.86% lat (usec) : 100=34.53%, 250=6.05%, 500=0.21%, 750=0.09%, 1000=0.05% lat (msec) : 2=0.08%, 4=0.16%, 10=3.65%, 20=16.43%, 50=3.60% lat (msec) : 100=0.01% cpu : usr=8.10%, sys=23.09%, ctx=298486, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=337839,84841,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=879MiB/s (922MB/s), 879MiB/s-879MiB/s (922MB/s-922MB/s), io=25.8GiB (27.7GB), run=30018-30018msec WRITE: bw=221MiB/s (232MB/s), 221MiB/s-221MiB/s (232MB/s-232MB/s), io=6628MiB (6950MB), run=30018-30018msec Disk stats (read/write): vdb: ios=336344/84426, merge=0/0, ticks=189541/1223579, in_queue=1479296, util=99.99% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=929905: Fri Jan 7 14:01:42 2022 read: IOPS=6900, BW=539MiB/s (565MB/s)(15.8GiB/30001msec) clat (usec): min=32, max=2684, avg=73.26, stdev=62.11 lat (usec): min=32, max=2685, avg=73.38, stdev=62.11 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 48], 10.00th=[ 50], 20.00th=[ 53], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 60], 60.00th=[ 63], | 70.00th=[ 66], 80.00th=[ 71], 90.00th=[ 81], 95.00th=[ 119], | 99.00th=[ 367], 99.50th=[ 441], 99.90th=[ 603], 99.95th=[ 668], | 99.99th=[ 1156] bw ( KiB/s): min=514629, max=593600, per=100.00%, avg=552391.68, stdev=19370.38, samples=59 iops : min= 6432, max= 7420, avg=6904.88, stdev=242.16, samples=59 write: IOPS=1728, BW=135MiB/s (142MB/s)(4052MiB/30001msec) clat (usec): min=235, max=2735, avg=271.32, stdev=25.81 lat (usec): min=242, max=2740, avg=278.73, stdev=25.97 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 253], 10.00th=[ 258], 20.00th=[ 260], | 30.00th=[ 265], 40.00th=[ 265], 50.00th=[ 269], 60.00th=[ 273], | 70.00th=[ 277], 80.00th=[ 281], 90.00th=[ 289], 95.00th=[ 293], | 99.00th=[ 322], 99.50th=[ 334], 99.90th=[ 449], 99.95th=[ 668], | 99.99th=[ 1303] bw ( KiB/s): min=130560, max=147680, per=100.00%, avg=138388.27, stdev=3903.65, samples=59 iops : min= 1632, max= 1846, avg=1729.85, stdev=48.80, samples=59 lat (usec) : 50=8.96%, 100=66.49%, 250=2.11%, 500=22.19%, 750=0.23% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=5.05%, sys=13.29%, ctx=258905, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=207026,51862,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=539MiB/s (565MB/s), 539MiB/s-539MiB/s (565MB/s-565MB/s), io=15.8GiB (16.0GB), run=30001-30001msec WRITE: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=4052MiB (4249MB), run=30001-30001msec Disk stats (read/write): vdb: ios=205840/51554, merge=0/0, ticks=24/11, in_queue=24182, util=80.83% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=930578: Fri Jan 7 14:02:19 2022 read: IOPS=12.2k, BW=950MiB/s (996MB/s)(27.8GiB/30011msec) slat (usec): min=2, max=413, avg= 9.23, stdev= 7.40 clat (nsec): min=606, max=60794k, avg=1794793.94, stdev=5959364.70 lat (usec): min=22, max=60975, avg=1804.14, stdev=5958.90 clat percentiles (usec): | 1.00th=[ 31], 5.00th=[ 38], 10.00th=[ 41], 20.00th=[ 47], | 30.00th=[ 53], 40.00th=[ 60], 50.00th=[ 68], 60.00th=[ 77], | 70.00th=[ 90], 80.00th=[ 112], 90.00th=[ 474], 95.00th=[16319], | 99.00th=[28705], 99.50th=[32637], 99.90th=[42730], 99.95th=[57410], | 99.99th=[60031] bw ( KiB/s): min=153440, max=1316320, per=100.00%, avg=973317.23, stdev=377057.77, samples=60 iops : min= 1918, max=16454, avg=12166.37, stdev=4713.19, samples=60 write: IOPS=3051, BW=238MiB/s (250MB/s)(7155MiB/30011msec) slat (usec): min=6, max=794, avg=23.04, stdev=14.89 clat (usec): min=1638, max=83040, avg=25546.72, stdev=9221.73 lat (usec): min=1726, max=83074, avg=25569.91, stdev=9221.93 clat percentiles (usec): | 1.00th=[ 7898], 5.00th=[ 9765], 10.00th=[12256], 20.00th=[17957], | 30.00th=[22414], 40.00th=[23987], 50.00th=[25297], 60.00th=[26870], | 70.00th=[28967], 80.00th=[32113], 90.00th=[39584], 95.00th=[41681], | 99.00th=[47973], 99.50th=[52167], 99.90th=[60031], 99.95th=[63701], | 99.99th=[78119] bw ( KiB/s): min=38560, max=316960, per=100.00%, avg=244183.80, stdev=94072.03, samples=60 iops : min= 482, max= 3962, avg=3052.27, stdev=1175.93, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.03%, 4=0.01%, 10=0.02%, 20=0.11%, 50=20.60% lat (usec) : 100=39.69%, 250=11.10%, 500=0.38%, 750=0.11%, 1000=0.08% lat (msec) : 2=0.13%, 4=0.05%, 10=2.70%, 20=6.55%, 50=18.20% lat (msec) : 100=0.22% cpu : usr=9.49%, sys=22.43%, ctx=257076, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=365043,91586,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=950MiB/s (996MB/s), 950MiB/s-950MiB/s (996MB/s-996MB/s), io=27.8GiB (29.9GB), run=30011-30011msec WRITE: bw=238MiB/s (250MB/s), 238MiB/s-238MiB/s (250MB/s-250MB/s), io=7155MiB (7503MB), run=30011-30011msec Disk stats (read/write): vdb: ios=364397/91410, merge=0/0, ticks=585119/2284316, in_queue=2954210, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=931406: Fri Jan 7 14:02:56 2022 read: IOPS=7190, BW=562MiB/s (589MB/s)(16.5GiB/30001msec) clat (usec): min=33, max=2373, avg=68.70, stdev=61.44 lat (usec): min=33, max=2373, avg=68.82, stdev=61.45 clat percentiles (usec): | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 49], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 58], | 70.00th=[ 61], 80.00th=[ 66], 90.00th=[ 76], 95.00th=[ 110], | 99.00th=[ 363], 99.50th=[ 433], 99.90th=[ 603], 99.95th=[ 660], | 99.99th=[ 816] bw ( KiB/s): min=522240, max=611040, per=99.95%, avg=574992.80, stdev=16061.72, samples=59 iops : min= 6528, max= 7638, avg=7187.36, stdev=200.78, samples=59 write: IOPS=1800, BW=141MiB/s (148MB/s)(4220MiB/30001msec) clat (usec): min=237, max=22282, avg=266.26, stdev=121.54 lat (usec): min=244, max=22289, avg=273.93, stdev=121.59 clat percentiles (usec): | 1.00th=[ 247], 5.00th=[ 251], 10.00th=[ 253], 20.00th=[ 255], | 30.00th=[ 258], 40.00th=[ 260], 50.00th=[ 265], 60.00th=[ 265], | 70.00th=[ 269], 80.00th=[ 273], 90.00th=[ 281], 95.00th=[ 289], | 99.00th=[ 314], 99.50th=[ 326], 99.90th=[ 420], 99.95th=[ 586], | 99.99th=[ 1123] bw ( KiB/s): min=134080, max=152320, per=100.00%, avg=144163.83, stdev=3826.94, samples=59 iops : min= 1676, max= 1904, avg=1802.02, stdev=47.83, samples=59 lat (usec) : 50=24.20%, 100=51.54%, 250=2.45%, 500=21.57%, 750=0.23% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=0.01% cpu : usr=5.29%, sys=12.49%, ctx=269764, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=215727,54021,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=562MiB/s (589MB/s), 562MiB/s-562MiB/s (589MB/s-589MB/s), io=16.5GiB (17.7GB), run=30001-30001msec WRITE: bw=141MiB/s (148MB/s), 141MiB/s-141MiB/s (148MB/s-148MB/s), io=4220MiB (4425MB), run=30001-30001msec Disk stats (read/write): vdb: ios=214442/53725, merge=0/0, ticks=22/43, in_queue=24756, util=82.72% 30s: (g=0): rw=randrw, bs=(R) 80.0KiB-80.0KiB, (W) 80.0KiB-80.0KiB, (T) 80.0KiB-80.0KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932348: Fri Jan 7 14:03:33 2022 read: IOPS=12.6k, BW=983MiB/s (1031MB/s)(28.8GiB/30029msec) slat (usec): min=2, max=55285, avg=59.85, stdev=543.53 clat (usec): min=65, max=95181, avg=6515.35, stdev=9310.12 lat (usec): min=1211, max=120386, avg=6575.31, stdev=9361.24 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2474], 10.00th=[ 2802], 20.00th=[ 3261], | 30.00th=[ 3621], 40.00th=[ 3949], 50.00th=[ 4293], 60.00th=[ 4686], | 70.00th=[ 5211], 80.00th=[ 6128], 90.00th=[ 8848], 95.00th=[15008], | 99.00th=[61604], 99.50th=[67634], 99.90th=[82314], 99.95th=[87557], | 99.99th=[93848] bw ( KiB/s): min=259360, max=1358720, per=100.00%, avg=1007091.22, stdev=350888.43, samples=60 iops : min= 3242, max=16984, avg=12588.63, stdev=4386.10, samples=60 write: IOPS=3157, BW=247MiB/s (259MB/s)(7407MiB/30029msec) slat (usec): min=6, max=40308, avg=70.17, stdev=480.35 clat (msec): min=4, max=153, avg=37.06, stdev=14.10 lat (msec): min=4, max=153, avg=37.14, stdev=14.12 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 18], 10.00th=[ 23], 20.00th=[ 30], | 30.00th=[ 32], 40.00th=[ 34], 50.00th=[ 35], 60.00th=[ 37], | 70.00th=[ 39], 80.00th=[ 42], 90.00th=[ 55], 95.00th=[ 69], | 99.00th=[ 88], 99.50th=[ 94], 99.90th=[ 108], 99.95th=[ 112], | 99.99th=[ 133] bw ( KiB/s): min=64160, max=339360, per=100.00%, avg=252740.37, stdev=87011.57, samples=60 iops : min= 802, max= 4242, avg=3159.23, stdev=1087.63, samples=60 lat (usec) : 100=0.01% lat (msec) : 2=0.88%, 4=32.19%, 10=39.87%, 20=5.33%, 50=17.63% lat (msec) : 100=4.06%, 250=0.04% cpu : usr=7.01%, sys=10.68%, ctx=50937, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=377795,94805,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=983MiB/s (1031MB/s), 983MiB/s-983MiB/s (1031MB/s-1031MB/s), io=28.8GiB (30.9GB), run=30029-30029msec WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=7407MiB (7766MB), run=30029-30029msec Disk stats (read/write): vdb: ios=377082/94619, merge=0/0, ticks=526177/3040974, in_queue=3686128, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=932851: Fri Jan 7 14:04:10 2022 read: IOPS=1748, BW=219MiB/s (229MB/s)(6556MiB/30002msec) clat (usec): min=48, max=10880, avg=97.59, stdev=98.62 lat (usec): min=48, max=10880, avg=97.73, stdev=98.63 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 63], 10.00th=[ 65], 20.00th=[ 68], | 30.00th=[ 71], 40.00th=[ 74], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 88], 80.00th=[ 96], 90.00th=[ 115], 95.00th=[ 237], | 99.00th=[ 469], 99.50th=[ 523], 99.90th=[ 775], 99.95th=[ 865], | 99.99th=[ 2245] bw ( KiB/s): min=74752, max=635392, per=100.00%, avg=223765.83, stdev=193445.72, samples=60 iops : min= 584, max= 4964, avg=1748.17, stdev=1511.30, samples=60 write: IOPS=439, BW=54.0MiB/s (57.7MB/s)(1650MiB/30002msec) clat (usec): min=360, max=27172, avg=1863.47, stdev=2081.55 lat (usec): min=372, max=27185, avg=1876.75, stdev=2082.38 clat percentiles (usec): | 1.00th=[ 371], 5.00th=[ 375], 10.00th=[ 379], 20.00th=[ 383], | 30.00th=[ 388], 40.00th=[ 396], 50.00th=[ 404], 60.00th=[ 1893], | 70.00th=[ 2442], 80.00th=[ 3949], 90.00th=[ 5211], 95.00th=[ 5932], | 99.00th=[ 7767], 99.50th=[ 8455], 99.90th=[ 9241], 99.95th=[ 9503], | 99.99th=[14222] bw ( KiB/s): min=19968, max=159232, per=100.00%, avg=56312.57, stdev=48816.23, samples=60 iops : min= 156, max= 1244, avg=439.93, stdev=381.38, samples=60 lat (usec) : 50=0.01%, 100=66.59%, 250=9.56%, 500=15.19%, 750=0.41% lat (usec) : 1000=0.08% lat (msec) : 2=0.33%, 4=3.88%, 10=3.94%, 20=0.01%, 50=0.01% cpu : usr=1.69%, sys=3.45%, ctx=65645, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=52444,13199,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=219MiB/s (229MB/s), 219MiB/s-219MiB/s (229MB/s-229MB/s), io=6556MiB (6874MB), run=30002-30002msec WRITE: bw=54.0MiB/s (57.7MB/s), 54.0MiB/s-54.0MiB/s (57.7MB/s-57.7MB/s), io=1650MiB (1730MB), run=30002-30002msec Disk stats (read/write): vdb: ios=52282/13151, merge=0/0, ticks=28/18822, in_queue=28296, util=94.51% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=933832: Fri Jan 7 14:04:47 2022 read: IOPS=2948, BW=369MiB/s (386MB/s)(10.8GiB/30001msec) slat (usec): min=8, max=288, avg=13.72, stdev= 4.25 clat (nsec): min=795, max=2369.7k, avg=70308.57, stdev=77119.21 lat (usec): min=40, max=2381, avg=84.17, stdev=77.38 clat percentiles (usec): | 1.00th=[ 38], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 52], 60.00th=[ 55], | 70.00th=[ 59], 80.00th=[ 64], 90.00th=[ 77], 95.00th=[ 206], | 99.00th=[ 445], 99.50th=[ 510], 99.90th=[ 750], 99.95th=[ 832], | 99.99th=[ 930] bw ( KiB/s): min=94464, max=740608, per=100.00%, avg=377437.87, stdev=223325.44, samples=60 iops : min= 738, max= 5786, avg=2948.73, stdev=1744.73, samples=60 write: IOPS=737, BW=92.1MiB/s (96.6MB/s)(2764MiB/30001msec) slat (usec): min=14, max=119, avg=26.76, stdev= 6.99 clat (usec): min=300, max=14479, avg=985.92, stdev=910.23 lat (usec): min=356, max=14507, avg=1012.84, stdev=911.49 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 351], 10.00th=[ 355], 20.00th=[ 355], | 30.00th=[ 359], 40.00th=[ 367], 50.00th=[ 371], 60.00th=[ 388], | 70.00th=[ 1778], 80.00th=[ 1876], 90.00th=[ 2024], 95.00th=[ 2245], | 99.00th=[ 3261], 99.50th=[ 4080], 99.90th=[ 7242], 99.95th=[ 8029], | 99.99th=[11600] bw ( KiB/s): min=20736, max=183808, per=100.00%, avg=94348.80, stdev=56445.50, samples=60 iops : min= 162, max= 1436, avg=737.10, stdev=440.98, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=36.28%, 100=38.66% lat (usec) : 250=1.85%, 500=15.37%, 750=0.36%, 1000=0.08% lat (msec) : 2=5.11%, 4=2.15%, 10=0.10%, 20=0.01% cpu : usr=2.60%, sys=8.51%, ctx=110562, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=88462,22114,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=369MiB/s (386MB/s), 369MiB/s-369MiB/s (386MB/s-386MB/s), io=10.8GiB (11.6GB), run=30001-30001msec WRITE: bw=92.1MiB/s (96.6MB/s), 92.1MiB/s-92.1MiB/s (96.6MB/s-96.6MB/s), io=2764MiB (2899MB), run=30001-30001msec Disk stats (read/write): vdb: ios=88318/22080, merge=0/0, ticks=5/11132, in_queue=27249, util=91.00% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=934415: Fri Jan 7 14:05:24 2022 read: IOPS=3818, BW=477MiB/s (500MB/s)(13.0GiB/30001msec) clat (usec): min=37, max=11606, avg=82.50, stdev=88.97 lat (usec): min=37, max=11607, avg=82.62, stdev=88.97 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 53], 20.00th=[ 56], | 30.00th=[ 58], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 67], | 70.00th=[ 71], 80.00th=[ 76], 90.00th=[ 89], 95.00th=[ 223], | 99.00th=[ 465], 99.50th=[ 545], 99.90th=[ 832], 99.95th=[ 889], | 99.99th=[ 1516] bw ( KiB/s): min=80640, max=738304, per=99.99%, avg=488722.08, stdev=229061.60, samples=60 iops : min= 630, max= 5768, avg=3818.12, stdev=1789.53, samples=60 write: IOPS=952, BW=119MiB/s (125MB/s)(3571MiB/30001msec) clat (usec): min=348, max=21802, avg=698.86, stdev=693.51 lat (usec): min=358, max=21814, avg=711.85, stdev=693.41 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 363], 10.00th=[ 367], 20.00th=[ 371], | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 379], 60.00th=[ 383], | 70.00th=[ 396], 80.00th=[ 1549], 90.00th=[ 1729], 95.00th=[ 1909], | 99.00th=[ 2835], 99.50th=[ 3392], 99.90th=[ 6063], 99.95th=[ 6521], | 99.99th=[ 8225] bw ( KiB/s): min=24064, max=188160, per=99.99%, avg=121856.20, stdev=57755.32, samples=60 iops : min= 188, max= 1470, avg=951.97, stdev=451.21, samples=60 lat (usec) : 50=1.45%, 100=72.63%, 250=2.48%, 500=18.64%, 750=0.49% lat (usec) : 1000=0.11% lat (msec) : 2=3.36%, 4=0.76%, 10=0.07%, 20=0.01%, 50=0.01% cpu : usr=3.55%, sys=7.19%, ctx=143134, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=114558,28564,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=477MiB/s (500MB/s), 477MiB/s-477MiB/s (500MB/s-500MB/s), io=13.0GiB (15.0GB), run=30001-30001msec WRITE: bw=119MiB/s (125MB/s), 119MiB/s-119MiB/s (125MB/s-125MB/s), io=3571MiB (3744MB), run=30001-30001msec Disk stats (read/write): vdb: ios=114240/28485, merge=0/0, ticks=34/7471, in_queue=26535, util=88.66% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=935716: Fri Jan 7 14:06:01 2022 read: IOPS=5702, BW=713MiB/s (747MB/s)(20.9GiB/30003msec) slat (usec): min=4, max=122, avg=10.13, stdev= 3.86 clat (nsec): min=728, max=26354k, avg=883305.00, stdev=1534914.04 lat (usec): min=30, max=26362, avg=893.54, stdev=1534.21 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 42], 10.00th=[ 44], 20.00th=[ 50], | 30.00th=[ 58], 40.00th=[ 70], 50.00th=[ 86], 60.00th=[ 119], | 70.00th=[ 363], 80.00th=[ 1975], 90.00th=[ 2933], 95.00th=[ 3851], | 99.00th=[ 6128], 99.50th=[ 7111], 99.90th=[12387], 99.95th=[16909], | 99.99th=[24511] bw ( KiB/s): min=121856, max=1603840, per=100.00%, avg=729932.25, stdev=563887.14, samples=60 iops : min= 952, max=12530, avg=5702.62, stdev=4405.36, samples=60 write: IOPS=1425, BW=178MiB/s (187MB/s)(5345MiB/30003msec) slat (usec): min=10, max=4638, avg=23.67, stdev=26.65 clat (usec): min=391, max=46931, avg=3412.41, stdev=2304.30 lat (usec): min=424, max=46962, avg=3436.22, stdev=2306.01 clat percentiles (usec): | 1.00th=[ 1467], 5.00th=[ 1811], 10.00th=[ 1991], 20.00th=[ 2180], | 30.00th=[ 2311], 40.00th=[ 2442], 50.00th=[ 2540], 60.00th=[ 2671], | 70.00th=[ 2900], 80.00th=[ 4555], 90.00th=[ 6390], 95.00th=[ 7767], | 99.00th=[10814], 99.50th=[12518], 99.90th=[27132], 99.95th=[32375], | 99.99th=[39584] bw ( KiB/s): min=32768, max=399104, per=100.00%, avg=182437.13, stdev=140059.34, samples=60 iops : min= 256, max= 3118, avg=1425.28, stdev=1094.22, samples=60 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=15.88%, 100=28.89% lat (usec) : 250=9.44%, 500=2.05%, 750=0.03%, 1000=0.01% lat (msec) : 2=9.88%, 4=25.56%, 10=7.83%, 20=0.35%, 50=0.07% cpu : usr=4.19%, sys=10.55%, ctx=107768, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=171084,42760,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=713MiB/s (747MB/s), 713MiB/s-713MiB/s (747MB/s-747MB/s), io=20.9GiB (22.4GB), run=30003-30003msec WRITE: bw=178MiB/s (187MB/s), 178MiB/s-178MiB/s (187MB/s-187MB/s), io=5345MiB (5605MB), run=30003-30003msec Disk stats (read/write): vdb: ios=170723/42666, merge=0/0, ticks=115810/123350, in_queue=289686, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=936630: Fri Jan 7 14:06:38 2022 read: IOPS=5444, BW=681MiB/s (714MB/s)(19.9GiB/30001msec) clat (usec): min=39, max=2789, avg=83.62, stdev=82.85 lat (usec): min=39, max=2789, avg=83.74, stdev=82.85 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 58], | 30.00th=[ 60], 40.00th=[ 62], 50.00th=[ 64], 60.00th=[ 68], | 70.00th=[ 71], 80.00th=[ 77], 90.00th=[ 88], 95.00th=[ 225], | 99.00th=[ 478], 99.50th=[ 562], 99.90th=[ 832], 99.95th=[ 881], | 99.99th=[ 1221] bw ( KiB/s): min=639488, max=762624, per=100.00%, avg=697071.51, stdev=25488.64, samples=59 iops : min= 4996, max= 5958, avg=5445.86, stdev=199.14, samples=59 write: IOPS=1361, BW=170MiB/s (179MB/s)(5107MiB/30001msec) clat (usec): min=346, max=26380, avg=380.42, stdev=134.21 lat (usec): min=353, max=26394, avg=392.55, stdev=134.27 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 363], 10.00th=[ 367], 20.00th=[ 367], | 30.00th=[ 371], 40.00th=[ 375], 50.00th=[ 375], 60.00th=[ 379], | 70.00th=[ 383], 80.00th=[ 388], 90.00th=[ 396], 95.00th=[ 404], | 99.00th=[ 424], 99.50th=[ 437], 99.90th=[ 529], 99.95th=[ 1004], | 99.99th=[ 1647] bw ( KiB/s): min=151808, max=188160, per=100.00%, avg=174324.76, stdev=5978.42, samples=59 iops : min= 1186, max= 1470, avg=1361.90, stdev=46.69, samples=59 lat (usec) : 50=0.95%, 100=73.35%, 250=2.19%, 500=22.81%, 750=0.54% lat (usec) : 1000=0.13% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=4.92%, sys=10.55%, ctx=204222, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=163347,40857,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=681MiB/s (714MB/s), 681MiB/s-681MiB/s (714MB/s-714MB/s), io=19.9GiB (21.4GB), run=30001-30001msec WRITE: bw=170MiB/s (179MB/s), 170MiB/s-170MiB/s (179MB/s-179MB/s), io=5107MiB (5355MB), run=30001-30001msec Disk stats (read/write): vdb: ios=162272/40598, merge=0/0, ticks=27/45, in_queue=25131, util=83.86% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=937323: Fri Jan 7 14:07:15 2022 read: IOPS=6674, BW=834MiB/s (875MB/s)(24.4GiB/30004msec) slat (usec): min=3, max=372, avg=10.96, stdev= 8.14 clat (nsec): min=680, max=46621k, avg=2635529.13, stdev=5604158.70 lat (usec): min=30, max=46631, avg=2646.64, stdev=5603.39 clat percentiles (usec): | 1.00th=[ 32], 5.00th=[ 40], 10.00th=[ 44], 20.00th=[ 49], | 30.00th=[ 56], 40.00th=[ 63], 50.00th=[ 72], 60.00th=[ 85], | 70.00th=[ 108], 80.00th=[ 293], 90.00th=[13042], 95.00th=[16057], | 99.00th=[20579], 99.50th=[22414], 99.90th=[28181], 99.95th=[34866], | 99.99th=[44303] bw ( KiB/s): min=239104, max=1411328, per=100.00%, avg=854276.57, stdev=391475.39, samples=60 iops : min= 1868, max=11026, avg=6674.07, stdev=3058.43, samples=60 write: IOPS=1671, BW=209MiB/s (219MB/s)(6268MiB/30004msec) slat (usec): min=9, max=908, avg=33.54, stdev=17.76 clat (usec): min=1888, max=86093, avg=19308.14, stdev=5215.49 lat (usec): min=1923, max=86137, avg=19341.86, stdev=5216.03 clat percentiles (usec): | 1.00th=[10814], 5.00th=[13042], 10.00th=[14222], 20.00th=[15533], | 30.00th=[16581], 40.00th=[17433], 50.00th=[18482], 60.00th=[19530], | 70.00th=[20579], 80.00th=[22152], 90.00th=[25560], 95.00th=[28967], | 99.00th=[37487], 99.50th=[40633], 99.90th=[50594], 99.95th=[58459], | 99.99th=[85459] bw ( KiB/s): min=66560, max=354304, per=99.99%, avg=213883.93, stdev=97110.18, samples=60 iops : min= 520, max= 2768, avg=1670.97, stdev=758.68, samples=60 lat (nsec) : 750=0.01%, 1000=0.04% lat (usec) : 2=0.07%, 4=0.01%, 10=0.03%, 20=0.09%, 50=17.03% lat (usec) : 100=36.77%, 250=9.82%, 500=0.31%, 750=0.08%, 1000=0.06% lat (msec) : 2=0.10%, 4=0.17%, 10=4.05%, 20=23.33%, 50=8.04% lat (msec) : 100=0.02% cpu : usr=7.13%, sys=13.78%, ctx=158310, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=200251,50140,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=834MiB/s (875MB/s), 834MiB/s-834MiB/s (875MB/s-875MB/s), io=24.4GiB (26.2GB), run=30004-30004msec WRITE: bw=209MiB/s (219MB/s), 209MiB/s-209MiB/s (219MB/s-219MB/s), io=6268MiB (6572MB), run=30004-30004msec Disk stats (read/write): vdb: ios=199730/50019, merge=0/0, ticks=481569/936028, in_queue=1471030, util=99.73% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938222: Fri Jan 7 14:07:52 2022 read: IOPS=5577, BW=697MiB/s (731MB/s)(20.4GiB/30001msec) clat (usec): min=38, max=2664, avg=80.32, stdev=83.17 lat (usec): min=38, max=2665, avg=80.44, stdev=83.17 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 55], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 64], | 70.00th=[ 68], 80.00th=[ 73], 90.00th=[ 86], 95.00th=[ 219], | 99.00th=[ 482], 99.50th=[ 570], 99.90th=[ 832], 99.95th=[ 873], | 99.99th=[ 1221] bw ( KiB/s): min=660480, max=785152, per=99.98%, avg=713819.12, stdev=23954.38, samples=59 iops : min= 5160, max= 6134, avg=5576.75, stdev=187.20, samples=59 write: IOPS=1395, BW=174MiB/s (183MB/s)(5232MiB/30001msec) clat (usec): min=348, max=20929, avg=375.96, stdev=111.56 lat (usec): min=358, max=20942, avg=388.47, stdev=111.66 clat percentiles (usec): | 1.00th=[ 359], 5.00th=[ 363], 10.00th=[ 363], 20.00th=[ 367], | 30.00th=[ 367], 40.00th=[ 371], 50.00th=[ 371], 60.00th=[ 375], | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 392], 95.00th=[ 400], | 99.00th=[ 416], 99.50th=[ 429], 99.90th=[ 457], 99.95th=[ 627], | 99.99th=[ 1369] bw ( KiB/s): min=163840, max=187392, per=100.00%, avg=178583.86, stdev=4613.91, samples=59 iops : min= 1280, max= 1464, avg=1395.19, stdev=36.05, samples=59 lat (usec) : 50=2.30%, 100=72.31%, 250=1.95%, 500=22.74%, 750=0.55% lat (usec) : 1000=0.14% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01% cpu : usr=4.84%, sys=10.14%, ctx=209217, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=167343,41858,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=697MiB/s (731MB/s), 697MiB/s-697MiB/s (731MB/s-731MB/s), io=20.4GiB (21.9GB), run=30001-30001msec WRITE: bw=174MiB/s (183MB/s), 174MiB/s-174MiB/s (183MB/s-183MB/s), io=5232MiB (5486MB), run=30001-30001msec Disk stats (read/write): vdb: ios=166462/41641, merge=0/0, ticks=28/35, in_queue=25492, util=85.13% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=938824: Fri Jan 7 14:08:29 2022 read: IOPS=7332, BW=917MiB/s (961MB/s)(26.9GiB/30003msec) slat (usec): min=3, max=363, avg=10.28, stdev= 9.52 clat (nsec): min=689, max=77009k, avg=3434806.59, stdev=8707596.17 lat (usec): min=32, max=77022, avg=3445.22, stdev=8706.85 clat percentiles (usec): | 1.00th=[ 35], 5.00th=[ 41], 10.00th=[ 43], 20.00th=[ 48], | 30.00th=[ 55], 40.00th=[ 64], 50.00th=[ 74], 60.00th=[ 84], | 70.00th=[ 101], 80.00th=[ 139], 90.00th=[18220], 95.00th=[27132], | 99.00th=[34866], 99.50th=[36963], 99.90th=[42730], 99.95th=[44827], | 99.99th=[53216] bw ( KiB/s): min=300288, max=1418752, per=99.98%, avg=938298.45, stdev=363502.13, samples=60 iops : min= 2346, max=11084, avg=7330.45, stdev=2839.86, samples=60 write: IOPS=1834, BW=229MiB/s (240MB/s)(6881MiB/30003msec) slat (usec): min=8, max=497, avg=33.86, stdev=19.48 clat (msec): min=5, max=124, avg=40.67, stdev=13.20 lat (msec): min=5, max=124, avg=40.70, stdev=13.20 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 17], 10.00th=[ 23], 20.00th=[ 32], | 30.00th=[ 36], 40.00th=[ 39], 50.00th=[ 41], 60.00th=[ 44], | 70.00th=[ 46], 80.00th=[ 50], 90.00th=[ 61], 95.00th=[ 66], | 99.00th=[ 72], 99.50th=[ 74], 99.90th=[ 87], 99.95th=[ 101], | 99.99th=[ 121] bw ( KiB/s): min=83712, max=344064, per=99.98%, avg=234808.55, stdev=90509.78, samples=60 iops : min= 654, max= 2688, avg=1834.43, stdev=707.10, samples=60 lat (nsec) : 750=0.01%, 1000=0.03% lat (usec) : 2=0.04%, 4=0.01%, 10=0.01%, 20=0.07%, 50=18.45% lat (usec) : 100=37.22%, 250=11.75%, 500=0.47%, 750=0.13%, 1000=0.08% lat (msec) : 2=0.18%, 4=0.15%, 10=0.14%, 20=5.53%, 50=22.19% lat (msec) : 100=3.55%, 250=0.01% cpu : usr=7.99%, sys=15.16%, ctx=183470, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=219986,55050,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=917MiB/s (961MB/s), 917MiB/s-917MiB/s (961MB/s-961MB/s), io=26.9GiB (28.8GB), run=30003-30003msec WRITE: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6881MiB (7216MB), run=30003-30003msec Disk stats (read/write): vdb: ios=219514/54922, merge=0/0, ticks=696678/2200926, in_queue=2953194, util=99.65% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=939505: Fri Jan 7 14:09:06 2022 read: IOPS=5572, BW=697MiB/s (730MB/s)(20.4GiB/30001msec) clat (usec): min=37, max=2621, avg=80.37, stdev=83.27 lat (usec): min=37, max=2621, avg=80.49, stdev=83.28 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 51], 10.00th=[ 52], 20.00th=[ 54], | 30.00th=[ 56], 40.00th=[ 58], 50.00th=[ 61], 60.00th=[ 63], | 70.00th=[ 68], 80.00th=[ 74], 90.00th=[ 87], 95.00th=[ 221], | 99.00th=[ 478], 99.50th=[ 562], 99.90th=[ 840], 99.95th=[ 898], | 99.99th=[ 1369] bw ( KiB/s): min=645888, max=774400, per=100.00%, avg=713254.40, stdev=25006.30, samples=60 iops : min= 5046, max= 6050, avg=5572.30, stdev=195.36, samples=60 write: IOPS=1393, BW=174MiB/s (183MB/s)(5228MiB/30001msec) clat (usec): min=345, max=19671, avg=376.39, stdev=108.43 lat (usec): min=354, max=19677, avg=388.95, stdev=108.51 clat percentiles (usec): | 1.00th=[ 355], 5.00th=[ 359], 10.00th=[ 363], 20.00th=[ 363], | 30.00th=[ 367], 40.00th=[ 371], 50.00th=[ 371], 60.00th=[ 375], | 70.00th=[ 379], 80.00th=[ 383], 90.00th=[ 392], 95.00th=[ 400], | 99.00th=[ 424], 99.50th=[ 441], 99.90th=[ 523], 99.95th=[ 725], | 99.99th=[ 1958] bw ( KiB/s): min=169728, max=187904, per=100.00%, avg=178432.00, stdev=3980.19, samples=60 iops : min= 1326, max= 1468, avg=1394.00, stdev=31.10, samples=60 lat (usec) : 50=2.67%, 100=71.73%, 250=2.11%, 500=22.81%, 750=0.53% lat (usec) : 1000=0.13% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=5.19%, sys=9.98%, ctx=209009, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=167170,41820,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=697MiB/s (730MB/s), 697MiB/s-697MiB/s (730MB/s-730MB/s), io=20.4GiB (21.9GB), run=30001-30001msec WRITE: bw=174MiB/s (183MB/s), 174MiB/s-174MiB/s (183MB/s-183MB/s), io=5228MiB (5481MB), run=30001-30001msec Disk stats (read/write): vdb: ios=166423/41629, merge=0/0, ticks=36/40, in_queue=25335, util=84.52% 30s: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=940610: Fri Jan 7 14:09:43 2022 read: IOPS=7865, BW=983MiB/s (1031MB/s)(28.8GiB/30006msec) slat (usec): min=2, max=73481, avg=94.79, stdev=952.72 clat (usec): min=1494, max=128038, avg=11172.92, stdev=14906.02 lat (usec): min=1497, max=171768, avg=11267.83, stdev=15006.17 clat percentiles (msec): | 1.00th=[ 4], 5.00th=[ 4], 10.00th=[ 5], 20.00th=[ 6], | 30.00th=[ 6], 40.00th=[ 7], 50.00th=[ 7], 60.00th=[ 8], | 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 22], 95.00th=[ 40], | 99.00th=[ 86], 99.50th=[ 94], 99.90th=[ 110], 99.95th=[ 121], | 99.99th=[ 128] bw ( KiB/s): min=262144, max=1447424, per=99.96%, avg=1006386.70, stdev=361107.36, samples=60 iops : min= 2048, max=11308, avg=7862.38, stdev=2821.17, samples=60 write: IOPS=1970, BW=246MiB/s (258MB/s)(7391MiB/30006msec) slat (usec): min=8, max=59426, avg=120.07, stdev=903.45 clat (msec): min=2, max=137, avg=56.39, stdev=14.16 lat (msec): min=8, max=157, avg=56.51, stdev=14.21 clat percentiles (msec): | 1.00th=[ 21], 5.00th=[ 29], 10.00th=[ 41], 20.00th=[ 48], | 30.00th=[ 51], 40.00th=[ 53], 50.00th=[ 56], 60.00th=[ 60], | 70.00th=[ 63], 80.00th=[ 66], 90.00th=[ 72], 95.00th=[ 80], | 99.00th=[ 96], 99.50th=[ 104], 99.90th=[ 120], 99.95th=[ 122], | 99.99th=[ 126] bw ( KiB/s): min=66048, max=350976, per=99.95%, avg=252077.67, stdev=89486.28, samples=60 iops : min= 516, max= 2742, avg=1969.35, stdev=699.12, samples=60 lat (msec) : 2=0.05%, 4=4.03%, 10=61.77%, 20=5.80%, 50=11.10% lat (msec) : 100=16.93%, 250=0.32% cpu : usr=7.52%, sys=7.30%, ctx=30581, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=236025,59124,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=983MiB/s (1031MB/s), 983MiB/s-983MiB/s (1031MB/s-1031MB/s), io=28.8GiB (30.9GB), run=30006-30006msec WRITE: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=7391MiB (7750MB), run=30006-30006msec Disk stats (read/write): vdb: ios=235517/58990, merge=0/0, ticks=777524/2806546, in_queue=3687088, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=941230: Fri Jan 7 14:10:20 2022 read: IOPS=1461, BW=365MiB/s (383MB/s)(10.7GiB/30006msec) clat (usec): min=70, max=6057, avg=138.18, stdev=129.41 lat (usec): min=71, max=6057, avg=138.33, stdev=129.42 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 84], 10.00th=[ 87], 20.00th=[ 90], | 30.00th=[ 94], 40.00th=[ 98], 50.00th=[ 103], 60.00th=[ 110], | 70.00th=[ 118], 80.00th=[ 130], 90.00th=[ 172], 95.00th=[ 363], | 99.00th=[ 742], 99.50th=[ 832], 99.90th=[ 1188], 99.95th=[ 1319], | 99.99th=[ 1647] bw ( KiB/s): min=117760, max=868352, per=100.00%, avg=374139.67, stdev=270471.68, samples=60 iops : min= 460, max= 3392, avg=1461.47, stdev=1056.52, samples=60 write: IOPS=368, BW=92.1MiB/s (96.6MB/s)(2765MiB/30006msec) clat (usec): min=660, max=34836, avg=2128.49, stdev=2339.71 lat (usec): min=674, max=34861, avg=2156.68, stdev=2340.59 clat percentiles (usec): | 1.00th=[ 668], 5.00th=[ 676], 10.00th=[ 676], 20.00th=[ 685], | 30.00th=[ 693], 40.00th=[ 693], 50.00th=[ 701], 60.00th=[ 717], | 70.00th=[ 2606], 80.00th=[ 4621], 90.00th=[ 5997], 95.00th=[ 6783], | 99.00th=[ 8586], 99.50th=[ 9110], 99.90th=[11731], 99.95th=[15664], | 99.99th=[25297] bw ( KiB/s): min=33280, max=204288, per=100.00%, avg=94379.18, stdev=68253.82, samples=60 iops : min= 130, max= 798, avg=368.65, stdev=266.62, samples=60 lat (usec) : 100=35.57%, 250=37.83%, 500=4.08%, 750=15.16%, 1000=0.72% lat (msec) : 2=0.20%, 4=1.75%, 10=4.64%, 20=0.03%, 50=0.01% cpu : usr=1.98%, sys=3.36%, ctx=54908, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=43840,11060,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=365MiB/s (383MB/s), 365MiB/s-365MiB/s (383MB/s-383MB/s), io=10.7GiB (11.5GB), run=30006-30006msec WRITE: bw=92.1MiB/s (96.6MB/s), 92.1MiB/s-92.1MiB/s (96.6MB/s-96.6MB/s), io=2765MiB (2899MB), run=30006-30006msec Disk stats (read/write): vdb: ios=43720/11036, merge=0/0, ticks=91/16286, in_queue=28187, util=94.21% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=942441: Fri Jan 7 14:10:57 2022 read: IOPS=2118, BW=530MiB/s (555MB/s)(15.5GiB/30001msec) slat (usec): min=12, max=201, avg=19.44, stdev= 5.09 clat (usec): min=2, max=2657, avg=113.99, stdev=125.80 lat (usec): min=67, max=2673, avg=133.57, stdev=126.09 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 66], 10.00th=[ 67], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 79], 60.00th=[ 84], | 70.00th=[ 90], 80.00th=[ 100], 90.00th=[ 137], 95.00th=[ 343], | 99.00th=[ 717], 99.50th=[ 799], 99.90th=[ 1221], 99.95th=[ 1369], | 99.99th=[ 1582] bw ( KiB/s): min=214016, max=872448, per=100.00%, avg=542380.72, stdev=255351.81, samples=60 iops : min= 836, max= 3408, avg=2118.67, stdev=997.48, samples=60 write: IOPS=532, BW=133MiB/s (140MB/s)(3994MiB/30001msec) slat (usec): min=23, max=305, avg=43.40, stdev= 9.20 clat (usec): min=602, max=20646, avg=1295.72, stdev=1062.87 lat (usec): min=670, max=20690, avg=1339.30, stdev=1064.16 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 660], 20.00th=[ 660], | 30.00th=[ 668], 40.00th=[ 668], 50.00th=[ 676], 60.00th=[ 685], | 70.00th=[ 766], 80.00th=[ 2376], 90.00th=[ 2671], 95.00th=[ 3261], | 99.00th=[ 4359], 99.50th=[ 4817], 99.90th=[ 8455], 99.95th=[ 9634], | 99.99th=[15270] bw ( KiB/s): min=56832, max=218624, per=100.00%, avg=136322.73, stdev=64685.92, samples=60 iops : min= 222, max= 854, avg=532.50, stdev=252.69, samples=60 lat (usec) : 4=0.01%, 20=0.01%, 50=0.04%, 100=64.13%, 250=9.80% lat (usec) : 500=3.69%, 750=15.70%, 1000=0.48% lat (msec) : 2=0.19%, 4=5.58%, 10=0.38%, 20=0.01%, 50=0.01% cpu : usr=2.57%, sys=7.51%, ctx=79553, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=63559,15976,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=530MiB/s (555MB/s), 530MiB/s-530MiB/s (555MB/s-555MB/s), io=15.5GiB (16.7GB), run=30001-30001msec WRITE: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=3994MiB (4188MB), run=30001-30001msec Disk stats (read/write): vdb: ios=63436/15934, merge=0/0, ticks=125/10932, in_queue=27410, util=91.54% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943444: Fri Jan 7 14:11:34 2022 read: IOPS=2650, BW=663MiB/s (695MB/s)(19.4GiB/30004msec) clat (usec): min=52, max=2982, avg=113.41, stdev=130.45 lat (usec): min=52, max=2982, avg=113.54, stdev=130.45 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 66], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 87], 80.00th=[ 95], 90.00th=[ 130], 95.00th=[ 347], | 99.00th=[ 734], 99.50th=[ 840], 99.90th=[ 1270], 99.95th=[ 1418], | 99.99th=[ 1680] bw ( KiB/s): min=230400, max=955392, per=100.00%, avg=678646.40, stdev=260888.44, samples=60 iops : min= 900, max= 3732, avg=2650.93, stdev=1019.10, samples=60 write: IOPS=662, BW=166MiB/s (174MB/s)(4970MiB/30004msec) clat (usec): min=637, max=15503, avg=1023.15, stdev=835.79 lat (usec): min=654, max=15526, avg=1047.90, stdev=836.08 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 668], 50.00th=[ 668], 60.00th=[ 676], | 70.00th=[ 685], 80.00th=[ 709], 90.00th=[ 2245], 95.00th=[ 2540], | 99.00th=[ 3621], 99.50th=[ 4359], 99.90th=[ 7963], 99.95th=[ 9372], | 99.99th=[13173] bw ( KiB/s): min=53248, max=233472, per=100.00%, avg=169627.53, stdev=66670.28, samples=60 iops : min= 208, max= 912, avg=662.58, stdev=260.44, samples=60 lat (usec) : 100=67.08%, 250=6.92%, 500=3.66%, 750=17.75%, 1000=0.56% lat (msec) : 2=0.27%, 4=3.63%, 10=0.12%, 20=0.01% cpu : usr=3.14%, sys=5.76%, ctx=99424, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=79524,19878,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=663MiB/s (695MB/s), 663MiB/s-663MiB/s (695MB/s-695MB/s), io=19.4GiB (20.8GB), run=30004-30004msec WRITE: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4970MiB (5211MB), run=30004-30004msec Disk stats (read/write): vdb: ios=79387/19844, merge=0/0, ticks=193/8221, in_queue=27147, util=90.57% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=943992: Fri Jan 7 14:12:11 2022 read: IOPS=3481, BW=870MiB/s (913MB/s)(25.5GiB/30015msec) slat (usec): min=7, max=1357, avg=15.00, stdev= 6.92 clat (nsec): min=819, max=38619k, avg=1220687.98, stdev=2262309.77 lat (usec): min=49, max=38654, avg=1235.82, stdev=2261.81 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 51], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 62], 40.00th=[ 69], 50.00th=[ 79], 60.00th=[ 105], | 70.00th=[ 277], 80.00th=[ 2343], 90.00th=[ 4752], 95.00th=[ 6390], | 99.00th=[ 8848], 99.50th=[ 9765], 99.90th=[14484], 99.95th=[17695], | 99.99th=[25822] bw ( KiB/s): min=362496, max=1357824, per=100.00%, avg=891726.27, stdev=331804.68, samples=60 iops : min= 1416, max= 5304, avg=3483.30, stdev=1296.13, samples=60 write: IOPS=871, BW=218MiB/s (228MB/s)(6537MiB/30015msec) slat (usec): min=18, max=2043, avg=43.42, stdev=18.92 clat (usec): min=763, max=52749, avg=6490.24, stdev=2359.40 lat (usec): min=807, max=52851, avg=6533.82, stdev=2360.39 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3654], 10.00th=[ 4146], 20.00th=[ 4686], | 30.00th=[ 5211], 40.00th=[ 5669], 50.00th=[ 6194], 60.00th=[ 6718], | 70.00th=[ 7242], 80.00th=[ 7963], 90.00th=[ 8979], 95.00th=[10028], | 99.00th=[13698], 99.50th=[16712], 99.90th=[27132], 99.95th=[33817], | 99.99th=[39060] bw ( KiB/s): min=79360, max=329216, per=100.00%, avg=223107.55, stdev=81677.79, samples=60 iops : min= 310, max= 1286, avg=871.50, stdev=319.04, samples=60 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=2.91%, 100=43.92% lat (usec) : 250=8.91%, 500=2.68%, 750=0.70%, 1000=0.11% lat (msec) : 2=3.43%, 4=8.56%, 10=27.38%, 20=1.30%, 50=0.08% lat (msec) : 100=0.01% cpu : usr=4.24%, sys=8.94%, ctx=88990, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=104490,26148,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=870MiB/s (913MB/s), 870MiB/s-870MiB/s (913MB/s-913MB/s), io=25.5GiB (27.4GB), run=30015-30015msec WRITE: bw=218MiB/s (228MB/s), 218MiB/s-218MiB/s (228MB/s-228MB/s), io=6537MiB (6855MB), run=30015-30015msec Disk stats (read/write): vdb: ios=104314/26102, merge=0/0, ticks=103139/155535, in_queue=291751, util=99.72% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=944969: Fri Jan 7 14:12:48 2022 read: IOPS=3487, BW=872MiB/s (914MB/s)(25.5GiB/30002msec) clat (usec): min=47, max=2687, avg=110.72, stdev=134.55 lat (usec): min=47, max=2687, avg=110.84, stdev=134.55 clat percentiles (usec): | 1.00th=[ 57], 5.00th=[ 62], 10.00th=[ 64], 20.00th=[ 68], | 30.00th=[ 70], 40.00th=[ 73], 50.00th=[ 76], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 90], 90.00th=[ 119], 95.00th=[ 347], | 99.00th=[ 758], 99.50th=[ 881], 99.90th=[ 1352], 99.95th=[ 1467], | 99.99th=[ 1795] bw ( KiB/s): min=809984, max=983552, per=100.00%, avg=892823.20, stdev=37308.88, samples=60 iops : min= 3164, max= 3842, avg=3487.58, stdev=145.75, samples=60 write: IOPS=872, BW=218MiB/s (229MB/s)(6546MiB/30002msec) clat (usec): min=633, max=18035, avg=671.10, stdev=151.66 lat (usec): min=645, max=18074, avg=695.39, stdev=151.91 clat percentiles (usec): | 1.00th=[ 644], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[ 668], 60.00th=[ 668], | 70.00th=[ 668], 80.00th=[ 676], 90.00th=[ 685], 95.00th=[ 693], | 99.00th=[ 717], 99.50th=[ 734], 99.90th=[ 2073], 99.95th=[ 2573], | 99.99th=[ 8979] bw ( KiB/s): min=194949, max=237568, per=100.00%, avg=223442.35, stdev=7533.44, samples=60 iops : min= 761, max= 928, avg=872.80, stdev=29.45, samples=60 lat (usec) : 50=0.01%, 100=69.58%, 250=4.50%, 500=3.48%, 750=21.52% lat (usec) : 1000=0.61% lat (msec) : 2=0.27%, 4=0.03%, 10=0.01%, 20=0.01% cpu : usr=4.71%, sys=5.92%, ctx=130819, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=104621,26184,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=872MiB/s (914MB/s), 872MiB/s-872MiB/s (914MB/s-914MB/s), io=25.5GiB (27.4GB), run=30002-30002msec WRITE: bw=218MiB/s (229MB/s), 218MiB/s-218MiB/s (229MB/s-229MB/s), io=6546MiB (6864MB), run=30002-30002msec Disk stats (read/write): vdb: ios=104234/26085, merge=0/0, ticks=318/50, in_queue=26410, util=88.27% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=945597: Fri Jan 7 14:13:25 2022 read: IOPS=3959, BW=990MiB/s (1038MB/s)(29.0GiB/30008msec) slat (usec): min=3, max=226, avg=12.63, stdev= 7.73 clat (nsec): min=1018, max=68449k, avg=4206029.70, stdev=9232282.99 lat (usec): min=47, max=68466, avg=4218.79, stdev=9231.86 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 53], 10.00th=[ 55], 20.00th=[ 59], | 30.00th=[ 66], 40.00th=[ 79], 50.00th=[ 106], 60.00th=[ 122], | 70.00th=[ 159], 80.00th=[ 1631], 90.00th=[18744], 95.00th=[27132], | 99.00th=[37487], 99.50th=[41681], 99.90th=[50070], 99.95th=[54789], | 99.99th=[58459] bw ( KiB/s): min=395776, max=1343615, per=100.00%, avg=1013683.88, stdev=263584.37, samples=60 iops : min= 1546, max= 5248, avg=3959.68, stdev=1029.62, samples=60 write: IOPS=987, BW=247MiB/s (259MB/s)(7406MiB/30008msec) slat (usec): min=16, max=321, avg=54.90, stdev=24.50 clat (usec): min=4043, max=94603, avg=33665.96, stdev=8315.14 lat (usec): min=4093, max=94649, avg=33721.04, stdev=8314.96 clat percentiles (usec): | 1.00th=[16712], 5.00th=[23200], 10.00th=[25297], 20.00th=[27395], | 30.00th=[29230], 40.00th=[30802], 50.00th=[32637], 60.00th=[34866], | 70.00th=[36439], 80.00th=[39060], 90.00th=[42730], 95.00th=[46924], | 99.00th=[63701], 99.50th=[68682], 99.90th=[81265], 99.95th=[86508], | 99.99th=[93848] bw ( KiB/s): min=100352, max=340480, per=99.98%, avg=252649.00, stdev=66760.65, samples=60 iops : min= 392, max= 1330, avg=986.90, stdev=260.77, samples=60 lat (usec) : 2=0.01%, 4=0.01%, 20=0.01%, 50=1.57%, 100=36.45% lat (usec) : 250=23.42%, 500=2.22%, 750=0.16%, 1000=0.08% lat (msec) : 2=0.15%, 4=0.19%, 10=1.28%, 20=7.67%, 50=25.98% lat (msec) : 100=0.81% cpu : usr=6.19%, sys=9.63%, ctx=105738, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=118810,29622,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=990MiB/s (1038MB/s), 990MiB/s-990MiB/s (1038MB/s-1038MB/s), io=29.0GiB (31.1GB), run=30008-30008msec WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=7406MiB (7765MB), run=30008-30008msec Disk stats (read/write): vdb: ios=118322/29490, merge=0/0, ticks=465665/976937, in_queue=1478500, util=99.70% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=946483: Fri Jan 7 14:14:02 2022 read: IOPS=3459, BW=865MiB/s (907MB/s)(25.3GiB/30001msec) clat (usec): min=52, max=2836, avg=113.40, stdev=134.07 lat (usec): min=52, max=2836, avg=113.54, stdev=134.08 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 72], 40.00th=[ 74], 50.00th=[ 77], 60.00th=[ 81], | 70.00th=[ 85], 80.00th=[ 93], 90.00th=[ 131], 95.00th=[ 347], | 99.00th=[ 750], 99.50th=[ 881], 99.90th=[ 1336], 99.95th=[ 1450], | 99.99th=[ 2024] bw ( KiB/s): min=814592, max=969728, per=99.92%, avg=884805.42, stdev=36044.65, samples=59 iops : min= 3182, max= 3788, avg=3456.27, stdev=140.80, samples=59 write: IOPS=866, BW=217MiB/s (227MB/s)(6496MiB/30001msec) clat (usec): min=636, max=17206, avg=668.59, stdev=114.92 lat (usec): min=657, max=17229, avg=693.77, stdev=115.21 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[ 668], 60.00th=[ 668], | 70.00th=[ 668], 80.00th=[ 676], 90.00th=[ 685], 95.00th=[ 693], | 99.00th=[ 717], 99.50th=[ 734], 99.90th=[ 1020], 99.95th=[ 1319], | 99.99th=[ 2343] bw ( KiB/s): min=202752, max=233472, per=100.00%, avg=221748.07, stdev=6614.24, samples=59 iops : min= 792, max= 912, avg=866.20, stdev=25.84, samples=59 lat (usec) : 100=67.46%, 250=6.57%, 500=3.54%, 750=21.55%, 1000=0.61% lat (msec) : 2=0.26%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=4.31%, sys=7.30%, ctx=129768, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=103774,25984,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=865MiB/s (907MB/s), 865MiB/s-865MiB/s (907MB/s-907MB/s), io=25.3GiB (27.2GB), run=30001-30001msec WRITE: bw=217MiB/s (227MB/s), 217MiB/s-217MiB/s (227MB/s-227MB/s), io=6496MiB (6812MB), run=30001-30001msec Disk stats (read/write): vdb: ios=103291/25858, merge=0/0, ticks=314/42, in_queue=26140, util=87.24% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=947439: Fri Jan 7 14:14:40 2022 read: IOPS=4209, BW=1052MiB/s (1103MB/s)(30.9GiB/30050msec) slat (usec): min=4, max=442, avg=11.66, stdev= 8.75 clat (usec): min=10, max=117132, avg=4604.89, stdev=11435.48 lat (usec): min=48, max=117145, avg=4616.69, stdev=11435.08 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 52], 10.00th=[ 54], 20.00th=[ 58], | 30.00th=[ 64], 40.00th=[ 76], 50.00th=[ 102], 60.00th=[ 118], | 70.00th=[ 141], 80.00th=[ 237], 90.00th=[ 25822], 95.00th=[ 32900], | 99.00th=[ 45351], 99.50th=[ 49021], 99.90th=[ 77071], 99.95th=[ 79168], | 99.99th=[101188] bw ( MiB/s): min= 486, max= 1290, per=100.00%, avg=1053.49, stdev=206.54, samples=60 iops : min= 1946, max= 5160, avg=4213.92, stdev=826.16, samples=60 write: IOPS=1050, BW=263MiB/s (275MB/s)(7895MiB/30050msec) slat (usec): min=17, max=636, avg=53.11, stdev=25.47 clat (msec): min=7, max=188, avg=76.60, stdev=20.33 lat (msec): min=7, max=188, avg=76.65, stdev=20.33 clat percentiles (msec): | 1.00th=[ 22], 5.00th=[ 34], 10.00th=[ 48], 20.00th=[ 65], | 30.00th=[ 71], 40.00th=[ 75], 50.00th=[ 80], 60.00th=[ 83], | 70.00th=[ 87], 80.00th=[ 91], 90.00th=[ 99], 95.00th=[ 106], | 99.00th=[ 125], 99.50th=[ 136], 99.90th=[ 163], 99.95th=[ 171], | 99.99th=[ 180] bw ( KiB/s): min=108544, max=326656, per=100.00%, avg=269295.73, stdev=49385.94, samples=60 iops : min= 424, max= 1276, avg=1051.92, stdev=192.92, samples=60 lat (usec) : 20=0.01%, 50=1.71%, 100=37.99%, 250=24.81%, 500=2.42% lat (usec) : 750=0.28%, 1000=0.15% lat (msec) : 2=0.43%, 4=0.34%, 10=0.41%, 20=0.78%, 50=12.60% lat (msec) : 100=16.39%, 250=1.69% cpu : usr=6.74%, sys=9.60%, ctx=117888, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=126487,31579,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1052MiB/s (1103MB/s), 1052MiB/s-1052MiB/s (1103MB/s-1103MB/s), io=30.9GiB (33.2GB), run=30050-30050msec WRITE: bw=263MiB/s (275MB/s), 263MiB/s-263MiB/s (275MB/s-275MB/s), io=7895MiB (8278MB), run=30050-30050msec Disk stats (read/write): vdb: ios=125991/31475, merge=0/0, ticks=532763/2394942, in_queue=2965286, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=948013: Fri Jan 7 14:15:17 2022 read: IOPS=3442, BW=861MiB/s (902MB/s)(25.2GiB/30001msec) clat (usec): min=52, max=8874, avg=114.45, stdev=138.11 lat (usec): min=52, max=8874, avg=114.58, stdev=138.11 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 67], 10.00th=[ 68], 20.00th=[ 70], | 30.00th=[ 73], 40.00th=[ 75], 50.00th=[ 78], 60.00th=[ 82], | 70.00th=[ 86], 80.00th=[ 94], 90.00th=[ 133], 95.00th=[ 351], | 99.00th=[ 758], 99.50th=[ 881], 99.90th=[ 1369], 99.95th=[ 1434], | 99.99th=[ 2089] bw ( KiB/s): min=777728, max=980480, per=99.96%, avg=880822.24, stdev=34272.99, samples=59 iops : min= 3038, max= 3830, avg=3440.71, stdev=133.88, samples=59 write: IOPS=861, BW=215MiB/s (226MB/s)(6464MiB/30001msec) clat (usec): min=637, max=10216, avg=670.45, stdev=77.83 lat (usec): min=656, max=10227, avg=694.94, stdev=78.14 clat percentiles (usec): | 1.00th=[ 652], 5.00th=[ 652], 10.00th=[ 652], 20.00th=[ 660], | 30.00th=[ 660], 40.00th=[ 660], 50.00th=[ 668], 60.00th=[ 668], | 70.00th=[ 676], 80.00th=[ 676], 90.00th=[ 685], 95.00th=[ 693], | 99.00th=[ 725], 99.50th=[ 750], 99.90th=[ 1074], 99.95th=[ 1336], | 99.99th=[ 5276] bw ( KiB/s): min=204800, max=235520, per=100.00%, avg=220689.36, stdev=6715.47, samples=59 iops : min= 800, max= 920, avg=862.07, stdev=26.23, samples=59 lat (usec) : 100=67.37%, 250=6.58%, 500=3.59%, 750=21.51%, 1000=0.66% lat (msec) : 2=0.27%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=4.27%, sys=7.46%, ctx=129137, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=103266,25854,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=861MiB/s (902MB/s), 861MiB/s-861MiB/s (902MB/s-902MB/s), io=25.2GiB (27.1GB), run=30001-30001msec WRITE: bw=215MiB/s (226MB/s), 215MiB/s-215MiB/s (226MB/s-226MB/s), io=6464MiB (6777MB), run=30001-30001msec Disk stats (read/write): vdb: ios=102541/25686, merge=0/0, ticks=330/43, in_queue=26297, util=87.88% 30s: (g=0): rw=randrw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=949366: Fri Jan 7 14:15:54 2022 read: IOPS=4316, BW=1079MiB/s (1132MB/s)(31.6GiB/30027msec) slat (usec): min=4, max=112270, avg=168.32, stdev=1061.26 clat (usec): min=697, max=161540, avg=18178.07, stdev=14412.20 lat (msec): min=3, max=185, avg=18.35, stdev=14.50 clat percentiles (msec): | 1.00th=[ 7], 5.00th=[ 9], 10.00th=[ 10], 20.00th=[ 11], | 30.00th=[ 12], 40.00th=[ 14], 50.00th=[ 15], 60.00th=[ 16], | 70.00th=[ 17], 80.00th=[ 20], 90.00th=[ 34], 95.00th=[ 46], | 99.00th=[ 88], 99.50th=[ 113], 99.90th=[ 122], 99.95th=[ 128], | 99.99th=[ 146] bw ( MiB/s): min= 457, max= 1337, per=99.97%, avg=1078.80, stdev=184.70, samples=60 iops : min= 1828, max= 5348, avg=4315.22, stdev=738.82, samples=60 write: IOPS=1077, BW=269MiB/s (283MB/s)(8091MiB/30027msec) slat (usec): min=22, max=69281, avg=243.74, stdev=917.11 clat (msec): min=21, max=243, avg=111.80, stdev=26.23 lat (msec): min=21, max=244, avg=112.04, stdev=26.24 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 62], 10.00th=[ 86], 20.00th=[ 96], | 30.00th=[ 102], 40.00th=[ 107], 50.00th=[ 112], 60.00th=[ 117], | 70.00th=[ 123], 80.00th=[ 130], 90.00th=[ 142], 95.00th=[ 155], | 99.00th=[ 182], 99.50th=[ 197], 99.90th=[ 226], 99.95th=[ 234], | 99.99th=[ 241] bw ( KiB/s): min=131584, max=352256, per=99.99%, avg=275883.73, stdev=41293.09, samples=60 iops : min= 514, max= 1376, avg=1077.67, stdev=161.30, samples=60 lat (usec) : 750=0.01% lat (msec) : 4=0.03%, 10=11.76%, 20=53.16%, 50=12.68%, 100=7.19% lat (msec) : 250=15.18% cpu : usr=9.77%, sys=4.93%, ctx=17027, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=129617,32364,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1079MiB/s (1132MB/s), 1079MiB/s-1079MiB/s (1132MB/s-1132MB/s), io=31.6GiB (33.0GB), run=30027-30027msec WRITE: bw=269MiB/s (283MB/s), 269MiB/s-269MiB/s (283MB/s-283MB/s), io=8091MiB (8484MB), run=30027-30027msec Disk stats (read/write): vdb: ios=128973/32238, merge=0/0, ticks=473054/3126456, in_queue=3690773, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950048: Fri Jan 7 14:16:30 2022 read: IOPS=1240, BW=620MiB/s (651MB/s)(18.2GiB/30005msec) clat (usec): min=96, max=3737, avg=197.92, stdev=204.19 lat (usec): min=96, max=3737, avg=198.07, stdev=204.19 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 111], 10.00th=[ 113], 20.00th=[ 117], | 30.00th=[ 121], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 141], | 70.00th=[ 153], 80.00th=[ 184], 90.00th=[ 359], 95.00th=[ 537], | 99.00th=[ 1237], 99.50th=[ 1369], 99.90th=[ 1844], 99.95th=[ 1991], | 99.99th=[ 2180] bw ( KiB/s): min=161792, max=1017856, per=100.00%, avg=635460.27, stdev=319979.56, samples=60 iops : min= 316, max= 1988, avg=1241.13, stdev=624.96, samples=60 write: IOPS=312, BW=156MiB/s (164MB/s)(4689MiB/30005msec) clat (usec): min=1233, max=24690, avg=2353.62, stdev=2388.45 lat (usec): min=1260, max=24807, avg=2405.30, stdev=2391.47 clat percentiles (usec): | 1.00th=[ 1254], 5.00th=[ 1254], 10.00th=[ 1254], 20.00th=[ 1254], | 30.00th=[ 1270], 40.00th=[ 1270], 50.00th=[ 1270], 60.00th=[ 1287], | 70.00th=[ 1287], 80.00th=[ 2835], 90.00th=[ 6849], 95.00th=[ 7635], | 99.00th=[ 9503], 99.50th=[11469], 99.90th=[19268], 99.95th=[21365], | 99.99th=[24773] bw ( KiB/s): min=50176, max=258048, per=100.00%, avg=160017.07, stdev=80939.79, samples=60 iops : min= 98, max= 504, avg=312.53, stdev=158.09, samples=60 lat (usec) : 100=0.04%, 250=68.25%, 500=7.09%, 750=2.18%, 1000=0.74% lat (msec) : 2=17.62%, 4=0.71%, 10=3.25%, 20=0.12%, 50=0.02% cpu : usr=2.44%, sys=3.43%, ctx=46621, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=37234,9377,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=620MiB/s (651MB/s), 620MiB/s-620MiB/s (651MB/s-651MB/s), io=18.2GiB (19.5GB), run=30005-30005msec WRITE: bw=156MiB/s (164MB/s), 156MiB/s-156MiB/s (164MB/s-164MB/s), io=4689MiB (4916MB), run=30005-30005msec Disk stats (read/write): vdb: ios=37171/9356, merge=0/0, ticks=750/18889, in_queue=27986, util=93.50% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=950863: Fri Jan 7 14:17:07 2022 read: IOPS=1502, BW=751MiB/s (788MB/s)(22.0GiB/30004msec) slat (usec): min=18, max=417, avg=28.07, stdev= 6.40 clat (usec): min=27, max=3099, avg=177.91, stdev=204.80 lat (usec): min=101, max=3131, avg=206.14, stdev=205.13 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 94], 10.00th=[ 96], 20.00th=[ 100], | 30.00th=[ 103], 40.00th=[ 109], 50.00th=[ 114], 60.00th=[ 122], | 70.00th=[ 131], 80.00th=[ 151], 90.00th=[ 334], 95.00th=[ 506], | 99.00th=[ 1205], 99.50th=[ 1336], 99.90th=[ 1975], 99.95th=[ 2073], | 99.99th=[ 2147] bw ( KiB/s): min=226304, max=1040384, per=100.00%, avg=769653.32, stdev=257240.09, samples=60 iops : min= 442, max= 2032, avg=1503.20, stdev=502.40, samples=60 write: IOPS=379, BW=190MiB/s (199MB/s)(5693MiB/30004msec) slat (usec): min=40, max=389, avg=73.90, stdev=13.37 clat (usec): min=1200, max=25763, avg=1737.35, stdev=1281.95 lat (usec): min=1270, max=25847, avg=1811.43, stdev=1284.18 clat percentiles (usec): | 1.00th=[ 1237], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1254], 40.00th=[ 1254], 50.00th=[ 1254], 60.00th=[ 1270], | 70.00th=[ 1270], 80.00th=[ 1303], 90.00th=[ 3392], 95.00th=[ 4113], | 99.00th=[ 6783], 99.50th=[ 8848], 99.90th=[12649], 99.95th=[15401], | 99.99th=[22152] bw ( KiB/s): min=61440, max=258048, per=100.00%, avg=194327.67, stdev=63379.97, samples=60 iops : min= 120, max= 504, avg=379.50, stdev=123.76, samples=60 lat (usec) : 50=0.01%, 100=17.26%, 250=51.60%, 500=6.92%, 750=1.86% lat (usec) : 1000=0.57% lat (msec) : 2=18.27%, 4=2.43%, 10=1.05%, 20=0.03%, 50=0.01% cpu : usr=2.75%, sys=6.80%, ctx=56501, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=45091,11386,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=751MiB/s (788MB/s), 751MiB/s-751MiB/s (788MB/s-788MB/s), io=22.0GiB (23.6GB), run=30004-30004msec WRITE: bw=190MiB/s (199MB/s), 190MiB/s-190MiB/s (199MB/s-199MB/s), io=5693MiB (5970MB), run=30004-30004msec Disk stats (read/write): vdb: ios=45019/11367, merge=0/0, ticks=959/16258, in_queue=27534, util=91.96% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=951816: Fri Jan 7 14:17:44 2022 read: IOPS=1726, BW=863MiB/s (905MB/s)(25.3GiB/30005msec) clat (usec): min=78, max=2168, avg=172.03, stdev=201.52 lat (usec): min=78, max=2169, avg=172.17, stdev=201.52 clat percentiles (usec): | 1.00th=[ 91], 5.00th=[ 94], 10.00th=[ 96], 20.00th=[ 99], | 30.00th=[ 103], 40.00th=[ 108], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 125], 80.00th=[ 141], 90.00th=[ 302], 95.00th=[ 474], | 99.00th=[ 1188], 99.50th=[ 1352], 99.90th=[ 1991], 99.95th=[ 2057], | 99.99th=[ 2114] bw ( KiB/s): min=316416, max=1134592, per=100.00%, avg=884121.67, stdev=240449.99, samples=60 iops : min= 618, max= 2216, avg=1726.80, stdev=469.63, samples=60 write: IOPS=434, BW=217MiB/s (228MB/s)(6516MiB/30005msec) clat (usec): min=1218, max=27584, avg=1560.62, stdev=925.81 lat (usec): min=1246, max=27638, avg=1610.13, stdev=926.22 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1254], 50.00th=[ 1254], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1270], 90.00th=[ 3130], 95.00th=[ 3523], | 99.00th=[ 5014], 99.50th=[ 6063], 99.90th=[ 8455], 99.95th=[10290], | 99.99th=[14222] bw ( KiB/s): min=86016, max=274432, per=100.00%, avg=222400.00, stdev=60874.33, samples=60 iops : min= 168, max= 536, avg=434.37, stdev=118.91, samples=60 lat (usec) : 100=18.26%, 250=50.93%, 500=7.18%, 750=1.51%, 1000=0.40% lat (msec) : 2=19.18%, 4=1.97%, 10=0.57%, 20=0.01%, 50=0.01% cpu : usr=3.27%, sys=4.89%, ctx=64845, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=51803,13032,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=863MiB/s (905MB/s), 863MiB/s-863MiB/s (905MB/s-905MB/s), io=25.3GiB (27.2GB), run=30005-30005msec WRITE: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s), io=6516MiB (6833MB), run=30005-30005msec Disk stats (read/write): vdb: ios=51703/13001, merge=0/0, ticks=1084/16555, in_queue=27154, util=90.68% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=952397: Fri Jan 7 14:18:21 2022 read: IOPS=2013, BW=1007MiB/s (1056MB/s)(29.5GiB/30008msec) slat (usec): min=5, max=130, avg=19.42, stdev= 7.29 clat (usec): min=2, max=30584, avg=2109.84, stdev=4186.89 lat (usec): min=73, max=30593, avg=2129.41, stdev=4186.70 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 74], 10.00th=[ 76], 20.00th=[ 83], | 30.00th=[ 92], 40.00th=[ 108], 50.00th=[ 133], 60.00th=[ 172], | 70.00th=[ 262], 80.00th=[ 3359], 90.00th=[ 9110], 95.00th=[12256], | 99.00th=[16909], 99.50th=[19006], 99.90th=[22938], 99.95th=[24249], | 99.99th=[27132] bw ( KiB/s): min=519168, max=1464320, per=100.00%, avg=1031209.13, stdev=209151.69, samples=60 iops : min= 1014, max= 2860, avg=2014.07, stdev=408.49, samples=60 write: IOPS=505, BW=253MiB/s (265MB/s)(7590MiB/30008msec) slat (usec): min=31, max=1220, avg=81.23, stdev=30.85 clat (usec): min=1876, max=40744, avg=11202.81, stdev=3937.47 lat (usec): min=1944, max=40958, avg=11284.22, stdev=3940.48 clat percentiles (usec): | 1.00th=[ 4047], 5.00th=[ 5669], 10.00th=[ 6652], 20.00th=[ 7963], | 30.00th=[ 8979], 40.00th=[ 9896], 50.00th=[10814], 60.00th=[11731], | 70.00th=[12780], 80.00th=[14091], 90.00th=[16057], 95.00th=[18220], | 99.00th=[22414], 99.50th=[24249], 99.90th=[33817], 99.95th=[36439], | 99.99th=[38011] bw ( KiB/s): min=125952, max=340992, per=100.00%, avg=259031.40, stdev=49132.13, samples=60 iops : min= 246, max= 666, avg=505.92, stdev=95.96, samples=60 lat (usec) : 4=0.01%, 50=0.01%, 100=29.05%, 250=26.44%, 500=3.50% lat (usec) : 750=0.94%, 1000=0.58% lat (msec) : 2=0.96%, 4=3.95%, 10=16.02%, 20=17.71%, 50=0.87% cpu : usr=4.42%, sys=6.23%, ctx=55333, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60427,15180,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1007MiB/s (1056MB/s), 1007MiB/s-1007MiB/s (1056MB/s-1056MB/s), io=29.5GiB (31.7GB), run=30008-30008msec WRITE: bw=253MiB/s (265MB/s), 253MiB/s-253MiB/s (265MB/s-265MB/s), io=7590MiB (7959MB), run=30008-30008msec Disk stats (read/write): vdb: ios=60333/15154, merge=0/0, ticks=111505/161656, in_queue=293978, util=99.71% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=953227: Fri Jan 7 14:18:59 2022 read: IOPS=2001, BW=1001MiB/s (1050MB/s)(29.3GiB/30001msec) clat (usec): min=79, max=2242, avg=171.22, stdev=206.30 lat (usec): min=80, max=2242, avg=171.35, stdev=206.30 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 95], 20.00th=[ 98], | 30.00th=[ 101], 40.00th=[ 105], 50.00th=[ 110], 60.00th=[ 115], | 70.00th=[ 123], 80.00th=[ 137], 90.00th=[ 306], 95.00th=[ 474], | 99.00th=[ 1188], 99.50th=[ 1434], 99.90th=[ 2040], 99.95th=[ 2073], | 99.99th=[ 2147] bw ( KiB/s): min=906240, max=1159168, per=99.89%, avg=1023859.12, stdev=53853.48, samples=59 iops : min= 1770, max= 2264, avg=1999.71, stdev=105.20, samples=59 write: IOPS=502, BW=251MiB/s (264MB/s)(7544MiB/30001msec) clat (usec): min=1214, max=2751, avg=1250.07, stdev=37.77 lat (usec): min=1245, max=2795, avg=1298.03, stdev=39.35 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1237], 50.00th=[ 1254], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1254], 90.00th=[ 1270], 95.00th=[ 1270], | 99.00th=[ 1303], 99.50th=[ 1319], 99.90th=[ 1909], 99.95th=[ 2180], | 99.99th=[ 2474] bw ( KiB/s): min=234496, max=280112, per=100.00%, avg=257528.19, stdev=7672.71, samples=59 iops : min= 458, max= 547, avg=502.98, stdev=14.98, samples=59 lat (usec) : 100=22.48%, 250=46.67%, 500=7.19%, 750=1.50%, 1000=0.44% lat (msec) : 2=21.61%, 4=0.12% cpu : usr=3.67%, sys=5.42%, ctx=75162, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=60060,15088,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1001MiB/s (1050MB/s), 1001MiB/s-1001MiB/s (1050MB/s-1050MB/s), io=29.3GiB (31.5GB), run=30001-30001msec WRITE: bw=251MiB/s (264MB/s), 251MiB/s-251MiB/s (264MB/s-264MB/s), io=7544MiB (7910MB), run=30001-30001msec Disk stats (read/write): vdb: ios=59718/14994, merge=0/0, ticks=1293/15006, in_queue=26953, util=90.05% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954253: Fri Jan 7 14:19:36 2022 read: IOPS=2093, BW=1047MiB/s (1098MB/s)(30.7GiB/30008msec) slat (usec): min=5, max=255, avg=13.62, stdev= 7.70 clat (usec): min=23, max=140384, avg=8712.68, stdev=19437.71 lat (usec): min=70, max=140393, avg=8726.44, stdev=19437.63 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 74], 20.00th=[ 79], | 30.00th=[ 89], 40.00th=[ 137], 50.00th=[ 188], 60.00th=[ 212], | 70.00th=[ 343], 80.00th=[ 799], 90.00th=[ 44303], 95.00th=[ 56886], | 99.00th=[ 73925], 99.50th=[ 81265], 99.90th=[ 98042], 99.95th=[104334], | 99.99th=[125305] bw ( MiB/s): min= 413, max= 1320, per=99.98%, avg=1046.71, stdev=178.57, samples=60 iops : min= 826, max= 2640, avg=2093.42, stdev=357.13, samples=60 write: IOPS=526, BW=263MiB/s (276MB/s)(7901MiB/30008msec) slat (usec): min=33, max=465, avg=104.66, stdev=52.60 clat (msec): min=8, max=192, avg=60.14, stdev=14.31 lat (msec): min=8, max=192, avg=60.24, stdev=14.31 clat percentiles (msec): | 1.00th=[ 31], 5.00th=[ 43], 10.00th=[ 46], 20.00th=[ 50], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 70], 90.00th=[ 78], 95.00th=[ 86], | 99.00th=[ 109], 99.50th=[ 117], 99.90th=[ 138], 99.95th=[ 142], | 99.99th=[ 182] bw ( KiB/s): min=115712, max=328704, per=99.95%, avg=269451.15, stdev=42335.53, samples=60 iops : min= 226, max= 642, avg=526.27, stdev=82.68, samples=60 lat (usec) : 50=0.02%, 100=27.47%, 250=24.31%, 500=8.74%, 750=3.11% lat (usec) : 1000=0.93% lat (msec) : 2=0.57%, 4=0.22%, 10=0.25%, 20=0.46%, 50=12.09% lat (msec) : 100=21.40%, 250=0.43% cpu : usr=6.10%, sys=4.83%, ctx=56392, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=62834,15801,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1047MiB/s (1098MB/s), 1047MiB/s-1047MiB/s (1098MB/s-1098MB/s), io=30.7GiB (32.9GB), run=30008-30008msec WRITE: bw=263MiB/s (276MB/s), 263MiB/s-263MiB/s (276MB/s-276MB/s), io=7901MiB (8284MB), run=30008-30008msec Disk stats (read/write): vdb: ios=62649/15737, merge=0/0, ticks=523774/937598, in_queue=1483968, util=99.69% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=954767: Fri Jan 7 14:20:12 2022 read: IOPS=1818, BW=909MiB/s (954MB/s)(26.6GiB/30001msec) clat (usec): min=113, max=15327, avg=210.11, stdev=215.52 lat (usec): min=113, max=15328, avg=210.25, stdev=215.53 clat percentiles (usec): | 1.00th=[ 122], 5.00th=[ 126], 10.00th=[ 128], 20.00th=[ 133], | 30.00th=[ 137], 40.00th=[ 141], 50.00th=[ 147], 60.00th=[ 155], | 70.00th=[ 167], 80.00th=[ 188], 90.00th=[ 343], 95.00th=[ 510], | 99.00th=[ 1237], 99.50th=[ 1450], 99.90th=[ 2073], 99.95th=[ 2114], | 99.99th=[ 2180] bw ( KiB/s): min=808960, max=1022976, per=100.00%, avg=931401.71, stdev=44981.00, samples=59 iops : min= 1580, max= 1998, avg=1819.14, stdev=87.86, samples=59 write: IOPS=457, BW=229MiB/s (240MB/s)(6856MiB/30001msec) clat (usec): min=1252, max=6493, avg=1293.20, stdev=62.28 lat (usec): min=1284, max=6545, avg=1343.37, stdev=64.17 clat percentiles (usec): | 1.00th=[ 1270], 5.00th=[ 1270], 10.00th=[ 1270], 20.00th=[ 1270], | 30.00th=[ 1287], 40.00th=[ 1287], 50.00th=[ 1287], 60.00th=[ 1287], | 70.00th=[ 1303], 80.00th=[ 1303], 90.00th=[ 1319], 95.00th=[ 1336], | 99.00th=[ 1369], 99.50th=[ 1385], 99.90th=[ 1729], 99.95th=[ 2147], | 99.99th=[ 4490] bw ( KiB/s): min=212992, max=250880, per=100.00%, avg=234087.59, stdev=8213.51, samples=59 iops : min= 416, max= 490, avg=457.19, stdev=16.03, samples=59 lat (usec) : 250=68.10%, 500=7.61%, 750=1.88%, 1000=0.65% lat (msec) : 2=21.64%, 4=0.12%, 10=0.01%, 20=0.01% cpu : usr=3.46%, sys=4.99%, ctx=68291, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=54566,13711,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=909MiB/s (954MB/s), 909MiB/s-909MiB/s (954MB/s-954MB/s), io=26.6GiB (28.6GB), run=30001-30001msec WRITE: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=6856MiB (7189MB), run=30001-30001msec Disk stats (read/write): vdb: ios=54270/13638, merge=0/0, ticks=1183/13650, in_queue=27054, util=90.32% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956004: Fri Jan 7 14:20:50 2022 read: IOPS=2123, BW=1062MiB/s (1114MB/s)(31.2GiB/30045msec) slat (usec): min=5, max=467, avg=13.38, stdev= 9.13 clat (usec): min=17, max=242518, avg=8944.77, stdev=21236.72 lat (usec): min=72, max=242525, avg=8958.27, stdev=21236.79 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 73], 10.00th=[ 75], 20.00th=[ 80], | 30.00th=[ 91], 40.00th=[ 147], 50.00th=[ 190], 60.00th=[ 221], | 70.00th=[ 367], 80.00th=[ 930], 90.00th=[ 46400], 95.00th=[ 58459], | 99.00th=[ 80217], 99.50th=[ 93848], 99.90th=[175113], 99.95th=[198181], | 99.99th=[225444] bw ( MiB/s): min= 681, max= 1385, per=100.00%, avg=1062.70, stdev=132.97, samples=60 iops : min= 1362, max= 2770, avg=2125.37, stdev=265.94, samples=60 write: IOPS=533, BW=267MiB/s (280MB/s)(8021MiB/30045msec) slat (usec): min=33, max=396, avg=99.41, stdev=48.85 clat (msec): min=18, max=368, avg=151.51, stdev=37.01 lat (msec): min=18, max=368, avg=151.61, stdev=37.01 clat percentiles (msec): | 1.00th=[ 39], 5.00th=[ 74], 10.00th=[ 118], 20.00th=[ 136], | 30.00th=[ 142], 40.00th=[ 148], 50.00th=[ 155], 60.00th=[ 159], | 70.00th=[ 165], 80.00th=[ 174], 90.00th=[ 186], 95.00th=[ 197], | 99.00th=[ 275], 99.50th=[ 309], 99.90th=[ 351], 99.95th=[ 355], | 99.99th=[ 368] bw ( KiB/s): min=157696, max=328704, per=99.91%, avg=273117.97, stdev=28265.98, samples=60 iops : min= 308, max= 642, avg=533.40, stdev=55.25, samples=60 lat (usec) : 20=0.01%, 50=0.01%, 100=26.79%, 250=23.73%, 500=9.04% lat (usec) : 750=3.35%, 1000=1.23% lat (msec) : 2=0.94%, 4=0.49%, 10=0.94%, 20=0.74%, 50=6.32% lat (msec) : 100=7.58%, 250=18.58%, 500=0.27% cpu : usr=5.87%, sys=4.84%, ctx=56258, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=63812,16042,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1062MiB/s (1114MB/s), 1062MiB/s-1062MiB/s (1114MB/s-1114MB/s), io=31.2GiB (33.5GB), run=30045-30045msec WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=8021MiB (8411MB), run=30045-30045msec Disk stats (read/write): vdb: ios=63556/15955, merge=0/0, ticks=535390/2411471, in_queue=2973070, util=99.61% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=956764: Fri Jan 7 14:21:27 2022 read: IOPS=1994, BW=997MiB/s (1046MB/s)(29.2GiB/30001msec) clat (usec): min=78, max=2645, avg=172.53, stdev=208.28 lat (usec): min=78, max=2645, avg=172.67, stdev=208.28 clat percentiles (usec): | 1.00th=[ 90], 5.00th=[ 93], 10.00th=[ 94], 20.00th=[ 97], | 30.00th=[ 99], 40.00th=[ 103], 50.00th=[ 109], 60.00th=[ 115], | 70.00th=[ 123], 80.00th=[ 145], 90.00th=[ 306], 95.00th=[ 478], | 99.00th=[ 1188], 99.50th=[ 1434], 99.90th=[ 2057], 99.95th=[ 2089], | 99.99th=[ 2147] bw ( KiB/s): min=904192, max=1135616, per=100.00%, avg=1021506.98, stdev=53917.49, samples=60 iops : min= 1766, max= 2218, avg=1995.12, stdev=105.32, samples=60 write: IOPS=501, BW=251MiB/s (263MB/s)(7520MiB/30001msec) clat (usec): min=1216, max=3520, avg=1249.03, stdev=41.90 lat (usec): min=1244, max=3574, avg=1300.06, stdev=44.87 clat percentiles (usec): | 1.00th=[ 1221], 5.00th=[ 1237], 10.00th=[ 1237], 20.00th=[ 1237], | 30.00th=[ 1237], 40.00th=[ 1237], 50.00th=[ 1237], 60.00th=[ 1254], | 70.00th=[ 1254], 80.00th=[ 1254], 90.00th=[ 1270], 95.00th=[ 1270], | 99.00th=[ 1303], 99.50th=[ 1336], 99.90th=[ 1598], 99.95th=[ 2147], | 99.99th=[ 3294] bw ( KiB/s): min=233472, max=278528, per=100.00%, avg=256682.95, stdev=7674.05, samples=60 iops : min= 456, max= 544, avg=501.32, stdev=15.00, samples=60 lat (usec) : 100=25.65%, 250=43.42%, 500=7.25%, 750=1.52%, 1000=0.40% lat (msec) : 2=21.63%, 4=0.13% cpu : usr=3.79%, sys=5.39%, ctx=74899, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=59851,15039,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=997MiB/s (1046MB/s), 997MiB/s-997MiB/s (1046MB/s-1046MB/s), io=29.2GiB (31.4GB), run=30001-30001msec WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=7520MiB (7885MB), run=30001-30001msec Disk stats (read/write): vdb: ios=59387/14937, merge=0/0, ticks=1314/14947, in_queue=26859, util=89.67% 30s: (g=0): rw=randrw, bs=(R) 512KiB-512KiB, (W) 512KiB-512KiB, (T) 512KiB-512KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=957723: Fri Jan 7 14:22:04 2022 read: IOPS=2138, BW=1069MiB/s (1121MB/s)(31.4GiB/30079msec) slat (usec): min=5, max=163952, avg=344.73, stdev=1879.48 clat (usec): min=506, max=341423, avg=37952.96, stdev=25550.25 lat (msec): min=9, max=341, avg=38.30, stdev=25.63 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 17], 10.00th=[ 20], 20.00th=[ 23], | 30.00th=[ 26], 40.00th=[ 28], 50.00th=[ 31], 60.00th=[ 33], | 70.00th=[ 37], 80.00th=[ 44], 90.00th=[ 75], 95.00th=[ 90], | 99.00th=[ 132], 99.50th=[ 169], 99.90th=[ 243], 99.95th=[ 262], | 99.99th=[ 317] bw ( MiB/s): min= 514, max= 1386, per=100.00%, avg=1069.91, stdev=149.44, samples=60 iops : min= 1028, max= 2772, avg=2139.80, stdev=298.88, samples=60 write: IOPS=537, BW=269MiB/s (282MB/s)(8077MiB/30079msec) slat (usec): min=65, max=223428, avg=478.23, stdev=2150.09 clat (msec): min=19, max=515, avg=219.30, stdev=46.72 lat (msec): min=33, max=515, avg=219.78, stdev=46.70 clat percentiles (msec): | 1.00th=[ 73], 5.00th=[ 140], 10.00th=[ 174], 20.00th=[ 192], | 30.00th=[ 205], 40.00th=[ 213], 50.00th=[ 222], 60.00th=[ 230], | 70.00th=[ 241], 80.00th=[ 251], 90.00th=[ 266], 95.00th=[ 279], | 99.00th=[ 347], 99.50th=[ 384], 99.90th=[ 489], 99.95th=[ 506], | 99.99th=[ 514] bw ( KiB/s): min=223232, max=320512, per=99.89%, avg=274647.13, stdev=23115.00, samples=60 iops : min= 436, max= 626, avg=536.40, stdev=45.16, samples=60 lat (usec) : 750=0.01% lat (msec) : 10=0.06%, 20=9.51%, 50=56.82%, 100=11.70%, 250=17.76% lat (msec) : 500=4.14%, 750=0.01% cpu : usr=9.23%, sys=3.53%, ctx=8093, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=64326,16153,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1069MiB/s (1121MB/s), 1069MiB/s-1069MiB/s (1121MB/s-1121MB/s), io=31.4GiB (33.7GB), run=30079-30079msec WRITE: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s), io=8077MiB (8469MB), run=30079-30079msec Disk stats (read/write): vdb: ios=64243/16107, merge=0/0, ticks=605851/3068621, in_queue=3719399, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=958708: Fri Jan 7 14:22:41 2022 read: IOPS=1015, BW=1015MiB/s (1064MB/s)(29.7GiB/30003msec) clat (usec): min=152, max=3611, avg=321.41, stdev=343.14 lat (usec): min=152, max=3611, avg=321.57, stdev=343.14 clat percentiles (usec): | 1.00th=[ 163], 5.00th=[ 167], 10.00th=[ 172], 20.00th=[ 176], | 30.00th=[ 184], 40.00th=[ 192], 50.00th=[ 202], 60.00th=[ 217], | 70.00th=[ 253], 80.00th=[ 375], 90.00th=[ 553], 95.00th=[ 816], | 99.00th=[ 2180], 99.50th=[ 2507], 99.90th=[ 3064], 99.95th=[ 3097], | 99.99th=[ 3130] bw ( KiB/s): min=376832, max=1218560, per=100.00%, avg=1039564.37, stdev=136253.48, samples=60 iops : min= 368, max= 1190, avg=1015.18, stdev=133.07, samples=60 write: IOPS=256, BW=256MiB/s (269MB/s)(7690MiB/30003msec) clat (usec): min=2391, max=17233, avg=2520.32, stdev=708.97 lat (usec): min=2446, max=17341, avg=2619.70, stdev=712.21 clat percentiles (usec): | 1.00th=[ 2409], 5.00th=[ 2409], 10.00th=[ 2409], 20.00th=[ 2409], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2442], 60.00th=[ 2442], | 70.00th=[ 2442], 80.00th=[ 2442], 90.00th=[ 2474], 95.00th=[ 2474], | 99.00th=[ 6718], 99.50th=[ 8291], 99.90th=[ 9896], 99.95th=[14091], | 99.99th=[17171] bw ( KiB/s): min=114688, max=290816, per=100.00%, avg=262469.52, stdev=29907.64, samples=60 iops : min= 112, max= 284, avg=256.30, stdev=29.20, samples=60 lat (usec) : 250=55.60%, 500=14.50%, 750=4.81%, 1000=2.11% lat (msec) : 2=1.57%, 4=21.06%, 10=0.33%, 20=0.02% cpu : usr=3.34%, sys=3.99%, ctx=38151, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=30454,7690,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1015MiB/s (1064MB/s), 1015MiB/s-1015MiB/s (1064MB/s-1064MB/s), io=29.7GiB (31.9GB), run=30003-30003msec WRITE: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=7690MiB (8064MB), run=30003-30003msec Disk stats (read/write): vdb: ios=30375/7665, merge=0/0, ticks=1547/15882, in_queue=27455, util=91.80% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=959287: Fri Jan 7 14:23:18 2022 read: IOPS=1001, BW=1002MiB/s (1051MB/s)(29.4GiB/30002msec) slat (usec): min=30, max=145, avg=43.36, stdev= 9.42 clat (usec): min=93, max=13141, avg=306.16, stdev=358.80 lat (usec): min=168, max=13213, avg=349.72, stdev=359.37 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 149], 10.00th=[ 153], 20.00th=[ 159], | 30.00th=[ 165], 40.00th=[ 176], 50.00th=[ 188], 60.00th=[ 204], | 70.00th=[ 245], 80.00th=[ 359], 90.00th=[ 537], 95.00th=[ 791], | 99.00th=[ 2180], 99.50th=[ 2606], 99.90th=[ 3064], 99.95th=[ 3097], | 99.99th=[ 3359] bw ( KiB/s): min=892928, max=1177600, per=100.00%, avg=1026078.83, stdev=62652.48, samples=60 iops : min= 872, max= 1150, avg=1002.02, stdev=61.21, samples=60 write: IOPS=252, BW=253MiB/s (265MB/s)(7587MiB/30002msec) slat (usec): min=68, max=400, avg=133.85, stdev=30.12 clat (usec): min=2207, max=7680, avg=2425.95, stdev=94.59 lat (usec): min=2455, max=7811, avg=2560.06, stdev=100.04 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2409], 10.00th=[ 2409], 20.00th=[ 2409], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2442], 80.00th=[ 2442], 90.00th=[ 2474], 95.00th=[ 2474], | 99.00th=[ 2540], 99.50th=[ 2704], 99.90th=[ 3130], 99.95th=[ 4359], | 99.99th=[ 7701] bw ( KiB/s): min=229376, max=282624, per=100.00%, avg=258944.48, stdev=10025.11, samples=60 iops : min= 224, max= 276, avg=252.87, stdev= 9.78, samples=60 lat (usec) : 100=0.01%, 250=56.42%, 500=14.09%, 750=4.62%, 1000=1.99% lat (msec) : 2=1.44%, 4=21.41%, 10=0.01%, 20=0.01% cpu : usr=3.07%, sys=6.41%, ctx=37665, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=30060,7587,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1002MiB/s (1051MB/s), 1002MiB/s-1002MiB/s (1051MB/s-1051MB/s), io=29.4GiB (31.5GB), run=30002-30002msec WRITE: bw=253MiB/s (265MB/s), 253MiB/s-253MiB/s (265MB/s-265MB/s), io=7587MiB (7956MB), run=30002-30002msec Disk stats (read/write): vdb: ios=29896/7548, merge=0/0, ticks=1592/15119, in_queue=27583, util=92.16% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=960014: Fri Jan 7 14:23:55 2022 read: IOPS=1038, BW=1039MiB/s (1089MB/s)(30.4GiB/30001msec) clat (usec): min=132, max=3307, avg=300.91, stdev=350.79 lat (usec): min=132, max=3307, avg=301.06, stdev=350.80 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 149], 10.00th=[ 151], 20.00th=[ 155], | 30.00th=[ 161], 40.00th=[ 169], 50.00th=[ 180], 60.00th=[ 192], | 70.00th=[ 225], 80.00th=[ 355], 90.00th=[ 537], 95.00th=[ 791], | 99.00th=[ 2180], 99.50th=[ 2671], 99.90th=[ 3032], 99.95th=[ 3064], | 99.99th=[ 3130] bw ( MiB/s): min= 452, max= 1226, per=100.00%, avg=1038.73, stdev=143.22, samples=60 iops : min= 452, max= 1226, avg=1038.73, stdev=143.22, samples=60 write: IOPS=262, BW=263MiB/s (276MB/s)(7884MiB/30001msec) clat (usec): min=2371, max=14841, avg=2507.34, stdev=658.40 lat (usec): min=2417, max=14954, avg=2606.91, stdev=660.36 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2409], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2442], 95.00th=[ 2474], | 99.00th=[ 6456], 99.50th=[ 7308], 99.90th=[ 8848], 99.95th=[11207], | 99.99th=[14877] bw ( KiB/s): min=131072, max=303104, per=99.99%, avg=269073.07, stdev=31893.06, samples=60 iops : min= 128, max= 296, avg=262.77, stdev=31.15, samples=60 lat (usec) : 250=57.60%, 500=13.02%, 750=4.46%, 1000=1.92% lat (msec) : 2=1.52%, 4=20.99%, 10=0.47%, 20=0.01% cpu : usr=3.30%, sys=4.17%, ctx=39055, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=31162,7884,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1039MiB/s (1089MB/s), 1039MiB/s-1039MiB/s (1089MB/s-1089MB/s), io=30.4GiB (32.7GB), run=30001-30001msec WRITE: bw=263MiB/s (276MB/s), 263MiB/s-263MiB/s (276MB/s-276MB/s), io=7884MiB (8267MB), run=30001-30001msec Disk stats (read/write): vdb: ios=31112/7871, merge=0/0, ticks=1580/16413, in_queue=27507, util=91.78% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961054: Fri Jan 7 14:24:32 2022 read: IOPS=1084, BW=1085MiB/s (1137MB/s)(31.8GiB/30004msec) slat (usec): min=9, max=137, avg=21.44, stdev= 9.91 clat (usec): min=58, max=53610, avg=4662.20, stdev=7952.84 lat (usec): min=116, max=53625, avg=4683.80, stdev=7952.86 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 114], 10.00th=[ 118], 20.00th=[ 129], | 30.00th=[ 190], 40.00th=[ 289], 50.00th=[ 343], 60.00th=[ 453], | 70.00th=[ 832], 80.00th=[11600], 90.00th=[17957], 95.00th=[22152], | 99.00th=[29754], 99.50th=[32637], 99.90th=[38536], 99.95th=[41157], | 99.99th=[46924] bw ( MiB/s): min= 622, max= 1446, per=99.99%, avg=1084.47, stdev=177.50, samples=60 iops : min= 622, max= 1446, avg=1084.47, stdev=177.50, samples=60 write: IOPS=274, BW=275MiB/s (288MB/s)(8237MiB/30004msec) slat (usec): min=58, max=611, avg=191.81, stdev=90.04 clat (usec): min=2543, max=52942, avg=17722.02, stdev=6696.98 lat (usec): min=2674, max=53240, avg=17914.02, stdev=6700.72 clat percentiles (usec): | 1.00th=[ 5866], 5.00th=[ 8225], 10.00th=[ 9896], 20.00th=[11994], | 30.00th=[13829], 40.00th=[15401], 50.00th=[16909], 60.00th=[18744], | 70.00th=[20579], 80.00th=[22938], 90.00th=[26346], 95.00th=[30016], | 99.00th=[37487], 99.50th=[41157], 99.90th=[46924], 99.95th=[47973], | 99.99th=[52691] bw ( KiB/s): min=126976, max=335872, per=99.97%, avg=281019.73, stdev=37712.56, samples=60 iops : min= 124, max= 328, avg=274.43, stdev=36.83, samples=60 lat (usec) : 100=0.08%, 250=29.02%, 500=20.09%, 750=5.93%, 1000=1.64% lat (msec) : 2=0.98%, 4=0.45%, 10=6.25%, 20=23.05%, 50=12.50% lat (msec) : 100=0.01% cpu : usr=5.44%, sys=3.48%, ctx=28880, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32540,8237,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1085MiB/s (1137MB/s), 1085MiB/s-1085MiB/s (1137MB/s-1137MB/s), io=31.8GiB (34.1GB), run=30004-30004msec WRITE: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=8237MiB (8637MB), run=30004-30004msec Disk stats (read/write): vdb: ios=32381/8196, merge=0/0, ticks=138627/141258, in_queue=294785, util=99.68% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=961493: Fri Jan 7 14:25:09 2022 read: IOPS=1084, BW=1085MiB/s (1137MB/s)(31.8GiB/30002msec) clat (usec): min=123, max=3686, avg=286.91, stdev=343.45 lat (usec): min=123, max=3686, avg=287.06, stdev=343.46 clat percentiles (usec): | 1.00th=[ 135], 5.00th=[ 139], 10.00th=[ 141], 20.00th=[ 145], | 30.00th=[ 151], 40.00th=[ 157], 50.00th=[ 167], 60.00th=[ 180], | 70.00th=[ 206], 80.00th=[ 343], 90.00th=[ 519], 95.00th=[ 783], | 99.00th=[ 2147], 99.50th=[ 2540], 99.90th=[ 2999], 99.95th=[ 3032], | 99.99th=[ 3064] bw ( MiB/s): min= 880, max= 1250, per=100.00%, avg=1084.71, stdev=70.46, samples=60 iops : min= 880, max= 1250, avg=1084.70, stdev=70.46, samples=60 write: IOPS=274, BW=275MiB/s (288MB/s)(8237MiB/30002msec) clat (usec): min=2365, max=3815, avg=2400.85, stdev=49.07 lat (usec): min=2418, max=3917, avg=2500.63, stdev=56.48 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2376], 40.00th=[ 2376], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2409], 95.00th=[ 2442], | 99.00th=[ 2507], 99.50th=[ 2671], 99.90th=[ 3163], 99.95th=[ 3458], | 99.99th=[ 3818] bw ( KiB/s): min=262144, max=301056, per=100.00%, avg=281149.90, stdev=8797.88, samples=60 iops : min= 256, max= 294, avg=274.53, stdev= 8.60, samples=60 lat (usec) : 250=58.66%, 500=12.57%, 750=4.09%, 1000=1.75% lat (msec) : 2=1.50%, 4=21.42% cpu : usr=3.48%, sys=3.23%, ctx=40793, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32538,8237,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1085MiB/s (1137MB/s), 1085MiB/s-1085MiB/s (1137MB/s-1137MB/s), io=31.8GiB (34.1GB), run=30002-30002msec WRITE: bw=275MiB/s (288MB/s), 275MiB/s-275MiB/s (288MB/s-288MB/s), io=8237MiB (8637MB), run=30002-30002msec Disk stats (read/write): vdb: ios=32384/8197, merge=0/0, ticks=1570/16406, in_queue=27684, util=92.37% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=962980: Fri Jan 7 14:25:46 2022 read: IOPS=1000, BW=1000MiB/s (1049MB/s)(29.4GiB/30090msec) slat (usec): min=9, max=394, avg=19.23, stdev=10.35 clat (usec): min=78, max=281869, avg=24095.97, stdev=44304.04 lat (usec): min=117, max=281888, avg=24115.35, stdev=44303.96 clat percentiles (usec): | 1.00th=[ 111], 5.00th=[ 115], 10.00th=[ 119], 20.00th=[ 143], | 30.00th=[ 322], 40.00th=[ 355], 50.00th=[ 537], 60.00th=[ 971], | 70.00th=[ 1926], 80.00th=[ 64226], 90.00th=[101188], 95.00th=[123208], | 99.00th=[160433], 99.50th=[183501], 99.90th=[217056], 99.95th=[235930], | 99.99th=[254804] bw ( KiB/s): min=698368, max=1333248, per=100.00%, avg=1026753.02, stdev=124903.06, samples=60 iops : min= 682, max= 1302, avg=1002.65, stdev=122.01, samples=60 write: IOPS=252, BW=252MiB/s (265MB/s)(7596MiB/30090msec) slat (usec): min=59, max=696, avg=203.32, stdev=99.86 clat (msec): min=10, max=279, avg=102.25, stdev=28.33 lat (msec): min=11, max=279, avg=102.46, stdev=28.33 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 62], 10.00th=[ 71], 20.00th=[ 81], | 30.00th=[ 88], 40.00th=[ 94], 50.00th=[ 101], 60.00th=[ 106], | 70.00th=[ 113], 80.00th=[ 123], 90.00th=[ 136], 95.00th=[ 153], | 99.00th=[ 197], 99.50th=[ 211], 99.90th=[ 266], 99.95th=[ 275], | 99.99th=[ 279] bw ( KiB/s): min=225280, max=317440, per=100.00%, avg=258516.05, stdev=19008.58, samples=60 iops : min= 220, max= 310, avg=252.43, stdev=18.58, samples=60 lat (usec) : 100=0.02%, 250=21.55%, 500=17.57%, 750=5.33%, 1000=3.84% lat (msec) : 2=7.91%, 4=2.62%, 10=0.65%, 20=0.62%, 50=1.54% lat (msec) : 100=20.30%, 250=18.01%, 500=0.04% cpu : usr=5.30%, sys=2.85%, ctx=22607, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=30105,7596,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1000MiB/s (1049MB/s), 1000MiB/s-1000MiB/s (1049MB/s-1049MB/s), io=29.4GiB (31.6GB), run=30090-30090msec WRITE: bw=252MiB/s (265MB/s), 252MiB/s-252MiB/s (265MB/s-265MB/s), io=7596MiB (7965MB), run=30090-30090msec Disk stats (read/write): vdb: ios=30094/7585, merge=0/0, ticks=709086/770764, in_queue=1496798, util=99.64% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=963564: Fri Jan 7 14:26:23 2022 read: IOPS=1071, BW=1072MiB/s (1124MB/s)(31.4GiB/30003msec) clat (usec): min=134, max=3408, avg=296.50, stdev=347.31 lat (usec): min=134, max=3408, avg=296.65, stdev=347.32 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 147], 10.00th=[ 149], 20.00th=[ 155], | 30.00th=[ 159], 40.00th=[ 167], 50.00th=[ 176], 60.00th=[ 188], | 70.00th=[ 217], 80.00th=[ 351], 90.00th=[ 529], 95.00th=[ 791], | 99.00th=[ 2180], 99.50th=[ 2671], 99.90th=[ 3032], 99.95th=[ 3064], | 99.99th=[ 3130] bw ( MiB/s): min= 924, max= 1254, per=100.00%, avg=1071.70, stdev=70.85, samples=60 iops : min= 924, max= 1254, avg=1071.70, stdev=70.85, samples=60 write: IOPS=271, BW=271MiB/s (284MB/s)(8135MiB/30003msec) clat (usec): min=2367, max=11289, avg=2408.51, stdev=111.19 lat (usec): min=2425, max=11382, avg=2507.54, stdev=114.68 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2409], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2442], 95.00th=[ 2442], | 99.00th=[ 2507], 99.50th=[ 2704], 99.90th=[ 3097], 99.95th=[ 3589], | 99.99th=[11338] bw ( KiB/s): min=251904, max=292864, per=100.00%, avg=277640.53, stdev=8748.99, samples=60 iops : min= 246, max= 286, avg=271.13, stdev= 8.54, samples=60 lat (usec) : 250=58.24%, 500=12.63%, 750=4.31%, 1000=1.86% lat (msec) : 2=1.51%, 4=21.44%, 10=0.01%, 20=0.01% cpu : usr=3.19%, sys=4.46%, ctx=40299, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32151,8135,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1072MiB/s (1124MB/s), 1072MiB/s-1072MiB/s (1124MB/s-1124MB/s), io=31.4GiB (33.7GB), run=30003-30003msec WRITE: bw=271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=8135MiB (8530MB), run=30003-30003msec Disk stats (read/write): vdb: ios=31893/8079, merge=0/0, ticks=1608/16177, in_queue=27363, util=91.39% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=964377: Fri Jan 7 14:27:00 2022 read: IOPS=995, BW=996MiB/s (1044MB/s)(29.4GiB/30249msec) slat (usec): min=9, max=1143, avg=18.30, stdev=15.93 clat (usec): min=59, max=434037, avg=27113.11, stdev=51490.89 lat (usec): min=119, max=434060, avg=27131.54, stdev=51490.79 clat percentiles (usec): | 1.00th=[ 113], 5.00th=[ 117], 10.00th=[ 120], 20.00th=[ 153], | 30.00th=[ 334], 40.00th=[ 383], 50.00th=[ 725], 60.00th=[ 1319], | 70.00th=[ 3621], 80.00th=[ 65799], 90.00th=[105382], 95.00th=[131597], | 99.00th=[212861], 99.50th=[270533], 99.90th=[350225], 99.95th=[371196], | 99.99th=[429917] bw ( KiB/s): min=569344, max=1783278, per=100.00%, avg=1027008.28, stdev=182453.89, samples=60 iops : min= 556, max= 1741, avg=1002.92, stdev=178.13, samples=60 write: IOPS=251, BW=251MiB/s (263MB/s)(7596MiB/30249msec) slat (usec): min=62, max=1143, avg=212.99, stdev=102.46 clat (msec): min=29, max=628, avg=290.34, stdev=73.60 lat (msec): min=29, max=628, avg=290.55, stdev=73.59 clat percentiles (msec): | 1.00th=[ 74], 5.00th=[ 153], 10.00th=[ 207], 20.00th=[ 247], | 30.00th=[ 268], 40.00th=[ 284], 50.00th=[ 296], 60.00th=[ 309], | 70.00th=[ 321], 80.00th=[ 338], 90.00th=[ 359], 95.00th=[ 397], | 99.00th=[ 514], 99.50th=[ 542], 99.90th=[ 625], 99.95th=[ 625], | 99.99th=[ 625] bw ( KiB/s): min=108544, max=406316, per=99.92%, avg=256927.38, stdev=37915.07, samples=60 iops : min= 106, max= 396, avg=250.87, stdev=36.97, samples=60 lat (usec) : 100=0.03%, 250=19.60%, 500=15.97%, 750=4.86%, 1000=3.78% lat (msec) : 2=8.40%, 4=3.54%, 10=1.41%, 20=1.07%, 50=2.91% lat (msec) : 100=9.68%, 250=12.45%, 500=16.06%, 750=0.25% cpu : usr=5.46%, sys=2.69%, ctx=20940, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30113,7596,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=996MiB/s (1044MB/s), 996MiB/s-996MiB/s (1044MB/s-1044MB/s), io=29.4GiB (31.6GB), run=30249-30249msec WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=7596MiB (7965MB), run=30249-30249msec Disk stats (read/write): vdb: ios=30099/7538, merge=0/0, ticks=791277/2174957, in_queue=2993131, util=99.56% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=965444: Fri Jan 7 14:27:37 2022 read: IOPS=1078, BW=1079MiB/s (1131MB/s)(31.6GiB/30003msec) clat (usec): min=125, max=3142, avg=291.29, stdev=349.85 lat (usec): min=125, max=3142, avg=291.43, stdev=349.85 clat percentiles (usec): | 1.00th=[ 137], 5.00th=[ 141], 10.00th=[ 143], 20.00th=[ 147], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 169], 60.00th=[ 182], | 70.00th=[ 212], 80.00th=[ 343], 90.00th=[ 519], 95.00th=[ 791], | 99.00th=[ 2147], 99.50th=[ 2671], 99.90th=[ 3032], 99.95th=[ 3032], | 99.99th=[ 3097] bw ( MiB/s): min= 928, max= 1296, per=100.00%, avg=1091.40, stdev=72.72, samples=60 iops : min= 928, max= 1296, avg=1090.98, stdev=72.71, samples=60 write: IOPS=273, BW=273MiB/s (286MB/s)(8194MiB/30003msec) clat (usec): min=2367, max=3884, avg=2402.64, stdev=45.42 lat (usec): min=2413, max=4004, avg=2502.48, stdev=52.54 clat percentiles (usec): | 1.00th=[ 2376], 5.00th=[ 2376], 10.00th=[ 2376], 20.00th=[ 2376], | 30.00th=[ 2376], 40.00th=[ 2409], 50.00th=[ 2409], 60.00th=[ 2409], | 70.00th=[ 2409], 80.00th=[ 2409], 90.00th=[ 2409], 95.00th=[ 2442], | 99.00th=[ 2474], 99.50th=[ 2606], 99.90th=[ 3032], 99.95th=[ 3163], | 99.99th=[ 3884] bw ( KiB/s): min=259634, max=302403, per=100.00%, avg=282941.00, stdev=9778.31, samples=60 iops : min= 253, max= 295, avg=275.87, stdev= 9.59, samples=60 lat (usec) : 250=58.53%, 500=12.62%, 750=3.99%, 1000=1.83% lat (msec) : 2=1.55%, 4=21.47% cpu : usr=3.31%, sys=3.68%, ctx=40565, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=32361,8194,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1079MiB/s (1131MB/s), 1079MiB/s-1079MiB/s (1131MB/s-1131MB/s), io=31.6GiB (33.9GB), run=30003-30003msec WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=8194MiB (8592MB), run=30003-30003msec Disk stats (read/write): vdb: ios=32350/8191, merge=0/0, ticks=1625/16390, in_queue=27759, util=91.95% 30s: (g=0): rw=randrw, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966050: Fri Jan 7 14:28:14 2022 read: IOPS=998, BW=999MiB/s (1047MB/s)(29.6GiB/30323msec) slat (usec): min=10, max=405402, avg=750.91, stdev=5523.61 clat (msec): min=15, max=569, avg=90.95, stdev=66.55 lat (msec): min=15, max=573, avg=91.70, stdev=66.76 clat percentiles (msec): | 1.00th=[ 23], 5.00th=[ 31], 10.00th=[ 38], 20.00th=[ 47], | 30.00th=[ 56], 40.00th=[ 63], 50.00th=[ 70], 60.00th=[ 79], | 70.00th=[ 94], 80.00th=[ 129], 90.00th=[ 171], 95.00th=[ 207], | 99.00th=[ 388], 99.50th=[ 439], 99.90th=[ 514], 99.95th=[ 535], | 99.99th=[ 558] bw ( KiB/s): min=227328, max=1728512, per=100.00%, avg=1030060.52, stdev=244942.28, samples=60 iops : min= 222, max= 1688, avg=1005.90, stdev=239.19, samples=60 write: IOPS=251, BW=252MiB/s (264MB/s)(7637MiB/30323msec) slat (usec): min=75, max=189521, avg=939.14, stdev=3492.24 clat (msec): min=87, max=974, avg=428.84, stdev=105.76 lat (msec): min=87, max=974, avg=429.78, stdev=105.81 clat percentiles (msec): | 1.00th=[ 138], 5.00th=[ 234], 10.00th=[ 309], 20.00th=[ 372], | 30.00th=[ 397], 40.00th=[ 418], 50.00th=[ 435], 60.00th=[ 447], | 70.00th=[ 464], 80.00th=[ 489], 90.00th=[ 527], 95.00th=[ 584], | 99.00th=[ 768], 99.50th=[ 827], 99.90th=[ 919], 99.95th=[ 944], | 99.99th=[ 978] bw ( KiB/s): min=145408, max=361772, per=99.84%, avg=257481.55, stdev=36035.05, samples=60 iops : min= 142, max= 353, avg=251.43, stdev=35.18, samples=60 lat (msec) : 20=0.27%, 50=18.66%, 100=39.14%, 250=20.75%, 500=17.90% lat (msec) : 750=3.05%, 1000=0.23% cpu : usr=8.46%, sys=2.37%, ctx=3374, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=30287,7637,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=999MiB/s (1047MB/s), 999MiB/s-999MiB/s (1047MB/s-1047MB/s), io=29.6GiB (31.8GB), run=30323-30323msec WRITE: bw=252MiB/s (264MB/s), 252MiB/s-252MiB/s (264MB/s-264MB/s), io=7637MiB (8008MB), run=30323-30323msec Disk stats (read/write): vdb: ios=30287/7621, merge=0/0, ticks=912830/2797200, in_queue=3733149, util=99.38% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=966912: Fri Jan 7 14:28:51 2022 read: IOPS=201, BW=402MiB/s (422MB/s)(11.8GiB/30020msec) clat (usec): min=258, max=5043, avg=610.49, stdev=562.62 lat (usec): min=258, max=5043, avg=610.68, stdev=562.63 clat percentiles (usec): | 1.00th=[ 273], 5.00th=[ 281], 10.00th=[ 289], 20.00th=[ 297], | 30.00th=[ 322], 40.00th=[ 363], 50.00th=[ 433], 60.00th=[ 553], | 70.00th=[ 693], 80.00th=[ 758], 90.00th=[ 947], 95.00th=[ 1336], | 99.00th=[ 4113], 99.50th=[ 4228], 99.90th=[ 4752], 99.95th=[ 4817], | 99.99th=[ 5014] bw ( KiB/s): min=229376, max=610304, per=100.00%, avg=412205.75, stdev=89463.78, samples=60 iops : min= 112, max= 298, avg=201.27, stdev=43.69, samples=60 write: IOPS=52, BW=105MiB/s (110MB/s)(3160MiB/30020msec) clat (usec): min=4691, max=34972, avg=16370.16, stdev=4628.42 lat (usec): min=4871, max=35164, avg=16654.96, stdev=4628.87 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 8979], 10.00th=[12649], 20.00th=[13304], | 30.00th=[13566], 40.00th=[13829], 50.00th=[16188], 60.00th=[17957], | 70.00th=[19006], 80.00th=[19530], 90.00th=[21365], 95.00th=[25035], | 99.00th=[28967], 99.50th=[30540], 99.90th=[32637], 99.95th=[34866], | 99.99th=[34866] bw ( KiB/s): min=90112, max=163840, per=100.00%, avg=107796.35, stdev=11321.60, samples=60 iops : min= 44, max= 80, avg=52.63, stdev= 5.53, samples=60 lat (usec) : 500=44.85%, 750=17.88%, 1000=9.50% lat (msec) : 2=4.91%, 4=1.02%, 10=2.28%, 20=16.38%, 50=3.16% cpu : usr=1.67%, sys=1.47%, ctx=7622, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=6038,1580,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=402MiB/s (422MB/s), 402MiB/s-402MiB/s (422MB/s-422MB/s), io=11.8GiB (12.7GB), run=30020-30020msec WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=3160MiB (3314MB), run=30020-30020msec Disk stats (read/write): vdb: ios=11966/3144, merge=0/0, ticks=1034/49571, in_queue=56695, util=96.44% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=967489: Fri Jan 7 14:29:28 2022 read: IOPS=353, BW=707MiB/s (742MB/s)(20.7GiB/30001msec) slat (usec): min=55, max=253, avg=75.95, stdev=14.47 clat (usec): min=109, max=4985, avg=518.36, stdev=562.11 lat (usec): min=280, max=5051, avg=594.56, stdev=563.03 clat percentiles (usec): | 1.00th=[ 227], 5.00th=[ 233], 10.00th=[ 237], 20.00th=[ 247], | 30.00th=[ 265], 40.00th=[ 297], 50.00th=[ 338], 60.00th=[ 404], | 70.00th=[ 510], 80.00th=[ 635], 90.00th=[ 857], 95.00th=[ 1270], | 99.00th=[ 4047], 99.50th=[ 4228], 99.90th=[ 4883], 99.95th=[ 4948], | 99.99th=[ 4948] bw ( KiB/s): min=344064, max=1355776, per=100.00%, avg=730035.29, stdev=301031.91, samples=59 iops : min= 168, max= 662, avg=356.44, stdev=146.99, samples=59 write: IOPS=89, BW=180MiB/s (189MB/s)(5396MiB/30001msec) slat (usec): min=135, max=865, avg=281.74, stdev=100.40 clat (usec): min=4648, max=34190, avg=8488.68, stdev=5204.95 lat (usec): min=4800, max=34484, avg=8770.73, stdev=5239.08 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4752], 60.00th=[ 6980], | 70.00th=[11600], 80.00th=[13173], 90.00th=[16581], 95.00th=[18744], | 99.00th=[23725], 99.50th=[26346], 99.90th=[30540], 99.95th=[33817], | 99.99th=[34341] bw ( KiB/s): min=86016, max=311919, per=100.00%, avg=185515.20, stdev=79740.19, samples=59 iops : min= 42, max= 152, avg=90.58, stdev=38.93, samples=59 lat (usec) : 250=18.17%, 500=37.21%, 750=13.13%, 1000=5.45% lat (msec) : 2=3.76%, 4=0.94%, 10=14.10%, 20=6.73%, 50=0.50% cpu : usr=2.24%, sys=3.60%, ctx=13349, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=10612,2698,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=707MiB/s (742MB/s), 707MiB/s-707MiB/s (742MB/s-742MB/s), io=20.7GiB (22.3GB), run=30001-30001msec WRITE: bw=180MiB/s (189MB/s), 180MiB/s-180MiB/s (189MB/s-189MB/s), io=5396MiB (5658MB), run=30001-30001msec Disk stats (read/write): vdb: ios=21158/5378, merge=0/0, ticks=1806/42271, in_queue=55077, util=95.03% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=968308: Fri Jan 7 14:30:05 2022 read: IOPS=453, BW=906MiB/s (951MB/s)(26.6GiB/30021msec) clat (usec): min=241, max=5024, avg=539.69, stdev=576.12 lat (usec): min=241, max=5024, avg=539.86, stdev=576.13 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 258], 10.00th=[ 262], 20.00th=[ 273], | 30.00th=[ 285], 40.00th=[ 314], 50.00th=[ 355], 60.00th=[ 424], | 70.00th=[ 515], 80.00th=[ 635], 90.00th=[ 889], 95.00th=[ 1319], | 99.00th=[ 4080], 99.50th=[ 4424], 99.90th=[ 4883], 99.95th=[ 4948], | 99.99th=[ 5014] bw ( KiB/s): min=364544, max=1376256, per=100.00%, avg=928940.78, stdev=301065.59, samples=60 iops : min= 178, max= 672, avg=453.58, stdev=147.00, samples=60 write: IOPS=113, BW=228MiB/s (239MB/s)(6844MiB/30021msec) clat (usec): min=4653, max=31033, avg=6397.08, stdev=3656.10 lat (usec): min=4758, max=31268, avg=6616.96, stdev=3678.73 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4752], 80.00th=[ 7308], 90.00th=[12256], 95.00th=[14484], | 99.00th=[19530], 99.50th=[22152], 99.90th=[29492], 99.95th=[30540], | 99.99th=[31065] bw ( KiB/s): min=106496, max=327680, per=100.00%, avg=233550.12, stdev=76620.47, samples=60 iops : min= 52, max= 160, avg=114.03, stdev=37.41, samples=60 lat (usec) : 250=0.53%, 500=54.45%, 750=13.64%, 1000=5.03% lat (msec) : 2=4.17%, 4=0.95%, 10=17.90%, 20=3.14%, 50=0.18% cpu : usr=2.65%, sys=2.85%, ctx=17040, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=13607,3422,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=906MiB/s (951MB/s), 906MiB/s-906MiB/s (951MB/s-951MB/s), io=26.6GiB (28.5GB), run=30021-30021msec WRITE: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=6844MiB (7176MB), run=30021-30021msec Disk stats (read/write): vdb: ios=27091/6824, merge=0/0, ticks=2303/33630, in_queue=49282, util=94.27% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=969564: Fri Jan 7 14:30:42 2022 read: IOPS=575, BW=1151MiB/s (1207MB/s)(33.7GiB/30010msec) slat (usec): min=19, max=289, avg=35.21, stdev=15.38 clat (usec): min=169, max=88115, avg=10740.57, stdev=11902.52 lat (usec): min=213, max=88141, avg=10775.94, stdev=11902.66 clat percentiles (usec): | 1.00th=[ 194], 5.00th=[ 206], 10.00th=[ 237], 20.00th=[ 635], | 30.00th=[ 750], 40.00th=[ 1434], 50.00th=[ 6325], 60.00th=[12649], | 70.00th=[16712], 80.00th=[20841], 90.00th=[26084], 95.00th=[31327], | 99.00th=[51643], 99.50th=[57934], 99.90th=[65799], 99.95th=[74974], | 99.99th=[87557] bw ( MiB/s): min= 324, max= 1900, per=100.00%, avg=1150.80, stdev=453.62, samples=60 iops : min= 162, max= 950, avg=575.38, stdev=226.82, samples=60 write: IOPS=145, BW=292MiB/s (306MB/s)(8760MiB/30010msec) slat (usec): min=125, max=1034, avg=477.24, stdev=196.27 clat (msec): min=4, max=123, avg=25.55, stdev=15.87 lat (msec): min=5, max=123, avg=26.02, stdev=15.90 clat percentiles (msec): | 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 13], 20.00th=[ 16], | 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 22], 60.00th=[ 24], | 70.00th=[ 27], 80.00th=[ 31], 90.00th=[ 42], 95.00th=[ 61], | 99.00th=[ 90], 99.50th=[ 105], 99.90th=[ 117], 99.95th=[ 118], | 99.99th=[ 124] bw ( KiB/s): min=114917, max=405504, per=99.95%, avg=298750.92, stdev=108630.05, samples=60 iops : min= 56, max= 198, avg=145.87, stdev=53.04, samples=60 lat (usec) : 250=8.59%, 500=3.74%, 750=11.58%, 1000=3.58% lat (msec) : 2=6.99%, 4=2.82%, 10=6.74%, 20=27.02%, 50=26.46% lat (msec) : 100=2.35%, 250=0.12% cpu : usr=6.90%, sys=2.68%, ctx=12589, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=17267,4380,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1151MiB/s (1207MB/s), 1151MiB/s-1151MiB/s (1207MB/s-1207MB/s), io=33.7GiB (36.2GB), run=30010-30010msec WRITE: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=8760MiB (9186MB), run=30010-30010msec Disk stats (read/write): vdb: ios=34454/8732, merge=0/0, ticks=261483/216367, in_queue=498869, util=99.66% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=970155: Fri Jan 7 14:31:19 2022 read: IOPS=562, BW=1124MiB/s (1179MB/s)(32.9GiB/30005msec) clat (usec): min=234, max=5339, avg=523.40, stdev=576.75 lat (usec): min=234, max=5339, avg=523.57, stdev=576.76 clat percentiles (usec): | 1.00th=[ 249], 5.00th=[ 255], 10.00th=[ 260], 20.00th=[ 269], | 30.00th=[ 281], 40.00th=[ 306], 50.00th=[ 343], 60.00th=[ 392], | 70.00th=[ 478], 80.00th=[ 603], 90.00th=[ 857], 95.00th=[ 1287], | 99.00th=[ 4080], 99.50th=[ 4621], 99.90th=[ 4948], 99.95th=[ 5014], | 99.99th=[ 5276] bw ( MiB/s): min= 864, max= 1372, per=100.00%, avg=1124.57, stdev=99.69, samples=60 iops : min= 432, max= 686, avg=562.28, stdev=49.84, samples=60 write: IOPS=142, BW=286MiB/s (300MB/s)(8572MiB/30005msec) clat (usec): min=4654, max=9247, avg=4726.08, stdev=179.33 lat (usec): min=4753, max=9808, avg=4931.13, stdev=190.31 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4686], 80.00th=[ 4686], 90.00th=[ 4752], 95.00th=[ 4948], | 99.00th=[ 5604], 99.50th=[ 5800], 99.90th=[ 6325], 99.95th=[ 6390], | 99.99th=[ 9241] bw ( KiB/s): min=270336, max=331776, per=100.00%, avg=292531.68, stdev=12899.56, samples=60 iops : min= 132, max= 162, avg=142.83, stdev= 6.31, samples=60 lat (usec) : 250=1.01%, 500=56.62%, 750=11.61%, 1000=4.68% lat (msec) : 2=3.72%, 4=1.02%, 10=21.36% cpu : usr=3.22%, sys=3.31%, ctx=21169, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=16868,4286,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1124MiB/s (1179MB/s), 1124MiB/s-1124MiB/s (1179MB/s-1179MB/s), io=32.9GiB (35.4GB), run=30005-30005msec WRITE: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=8572MiB (8988MB), run=30005-30005msec Disk stats (read/write): vdb: ios=33500/8512, merge=0/0, ticks=2792/27616, in_queue=47120, util=92.98% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=971110: Fri Jan 7 14:31:56 2022 read: IOPS=695, BW=1391MiB/s (1459MB/s)(40.8GiB/30058msec) slat (usec): min=19, max=739, avg=33.51, stdev=30.97 clat (usec): min=120, max=221120, avg=36837.62, stdev=37224.76 lat (usec): min=208, max=221150, avg=36871.29, stdev=37224.97 clat percentiles (usec): | 1.00th=[ 196], 5.00th=[ 212], 10.00th=[ 529], 20.00th=[ 848], | 30.00th=[ 1991], 40.00th=[ 4047], 50.00th=[ 41157], 60.00th=[ 49021], | 70.00th=[ 57934], 80.00th=[ 72877], 90.00th=[ 83362], 95.00th=[ 96994], | 99.00th=[135267], 99.50th=[158335], 99.90th=[185598], 99.95th=[196084], | 99.99th=[212861] bw ( MiB/s): min= 612, max= 1856, per=100.00%, avg=1391.71, stdev=257.46, samples=60 iops : min= 306, max= 928, avg=695.82, stdev=128.72, samples=60 write: IOPS=175, BW=351MiB/s (368MB/s)(10.3GiB/30058msec) slat (usec): min=118, max=1584, avg=515.25, stdev=207.56 clat (msec): min=20, max=300, avg=138.18, stdev=32.28 lat (msec): min=21, max=300, avg=138.70, stdev=32.26 clat percentiles (msec): | 1.00th=[ 61], 5.00th=[ 89], 10.00th=[ 102], 20.00th=[ 113], | 30.00th=[ 124], 40.00th=[ 132], 50.00th=[ 140], 60.00th=[ 146], | 70.00th=[ 150], 80.00th=[ 161], 90.00th=[ 174], 95.00th=[ 190], | 99.00th=[ 243], 99.50th=[ 255], 99.90th=[ 292], 99.95th=[ 296], | 99.99th=[ 300] bw ( KiB/s): min=188416, max=409600, per=99.91%, avg=359143.92, stdev=55448.84, samples=60 iops : min= 92, max= 200, avg=175.33, stdev=27.07, samples=60 lat (usec) : 250=5.79%, 500=2.06%, 750=7.00%, 1000=2.27% lat (msec) : 2=6.86%, 4=7.81%, 10=4.76%, 20=0.65%, 50=11.87% lat (msec) : 100=28.99%, 250=21.81%, 500=0.13% cpu : usr=8.91%, sys=3.01%, ctx=10194, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=20906,5276,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1391MiB/s (1459MB/s), 1391MiB/s-1391MiB/s (1459MB/s-1459MB/s), io=40.8GiB (43.8GB), run=30058-30058msec WRITE: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=10.3GiB (11.1GB), run=30058-30058msec Disk stats (read/write): vdb: ios=41744/10520, merge=0/0, ticks=1091195/1437969, in_queue=2560799, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972157: Fri Jan 7 14:32:33 2022 read: IOPS=552, BW=1105MiB/s (1158MB/s)(32.4GiB/30001msec) clat (usec): min=253, max=5784, avg=544.12, stdev=581.24 lat (usec): min=254, max=5784, avg=544.29, stdev=581.24 clat percentiles (usec): | 1.00th=[ 265], 5.00th=[ 273], 10.00th=[ 277], 20.00th=[ 289], | 30.00th=[ 302], 40.00th=[ 326], 50.00th=[ 359], 60.00th=[ 412], | 70.00th=[ 498], 80.00th=[ 627], 90.00th=[ 881], 95.00th=[ 1303], | 99.00th=[ 4113], 99.50th=[ 4621], 99.90th=[ 4948], 99.95th=[ 5014], | 99.99th=[ 5735] bw ( MiB/s): min= 863, max= 1382, per=100.00%, avg=1117.80, stdev=106.31, samples=59 iops : min= 431, max= 691, avg=558.61, stdev=53.15, samples=59 write: IOPS=140, BW=281MiB/s (295MB/s)(8436MiB/30001msec) clat (usec): min=4665, max=7232, avg=4762.57, stdev=202.20 lat (usec): min=4770, max=7445, avg=4964.74, stdev=207.78 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4686], 40.00th=[ 4686], 50.00th=[ 4686], 60.00th=[ 4686], | 70.00th=[ 4752], 80.00th=[ 4752], 90.00th=[ 4817], 95.00th=[ 5145], | 99.00th=[ 5800], 99.50th=[ 6063], 99.90th=[ 6325], 99.95th=[ 6390], | 99.99th=[ 7242] bw ( KiB/s): min=241922, max=315392, per=100.00%, avg=290045.15, stdev=12739.24, samples=59 iops : min= 118, max= 154, avg=141.41, stdev= 6.21, samples=59 lat (usec) : 500=56.14%, 750=12.40%, 1000=5.20% lat (msec) : 2=3.80%, 4=1.06%, 10=21.39% cpu : usr=3.14%, sys=3.81%, ctx=20803, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=16573,4218,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1105MiB/s (1158MB/s), 1105MiB/s-1105MiB/s (1158MB/s-1158MB/s), io=32.4GiB (34.8GB), run=30001-30001msec WRITE: bw=281MiB/s (295MB/s), 281MiB/s-281MiB/s (295MB/s-295MB/s), io=8436MiB (8846MB), run=30001-30001msec Disk stats (read/write): vdb: ios=33140/8430, merge=0/0, ticks=2794/33201, in_queue=51914, util=92.28% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=972681: Fri Jan 7 14:33:10 2022 read: IOPS=724, BW=1449MiB/s (1519MB/s)(42.6GiB/30089msec) slat (usec): min=20, max=221446, avg=977.80, stdev=4388.43 clat (msec): min=8, max=383, avg=67.64, stdev=50.05 lat (msec): min=8, max=457, avg=68.61, stdev=50.32 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 20], 10.00th=[ 23], 20.00th=[ 30], | 30.00th=[ 35], 40.00th=[ 40], 50.00th=[ 47], 60.00th=[ 72], | 70.00th=[ 88], 80.00th=[ 102], 90.00th=[ 121], 95.00th=[ 153], | 99.00th=[ 268], 99.50th=[ 309], 99.90th=[ 334], 99.95th=[ 347], | 99.99th=[ 368] bw ( MiB/s): min= 704, max= 2000, per=99.98%, avg=1448.55, stdev=289.45, samples=60 iops : min= 352, max= 1000, avg=724.27, stdev=144.72, samples=60 write: IOPS=182, BW=365MiB/s (382MB/s)(10.7GiB/30089msec) slat (usec): min=343, max=251895, avg=1583.04, stdev=7075.79 clat (msec): min=12, max=555, avg=273.91, stdev=60.29 lat (msec): min=41, max=568, avg=275.50, stdev=59.97 clat percentiles (msec): | 1.00th=[ 99], 5.00th=[ 163], 10.00th=[ 213], 20.00th=[ 251], | 30.00th=[ 262], 40.00th=[ 271], 50.00th=[ 275], 60.00th=[ 284], | 70.00th=[ 292], 80.00th=[ 300], 90.00th=[ 321], 95.00th=[ 359], | 99.00th=[ 489], 99.50th=[ 514], 99.90th=[ 542], 99.95th=[ 550], | 99.99th=[ 558] bw ( KiB/s): min=159744, max=512000, per=99.69%, avg=372177.73, stdev=54999.20, samples=60 iops : min= 78, max= 250, avg=181.72, stdev=26.86, samples=60 lat (msec) : 10=0.03%, 20=4.96%, 50=36.57%, 100=22.08%, 250=19.27% lat (msec) : 500=16.95%, 750=0.14% cpu : usr=12.19%, sys=3.41%, ctx=4440, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=21796,5485,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1449MiB/s (1519MB/s), 1449MiB/s-1449MiB/s (1519MB/s-1519MB/s), io=42.6GiB (45.7GB), run=30089-30089msec WRITE: bw=365MiB/s (382MB/s), 365MiB/s-365MiB/s (382MB/s-382MB/s), io=10.7GiB (11.5GB), run=30089-30089msec Disk stats (read/write): vdb: ios=43506/10926, merge=0/0, ticks=1067909/2616117, in_queue=3726729, util=99.38% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=973640: Fri Jan 7 14:33:47 2022 read: IOPS=546, BW=1093MiB/s (1146MB/s)(32.0GiB/30003msec) clat (usec): min=271, max=5287, avg=560.45, stdev=570.20 lat (usec): min=271, max=5287, avg=560.64, stdev=570.21 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 293], 10.00th=[ 297], 20.00th=[ 310], | 30.00th=[ 322], 40.00th=[ 351], 50.00th=[ 379], 60.00th=[ 429], | 70.00th=[ 510], 80.00th=[ 635], 90.00th=[ 898], 95.00th=[ 1352], | 99.00th=[ 4146], 99.50th=[ 4359], 99.90th=[ 4948], 99.95th=[ 5014], | 99.99th=[ 5080] bw ( MiB/s): min= 860, max= 1488, per=100.00%, avg=1093.43, stdev=122.61, samples=60 iops : min= 430, max= 744, avg=546.72, stdev=61.31, samples=60 write: IOPS=138, BW=278MiB/s (291MB/s)(8330MiB/30003msec) clat (usec): min=4683, max=7292, avg=4792.53, stdev=215.85 lat (usec): min=4789, max=7473, avg=4986.04, stdev=218.50 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 4686], 10.00th=[ 4686], 20.00th=[ 4686], | 30.00th=[ 4752], 40.00th=[ 4752], 50.00th=[ 4752], 60.00th=[ 4752], | 70.00th=[ 4752], 80.00th=[ 4752], 90.00th=[ 4817], 95.00th=[ 5145], | 99.00th=[ 5866], 99.50th=[ 6128], 99.90th=[ 6652], 99.95th=[ 6915], | 99.99th=[ 7308] bw ( KiB/s): min=245760, max=311296, per=99.99%, avg=284272.38, stdev=13589.85, samples=60 iops : min= 120, max= 152, avg=138.80, stdev= 6.63, samples=60 lat (usec) : 500=54.95%, 750=13.29%, 1000=5.22% lat (msec) : 2=4.22%, 4=0.96%, 10=21.36% cpu : usr=3.06%, sys=3.89%, ctx=20582, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=16401,4165,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1093MiB/s (1146MB/s), 1093MiB/s-1093MiB/s (1146MB/s-1146MB/s), io=32.0GiB (34.4GB), run=30003-30003msec WRITE: bw=278MiB/s (291MB/s), 278MiB/s-278MiB/s (291MB/s-291MB/s), io=8330MiB (8735MB), run=30003-30003msec Disk stats (read/write): vdb: ios=32730/8294, merge=0/0, ticks=2740/33685, in_queue=52903, util=92.31% 30s: (g=0): rw=randrw, bs=(R) 2048KiB-2048KiB, (W) 2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=974265: Fri Jan 7 14:34:24 2022 read: IOPS=737, BW=1474MiB/s (1546MB/s)(43.3GiB/30080msec) slat (usec): min=22, max=221549, avg=957.36, stdev=4206.49 clat (msec): min=10, max=517, avg=172.70, stdev=59.49 lat (msec): min=69, max=629, avg=173.66, stdev=59.63 clat percentiles (msec): | 1.00th=[ 94], 5.00th=[ 109], 10.00th=[ 117], 20.00th=[ 129], | 30.00th=[ 138], 40.00th=[ 146], 50.00th=[ 159], 60.00th=[ 176], | 70.00th=[ 192], 80.00th=[ 209], 90.00th=[ 234], 95.00th=[ 284], | 99.00th=[ 393], 99.50th=[ 443], 99.90th=[ 502], 99.95th=[ 502], | 99.99th=[ 518] bw ( MiB/s): min= 501, max= 2116, per=99.64%, avg=1469.02, stdev=238.72, samples=60 iops : min= 250, max= 1058, avg=734.48, stdev=119.40, samples=60 write: IOPS=185, BW=372MiB/s (390MB/s)(10.9GiB/30080msec) slat (usec): min=380, max=219880, avg=1564.48, stdev=5250.71 clat (msec): min=96, max=737, avg=383.28, stdev=74.84 lat (msec): min=96, max=738, avg=384.85, stdev=74.88 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 243], 10.00th=[ 313], 20.00th=[ 351], | 30.00th=[ 368], 40.00th=[ 376], 50.00th=[ 384], 60.00th=[ 393], | 70.00th=[ 401], 80.00th=[ 414], 90.00th=[ 443], 95.00th=[ 542], | 99.00th=[ 617], 99.50th=[ 634], 99.90th=[ 726], 99.95th=[ 735], | 99.99th=[ 735] bw ( KiB/s): min=225731, max=487424, per=99.22%, avg=377833.95, stdev=50659.51, samples=60 iops : min= 110, max= 238, avg=184.47, stdev=24.74, samples=60 lat (msec) : 20=0.01%, 100=1.77%, 250=73.62%, 500=23.24%, 750=1.36% cpu : usr=13.03%, sys=3.45%, ctx=4625, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=22174,5593,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1474MiB/s (1546MB/s), 1474MiB/s-1474MiB/s (1546MB/s-1546MB/s), io=43.3GiB (46.5GB), run=30080-30080msec WRITE: bw=372MiB/s (390MB/s), 372MiB/s-372MiB/s (390MB/s-390MB/s), io=10.9GiB (11.7GB), run=30080-30080msec Disk stats (read/write): vdb: ios=44174/11122, merge=0/0, ticks=1002603/2679105, in_queue=3716327, util=99.04% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=975108: Fri Jan 7 14:35:01 2022 read: IOPS=138, BW=554MiB/s (581MB/s)(16.2GiB/30001msec) clat (usec): min=410, max=8797, avg=1034.79, stdev=1043.58 lat (usec): min=411, max=8797, avg=1035.00, stdev=1043.59 clat percentiles (usec): | 1.00th=[ 424], 5.00th=[ 441], 10.00th=[ 453], 20.00th=[ 478], | 30.00th=[ 523], 40.00th=[ 594], 50.00th=[ 701], 60.00th=[ 857], | 70.00th=[ 1172], 80.00th=[ 1352], 90.00th=[ 1598], 95.00th=[ 2343], | 99.00th=[ 7898], 99.50th=[ 8029], 99.90th=[ 8717], 99.95th=[ 8717], | 99.99th=[ 8848] bw ( KiB/s): min=270336, max=1255887, per=100.00%, avg=567212.80, stdev=168128.42, samples=60 iops : min= 66, max= 306, avg=138.47, stdev=41.01, samples=60 write: IOPS=36, BW=147MiB/s (154MB/s)(4408MiB/30001msec) clat (usec): min=9204, max=67694, avg=22713.37, stdev=7337.28 lat (usec): min=9502, max=68906, avg=23311.05, stdev=7351.21 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9765], 10.00th=[12256], 20.00th=[17957], | 30.00th=[19530], 40.00th=[20317], 50.00th=[22938], 60.00th=[24249], | 70.00th=[25035], 80.00th=[28181], 90.00th=[32900], 95.00th=[34341], | 99.00th=[42206], 99.50th=[44303], 99.90th=[53216], 99.95th=[67634], | 99.99th=[67634] bw ( KiB/s): min=131072, max=237568, per=99.92%, avg=150334.42, stdev=21064.67, samples=60 iops : min= 32, max= 58, avg=36.70, stdev= 5.14, samples=60 lat (usec) : 500=20.22%, 750=22.01%, 1000=9.87% lat (msec) : 2=21.92%, 4=2.70%, 10=3.58%, 20=6.13%, 50=13.53% lat (msec) : 100=0.04% cpu : usr=2.36%, sys=1.67%, ctx=5261, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=4154,1102,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=554MiB/s (581MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), io=16.2GiB (17.4GB), run=30001-30001msec WRITE: bw=147MiB/s (154MB/s), 147MiB/s-147MiB/s (154MB/s-154MB/s), io=4408MiB (4622MB), run=30001-30001msec Disk stats (read/write): vdb: ios=16536/4384, merge=0/0, ticks=5461/92087, in_queue=106733, util=96.96% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=976334: Fri Jan 7 14:35:38 2022 read: IOPS=211, BW=846MiB/s (887MB/s)(24.8GiB/30008msec) slat (usec): min=73, max=308, avg=94.35, stdev=18.36 clat (usec): min=294, max=8758, avg=922.52, stdev=1062.48 lat (usec): min=417, max=8845, avg=1017.13, stdev=1063.85 clat percentiles (usec): | 1.00th=[ 347], 5.00th=[ 363], 10.00th=[ 375], 20.00th=[ 400], | 30.00th=[ 445], 40.00th=[ 515], 50.00th=[ 611], 60.00th=[ 742], | 70.00th=[ 914], 80.00th=[ 1188], 90.00th=[ 1467], 95.00th=[ 2212], | 99.00th=[ 7832], 99.50th=[ 8029], 99.90th=[ 8586], 99.95th=[ 8717], | 99.99th=[ 8717] bw ( KiB/s): min=286720, max=1507328, per=100.00%, avg=866850.13, stdev=292322.28, samples=60 iops : min= 70, max= 368, avg=211.63, stdev=71.37, samples=60 write: IOPS=54, BW=220MiB/s (230MB/s)(6596MiB/30008msec) slat (usec): min=262, max=1696, avg=603.37, stdev=251.00 clat (usec): min=9104, max=49301, avg=13666.30, stdev=6565.70 lat (usec): min=9452, max=49793, avg=14270.05, stdev=6604.92 clat percentiles (usec): | 1.00th=[ 9110], 5.00th=[ 9110], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9503], 50.00th=[ 9896], 60.00th=[10552], | 70.00th=[14615], 80.00th=[19530], 90.00th=[23725], 95.00th=[26870], | 99.00th=[32375], 99.50th=[35390], 99.90th=[43254], 99.95th=[49546], | 99.99th=[49546] bw ( KiB/s): min=122880, max=319488, per=99.97%, avg=225006.93, stdev=70526.18, samples=60 iops : min= 30, max= 78, avg=54.93, stdev=17.22, samples=60 lat (usec) : 500=30.57%, 750=17.55%, 1000=10.25% lat (msec) : 2=16.44%, 4=2.46%, 10=12.68%, 20=5.99%, 50=4.05% cpu : usr=3.10%, sys=2.58%, ctx=8011, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=6349,1649,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=846MiB/s (887MB/s), 846MiB/s-846MiB/s (887MB/s-887MB/s), io=24.8GiB (26.6GB), run=30008-30008msec WRITE: bw=220MiB/s (230MB/s), 220MiB/s-220MiB/s (230MB/s-230MB/s), io=6596MiB (6916MB), run=30008-30008msec Disk stats (read/write): vdb: ios=25288/6568, merge=0/0, ticks=6670/76768, in_queue=98165, util=95.86% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977095: Fri Jan 7 14:36:15 2022 read: IOPS=254, BW=1016MiB/s (1065MB/s)(29.8GiB/30009msec) clat (usec): min=388, max=8807, avg=947.01, stdev=1045.02 lat (usec): min=388, max=8808, avg=947.20, stdev=1045.03 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 416], 10.00th=[ 429], 20.00th=[ 449], | 30.00th=[ 494], 40.00th=[ 562], 50.00th=[ 652], 60.00th=[ 758], | 70.00th=[ 906], 80.00th=[ 1156], 90.00th=[ 1483], 95.00th=[ 2180], | 99.00th=[ 7898], 99.50th=[ 8160], 99.90th=[ 8717], 99.95th=[ 8717], | 99.99th=[ 8848] bw ( KiB/s): min=499712, max=1469306, per=100.00%, avg=1040873.48, stdev=231783.09, samples=60 iops : min= 122, max= 358, avg=254.10, stdev=56.57, samples=60 write: IOPS=64, BW=259MiB/s (272MB/s)(7780MiB/30009msec) clat (usec): min=9189, max=48805, avg=11216.38, stdev=4274.06 lat (usec): min=9424, max=49210, avg=11706.07, stdev=4302.64 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9372], 50.00th=[ 9634], 60.00th=[ 9896], | 70.00th=[10290], 80.00th=[10945], 90.00th=[16909], 95.00th=[21365], | 99.00th=[28967], 99.50th=[33162], 99.90th=[40109], 99.95th=[49021], | 99.99th=[49021] bw ( KiB/s): min=122880, max=327680, per=99.98%, avg=265434.47, stdev=55964.46, samples=60 iops : min= 30, max= 80, avg=64.80, stdev=13.67, samples=60 lat (usec) : 500=24.90%, 750=22.19%, 1000=12.50% lat (msec) : 2=15.46%, 4=2.48%, 10=14.71%, 20=6.45%, 50=1.33% cpu : usr=3.44%, sys=2.18%, ctx=9579, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=7623,1945,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1016MiB/s (1065MB/s), 1016MiB/s-1016MiB/s (1065MB/s-1065MB/s), io=29.8GiB (31.0GB), run=30009-30009msec WRITE: bw=259MiB/s (272MB/s), 259MiB/s-259MiB/s (272MB/s-272MB/s), io=7780MiB (8158MB), run=30009-30009msec Disk stats (read/write): vdb: ios=30400/7752, merge=0/0, ticks=6566/69932, in_queue=94031, util=95.61% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=10 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=977780: Fri Jan 7 14:36:52 2022 read: IOPS=316, BW=1266MiB/s (1328MB/s)(37.1GiB/30017msec) slat (usec): min=38, max=630, avg=64.71, stdev=30.25 clat (usec): min=323, max=104300, avg=20860.52, stdev=16293.93 lat (usec): min=395, max=104569, avg=20925.45, stdev=16293.91 clat percentiles (usec): | 1.00th=[ 367], 5.00th=[ 1012], 10.00th=[ 1270], 20.00th=[ 2671], | 30.00th=[ 8586], 40.00th=[ 14877], 50.00th=[ 20317], 60.00th=[ 24249], | 70.00th=[ 29230], 80.00th=[ 34341], 90.00th=[ 42730], 95.00th=[ 49546], | 99.00th=[ 64226], 99.50th=[ 69731], 99.90th=[ 93848], 99.95th=[101188], | 99.99th=[104334] bw ( MiB/s): min= 632, max= 1944, per=100.00%, avg=1266.62, stdev=300.37, samples=60 iops : min= 158, max= 486, avg=316.60, stdev=75.08, samples=60 write: IOPS=80, BW=323MiB/s (338MB/s)(9688MiB/30017msec) slat (usec): min=230, max=3310, avg=1124.48, stdev=297.73 clat (msec): min=9, max=140, avg=40.69, stdev=17.92 lat (msec): min=10, max=141, avg=41.81, stdev=17.94 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 19], 10.00th=[ 22], 20.00th=[ 27], | 30.00th=[ 31], 40.00th=[ 35], 50.00th=[ 39], 60.00th=[ 42], | 70.00th=[ 47], 80.00th=[ 53], 90.00th=[ 63], 95.00th=[ 74], | 99.00th=[ 103], 99.50th=[ 113], 99.90th=[ 138], 99.95th=[ 140], | 99.99th=[ 142] bw ( KiB/s): min=180224, max=401408, per=99.87%, avg=330056.57, stdev=60095.23, samples=60 iops : min= 44, max= 98, avg=80.57, stdev=14.67, samples=60 lat (usec) : 500=2.71%, 750=0.67%, 1000=0.55% lat (msec) : 2=8.91%, 4=5.73%, 10=6.98%, 20=15.47%, 50=50.54% lat (msec) : 100=8.13%, 250=0.31% cpu : usr=8.85%, sys=2.60%, ctx=6138, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=99.9%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=9503,2422,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=10 Run status group 0 (all jobs): READ: bw=1266MiB/s (1328MB/s), 1266MiB/s-1266MiB/s (1328MB/s-1328MB/s), io=37.1GiB (39.9GB), run=30017-30017msec WRITE: bw=323MiB/s (338MB/s), 323MiB/s-323MiB/s (338MB/s-338MB/s), io=9688MiB (10.2GB), run=30017-30017msec Disk stats (read/write): vdb: ios=37896/9636, merge=0/0, ticks=423177/369345, in_queue=816250, util=99.59% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=978557: Fri Jan 7 14:37:29 2022 read: IOPS=282, BW=1129MiB/s (1184MB/s)(33.1GiB/30005msec) clat (usec): min=382, max=8935, avg=946.46, stdev=1055.63 lat (usec): min=382, max=8936, avg=946.66, stdev=1055.64 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 416], 10.00th=[ 424], 20.00th=[ 449], | 30.00th=[ 494], 40.00th=[ 562], 50.00th=[ 652], 60.00th=[ 758], | 70.00th=[ 898], 80.00th=[ 1123], 90.00th=[ 1483], 95.00th=[ 2180], | 99.00th=[ 7898], 99.50th=[ 8029], 99.90th=[ 8717], 99.95th=[ 8717], | 99.99th=[ 8979] bw ( MiB/s): min= 650, max= 1716, per=100.00%, avg=1136.52, stdev=168.69, samples=60 iops : min= 162, max= 429, avg=283.67, stdev=42.18, samples=60 write: IOPS=71, BW=286MiB/s (300MB/s)(8596MiB/30005msec) clat (usec): min=9186, max=21072, avg=9744.10, stdev=735.86 lat (usec): min=9406, max=21491, avg=10221.49, stdev=765.05 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[ 9896], 80.00th=[10290], 90.00th=[10683], 95.00th=[11207], | 99.00th=[12125], 99.50th=[12518], 99.90th=[13173], 99.95th=[14091], | 99.99th=[21103] bw ( KiB/s): min=255484, max=338580, per=100.00%, avg=295169.40, stdev=17623.39, samples=60 iops : min= 62, max= 82, avg=71.60, stdev= 4.29, samples=60 lat (usec) : 500=24.49%, 750=22.53%, 1000=13.18% lat (msec) : 2=14.93%, 4=2.37%, 10=17.08%, 20=5.42%, 50=0.01% cpu : usr=3.73%, sys=2.17%, ctx=10636, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8468,2149,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1129MiB/s (1184MB/s), 1129MiB/s-1129MiB/s (1184MB/s-1184MB/s), io=33.1GiB (35.5GB), run=30005-30005msec WRITE: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=8596MiB (9014MB), run=30005-30005msec Disk stats (read/write): vdb: ios=33872/8592, merge=0/0, ticks=7044/64719, in_queue=91710, util=95.16% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=50 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=979525: Fri Jan 7 14:38:06 2022 read: IOPS=337, BW=1348MiB/s (1414MB/s)(39.6GiB/30041msec) slat (usec): min=39, max=234686, avg=2106.38, stdev=7568.23 clat (usec): min=352, max=397353, avg=75665.65, stdev=53924.79 lat (usec): min=401, max=498516, avg=77772.54, stdev=54761.37 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 9], 10.00th=[ 14], 20.00th=[ 24], | 30.00th=[ 42], 40.00th=[ 63], 50.00th=[ 75], 60.00th=[ 86], | 70.00th=[ 95], 80.00th=[ 108], 90.00th=[ 131], 95.00th=[ 169], | 99.00th=[ 279], 99.50th=[ 305], 99.90th=[ 355], 99.95th=[ 363], | 99.99th=[ 397] bw ( MiB/s): min= 601, max= 1907, per=100.00%, avg=1359.60, stdev=329.27, samples=60 iops : min= 150, max= 476, avg=339.43, stdev=82.37, samples=60 write: IOPS=85, BW=343MiB/s (360MB/s)(10.1GiB/30041msec) slat (usec): min=384, max=200855, avg=2930.22, stdev=6280.77 clat (msec): min=45, max=548, avg=273.94, stdev=59.85 lat (msec): min=46, max=549, avg=276.87, stdev=60.01 clat percentiles (msec): | 1.00th=[ 90], 5.00th=[ 161], 10.00th=[ 209], 20.00th=[ 247], | 30.00th=[ 262], 40.00th=[ 271], 50.00th=[ 279], 60.00th=[ 288], | 70.00th=[ 296], 80.00th=[ 305], 90.00th=[ 326], 95.00th=[ 347], | 99.00th=[ 485], 99.50th=[ 514], 99.90th=[ 550], 99.95th=[ 550], | 99.99th=[ 550] bw ( KiB/s): min=141528, max=537858, per=100.00%, avg=353957.58, stdev=76777.12, samples=60 iops : min= 34, max= 131, avg=85.92, stdev=18.78, samples=60 lat (usec) : 500=0.13%, 750=0.02%, 1000=0.02% lat (msec) : 2=0.50%, 4=0.87%, 10=3.60%, 20=8.55%, 50=12.63% lat (msec) : 100=33.76%, 250=23.07%, 500=16.68%, 750=0.16% cpu : usr=11.24%, sys=2.93%, ctx=4087, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.8%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwt: total=10127,2577,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=50 Run status group 0 (all jobs): READ: bw=1348MiB/s (1414MB/s), 1348MiB/s-1348MiB/s (1414MB/s-1414MB/s), io=39.6GiB (42.5GB), run=30041-30041msec WRITE: bw=343MiB/s (360MB/s), 343MiB/s-343MiB/s (360MB/s-360MB/s), io=10.1GiB (10.8GB), run=30041-30041msec Disk stats (read/write): vdb: ios=40422/10262, merge=0/0, ticks=1145244/2544126, in_queue=3723609, util=99.42% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=980489: Fri Jan 7 14:38:43 2022 read: IOPS=284, BW=1137MiB/s (1192MB/s)(33.3GiB/30001msec) clat (usec): min=378, max=9002, avg=932.56, stdev=1060.82 lat (usec): min=379, max=9002, avg=932.76, stdev=1060.83 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 408], 10.00th=[ 416], 20.00th=[ 441], | 30.00th=[ 486], 40.00th=[ 562], 50.00th=[ 644], 60.00th=[ 742], | 70.00th=[ 881], 80.00th=[ 1074], 90.00th=[ 1467], 95.00th=[ 2180], | 99.00th=[ 7898], 99.50th=[ 8291], 99.90th=[ 8717], 99.95th=[ 8848], | 99.99th=[ 8979] bw ( MiB/s): min= 712, max= 1632, per=100.00%, avg=1136.70, stdev=178.17, samples=60 iops : min= 178, max= 408, avg=284.17, stdev=44.55, samples=60 write: IOPS=72, BW=288MiB/s (302MB/s)(8648MiB/30001msec) clat (usec): min=9191, max=13527, avg=9724.27, stdev=687.30 lat (usec): min=9405, max=13940, avg=10188.40, stdev=711.05 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[ 9896], 80.00th=[10290], 90.00th=[10683], 95.00th=[11207], | 99.00th=[12125], 99.50th=[12387], 99.90th=[12780], 99.95th=[13435], | 99.99th=[13566] bw ( KiB/s): min=245760, max=335872, per=99.96%, avg=295058.92, stdev=17950.65, samples=60 iops : min= 60, max= 82, avg=72.03, stdev= 4.38, samples=60 lat (usec) : 500=25.78%, 750=22.48%, 1000=12.94% lat (msec) : 2=13.99%, 4=2.34%, 10=17.25%, 20=5.22% cpu : usr=3.73%, sys=2.00%, ctx=10704, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8525,2162,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1137MiB/s (1192MB/s), 1137MiB/s-1137MiB/s (1192MB/s-1192MB/s), io=33.3GiB (35.8GB), run=30001-30001msec WRITE: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=8648MiB (9068MB), run=30001-30001msec Disk stats (read/write): vdb: ios=33920/8609, merge=0/0, ticks=6551/64771, in_queue=91167, util=94.73% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=100 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981070: Fri Jan 7 14:39:20 2022 read: IOPS=343, BW=1376MiB/s (1443MB/s)(40.4GiB/30061msec) slat (usec): min=46, max=230593, avg=2143.79, stdev=6615.81 clat (msec): min=9, max=636, avg=187.53, stdev=65.30 lat (msec): min=65, max=736, avg=189.68, stdev=65.81 clat percentiles (msec): | 1.00th=[ 85], 5.00th=[ 104], 10.00th=[ 115], 20.00th=[ 134], | 30.00th=[ 150], 40.00th=[ 169], 50.00th=[ 184], 60.00th=[ 197], | 70.00th=[ 211], 80.00th=[ 226], 90.00th=[ 255], 95.00th=[ 305], | 99.00th=[ 443], 99.50th=[ 460], 99.90th=[ 558], 99.95th=[ 634], | 99.99th=[ 634] bw ( MiB/s): min= 552, max= 2168, per=99.57%, avg=1369.91, stdev=302.16, samples=60 iops : min= 138, max= 542, avg=342.47, stdev=75.54, samples=60 write: IOPS=87, BW=351MiB/s (368MB/s)(10.3GiB/30061msec) slat (usec): min=1093, max=220100, avg=2968.89, stdev=6547.22 clat (msec): min=127, max=743, avg=391.47, stdev=74.07 lat (msec): min=139, max=745, avg=394.44, stdev=73.95 clat percentiles (msec): | 1.00th=[ 171], 5.00th=[ 262], 10.00th=[ 313], 20.00th=[ 355], | 30.00th=[ 372], 40.00th=[ 384], 50.00th=[ 393], 60.00th=[ 405], | 70.00th=[ 414], 80.00th=[ 430], 90.00th=[ 460], 95.00th=[ 493], | 99.00th=[ 625], 99.50th=[ 667], 99.90th=[ 743], 99.95th=[ 743], | 99.99th=[ 743] bw ( KiB/s): min=73728, max=507904, per=98.91%, avg=355406.65, stdev=71538.50, samples=60 iops : min= 18, max= 124, avg=86.77, stdev=17.47, samples=60 lat (msec) : 10=0.01%, 100=2.93%, 250=68.85%, 500=27.03%, 750=1.18% cpu : usr=11.75%, sys=3.08%, ctx=4208, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10340,2637,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=100 Run status group 0 (all jobs): READ: bw=1376MiB/s (1443MB/s), 1376MiB/s-1376MiB/s (1443MB/s-1443MB/s), io=40.4GiB (43.4GB), run=30061-30061msec WRITE: bw=351MiB/s (368MB/s), 351MiB/s-351MiB/s (368MB/s-368MB/s), io=10.3GiB (11.1GB), run=30061-30061msec Disk stats (read/write): vdb: ios=41292/10484, merge=0/0, ticks=1026667/2659464, in_queue=3722228, util=99.06% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=psync, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=981879: Fri Jan 7 14:39:57 2022 read: IOPS=282, BW=1130MiB/s (1185MB/s)(33.1GiB/30006msec) clat (usec): min=373, max=8822, avg=944.93, stdev=1048.46 lat (usec): min=373, max=8822, avg=945.13, stdev=1048.47 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 416], 10.00th=[ 429], 20.00th=[ 453], | 30.00th=[ 498], 40.00th=[ 570], 50.00th=[ 660], 60.00th=[ 766], | 70.00th=[ 898], 80.00th=[ 1106], 90.00th=[ 1467], 95.00th=[ 2212], | 99.00th=[ 7898], 99.50th=[ 8029], 99.90th=[ 8717], 99.95th=[ 8717], | 99.99th=[ 8848] bw ( MiB/s): min= 696, max= 1488, per=100.00%, avg=1130.67, stdev=158.92, samples=60 iops : min= 174, max= 372, avg=282.63, stdev=39.71, samples=60 write: IOPS=71, BW=287MiB/s (301MB/s)(8612MiB/30006msec) clat (usec): min=9188, max=13122, avg=9729.95, stdev=677.85 lat (usec): min=9392, max=13974, avg=10203.77, stdev=703.08 clat percentiles (usec): | 1.00th=[ 9241], 5.00th=[ 9241], 10.00th=[ 9241], 20.00th=[ 9241], | 30.00th=[ 9241], 40.00th=[ 9241], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[ 9896], 80.00th=[10290], 90.00th=[10683], 95.00th=[11076], | 99.00th=[11994], 99.50th=[12518], 99.90th=[13042], 99.95th=[13042], | 99.99th=[13173] bw ( KiB/s): min=253952, max=327680, per=99.97%, avg=293819.73, stdev=17359.30, samples=60 iops : min= 62, max= 80, avg=71.73, stdev= 4.24, samples=60 lat (usec) : 500=24.25%, 750=22.52%, 1000=13.60% lat (msec) : 2=14.85%, 4=2.27%, 10=17.20%, 20=5.31% cpu : usr=3.69%, sys=2.74%, ctx=10654, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=8480,2153,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1130MiB/s (1185MB/s), 1130MiB/s-1130MiB/s (1185MB/s-1185MB/s), io=33.1GiB (35.6GB), run=30006-30006msec WRITE: bw=287MiB/s (301MB/s), 287MiB/s-287MiB/s (301MB/s-301MB/s), io=8612MiB (9030MB), run=30006-30006msec Disk stats (read/write): vdb: ios=33664/8556, merge=0/0, ticks=6797/64016, in_queue=90258, util=93.52% 30s: (g=0): rw=randrw, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=200 fio-3.0 Starting 1 process 30s: (groupid=0, jobs=1): err= 0: pid=983241: Fri Jan 7 14:40:34 2022 read: IOPS=349, BW=1398MiB/s (1466MB/s)(41.0GiB/30048msec) slat (usec): min=45, max=233315, avg=2086.77, stdev=5586.82 clat (msec): min=24, max=713, avg=411.32, stdev=78.22 lat (msec): min=74, max=713, avg=413.40, stdev=78.39 clat percentiles (msec): | 1.00th=[ 257], 5.00th=[ 305], 10.00th=[ 321], 20.00th=[ 351], | 30.00th=[ 372], 40.00th=[ 388], 50.00th=[ 405], 60.00th=[ 422], | 70.00th=[ 443], 80.00th=[ 468], 90.00th=[ 510], 95.00th=[ 550], | 99.00th=[ 634], 99.50th=[ 659], 99.90th=[ 701], 99.95th=[ 701], | 99.99th=[ 709] bw ( MiB/s): min= 48, max= 1976, per=98.68%, avg=1379.33, stdev=273.41, samples=60 iops : min= 12, max= 494, avg=344.83, stdev=68.35, samples=60 write: IOPS=88, BW=355MiB/s (372MB/s)(10.4GiB/30048msec) slat (usec): min=1107, max=235200, avg=3018.84, stdev=7275.88 clat (msec): min=75, max=912, avg=612.13, stdev=87.77 lat (msec): min=76, max=914, avg=615.15, stdev=87.48 clat percentiles (msec): | 1.00th=[ 376], 5.00th=[ 481], 10.00th=[ 514], 20.00th=[ 558], | 30.00th=[ 584], 40.00th=[ 592], 50.00th=[ 617], 60.00th=[ 625], | 70.00th=[ 651], 80.00th=[ 667], 90.00th=[ 718], 95.00th=[ 760], | 99.00th=[ 844], 99.50th=[ 869], 99.90th=[ 894], 99.95th=[ 894], | 99.99th=[ 911] bw ( KiB/s): min=229376, max=516096, per=100.00%, avg=364196.88, stdev=66488.15, samples=59 iops : min= 56, max= 126, avg=88.92, stdev=16.23, samples=59 lat (msec) : 50=0.01%, 100=0.23%, 250=0.49%, 500=71.01%, 750=27.11% lat (msec) : 1000=1.15% cpu : usr=12.01%, sys=3.18%, ctx=4247, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.5% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwt: total=10500,2668,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=200 Run status group 0 (all jobs): READ: bw=1398MiB/s (1466MB/s), 1398MiB/s-1398MiB/s (1466MB/s-1466MB/s), io=41.0GiB (44.0GB), run=30048-30048msec WRITE: bw=355MiB/s (372MB/s), 355MiB/s-355MiB/s (372MB/s-372MB/s), io=10.4GiB (11.2GB), run=30048-30048msec Disk stats (read/write): vdb: ios=41785/10636, merge=0/0, ticks=1021959/2673559, in_queue=3725094, util=98.38%