Extracting prog: 3m17.015850145s Minimizing prog: 23m58.809461624s Simplifying prog options: 0s Extracting C: 1m18.281343924s Simplifying C: 29m15.704210151s 1 programs, timeouts [15s 1m40s 6m0s] extracting reproducer from 1 programs single: executing 1 programs separately with timeout 15s testing program (duration=15s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash single: failed to extract reproducer single: executing 1 programs separately with timeout 1m40s testing program (duration=1m40s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program crashed: BUG: workqueue lockup single: successfully extracted reproducer found reproducer with 4 syscalls minimizing guilty program testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, 0x0, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, 0x0) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, 0x0, &(0x7f0000000100)) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, 0x0) timer_settime(0x0, 0x0, &(0x7f000006b000)={{0x0, 0x8}, {0x0, 0x9}}, 0x0) program did not crash testing program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime detailed listing: executing program 0: prlimit64(0x0, 0xe, &(0x7f0000000140)={0xc, 0x8b}, 0x0) sched_setscheduler(0x0, 0x1, &(0x7f0000000300)=0x9) timer_create(0x0, &(0x7f0000000680)={0x0, 0x21, 0x0, @tid=0xffffffffffffffff}, &(0x7f0000000100)) timer_settime(0x0, 0x0, 0x0, 0x0) program did not crash extracting C reproducer testing compiled C program (duration=2m30s, {Threaded:true Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup simplifying C reproducer testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:1 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program did not crash testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox: SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:false KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:false IEEE802154:true Sysctl:true Swap:false UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program did not crash testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:false Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program did not crash testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:false IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:false IEEE802154:false Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:false IEEE802154:false Sysctl:true Swap:true UseTmpDir:false HandleSegv:true Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program did not crash testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:false IEEE802154:false Sysctl:true Swap:true UseTmpDir:true HandleSegv:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:false IEEE802154:false Sysctl:false Swap:true UseTmpDir:true HandleSegv:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup testing compiled C program (duration=2m30s, {Threaded:false Repeat:true RepeatTimes:0 Procs:5 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:false NetDevices:false NetReset:false Cgroups:false BinfmtMisc:false CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:false VhciInjection:true Wifi:false IEEE802154:false Sysctl:false Swap:false UseTmpDir:true HandleSegv:false Trace:false LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}): prlimit64-sched_setscheduler-timer_create-timer_settime program crashed: BUG: workqueue lockup reproducing took 57m49.810888314s repro crashed as (corrupted=false): BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 50s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 50s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=-20 stuck for 48s! BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 51s! BUG: workqueue lockup - pool cpus=0-1 node=0 flags=0x4 nice=-20 stuck for 49s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8 pending: vmstat_shepherd, l2cap_info_timeout, 4*ovs_dp_masks_rebalance, kfree_rcu_monitor pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=9 refcnt=10 in-flight: 2232:request_firmware_work_func pending: psi_avgs_work, ovs_dp_masks_rebalance, 4*l2cap_info_timeout, ovs_dp_masks_rebalance, kfree_rcu_monitor workqueue events_long: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 pending: 4*defense_work_handler pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: 2*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: toggle_allocation_gate pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3 pending: flush_memcg_stats_dwork, crng_reseed workqueue events_power_efficient: flags=0x82 pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3 pending: neigh_managed_work, do_cache_clean pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6 pending: neigh_managed_work, 2*neigh_periodic_work, gc_worker, reg_check_chans_work workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 in-flight: 291:wb_workfn pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wb_update_bandwidth_workfn workqueue kblockd: flags=0x18 pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=1 refcnt=2 pending: blk_mq_timeout_work workqueue dm_bufio_cache: flags=0x8 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: work_fn workqueue krxrpcd: flags=0x2001a pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=9 pending: rxrpc_peer_keepalive_worker inactive: 5*rxrpc_peer_keepalive_worker workqueue ext4-rsv-conversion: flags=0xa pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: ext4_end_io_rsv_work workqueue hci0: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci1: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci2: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci3: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci4: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=51s workers=4 idle: 6294 25 27 pool 8: cpus=0-1 flags=0x4 nice=0 hung=51s workers=10 idle: 4243 518 12 11 708 42 547 43 505 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 6: task:kworker/1:2 state:R running task stack:0 pid:2232 tgid:2232 ppid:2 flags:0x00000008 Workqueue: events request_firmware_work_func Call trace: __switch_to+0x314/0x560 arch/arm64/kernel/process.c:553 context_switch kernel/sched/core.c:5408 [inline] __schedule+0x14ac/0x24d8 kernel/sched/core.c:6745 __schedule_loop kernel/sched/core.c:6822 [inline] schedule+0xbc/0x238 kernel/sched/core.c:6837 schedule_timeout+0x1cc/0x33c kernel/time/timer.c:2581 do_wait_for_common+0x30c/0x468 kernel/sched/completion.c:95 __wait_for_common kernel/sched/completion.c:116 [inline] wait_for_common kernel/sched/completion.c:127 [inline] wait_for_completion_killable_timeout+0x50/0x74 kernel/sched/completion.c:288 __fw_state_wait_common drivers/base/firmware_loader/firmware.h:104 [inline] fw_sysfs_wait_timeout drivers/base/firmware_loader/fallback.c:44 [inline] fw_load_sysfs_fallback drivers/base/firmware_loader/fallback.c:110 [inline] fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:162 [inline] firmware_fallback_sysfs+0x528/0x918 drivers/base/firmware_loader/fallback.c:238 _request_firmware+0xd1c/0xf7c drivers/base/firmware_loader/main.c:914 request_firmware_work_func+0xfc/0x214 drivers/base/firmware_loader/main.c:1165 process_one_work+0x79c/0x15b8 kernel/workqueue.c:3248 process_scheduled_works kernel/workqueue.c:3329 [inline] worker_thread+0x938/0xecc kernel/workqueue.c:3409 kthread+0x288/0x310 kernel/kthread.c:389 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860 final repro crashed as (corrupted=false): BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 50s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 50s! BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=-20 stuck for 48s! BUG: workqueue lockup - pool cpus=0-1 flags=0x4 nice=0 stuck for 51s! BUG: workqueue lockup - pool cpus=0-1 node=0 flags=0x4 nice=-20 stuck for 49s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8 pending: vmstat_shepherd, l2cap_info_timeout, 4*ovs_dp_masks_rebalance, kfree_rcu_monitor pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=9 refcnt=10 in-flight: 2232:request_firmware_work_func pending: psi_avgs_work, ovs_dp_masks_rebalance, 4*l2cap_info_timeout, ovs_dp_masks_rebalance, kfree_rcu_monitor workqueue events_long: flags=0x0 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5 pending: 4*defense_work_handler pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=2 refcnt=3 pending: 2*defense_work_handler workqueue events_unbound: flags=0x2 pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: toggle_allocation_gate pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3 pending: flush_memcg_stats_dwork, crng_reseed workqueue events_power_efficient: flags=0x82 pwq 8: cpus=0-1 flags=0x4 nice=0 active=2 refcnt=3 pending: neigh_managed_work, do_cache_clean pwq 8: cpus=0-1 flags=0x4 nice=0 active=5 refcnt=6 pending: neigh_managed_work, 2*neigh_periodic_work, gc_worker, reg_check_chans_work workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: vmstat_update workqueue writeback: flags=0x4a pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 in-flight: 291:wb_workfn pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: wb_update_bandwidth_workfn workqueue kblockd: flags=0x18 pwq 7: cpus=1 node=0 flags=0x0 nice=-20 active=1 refcnt=2 pending: blk_mq_timeout_work workqueue dm_bufio_cache: flags=0x8 pwq 6: cpus=1 node=0 flags=0x0 nice=0 active=1 refcnt=2 pending: work_fn workqueue krxrpcd: flags=0x2001a pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=9 pending: rxrpc_peer_keepalive_worker inactive: 5*rxrpc_peer_keepalive_worker workqueue ext4-rsv-conversion: flags=0xa pwq 8: cpus=0-1 flags=0x4 nice=0 active=1 refcnt=2 pending: ext4_end_io_rsv_work workqueue hci0: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci1: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci2: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci3: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout workqueue hci4: flags=0x20012 pwq 9: cpus=0-1 node=0 flags=0x4 nice=-20 active=1 refcnt=4 pending: hci_cmd_timeout pool 6: cpus=1 node=0 flags=0x0 nice=0 hung=51s workers=4 idle: 6294 25 27 pool 8: cpus=0-1 flags=0x4 nice=0 hung=51s workers=10 idle: 4243 518 12 11 708 42 547 43 505 Showing backtraces of running workers in stalled CPU-bound worker pools: pool 6: task:kworker/1:2 state:R running task stack:0 pid:2232 tgid:2232 ppid:2 flags:0x00000008 Workqueue: events request_firmware_work_func Call trace: __switch_to+0x314/0x560 arch/arm64/kernel/process.c:553 context_switch kernel/sched/core.c:5408 [inline] __schedule+0x14ac/0x24d8 kernel/sched/core.c:6745 __schedule_loop kernel/sched/core.c:6822 [inline] schedule+0xbc/0x238 kernel/sched/core.c:6837 schedule_timeout+0x1cc/0x33c kernel/time/timer.c:2581 do_wait_for_common+0x30c/0x468 kernel/sched/completion.c:95 __wait_for_common kernel/sched/completion.c:116 [inline] wait_for_common kernel/sched/completion.c:127 [inline] wait_for_completion_killable_timeout+0x50/0x74 kernel/sched/completion.c:288 __fw_state_wait_common drivers/base/firmware_loader/firmware.h:104 [inline] fw_sysfs_wait_timeout drivers/base/firmware_loader/fallback.c:44 [inline] fw_load_sysfs_fallback drivers/base/firmware_loader/fallback.c:110 [inline] fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:162 [inline] firmware_fallback_sysfs+0x528/0x918 drivers/base/firmware_loader/fallback.c:238 _request_firmware+0xd1c/0xf7c drivers/base/firmware_loader/main.c:914 request_firmware_work_func+0xfc/0x214 drivers/base/firmware_loader/main.c:1165 process_one_work+0x79c/0x15b8 kernel/workqueue.c:3248 process_scheduled_works kernel/workqueue.c:3329 [inline] worker_thread+0x938/0xecc kernel/workqueue.c:3409 kthread+0x288/0x310 kernel/kthread.c:389 ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:860