syzbot


INFO: task hung in switchdev_deferred_process_work

Status: upstream: reported on 2020/03/08 04:06
Reported-by: syzbot+f8665ab78456deea5754@syzkaller.appspotmail.com
First crash: 1502d, last: 424d
Similar bugs (6)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in switchdev_deferred_process_work (2) net C inconclusive 992 37d 1361d 0/26 upstream: reported C repro on 2020/07/27 17:04
upstream INFO: task hung in switchdev_deferred_process_work net 35 1651d 2187d 0/26 closed as invalid on 2019/10/23 07:26
linux-5.15 INFO: task hung in switchdev_deferred_process_work 1 372d 372d 0/3 auto-obsoleted due to no activity on 2023/08/10 05:46
linux-4.14 INFO: task hung in switchdev_deferred_process_work 1 713d 713d 0/1 auto-obsoleted due to no activity on 2022/09/03 16:45
linux-5.15 INFO: task hung in switchdev_deferred_process_work (2) 36 26d 98d 0/3 upstream: reported on 2024/01/11 18:09
linux-6.1 INFO: task hung in switchdev_deferred_process_work 3 39d 80d 0/3 upstream: reported on 2024/01/29 04:44

Sample crash report:
systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy1 wpan1: encryption failed: -22
ieee802154 phy0 wpan0: encryption failed: -22
ieee802154 phy1 wpan1: encryption failed: -22
INFO: task kworker/0:1:14 blocked for more than 140 seconds.
      Not tainted 4.19.211-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:1     D26128    14      2 0x80000000
Workqueue: events switchdev_deferred_process_work
Call Trace:
 context_switch kernel/sched/core.c:2828 [inline]
 __schedule+0x887/0x2040 kernel/sched/core.c:3517
 schedule+0x8d/0x1b0 kernel/sched/core.c:3561
 schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:3619
 __mutex_lock_common kernel/locking/mutex.c:1016 [inline]
 __mutex_lock+0x5f0/0x1190 kernel/locking/mutex.c:1078
 switchdev_deferred_process_work+0xa/0x20 net/switchdev/switchdev.c:150
 process_one_work+0x864/0x1570 kernel/workqueue.c:2153
 worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

Showing all locks held in the system:
2 locks held by systemd/1:
3 locks held by kworker/0:1/14:
 #0: 0000000001d77a09 ((wq_completion)"events"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2124
 #1: 000000002d9a5a8c (deferred_process_work){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128
 #2: 00000000e7ce8302 (rtnl_mutex){+.+.}, at: switchdev_deferred_process_work+0xa/0x20 net/switchdev/switchdev.c:150
6 locks held by kworker/u4:4/229:
1 lock held by khungtaskd/1571:
 #0: 000000004ed3e90d (rcu_read_lock){....}, at: debug_show_all_locks+0x53/0x265 kernel/locking/lockdep.c:4441
3 locks held by kworker/1:2/3498:
4 locks held by systemd-udevd/4697:
1 lock held by in:imklog/7811:
 #0: 000000009b551b96 (&f->f_pos_lock){+.+.}, at: __fdget_pos+0x26f/0x310 fs/file.c:767
3 locks held by kworker/0:4/9332:
 #0: 00000000011bedba ((wq_completion)"%s"("ipv6_addrconf")){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2124
 #1: 000000003511d8d9 ((addr_chk_work).work){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128
 #2: 00000000e7ce8302 (rtnl_mutex){+.+.}, at: addrconf_verify_work+0xa/0x20 net/ipv6/addrconf.c:4476
2 locks held by syz-executor.4/11426:
3 locks held by kworker/1:3/21871:
 #0: 0000000001d77a09 ((wq_completion)"events"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2124
 #1: 0000000054c89fac ((linkwatch_work).work){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2128
 #2: 00000000e7ce8302 (rtnl_mutex){+.+.}, at: linkwatch_event+0xb/0x60 net/core/link_watch.c:236
3 locks held by kworker/0:2/22184:
2 locks held by kworker/u4:3/26847:
5 locks held by kworker/u4:12/27440:
1 lock held by syz-executor.2/29920:
 #0: 00000000e7ce8302 (rtnl_mutex){+.+.}, at: netdev_run_todo+0x719/0xab0 net/core/dev.c:9005
1 lock held by syz-executor.1/31295:
 #0: 00000000e7ce8302 (rtnl_mutex){+.+.}, at: tun_detach drivers/net/tun.c:759 [inline]
 #0: 00000000e7ce8302 (rtnl_mutex){+.+.}, at: tun_chr_close+0x3a/0x180 drivers/net/tun.c:3323
2 locks held by syz-executor.0/31310:
3 locks held by syz-executor.3/31315:
2 locks held by systemd-cgroups/31339:

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 1571 Comm: khungtaskd Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/21/2023
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1fc/0x2ef lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x63/0xa2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x1a6/0x1f0 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:203 [inline]
 watchdog+0x991/0xe60 kernel/hung_task.c:287
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 229 Comm: kworker/u4:4 Not tainted 4.19.211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/21/2023
Workqueue:            (null) (phy34)
RIP: 0010:kasan_check_read+0x4/0x10 mm/kasan/kasan.c:272
Code: 89 d0 49 39 c1 75 11 49 0f be 01 41 83 e2 07 49 39 c2 0f 8c 4e ff ff ff 5b 44 89 c2 5d 41 5c e9 b2 18 00 00 66 90 48 8b 0c 24 <89> f6 31 d2 e9 83 fe ff ff 0f 1f 00 48 8b 0c 24 89 f6 ba 01 00 00
RSP: 0018:ffff8880b48a7c18 EFLAGS: 00000046
RAX: 0000000000000007 RBX: 0000000000000086 RCX: ffffffff814be140
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff8d44f308
RBP: ffffffff8d44f308 R08: 0000000000000086 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
R13: 1ffff11016914f8d R14: ffff8880976bad28 R15: ffffffff8d44f308
FS:  0000000000000000(0000) GS:ffff8880ba000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000130 CR3: 00000000b2aaa000 CR4: 00000000003426f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 atomic_read include/asm-generic/atomic-instrumented.h:21 [inline]
 queued_spin_is_locked include/asm-generic/qspinlock.h:35 [inline]
 debug_spin_unlock kernel/locking/spinlock_debug.c:98 [inline]
 do_raw_spin_unlock+0x50/0x230 kernel/locking/spinlock_debug.c:134
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:159 [inline]
 _raw_spin_unlock_irqrestore+0x23/0xe0 kernel/locking/spinlock.c:184
 debug_object_deactivate lib/debugobjects.c:568 [inline]
 debug_object_deactivate+0x1f9/0x2e0 lib/debugobjects.c:529
 debug_work_deactivate kernel/workqueue.c:497 [inline]
 process_one_work+0x375/0x1570 kernel/workqueue.c:2084
 worker_thread+0x64c/0x1130 kernel/workqueue.c:2296
 kthread+0x33f/0x460 kernel/kthread.c:259
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

Crashes (95):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/02/19 10:00 linux-4.19.y 3f8a27f9e27b bcdf85f8 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2023/02/11 22:38 linux-4.19.y 3f8a27f9e27b 93e26d60 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2023/02/10 18:36 linux-4.19.y 3f8a27f9e27b e29a17f5 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2023/01/18 15:15 linux-4.19.y 3f8a27f9e27b 4620c2d9 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/12/08 16:49 linux-4.19.y 3f8a27f9e27b 1034e5fa .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/10/24 10:32 linux-4.19.y 3f8a27f9e27b faae2fda .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/10/22 19:17 linux-4.19.y 3f8a27f9e27b c0b80a55 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/10/08 03:36 linux-4.19.y 3f8a27f9e27b 79a59635 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/09/29 02:14 linux-4.19.y 3f8a27f9e27b a41a2080 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/09/28 01:11 linux-4.19.y 3f8a27f9e27b 75c78242 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/09/20 00:44 linux-4.19.y 3f8a27f9e27b dd9a85ff .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/09/15 03:33 linux-4.19.y 3f8a27f9e27b b884348d .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/09/07 22:00 linux-4.19.y 3f8a27f9e27b c5b7bc57 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/09/03 00:27 linux-4.19.y 3f8a27f9e27b 25194605 .config console log report info [disk image] [vmlinux] ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/08/08 15:54 linux-4.19.y 3f8a27f9e27b da700653 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/08/05 09:38 linux-4.19.y 3f8a27f9e27b a65a7ce9 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/08/03 16:33 linux-4.19.y 3f8a27f9e27b 1c9013ac .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/08/02 19:49 linux-4.19.y 3f8a27f9e27b 1c9013ac .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/07/18 04:30 linux-4.19.y 3f8a27f9e27b 95cb00d1 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/06/22 21:47 linux-4.19.y 3f8a27f9e27b 912f5df7 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/06/03 08:46 linux-4.19.y 3f8a27f9e27b 02dddea8 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/05/26 11:52 linux-4.19.y 3f8a27f9e27b 3037caa9 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/05/24 22:19 linux-4.19.y 3f8a27f9e27b 647c0e27 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/05/24 07:50 linux-4.19.y 3f8a27f9e27b e7f9308d .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/05/05 20:07 linux-4.19.y 3f8a27f9e27b b3f09415 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/03/23 07:31 linux-4.19.y 3f8a27f9e27b 5ff41e94 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/02/20 09:51 linux-4.19.y 3f8a27f9e27b 3cd800e4 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/02/19 13:37 linux-4.19.y 3f8a27f9e27b 3cd800e4 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/01/12 19:56 linux-4.19.y 3f8a27f9e27b 44d1319a .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/01/10 15:06 linux-4.19.y 3f8a27f9e27b 2ca0d385 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/01/09 08:37 linux-4.19.y 3f8a27f9e27b 2ca0d385 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/01/09 03:09 linux-4.19.y 3f8a27f9e27b 2ca0d385 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/01/09 00:55 linux-4.19.y 3f8a27f9e27b 2ca0d385 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2022/01/04 16:29 linux-4.19.y 3f8a27f9e27b 7f723fbe .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/31 23:29 linux-4.19.y 3f8a27f9e27b 36bd2e48 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/25 14:32 linux-4.19.y 3f8a27f9e27b 6caa12e4 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/22 11:49 linux-4.19.y 3f8a27f9e27b 6caa12e4 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/19 23:10 linux-4.19.y 3f8a27f9e27b 44068e19 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/18 20:35 linux-4.19.y 3f8a27f9e27b 44068e19 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/13 09:34 linux-4.19.y 3f8a27f9e27b 49ca1f59 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/05 07:31 linux-4.19.y 3f8a27f9e27b a617004c .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/04 11:26 linux-4.19.y 3f8a27f9e27b a617004c .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/12/02 04:20 linux-4.19.y 3f8a27f9e27b 61f86278 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/11/29 20:48 linux-4.19.y 3f8a27f9e27b d0830353 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/11/28 12:45 linux-4.19.y 3f8a27f9e27b 63eeac02 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2021/11/27 02:23 linux-4.19.y 3f8a27f9e27b 63eeac02 .config console log report info ci2-linux-4-19 INFO: task hung in switchdev_deferred_process_work
2020/11/14 17:10 linux-4.19.y 31acccdc8774 1bf9a662 .config console log report info ci2-linux-4-19
2020/05/10 00:22 linux-4.19.y 84920cc7fbe1 88cb3e92 .config console log report ci2-linux-4-19
2020/04/28 00:37 linux-4.19.y 7edd66cf6167 0ce7569e .config console log report ci2-linux-4-19
2020/04/20 18:11 linux-4.19.y 8488c3f3bc86 347a5dc3 .config console log report ci2-linux-4-19
2020/04/01 10:38 linux-4.19.y 54b4fa6d3955 a34e2c33 .config console log report ci2-linux-4-19
2020/03/17 09:27 linux-4.19.y 339485c9a80f 749688d2 .config console log report ci2-linux-4-19
2020/03/16 20:40 linux-4.19.y 339485c9a80f 749688d2 .config console log report ci2-linux-4-19
2020/03/15 13:36 linux-4.19.y 569209711609 749688d2 .config console log report ci2-linux-4-19
2020/03/08 04:05 linux-4.19.y 7472c4028e23 2e9971bb .config console log report ci2-linux-4-19
* Struck through repros no longer work on HEAD.