syzbot

 sign-in | mailing list | source | docs
🐞 Open [987] ≡ Subsystems 🐞 Fixed [5236] 🐞 Invalid [12505] ⬇ Missing Backports [83] 📈 Kernel Health 📈 Bug Lifetimes 📈 Fuzzing 📈 Crashes 💬 Send us feedback

INFO: task hung in process_one_work (3)

Status: closed as invalid on 2019/11/30 16:54
Subsystems: net
[Documentation on labels]
First crash: 1608d, last: 1608d
▶ ▼ Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in process_one_work trace 1 2208d 2207d 0/26 auto-closed as invalid on 2019/02/22 10:22
upstream INFO: task hung in process_one_work (4) net syz error error 49 56d 1343d 0/26 upstream: reported syz repro on 2020/08/21 00:08
upstream INFO: task hung in process_one_work (2) bpf net 1 1772d 1772d 0/26 auto-closed as invalid on 2019/10/25 10:51

Sample crash report:
INFO: task kworker/2:2:3109 blocked for more than 143 seconds.
      Not tainted 5.4.0-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/2:2     D26888  3109      2 0x80004000
Workqueue: events nsim_dev_trap_report_work
Call Trace:
 context_switch kernel/sched/core.c:3385 [inline]
 __schedule+0x8e1/0x1f30 kernel/sched/core.c:4081
 schedule+0xdc/0x2b0 kernel/sched/core.c:4155
 schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:4214
 __mutex_lock_common kernel/locking/mutex.c:1036 [inline]
 __mutex_lock+0x7ab/0x13c0 kernel/locking/mutex.c:1106
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1121
 nsim_dev_trap_report_work+0x61/0xaf0 drivers/net/netdevsim/dev.c:407
 process_one_work+0x9af/0x1740 kernel/workqueue.c:2264
 worker_thread+0x98/0xe40 kernel/workqueue.c:2410
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

Showing all locks held in the system:
1 lock held by khungtaskd/1114:
 #0: ffffffff895a4080 (rcu_read_lock){....}, at: debug_show_all_locks+0x5f/0x279 kernel/locking/lockdep.c:5334
3 locks held by kworker/2:2/3109:
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: set_work_data kernel/workqueue.c:615 [inline]
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:642 [inline]
 #0: ffff88802cc27d28 ((wq_completion)events){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2235
 #1: ffff88802574fdc0 ((work_completion)(&(&nsim_dev->trap_data->trap_report_dw)->work)){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2239
 #2: ffff88802a645ae0 (&nsim_dev->port_list_lock){+.+.}, at: nsim_dev_trap_report_work+0x61/0xaf0 drivers/net/netdevsim/dev.c:407
1 lock held by rsyslogd/7851:
 #0: ffff88800e63a0e0 (&f->f_pos_lock){+.+.}, at: __fdget_pos+0xee/0x110 fs/file.c:801
2 locks held by getty/7973:
 #0: ffff88801e5b8090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc900040022e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/7974:
 #0: ffff88802687f090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc9000400a2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/7975:
 #0: ffff8880224fd090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc9000401a2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/7976:
 #0: ffff888025280090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc900040222e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/7977:
 #0: ffff888025a50090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc9000401e2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/7978:
 #0: ffff88802662a090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc900040162e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/7979:
 #0: ffff88801e6a0090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc900026f22e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
7 locks held by kworker/u16:3/8037:
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: set_work_data kernel/workqueue.c:615 [inline]
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:642 [inline]
 #0: ffff88802c7e4d28 ((wq_completion)netns){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2235
 #1: ffff88806ce4fdc0 (net_cleanup_work){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2239
 #2: ffffffff8a087e08 (pernet_ops_rwsem){++++}, at: cleanup_net+0xae/0xaf0 net/core/net_namespace.c:559
 #3: ffffffff8a0b12e0 (devlink_mutex){+.+.}, at: devlink_pernet_pre_exit+0x21/0x1a0 net/core/devlink.c:8255
 #4: ffff88802a645ae0 (&nsim_dev->port_list_lock){+.+.}, at: nsim_dev_port_del_all+0x2d/0xe0 drivers/net/netdevsim/dev.c:684
 #5: ffffffff8a094100 (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72
 #6: ffffffff895a7e38 (rcu_state.exp_mutex){+.+.}, at: exp_funnel_lock kernel/rcu/tree_exp.h:285 [inline]
 #6: ffffffff895a7e38 (rcu_state.exp_mutex){+.+.}, at: synchronize_rcu_expedited+0x4d6/0x5f0 kernel/rcu/tree_exp.h:817
3 locks held by syz-executor.0/8040:
2 locks held by kworker/2:3/8067:
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: set_work_data kernel/workqueue.c:615 [inline]
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:642 [inline]
 #0: ffff88802cc38928 ((wq_completion)rcu_gp){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2235
 #1: ffff88806d33fdc0 ((work_completion)(&rew.rew_work)){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2239

=============================================

NMI backtrace for cpu 0
CPU: 0 PID: 1114 Comm: khungtaskd Not tainted 5.4.0-syzkaller #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x197/0x210 lib/dump_stack.c:118
 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:205 [inline]
 watchdog+0xb11/0x10c0 kernel/hung_task.c:289
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Sending NMI from CPU 0 to CPUs 1-3:
NMI backtrace for cpu 1 skipped: idling at native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:60
NMI backtrace for cpu 2 skipped: idling at native_safe_halt+0xe/0x10 arch/x86/include/asm/irqflags.h:60
NMI backtrace for cpu 3
CPU: 3 PID: 8040 Comm: syz-executor.0 Not tainted 5.4.0-syzkaller #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.12.0-59-gc9ba5276e321-prebuilt.qemu.org 04/01/2014
RIP: 0010:skb_network_offset include/linux/skbuff.h:2576 [inline]
RIP: 0010:neigh_resolve_output net/core/neighbour.c:1483 [inline]
RIP: 0010:neigh_resolve_output+0x266/0x990 net/core/neighbour.c:1470
Code: 45 97 7c 08 84 c0 0f 85 4d 05 00 00 48 b9 00 00 00 00 00 fc ff df 48 8b 45 a8 45 0f b7 a7 b4 00 00 00 48 c1 e8 03 80 3c 08 00 <0f> 85 37 06 00 00 48 b9 00 00 00 00 00 fc ff df 48 8b 45 a0 4d 03
RSP: 0018:ffff88802d509738 EFLAGS: 00000246
RAX: 1ffff11004f4ed80 RBX: ffff8880225f8800 RCX: dffffc0000000000
RDX: 0000000000000100 RSI: ffffffff860f3934 RDI: dffffc0000000000
RBP: ffff88802d5097b0 R08: 1ffffffff15d29b3 R09: fffffbfff15d29b4
R10: fffffbfff15d29b3 R11: ffffffff8ae94d9f R12: 0000000000000010
R13: ffff8880225f8828 R14: ffffffff88b5e440 R15: ffff888027a76b40
FS:  00007ff8557fa700(0000) GS:ffff88802d500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006fb194 CR3: 000000006c81e000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 neigh_output include/net/neighbour.h:511 [inline]
 ip6_finish_output2+0x109a/0x25c0 net/ipv6/ip6_output.c:116
 __ip6_finish_output+0x444/0xaa0 net/ipv6/ip6_output.c:142
 ip6_finish_output+0x38/0x1f0 net/ipv6/ip6_output.c:152
 NF_HOOK_COND include/linux/netfilter.h:296 [inline]
 ip6_output+0x25e/0x880 net/ipv6/ip6_output.c:175
 dst_output include/net/dst.h:436 [inline]
 NF_HOOK include/linux/netfilter.h:307 [inline]
 ndisc_send_skb+0xf1f/0x1490 net/ipv6/ndisc.c:505
 ndisc_send_rs+0x134/0x720 net/ipv6/ndisc.c:699
 addrconf_rs_timer+0x30f/0x6e0 net/ipv6/addrconf.c:3879
 call_timer_fn+0x1ac/0x780 kernel/time/timer.c:1404
 expire_timers kernel/time/timer.c:1449 [inline]
 __run_timers kernel/time/timer.c:1773 [inline]
 __run_timers kernel/time/timer.c:1740 [inline]
 run_timer_softirq+0x6c3/0x1790 kernel/time/timer.c:1786
 __do_softirq+0x262/0x98c kernel/softirq.c:292
 invoke_softirq kernel/softirq.c:373 [inline]
 irq_exit+0x19b/0x1e0 kernel/softirq.c:413
 exiting_irq arch/x86/include/asm/apic.h:536 [inline]
 smp_apic_timer_interrupt+0x1a3/0x610 arch/x86/kernel/apic/apic.c:1137
 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:829
 </IRQ>
RIP: 0010:__raw_spin_unlock_irq include/linux/spinlock_api_smp.h:169 [inline]
RIP: 0010:_raw_spin_unlock_irq+0x4f/0x80 kernel/locking/spinlock.c:199
Code: c0 68 34 53 89 48 ba 00 00 00 00 00 fc ff df 48 c1 e8 03 80 3c 10 00 75 33 48 83 3d 12 2a 99 01 00 74 20 fb 66 0f 1f 44 00 00 <bf> 01 00 00 00 e8 e7 6b 96 f9 65 8b 05 18 14 48 78 85 c0 74 06 41
RSP: 0018:ffff88806c90fb20 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13
RAX: 1ffffffff12a668d RBX: ffff88806c900040 RCX: 0000000000000000
RDX: dffffc0000000000 RSI: 0000000000000006 RDI: ffff88806c9008d4
RBP: ffff88806c90fb28 R08: ffff88806c900040 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88802d537380
R13: ffff88802c44c200 R14: ffff8880293244c0 R15: 0000000000000402
 finish_lock_switch kernel/sched/core.c:3124 [inline]
 finish_task_switch+0x147/0x750 kernel/sched/core.c:3224
 context_switch kernel/sched/core.c:3388 [inline]
 __schedule+0x8e9/0x1f30 kernel/sched/core.c:4081
 preempt_schedule_irq+0xb5/0x160 kernel/sched/core.c:4338
 retint_kernel+0x1b/0x2b
RIP: 0010:__do_sys_perf_event_open+0x1eff/0x2c70 kernel/events/core.c:11567
Code: 9d a8 fe ff ff 31 ff 89 de e8 bd 2f e6 ff 85 db 0f 85 78 07 00 00 e8 30 2e e6 ff 8b 9d b4 fe ff ff 48 8b b5 78 fe ff ff 89 df <4c> 63 f3 e8 39 c8 35 00 e9 d0 e1 ff ff 48 89 c3 48 89 85 80 fe ff
RSP: 0018:ffff88806c90fd40 EFLAGS: 00010246 ORIG_RAX: ffffffffffffff13
RAX: 0000000000040000 RBX: 0000000000000004 RCX: ffffc9000409c000
RDX: 0000000000040000 RSI: ffff888020208300 RDI: 0000000000000004
RBP: ffff88806c90fed8 R08: ffff88806c900040 R09: ffffed100d920242
R10: ffffed100d920241 R11: ffff88806c90120f R12: ffff8880292fa2e0
R13: ffff88806c900040 R14: ffff88806c901290 R15: 1ffff1100d921fb6
 __se_sys_perf_event_open kernel/events/core.c:11151 [inline]
 __x64_sys_perf_event_open+0xbe/0x150 kernel/events/core.c:11151
 do_syscall_64+0xfa/0x790 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x45a759
Code: bd b1 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 8b b1 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007ff8557f9c88 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: ffffffffffffffda RBX: 000000000071bf00 RCX: 000000000045a759
RDX: ffffffffffffffff RSI: 0000000000000000 RDI: 000000002001d000
RBP: 0000000000000005 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 00007ff8557fa6d4
R13: 00000000004aec2b R14: 00000000006f1ca8 R15: 00000000ffffffff

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/11/30 04:34 upstream 81b6b96475ac 3a75be00 .config console log report ci-qemu-upstream
* Struck through repros no longer work on HEAD.