syzbot


INFO: task hung in rcu_barrier (5)

Status: closed as invalid on 2023/03/27 10:32
Subsystems: net
[Documentation on labels]
First crash: 402d, last: 402d
Similar bugs (4)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in rcu_barrier (2) net 1 1236d 1236d 0/26 auto-closed as invalid on 2021/03/12 23:17
upstream INFO: task hung in rcu_barrier (3) net 13 836d 1039d 0/26 closed as invalid on 2022/02/07 19:19
upstream INFO: task hung in rcu_barrier net 1 1874d 1874d 0/26 auto-closed as invalid on 2019/09/11 09:47
upstream INFO: task hung in rcu_barrier (4) net 1 564d 564d 0/26 auto-obsoleted due to no activity on 2023/01/14 10:47

Sample crash report:
INFO: task kworker/u4:15:6711 blocked for more than 143 seconds.
      Not tainted 6.3.0-rc3-syzkaller-00338-gda8e7da11e4b #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u4:15   state:D stack:20216 pid:6711  ppid:2      flags:0x00004000
Workqueue: netns cleanup_net
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5304 [inline]
 __schedule+0x173b/0x4630 kernel/sched/core.c:6622
 schedule+0xc3/0x180 kernel/sched/core.c:6698
 schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:6757
 __mutex_lock_common+0xe33/0x2530 kernel/locking/mutex.c:679
 __mutex_lock kernel/locking/mutex.c:747 [inline]
 mutex_lock_nested+0x1b/0x20 kernel/locking/mutex.c:799
 rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
 l2tp_exit_net+0x26e/0x310 net/l2tp/l2tp_core.c:1672
 ops_exit_list net/core/net_namespace.c:169 [inline]
 cleanup_net+0x6d2/0xb80 net/core/net_namespace.c:613
 process_one_work+0x8a0/0x10e0 kernel/workqueue.c:2390
 worker_thread+0xa63/0x1210 kernel/workqueue.c:2537
 kthread+0x270/0x300 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/12:
 #0: ffffffff8cf255f0 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x29/0xd20 kernel/rcu/tasks.h:510
1 lock held by rcu_tasks_trace/13:
 #0: ffffffff8cf25df0 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x29/0xd20 kernel/rcu/tasks.h:510
1 lock held by khungtaskd/28:
 #0: ffffffff8cf25420 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x0/0x30
2 locks held by getty/4755:
 #0: ffff88814a176098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:244
 #1: ffffc900015902f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6ab/0x1db0 drivers/tty/n_tty.c:2177
4 locks held by kworker/u4:15/6711:
 #0: ffff888016612938 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0x77e/0x10e0 kernel/workqueue.c:2363
 #1: ffffc9000606fd20 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0x7c8/0x10e0 kernel/workqueue.c:2365
 #2: ffffffff8e07e590 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0xf5/0xb80 net/core/net_namespace.c:575
 #3: ffffffff8cf2a7c0 (rcu_state.barrier_mutex){+.+.}-{3:3}, at: rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
2 locks held by kworker/1:30/7262:
 #0: ffff888012472538 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x77e/0x10e0 kernel/workqueue.c:2363
 #1: ffffc90004b2fd20 ((work_completion)(&rew->rew_work)){+.+.}-{0:0}, at: process_one_work+0x7c8/0x10e0 kernel/workqueue.c:2365
4 locks held by kworker/u4:8/15307:
1 lock held by syz-executor.1/15322:
 #0: ffffffff8cf2a7c0 (rcu_state.barrier_mutex){+.+.}-{3:3}, at: rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
2 locks held by kworker/1:8/16424:
3 locks held by kworker/0:25/18864:
 #0: ffff888012470d38 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x77e/0x10e0 kernel/workqueue.c:2363
 #1: ffffc900058a7d20 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0x7c8/0x10e0 kernel/workqueue.c:2365
 #2: ffffffff8e08a928 (rtnl_mutex){+.+.}-{3:3}, at: linkwatch_event+0xe/0x60 net/core/link_watch.c:277
1 lock held by syz-executor.2/20622:
 #0: ffffffff8cf2a7c0 (rcu_state.barrier_mutex){+.+.}-{3:3}, at: rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
1 lock held by syz-executor.1/20644:
 #0: ffffffff8cf2a7c0 (rcu_state.barrier_mutex){+.+.}-{3:3}, at: rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
7 locks held by syz-executor.2/20675:
 #0: ffff888027a4c460 (sb_writers#8){.+.+}-{0:0}, at: vfs_write+0x26d/0xbb0 fs/read_write.c:580
 #1: ffff88804d559888 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1eb/0x4f0 fs/kernfs/file.c:325
 #2: ffff888021e09ae8 (kn->active#51){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x20f/0x4f0 fs/kernfs/file.c:326
 #3: ffffffff8d9fe908 (nsim_bus_dev_list_lock){+.+.}-{3:3}, at: del_device_store+0xfc/0x480 drivers/net/netdevsim/bus.c:209
 #4: ffff888050b570e8 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:832 [inline]
 #4: ffff888050b570e8 (&dev->mutex){....}-{3:3}, at: __device_driver_lock drivers/base/dd.c:1063 [inline]
 #4: ffff888050b570e8 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0xce/0x7a0 drivers/base/dd.c:1260
 #5: ffff888050b50250 (&devlink->lock_key#13){+.+.}-{3:3}, at: nsim_drv_remove+0x50/0x160 drivers/net/netdevsim/dev.c:1675
 #6: ffffffff8cf2a7c0 (rcu_state.barrier_mutex){+.+.}-{3:3}, at: rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
2 locks held by syz-executor.4/20711:
 #0: ffffffff8e07e590 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x4af/0x790 net/core/net_namespace.c:486
 #1: ffffffff8cf2a7c0 (rcu_state.barrier_mutex){+.+.}-{3:3}, at: rcu_barrier+0x4c/0x540 kernel/rcu/tree.c:3945
4 locks held by syz-executor.1/20728:
 #0: ffff888027a4c460 (sb_writers#8){.+.+}-{0:0}, at: vfs_write+0x26d/0xbb0 fs/read_write.c:580
 #1: ffff88804d0cb888 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1eb/0x4f0 fs/kernfs/file.c:325
 #2: ffff888021e09ae8 (kn->active#51){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x20f/0x4f0 fs/kernfs/file.c:326
 #3: ffffffff8d9fe908 (nsim_bus_dev_list_lock){+.+.}-{3:3}, at: del_device_store+0xfc/0x480 drivers/net/netdevsim/bus.c:209
4 locks held by syz-executor.2/20752:
 #0: ffff888027a4c460 (sb_writers#8){.+.+}-{0:0}, at: vfs_write+0x26d/0xbb0 fs/read_write.c:580
 #1: ffff888028cabc88 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1eb/0x4f0 fs/kernfs/file.c:325
 #2: ffff888021e09ae8 (kn->active#51){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x20f/0x4f0 fs/kernfs/file.c:326
 #3: ffffffff8d9fe908 (nsim_bus_dev_list_lock){+.+.}-{3:3}, at: del_device_store+0xfc/0x480 drivers/net/netdevsim/bus.c:209
4 locks held by syz-executor.1/20767:
 #0: ffff888027a4c460 (sb_writers#8){.+.+}-{0:0}, at: vfs_write+0x26d/0xbb0 fs/read_write.c:580
 #1: ffff88804e9f9088 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1eb/0x4f0 fs/kernfs/file.c:325
 #2: ffff888021e09ae8 (kn->active#51){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x20f/0x4f0 fs/kernfs/file.c:326
 #3: ffffffff8d9fe908 (nsim_bus_dev_list_lock){+.+.}-{3:3}, at: del_device_store+0xfc/0x480 drivers/net/netdevsim/bus.c:209
2 locks held by syz-executor.2/20779:
 #0: ffffffff8e08a928 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:75 [inline]
 #0: ffffffff8e08a928 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x7ce/0xf40 net/core/rtnetlink.c:6171
 #1: ffffffff8cf2a8f8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:293 [inline]
 #1: ffffffff8cf2a8f8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x3a3/0x890 kernel/rcu/tree_exp.h:989

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 28 Comm: khungtaskd Not tainted 6.3.0-rc3-syzkaller-00338-gda8e7da11e4b #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e7/0x2d0 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x498/0x4d0 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x187/0x300 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:222 [inline]
 watchdog+0xec2/0xf00 kernel/hung_task.c:379
 kthread+0x270/0x300 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 15 Comm: ksoftirqd/0 Not tainted 6.3.0-rc3-syzkaller-00338-gda8e7da11e4b #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 03/02/2023
RIP: 0010:check_preemption_disabled+0x37/0x110
Code: 8b 04 25 28 00 00 00 48 89 44 24 08 65 8b 1d 1c 1b 63 75 65 8b 05 11 1b 63 75 a9 ff ff ff 7f 74 22 65 48 8b 04 25 28 00 00 00 <48> 3b 44 24 08 0f 85 c7 00 00 00 89 d8 48 83 c4 10 5b 41 5c 41 5e
RSP: 0018:ffffc90000146c68 EFLAGS: 00000002
RAX: 77843f2984270300 RBX: 0000000000000000 RCX: ffffffff91ada003
RDX: dffffc0000000000 RSI: ffffffff8aea8ee0 RDI: ffffffff8b37fe40
RBP: ffffc90000146d50 R08: dffffc0000000000 R09: fffffbfff1ca701e
R10: 0000000000000000 R11: dffffc0000000001 R12: ffff8880167fc528
R13: 0000000000000002 R14: ffff8880167fc4b0 R15: 1ffff11002cff896
FS:  0000000000000000(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0b1e203300 CR3: 000000000cd30000 CR4: 00000000003526f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 lockdep_recursion_finish kernel/locking/lockdep.c:467 [inline]
 lockdep_hardirqs_on_prepare+0x43c/0x7a0 kernel/locking/lockdep.c:4323
 trace_hardirqs_on+0x28/0x40 kernel/trace/trace_preemptirq.c:61
 ___slab_alloc+0x303/0x10a0 mm/slub.c:3126
 __slab_alloc mm/slub.c:3292 [inline]
 __slab_alloc_node mm/slub.c:3345 [inline]
 slab_alloc_node mm/slub.c:3442 [inline]
 kmem_cache_alloc_node+0x1f3/0x350 mm/slub.c:3497
 kmalloc_reserve+0x8a/0x240 net/core/skbuff.c:520
 __alloc_skb+0x1b1/0x420 net/core/skbuff.c:606
 __pskb_copy_fclone+0xb7/0x11d0 net/core/skbuff.c:1916
 __pskb_copy include/linux/skbuff.h:1341 [inline]
 pskb_copy include/linux/skbuff.h:3521 [inline]
 nf_dup_ipv4+0x13f/0x8d0 net/ipv4/netfilter/nf_dup_ipv4.c:62
 nft_dup_ipv4_eval+0x1db/0x300 net/ipv4/netfilter/nft_dup_ipv4.c:30
 expr_call_ops_eval net/netfilter/nf_tables_core.c:245 [inline]
 nft_do_chain+0x4ef/0x1bc0 net/netfilter/nf_tables_core.c:295
 nft_do_chain_ipv4+0x202/0x320 net/netfilter/nft_chain_filter.c:23
 nf_hook_entry_hookfn include/linux/netfilter.h:142 [inline]
 nf_hook_slow+0xb2/0x1f0 net/netfilter/core.c:614
 nf_hook include/linux/netfilter.h:257 [inline]
 NF_HOOK+0x26f/0x410 include/linux/netfilter.h:300
 __netif_receive_skb_one_core net/core/dev.c:5479 [inline]
 __netif_receive_skb+0x1ca/0x530 net/core/dev.c:5593
 process_backlog+0x34b/0x710 net/core/dev.c:5921
 __napi_poll+0xc7/0x470 net/core/dev.c:6482
 napi_poll net/core/dev.c:6549 [inline]
 net_rx_action+0x70a/0xeb0 net/core/dev.c:6659
 __do_softirq+0x2ab/0x908 kernel/softirq.c:571
 run_ksoftirqd+0xc5/0x120 kernel/softirq.c:934
 smpboot_thread_fn+0x533/0x9f0 kernel/smpboot.c:164
 kthread+0x270/0x300 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/03/26 10:59 upstream da8e7da11e4b fbf0499a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in rcu_barrier
* Struck through repros no longer work on HEAD.