syzbot


INFO: task hung in devlink_pernet_pre_exit (2)

Status: internal: reported on 2024/10/20 07:50
Subsystems: net
[Documentation on labels]
Fix commit: a1494d532e28 netdevsim: use cond_resched() in nsim_dev_trap_report_work()
Patched on: [ci-qemu-gce-upstream-auto ci-qemu-upstream ci-qemu-upstream-386 ci-qemu2-arm32 ci-qemu2-arm64 ci-qemu2-arm64-compat ci-qemu2-arm64-mte ci-snapshot-upstream-root ci-upstream-bpf-kasan-gce ci-upstream-gce-arm64 ci-upstream-gce-leak ci-upstream-kasan-badwrites-root ci-upstream-kasan-gce ci-upstream-kasan-gce-386 ci-upstream-kasan-gce-root ci-upstream-kasan-gce-selinux-root ci-upstream-kasan-gce-smack-root ci-upstream-kmsan-gce-386-root ci-upstream-kmsan-gce-root ci-upstream-linux-next-kasan-gce-root ci-upstream-net-this-kasan-gce ci2-upstream-fs ci2-upstream-kcsan-gce ci2-upstream-usb], missing on: [ci-qemu-native-arm64-kvm ci-qemu2-riscv64 ci-upstream-bpf-next-kasan-gce ci-upstream-net-kasan-gce]
First crash: 179d, last: 1d17h
Similar bugs (2)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in devlink_pernet_pre_exit net 1 1098d 1098d 0/28 auto-closed as invalid on 2022/01/18 07:25
linux-5.15 INFO: task hung in devlink_pernet_pre_exit 1 40d 40d 0/3 upstream: reported on 2024/09/12 18:03

Sample crash report:
INFO: task kworker/u8:14:7210 blocked for more than 143 seconds.
      Not tainted 6.12.0-rc3-syzkaller-00454-gdb87114dcf13 #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u8:14   state:D stack:20496 pid:7210  tgid:7210  ppid:2      flags:0x00004000
Workqueue: netns cleanup_net
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5328 [inline]
 __schedule+0x18af/0x4bd0 kernel/sched/core.c:6690
 __schedule_loop kernel/sched/core.c:6767 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6782
 schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6839
 __mutex_lock_common kernel/locking/mutex.c:684 [inline]
 __mutex_lock+0x6a7/0xd70 kernel/locking/mutex.c:752
 device_lock include/linux/device.h:1014 [inline]
 devl_dev_lock net/devlink/devl_internal.h:108 [inline]
 devlink_pernet_pre_exit+0x13b/0x440 net/devlink/core.c:506
 ops_pre_exit_list net/core/net_namespace.c:163 [inline]
 cleanup_net+0x617/0xcc0 net/core/net_namespace.c:606
 process_one_work kernel/workqueue.c:3229 [inline]
 process_scheduled_works+0xa65/0x1850 kernel/workqueue.c:3310
 worker_thread+0x870/0xd30 kernel/workqueue.c:3391
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Showing all locks held in the system:
2 locks held by ksoftirqd/1/24:
1 lock held by khungtaskd/30:
 #0: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:337 [inline]
 #0: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:849 [inline]
 #0: ffffffff8e937e20 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x2a0 kernel/locking/lockdep.c:6720
1 lock held by klogd/4670:
1 lock held by dhcpcd/4895:
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: devinet_ioctl+0x2ce/0x1aa0 net/ipv4/devinet.c:1112
2 locks held by getty/4978:
 #0: ffff88802ea7e0a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
 #1: ffffc90002f062f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6a6/0x1e00 drivers/tty/n_tty.c:2211
2 locks held by kworker/u8:13/6336:
4 locks held by kworker/u8:14/7210:
 #0: ffff88801bae5948 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3204 [inline]
 #0: ffff88801bae5948 ((wq_completion)netns){+.+.}-{0:0}, at: process_scheduled_works+0x93b/0x1850 kernel/workqueue.c:3310
 #1: ffffc90002fdfd00 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3205 [inline]
 #1: ffffc90002fdfd00 (net_cleanup_work){+.+.}-{0:0}, at: process_scheduled_works+0x976/0x1850 kernel/workqueue.c:3310
 #2: ffffffff8fcc63d0 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0x16a/0xcc0 net/core/net_namespace.c:580
 #3: ffff88807779a0e8 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:1014 [inline]
 #3: ffff88807779a0e8 (&dev->mutex){....}-{3:3}, at: devl_dev_lock net/devlink/devl_internal.h:108 [inline]
 #3: ffff88807779a0e8 (&dev->mutex){....}-{3:3}, at: devlink_pernet_pre_exit+0x13b/0x440 net/devlink/core.c:506
7 locks held by syz-executor/10952:
 #0: ffff888032cd4420 (sb_writers#8){.+.+}-{0:0}, at: file_start_write include/linux/fs.h:2931 [inline]
 #0: ffff888032cd4420 (sb_writers#8){.+.+}-{0:0}, at: vfs_write+0x224/0xc90 fs/read_write.c:679
 #1: ffff88805f8bd488 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x1ea/0x500 fs/kernfs/file.c:325
 #2: ffff888027edba58 (kn->active#49){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x20e/0x500 fs/kernfs/file.c:326
 #3: ffffffff8f570e28 (nsim_bus_dev_list_lock){+.+.}-{3:3}, at: del_device_store+0xfc/0x480 drivers/net/netdevsim/bus.c:216
 #4: ffff88807779a0e8 (&dev->mutex){....}-{3:3}, at: device_lock include/linux/device.h:1014 [inline]
 #4: ffff88807779a0e8 (&dev->mutex){....}-{3:3}, at: __device_driver_lock drivers/base/dd.c:1095 [inline]
 #4: ffff88807779a0e8 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0xce/0x7c0 drivers/base/dd.c:1293
 #5: ffff88807779b250 (&devlink->lock_key#62){+.+.}-{3:3}, at: nsim_drv_remove+0x50/0x160 drivers/net/netdevsim/dev.c:1675
 #6: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: unregister_netdevice_notifier_net+0x89/0x3a0 net/core/dev.c:1917
2 locks held by syz-executor/11249:
 #0: ffffffff8fcc63d0 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x328/0x570 net/core/net_namespace.c:490
 #1: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: mpls_net_exit+0x7d/0x2a0 net/mpls/af_mpls.c:2706
2 locks held by syz-executor/11260:
 #0: ffffffff8fcc63d0 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x328/0x570 net/core/net_namespace.c:490
 #1: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: mpls_net_exit+0x7d/0x2a0 net/mpls/af_mpls.c:2706
3 locks held by syz-executor/11270:
2 locks held by syz-executor/11276:
 #0: ffffffff8fcc63d0 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x328/0x570 net/core/net_namespace.c:490
 #1: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: ip_tunnel_init_net+0x20e/0x720 net/ipv4/ip_tunnel.c:1159
2 locks held by syz-executor/11281:
 #0: ffffffff8fcc63d0 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x328/0x570 net/core/net_namespace.c:490
 #1: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: setup_net+0x602/0x9e0 net/core/net_namespace.c:378
2 locks held by syz-executor/11286:
 #0: ffffffff8fcc63d0 (pernet_ops_rwsem){++++}-{3:3}, at: copy_net_ns+0x328/0x570 net/core/net_namespace.c:490
 #1: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: tc_action_net_exit include/net/act_api.h:173 [inline]
 #1: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: gate_exit_net+0x30/0x100 net/sched/act_gate.c:659
1 lock held by syz-executor/11340:
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x6e6/0xcf0 net/core/rtnetlink.c:6672
1 lock held by syz-executor/11348:
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x6e6/0xcf0 net/core/rtnetlink.c:6672
1 lock held by syz-executor/11352:
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x6e6/0xcf0 net/core/rtnetlink.c:6672
1 lock held by syz-executor/11355:
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:79 [inline]
 #0: ffffffff8fcd2ec8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x6e6/0xcf0 net/core/rtnetlink.c:6672

=============================================

NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 30 Comm: khungtaskd Not tainted 6.12.0-rc3-syzkaller-00454-gdb87114dcf13 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:120
 nmi_cpu_backtrace+0x49c/0x4d0 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x198/0x320 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:162 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:223 [inline]
 watchdog+0xff4/0x1040 kernel/hung_task.c:379
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 24 Comm: ksoftirqd/1 Not tainted 6.12.0-rc3-syzkaller-00454-gdb87114dcf13 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:arch_stack_walk+0x112/0x150 arch/x86/kernel/stacktrace.c:27
Code: ff ff ff 00 74 37 48 8d 9d 70 ff ff ff 48 89 df e8 b3 9c 09 00 48 85 c0 74 23 4c 89 f7 48 89 c6 4d 89 eb e8 60 b3 98 0a 84 c0 <74> 11 48 89 df e8 e4 9d 09 00 83 bd 70 ff ff ff 00 75 d0 65 48 8b
RSP: 0018:ffffc900001e69c0 EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffc900001e69c0 RCX: ffffffff917bc000
RDX: dffffc0000000000 RSI: ffffffff89db18c5 RDI: ffffc900001e6aac
RBP: ffffc900001e6a50 R08: 0000000000000008 R09: ffffc900001e6ab0
R10: dffffc0000000000 R11: ffffffff8180a0c0 R12: ffff88801d6e8000
R13: ffffffff8180a0c0 R14: ffffc900001e6aa0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8880b8700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe50c204f98 CR3: 000000000e734000 CR4: 0000000000350ef0
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 stack_trace_save+0x118/0x1d0 kernel/stacktrace.c:122
 kasan_save_stack mm/kasan/common.c:47 [inline]
 kasan_save_track+0x3f/0x80 mm/kasan/common.c:68
 unpoison_slab_object mm/kasan/common.c:319 [inline]
 __kasan_slab_alloc+0x66/0x80 mm/kasan/common.c:345
 kasan_slab_alloc include/linux/kasan.h:247 [inline]
 slab_post_alloc_hook mm/slub.c:4085 [inline]
 slab_alloc_node mm/slub.c:4134 [inline]
 kmem_cache_alloc_node_noprof+0x16b/0x320 mm/slub.c:4186
 __alloc_skb+0x1c3/0x440 net/core/skbuff.c:668
 alloc_skb include/linux/skbuff.h:1322 [inline]
 synproxy_send_client_synack_ipv6+0x1ba/0xc30 net/netfilter/nf_synproxy_core.c:873
 nft_synproxy_eval_v6 net/netfilter/nft_synproxy.c:90 [inline]
 nft_synproxy_do_eval+0x739/0xa60 net/netfilter/nft_synproxy.c:145
 expr_call_ops_eval net/netfilter/nf_tables_core.c:240 [inline]
 nft_do_chain+0x4af/0x1da0 net/netfilter/nf_tables_core.c:288
 nft_do_chain_inet+0x418/0x6b0 net/netfilter/nft_chain_filter.c:161
 nf_hook_entry_hookfn include/linux/netfilter.h:154 [inline]
 nf_hook_slow+0xc5/0x220 net/netfilter/core.c:626
 nf_hook include/linux/netfilter.h:269 [inline]
 NF_HOOK+0x29e/0x450 include/linux/netfilter.h:312
 NF_HOOK+0x3a6/0x450 include/linux/netfilter.h:314
 __netif_receive_skb_one_core net/core/dev.c:5666 [inline]
 __netif_receive_skb+0x1ea/0x650 net/core/dev.c:5779
 process_backlog+0x662/0x15b0 net/core/dev.c:6111
 __napi_poll+0xcd/0x490 net/core/dev.c:6775
 napi_poll net/core/dev.c:6844 [inline]
 net_rx_action+0x89b/0x1240 net/core/dev.c:6966
 handle_softirqs+0x2c7/0x980 kernel/softirq.c:554
 run_ksoftirqd+0xca/0x130 kernel/softirq.c:927
 smpboot_thread_fn+0x546/0xa30 kernel/smpboot.c:164
 kthread+0x2f2/0x390 kernel/kthread.c:389
 ret_from_fork+0x4d/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (31):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/10/21 02:53 upstream db87114dcf13 cd6fc0a3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/10/15 04:26 upstream eca631b8fe80 b01b6661 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/10/15 03:37 upstream eca631b8fe80 b01b6661 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in devlink_pernet_pre_exit
2024/10/08 02:21 upstream 87d6aab2389e 402f1df0 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in devlink_pernet_pre_exit
2024/10/06 22:54 upstream 2a130b7e1fcd d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/10/02 12:23 upstream e32cde8d2bd7 ea2b66a6 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/09/30 04:12 upstream e7ed34365879 ba29ff75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in devlink_pernet_pre_exit
2024/09/25 21:43 upstream 684a64bf32b6 349a68c4 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in devlink_pernet_pre_exit
2024/09/23 04:16 upstream af9c191ac2a0 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in devlink_pernet_pre_exit
2024/09/21 07:18 upstream 1868f9d0260e 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in devlink_pernet_pre_exit
2024/08/24 18:33 upstream d2bafcf224f3 d7d32352 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/08/14 08:00 upstream 6b0f8db921ab bde81f6f .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in devlink_pernet_pre_exit
2024/08/13 10:24 upstream d74da846046a 7b0f4b46 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in devlink_pernet_pre_exit
2024/07/15 08:56 upstream 0c3836482481 c605e6a2 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/06/25 12:56 upstream 55027e689933 215eef4a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/06/23 03:29 upstream 5f583a3162ff edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/06/12 01:46 upstream 2ef5971ff345 c7d7d815 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/06/08 02:48 upstream 96e09b8f8166 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in devlink_pernet_pre_exit
2024/06/04 00:34 upstream f06ce441457d 0aba2352 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/05/27 12:12 upstream 1613e604df0c a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/10/14 09:42 upstream 6485cf5ea253 084d8178 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in devlink_pernet_pre_exit
2024/10/05 20:04 upstream fc20a3e57247 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in devlink_pernet_pre_exit
2024/09/22 00:04 upstream 88264981f208 6f888b75 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in devlink_pernet_pre_exit
2024/10/10 19:02 net-next 09cf85ef183a 8fbfc0c8 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in devlink_pernet_pre_exit
2024/10/20 07:40 linux-next 15e7d45e786a cd6fc0a3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/10/15 23:15 linux-next b852e1e7a038 bde2d81c .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/10/14 21:09 linux-next 7f773fd61baa b01b6661 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/04/26 16:50 linux-next bb7a2467e6be 059e9963 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in devlink_pernet_pre_exit
2024/10/04 04:57 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 5f5673607153 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 INFO: task hung in devlink_pernet_pre_exit
2024/05/28 22:11 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci fda5695d692c 34889ee3 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 INFO: task hung in devlink_pernet_pre_exit
2024/10/12 02:16 git://git.kernel.org/pub/scm/linux/kernel/git/riscv/linux.git fixes cfb10de18538 084d8178 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu2-riscv64 INFO: task hung in devlink_pernet_pre_exit
* Struck through repros no longer work on HEAD.