syzbot


INFO: task hung in synchronize_rcu (4)

Status: upstream: reported C repro on 2023/05/04 02:01
Subsystems: net fs
[Documentation on labels]
Reported-by: syzbot+222aa26d0a5dbc2e84fe@syzkaller.appspotmail.com
First crash: 261d, last: 5d01h
Cause bisection: introduced by (bisect log) :
commit 3b5d4ddf8fe1f60082513f94bae586ac80188a03
Author: Martin KaFai Lau <kafai@fb.com>
Date: Wed Mar 9 09:04:50 2022 +0000

  bpf: net: Remove TC_AT_INGRESS_OFFSET and SKB_MONO_DELIVERY_TIME_OFFSET macro

Crash: INFO: task hung in synchronize_rcu (log)
Repro: syz .config
  
Fix bisection: failed (error log, bisect log)
  
Discussions (4)
Title Replies (including bot) Last reply
[syzbot] Monthly fs report (Aug 2023) 0 (1) 2023/08/08 12:55
[syzbot] Monthly fs report (Jul 2023) 0 (1) 2023/07/10 09:36
[syzbot] [fs?] INFO: task hung in synchronize_rcu (4) 2 (4) 2023/05/21 05:04
Re: [syzbot] [fs?] INFO: task hung in synchronize_rcu (4) 4 (4) 2023/05/05 09:49
Similar bugs (9)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in synchronize_rcu (3) C done done 374 295d 1233d 24/25 fixed on 2023/02/24 13:50
linux-6.1 INFO: task hung in synchronize_rcu origin:lts-only C unreliable 3 33d 250d 0/3 upstream: reported C repro on 2023/04/01 04:49
android-49 INFO: task hung in synchronize_rcu 1 1469d 1469d 0/3 auto-closed as invalid on 2020/03/28 16:57
linux-4.14 INFO: task hung in synchronize_rcu vfs C error 162 309d 1491d 0/1 upstream: reported C repro on 2019/11/07 04:32
linux-4.19 INFO: task hung in synchronize_rcu C error 70 299d 1489d 0/1 upstream: reported C repro on 2019/11/08 21:27
linux-5.15 INFO: task hung in synchronize_rcu 1 235d 235d 0/3 auto-obsoleted due to no activity on 2023/08/14 02:05
upstream INFO: task hung in synchronize_rcu syz done 1206 1469d 1495d 15/25 fixed on 2019/11/29 15:48
android-414 INFO: task hung in synchronize_rcu 1 1487d 1487d 0/1 auto-closed as invalid on 2020/03/10 10:42
upstream INFO: task hung in synchronize_rcu (2) trace 8 1468d 1468d 0/25 closed as invalid on 2019/11/30 16:54
Last patch testing requests (10)
Created Duration User Patch Repo Result
2023/08/28 15:24 45m (2) retest repro upstream OK log
2023/08/28 15:24 34m retest repro net-next OK log
2023/08/28 15:24 33m retest repro net-next OK log
2023/08/28 15:24 22m retest repro net OK log
2023/08/28 15:24 26m retest repro net OK log
2023/05/25 12:07 18m penguin-kernel@i-love.sakura.ne.jp patch upstream report log
2023/05/25 11:18 22m penguin-kernel@i-love.sakura.ne.jp patch upstream OK log
2023/05/25 10:47 21m penguin-kernel@i-love.sakura.ne.jp patch upstream OK log
2023/05/24 02:02 30m penguin-kernel@i-love.sakura.ne.jp patch upstream report log
2023/05/23 13:13 19m penguin-kernel@i-love.sakura.ne.jp patch upstream error OK

Sample crash report:
INFO: task dhcpcd:5133 blocked for more than 150 seconds.
      Not tainted 6.4.0-rc7-syzkaller-01944-g3674fbf0451d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dhcpcd          state:D stack:29016 pid:5133  ppid:4676   flags:0x00000002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5343 [inline]
 __schedule+0xc9a/0x5880 kernel/sched/core.c:6669
 schedule+0xde/0x1a0 kernel/sched/core.c:6745
 synchronize_rcu_expedited+0x5d4/0x770 kernel/rcu/tree_exp.h:1007
 synchronize_rcu+0x2f1/0x3a0 kernel/rcu/tree.c:3499
 synchronize_net+0x4e/0x60 net/core/dev.c:10759
 __unregister_prot_hook+0x4b3/0x5c0 net/packet/af_packet.c:380
 packet_do_bind+0x994/0xe90 net/packet/af_packet.c:3238
 packet_bind+0x105/0x150 net/packet/af_packet.c:3322
 __sys_bind+0x1ed/0x260 net/socket.c:1791
 __do_sys_bind net/socket.c:1802 [inline]
 __se_sys_bind net/socket.c:1800 [inline]
 __x64_sys_bind+0x73/0xb0 net/socket.c:1800
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f2419686677
RSP: 002b:00007ffde955d438 EFLAGS: 00000217 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 000055d51aa09ca3 RCX: 00007f2419686677
RDX: 0000000000000014 RSI: 00007ffde955d448 RDI: 0000000000000005
RBP: 0000000000000000 R08: 000055d51bb38f30 R09: 0000000000200000
R10: 000000000000006d R11: 0000000000000217 R12: 000055d51bb351e0
R13: 000055d51bb3a138 R14: 00007ffde957dd2c R15: 000055d51bb3a104
 </TASK>

Showing all locks held in the system:
3 locks held by kworker/0:1/9:
1 lock held by rcu_tasks_kthre/13:
 #0: ffffffff8c7984f0 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0xd80 kernel/rcu/tasks.h:518
1 lock held by rcu_tasks_trace/14:
 #0: ffffffff8c7981f0 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0xd80 kernel/rcu/tasks.h:518
1 lock held by khungtaskd/28:
 #0: ffffffff8c799100 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x340 kernel/locking/lockdep.c:6559
3 locks held by kworker/u4:4/75:
2 locks held by getty/4764:
 #0: ffff88814a9cc098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x26/0x80 drivers/tty/tty_ldisc.c:243
 #1: ffffc900015802f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0xef4/0x13e0 drivers/tty/n_tty.c:2176
2 locks held by dhcpcd/5133:
 #0: ffff88806eb3a130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1708 [inline]
 #0: ffff88806eb3a130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x32/0xe90 net/packet/af_packet.c:3202
 #1: ffffffff8c7a4578 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:293 [inline]
 #1: ffffffff8c7a4578 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x64a/0x770 kernel/rcu/tree_exp.h:992
2 locks held by dhcpcd/5146:
 #0: ffff88807dfd6130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1708 [inline]
 #0: ffff88807dfd6130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x32/0xe90 net/packet/af_packet.c:3202
 #1: ffffffff8c7a4578 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:325 [inline]
 #1: ffffffff8c7a4578 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x3e8/0x770 kernel/rcu/tree_exp.h:992
1 lock held by dhcpcd/5147:
 #0: ffff88807b7ee130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1708 [inline]
 #0: ffff88807b7ee130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x32/0xe90 net/packet/af_packet.c:3202
1 lock held by dhcpcd/5406:
 #0: ffff88807d064130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1708 [inline]
 #0: ffff88807d064130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x32/0xe90 net/packet/af_packet.c:3202
1 lock held by dhcpcd/5407:
 #0: ffff88807d066130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1708 [inline]
 #0: ffff88807d066130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x32/0xe90 net/packet/af_packet.c:3202
1 lock held by dhcpcd/5408:
 #0: ffff88802b9d0130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1708 [inline]
 #0: ffff88802b9d0130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x32/0xe90 net/packet/af_packet.c:3202

=============================================

NMI backtrace for cpu 0
CPU: 0 PID: 28 Comm: khungtaskd Not tainted 6.4.0-rc7-syzkaller-01944-g3674fbf0451d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/27/2023
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xd9/0x150 lib/dump_stack.c:106
 nmi_cpu_backtrace+0x29c/0x350 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x2a4/0x300 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:222 [inline]
 watchdog+0xe16/0x1090 kernel/hung_task.c:379
 kthread+0x344/0x440 kernel/kthread.c:379
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 5461 Comm: syz-executor138 Not tainted 6.4.0-rc7-syzkaller-01944-g3674fbf0451d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 05/27/2023
RIP: 0010:__might_sleep+0x0/0x160 kernel/sched/core.c:10069
Code: 9b fd ff ff 48 c7 c7 00 f2 40 8c e8 3a 29 7f 00 e9 c5 fb ff ff e8 30 29 7f 00 e9 10 fc ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 <f3> 0f 1e fa 48 b8 00 00 00 00 00 fc ff df 41 54 41 89 f4 55 48 89
RSP: 0018:ffffc9000352f4f8 EFLAGS: 00000206
RAX: 0000000000000000 RBX: ffffc9000352f5b8 RCX: ffffc9000352f488
RDX: 1ffff110040bb8c7 RSI: 0000000000000132 RDI: ffffffff8a5925c0
RBP: 0000000000000000 R08: 0000000000000001 R09: ffffffff8e7b1f17
R10: fffffbfff1cf63e2 R11: 0000000000094001 R12: 0000000000000000
R13: 0000000000000cc0 R14: ffffc9000352f5c0 R15: 0000000000000000
FS:  00007f83b5502700(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83b55a9528 CR3: 0000000059262000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 might_alloc include/linux/sched/mm.h:306 [inline]
 prepare_alloc_pages+0x3da/0x570 mm/page_alloc.c:4539
 __alloc_pages+0x149/0x4a0 mm/page_alloc.c:4757
 alloc_pages+0x1aa/0x270 mm/mempolicy.c:2279
 __get_free_pages+0xc/0x40 mm/page_alloc.c:4818
 kasan_populate_vmalloc_pte mm/kasan/shadow.c:323 [inline]
 kasan_populate_vmalloc_pte+0x2e/0x180 mm/kasan/shadow.c:314
 apply_to_pte_range mm/memory.c:2585 [inline]
 apply_to_pmd_range mm/memory.c:2629 [inline]
 apply_to_pud_range mm/memory.c:2665 [inline]
 apply_to_p4d_range mm/memory.c:2701 [inline]
 __apply_to_page_range+0x68c/0x1030 mm/memory.c:2735
 alloc_vmap_area+0x500/0x1e00 mm/vmalloc.c:1642
 __get_vm_area_node+0x145/0x3f0 mm/vmalloc.c:2551
 __vmalloc_node_range+0x252/0x14c0 mm/vmalloc.c:3226
 kvmalloc_node+0x156/0x1a0 mm/util.c:629
 kvmalloc include/linux/slab.h:697 [inline]
 xt_alloc_table_info+0x40/0xa0 net/netfilter/x_tables.c:1192
 do_replace net/ipv4/netfilter/ip_tables.c:1122 [inline]
 do_ipt_set_ctl+0x514/0xb10 net/ipv4/netfilter/ip_tables.c:1627
 nf_setsockopt+0x87/0xe0 net/netfilter/nf_sockopt.c:101
 ip_setsockopt+0xf2/0x110 net/ipv4/ip_sockglue.c:1467
 raw_setsockopt+0x23e/0x2d0 net/ipv4/raw.c:835
 __sys_setsockopt+0x2c6/0x5b0 net/socket.c:2262
 __do_sys_setsockopt net/socket.c:2273 [inline]
 __se_sys_setsockopt net/socket.c:2270 [inline]
 __x64_sys_setsockopt+0xbe/0x160 net/socket.c:2270
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f83b555bd79
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 91 18 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f83b5502318 EFLAGS: 00000246 ORIG_RAX: 0000000000000036
RAX: ffffffffffffffda RBX: 00007f83b55dd488 RCX: 00007f83b555bd79
RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 00007f83b55dd480 R08: 0000000000000258 R09: 0000000000000000
R10: 0000000020000500 R11: 0000000000000246 R12: 0000000020000560
R13: 00007ffd1bdb827f R14: 00007f83b5502400 R15: 0000000000022000
 </TASK>

Crashes (25):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/07/05 16:36 net 3674fbf0451d 80298b6f .config strace log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in synchronize_rcu
2023/07/05 16:01 net-next ae230642190a 80298b6f .config strace log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/05/20 22:13 net-next dcbe4ea1985d 4bce1a3e .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/05/15 07:26 upstream 838a854820ee 2b9ba477 .config strace log report syz [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in synchronize_rcu
2023/04/30 01:55 net 6686317855c6 62df2017 .config console log report syz [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in synchronize_rcu
2023/12/02 17:22 upstream 815fb87b7530 f819d6f7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in synchronize_rcu
2023/09/01 12:18 upstream 99d99825fc07 696ea0d2 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in synchronize_rcu
2023/07/29 11:09 upstream ffabf7c73176 92476829 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in synchronize_rcu
2023/07/09 14:30 upstream 1c7873e33645 668cb1fa .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in synchronize_rcu
2023/07/07 09:20 upstream a452483508d7 22ae5830 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in synchronize_rcu
2023/06/15 10:19 upstream b6dad5178cea 76decb82 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in synchronize_rcu
2023/06/06 04:39 upstream f8dba31b0a82 a4ae4f42 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in synchronize_rcu
2023/04/09 04:38 upstream a79d5c76f705 71147e29 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in synchronize_rcu
2023/03/20 22:00 upstream 7d31677bb7b1 7939252e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in synchronize_rcu
2023/06/05 15:39 upstream 9561de3a55be a4ae4f42 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in synchronize_rcu
2023/06/05 03:07 upstream 6f64a5ebe1dc a4ae4f42 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-386 INFO: task hung in synchronize_rcu
2023/06/29 15:45 net 3674fbf0451d ca69c785 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in synchronize_rcu
2023/05/18 11:07 net ab87603b2511 3bb7af1d .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in synchronize_rcu
2023/10/17 19:20 net-next 7713ec844756 342b9c55 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/06/15 12:46 net-next 97c5209b3d37 757d26ed .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/04/20 10:56 net-next a5962e0a8b7b 2b32bd34 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/09/16 14:05 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci fe4469582053 0b6a67ac .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 INFO: task hung in synchronize_rcu
2023/06/03 21:01 linux-next 715abedee4cd a4ae4f42 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in synchronize_rcu
2023/05/23 00:47 linux-next 715abedee4cd 4bce1a3e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in synchronize_rcu
2023/05/21 03:03 linux-next 715abedee4cd 4bce1a3e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in synchronize_rcu
* Struck through repros no longer work on HEAD.