syzbot


INFO: task hung in synchronize_rcu (4)

Status: upstream: reported C repro on 2023/05/04 02:01
Labels: fs (incorrect?)
Reported-by: syzbot+222aa26d0a5dbc2e84fe@syzkaller.appspotmail.com
First crash: 72d, last: 9d01h

Cause bisection: introduced by (bisect log) :
commit 3b5d4ddf8fe1f60082513f94bae586ac80188a03
Author: Martin KaFai Lau <kafai@fb.com>
Date: Wed Mar 9 09:04:50 2022 +0000

  bpf: net: Remove TC_AT_INGRESS_OFFSET and SKB_MONO_DELIVERY_TIME_OFFSET macro

Crash: INFO: task hung in synchronize_rcu (log)
Repro: syz .config
Discussions (2)
Title Replies (including bot) Last reply
[syzbot] [fs?] INFO: task hung in synchronize_rcu (4) 2 (4) 2023/05/21 05:04
Re: [syzbot] [fs?] INFO: task hung in synchronize_rcu (4) 4 (4) 2023/05/05 09:49
Similar bugs (9)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in synchronize_rcu (3) C done done 374 105d 1043d 24/24 fixed on 2023/02/24 13:50
linux-6.1 INFO: task hung in synchronize_rcu origin:lts-only C 1 30d 60d 0/3 upstream: reported C repro on 2023/04/01 04:49
android-49 INFO: task hung in synchronize_rcu 1 1279d 1279d 0/3 auto-closed as invalid on 2020/03/28 16:57
linux-4.14 INFO: task hung in synchronize_rcu vfs C error 162 120d 1301d 0/1 upstream: reported C repro on 2019/11/07 04:32
linux-4.19 INFO: task hung in synchronize_rcu C error 70 109d 1300d 0/1 upstream: reported C repro on 2019/11/08 21:27
linux-5.15 INFO: task hung in synchronize_rcu 1 46d 46d 0/3 upstream: reported on 2023/04/16 02:04
upstream INFO: task hung in synchronize_rcu syz done 1206 1279d 1305d 15/24 fixed on 2019/11/29 15:48
android-414 INFO: task hung in synchronize_rcu 1 1297d 1297d 0/1 auto-closed as invalid on 2020/03/10 10:42
upstream INFO: task hung in synchronize_rcu (2) 8 1278d 1279d 0/24 closed as invalid on 2019/11/30 16:54
Last patch testing requests (5)
Created Duration User Patch Repo Result
2023/05/25 12:07 18m penguin-kernel@i-love.sakura.ne.jp patch upstream report log
2023/05/25 11:18 22m penguin-kernel@i-love.sakura.ne.jp patch upstream OK log
2023/05/25 10:47 21m penguin-kernel@i-love.sakura.ne.jp patch upstream OK log
2023/05/24 02:02 30m penguin-kernel@i-love.sakura.ne.jp patch upstream report log
2023/05/23 13:13 19m penguin-kernel@i-love.sakura.ne.jp patch upstream error

Sample crash report:
INFO: task dhcpcd:10860 blocked for more than 143 seconds.
      Not tainted 6.4.0-rc2-syzkaller-00481-gdcbe4ea1985d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dhcpcd          state:D stack:29024 pid:10860 ppid:4670   flags:0x00000002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5343 [inline]
 __schedule+0xc9a/0x5880 kernel/sched/core.c:6669
 schedule+0xde/0x1a0 kernel/sched/core.c:6745
 exp_funnel_lock kernel/rcu/tree_exp.h:316 [inline]
 synchronize_rcu_expedited+0x6f8/0x770 kernel/rcu/tree_exp.h:992
 synchronize_rcu+0x2f1/0x3a0 kernel/rcu/tree.c:3499
 synchronize_net+0x4e/0x60 net/core/dev.c:10791
 __unregister_prot_hook+0x4b3/0x5c0 net/packet/af_packet.c:380
 packet_do_bind+0x93f/0xe30 net/packet/af_packet.c:3235
 packet_bind+0x15f/0x1c0 net/packet/af_packet.c:3319
 __sys_bind+0x1ed/0x260 net/socket.c:1803
 __do_sys_bind net/socket.c:1814 [inline]
 __se_sys_bind net/socket.c:1812 [inline]
 __x64_sys_bind+0x73/0xb0 net/socket.c:1812
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f522deb3677
RSP: 002b:00007ffec7fc94f8 EFLAGS: 00000217 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 000055e71e865ca3 RCX: 00007f522deb3677
RDX: 0000000000000014 RSI: 00007ffec7fc9508 RDI: 0000000000000005
RBP: 0000000000000000 R08: 000055e7200048a0 R09: 0000000000000004
R10: 000000000000006d R11: 0000000000000217 R12: 000055e71ffff250
R13: 000055e720004788 R14: 00007ffec7fe9dec R15: 000055e720004754
 </TASK>
INFO: task dhcpcd:10906 blocked for more than 145 seconds.
      Not tainted 6.4.0-rc2-syzkaller-00481-gdcbe4ea1985d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dhcpcd          state:D stack:28864 pid:10906 ppid:4670   flags:0x00000002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5343 [inline]
 __schedule+0xc9a/0x5880 kernel/sched/core.c:6669
 schedule+0xde/0x1a0 kernel/sched/core.c:6745
 exp_funnel_lock kernel/rcu/tree_exp.h:316 [inline]
 synchronize_rcu_expedited+0x6f8/0x770 kernel/rcu/tree_exp.h:992
 synchronize_rcu+0x2f1/0x3a0 kernel/rcu/tree.c:3499
 synchronize_net+0x4e/0x60 net/core/dev.c:10791
 __unregister_prot_hook+0x4b3/0x5c0 net/packet/af_packet.c:380
 packet_do_bind+0x93f/0xe30 net/packet/af_packet.c:3235
 packet_bind+0x15f/0x1c0 net/packet/af_packet.c:3319
 __sys_bind+0x1ed/0x260 net/socket.c:1803
 __do_sys_bind net/socket.c:1814 [inline]
 __se_sys_bind net/socket.c:1812 [inline]
 __x64_sys_bind+0x73/0xb0 net/socket.c:1812
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f522deb3677
RSP: 002b:00007ffec7fc94f8 EFLAGS: 00000217 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 000055e71e865ca3 RCX: 00007f522deb3677
RDX: 0000000000000014 RSI: 00007ffec7fc9508 RDI: 0000000000000005
RBP: 0000000000000000 R08: 000055e720004a20 R09: 0000000000000004
R10: 000000000000006d R11: 0000000000000217 R12: 000055e71ffff250
R13: 000055e720004908 R14: 00007ffec7fe9dec R15: 000055e7200048d4
 </TASK>
INFO: task dhcpcd:11298 blocked for more than 147 seconds.
      Not tainted 6.4.0-rc2-syzkaller-00481-gdcbe4ea1985d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dhcpcd          state:D stack:29008 pid:11298 ppid:4670   flags:0x00000002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5343 [inline]
 __schedule+0xc9a/0x5880 kernel/sched/core.c:6669
 schedule+0xde/0x1a0 kernel/sched/core.c:6745
 exp_funnel_lock kernel/rcu/tree_exp.h:316 [inline]
 synchronize_rcu_expedited+0x6f8/0x770 kernel/rcu/tree_exp.h:992
 synchronize_rcu+0x2f1/0x3a0 kernel/rcu/tree.c:3499
 synchronize_net+0x4e/0x60 net/core/dev.c:10791
 __unregister_prot_hook+0x4b3/0x5c0 net/packet/af_packet.c:380
 packet_do_bind+0x93f/0xe30 net/packet/af_packet.c:3235
 packet_bind+0x15f/0x1c0 net/packet/af_packet.c:3319
 __sys_bind+0x1ed/0x260 net/socket.c:1803
 __do_sys_bind net/socket.c:1814 [inline]
 __se_sys_bind net/socket.c:1812 [inline]
 __x64_sys_bind+0x73/0xb0 net/socket.c:1812
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f522deb3677
RSP: 002b:00007ffec7fc94f8 EFLAGS: 00000217
 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 000055e71e865ca3 RCX: 00007f522deb3677
RDX: 0000000000000014 RSI: 00007ffec7fc9508 RDI: 0000000000000005
RBP: 0000000000000000 R08: 000055e720004420 R09: 0000000000000004
R10: 000000000000006d R11: 0000000000000217 R12: 000055e71ffff250
R13: 000055e720004a88 R14: 00007ffec7fe9dec R15: 000055e720004a54
 </TASK>
INFO: task dhcpcd:11328 blocked for more than 149 seconds.
      Not tainted 6.4.0-rc2-syzkaller-00481-gdcbe4ea1985d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dhcpcd          state:D stack:28320 pid:11328 ppid:4670   flags:0x00000002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5343 [inline]
 __schedule+0xc9a/0x5880 kernel/sched/core.c:6669
 schedule+0xde/0x1a0 kernel/sched/core.c:6745
 exp_funnel_lock kernel/rcu/tree_exp.h:316 [inline]
 synchronize_rcu_expedited+0x6f8/0x770 kernel/rcu/tree_exp.h:992
 synchronize_rcu+0x2f1/0x3a0 kernel/rcu/tree.c:3499
 synchronize_net+0x4e/0x60 net/core/dev.c:10791
 __unregister_prot_hook+0x4b3/0x5c0 net/packet/af_packet.c:380
 packet_do_bind+0x93f/0xe30 net/packet/af_packet.c:3235
 packet_bind+0x15f/0x1c0 net/packet/af_packet.c:3319
 __sys_bind+0x1ed/0x260 net/socket.c:1803
 __do_sys_bind net/socket.c:1814 [inline]
 __se_sys_bind net/socket.c:1812 [inline]
 __x64_sys_bind+0x73/0xb0 net/socket.c:1812
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x39/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f522deb3677
RSP: 002b:00007ffec7fc94f8 EFLAGS: 00000217 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 000055e71e865ca3 RCX: 00007f522deb3677
RDX: 0000000000000014 RSI: 00007ffec7fc9508 RDI: 0000000000000005
RBP: 0000000000000000 R08: 000055e720004420 R09: 0000000000000004
R10: 000000000000006d R11: 0000000000000217 R12: 000055e71ffff250
R13: 000055e720004c08 R14: 00007ffec7fe9dec R15: 000055e720004bd4
 </TASK>

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/13:
 #0: ffffffff8c798430 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0xd80 kernel/rcu/tasks.h:518
1 lock held by rcu_tasks_trace/14:
 #0: ffffffff8c798130 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x31/0xd80 kernel/rcu/tasks.h:518
1 lock held by khungtaskd/28:
 #0: ffffffff8c799040 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x340 kernel/locking/lockdep.c:6545
1 lock held by kswapd0/84:
2 locks held by kswapd1/85:
3 locks held by kworker/0:2/760:
3 locks held by kworker/1:2/1126:
1 lock held by syslogd/4438:
2 locks held by klogd/4445:
1 lock held by dhcpcd/4669:
2 locks held by getty/4756:
 #0: ffff8880286bf098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x26/0x80 drivers/tty/tty_ldisc.c:243
 #1: ffffc900015902f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0xef4/0x13e0 drivers/tty/n_tty.c:2176
2 locks held by sshd/5018:
2 locks held by syz-executor300/5025:
2 locks held by dhcpcd/10797:
 #0: ffff8880734ebe10 (&sb->s_type->i_mutex_key#10){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:775 [inline]
 #0: ffff8880734ebe10 (&sb->s_type->i_mutex_key#10){+.+.}-{3:3}, at: __sock_release+0x86/0x290 net/socket.c:652
 #1: ffffffff8c7a44b8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:325 [inline]
 #1: ffffffff8c7a44b8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x3e8/0x770 kernel/rcu/tree_exp.h:992
2 locks held by dhcpcd/10818:
 #0: ffff8880217f2130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1697 [inline]
 #0: ffff8880217f2130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x2f/0xe30 net/packet/af_packet.c:3202
 #1: ffffffff8c7a44b8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:325 [inline]
 #1: ffffffff8c7a44b8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x3e8/0x770 kernel/rcu/tree_exp.h:992
1 lock held by dhcpcd/10860:
 #0: ffff88807b95c130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1697 [inline]
 #0: ffff88807b95c130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x2f/0xe30 net/packet/af_packet.c:3202
1 lock held by dhcpcd/10906:
 #0: ffff888076080130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1697 [inline]
 #0: ffff888076080130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x2f/0xe30 net/packet/af_packet.c:3202
1 lock held by dhcpcd/11298:
 #0: ffff888027a66130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: lock_sock include/net/sock.h:1697 [inline]
 #0: ffff888027a66130 (sk_lock-AF_PACKET){+.+.}-{0:0}, at: packet_do_bind+0x2f/0xe30 net/packet/af_packet.c:3202

Crashes (9):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Manager Title
2023/05/20 22:13 net-next dcbe4ea1985d 4bce1a3e .config console log report syz C [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/05/15 07:26 upstream 838a854820ee 2b9ba477 .config strace log report syz [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in synchronize_rcu
2023/04/30 01:55 net 6686317855c6 62df2017 .config console log report syz [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in synchronize_rcu
2023/04/09 04:38 upstream a79d5c76f705 71147e29 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in synchronize_rcu
2023/03/20 22:00 upstream 7d31677bb7b1 7939252e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce INFO: task hung in synchronize_rcu
2023/05/18 11:07 net ab87603b2511 3bb7af1d .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-this-kasan-gce INFO: task hung in synchronize_rcu
2023/04/20 10:56 net-next a5962e0a8b7b 2b32bd34 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-net-kasan-gce INFO: task hung in synchronize_rcu
2023/05/23 00:47 linux-next 715abedee4cd 4bce1a3e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in synchronize_rcu
2023/05/21 03:03 linux-next 715abedee4cd 4bce1a3e .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in synchronize_rcu
* Struck through repros no longer work on HEAD.