syzbot


INFO: task hung in gfs2_gl_hash_clear (3)

Status: upstream: reported C repro on 2022/11/29 08:32
Reported-by: syzbot+ed7d0f71a89e28557a77@syzkaller.appspotmail.com
First crash: 68d, last: 5d00h

Cause bisection: failed (bisect log)
similar bugs (4):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream INFO: task hung in gfs2_gl_hash_clear 156 822d 827d 0/24 auto-closed as invalid on 2021/02/02 22:47
upstream INFO: task hung in gfs2_gl_hash_clear (2) 1 243d 241d 0/24 auto-closed as invalid on 2022/09/04 20:20
linux-4.19 INFO: task hung in gfs2_gl_hash_clear gfs2 1 32d 32d 0/1 upstream: reported on 2023/01/04 01:01
upstream INFO: task can't die in gfs2_gl_hash_clear (2) 3 654d 748d 0/24 auto-closed as invalid on 2021/07/21 03:15
Last patch testing requests:
Created Duration User Patch Repo Result
2022/12/01 06:15 0m liaoyu15@huawei.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git b7b275e60bcd error
2022/12/01 01:34 0m bobo.shaobowang@huawei.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git b7b275e60bcd error
2022/11/30 10:02 0m bobo.shaobowang@huawei.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git b7b275e60bcd error

Sample crash report:
INFO: task syz-executor786:5062 blocked for more than 143 seconds.
      Not tainted 6.1.0-syzkaller-13052-ged56954cf5a8 #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz-executor786 state:D stack:26624 pid:5062  ppid:5059   flags:0x00004002
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5244 [inline]
 __schedule+0xb8a/0x5450 kernel/sched/core.c:6555
 schedule+0xde/0x1b0 kernel/sched/core.c:6631
 schedule_timeout+0x14e/0x2a0 kernel/time/timer.c:2167
 gfs2_gl_hash_clear+0x247/0x270 fs/gfs2/glock.c:2194
 gfs2_put_super+0x497/0x670 fs/gfs2/super.c:627
 generic_shutdown_super+0x158/0x410 fs/super.c:492
 kill_block_super+0x9b/0xf0 fs/super.c:1386
 gfs2_kill_sb+0x108/0x170 fs/gfs2/ops_fstype.c:1738
 deactivate_locked_super+0x98/0x160 fs/super.c:332
 deactivate_super+0xb1/0xd0 fs/super.c:363
 cleanup_mnt+0x2ae/0x3d0 fs/namespace.c:1291
 task_work_run+0x16f/0x270 kernel/task_work.c:179
 ptrace_notify+0x118/0x140 kernel/signal.c:2354
 ptrace_report_syscall include/linux/ptrace.h:411 [inline]
 ptrace_report_syscall_exit include/linux/ptrace.h:473 [inline]
 syscall_exit_work kernel/entry/common.c:251 [inline]
 syscall_exit_to_user_mode_prepare+0x129/0x280 kernel/entry/common.c:278
 __syscall_exit_to_user_mode_work kernel/entry/common.c:283 [inline]
 syscall_exit_to_user_mode+0xd/0x50 kernel/entry/common.c:296
 do_syscall_64+0x46/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f8f9521acf7
RSP: 002b:00007ffe972267b8 EFLAGS: 00000206 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007f8f9521acf7
RDX: 00007ffe97226879 RSI: 000000000000000a RDI: 00007ffe97226870
RBP: 00007ffe97226870 R08: 00000000ffffffff R09: 00007ffe97226650
R10: 0000555555c24653 R11: 0000000000000206 R12: 00007ffe972278e0
R13: 0000555555c245f0 R14: 00007ffe972267e0 R15: 0000000000000001
 </TASK>

Showing all locks held in the system:
1 lock held by rcu_tasks_kthre/12:
 #0: ffffffff8c590b70 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x26/0xc70 kernel/rcu/tasks.h:507
1 lock held by rcu_tasks_trace/13:
 #0: ffffffff8c590870 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x26/0xc70 kernel/rcu/tasks.h:507
1 lock held by khungtaskd/27:
 #0: ffffffff8c5916c0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x57/0x264 kernel/locking/lockdep.c:6494
2 locks held by getty/4739:
 #0: ffff888029b60098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x26/0x80 drivers/tty/tty_ldisc.c:244
 #1: ffffc900015a02f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0xef4/0x13e0 drivers/tty/n_tty.c:2177
1 lock held by syz-executor786/5062:
 #0: ffff8880294f80e0 (&type->s_umount_key#42){+.+.}-{3:3}, at: deactivate_super+0xa9/0xd0 fs/super.c:362

=============================================

NMI backtrace for cpu 0
CPU: 0 PID: 27 Comm: khungtaskd Not tainted 6.1.0-syzkaller-13052-ged56954cf5a8 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xd1/0x138 lib/dump_stack.c:106
 nmi_cpu_backtrace.cold+0x24/0x18a lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x333/0x3c0 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:220 [inline]
 watchdog+0xc75/0xfc0 kernel/hung_task.c:377
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 46 Comm: kworker/u4:3 Not tainted 6.1.0-syzkaller-13052-ged56954cf5a8 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: events_unbound toggle_allocation_gate
RIP: 0010:__default_send_IPI_dest_field+0x80/0x130 arch/x86/kernel/apic/ipi.c:173
Code: 00 c3 5f ff f6 c4 10 75 e2 44 89 e7 c1 e7 18 89 3c 25 10 c3 5f ff 89 f0 09 d8 80 cf 04 83 fe 02 0f 44 c3 89 04 25 00 c3 5f ff <48> 83 c4 08 5b 5d 41 5c c3 48 c7 c0 40 bd f2 8b 48 ba 00 00 00 00
RSP: 0018:ffffc90000b778a8 EFLAGS: 00000006
RAX: 00000000000008fb RBX: 0000000000000c00 RCX: 0000000000000001
RDX: 0000000000000800 RSI: 00000000000000fb RDI: 0000000001000000
RBP: fffffbffffebf860 R08: 0000000000000005 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000001
R13: ffffc90000b77910 R14: 0000000000000002 R15: dffffc0000000000
FS:  0000000000000000(0000) GS:ffff8880b9b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055bed656a680 CR3: 000000000c28e000 CR4: 0000000000350ee0
Call Trace:
 <TASK>
 _flat_send_IPI_mask+0x4e/0x60 arch/x86/kernel/apic/apic_flat_64.c:56
 arch_send_call_function_single_ipi arch/x86/include/asm/smp.h:108 [inline]
 send_call_function_single_ipi+0x1ed/0x3b0 kernel/sched/core.c:3774
 smp_call_function_many_cond+0xe64/0x10a0 kernel/smp.c:967
 on_each_cpu_cond_mask+0x5a/0xa0 kernel/smp.c:1155
 on_each_cpu include/linux/smp.h:71 [inline]
 text_poke_sync arch/x86/kernel/alternative.c:1772 [inline]
 text_poke_bp_batch+0x3f1/0x6b0 arch/x86/kernel/alternative.c:2016
 text_poke_flush arch/x86/kernel/alternative.c:2131 [inline]
 text_poke_flush arch/x86/kernel/alternative.c:2128 [inline]
 text_poke_finish+0x1a/0x30 arch/x86/kernel/alternative.c:2138
 arch_jump_label_transform_apply+0x17/0x30 arch/x86/kernel/jump_label.c:146
 jump_label_update+0x32f/0x410 kernel/jump_label.c:829
 static_key_disable_cpuslocked+0x156/0x1b0 kernel/jump_label.c:235
 static_key_disable+0x1a/0x20 kernel/jump_label.c:243
 toggle_allocation_gate mm/kfence/core.c:804 [inline]
 toggle_allocation_gate+0x143/0x230 mm/kfence/core.c:791
 process_one_work+0x9bf/0x1710 kernel/workqueue.c:2289
 worker_thread+0x669/0x1090 kernel/workqueue.c:2436
 kthread+0x2e8/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:308
 </TASK>

Fix bisection attempts:
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Title
ci-upstream-kasan-gce-root 2023/01/31 11:59 upstream 22b8077d0fce 05494336 .config console log report syz C
* Struck through repros no longer work on HEAD.
Crashes (8):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets Title
ci-upstream-kasan-gce-root 2022/12/18 03:47 upstream ed56954cf5a8 05494336 .config strace log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] INFO: task hung in gfs2_gl_hash_clear
ci-upstream-gce-arm64 2022/11/29 05:54 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 6d464646530f ca9683b8 .config console log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] INFO: task hung in gfs2_gl_hash_clear
ci2-upstream-fs 2022/12/21 09:48 upstream 6feb57c2fd7c d3e76707 .config console log report info [disk image] [vmlinux] [kernel image] INFO: task hung in gfs2_gl_hash_clear
ci2-upstream-fs 2022/12/18 14:39 upstream f9ff5644bcc0 05494336 .config console log report info [disk image] [vmlinux] [kernel image] INFO: task hung in gfs2_gl_hash_clear
ci2-upstream-fs 2022/12/18 11:42 upstream f9ff5644bcc0 05494336 .config console log report info [disk image] [vmlinux] [kernel image] INFO: task hung in gfs2_gl_hash_clear
ci2-upstream-fs 2022/12/11 23:23 upstream 4cee37b3a4e6 67be1ae7 .config console log report info [disk image] [vmlinux] [kernel image] INFO: task hung in gfs2_gl_hash_clear
ci2-upstream-fs 2022/12/05 11:16 upstream 0ba09b173387 e080de16 .config console log report info [disk image] [vmlinux] [kernel image] INFO: task hung in gfs2_gl_hash_clear
ci-upstream-gce-arm64 2022/12/07 11:27 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci a5541c0811a0 d88f3abb .config console log report info [disk image] [vmlinux] [kernel image] INFO: task hung in gfs2_gl_hash_clear
* Struck through repros no longer work on HEAD.