syzbot


possible deadlock in __schedule (2)

Status: auto-obsoleted due to no activity on 2024/01/09 23:41
Subsystems: kernel
[Documentation on labels]
Reported-by: syzbot+39a85bc0224f82336405@syzkaller.appspotmail.com
First crash: 203d, last: 188d
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [kernel?] possible deadlock in __schedule (2) 0 (1) 2023/10/20 11:28
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in __schedule kernel 1 1091d 1085d 0/26 auto-closed as invalid on 2021/09/09 01:04

Sample crash report:
EEVDF scheduling fail, picking leftmost
======================================================
WARNING: possible circular locking dependency detected
6.6.0-rc6-syzkaller-00285-g9c5d00cb7b6b #0 Not tainted
------------------------------------------------------
udevd/2923 is trying to acquire lock:
ffff8000865271b8 ((console_sem).lock){-.-.}-{2:2}, at: down_trylock+0x18/0x80 kernel/locking/semaphore.c:139

but task is already holding lock:
ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested kernel/sched/core.c:558 [inline]
ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock kernel/sched/sched.h:1372 [inline]
ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: rq_lock kernel/sched/sched.h:1681 [inline]
ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x268/0x2ae4 kernel/sched/core.c:6612

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&rq->__lock){-.-.}-{2:2}:
       _raw_spin_lock_nested+0x50/0x6c kernel/locking/spinlock.c:378
       raw_spin_rq_lock_nested+0x2c/0x44 kernel/sched/core.c:558
       raw_spin_rq_lock kernel/sched/sched.h:1372 [inline]
       rq_lock kernel/sched/sched.h:1681 [inline]
       task_fork_fair+0x70/0x13c kernel/sched/fair.c:12416
       sched_cgroup_fork+0x35c/0x520 kernel/sched/core.c:4816
       copy_process+0x2fb0/0x5520 kernel/fork.c:2609
       kernel_clone+0x140/0x7e8 kernel/fork.c:2909
       user_mode_thread+0xb4/0xf0 kernel/fork.c:2987
       rest_init+0x2c/0x210 init/main.c:691
       arch_post_acpi_subsys_init+0x0/0x8 init/main.c:823
       start_kernel+0x328/0x3a0 init/main.c:1068
       __primary_switched+0xb8/0xc0 arch/arm64/kernel/head.S:523

-> #1 (&p->pi_lock){-.-.}-{2:2}:
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x58/0x80 kernel/locking/spinlock.c:162
       class_raw_spinlock_irqsave_constructor include/linux/spinlock.h:518 [inline]
       try_to_wake_up+0xac/0x1924 kernel/sched/core.c:4230
       wake_up_process+0x18/0x24 kernel/sched/core.c:4478
       __up.isra.0+0x124/0x18c kernel/locking/semaphore.c:278
       up+0x94/0xd4 kernel/locking/semaphore.c:191
       __up_console_sem kernel/printk/printk.c:346 [inline]
       __console_unlock kernel/printk/printk.c:2718 [inline]
       console_unlock+0x1b8/0x1d8 kernel/printk/printk.c:3037
       fb_flashcursor drivers/video/fbdev/core/fbcon.c:382 [inline]
       fb_flashcursor+0x220/0x340 drivers/video/fbdev/core/fbcon.c:348
       process_one_work+0x670/0x143c kernel/workqueue.c:2630
       process_scheduled_works kernel/workqueue.c:2703 [inline]
       worker_thread+0x5a8/0xfb0 kernel/workqueue.c:2784
       kthread+0x27c/0x300 kernel/kthread.c:388
       ret_from_fork+0x10/0x20 arch/arm64/kernel/entry.S:857

-> #0 ((console_sem).lock){-.-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3134 [inline]
       check_prevs_add kernel/locking/lockdep.c:3253 [inline]
       validate_chain kernel/locking/lockdep.c:3868 [inline]
       __lock_acquire+0x2cac/0x6b70 kernel/locking/lockdep.c:5136
       lock_acquire kernel/locking/lockdep.c:5753 [inline]
       lock_acquire+0x480/0x7c8 kernel/locking/lockdep.c:5718
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x58/0x80 kernel/locking/spinlock.c:162
       down_trylock+0x18/0x80 kernel/locking/semaphore.c:139
       __down_trylock_console_sem+0x38/0xd8 kernel/printk/printk.c:329
       console_trylock kernel/printk/printk.c:2671 [inline]
       console_trylock_spinning kernel/printk/printk.c:1927 [inline]
       vprintk_emit+0x334/0x4e4 kernel/printk/printk.c:2306
       vprintk_default+0x38/0x44 kernel/printk/printk.c:2322
       vprintk+0x17c/0x1bc kernel/printk/printk_safe.c:45
       _printk+0xa8/0xe0 kernel/printk/printk.c:2332
       pick_eevdf kernel/sched/fair.c:976 [inline]
       pick_next_entity kernel/sched/fair.c:5278 [inline]
       pick_next_task_fair+0x1a4/0xd3c kernel/sched/fair.c:8222
       __pick_next_task kernel/sched/core.c:6004 [inline]
       pick_next_task kernel/sched/core.c:6514 [inline]
       __schedule+0x3b8/0x2ae4 kernel/sched/core.c:6659
       preempt_schedule_common kernel/sched/core.c:6864 [inline]
       preempt_schedule+0xf4/0x254 kernel/sched/core.c:6888
       __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
       _raw_spin_unlock_irqrestore+0x94/0xac kernel/locking/spinlock.c:194
       debug_object_activate+0x230/0x3f0 lib/debugobjects.c:745
       debug_rcu_head_queue kernel/rcu/rcu.h:226 [inline]
       __call_rcu_common.constprop.0+0x38/0x8c0 kernel/rcu/tree.c:2638
       call_rcu+0x10/0x1c kernel/rcu/tree.c:2767
       file_free fs/file_table.c:82 [inline]
       __fput+0x3f8/0x830 fs/file_table.c:397
       __fput_sync+0x74/0x84 fs/file_table.c:465
       __do_sys_close fs/open.c:1572 [inline]
       __se_sys_close fs/open.c:1557 [inline]
       __arm64_sys_close+0x6c/0xbc fs/open.c:1557
       __invoke_syscall arch/arm64/kernel/syscall.c:37 [inline]
       invoke_syscall+0x6c/0x258 arch/arm64/kernel/syscall.c:51
       el0_svc_common.constprop.0+0xac/0x230 arch/arm64/kernel/syscall.c:136
       do_el0_svc+0x40/0x58 arch/arm64/kernel/syscall.c:155
       el0_svc+0x58/0x140 arch/arm64/kernel/entry-common.c:678
       el0t_64_sync_handler+0x100/0x12c arch/arm64/kernel/entry-common.c:696
       el0t_64_sync+0x190/0x194 arch/arm64/kernel/entry.S:595

other info that might help us debug this:

Chain exists of:
  (console_sem).lock --> &p->pi_lock --> &rq->__lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&rq->__lock);
                               lock(&p->pi_lock);
                               lock(&rq->__lock);
  lock((console_sem).lock);

 *** DEADLOCK ***

1 lock held by udevd/2923:
 #0: ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested kernel/sched/core.c:558 [inline]
 #0: ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock kernel/sched/sched.h:1372 [inline]
 #0: ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: rq_lock kernel/sched/sched.h:1681 [inline]
 #0: ffff00006a8fe758 (&rq->__lock){-.-.}-{2:2}, at: __schedule+0x268/0x2ae4 kernel/sched/core.c:6612

stack backtrace:
CPU: 1 PID: 2923 Comm: udevd Not tainted 6.6.0-rc6-syzkaller-00285-g9c5d00cb7b6b #0
Hardware name: linux,dummy-virt (DT)
Call trace:
 dump_backtrace+0x9c/0x11c arch/arm64/kernel/stacktrace.c:233
 show_stack+0x18/0x24 arch/arm64/kernel/stacktrace.c:240
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x74/0xd4 lib/dump_stack.c:106
 dump_stack+0x1c/0x28 lib/dump_stack.c:113
 print_circular_bug+0x420/0x6f8 kernel/locking/lockdep.c:2060
 check_noncircular+0x2dc/0x364 kernel/locking/lockdep.c:2187
 check_prev_add kernel/locking/lockdep.c:3134 [inline]
 check_prevs_add kernel/locking/lockdep.c:3253 [inline]
 validate_chain kernel/locking/lockdep.c:3868 [inline]
 __lock_acquire+0x2cac/0x6b70 kernel/locking/lockdep.c:5136
 lock_acquire kernel/locking/lockdep.c:5753 [inline]
 lock_acquire+0x480/0x7c8 kernel/locking/lockdep.c:5718
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0x58/0x80 kernel/locking/spinlock.c:162
 down_trylock+0x18/0x80 kernel/locking/semaphore.c:139
 __down_trylock_console_sem+0x38/0xd8 kernel/printk/printk.c:329
 console_trylock kernel/printk/printk.c:2671 [inline]
 console_trylock_spinning kernel/printk/printk.c:1927 [inline]
 vprintk_emit+0x334/0x4e4 kernel/printk/printk.c:2306
 vprintk_default+0x38/0x44 kernel/printk/printk.c:2322
 vprintk+0x17c/0x1bc kernel/printk/printk_safe.c:45
 _printk+0xa8/0xe0 kernel/printk/printk.c:2332
 pick_eevdf kernel/sched/fair.c:976 [inline]
 pick_next_entity kernel/sched/fair.c:5278 [inline]
 pick_next_task_fair+0x1a4/0xd3c kernel/sched/fair.c:8222
 __pick_next_task kernel/sched/core.c:6004 [inline]
 pick_next_task kernel/sched/core.c:6514 [inline]
 __schedule+0x3b8/0x2ae4 kernel/sched/core.c:6659
 preempt_schedule_common kernel/sched/core.c:6864 [inline]
 preempt_schedule+0xf4/0x254 kernel/sched/core.c:6888
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:152 [inline]
 _raw_spin_unlock_irqrestore+0x94/0xac kernel/locking/spinlock.c:194
 debug_object_activate+0x230/0x3f0 lib/debugobjects.c:745
 debug_rcu_head_queue kernel/rcu/rcu.h:226 [inline]
 __call_rcu_common.constprop.0+0x38/0x8c0 kernel/rcu/tree.c:2638
 call_rcu+0x10/0x1c kernel/rcu/tree.c:2767
 file_free fs/file_table.c:82 [inline]
 __fput+0x3f8/0x830 fs/file_table.c:397
 __fput_sync+0x74/0x84 fs/file_table.c:465
 __do_sys_close fs/open.c:1572 [inline]
 __se_sys_close fs/open.c:1557 [inline]
 __arm64_sys_close+0x6c/0xbc fs/open.c:1557
 __invoke_syscall arch/arm64/kernel/syscall.c:37 [inline]
 invoke_syscall+0x6c/0x258 arch/arm64/kernel/syscall.c:51
 el0_svc_common.constprop.0+0xac/0x230 arch/arm64/kernel/syscall.c:136
 do_el0_svc+0x40/0x58 arch/arm64/kernel/syscall.c:155
 el0_svc+0x58/0x140 arch/arm64/kernel/entry-common.c:678
 el0t_64_sync_handler+0x100/0x12c arch/arm64/kernel/entry-common.c:696
 el0t_64_sync+0x190/0x194 arch/arm64/kernel/entry.S:595

Crashes (13):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2023/10/21 13:47 upstream 9c5d00cb7b6b 361b23dc .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu2-arm64 possible deadlock in __schedule
2023/10/18 16:21 upstream 06dc10eae55b 342b9c55 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu2-arm64 possible deadlock in __schedule
2023/10/18 15:44 upstream 06dc10eae55b 342b9c55 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu2-arm64-compat possible deadlock in __schedule
2023/10/16 11:23 upstream 58720809f527 f757a323 .config console log report info [disk image (non-bootable)] [vmlinux] [kernel image] ci-qemu2-arm64 possible deadlock in __schedule
2023/10/31 23:40 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing c70793fb7632 58499c95 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb possible deadlock in __schedule
2023/10/31 14:23 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing c70793fb7632 58499c95 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-usb possible deadlock in __schedule
2023/10/23 21:08 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 989a3687 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
2023/10/21 20:05 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 361b23dc .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
2023/10/21 03:16 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 361b23dc .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
2023/10/20 01:51 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 42e1d524 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
2023/10/19 14:42 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 42e1d524 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
2023/10/19 06:20 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 342b9c55 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
2023/10/19 02:37 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci 78124b0c1d10 342b9c55 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 possible deadlock in __schedule
* Struck through repros no longer work on HEAD.