syzbot


possible deadlock in __mod_timer (2)

Status: upstream: reported on 2024/08/11 05:04
Reported-by: syzbot+1e175fa5b198b9e3206f@syzkaller.appspotmail.com
First crash: 191d, last: 31d
Similar bugs (4)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in __mod_timer (4) bpf trace C 4 55d 149d 0/28 upstream: reported C repro on 2024/09/22 06:39
upstream possible deadlock in __mod_timer kernel 35 801d 875d 0/28 auto-obsoleted due to no activity on 2023/04/09 11:04
linux-5.15 possible deadlock in __mod_timer 4 297d 316d 0/3 auto-obsoleted due to no activity on 2024/08/05 08:07
upstream possible deadlock in __mod_timer (2) kernel 1 650d 650d 22/28 fixed on 2023/07/01 16:05

Sample crash report:
------------[ cut here ]------------
======================================================
WARNING: possible circular locking dependency detected
5.15.176-syzkaller #0 Not tainted
------------------------------------------------------
ksoftirqd/1/20 is trying to acquire lock:
ffffffff8c9fc538 ((console_sem).lock){..-.}-{2:2}, at: down_trylock+0x1c/0xa0 kernel/locking/semaphore.c:138

but task is already holding lock:
ffff8880b8f28098 (&base->lock){-.-.}-{2:2}, at: __mod_timer+0x772/0xeb0 kernel/time/timer.c:1043

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (&base->lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       lock_timer_base+0x120/0x260 kernel/time/timer.c:946
       __mod_timer+0x1d6/0xeb0 kernel/time/timer.c:1019
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       psi_enqueue kernel/sched/stats.h:104 [inline]
       enqueue_task+0x2fe/0x3a0 kernel/sched/core.c:1970
       activate_task kernel/sched/core.c:2002 [inline]
       wake_up_new_task+0x515/0xb60 kernel/sched/core.c:4519
       kernel_clone+0x44e/0x960 kernel/fork.c:2627
       kernel_thread+0x168/0x1e0 kernel/fork.c:2655
       rest_init+0x21/0x330 init/main.c:706
       start_kernel+0x48c/0x540 init/main.c:1140
       secondary_startup_64_no_verify+0xb1/0xbb

-> #2 (&rq->__lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       _raw_spin_lock_nested+0x2d/0x40 kernel/locking/spinlock.c:368
       raw_spin_rq_lock_nested+0x26/0x140 kernel/sched/core.c:475
       raw_spin_rq_lock kernel/sched/sched.h:1326 [inline]
       rq_lock kernel/sched/sched.h:1621 [inline]
       task_fork_fair+0x5d/0x350 kernel/sched/fair.c:11488
       sched_cgroup_fork+0x2d3/0x330 kernel/sched/core.c:4463
       copy_process+0x224a/0x3ef0 kernel/fork.c:2319
       kernel_clone+0x210/0x960 kernel/fork.c:2603
       kernel_thread+0x168/0x1e0 kernel/fork.c:2655
       rest_init+0x21/0x330 init/main.c:706
       start_kernel+0x48c/0x540 init/main.c:1140
       secondary_startup_64_no_verify+0xb1/0xbb

-> #1 (&p->pi_lock){-.-.}-{2:2}:
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       try_to_wake_up+0xae/0x1300 kernel/sched/core.c:4027
       up+0x6e/0x90 kernel/locking/semaphore.c:190
       __up_console_sem+0x11a/0x1e0 kernel/printk/printk.c:256
       console_unlock+0x1145/0x12b0 kernel/printk/printk.c:2760
       vprintk_emit+0xbf/0x150 kernel/printk/printk.c:2274
       dev_vprintk_emit+0x2aa/0x330 drivers/base/core.c:4638
       dev_printk_emit+0xd9/0x120 drivers/base/core.c:4649
       _dev_warn+0x11e/0x170 drivers/base/core.c:4705
       firmware_fallback_sysfs+0x681/0xc90 drivers/base/firmware_loader/fallback.c:654
       _request_firmware+0xc72/0x12c0 drivers/base/firmware_loader/main.c:876
       request_firmware_work_func+0x126/0x270 drivers/base/firmware_loader/main.c:1127
       process_one_work+0x8a1/0x10c0 kernel/workqueue.c:2310
       worker_thread+0xaca/0x1280 kernel/workqueue.c:2457
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

-> #0 ((console_sem).lock){..-.}-{2:2}:
       check_prev_add kernel/locking/lockdep.c:3053 [inline]
       check_prevs_add kernel/locking/lockdep.c:3172 [inline]
       validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
       __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
       lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
       down_trylock+0x1c/0xa0 kernel/locking/semaphore.c:138
       __down_trylock_console_sem+0x105/0x250 kernel/printk/printk.c:239
       console_trylock kernel/printk/printk.c:2575 [inline]
       console_trylock_spinning+0x8a/0x3f0 kernel/printk/printk.c:1867
       vprintk_emit+0xa6/0x150 kernel/printk/printk.c:2273
       _printk+0xd1/0x120 kernel/printk/printk.c:2299
       report_bug+0x1e5/0x2e0 lib/bug.c:195
       handle_bug+0x3d/0x70 arch/x86/kernel/traps.c:239
       exc_invalid_op+0x16/0x40 arch/x86/kernel/traps.c:259
       asm_exc_invalid_op+0x16/0x20 arch/x86/include/asm/idtentry.h:604
       copy_from_user_nofault+0x15c/0x1c0
       bpf_probe_read_user_common kernel/trace/bpf_trace.c:157 [inline]
       ____bpf_probe_read_user kernel/trace/bpf_trace.c:166 [inline]
       bpf_probe_read_user+0x26/0x70 kernel/trace/bpf_trace.c:163
       bpf_prog_690f12e098ef46de+0x32/0x7d0
       bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
       __bpf_prog_run include/linux/filter.h:628 [inline]
       bpf_prog_run include/linux/filter.h:635 [inline]
       __bpf_trace_run kernel/trace/bpf_trace.c:1878 [inline]
       bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1916
       trace_timer_start include/trace/events/timer.h:52 [inline]
       enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
       internal_add_timer kernel/time/timer.c:611 [inline]
       __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
       queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
       queue_delayed_work include/linux/workqueue.h:527 [inline]
       wb_inode_writeback_end mm/page-writeback.c:2791 [inline]
       test_clear_page_writeback+0x8e7/0xd30 mm/page-writeback.c:2818
       end_page_writeback+0x2a6/0x690 mm/filemap.c:1603
       end_bio_bh_io_sync+0xb3/0x110 fs/buffer.c:3007
       req_bio_endio block/blk-core.c:261 [inline]
       blk_update_request+0x87c/0x1470 block/blk-core.c:1441
       blk_mq_end_request+0x3a/0x70 block/blk-mq.c:575
       blk_complete_reqs block/blk-mq.c:587 [inline]
       blk_done_softirq+0xf2/0x130 block/blk-mq.c:592
       handle_softirqs+0x3a7/0x930 kernel/softirq.c:558
       run_ksoftirqd+0xc6/0x120 kernel/softirq.c:925
       smpboot_thread_fn+0x51b/0x9d0 kernel/smpboot.c:164
       kthread+0x3f6/0x4f0 kernel/kthread.c:334
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287

other info that might help us debug this:

Chain exists of:
  (console_sem).lock --> &rq->__lock --> &base->lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&base->lock);
                               lock(&rq->__lock);
                               lock(&base->lock);
  lock((console_sem).lock);

 *** DEADLOCK ***

5 locks held by ksoftirqd/1/20:
 #0: ffffffff8cb1fce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x0/0x30
 #1: ffff88801790b7d0 (&xa->xa_lock#5){..-.}-{2:2}, at: test_clear_page_writeback+0x30d/0xd30 mm/page-writeback.c:2806
 #2: ffff8881473fc388 (&wb->work_lock){..-.}-{2:2}, at: wb_inode_writeback_end mm/page-writeback.c:2789 [inline]
 #2: ffff8881473fc388 (&wb->work_lock){..-.}-{2:2}, at: test_clear_page_writeback+0x82f/0xd30 mm/page-writeback.c:2818
 #3: ffff8880b8f28098 (&base->lock){-.-.}-{2:2}, at: __mod_timer+0x772/0xeb0 kernel/time/timer.c:1043
 #4: ffffffff8cb1fce0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire+0x5/0x30 include/linux/rcupdate.h:311

stack backtrace:
CPU: 1 PID: 20 Comm: ksoftirqd/1 Not tainted 5.15.176-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2d0 lib/dump_stack.c:106
 check_noncircular+0x2f8/0x3b0 kernel/locking/lockdep.c:2133
 check_prev_add kernel/locking/lockdep.c:3053 [inline]
 check_prevs_add kernel/locking/lockdep.c:3172 [inline]
 validate_chain+0x1649/0x5930 kernel/locking/lockdep.c:3788
 __lock_acquire+0x1295/0x1ff0 kernel/locking/lockdep.c:5012
 lock_acquire+0x1db/0x4f0 kernel/locking/lockdep.c:5623
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0xd1/0x120 kernel/locking/spinlock.c:162
 down_trylock+0x1c/0xa0 kernel/locking/semaphore.c:138
 __down_trylock_console_sem+0x105/0x250 kernel/printk/printk.c:239
 console_trylock kernel/printk/printk.c:2575 [inline]
 console_trylock_spinning+0x8a/0x3f0 kernel/printk/printk.c:1867
 vprintk_emit+0xa6/0x150 kernel/printk/printk.c:2273
 _printk+0xd1/0x120 kernel/printk/printk.c:2299
 report_bug+0x1e5/0x2e0 lib/bug.c:195
 handle_bug+0x3d/0x70 arch/x86/kernel/traps.c:239
 exc_invalid_op+0x16/0x40 arch/x86/kernel/traps.c:259
 asm_exc_invalid_op+0x16/0x20 arch/x86/include/asm/idtentry.h:604
RIP: 0010:copy_from_user_nofault+0x15c/0x1c0 mm/maccess.c:226
Code: db 48 c7 c0 f2 ff ff ff 48 0f 44 c5 eb 0c e8 cb ba d5 ff 48 c7 c0 f2 ff ff ff 5b 41 5c 41 5d 41 5e 41 5f 5d c3 e8 b4 ba d5 ff <0f> 0b e9 1e ff ff ff 89 d9 80 e1 07 80 c1 03 38 c1 0f 8c ef fe ff
RSP: 0018:ffffc90000da7668 EFLAGS: 00010046
RAX: ffffffff81aaca2c RBX: 0000000000000000 RCX: ffff88813fe28000
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff81aac93d R09: fffffbfff2131421
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: ffffc90000da76c8
 bpf_probe_read_user_common kernel/trace/bpf_trace.c:157 [inline]
 ____bpf_probe_read_user kernel/trace/bpf_trace.c:166 [inline]
 bpf_probe_read_user+0x26/0x70 kernel/trace/bpf_trace.c:163
 bpf_prog_690f12e098ef46de+0x32/0x7d0
 bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1878 [inline]
 bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1916
 trace_timer_start include/trace/events/timer.h:52 [inline]
 enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
 internal_add_timer kernel/time/timer.c:611 [inline]
 __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
 queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
 queue_delayed_work include/linux/workqueue.h:527 [inline]
 wb_inode_writeback_end mm/page-writeback.c:2791 [inline]
 test_clear_page_writeback+0x8e7/0xd30 mm/page-writeback.c:2818
 end_page_writeback+0x2a6/0x690 mm/filemap.c:1603
 end_bio_bh_io_sync+0xb3/0x110 fs/buffer.c:3007
 req_bio_endio block/blk-core.c:261 [inline]
 blk_update_request+0x87c/0x1470 block/blk-core.c:1441
 blk_mq_end_request+0x3a/0x70 block/blk-mq.c:575
 blk_complete_reqs block/blk-mq.c:587 [inline]
 blk_done_softirq+0xf2/0x130 block/blk-mq.c:592
 handle_softirqs+0x3a7/0x930 kernel/softirq.c:558
 run_ksoftirqd+0xc6/0x120 kernel/softirq.c:925
 smpboot_thread_fn+0x51b/0x9d0 kernel/smpboot.c:164
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>
WARNING: CPU: 1 PID: 20 at mm/maccess.c:226 copy_from_user_nofault+0x15c/0x1c0
Modules linked in:
CPU: 1 PID: 20 Comm: ksoftirqd/1 Not tainted 5.15.176-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
RIP: 0010:copy_from_user_nofault+0x15c/0x1c0 mm/maccess.c:226
Code: db 48 c7 c0 f2 ff ff ff 48 0f 44 c5 eb 0c e8 cb ba d5 ff 48 c7 c0 f2 ff ff ff 5b 41 5c 41 5d 41 5e 41 5f 5d c3 e8 b4 ba d5 ff <0f> 0b e9 1e ff ff ff 89 d9 80 e1 07 80 c1 03 38 c1 0f 8c ef fe ff
RSP: 0018:ffffc90000da7668 EFLAGS: 00010046
RAX: ffffffff81aaca2c RBX: 0000000000000000 RCX: ffff88813fe28000
RDX: 0000000000000100 RSI: 0000000000000000 RDI: 0000000000000000
RBP: dffffc0000000000 R08: ffffffff81aac93d R09: fffffbfff2131421
R10: 0000000000000000 R11: dffffc0000000001 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: ffffc90000da76c8
FS:  0000000000000000(0000) GS:ffff8880b8f00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000110c44820c CR3: 0000000064d5e000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 bpf_probe_read_user_common kernel/trace/bpf_trace.c:157 [inline]
 ____bpf_probe_read_user kernel/trace/bpf_trace.c:166 [inline]
 bpf_probe_read_user+0x26/0x70 kernel/trace/bpf_trace.c:163
 bpf_prog_690f12e098ef46de+0x32/0x7d0
 bpf_dispatcher_nop_func include/linux/bpf.h:790 [inline]
 __bpf_prog_run include/linux/filter.h:628 [inline]
 bpf_prog_run include/linux/filter.h:635 [inline]
 __bpf_trace_run kernel/trace/bpf_trace.c:1878 [inline]
 bpf_trace_run3+0x1d1/0x380 kernel/trace/bpf_trace.c:1916
 trace_timer_start include/trace/events/timer.h:52 [inline]
 enqueue_timer+0x3ae/0x540 kernel/time/timer.c:586
 internal_add_timer kernel/time/timer.c:611 [inline]
 __mod_timer+0xa60/0xeb0 kernel/time/timer.c:1062
 queue_delayed_work_on+0x156/0x250 kernel/workqueue.c:1715
 queue_delayed_work include/linux/workqueue.h:527 [inline]
 wb_inode_writeback_end mm/page-writeback.c:2791 [inline]
 test_clear_page_writeback+0x8e7/0xd30 mm/page-writeback.c:2818
 end_page_writeback+0x2a6/0x690 mm/filemap.c:1603
 end_bio_bh_io_sync+0xb3/0x110 fs/buffer.c:3007
 req_bio_endio block/blk-core.c:261 [inline]
 blk_update_request+0x87c/0x1470 block/blk-core.c:1441
 blk_mq_end_request+0x3a/0x70 block/blk-mq.c:575
 blk_complete_reqs block/blk-mq.c:587 [inline]
 blk_done_softirq+0xf2/0x130 block/blk-mq.c:592
 handle_softirqs+0x3a7/0x930 kernel/softirq.c:558
 run_ksoftirqd+0xc6/0x120 kernel/softirq.c:925
 smpboot_thread_fn+0x51b/0x9d0 kernel/smpboot.c:164
 kthread+0x3f6/0x4f0 kernel/kthread.c:334
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:287
 </TASK>

Crashes (6):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2025/01/18 08:27 linux-5.15.y 4735586da88e f2cb035c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __mod_timer
2024/11/03 12:30 linux-5.15.y 72244eab0dad f00eed24 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __mod_timer
2024/10/26 10:38 linux-5.15.y 74cdd62cb470 65e8686b .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __mod_timer
2024/10/22 02:04 linux-5.15.y 584a40a22cb9 f1e4447c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __mod_timer
2024/10/06 08:10 linux-5.15.y 3a5928702e71 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan-perf possible deadlock in __mod_timer
2024/08/11 05:03 linux-5.15.y 7e89efd3ae1c 6f4edef4 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-5-15-kasan possible deadlock in __mod_timer
* Struck through repros no longer work on HEAD.