syzbot


possible deadlock in dpm_for_each_dev

Status: upstream: reported C repro on 2024/07/07 17:25
Subsystems: pm usb
[Documentation on labels]
Reported-by: syzbot+2a03726f1d4eff48b278@syzkaller.appspotmail.com
First crash: 140d, last: 30d
Cause bisection: failed (error log, bisect log)
  
Discussions (4)
Title Replies (including bot) Last reply
[syzbot] Monthly pm report (Oct 2024) 0 (1) 2024/10/28 08:24
[syzbot] Monthly pm report (Sep 2024) 0 (1) 2024/09/26 12:43
[syzbot] Monthly pm report (Aug 2024) 0 (1) 2024/08/26 13:47
[syzbot] [pm?] [usb?] possible deadlock in dpm_for_each_dev 0 (1) 2024/07/07 17:25
Last patch testing requests (3)
Created Duration User Patch Repo Result
2024/10/04 22:24 23m retest repro upstream report log
2024/08/02 10:52 16m retest repro upstream report log
2024/07/17 18:13 19m retest repro upstream report log
Fix bisection attempts (3)
Created Duration User Patch Repo Result
2024/10/21 13:43 3h25m bisect fix upstream OK (0) job log log
2024/09/20 19:11 1h51m bisect fix upstream OK (0) job log log
2024/08/19 04:48 3h12m bisect fix upstream OK (0) job log log

Sample crash report:
RBP: 00007fa6014b83d0 R08: 00007fa6013c3fa7 R09: 0000000000000038
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa6013c4210
R13: 0000000000000001 R14: 00746f687370616e R15: 616e732f7665642f
 </TASK>
(NULL device *): loading /lib/firmware/dvb-usb-dib0700-1.20.fw failed with error -12
(NULL device *): Direct firmware load for dvb-usb-dib0700-1.20.fw failed with error -12
(NULL device *): Falling back to sysfs fallback for: dvb-usb-dib0700-1.20.fw
======================================================
WARNING: possible circular locking dependency detected
6.10.0-syzkaller-08280-g68b59730459e #0 Not tainted
------------------------------------------------------
syz-executor800/5113 is trying to acquire lock:
ffffffff8e1df830 (umhelper_sem){++++}-{3:3}, at: usermodehelper_read_trylock+0x140/0x300 kernel/umh.c:215

but task is already holding lock:
ffffffff8ec7df88 (dpm_list_mtx){+.+.}-{3:3}, at: device_pm_lock drivers/base/power/main.c:113 [inline]
ffffffff8ec7df88 (dpm_list_mtx){+.+.}-{3:3}, at: dpm_for_each_dev+0x2b/0xc0 drivers/base/power/main.c:1961

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (dpm_list_mtx){+.+.}-{3:3}:
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5753
       __mutex_lock_common kernel/locking/mutex.c:608 [inline]
       __mutex_lock+0x136/0xd70 kernel/locking/mutex.c:752
       device_pm_add+0x78/0x320 drivers/base/power/main.c:137
       device_add+0x5e8/0xbf0 drivers/base/core.c:3642
       fw_load_sysfs_fallback drivers/base/firmware_loader/fallback.c:86 [inline]
       fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:162 [inline]
       firmware_fallback_sysfs+0x307/0x9e0 drivers/base/firmware_loader/fallback.c:238
       _request_firmware+0xcf5/0x12b0 drivers/base/firmware_loader/main.c:914
       request_firmware_work_func+0x12a/0x280 drivers/base/firmware_loader/main.c:1165
       process_one_work kernel/workqueue.c:3231 [inline]
       process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
       worker_thread+0x86d/0xd40 kernel/workqueue.c:3390
       kthread+0x2f0/0x390 kernel/kthread.c:389
       ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
       ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244

-> #0 (umhelper_sem){++++}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:3133 [inline]
       check_prevs_add kernel/locking/lockdep.c:3252 [inline]
       validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3868
       __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5136
       lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5753
       down_read+0xb1/0xa40 kernel/locking/rwsem.c:1526
       usermodehelper_read_trylock+0x140/0x300 kernel/umh.c:215
       fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:147 [inline]
       firmware_fallback_sysfs+0x184/0x9e0 drivers/base/firmware_loader/fallback.c:238
       _request_firmware+0xcf5/0x12b0 drivers/base/firmware_loader/main.c:914
       request_firmware drivers/base/firmware_loader/main.c:963 [inline]
       cache_firmware drivers/base/firmware_loader/main.c:1265 [inline]
       __async_dev_cache_fw_image+0xe7/0x320 drivers/base/firmware_loader/main.c:1379
       async_schedule_node_domain+0xdc/0x110 kernel/async.c:221
       async_schedule_domain include/linux/async.h:72 [inline]
       dev_cache_fw_image+0x36d/0x3e0 drivers/base/firmware_loader/main.c:1435
       dpm_for_each_dev+0x58/0xc0 drivers/base/power/main.c:1963
       device_cache_fw_images drivers/base/firmware_loader/main.c:1485 [inline]
       fw_pm_notify+0x24a/0x2f0 drivers/base/firmware_loader/main.c:1536
       notifier_call_chain+0x19f/0x3e0 kernel/notifier.c:93
       notifier_call_chain_robust kernel/notifier.c:128 [inline]
       blocking_notifier_call_chain_robust+0xe8/0x1e0 kernel/notifier.c:353
       pm_notifier_call_chain_robust+0x2c/0x60 kernel/power/main.c:102
       snapshot_open+0x1a1/0x280 kernel/power/user.c:77
       misc_open+0x313/0x390 drivers/char/misc.c:165
       chrdev_open+0x5b0/0x630 fs/char_dev.c:414
       do_dentry_open+0x970/0x1440 fs/open.c:959
       vfs_open+0x3e/0x330 fs/open.c:1089
       do_open fs/namei.c:3727 [inline]
       path_openat+0x2b3e/0x3470 fs/namei.c:3886
       do_filp_open+0x235/0x490 fs/namei.c:3913
       do_sys_openat2+0x13e/0x1d0 fs/open.c:1416
       do_sys_open fs/open.c:1431 [inline]
       __do_sys_openat fs/open.c:1447 [inline]
       __se_sys_openat fs/open.c:1442 [inline]
       __x64_sys_openat+0x247/0x2a0 fs/open.c:1442
       do_syscall_x64 arch/x86/entry/common.c:52 [inline]
       do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
       entry_SYSCALL_64_after_hwframe+0x77/0x7f

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(dpm_list_mtx);
                               lock(umhelper_sem);
                               lock(dpm_list_mtx);
  rlock(umhelper_sem);

 *** DEADLOCK ***

5 locks held by syz-executor800/5113:
 #0: ffffffff8eb397e8 (misc_mtx){+.+.}-{3:3}, at: misc_open+0x5c/0x390 drivers/char/misc.c:129
 #1: ffffffff8e1e9468 (system_transition_mutex){+.+.}-{3:3}, at: lock_system_sleep+0x60/0xa0 kernel/power/main.c:56
 #2: ffffffff8e209910 ((pm_chain_head).rwsem){++++}-{3:3}, at: blocking_notifier_call_chain_robust+0xac/0x1e0 kernel/notifier.c:352
 #3: ffffffff8ec82ee8 (fw_lock){+.+.}-{3:3}, at: device_cache_fw_images drivers/base/firmware_loader/main.c:1483 [inline]
 #3: ffffffff8ec82ee8 (fw_lock){+.+.}-{3:3}, at: fw_pm_notify+0x232/0x2f0 drivers/base/firmware_loader/main.c:1536
 #4: ffffffff8ec7df88 (dpm_list_mtx){+.+.}-{3:3}, at: device_pm_lock drivers/base/power/main.c:113 [inline]
 #4: ffffffff8ec7df88 (dpm_list_mtx){+.+.}-{3:3}, at: dpm_for_each_dev+0x2b/0xc0 drivers/base/power/main.c:1961

stack backtrace:
CPU: 0 PID: 5113 Comm: syz-executor800 Not tainted 6.10.0-syzkaller-08280-g68b59730459e #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2186
 check_prev_add kernel/locking/lockdep.c:3133 [inline]
 check_prevs_add kernel/locking/lockdep.c:3252 [inline]
 validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3868
 __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5136
 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5753
 down_read+0xb1/0xa40 kernel/locking/rwsem.c:1526
 usermodehelper_read_trylock+0x140/0x300 kernel/umh.c:215
 fw_load_from_user_helper drivers/base/firmware_loader/fallback.c:147 [inline]
 firmware_fallback_sysfs+0x184/0x9e0 drivers/base/firmware_loader/fallback.c:238
 _request_firmware+0xcf5/0x12b0 drivers/base/firmware_loader/main.c:914
 request_firmware drivers/base/firmware_loader/main.c:963 [inline]
 cache_firmware drivers/base/firmware_loader/main.c:1265 [inline]
 __async_dev_cache_fw_image+0xe7/0x320 drivers/base/firmware_loader/main.c:1379
 async_schedule_node_domain+0xdc/0x110 kernel/async.c:221
 async_schedule_domain include/linux/async.h:72 [inline]
 dev_cache_fw_image+0x36d/0x3e0 drivers/base/firmware_loader/main.c:1435
 dpm_for_each_dev+0x58/0xc0 drivers/base/power/main.c:1963
 device_cache_fw_images drivers/base/firmware_loader/main.c:1485 [inline]
 fw_pm_notify+0x24a/0x2f0 drivers/base/firmware_loader/main.c:1536
 notifier_call_chain+0x19f/0x3e0 kernel/notifier.c:93
 notifier_call_chain_robust kernel/notifier.c:128 [inline]
 blocking_notifier_call_chain_robust+0xe8/0x1e0 kernel/notifier.c:353
 pm_notifier_call_chain_robust+0x2c/0x60 kernel/power/main.c:102
 snapshot_open+0x1a1/0x280 kernel/power/user.c:77
 misc_open+0x313/0x390 drivers/char/misc.c:165
 chrdev_open+0x5b0/0x630 fs/char_dev.c:414
 do_dentry_open+0x970/0x1440 fs/open.c:959
 vfs_open+0x3e/0x330 fs/open.c:1089
 do_open fs/namei.c:3727 [inline]
 path_openat+0x2b3e/0x3470 fs/namei.c:3886
 do_filp_open+0x235/0x490 fs/namei.c:3913
 do_sys_openat2+0x13e/0x1d0 fs/open.c:1416
 do_sys_open fs/open.c:1431 [inline]
 __do_sys_openat fs/open.c:1447 [inline]
 __se_sys_openat fs/open.c:1442 [inline]
 __x64_sys_openat+0x247/0x2a0 fs/open.c:1442
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fa6014370c9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 a1 1d 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007fa6013c4208 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00007fa6014b83d8 RCX: 00007fa6014370c9
RDX: 0000000000000000 RSI: 0000000020000080 RDI: ffffffffffffff9c
RBP: 00007fa6014b83d0 R08: 00007fa6013c3fa7 R09: 0000000000000038
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fa6013c4210
R13: 0000000000000001 R14: 00746f687370616e R15: 616e732f7665642f
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/07/19 10:43 upstream 68b59730459e ee4e11c8 .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce possible deadlock in dpm_for_each_dev
2024/07/03 17:17 upstream e9d22f7a6655 409d975c .config console log report syz / log C [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce possible deadlock in dpm_for_each_dev
* Struck through repros no longer work on HEAD.