syzbot


possible deadlock in blkdev_put (2)

Status: fixed on 2022/03/08 16:11
Reported-by: syzbot+643e4ce4b6ad1347d372@syzkaller.appspotmail.com
Fix commit: 322c4293ecc5 loop: make autoclear operation asynchronous
First crash: 257d, last: 152d

Cause bisection: introduced by (bisect log) :
commit f9006acc8dfe59e25aa75729728ac57a8d84fc32
Author: Florian Westphal <fw@strlen.de>
Date: Wed Apr 21 07:51:08 2021 +0000

  netfilter: arp_tables: pass table pointer via nf_hook_ops

Crash: WARNING in __nf_unregister_net_hook (log)
Repro: C syz .config
duplicates (1):
Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
possible deadlock in blkdev_get_by_dev 1 223d 219d 0/23 closed as dup on 2021/12/31 02:53
similar bugs (3):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in blkdev_put 1 686d 685d 0/23 auto-closed as invalid on 2021/01/18 22:03
upstream possible deadlock in blkdev_put (3) C 870 75d 148d 21/23 upstream: reported C repro on 2022/03/12 16:25
linux-4.14 possible deadlock in blkdev_put C 5 17d 675d 0/1 upstream: reported C repro on 2020/10/01 09:27
Patch testing requests:
Created Duration User Patch Repo Result
2022/01/28 07:21 18m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.3 OK
2022/01/26 15:28 18m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.2 OK
2022/01/26 08:29 17m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.2 OK
2022/01/26 08:02 10m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.2 report log
2022/01/25 20:21 18m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.2 report log
2022/01/25 15:36 0m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.2 error
2022/01/25 08:17 7m hch@lst.de git://git.infradead.org/users/hch/block.git loop-fix.2 error
2022/01/20 07:39 18m hch@lst.de git://git.infradead.org/users/hch/block.git part_tbl_mutex OK
2022/01/19 08:49 10m hch@lst.de git://git.infradead.org/users/hch/block.git a0b72af75eb90399b0527226eeb43714313505ce report log
2022/01/19 08:12 25m hch@lst.de git://git.infradead.org/users/hch/block.git part_tbl_mutex error
2021/12/24 15:47 17m penguin-kernel@i-love.sakura.ne.jp patch git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git next-20211224 OK
2021/12/08 10:10 19m penguin-kernel@i-love.sakura.ne.jp patch git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git next-20211207 OK

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.16.0-rc4-next-20211207-syzkaller #0 Not tainted
------------------------------------------------------
systemd-udevd/7081 is trying to acquire lock:
ffff88802358c938 ((wq_completion)loop3){+.+.}-{0:0}, at: flush_workqueue+0xe1/0x15b0 kernel/workqueue.c:2835

but task is already holding lock:
ffff88801a57a118 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_put+0x99/0x980 block/bdev.c:907

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #6 (&disk->open_mutex){+.+.}-{3:3}:
       __mutex_lock_common kernel/locking/mutex.c:607 [inline]
       __mutex_lock+0x12f/0x12f0 kernel/locking/mutex.c:740
       blkdev_get_by_dev.part.0+0x40e/0xc70 block/bdev.c:809
       blkdev_get_by_dev+0x6b/0x80 block/bdev.c:852
       swsusp_check+0x97/0x2f0 kernel/power/swap.c:1520
       software_resume.part.0+0x102/0x1f0 kernel/power/hibernate.c:979
       software_resume kernel/power/hibernate.c:86 [inline]
       resume_store+0x161/0x190 kernel/power/hibernate.c:1181
       kobj_attr_store+0x50/0x80 lib/kobject.c:856
       sysfs_kf_write+0x110/0x160 fs/sysfs/file.c:136
       kernfs_fop_write_iter+0x342/0x500 fs/kernfs/file.c:296
       call_write_iter include/linux/fs.h:2079 [inline]
       new_sync_write+0x429/0x660 fs/read_write.c:503
       vfs_write+0x7cd/0xae0 fs/read_write.c:590
       ksys_write+0x12d/0x250 fs/read_write.c:643
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #5 (system_transition_mutex/1){+.+.}-{3:3}:
       __mutex_lock_common kernel/locking/mutex.c:607 [inline]
       __mutex_lock+0x12f/0x12f0 kernel/locking/mutex.c:740
       software_resume.part.0+0x19/0x1f0 kernel/power/hibernate.c:934
       software_resume kernel/power/hibernate.c:86 [inline]
       resume_store+0x161/0x190 kernel/power/hibernate.c:1181
       kobj_attr_store+0x50/0x80 lib/kobject.c:856
       sysfs_kf_write+0x110/0x160 fs/sysfs/file.c:136
       kernfs_fop_write_iter+0x342/0x500 fs/kernfs/file.c:296
       call_write_iter include/linux/fs.h:2079 [inline]
       new_sync_write+0x429/0x660 fs/read_write.c:503
       vfs_write+0x7cd/0xae0 fs/read_write.c:590
       ksys_write+0x12d/0x250 fs/read_write.c:643
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #4 (&of->mutex){+.+.}-{3:3}:
       __mutex_lock_common kernel/locking/mutex.c:607 [inline]
       __mutex_lock+0x12f/0x12f0 kernel/locking/mutex.c:740
       kernfs_seq_start+0x4b/0x260 fs/kernfs/file.c:112
       seq_read_iter+0x2c7/0x1240 fs/seq_file.c:225
       kernfs_fop_read_iter+0x44f/0x5f0 fs/kernfs/file.c:241
       call_read_iter include/linux/fs.h:2073 [inline]
       new_sync_read+0x421/0x6e0 fs/read_write.c:400
       vfs_read+0x35c/0x600 fs/read_write.c:481
       ksys_read+0x12d/0x250 fs/read_write.c:619
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #3 (&p->lock){+.+.}-{3:3}:
       __mutex_lock_common kernel/locking/mutex.c:607 [inline]
       __mutex_lock+0x12f/0x12f0 kernel/locking/mutex.c:740
       seq_read_iter+0xdf/0x1240 fs/seq_file.c:182
       call_read_iter include/linux/fs.h:2073 [inline]
       generic_file_splice_read+0x453/0x6d0 fs/splice.c:311
       do_splice_to+0x1bf/0x250 fs/splice.c:796
       splice_direct_to_actor+0x2c2/0x8c0 fs/splice.c:870
       do_splice_direct+0x1b3/0x280 fs/splice.c:979
       do_sendfile+0xaf2/0x1250 fs/read_write.c:1245
       __do_sys_sendfile64 fs/read_write.c:1310 [inline]
       __se_sys_sendfile64 fs/read_write.c:1296 [inline]
       __x64_sys_sendfile64+0x1cc/0x210 fs/read_write.c:1296
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x44/0xae

-> #2 (sb_writers#3){.+.+}-{0:0}:
       percpu_down_read include/linux/percpu-rwsem.h:51 [inline]
       __sb_start_write include/linux/fs.h:1727 [inline]
       sb_start_write include/linux/fs.h:1797 [inline]
       file_start_write include/linux/fs.h:2942 [inline]
       lo_write_bvec drivers/block/loop.c:242 [inline]
       lo_write_simple drivers/block/loop.c:265 [inline]
       do_req_filebacked drivers/block/loop.c:494 [inline]
       loop_handle_cmd drivers/block/loop.c:1846 [inline]
       loop_process_work+0x1499/0x1db0 drivers/block/loop.c:1886
       process_one_work+0x9b2/0x1690 kernel/workqueue.c:2318
       worker_thread+0x658/0x11f0 kernel/workqueue.c:2465
       kthread+0x405/0x4f0 kernel/kthread.c:345
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #1 ((work_completion)(&lo->rootcg_work)){+.+.}-{0:0}:
       process_one_work+0x921/0x1690 kernel/workqueue.c:2294
       worker_thread+0x658/0x11f0 kernel/workqueue.c:2465
       kthread+0x405/0x4f0 kernel/kthread.c:345
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295

-> #0 ((wq_completion)loop3){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3063 [inline]
       check_prevs_add kernel/locking/lockdep.c:3186 [inline]
       validate_chain kernel/locking/lockdep.c:3801 [inline]
       __lock_acquire+0x2a07/0x54a0 kernel/locking/lockdep.c:5027
       lock_acquire kernel/locking/lockdep.c:5637 [inline]
       lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5602
       flush_workqueue+0x110/0x15b0 kernel/workqueue.c:2838
       drain_workqueue+0x1a5/0x3c0 kernel/workqueue.c:3003
       destroy_workqueue+0x71/0x800 kernel/workqueue.c:4440
       __loop_clr_fd+0x1ab/0xe20 drivers/block/loop.c:1118
       lo_release+0x1ac/0x1f0 drivers/block/loop.c:1750
       blkdev_put_whole block/bdev.c:694 [inline]
       blkdev_put+0x2de/0x980 block/bdev.c:949
       blkdev_close+0x6a/0x80 block/fops.c:516
       __fput+0x286/0x9f0 fs/file_table.c:311
       task_work_run+0xdd/0x1a0 kernel/task_work.c:164
       tracehook_notify_resume include/linux/tracehook.h:189 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:175 [inline]
       exit_to_user_mode_prepare+0x27e/0x290 kernel/entry/common.c:207
       __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline]
       syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300
       do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
       entry_SYSCALL_64_after_hwframe+0x44/0xae

other info that might help us debug this:

Chain exists of:
  (wq_completion)loop3 --> system_transition_mutex/1 --> &disk->open_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&disk->open_mutex);
                               lock(system_transition_mutex/1);
                               lock(&disk->open_mutex);
  lock((wq_completion)loop3);

 *** DEADLOCK ***

1 lock held by systemd-udevd/7081:
 #0: ffff88801a57a118 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_put+0x99/0x980 block/bdev.c:907

stack backtrace:
CPU: 1 PID: 7081 Comm: systemd-udevd Not tainted 5.16.0-rc4-next-20211207-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2143
 check_prev_add kernel/locking/lockdep.c:3063 [inline]
 check_prevs_add kernel/locking/lockdep.c:3186 [inline]
 validate_chain kernel/locking/lockdep.c:3801 [inline]
 __lock_acquire+0x2a07/0x54a0 kernel/locking/lockdep.c:5027
 lock_acquire kernel/locking/lockdep.c:5637 [inline]
 lock_acquire+0x1ab/0x510 kernel/locking/lockdep.c:5602
 flush_workqueue+0x110/0x15b0 kernel/workqueue.c:2838
 drain_workqueue+0x1a5/0x3c0 kernel/workqueue.c:3003
 destroy_workqueue+0x71/0x800 kernel/workqueue.c:4440
 __loop_clr_fd+0x1ab/0xe20 drivers/block/loop.c:1118
 lo_release+0x1ac/0x1f0 drivers/block/loop.c:1750
 blkdev_put_whole block/bdev.c:694 [inline]
 blkdev_put+0x2de/0x980 block/bdev.c:949
 blkdev_close+0x6a/0x80 block/fops.c:516
 __fput+0x286/0x9f0 fs/file_table.c:311
 task_work_run+0xdd/0x1a0 kernel/task_work.c:164
 tracehook_notify_resume include/linux/tracehook.h:189 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:175 [inline]
 exit_to_user_mode_prepare+0x27e/0x290 kernel/entry/common.c:207
 __syscall_exit_to_user_mode_work kernel/entry/common.c:289 [inline]
 syscall_exit_to_user_mode+0x19/0x60 kernel/entry/common.c:300
 do_syscall_64+0x42/0xb0 arch/x86/entry/common.c:86
 entry_SYSCALL_64_after_hwframe+0x44/0xae
RIP: 0033:0x7f5387c2b270
Code: 73 01 c3 48 8b 0d 38 7d 20 00 f7 d8 64 89 01 48 83 c8 ff c3 66 0f 1f 44 00 00 83 3d 59 c1 20 00 00 75 10 b8 03 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 ee fb ff ff 48 89 04 24
RSP: 002b:00007ffce06abcb8 EFLAGS: 00000246 ORIG_RAX: 0000000000000003
RAX: 0000000000000000 RBX: 0000000000000007 RCX: 00007f5387c2b270
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000007
RBP: 00007f5388ae5710 R08: 00005621f86a7af0 R09: 00005621f86b5700
R10: 00007f5388ae58c0 R11: 0000000000000246 R12: 0000000000000000
R13: 00005621f86a3e40 R14: 0000000000000003 R15: 000000000000000e
 </TASK>
I/O error, dev loop2, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
I/O error, dev loop2, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
I/O error, dev loop2, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Buffer I/O error on dev loop2, logical block 0, async page read
I/O error, dev loop3, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Buffer I/O error on dev loop3, logical block 1, async page read
I/O error, dev loop3, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Buffer I/O error on dev loop3, logical block 1, async page read

Crashes (502):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci-upstream-linux-next-kasan-gce-root 2021/12/08 05:04 linux-next 04fe99a8d936 0230ba3e .config log report syz C possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/08 15:03 upstream ea4424be1688 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/08 12:31 upstream ea4424be1688 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/08 04:15 upstream ea4424be1688 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/07 22:10 upstream ea4424be1688 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/07 07:36 upstream ffb217a13a2e 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/03/06 23:37 upstream ffb217a13a2e 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/06 11:54 upstream dcde98da9970 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/03/06 10:26 upstream dcde98da9970 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/03/05 20:09 upstream ac84e82f78cb 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-smack-root 2022/03/05 18:42 upstream ac84e82f78cb 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/05 10:31 upstream ac84e82f78cb 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/05 00:14 upstream 38f80f42147f 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/04 12:45 upstream b08968f196d4 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/04 03:32 upstream b08968f196d4 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/04 02:11 upstream b08968f196d4 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/03 13:48 upstream 5859a2b19911 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/02 11:30 upstream fb184c4af9b9 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/02 10:00 upstream fb184c4af9b9 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/02 08:00 upstream 575115360652 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/01 22:51 upstream 575115360652 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-smack-root 2022/03/01 22:09 upstream 575115360652 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/03/01 04:44 upstream 719fce7539cd 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-root 2022/03/01 03:21 upstream 719fce7539cd 45a13a73 .config log report info possible deadlock in blkdev_put
ci-qemu-upstream 2022/03/01 01:08 upstream 719fce7539cd 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-root 2022/02/28 15:41 upstream 7e57714cd0ad 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/02/28 13:48 upstream 52a025546731 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/02/28 03:56 upstream 52a025546731 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/02/28 01:57 upstream 52a025546731 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/02/27 23:48 upstream 2293be58d6a1 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/02/27 15:23 upstream 2293be58d6a1 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce 2022/02/27 02:04 upstream 2293be58d6a1 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/02/26 23:04 upstream 2293be58d6a1 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-selinux-root 2022/02/26 09:59 upstream 9137eda53752 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-root 2022/02/26 08:00 upstream 9137eda53752 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-root 2022/02/25 19:50 upstream 53ab78cd6d5a 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-root 2022/02/25 18:38 upstream 53ab78cd6d5a 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-386 2022/03/06 14:49 upstream dcde98da9970 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-386 2022/03/06 00:32 upstream 0014404f9c18 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-qemu-upstream-386 2022/03/05 15:47 upstream ac84e82f78cb 45a13a73 .config log report info possible deadlock in blkdev_put
ci-qemu-upstream-386 2022/03/05 02:11 upstream 07ebd38a0da2 45a13a73 .config log report info possible deadlock in blkdev_put
ci-qemu-upstream-386 2022/03/04 22:15 upstream 07ebd38a0da2 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-386 2022/03/03 15:14 upstream 5859a2b19911 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-386 2022/03/02 14:09 upstream fb184c4af9b9 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-386 2022/03/02 01:11 upstream 575115360652 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-kasan-gce-386 2022/02/28 17:44 upstream 7e57714cd0ad 45a13a73 .config log report info possible deadlock in blkdev_put
ci-qemu-upstream-386 2022/02/28 12:00 upstream 7e57714cd0ad 45a13a73 .config log report info possible deadlock in blkdev_put
ci-qemu-upstream-386 2022/02/25 21:35 upstream 53ab78cd6d5a 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-linux-next-kasan-gce-root 2022/03/08 02:40 linux-next 91265a6da44d 7bdd8b2c .config log report info possible deadlock in blkdev_put
ci-upstream-linux-next-kasan-gce-root 2022/03/03 11:55 linux-next 91265a6da44d 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-linux-next-kasan-gce-root 2022/03/01 12:35 linux-next e6ada6df471f 45a13a73 .config log report info possible deadlock in blkdev_put
ci-upstream-linux-next-kasan-gce-root 2021/11/27 06:06 linux-next f81e94e91878 63eeac02 .config log report info possible deadlock in blkdev_put
ci-upstream-linux-next-kasan-gce-root 2021/11/23 07:01 linux-next aacdecce8147 545ab074 .config log report info possible deadlock in blkdev_put