syzbot


possible deadlock in btrfs_scrub_dev

Status: upstream: reported C repro on 2022/12/17 16:53
Subsystems: btrfs
[Documentation on labels]
Reported-by: syzbot+b1a530302b8cbd3b85d7@syzkaller.appspotmail.com
First crash: 740d, last: 663d

Sample crash report:
BTRFS: device fsid 24c7a497-3402-47dd-bef8-82358f5f30e0 devid 1 transid 8 /dev/loop0
BTRFS info (device loop0): using free space tree
BTRFS info (device loop0): has skinny extents
======================================================
WARNING: possible circular locking dependency detected
4.14.302-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor100/7968 is trying to acquire lock:
 ("%s-%s""btrfs", name){+.+.}, at: [<ffffffff8135cb6b>] flush_workqueue+0xcb/0x1310 kernel/workqueue.c:2622

but task is already holding lock:
 (&fs_info->scrub_lock){+.+.}, at: [<ffffffff82b1ba96>] btrfs_scrub_dev+0x506/0xcd0 fs/btrfs/scrub.c:4217

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (&fs_info->scrub_lock){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       btrfs_scrub_dev+0x1f3/0xcd0 fs/btrfs/scrub.c:4150
       btrfs_ioctl_scrub fs/btrfs/ioctl.c:4451 [inline]
       btrfs_ioctl+0xba8/0x5b20 fs/btrfs/ioctl.c:5681
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:500 [inline]
       do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
       SYSC_ioctl fs/ioctl.c:701 [inline]
       SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x5e/0xd3

-> #2 (&fs_devs->device_list_mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       __reada_start_machine fs/btrfs/reada.c:765 [inline]
       reada_start_machine_worker+0x1d2/0xa90 fs/btrfs/reada.c:746
       normal_work_helper+0x304/0x1330 fs/btrfs/async-thread.c:376
       process_one_work+0x793/0x14a0 kernel/workqueue.c:2117
       worker_thread+0x5cc/0xff0 kernel/workqueue.c:2251
       kthread+0x30d/0x420 kernel/kthread.c:232
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:406

-> #1 ((&work->normal_work)){+.+.}:
       process_one_work+0x736/0x14a0 kernel/workqueue.c:2093
       worker_thread+0x5cc/0xff0 kernel/workqueue.c:2251
       kthread+0x30d/0x420 kernel/kthread.c:232
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:406

-> #0 ("%s-%s""btrfs", name){+.+.}:
       lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
       flush_workqueue+0xfa/0x1310 kernel/workqueue.c:2625
       drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2790
       destroy_workqueue+0x71/0x710 kernel/workqueue.c:4116
       __btrfs_destroy_workqueue fs/btrfs/async-thread.c:436 [inline]
       btrfs_destroy_workqueue+0xf8/0x630 fs/btrfs/async-thread.c:447
       scrub_workers_put+0x90/0x1a0 fs/btrfs/scrub.c:4075
       btrfs_scrub_dev+0x536/0xcd0 fs/btrfs/scrub.c:4219
       btrfs_ioctl_scrub fs/btrfs/ioctl.c:4451 [inline]
       btrfs_ioctl+0xba8/0x5b20 fs/btrfs/ioctl.c:5681
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:500 [inline]
       do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
       SYSC_ioctl fs/ioctl.c:701 [inline]
       SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x5e/0xd3

other info that might help us debug this:

Chain exists of:
  "%s-%s""btrfs", name --> &fs_devs->device_list_mutex --> &fs_info->scrub_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&fs_info->scrub_lock);
                               lock(&fs_devs->device_list_mutex);
                               lock(&fs_info->scrub_lock);
  lock("%s-%s""btrfs", name);

 *** DEADLOCK ***

1 lock held by syz-executor100/7968:
 #0:  (&fs_info->scrub_lock){+.+.}, at: [<ffffffff82b1ba96>] btrfs_scrub_dev+0x506/0xcd0 fs/btrfs/scrub.c:4217

stack backtrace:
CPU: 0 PID: 7968 Comm: syz-executor100 Not tainted 4.14.302-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x1b2/0x281 lib/dump_stack.c:58
 print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1905 [inline]
 check_prevs_add kernel/locking/lockdep.c:2022 [inline]
 validate_chain kernel/locking/lockdep.c:2464 [inline]
 __lock_acquire+0x2e0e/0x3f20 kernel/locking/lockdep.c:3491
 lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
 flush_workqueue+0xfa/0x1310 kernel/workqueue.c:2625
 drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2790
 destroy_workqueue+0x71/0x710 kernel/workqueue.c:4116
 __btrfs_destroy_workqueue fs/btrfs/async-thread.c:436 [inline]
 btrfs_destroy_workqueue+0xf8/0x630 fs/btrfs/async-thread.c:447
 scrub_workers_put+0x90/0x1a0 fs/btrfs/scrub.c:4075
 btrfs_scrub_dev+0x536/0xcd0 fs/btrfs/scrub.c:4219
 btrfs_ioctl_scrub fs/btrfs/ioctl.c:4451 [inline]
 btrfs_ioctl+0xba8/0x5b20 fs/btrfs/ioctl.c:5681
 vfs_ioctl fs/ioctl.c:46 [inline]
 file_ioctl fs/ioctl.c:500 [inline]
 do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
 SYSC_ioctl fs/ioctl.c:701 [inline]
 SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
 do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x5e/0xd3
RIP: 0033:0x7f3243f808c

Crashes (456):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/12/26 01:27 linux-4.14.y c4215ee4771b 9da18ae8 .config console log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2022/12/17 17:05 linux-4.14.y c4215ee4771b 05494336 .config console log report syz C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/04 05:52 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/04 03:06 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/03 11:30 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/02 03:36 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/02 02:16 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/02 00:51 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 22:42 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 21:02 linux-4.14.y 7878a41b6cc1 f8902b57 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 18:14 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 17:02 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 15:47 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 12:54 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 11:06 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 09:34 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 08:24 linux-4.14.y 7878a41b6cc1 ef65e6cb .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/03/01 04:24 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/28 14:46 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/28 04:43 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/28 02:17 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/28 00:43 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/28 00:10 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/27 22:14 linux-4.14.y 7878a41b6cc1 95aee97a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/27 15:32 linux-4.14.y 7878a41b6cc1 9189cb53 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/27 14:13 linux-4.14.y 7878a41b6cc1 9189cb53 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/27 12:37 linux-4.14.y 7878a41b6cc1 9189cb53 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/27 10:59 linux-4.14.y 7878a41b6cc1 9189cb53 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/27 09:08 linux-4.14.y 7878a41b6cc1 9189cb53 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/26 07:52 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/26 02:26 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/26 00:48 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 23:08 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 22:02 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 18:56 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 17:24 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 15:16 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 12:44 linux-4.14.y 7878a41b6cc1 ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/25 05:33 linux-4.14.y 1e61bd26fa2c ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/24 17:37 linux-4.14.y 1e61bd26fa2c ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/24 15:00 linux-4.14.y 1e61bd26fa2c ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/24 13:58 linux-4.14.y 1e61bd26fa2c ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/24 11:56 linux-4.14.y 1e61bd26fa2c ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/24 10:53 linux-4.14.y 1e61bd26fa2c ee50e71c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/23 12:54 linux-4.14.y 1e61bd26fa2c 9e2ebb3c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/23 10:27 linux-4.14.y 1e61bd26fa2c 9e2ebb3c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/23 07:37 linux-4.14.y 1e61bd26fa2c 9f1e2cb3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/22 22:58 linux-4.14.y 1e61bd26fa2c 9f1e2cb3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/22 16:46 linux-4.14.y 1e61bd26fa2c 9f1e2cb3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/22 00:56 linux-4.14.y a8ad60f2af58 42a4d508 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2023/02/21 21:44 linux-4.14.y a8ad60f2af58 42a4d508 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
2022/12/17 16:53 linux-4.14.y c4215ee4771b 05494336 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in btrfs_scrub_dev
* Struck through repros no longer work on HEAD.