syzbot


possible deadlock in start_transaction

Status: upstream: reported syz repro on 2022/12/27 19:04
Subsystems: btrfs
[Documentation on labels]
Reported-by: syzbot+5f0cb326a365dd7d0ea1@syzkaller.appspotmail.com
First crash: 544d, last: 502d
Similar bugs (3)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 possible deadlock in start_transaction (2) btrfs C 424 475d 574d 0/1 upstream: reported C repro on 2022/11/27 06:33
upstream possible deadlock in start_transaction btrfs 2 1362d 1358d 0/27 auto-closed as invalid on 2021/01/28 08:05
linux-4.19 possible deadlock in start_transaction 1 891d 891d 0/1 auto-closed as invalid on 2022/05/14 23:34

Sample crash report:
audit: type=1804 audit(1672206184.336:11): pid=9687 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.2" name="/root/syzkaller-testdir1518096429/syzkaller.iMp60r/4/file0/.log" dev="loop2" ino=263 res=1
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
======================================================
WARNING: possible circular locking dependency detected
4.14.302-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:0/5 is trying to acquire lock:
 (sb_internal#2){.+.+}, at: [<ffffffff829edf5e>] sb_start_intwrite include/linux/fs.h:1598 [inline]
 (sb_internal#2){.+.+}, at: [<ffffffff829edf5e>] start_transaction+0x6de/0xf30 fs/btrfs/transaction.c:548

but task is already holding lock:
 ((&work->normal_work)){+.+.}, at: [<ffffffff81366166>] process_one_work+0x6e6/0x14a0 kernel/workqueue.c:2092

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 ((&work->normal_work)){+.+.}:
       process_one_work+0x736/0x14a0 kernel/workqueue.c:2093
       worker_thread+0x5cc/0xff0 kernel/workqueue.c:2251
       kthread+0x30d/0x420 kernel/kthread.c:232
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:406

-> #3 ("%s-%s""btrfs", name){+.+.}:
       flush_workqueue+0xfa/0x1310 kernel/workqueue.c:2625
       drain_workqueue+0x177/0x3e0 kernel/workqueue.c:2790
       destroy_workqueue+0x71/0x710 kernel/workqueue.c:4116
       __btrfs_destroy_workqueue fs/btrfs/async-thread.c:436 [inline]
       btrfs_destroy_workqueue+0xf8/0x630 fs/btrfs/async-thread.c:447
       scrub_workers_put+0x90/0x1a0 fs/btrfs/scrub.c:4075
       btrfs_scrub_dev+0x536/0xcd0 fs/btrfs/scrub.c:4219
       btrfs_ioctl_scrub fs/btrfs/ioctl.c:4451 [inline]
       btrfs_ioctl+0xba8/0x5b20 fs/btrfs/ioctl.c:5681
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:500 [inline]
       do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
       SYSC_ioctl fs/ioctl.c:701 [inline]
       SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x5e/0xd3

-> #2 (&fs_info->scrub_lock){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       btrfs_scrub_dev+0x1f3/0xcd0 fs/btrfs/scrub.c:4150
       btrfs_ioctl_scrub fs/btrfs/ioctl.c:4451 [inline]
       btrfs_ioctl+0xba8/0x5b20 fs/btrfs/ioctl.c:5681
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:500 [inline]
       do_vfs_ioctl+0x75a/0xff0 fs/ioctl.c:684
       SYSC_ioctl fs/ioctl.c:701 [inline]
       SyS_ioctl+0x7f/0xb0 fs/ioctl.c:692
       do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x5e/0xd3

-> #1 (&fs_devs->device_list_mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xc4/0x1310 kernel/locking/mutex.c:893
       btrfs_finish_chunk_alloc+0x221/0xe90 fs/btrfs/volumes.c:4923
       btrfs_create_pending_block_groups+0x1fd/0x540 fs/btrfs/extent-tree.c:10388
       __btrfs_end_transaction+0x1f2/0xaa0 fs/btrfs/transaction.c:851
       flush_space+0x8de/0xde0 fs/btrfs/extent-tree.c:5046
       btrfs_async_reclaim_metadata_space+0x414/0xc20 fs/btrfs/extent-tree.c:5162
       process_one_work+0x793/0x14a0 kernel/workqueue.c:2117
       worker_thread+0x5cc/0xff0 kernel/workqueue.c:2251
       kthread+0x30d/0x420 kernel/kthread.c:232
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:406

-> #0 (sb_internal#2){.+.+}:
       lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
       percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
       percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
       __sb_start_write+0x64/0x260 fs/super.c:1342
       sb_start_intwrite include/linux/fs.h:1598 [inline]
       start_transaction+0x6de/0xf30 fs/btrfs/transaction.c:548
       btrfs_qgroup_rescan_worker+0x176/0x1060 fs/btrfs/qgroup.c:2632
       normal_work_helper+0x304/0x1330 fs/btrfs/async-thread.c:376
       process_one_work+0x793/0x14a0 kernel/workqueue.c:2117
       worker_thread+0x5cc/0xff0 kernel/workqueue.c:2251
       kthread+0x30d/0x420 kernel/kthread.c:232
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:406

other info that might help us debug this:

Chain exists of:
  sb_internal#2 --> "%s-%s""btrfs", name --> (&work->normal_work)

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((&work->normal_work));
                               lock("%s-%s""btrfs", name);
                               lock((&work->normal_work));
  lock(sb_internal#2);

 *** DEADLOCK ***

2 locks held by kworker/u4:0/5:
 #0:  ("%s-%s""btrfs", name){+.+.}, at: [<ffffffff81366130>] process_one_work+0x6b0/0x14a0 kernel/workqueue.c:2088
 #1:  ((&work->normal_work)){+.+.}, at: [<ffffffff81366166>] process_one_work+0x6e6/0x14a0 kernel/workqueue.c:2092

stack backtrace:
CPU: 1 PID: 5 Comm: kworker/u4:0 Not tainted 4.14.302-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
Workqueue: btrfs-qgroup-rescan btrfs_qgroup_rescan_helper
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x1b2/0x281 lib/dump_stack.c:58
 print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1905 [inline]
 check_prevs_add kernel/locking/lockdep.c:2022 [inline]
 validate_chain kernel/locking/lockdep.c:2464 [inline]
 __lock_acquire+0x2e0e/0x3f20 kernel/locking/lockdep.c:3491
 lock_acquire+0x170/0x3f0 kernel/locking/lockdep.c:3998
 percpu_down_read_preempt_disable include/linux/percpu-rwsem.h:36 [inline]
 percpu_down_read include/linux/percpu-rwsem.h:59 [inline]
 __sb_start_write+0x64/0x260 fs/super.c:1342
 sb_start_intwrite include/linux/fs.h:1598 [inline]
 start_transaction+0x6de/0xf30 fs/btrfs/transaction.c:548
 btrfs_qgroup_rescan_worker+0x176/0x1060 fs/btrfs/qgroup.c:2632
 normal_work_helper+0x304/0x1330 fs/btrfs/async-thread.c:376
 process_one_work+0x793/0x14a0 kernel/workqueue.c:2117
 worker_thread+0x5cc/0xff0 kernel/workqueue.c:2251
 kthread+0x30d/0x420 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:406
kauditd_printk_skb: 3 callbacks suppressed
audit: type=1804 audit(1672206185.917:15): pid=9782 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.5" name="/root/syzkaller-testdir7186275/syzkaller.ui2VsU/3/file0/.log" dev="loop2" ino=263 res=1
audit: type=1804 audit(1672206185.917:16): pid=9775 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.3" name="/root/syzkaller-testdir709057861/syzkaller.eZgcEB/3/file0/.log" dev="loop2" ino=263 res=1
audit: type=1804 audit(1672206185.927:17): pid=9778 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.1" name="/root/syzkaller-testdir2753593569/syzkaller.28xMi4/3/file0/.log" dev="loop2" ino=263 res=1
audit: type=1804 audit(1672206185.927:18): pid=9780 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.4" name="/root/syzkaller-testdir2476623929/syzkaller.nVRzQ3/3/file0/.log" dev="loop2" ino=263 res=1
audit: type=1804 audit(1672206185.957:19): pid=9790 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.0" name="/root/syzkaller-testdir905495311/syzkaller.WLmpgv/4/file0/.log" dev="loop2" ino=263 res=1
audit: type=1804 audit(1672206186.307:20): pid=9850 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.2" name="/root/syzkaller-testdir1518096429/syzkaller.iMp60r/6/.log" dev="sda1" ino=13933 res=1
BTRFS error (device loop2): fail to start transaction for status update: -28
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
audit: type=1804 audit(1672206186.547:21): pid=9866 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.2" name="/root/syzkaller-testdir1518096429/syzkaller.iMp60r/7/.log" dev="sda1" ino=13940 res=1
audit: type=1804 audit(1672206186.697:22): pid=9900 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.2" name="/root/syzkaller-testdir1518096429/syzkaller.iMp60r/8/.log" dev="sda1" ino=13941 res=1
audit: type=1804 audit(1672206186.807:23): pid=9919 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.2" name="/root/syzkaller-testdir1518096429/syzkaller.iMp60r/9/.log" dev="sda1" ino=13941 res=1
audit: type=1804 audit(1672206186.927:24): pid=9857 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.3" name="/root/syzkaller-testdir709057861/syzkaller.eZgcEB/4/file0/.log" dev="loop2" ino=263 res=1
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop2): using free space tree
BTRFS info (device loop2): has skinny extents
BTRFS info (device loop0): using free space tree
BTRFS info (device loop0): has skinny extents
BTRFS info (device loop1): using free space tree
BTRFS info (device loop1): has skinny extents
BTRFS info (device loop1): using free space tree
BTRFS info (device loop1): has skinny extents
kauditd_printk_skb: 21 callbacks suppressed
audit: type=1804 audit(1672206190.978:46): pid=10304 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.1" name="/root/syzkaller-testdir2753593569/syzkaller.28xMi4/10/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.068:47): pid=10316 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.4" name="/root/syzkaller-testdir2476623929/syzkaller.nVRzQ3/10/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.118:48): pid=10325 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.3" name="/root/syzkaller-testdir709057861/syzkaller.eZgcEB/10/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.118:49): pid=10324 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.5" name="/root/syzkaller-testdir7186275/syzkaller.ui2VsU/10/file0/.log" dev="loop1" ino=263 res=1
BTRFS info (device loop1): using free space tree
BTRFS info (device loop1): has skinny extents
audit: type=1804 audit(1672206191.768:50): pid=10376 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.1" name="/root/syzkaller-testdir2753593569/syzkaller.28xMi4/11/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.768:51): pid=10397 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.5" name="/root/syzkaller-testdir7186275/syzkaller.ui2VsU/11/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.768:52): pid=10397 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.5" name="/root/syzkaller-testdir7186275/syzkaller.ui2VsU/11/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.898:53): pid=10395 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.2" name="/root/syzkaller-testdir1518096429/syzkaller.iMp60r/18/file0/.log" dev="loop1" ino=263 res=1
audit: type=1804 audit(1672206191.898:54): pid=10389 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="ToMToU" comm="syz-executor.0" name="/root/syzkaller-testdir905495311/syzkaller.WLmpgv/12/file0/.log" dev="loop1" ino=263 res=1
BTRFS info (device loop1): using free space tree
BTRFS info (device loop1): has skinny extents
audit: type=1804 audit(1672206192.478:55): pid=10461 uid=0 auid=4294967295 ses=4294967295 op="invalid_pcr" cause="open_writers" comm="syz-executor.1" name="/root/syzkaller-testdir2753593569/syzkaller.28xMi4/12/file0/.log" dev="loop1" ino=263 res=1
BTRFS info (device loop4): using free space tree
BTRFS info (device loop4): has skinny extents
BTRFS info (device loop4): using free space tree
BTRFS info (device loop4): has skinny extents
BTRFS info (device loop4): using free space tree
BTRFS info (device loop4): has skinny extents

Crashes (6):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/12/28 05:47 linux-4.14.y c4215ee4771b 44712fbc .config console log report syz [disk image] [vmlinux] [kernel image] [mounted in repro #1] [mounted in repro #2] ci2-linux-4-14 possible deadlock in start_transaction
2023/02/07 12:37 linux-4.14.y a8ad60f2af58 5bc3be51 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in start_transaction
2023/01/10 02:59 linux-4.14.y c4215ee4771b 48bc529a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in start_transaction
2023/01/10 00:45 linux-4.14.y c4215ee4771b 48bc529a .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in start_transaction
2022/12/28 06:32 linux-4.14.y c4215ee4771b 44712fbc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in start_transaction
2022/12/27 19:04 linux-4.14.y c4215ee4771b 44712fbc .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-4-14 possible deadlock in start_transaction
* Struck through repros no longer work on HEAD.