syzbot


possible deadlock in strp_work

Status: fixed on 2023/02/24 13:50
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+9fc084a4348493ef65d2@syzkaller.appspotmail.com
Fix commit: 8fc29ff3910f kcm: fix strp_init() order and cleanup
First crash: 611d, last: 606d
Cause bisection: failed (error log, bisect log)
  
Discussions (9)
Title Replies (including bot) Last reply
[PATCH 5.4 000/108] 5.4.212-rc1 review 114 (114) 2022/09/17 03:06
[PATCH 4.19 00/79] 4.19.257-rc1 review 88 (88) 2022/09/17 02:04
[PATCH 5.15 000/107] 5.15.66-rc1 review 131 (131) 2022/09/16 16:25
[PATCH 4.14 00/61] 4.14.293-rc1 review 64 (64) 2022/09/15 00:15
[PATCH 5.19 000/155] 5.19.8-rc1 review 167 (167) 2022/09/08 09:40
[PATCH 5.10 00/80] 5.10.142-rc1 review 88 (88) 2022/09/08 04:04
[Patch net v2] kcm: fix strp_init() order and cleanup 2 (2) 2022/08/31 19:20
[Patch net] kcm: get rid of unnecessary cleanup 2 (2) 2022/08/23 23:39
[syzbot] possible deadlock in strp_work 0 (1) 2022/08/21 10:17
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.14 possible deadlock in strp_work C error 214 588d 1505d 0/1 upstream: reported C repro on 2020/03/06 14:18
Last patch testing requests (2)
Created Duration User Patch Repo Result
2022/08/27 17:40 16m xiyou.wangcong@gmail.com https://github.com/congwang/linux.git net OK log
2022/08/21 18:12 16m xiyou.wangcong@gmail.com patch git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git master OK log

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
6.0.0-rc1-next-20220819-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:3/51 is trying to acquire lock:
ffff88801c488d30 (sk_lock-AF_INET){+.+.}-{0:0}, at: do_strp_work net/strparser/strparser.c:398 [inline]
ffff88801c488d30 (sk_lock-AF_INET){+.+.}-{0:0}, at: strp_work+0x40/0x130 net/strparser/strparser.c:415

but task is already holding lock:
ffffc90000bc7da8 ((work_completion)(&strp->work)){+.+.}-{0:0}, at: process_one_work+0x8ae/0x1610 kernel/workqueue.c:2264

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 ((work_completion)(&strp->work)){+.+.}-{0:0}:
       __flush_work+0x105/0xae0 kernel/workqueue.c:3069
       __cancel_work_timer+0x3f9/0x570 kernel/workqueue.c:3160
       strp_done+0x64/0xf0 net/strparser/strparser.c:513
       kcm_attach net/kcm/kcmsock.c:1429 [inline]
       kcm_attach_ioctl net/kcm/kcmsock.c:1490 [inline]
       kcm_ioctl+0x913/0x1180 net/kcm/kcmsock.c:1696
       sock_do_ioctl+0xcc/0x230 net/socket.c:1169
       sock_ioctl+0x2f1/0x640 net/socket.c:1286
       vfs_ioctl fs/ioctl.c:51 [inline]
       __do_sys_ioctl fs/ioctl.c:870 [inline]
       __se_sys_ioctl fs/ioctl.c:856 [inline]
       __x64_sys_ioctl+0x193/0x200 fs/ioctl.c:856
       do_syscall_x64 arch/x86/entry/common.c:50 [inline]
       do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
       entry_SYSCALL_64_after_hwframe+0x63/0xcd

-> #0 (sk_lock-AF_INET){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:3095 [inline]
       check_prevs_add kernel/locking/lockdep.c:3214 [inline]
       validate_chain kernel/locking/lockdep.c:3829 [inline]
       __lock_acquire+0x2a43/0x56d0 kernel/locking/lockdep.c:5053
       lock_acquire kernel/locking/lockdep.c:5666 [inline]
       lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5631
       lock_sock_nested+0x36/0xf0 net/core/sock.c:3391
       do_strp_work net/strparser/strparser.c:398 [inline]
       strp_work+0x40/0x130 net/strparser/strparser.c:415
       process_one_work+0x991/0x1610 kernel/workqueue.c:2289
       worker_thread+0x665/0x1080 kernel/workqueue.c:2436
       kthread+0x2e4/0x3a0 kernel/kthread.c:376
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((work_completion)(&strp->work));
                               lock(sk_lock-AF_INET);
                               lock((work_completion)(&strp->work));
  lock(sk_lock-AF_INET);

 *** DEADLOCK ***

2 locks held by kworker/u4:3/51:
 #0: ffff88802646f138 ((wq_completion)kstrp){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff88802646f138 ((wq_completion)kstrp){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline]
 #0: ffff88802646f138 ((wq_completion)kstrp){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1280 [inline]
 #0: ffff88802646f138 ((wq_completion)kstrp){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:636 [inline]
 #0: ffff88802646f138 ((wq_completion)kstrp){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:663 [inline]
 #0: ffff88802646f138 ((wq_completion)kstrp){+.+.}-{0:0}, at: process_one_work+0x87a/0x1610 kernel/workqueue.c:2260
 #1: ffffc90000bc7da8 ((work_completion)(&strp->work)){+.+.}-{0:0}, at: process_one_work+0x8ae/0x1610 kernel/workqueue.c:2264

stack backtrace:
CPU: 1 PID: 51 Comm: kworker/u4:3 Not tainted 6.0.0-rc1-next-20220819-syzkaller #0
kworker/u4:3[51] cmdline: a
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 07/22/2022
Workqueue: kstrp strp_work
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:122 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:140
 check_noncircular+0x25f/0x2e0 kernel/locking/lockdep.c:2175
 check_prev_add kernel/locking/lockdep.c:3095 [inline]
 check_prevs_add kernel/locking/lockdep.c:3214 [inline]
 validate_chain kernel/locking/lockdep.c:3829 [inline]
 __lock_acquire+0x2a43/0x56d0 kernel/locking/lockdep.c:5053
 lock_acquire kernel/locking/lockdep.c:5666 [inline]
 lock_acquire+0x1ab/0x570 kernel/locking/lockdep.c:5631
 lock_sock_nested+0x36/0xf0 net/core/sock.c:3391
 do_strp_work net/strparser/strparser.c:398 [inline]
 strp_work+0x40/0x130 net/strparser/strparser.c:415
 process_one_work+0x991/0x1610 kernel/workqueue.c:2289
 worker_thread+0x665/0x1080 kernel/workqueue.c:2436
 kthread+0x2e4/0x3a0 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:306
 </TASK>

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2022/08/22 15:19 linux-next 8755ae45a9e8 26a13b38 .config console log report syz C ci-upstream-linux-next-kasan-gce-root possible deadlock in strp_work
2022/08/22 08:02 linux-next 8755ae45a9e8 26a13b38 .config console log report syz C ci-upstream-linux-next-kasan-gce-root possible deadlock in strp_work
2022/08/20 21:46 linux-next 8755ae45a9e8 26a13b38 .config console log report syz C ci-upstream-linux-next-kasan-gce-root possible deadlock in strp_work
2022/08/17 10:06 linux-next 95d10484d66e 4e72d229 .config console log report info ci-upstream-linux-next-kasan-gce-root possible deadlock in strp_work
* Struck through repros no longer work on HEAD.