syzbot


possible deadlock in cleanup_net

Status: fixed on 2021/08/30 09:32
Reported-by: syzbot+bf91330c3df22a4c2b7c@syzkaller.appspotmail.com
Fix commit: 3719acc161d5 Bluetooth: defer cleanup of resources in hci_unregister_dev()
First crash: 1472d, last: 1210d
Fix bisection: fixed by (bisect log) :
commit 3719acc161d5c1ce09912cc1c9eddc2c5faa3c66
Author: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Date: Wed Aug 4 10:26:56 2021 +0000

  Bluetooth: defer cleanup of resources in hci_unregister_dev()

  
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in cleanup_net net C done 281 1535d 1542d 15/28 fixed on 2020/11/16 12:12
Fix bisection attempts (2)
Created Duration User Patch Repo Result
2021/08/29 11:02 4h07m bisect fix linux-4.19.y OK (1) job log
2021/07/30 10:30 32m bisect fix linux-4.19.y OK (0) job log log

Sample crash report:
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
======================================================
WARNING: possible circular locking dependency detected
4.19.161-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:3/110 is trying to acquire lock:
00000000f710179f ((wq_completion)"events"){+.+.}, at: flush_workqueue+0xe8/0x13e0 kernel/workqueue.c:2660

but task is already holding lock:
00000000a7c48411 (pernet_ops_rwsem){++++}, at: cleanup_net+0xa8/0x8b0 net/core/net_namespace.c:520

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (pernet_ops_rwsem){++++}:
       unregister_netdevice_notifier+0x7b/0x330 net/core/dev.c:1708
       raw_release+0x58/0x820 net/can/raw.c:358
       __sock_release+0xcd/0x2a0 net/socket.c:579
       sock_close+0x15/0x20 net/socket.c:1140
       __fput+0x2ce/0x890 fs/file_table.c:278
       task_work_run+0x148/0x1c0 kernel/task_work.c:113
       tracehook_notify_resume include/linux/tracehook.h:193 [inline]
       exit_to_usermode_loop+0x251/0x2a0 arch/x86/entry/common.c:167
       prepare_exit_to_usermode arch/x86/entry/common.c:198 [inline]
       syscall_return_slowpath arch/x86/entry/common.c:271 [inline]
       do_syscall_64+0x538/0x620 arch/x86/entry/common.c:296
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #2 (&sb->s_type->i_mutex_key#13){+.+.}:
       inode_lock include/linux/fs.h:748 [inline]
       __sock_release+0x86/0x2a0 net/socket.c:578
       sock_close+0x15/0x20 net/socket.c:1140
       __fput+0x2ce/0x890 fs/file_table.c:278
       delayed_fput+0x56/0x70 fs/file_table.c:304
       process_one_work+0x864/0x1570 kernel/workqueue.c:2155
       worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
       kthread+0x33f/0x460 kernel/kthread.c:259
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #1 ((delayed_fput_work).work){+.+.}:
       worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
       kthread+0x33f/0x460 kernel/kthread.c:259
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

-> #0 ((wq_completion)"events"){+.+.}:
       flush_workqueue+0x117/0x13e0 kernel/workqueue.c:2663
       flush_scheduled_work include/linux/workqueue.h:599 [inline]
       tipc_exit_net+0x38/0x60 net/tipc/core.c:100
       ops_exit_list+0xa5/0x150 net/core/net_namespace.c:153
       cleanup_net+0x3b4/0x8b0 net/core/net_namespace.c:553
       process_one_work+0x864/0x1570 kernel/workqueue.c:2155
       worker_thread+0x64c/0x1130 kernel/workqueue.c:2298
       kthread+0x33f/0x460 kernel/kthread.c:259
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:415

other info that might help us debug this:

Chain exists of:
  (wq_completion)"events" --> &sb->s_type->i_mutex_key#13 --> pernet_ops_rwsem

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(pernet_ops_rwsem);
                               lock(&sb->s_type->i_mutex_key#13);
                               lock(pernet_ops_rwsem);
  lock((wq_completion)"events");

 *** DEADLOCK ***

3 locks held by kworker/u4:3/110:
 #0: 000000000251fa38 ((wq_completion)"%s""netns"){+.+.}, at: process_one_work+0x767/0x1570 kernel/workqueue.c:2126
 #1: 00000000b133b634 (net_cleanup_work){+.+.}, at: process_one_work+0x79c/0x1570 kernel/workqueue.c:2130
 #2: 00000000a7c48411 (pernet_ops_rwsem){++++}, at: cleanup_net+0xa8/0x8b0 net/core/net_namespace.c:520

stack backtrace:
CPU: 0 PID: 110 Comm: kworker/u4:3 Not tainted 4.19.161-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1fc/0x2fe lib/dump_stack.c:118
 print_circular_bug.constprop.0.cold+0x2d7/0x41e kernel/locking/lockdep.c:1221
 check_prev_add kernel/locking/lockdep.c:1865 [inline]
 check_prevs_add kernel/locking/lockdep.c:1978 [inline]
 validate_chain kernel/locking/lockdep.c:2419 [inline]
 __lock_acquire+0x30c9/0x3ff0 kernel/locking/lockdep.c:3415
 lock_acquire+0x170/0x3c0 kernel/locking/lockdep.c:3907
 flush_workqueue+0x117/0x13e0 kernel/workqueue.c:2663
 flush_scheduled_work include/linux/workqueue.h:599 [inline]
 tipc_exit_net+0x38/0x60 net/tipc/core.c:100
 ops_exit_list+0xa5/0x150 net/core/net_namespace.c:153
 cleanup_net+0x3b4/0x8b0 net/core/net_namespace.c:553
 process_one_work+0x864/0x1570 kernel/workqueue.c:2155
 worke

Crashes (5212):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/12/06 04:36 linux-4.19.y daefdc9eb24b 50503117 .config console log report syz C ci2-linux-4-19
2021/06/30 10:30 linux-4.19.y eb575cd5d7f6 84fd4c77 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/30 08:04 linux-4.19.y eb575cd5d7f6 84fd4c77 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/30 05:20 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/30 03:46 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/30 01:13 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/30 00:11 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 22:38 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 22:05 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 21:02 linux-4.19.y eb575cd5d7f6 a4fccb01 .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 18:39 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 11:14 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 09:20 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 08:10 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 06:19 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 04:09 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 03:08 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/29 02:04 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 22:52 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 21:15 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 17:43 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 15:55 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 14:50 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 13:33 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 12:14 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 08:15 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 07:08 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 05:18 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 03:30 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 01:25 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/28 00:01 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 21:18 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 18:49 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 17:41 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 16:16 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 13:36 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 12:11 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 11:39 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 10:29 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 09:13 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 08:08 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 06:56 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 05:35 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 03:48 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/27 03:35 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/26 22:56 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/26 21:31 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/26 20:10 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/06/26 18:55 linux-4.19.y eb575cd5d7f6 9d2ab5df .config console log report info ci2-linux-4-19 possible deadlock in cleanup_net
2021/01/17 14:13 linux-4.19.y c110fed0e606 813be542 .config console log report info ci2-linux-4-19
2020/11/10 12:25 linux-4.19.y 53fff24aaf01 cca87986 .config console log report info ci2-linux-4-19
* Struck through repros no longer work on HEAD.