syzbot


possible deadlock in cleanup_net

Status: fixed on 2020/11/16 12:12
Subsystems: net
[Documentation on labels]
Reported-by: syzbot+d5aa7e0385f6a5d0f4fd@syzkaller.appspotmail.com
Fix commit: d966ddcc3821 tipc: fix a deadlock when flushing scheduled work
First crash: 1330d, last: 1318d
Cause bisection: introduced by (bisect log) :
commit fdeba99b1e58ecd18c2940c453e19e4ef20ff591
Author: Hoang Huu Le <hoang.h.le@dektech.com.au>
Date: Thu Aug 27 02:56:51 2020 +0000

  tipc: fix use-after-free in tipc_bcast_get_mode

Crash: possible deadlock in __sock_release (log)
Repro: syz .config
  
Discussions (5)
Title Replies (including bot) Last reply
[PATCH 5.9 00/75] 5.9.14-rc1 review 83 (83) 2020/12/11 14:23
[PATCH 5.4 00/54] 5.4.83-rc1 review 57 (57) 2020/12/10 16:46
[net-next v2] tipc: fix a deadlock when flushing scheduled work 2 (2) 2020/09/07 19:10
[net-next] tipc: fix a deadlock when flushing scheduled work 5 (5) 2020/09/07 15:05
possible deadlock in cleanup_net 0 (1) 2020/08/31 17:17
Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 possible deadlock in cleanup_net C done 5212 994d 1255d 1/1 fixed on 2021/08/30 09:32

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.9.0-rc2-next-20200828-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:6/349 is trying to acquire lock:
ffff8880aa063d38 ((wq_completion)events){+.+.}-{0:0}, at: flush_workqueue+0xe1/0x13e0 kernel/workqueue.c:2777

but task is already holding lock:
ffffffff8a879430 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0x9b/0xb10 net/core/net_namespace.c:565

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (pernet_ops_rwsem){++++}-{3:3}:
       down_write+0x8d/0x150 kernel/locking/rwsem.c:1531
       unregister_netdevice_notifier+0x1e/0x170 net/core/dev.c:1861
       bcm_release+0x94/0x750 net/can/bcm.c:1474
       __sock_release+0xcd/0x280 net/socket.c:596
       sock_close+0x18/0x20 net/socket.c:1277
       __fput+0x285/0x920 fs/file_table.c:281
       task_work_run+0xdd/0x190 kernel/task_work.c:141
       tracehook_notify_resume include/linux/tracehook.h:188 [inline]
       exit_to_user_mode_loop kernel/entry/common.c:140 [inline]
       exit_to_user_mode_prepare+0x195/0x1c0 kernel/entry/common.c:167
       syscall_exit_to_user_mode+0x59/0x2b0 kernel/entry/common.c:242
       entry_SYSCALL_64_after_hwframe+0x44/0xa9

-> #2 (&sb->s_type->i_mutex_key#13){+.+.}-{3:3}:
       down_write+0x8d/0x150 kernel/locking/rwsem.c:1531
       inode_lock include/linux/fs.h:779 [inline]
       __sock_release+0x86/0x280 net/socket.c:595
       sock_close+0x18/0x20 net/socket.c:1277
       __fput+0x285/0x920 fs/file_table.c:281
       delayed_fput+0x56/0x70 fs/file_table.c:309
       process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
       worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
       kthread+0x3b5/0x4a0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #1 ((delayed_fput_work).work){+.+.}-{0:0}:
       process_one_work+0x8bb/0x1670 kernel/workqueue.c:2245
       worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
       kthread+0x3b5/0x4a0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #0 ((wq_completion)events){+.+.}-{0:0}:
       check_prev_add kernel/locking/lockdep.c:2496 [inline]
       check_prevs_add kernel/locking/lockdep.c:2601 [inline]
       validate_chain kernel/locking/lockdep.c:3218 [inline]
       __lock_acquire+0x2a6b/0x5640 kernel/locking/lockdep.c:4426
       lock_acquire+0x1f1/0xad0 kernel/locking/lockdep.c:5005
       flush_workqueue+0x110/0x13e0 kernel/workqueue.c:2780
       flush_scheduled_work include/linux/workqueue.h:597 [inline]
       tipc_exit_net+0x47/0x2a0 net/tipc/core.c:116
       ops_exit_list+0xb0/0x160 net/core/net_namespace.c:186
       cleanup_net+0x4ea/0xb10 net/core/net_namespace.c:603
       process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
       worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
       kthread+0x3b5/0x4a0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

other info that might help us debug this:

Chain exists of:
  (wq_completion)events --> &sb->s_type->i_mutex_key#13 --> pernet_ops_rwsem

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(pernet_ops_rwsem);
                               lock(&sb->s_type->i_mutex_key#13);
                               lock(pernet_ops_rwsem);
  lock((wq_completion)events);

 *** DEADLOCK ***

3 locks held by kworker/u4:6/349:
 #0: ffff8880a97b1138 ((wq_completion)netns){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff8880a97b1138 ((wq_completion)netns){+.+.}-{0:0}, at: atomic64_set include/asm-generic/atomic-instrumented.h:856 [inline]
 #0: ffff8880a97b1138 ((wq_completion)netns){+.+.}-{0:0}, at: atomic_long_set include/asm-generic/atomic-long.h:41 [inline]
 #0: ffff8880a97b1138 ((wq_completion)netns){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:616 [inline]
 #0: ffff8880a97b1138 ((wq_completion)netns){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:643 [inline]
 #0: ffff8880a97b1138 ((wq_completion)netns){+.+.}-{0:0}, at: process_one_work+0x82b/0x1670 kernel/workqueue.c:2240
 #1: ffffc900020e7da8 (net_cleanup_work){+.+.}-{0:0}, at: process_one_work+0x85f/0x1670 kernel/workqueue.c:2244
 #2: ffffffff8a879430 (pernet_ops_rwsem){++++}-{3:3}, at: cleanup_net+0x9b/0xb10 net/core/net_namespace.c:565

stack backtrace:
CPU: 1 PID: 349 Comm: kworker/u4:6 Not tainted 5.9.0-rc2-next-20200828-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x18f/0x20d lib/dump_stack.c:118
 check_noncircular+0x324/0x3e0 kernel/locking/lockdep.c:1827
 check_prev_add kernel/locking/lockdep.c:2496 [inline]
 check_prevs_add kernel/locking/lockdep.c:2601 [inline]
 validate_chain kernel/locking/lockdep.c:3218 [inline]
 __lock_acquire+0x2a6b/0x5640 kernel/locking/lockdep.c:4426
 lock_acquire+0x1f1/0xad0 kernel/locking/lockdep.c:5005
 flush_workqueue+0x110/0x13e0 kernel/workqueue.c:2780
 flush_scheduled_work include/linux/workqueue.h:597 [inline]
 tipc_exit_net+0x47/0x2a0 net/tipc/core.c:116
 ops_exit_list+0xb0/0x160 net/core/net_namespace.c:186
 cleanup_net+0x4ea/0xb10 net/core/net_namespace.c:603
 process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
 worker_thread+0x64c/0x1120 kernel/workqueue.c:2415

Crashes (281):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/08/30 15:01 linux-next b36c969764ab d5a3ae1f .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2020/08/29 03:25 linux-next b36c969764ab d5a3ae1f .config console log report syz C ci-upstream-linux-next-kasan-gce-root
2020/08/28 09:11 net-next-old 50aba46c234e 816e0689 .config console log report syz ci-upstream-net-kasan-gce
2020/09/08 15:26 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 14:38 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 11:57 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 10:32 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 09:15 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 07:28 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 07:04 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 05:06 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 03:01 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/08 00:47 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 23:46 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 22:47 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 21:20 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 19:16 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 17:57 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 13:58 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 09:44 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 04:29 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 03:51 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 02:13 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/07 01:01 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 19:53 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 18:45 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 15:06 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 14:03 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 12:58 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 12:43 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 09:22 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 07:38 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 04:48 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/06 01:46 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 23:53 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 18:44 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 14:41 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 13:35 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 12:45 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 11:24 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 10:07 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 08:52 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 07:45 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/09/05 02:34 linux-next 7a6956579ce6 abf9ba4f .config console log report ci-upstream-linux-next-kasan-gce-root
2020/08/28 08:49 linux-next b36c969764ab 816e0689 .config console log report ci-upstream-linux-next-kasan-gce-root
* Struck through repros no longer work on HEAD.