possible deadlock in __sock

possible deadlock in __sock_release
Status: upstream: reported C repro on 2020/08/31 20:58
Reported-by: syzbot+8e467b009209f1fcf666@syzkaller.appspotmail.com
First crash: 237d, last: 211d

Cause bisection: introduced by (bisect log) [merge commit]:
commit edea44f5872a14d0e8698e0e0e540320833db9cb
Author: Jiri Kosina <jkosina@suse.cz>
Date: Wed Feb 12 13:20:41 2020 +0000

Merge branch 'for-5.7/core' into for-next

Crash: WARNING: ODEBUG bug in __do_softirq (log)
Repro: C syz .config

similar bugs (1):
Kernel	Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
linux-4.19	possible deadlock in __sock_release	C			39812	11m	162d	0/1	upstream: reported C repro on 2020/11/10 12:29

Sample crash report:

IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
======================================================
WARNING: possible circular locking dependency detected
5.9.0-rc3-next-20200903-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor985/6864 is trying to acquire lock:
ffffffff8a87d730 (pernet_ops_rwsem){++++}-{3:3}, at: unregister_netdevice_notifier+0x1e/0x170 net/core/dev.c:1861

but task is already holding lock:
ffff888085909c90 (&sb->s_type->i_mutex_key#12){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:779 [inline]
ffff888085909c90 (&sb->s_type->i_mutex_key#12){+.+.}-{3:3}, at: __sock_release+0x86/0x280 net/socket.c:595

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (&sb->s_type->i_mutex_key#12){+.+.}-{3:3}:
       down_write+0x8d/0x150 kernel/locking/rwsem.c:1531
       inode_lock include/linux/fs.h:779 [inline]
       __sock_release+0x86/0x280 net/socket.c:595
       sock_close+0x18/0x20 net/socket.c:1277
       __fput+0x285/0x920 fs/file_table.c:281
       delayed_fput+0x56/0x70 fs/file_table.c:309
       process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
       worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
       kthread+0x3b5/0x4a0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #2 ((delayed_fput_work).work){+.+.}-{0:0}:
       process_one_work+0x8bb/0x1670 kernel/workqueue.c:2245
       worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
       kthread+0x3b5/0x4a0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #1 ((wq_completion)events){+.+.}-{0:0}:
       flush_workqueue+0x110/0x13e0 kernel/workqueue.c:2780
       flush_scheduled_work include/linux/workqueue.h:597 [inline]
       tipc_exit_net+0x47/0x2a0 net/tipc/core.c:116
       ops_exit_list+0xb0/0x160 net/core/net_namespace.c:186
       cleanup_net+0x4ea/0xb10 net/core/net_namespace.c:603
       process_one_work+0x94c/0x1670 kernel/workqueue.c:2269
       worker_thread+0x64c/0x1120 kernel/workqueue.c:2415
       kthread+0x3b5/0x4a0 kernel/kthread.c:292
       ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:294

-> #0 (pernet_ops_rwsem){++++}-{3:3}:
       check_prev_add kernel/locking/lockdep.c:2496 [inline]
       check_prevs_add kernel/locking/lockdep.c:2601 [inline]
       validate_chain kernel/locking/lockdep.c:3218 [inline]
       __lock_acquire+0x29bb/0x5570 kernel/locking/lockdep.c:4426
       lock_acquire+0x1f3/0xae0 kernel/locking/lockdep.c:5006
       down_write+0x8d/0x150 kernel/locking/rwsem.c:1531
       unregister_netdevice_notifier+0x1e/0x170 net/core/dev.c:1861
       raw_release+0x58/0x890 net/can/raw.c:354
       __sock_release+0xcd/0x280 net/socket.c:596
       sock_close+0x18/0x20 net/socket.c:1277
       __fput+0x285/0x920 fs/file_table.c:281
       task_work_run+0xdd/0x190 kernel/task_work.c:141
       exit_task_work include/linux/task_work.h:25 [inline]
       do_exit+0xb7d/0x29f0 kernel/exit.c:806
       do_group_exit+0x125/0x310 kernel/exit.c:903
       __do_sys_exit_group kernel/exit.c:914 [inline]
       __se_sys_exit_group kernel/exit.c:912 [inline]
       __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:912
       do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
       entry_SYSCALL_64_after_hwframe+0x44/0xa9

other info that might help us debug this:

Chain exists of:
  pernet_ops_rwsem --> (delayed_fput_work).work --> &sb->s_type->i_mutex_key#12

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&sb->s_type->i_mutex_key#12);
                               lock((delayed_fput_work).work);
                               lock(&sb->s_type->i_mutex_key#12);
  lock(pernet_ops_rwsem);

 *** DEADLOCK ***

1 lock held by syz-executor985/6864:
 #0: ffff888085909c90 (&sb->s_type->i_mutex_key#12){+.+.}-{3:3}, at: inode_lock include/linux/fs.h:779 [inline]
 #0: ffff888085909c90 (&sb->s_type->i_mutex_key#12){+.+.}-{3:3}, at: __sock_release+0x86/0x280 net/socket.c:595

stack backtrace:
CPU: 0 PID: 6864 Comm: syz-executor985 Not tainted 5.9.0-rc3-next-20200903-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x198/0x1fd lib/dump_stack.c:118
 check_noncircular+0x324/0x3e0 kernel/locking/lockdep.c:1827
 check_prev_add kernel/locking/lockdep.c:2496 [inline]
 check_prevs_add kernel/locking/lockdep.c:2601 [inline]
 validate_chain kernel/locking/lockdep.c:3218 [inline]
 __lock_acquire+0x29bb/0x5570 kernel/locking/lockdep.c:4426
 lock_acquire+0x1f3/0xae0 kernel/locking/lockdep.c:5006
 down_write+0x8d/0x150 kernel/locking/rwsem.c:1531
 unregister_netdevice_notifier+0x1e/0x170 net/core/dev.c:1861
 raw_release+0x58/0x890 net/can/raw.c:354
 __sock_release+0xcd/0x280 net/socket.c:596
 sock_close+0x18/0x20 net/socket.c:1277
 __fput+0x285/0x920 fs/file_table.c:281
 task_work_run+0xdd/0x190 kernel/task_work.c:141
 exit_task_work include/linux/task_work.h:25 [inline]
 do_exit+0xb7d/0x29f0 kernel/exit.c:806
 do_group_exit+0x125/0x310 kernel/exit.c:903
 __do_sys_exit_group kernel/exit.c:914 [inline]
 __se_sys_exit_group kernel/exit.c:912 [inline]
 __x64_sys_exit_group+0x3a/0x50 kernel/exit.c:912
 do_syscall_64+0x2d/0x70 arch/x86/entry/common.c:46
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x43f9b8
Code: Bad RIP value.
RSP: 002b:00007ffe08a46098 EFLAGS: 00000246 ORIG_RAX: 00000000000000e7
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 000000000043f9b8
RDX: 0000000000000

Crashes (2727):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	VM info
ci-upstream-linux-next-kasan-gce-root	2020/09/07 21:40	linux-next	7a695657	abf9ba4f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/09/06 15:00	linux-next	7a695657	abf9ba4f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/31 09:48	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/30 14:20	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/30 13:00	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/30 12:39	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/30 12:00	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/29 01:53	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/28 23:59	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/28 22:52	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/28 21:23	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/28 13:16	linux-next	b36c9697	816e0689	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/08/29 11:28	linux-next	b36c9697	d5a3ae1f	.config	log	report	syz
ci-upstream-bpf-next-kasan-gce	2020/09/22 14:55	bpf-next	a8a71796	3e8f6c27	.config	log	report			info
ci-upstream-net-kasan-gce	2020/09/07 16:06	net-next	02a20d4f	abf9ba4f	.config	log	report
ci-upstream-net-kasan-gce	2020/08/27 20:48	net-next	50aba46c	816e0689	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 17:27	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 17:15	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 16:09	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 15:17	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 14:37	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 12:39	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 12:05	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 11:23	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 11:17	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 10:16	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 09:45	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 09:18	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 08:47	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 08:07	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 07:35	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 07:01	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 05:28	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 05:06	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 03:46	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 03:46	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 02:42	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 01:52	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/08 00:23	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 23:42	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 23:09	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 22:41	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 22:41	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 21:09	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 20:08	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 19:24	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 19:17	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 18:13	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 18:05	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 17:02	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 14:00	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 12:37	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 11:04	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 10:04	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 09:30	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 08:40	linux-next	7a695657	abf9ba4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/09/07 07:16	linux-next	7a695657	abf9ba4f	.config	log	report