possible deadlock in rtnl

possible deadlock in rtnl_lock (6)
Status: upstream: reported C repro on 2019/07/08 19:37
Reported-by: syzbot+174ce29c2308dec5bc68@syzkaller.appspotmail.com
First crash: 228d, last: 212d

Cause bisection: introduced by (bisect log):

commit 455302d1c9ae9318660aaeb9748a01ff414c9741
Author: Ilya Maximets <i.maximets@samsung.com>
Date: Fri Jun 28 08:04:07 2019 +0000

xdp: fix hang while unregistering device bound to xdp socket

Crash: possible deadlock in xsk_notifier (log)
Repro: C syz .config

Fix bisection: failed (bisect log)

similar bugs (9):
Kernel	Title	Repro	Count	Last	Reported	Patched	Status
android-44	possible deadlock in rtnl_lock	C	4676	118d	311d	0/2	public: reported C repro on 2019/04/12 00:00
upstream	possible deadlock in rtnl_lock (5)	C	1009	663d	691d	6/16	fixed on 2018/05/08 18:30
upstream	possible deadlock in rtnl_lock (3)	C	3633	739d	740d	4/16	fixed on 2018/02/07 13:48
linux-4.19	possible deadlock in rtnl_lock	C	31	2d05h	42d	0/1	upstream: reported C repro on 2020/01/06 01:18
upstream	possible deadlock in rtnl_lock (4)	C	73333	722d	739d	4/16	fixed on 2018/02/26 20:04
android-49	possible deadlock in rtnl_lock	C	2906	541d	692d	0/3	closed as invalid on 2019/04/05 05:43
upstream	possible deadlock in rtnl_lock	C	15711	746d	802d	4/16	fixed on 2018/02/01 04:00
upstream	possible deadlock in rtnl_lock (2)	C	10369	742d	745d	4/16	fixed on 2018/02/04 23:45
linux-4.14	possible deadlock in rtnl_lock	C	218	8h18m	42d	0/1	upstream: reported C repro on 2020/01/06 01:45

Sample crash report:

======================================================
WARNING: possible circular locking dependency detected
5.2.0-rc6+ #76 Not tainted
------------------------------------------------------
syz-executor613/9114 is trying to acquire lock:
000000002c564901 (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72

but task is already holding lock:
0000000039d6ee9b (&xs->mutex){+.+.}, at: xsk_bind+0x16a/0xe70 net/xdp/xsk.c:422

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&xs->mutex){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:926 [inline]
       __mutex_lock+0xf7/0x1310 kernel/locking/mutex.c:1073
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
       xsk_notifier+0x149/0x290 net/xdp/xsk.c:730
       notifier_call_chain+0xc2/0x230 kernel/notifier.c:95
       __raw_notifier_call_chain kernel/notifier.c:396 [inline]
       raw_notifier_call_chain+0x2e/0x40 kernel/notifier.c:403
       call_netdevice_notifiers_info+0x3f/0x90 net/core/dev.c:1749
       call_netdevice_notifiers_extack net/core/dev.c:1761 [inline]
       call_netdevice_notifiers net/core/dev.c:1775 [inline]
       rollback_registered_many+0x9b9/0xfc0 net/core/dev.c:8206
       rollback_registered+0x109/0x1d0 net/core/dev.c:8248
       unregister_netdevice_queue net/core/dev.c:9295 [inline]
       unregister_netdevice_queue+0x1ee/0x2c0 net/core/dev.c:9288
       br_dev_delete+0x145/0x1a0 net/bridge/br_if.c:383
       br_del_bridge+0xd7/0x120 net/bridge/br_if.c:483
       br_ioctl_deviceless_stub+0x2a4/0x7b0 net/bridge/br_ioctl.c:376
       sock_ioctl+0x44b/0x780 net/socket.c:1141
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:509 [inline]
       do_vfs_ioctl+0xd5f/0x1380 fs/ioctl.c:696
       ksys_ioctl+0xab/0xd0 fs/ioctl.c:713
       __do_sys_ioctl fs/ioctl.c:720 [inline]
       __se_sys_ioctl fs/ioctl.c:718 [inline]
       __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:718
       do_syscall_64+0xfd/0x680 arch/x86/entry/common.c:301
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&net->xdp.lock){+.+.}:
       __mutex_lock_common kernel/locking/mutex.c:926 [inline]
       __mutex_lock+0xf7/0x1310 kernel/locking/mutex.c:1073
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
       xsk_notifier+0xa7/0x290 net/xdp/xsk.c:726
       notifier_call_chain+0xc2/0x230 kernel/notifier.c:95
       __raw_notifier_call_chain kernel/notifier.c:396 [inline]
       raw_notifier_call_chain+0x2e/0x40 kernel/notifier.c:403
       call_netdevice_notifiers_info+0x3f/0x90 net/core/dev.c:1749
       call_netdevice_notifiers_extack net/core/dev.c:1761 [inline]
       call_netdevice_notifiers net/core/dev.c:1775 [inline]
       rollback_registered_many+0x9b9/0xfc0 net/core/dev.c:8206
       rollback_registered+0x109/0x1d0 net/core/dev.c:8248
       unregister_netdevice_queue net/core/dev.c:9295 [inline]
       unregister_netdevice_queue+0x1ee/0x2c0 net/core/dev.c:9288
       br_dev_delete+0x145/0x1a0 net/bridge/br_if.c:383
       br_del_bridge+0xd7/0x120 net/bridge/br_if.c:483
       br_ioctl_deviceless_stub+0x2a4/0x7b0 net/bridge/br_ioctl.c:376
       sock_ioctl+0x44b/0x780 net/socket.c:1141
       vfs_ioctl fs/ioctl.c:46 [inline]
       file_ioctl fs/ioctl.c:509 [inline]
       do_vfs_ioctl+0xd5f/0x1380 fs/ioctl.c:696
       ksys_ioctl+0xab/0xd0 fs/ioctl.c:713
       __do_sys_ioctl fs/ioctl.c:720 [inline]
       __se_sys_ioctl fs/ioctl.c:718 [inline]
       __x64_sys_ioctl+0x73/0xb0 fs/ioctl.c:718
       do_syscall_64+0xfd/0x680 arch/x86/entry/common.c:301
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (rtnl_mutex){+.+.}:
       lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4303
       __mutex_lock_common kernel/locking/mutex.c:926 [inline]
       __mutex_lock+0xf7/0x1310 kernel/locking/mutex.c:1073
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
       rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72
       xdp_umem_assign_dev+0xbe/0x8b0 net/xdp/xdp_umem.c:96
       xsk_bind+0x4d7/0xe70 net/xdp/xsk.c:488
       __sys_bind+0x239/0x290 net/socket.c:1653
       __do_sys_bind net/socket.c:1664 [inline]
       __se_sys_bind net/socket.c:1662 [inline]
       __x64_sys_bind+0x73/0xb0 net/socket.c:1662
       do_syscall_64+0xfd/0x680 arch/x86/entry/common.c:301
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
  rtnl_mutex --> &net->xdp.lock --> &xs->mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&xs->mutex);
                               lock(&net->xdp.lock);
                               lock(&xs->mutex);
  lock(rtnl_mutex);

 *** DEADLOCK ***

1 lock held by syz-executor613/9114:
 #0: 0000000039d6ee9b (&xs->mutex){+.+.}, at: xsk_bind+0x16a/0xe70 net/xdp/xsk.c:422

stack backtrace:
CPU: 1 PID: 9114 Comm: syz-executor613 Not tainted 5.2.0-rc6+ #76
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 lib/dump_stack.c:113
 print_circular_bug.cold+0x1cc/0x28f kernel/locking/lockdep.c:1565
 check_prev_add kernel/locking/lockdep.c:2310 [inline]
 check_prevs_add kernel/locking/lockdep.c:2418 [inline]
 validate_chain kernel/locking/lockdep.c:2800 [inline]
 __lock_acquire+0x3755/0x5490 kernel/locking/lockdep.c:3793
 lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4303
 __mutex_lock_common kernel/locking/mutex.c:926 [inline]
 __mutex_lock+0xf7/0x1310 kernel/locking/mutex.c:1073
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1088
 rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72
 xdp_umem_assign_dev+0xbe/0x8b0 net/xdp/xdp_umem.c:96
 xsk_bind+0x4d7/0xe70 net/xdp/xsk.c:488
 __sys_bind+0x239/0x290 net/socket.c:1653
 __do_sys_bind net/socket.c:1664 [inline]
 __se_sys_bind net/socket.c:1662 [inline]
 __x64_sys_bind+0x73/0xb0 net/socket.c:1662
 do_syscall_64+0xfd/0x680 arch/x86/entry/common.c:301
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x447909
Code: e8 cc e7 ff ff 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 3b 08 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007fb2a478fd98 EFLAGS: 00000246 ORIG_RAX: 0000000000000031
RAX: ffffffffffffffda RBX: 00000000006dcc58 RCX: 0000000000447909
RDX: 0000000000000010 RSI: 0000000020000040 RDI: 0000000000000005
RBP: 00000000006dcc50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000006dcc5c
R13: 0000003066736362 R14: 0000000000000000 R15: 0000003066736362

Crashes (417):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro	Maintainers
ci-upstream-net-this-kasan-gce	2019/07/07 11:38	net	537de0c8	f62e1e85	.config	log	report	syz	C	ast@kernel.org, bpf@vger.kernel.org, christian@brauner.io, daniel@iogearbox.net, davem@davemloft.net, dsahern@gmail.com, edumazet@google.com, hawk@kernel.org, idosch@mellanox.com, jakub.kicinski@netronome.com, johannes.berg@intel.com, john.fastabend@gmail.com, kafai@fb.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, petrm@mellanox.com, roopa@cumulusnetworks.com, songliubraving@fb.com, xdp-newbies@vger.kernel.org, yhs@fb.com
ci-upstream-linux-next-kasan-gce-root	2019/07/10 12:20	linux-next	4608a726	f62e1e85	.config	log	report	syz	C	ast@kernel.org, bpf@vger.kernel.org, christian@brauner.io, daniel@iogearbox.net, davem@davemloft.net, dsahern@gmail.com, edumazet@google.com, hawk@kernel.org, idosch@mellanox.com, jakub.kicinski@netronome.com, johannes.berg@intel.com, john.fastabend@gmail.com, kafai@fb.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, petrm@mellanox.com, roopa@cumulusnetworks.com, songliubraving@fb.com, xdp-newbies@vger.kernel.org, yhs@fb.com
ci-upstream-kasan-gce-smack-root	2019/07/20 05:21	upstream	3bfe1fc4	1656845f	.config	log	report
ci-upstream-kasan-gce-smack-root	2019/07/19 12:34	upstream	3bfe1fc4	1656845f	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/07/17 11:23	upstream	3eb51486	0d10349c	.config	log	report			ast@kernel.org, bpf@vger.kernel.org, christian@brauner.io, daniel@iogearbox.net, davem@davemloft.net, dsahern@gmail.com, hawk@kernel.org, idosch@mellanox.com, jakub.kicinski@netronome.com, jiri@mellanox.com, johannes.berg@intel.com, john.fastabend@gmail.com, kafai@fb.com, linux-kernel@vger.kernel.org, mkubecek@suse.cz, netdev@vger.kernel.org, petrm@mellanox.com, roopa@cumulusnetworks.com, songliubraving@fb.com, xdp-newbies@vger.kernel.org, yhs@fb.com
ci-upstream-net-this-kasan-gce	2019/07/19 04:02	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/19 03:01	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/19 03:00	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/18 22:44	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/18 14:45	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/18 14:32	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/18 09:05	net	49d05fe2	7bb222f7	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/17 23:33	net	a5b64700	f613a7c4	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/17 22:26	net	a5b64700	f613a7c4	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/17 21:00	net	a5b64700	f613a7c4	.config	log	report
ci-upstream-net-this-kasan-gce	2019/07/04 08:14	net	9d1bc24b	55565fa0	.config	log	report			ast@kernel.org, bpf@vger.kernel.org, christian@brauner.io, daniel@iogearbox.net, davem@davemloft.net, dsahern@gmail.com, edumazet@google.com, hawk@kernel.org, idosch@mellanox.com, jakub.kicinski@netronome.com, johannes.berg@intel.com, john.fastabend@gmail.com, kafai@fb.com, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, petrm@mellanox.com, roopa@cumulusnetworks.com, songliubraving@fb.com, xdp-newbies@vger.kernel.org, yhs@fb.com
ci-upstream-net-kasan-gce	2019/07/19 20:33	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 19:18	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 18:14	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 17:09	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 16:04	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 15:01	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 12:14	net-next	192f0f8e	1656845f	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 11:09	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 10:35	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 09:33	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 09:14	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 07:54	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 07:12	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 06:04	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 05:05	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 02:00	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 01:15	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 00:14	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/19 00:13	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 23:08	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 21:26	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 21:13	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 20:11	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 19:56	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 18:49	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 18:18	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 16:55	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 15:49	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 13:29	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 11:50	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 10:42	net-next	192f0f8e	7bb222f7	.config	log	report
ci-upstream-net-kasan-gce	2019/07/18 07:49	net-next	192f0f8e	f613a7c4	.config	log	report
ci-upstream-net-kasan-gce	2019/07/17 19:35	net-next	192f0f8e	f613a7c4	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/07/14 15:18	linux-next	2c51253c	2bbe2f05	.config	log	report			ast@kernel.org, bpf@vger.kernel.org, christian@brauner.io, daniel@iogearbox.net, davem@davemloft.net, dsahern@gmail.com, edumazet@google.com, hawk@kernel.org, idosch@mellanox.com, jakub.kicinski@netronome.com, jiri@mellanox.com, johannes.berg@intel.com, john.fastabend@gmail.com, kafai@fb.com, linux-kernel@vger.kernel.org, mkubecek@suse.cz, netdev@vger.kernel.org, petrm@mellanox.com, roopa@cumulusnetworks.com, songliubraving@fb.com, xdp-newbies@vger.kernel.org, yhs@fb.com