possible deadlock in static_key_slow

possible deadlock in static_key_slow_dec
Status: upstream: reported C repro on 2018/11/07 01:38
Reported-by: syzbot+b011e55d1b4c015100d2@syzkaller.appspotmail.com
First crash: 1078d, last: 573d

Cause bisection: introduced by (bisect log) [merge commit]:
commit 7cd18c11799720bf077309be1b84b8395af1e2ff
Author: Stephen Rothwell <sfr@canb.auug.org.au>
Date: Wed Aug 14 04:46:44 2019 +0000

Merge remote-tracking branch 'tip/auto-latest'

Crash: possible deadlock in static_key_slow_dec (log)
Repro: C syz .config

Fix bisection: failed (bisect log)

duplicates (1):
Title	Repro	Cause bisect	Fix bisect	Count	Last	Reported	Patched	Status
possible deadlock in __static_key_slow_dec	C	done	error	2	590d	590d	0/22	closed as dup on 2021/01/20 05:22

similar bugs (3):
Kernel	Title	Count	Last	Reported	Patched	Status
linux-4.19	possible deadlock in static_key_slow_dec (3)	1	12d	12d	0/1	upstream: reported on 2021/10/04 21:46
linux-4.19	possible deadlock in static_key_slow_dec	1	777d	776d	0/1	auto-closed as invalid on 2019/12/30 04:27
linux-4.19	possible deadlock in static_key_slow_dec (2)	2	281d	365d	0/1	auto-closed as invalid on 2021/05/08 21:09

Sample crash report:

======================================================
WARNING: possible circular locking dependency detected
5.6.0-rc5-syzkaller #0 Not tainted
------------------------------------------------------
syz-executor754/10070 is trying to acquire lock:
ffffffff89768e70 (cpu_hotplug_lock.rw_sem){++++}, at: __static_key_slow_dec kernel/jump_label.c:254 [inline]
ffffffff89768e70 (cpu_hotplug_lock.rw_sem){++++}, at: static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:270

but task is already holding lock:
ffff88809011c7d8 (&mm->mmap_sem#2){++++}, at: vm_mmap_pgoff+0x152/0x200 mm/util.c:504

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (&mm->mmap_sem#2){++++}:
       down_write+0x8d/0x150 kernel/locking/rwsem.c:1534
       mpol_rebind_mm+0x20/0xc0 mm/mempolicy.c:382
       cpuset_attach+0x214/0x400 kernel/cgroup/cpuset.c:2203
       cgroup_migrate_execute+0xc34/0x1210 kernel/cgroup/cgroup.c:2445
       cgroup_attach_task+0x57c/0x8e0 kernel/cgroup/cgroup.c:2738
       __cgroup1_procs_write.constprop.0+0x3a7/0x490 kernel/cgroup/cgroup-v1.c:522
       cgroup_file_write+0x20d/0x710 kernel/cgroup/cgroup.c:3695
       kernfs_fop_write+0x268/0x490 fs/kernfs/file.c:315
       __vfs_write+0x76/0x100 fs/read_write.c:494
       vfs_write+0x262/0x5c0 fs/read_write.c:558
       ksys_write+0x127/0x250 fs/read_write.c:611
       do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #1 (&cpuset_rwsem){++++}:
       percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
       cpuset_read_lock+0x3b/0x130 kernel/cgroup/cpuset.c:340
       __sched_setscheduler.constprop.0+0x251/0x2000 kernel/sched/core.c:4869
       _sched_setscheduler+0xee/0x180 kernel/sched/core.c:5041
       __kthread_create_on_node+0x2ea/0x410 kernel/kthread.c:349
       kthread_create_on_node+0xbb/0xf0 kernel/kthread.c:388
       create_worker+0x23b/0x530 kernel/workqueue.c:1926
       workqueue_prepare_cpu+0x9c/0xf0 kernel/workqueue.c:5026
       cpuhp_invoke_callback+0x22c/0x1d20 kernel/cpu.c:172
       cpuhp_up_callbacks kernel/cpu.c:599 [inline]
       _cpu_up+0x27d/0x540 kernel/cpu.c:1165
       do_cpu_up kernel/cpu.c:1200 [inline]
       do_cpu_up+0xfe/0x1a0 kernel/cpu.c:1172
       smp_init+0x239/0x24e kernel/smp.c:604
       kernel_init_freeable+0x32e/0x5ae init/main.c:1438
       kernel_init+0xd/0x1bb init/main.c:1352
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

-> #0 (cpu_hotplug_lock.rw_sem){++++}:
       check_prev_add kernel/locking/lockdep.c:2475 [inline]
       check_prevs_add kernel/locking/lockdep.c:2580 [inline]
       validate_chain kernel/locking/lockdep.c:2970 [inline]
       __lock_acquire+0x201b/0x3ca0 kernel/locking/lockdep.c:3954
       lock_acquire+0x197/0x420 kernel/locking/lockdep.c:4484
       percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
       cpus_read_lock+0x3b/0x130 kernel/cpu.c:292
       __static_key_slow_dec kernel/jump_label.c:254 [inline]
       static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:270
       sw_perf_event_destroy+0x81/0x130 kernel/events/core.c:8840
       _free_event+0x33b/0x12b0 kernel/events/core.c:4616
       put_event+0x40/0x50 kernel/events/core.c:4710
       perf_mmap_close+0x4f7/0xcb0 kernel/events/core.c:5754
       remove_vma+0xa9/0x170 mm/mmap.c:177
       remove_vma_list mm/mmap.c:2568 [inline]
       __do_munmap+0x729/0x10e0 mm/mmap.c:2812
       do_munmap mm/mmap.c:2820 [inline]
       mmap_region+0x1ef/0x1540 mm/mmap.c:1713
       do_mmap+0x843/0x1140 mm/mmap.c:1543
       do_mmap_pgoff include/linux/mm.h:2334 [inline]
       vm_mmap_pgoff+0x197/0x200 mm/util.c:506
       ksys_mmap_pgoff+0x457/0x5b0 mm/mmap.c:1593
       do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

Chain exists of:
  cpu_hotplug_lock.rw_sem --> &cpuset_rwsem --> &mm->mmap_sem#2

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(&mm->mmap_sem#2);
                               lock(&cpuset_rwsem);
                               lock(&mm->mmap_sem#2);
  lock(cpu_hotplug_lock.rw_sem);

 *** DEADLOCK ***

1 lock held by syz-executor754/10070:
 #0: ffff88809011c7d8 (&mm->mmap_sem#2){++++}, at: vm_mmap_pgoff+0x152/0x200 mm/util.c:504

stack backtrace:
CPU: 1 PID: 10070 Comm: syz-executor754 Not tainted 5.6.0-rc5-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x188/0x20d lib/dump_stack.c:118
 check_noncircular+0x32e/0x3e0 kernel/locking/lockdep.c:1808
 check_prev_add kernel/locking/lockdep.c:2475 [inline]
 check_prevs_add kernel/locking/lockdep.c:2580 [inline]
 validate_chain kernel/locking/lockdep.c:2970 [inline]
 __lock_acquire+0x201b/0x3ca0 kernel/locking/lockdep.c:3954
 lock_acquire+0x197/0x420 kernel/locking/lockdep.c:4484
 percpu_down_read include/linux/percpu-rwsem.h:40 [inline]
 cpus_read_lock+0x3b/0x130 kernel/cpu.c:292
 __static_key_slow_dec kernel/jump_label.c:254 [inline]
 static_key_slow_dec+0x4f/0x90 kernel/jump_label.c:270
 sw_perf_event_destroy+0x81/0x130 kernel/events/core.c:8840
 _free_event+0x33b/0x12b0 kernel/events/core.c:4616
 put_event+0x40/0x50 kernel/events/core.c:4710
 perf_mmap_close+0x4f7/0xcb0 kernel/events/core.c:5754
 remove_vma+0xa9/0x170 mm/mmap.c:177
 remove_vma_list mm/mmap.c:2568 [inline]
 __do_munmap+0x729/0x10e0 mm/mmap.c:2812
 do_munmap mm/mmap.c:2820 [inline]
 mmap_region+0x1ef/0x1540 mm/mmap.c:1713
 do_mmap+0x843/0x1140 mm/mmap.c:1543
 do_mmap_pgoff include/linux/mm.h:2334 [inline]
 vm_mmap_pgoff+0x197/0x200 mm/util.c:506
 ksys_mmap_pgoff+0x457/0x5b0 mm/mmap.c:1593
 do_syscall_64+0xf6/0x7d0 arch/x86/entry/common.c:294
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x4472e9
Code: e8 2c bb 02 00 48 83 c4 18 c3 0f 1f 80 00 00 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 5b 07 fc ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f8789a02da8 EFLAGS: 00000246 ORIG_RAX: 0000000000000009
RAX: ffffffffffffffda RBX: 00000000006dcc38 RCX: 00000000004472e9
RDX: 0000000000000000 RSI: 0000000000003000 RDI: 0000000020ffd000
RBP: 00000000006dcc30 R08: 0000000000000004 R09: 0000000000000000
R10: 0000000000000011 R11: 0000000000000246 R12: 00000000006dcc3c
R13: 00007ffef74e260f R14: 00007f8789a039c0 R15: 0000000000000000

Crashes (35):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro
ci-upstream-kasan-gce-selinux-root	2020/03/13 06:06	upstream	3cc6e2c599cd	d850e9d0	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2020/03/10 14:59	upstream	30bb5572ce7a	35f53e45	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2020/03/23 04:57	linux-next	770fbb32d34e	78267cec	.config	log	report	syz	C
ci-upstream-linux-next-kasan-gce-root	2019/08/16 07:06	linux-next	17da61ae48ec	8fd428a1	.config	log	report	syz	C
ci-upstream-kasan-gce-root	2020/03/19 13:44	upstream	5076190daded	2c31c529	.config	log	report
ci-upstream-kasan-gce-selinux-root	2020/03/06 21:41	upstream	63623fd44972	c88c7b75	.config	log	report
ci-upstream-kasan-gce	2020/03/04 21:26	upstream	63623fd44972	c88c7b75	.config	log	report
ci-upstream-kasan-gce	2020/02/27 12:58	upstream	f8788d86ab28	59b57593	.config	log	report
ci-upstream-kasan-gce-selinux-root	2020/02/16 15:51	upstream	db70e26e33ee	cf914200	.config	log	report
ci-upstream-kasan-gce-selinux-root	2020/01/30 01:56	upstream	b3a608222336	5ed23f9a	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/12/20 15:33	upstream	7e0165b2f1a9	e30cbdae	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/10/21 20:20	upstream	7d194c2100ad	b24d2b8a	.config	log	report
ci-upstream-kasan-gce	2019/10/17 21:26	upstream	283ea345934d	8c88c9c1	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/09/24 15:34	upstream	4c07e2ddab5b	0942eab8	.config	log	report
ci-upstream-kasan-gce	2019/09/21 13:53	upstream	f97c81dc6ca5	d96e88f3	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/03/26 07:15	upstream	a3ac7917b730	55684ce1	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/03/25 19:48	upstream	8c2ffd917477	2c86e0a5	.config	log	report
ci-upstream-kasan-gce-smack-root	2019/03/06 07:32	upstream	63bdf4284c38	16559f86	.config	log	report
ci-upstream-kasan-gce-smack-root	2019/01/11 18:43	upstream	de6629eb262e	c3f3344c	.config	log	report
ci-upstream-kasan-gce-selinux-root	2019/01/06 11:46	upstream	b5aef86e089a	94f8adb5	.config	log	report
ci-upstream-kasan-gce-smack-root	2018/11/28 05:34	upstream	ef78e5ec9214	4b6d14f2	.config	log	report
ci-upstream-kasan-gce-root	2018/11/27 13:40	upstream	ef78e5ec9214	4b6d14f2	.config	log	report
ci-upstream-kasan-gce-selinux-root	2018/11/25 17:56	upstream	e195ca6cb6f2	3d3ec907	.config	log	report
ci-upstream-kasan-gce-root	2018/11/04 05:07	upstream	83650fd58a93	8bd6bd63	.config	log	report
ci-upstream-kasan-gce-386	2020/01/24 13:39	upstream	4703d9119972	2e95ab33	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/03/13 17:52	linux-next	770fbb32d34e	d850e9d0	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/02/09 01:14	linux-next	6dff1565d69c	06150bf1	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/01/12 14:42	linux-next	6c09d7dbb7d3	31290a45	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/11/24 12:19	linux-next	b9d3d0140506	598ca6c8	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/11/05 14:16	linux-next	51309b9d73f5	0f3ec414	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/10/05 11:40	linux-next	311ef88adfa3	f3f7d9c8	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/10/04 03:56	linux-next	2521ffab5375	fc17ba49	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/08/16 06:36	linux-next	17da61ae48ec	8fd428a1	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2019/02/02 03:08	linux-next	dc4c89997735	564f9a4f	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2018/11/04 23:08	linux-next	25e9471b6a27	8bd6bd63	.config	log	report