syzbot

 sign-in | mailing list | source | docs
🐞 Open [1429]
≡ Subsystems
🐞 Fixed [5832]
🐞 Invalid [14241]
⬇ Missing Backports [92]
≡ Crashes
Kernel Health Bugs/Month Bug Lifetimes Fuzzing
Total Repo Heatmap Subsystems Heatmap
💬 Send us feedback

possible deadlock in __synchronize_srcu

Status: closed as dup on 2017/10/31 13:20
Reported-by: syzbot+b8ff4d5c3fa77f2e2f0f9be34e6b2795ffc3c65e@syzkaller.appspotmail.com
First crash: 2647d, last: 2644d
▶ ▼ Duplicate of
Title Repro Cause bisect Fix bisect Count Last Reported
possible deadlock in process_one_work C 18291 2584d 2580d
▶ ▼ Similar bugs (1)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in __synchronize_srcu (2) block 1 4d14h 14h33m 0/28 upstream: reported on 2024/11/22 13:47

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
4.13.0-rc6-next-20170825+ #9 Not tainted
------------------------------------------------------
kworker/u4:6/3517 is trying to acquire lock:
 ((complete)&rcu.completion){+.+.}, at: [<ffffffff815b4075>] __synchronize_srcu+0x1b5/0x250 kernel/rcu/srcutree.c:898

but task is already holding lock:
 (slab_mutex){+.+.}, at: [<ffffffff8192f390>] kmem_cache_destroy+0x30/0x250 mm/slab_common.c:821

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (slab_mutex){+.+.}:
       check_prevs_add kernel/locking/lockdep.c:2020 [inline]
       validate_chain kernel/locking/lockdep.c:2469 [inline]
       __lock_acquire+0x3286/0x4620 kernel/locking/lockdep.c:3498
       lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0x16f/0x1870 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       kmem_cache_create+0x39/0x2a0 mm/slab_common.c:435
       ptlock_cache_init+0x24/0x2d mm/memory.c:4632
       pgtable_init include/linux/mm.h:1756 [inline]
       mm_init init/main.c:504 [inline]
       start_kernel+0x3d4/0x7ad init/main.c:569
       x86_64_start_reservations+0x2a/0x2c arch/x86/kernel/head64.c:381
       x86_64_start_kernel+0x13c/0x149 arch/x86/kernel/head64.c:362
       verify_cpu+0x0/0xfb

-> #2 (memcg_cache_ids_sem){.+.+}:
       check_prevs_add kernel/locking/lockdep.c:2020 [inline]
       validate_chain kernel/locking/lockdep.c:2469 [inline]
       __lock_acquire+0x3286/0x4620 kernel/locking/lockdep.c:3498
       lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002
       down_read+0x96/0x150 kernel/locking/rwsem.c:23
       memcg_get_cache_ids+0x10/0x20 mm/memcontrol.c:274
       list_lru_destroy+0x96/0x490 mm/list_lru.c:573
       deactivate_locked_super+0x94/0xd0 fs/super.c:315
       deactivate_super+0x141/0x1b0 fs/super.c:339
       cleanup_mnt+0xb2/0x150 fs/namespace.c:1113
       mntput_no_expire+0x6e0/0xa90 fs/namespace.c:1179
       mntput fs/namespace.c:1189 [inline]
       kern_unmount+0x9c/0xd0 fs/namespace.c:2934
       pid_ns_release_proc+0x37/0x50 fs/proc/root.c:231
       proc_cleanup_work+0x19/0x20 kernel/pid_namespace.c:79
       process_one_work+0xbfd/0x1be0 kernel/workqueue.c:2098
       worker_thread+0x223/0x1860 kernel/workqueue.c:2233
       kthread+0x39c/0x470 kernel/kthread.c:231
       ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431

-> #1 ((&ns->proc_work)){+.+.}:
       process_one_work+0xba5/0x1be0 kernel/workqueue.c:2095
       worker_thread+0x223/0x1860 kernel/workqueue.c:2233
       kthread+0x39c/0x470 kernel/kthread.c:231
       ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431
       0xffffffffffffffff

-> #0 ((complete)&rcu.completion){+.+.}:
       check_prev_add+0x865/0x1520 kernel/locking/lockdep.c:1894
       check_prevs_add kernel/locking/lockdep.c:2020 [inline]
       validate_chain kernel/locking/lockdep.c:2469 [inline]
       __lock_acquire+0x3286/0x4620 kernel/locking/lockdep.c:3498
       lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002
       complete_acquire include/linux/completion.h:39 [inline]
       __wait_for_common kernel/sched/completion.c:108 [inline]
       wait_for_common kernel/sched/completion.c:122 [inline]
       wait_for_completion+0xc8/0x770 kernel/sched/completion.c:143
       __synchronize_srcu+0x1b5/0x250 kernel/rcu/srcutree.c:898
       synchronize_srcu_expedited kernel/rcu/srcutree.c:923 [inline]
       synchronize_srcu+0x1a3/0x560 kernel/rcu/srcutree.c:974
       quarantine_remove_cache+0xd7/0xf0 mm/kasan/quarantine.c:327
       kasan_cache_shutdown+0x9/0x10 mm/kasan/kasan.c:381
       shutdown_cache+0x15/0x1b0 mm/slab_common.c:531
       kmem_cache_destroy+0x236/0x250 mm/slab_common.c:829
       tipc_server_stop+0x13f/0x190 net/tipc/server.c:636
       tipc_topsrv_stop+0x1fe/0x350 net/tipc/subscr.c:390
       tipc_exit_net+0x15/0x40 net/tipc/core.c:96
       ops_exit_list.isra.6+0xae/0x150 net/core/net_namespace.c:142
       cleanup_net+0x5c7/0xb60 net/core/net_namespace.c:483
       process_one_work+0xbfd/0x1be0 kernel/workqueue.c:2098
       worker_thread+0x223/0x1860 kernel/workqueue.c:2233
       kthread+0x39c/0x470 kernel/kthread.c:231
       ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431

other info that might help us debug this:

Chain exists of:
  (complete)&rcu.completion --> memcg_cache_ids_sem --> slab_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(slab_mutex);
                               lock(memcg_cache_ids_sem);
                               lock(slab_mutex);
  lock((complete)&rcu.completion);

 *** DEADLOCK ***

5 locks held by kworker/u4:6/3517:
 #0:  ("%s""netns"){.+.+}, at: [<ffffffff81468534>] __write_once_size include/linux/compiler.h:305 [inline]
 #0:  ("%s""netns"){.+.+}, at: [<ffffffff81468534>] atomic64_set arch/x86/include/asm/atomic64_64.h:33 [inline]
 #0:  ("%s""netns"){.+.+}, at: [<ffffffff81468534>] atomic_long_set include/asm-generic/atomic-long.h:56 [inline]
 #0:  ("%s""netns"){.+.+}, at: [<ffffffff81468534>] set_work_data kernel/workqueue.c:617 [inline]
 #0:  ("%s""netns"){.+.+}, at: [<ffffffff81468534>] set_work_pool_and_clear_pending kernel/workqueue.c:644 [inline]
 #0:  ("%s""netns"){.+.+}, at: [<ffffffff81468534>] process_one_work+0xad4/0x1be0 kernel/workqueue.c:2090
 #1:  (net_cleanup_work){+.+.}, at: [<ffffffff8146858c>] process_one_work+0xb2c/0x1be0 kernel/workqueue.c:2094
 #2:  (net_mutex){+.+.}, at: [<ffffffff83e57bb7>] cleanup_net+0x247/0xb60 net/core/net_namespace.c:449
 #3:  (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8192f382>] get_online_cpus include/linux/cpu.h:126 [inline]
 #3:  (cpu_hotplug_lock.rw_sem){++++}, at: [<ffffffff8192f382>] kmem_cache_destroy+0x22/0x250 mm/slab_common.c:818
 #4:  (slab_mutex){+.+.}, at: [<ffffffff8192f390>] kmem_cache_destroy+0x30/0x250 mm/slab_common.c:821

stack backtrace:
CPU: 0 PID: 3517 Comm: kworker/u4:6 Not tainted 4.13.0-rc6-next-20170825+ #9
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
 __dump_stack lib/dump_stack.c:16 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:52
 print_circular_bug+0x503/0x710 kernel/locking/lockdep.c:1259
 check_prev_add+0x865/0x1520 kernel/locking/lockdep.c:1894
 check_prevs_add kernel/locking/lockdep.c:2020 [inline]
 validate_chain kernel/locking/lockdep.c:2469 [inline]
 __lock_acquire+0x3286/0x4620 kernel/locking/lockdep.c:3498
 lock_acquire+0x1d5/0x580 kernel/locking/lockdep.c:4002
 complete_acquire include/linux/completion.h:39 [inline]
 __wait_for_common kernel/sched/completion.c:108 [inline]
 wait_for_common kernel/sched/completion.c:122 [inline]
 wait_for_completion+0xc8/0x770 kernel/sched/completion.c:143
 __synchronize_srcu+0x1b5/0x250 kernel/rcu/srcutree.c:898
 synchronize_srcu_expedited kernel/rcu/srcutree.c:923 [inline]
 synchronize_srcu+0x1a3/0x560 kernel/rcu/srcutree.c:974
 quarantine_remove_cache+0xd7/0xf0 mm/kasan/quarantine.c:327
 kasan_cache_shutdown+0x9/0x10 mm/kasan/kasan.c:381
 shutdown_cache+0x15/0x1b0 mm/slab_common.c:531
 kmem_cache_destroy+0x236/0x250 mm/slab_common.c:829
 tipc_server_stop+0x13f/0x190 net/tipc/server.c:636
 tipc_topsrv_stop+0x1fe/0x350 net/tipc/subscr.c:390
 tipc_exit_net+0x15/0x40 net/tipc/core.c:96
 ops_exit_list.isra.6+0xae/0x150 net/core/net_namespace.c:142
 cleanup_net+0x5c7/0xb60 net/core/net_namespace.c:483
 process_one_work+0xbfd/0x1be0 kernel/workqueue.c:2098
 worker_thread+0x223/0x1860 kernel/workqueue.c:2233
 kthread+0x39c/0x470 kernel/kthread.c:231
 ret_from_fork+0x2a/0x40 arch/x86/entry/entry_64.S:431
TCP: tcp_parse_options: Illegal window scaling value 243 > 14 received
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=3 sclass=netlink_route_socket pig=6381 comm=syz-executor0
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=netlink_route_socket pig=6389 comm=syz-executor0
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=51 sclass=netlink_route_socket pig=6389 comm=syz-executor0
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=netlink_route_socket pig=6389 comm=syz-executor0
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=3 sclass=netlink_route_socket pig=6398 comm=syz-executor0
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=netlink_route_socket pig=6398 comm=syz-executor0
netlink: 5 bytes leftover after parsing attributes in process `syz-executor5'.
IPv6: NLM_F_REPLACE set, but no existing node found!
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=51 sclass=netlink_route_socket pig=6403 comm=syz-executor0
SELinux: unrecognized netlink message: protocol=0 nlmsg_type=0 sclass=netlink_route_socket pig=6403 comm=syz-executor0
netlink: 5 bytes leftover after parsing attributes in process `syz-executor5'.
IPv6: NLM_F_REPLACE set, but no existing node found!
netlink: 11 bytes leftover after parsing attributes in process `syz-executor5'.
netlink: 11 bytes leftover after parsing attributes in process `syz-executor5'.
audit: type=1326 audit(1503891590.907:13): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=6559 comm="syz-executor5" exe="/root/syz-executor5" sig=9 arch=c000003e syscall=202 compat=0 ip=0x4512e9 code=0x0
audit: type=1326 audit(1503891591.063:14): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=6559 comm="syz-executor5" exe="/root/syz-executor5" sig=9 arch=c000003e syscall=202 compat=0 ip=0x4512e9 code=0x0
QAT: Invalid ioctl
QAT: Invalid ioctl
netlink: 1 bytes leftover after parsing attributes in process `syz-executor6'.
QAT: Invalid ioctl
QAT: Invalid ioctl
netlink: 1 bytes leftover after parsing attributes in process `syz-executor6'.
audit: type=1326 audit(1503891591.521:15): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=6680 comm="syz-executor2" exe="/root/syz-executor2" sig=31 arch=c000003e syscall=202 compat=0 ip=0x4512e9 code=0xffff0000
netlink: 21 bytes leftover after parsing attributes in process `syz-executor2'.
netlink: 8 bytes leftover after parsing attributes in process `syz-executor0'.
netlink: 8 bytes leftover after parsing attributes in process `syz-executor0'.
netlink: 21 bytes leftover after parsing attributes in process `syz-executor2'.
QAT: Invalid ioctl
audit: type=1326 audit(1503891591.587:16): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=kernel pid=6680 comm="syz-executor2" exe="/root/syz-executor2" sig=31 arch=c000003e syscall=202 compat=0 ip=0x4512e9 code=0xffff0000
==================================================================
BUG: KASAN: use-after-free in hlist_add_head include/linux/list.h:673 [inline]
BUG: KASAN: use-after-free in enqueue_timer kernel/time/timer.c:519 [inline]
BUG: KASAN: use-after-free in __internal_add_timer+0x275/0x2d0 kernel/time/timer.c:530
Write of size 8 at addr ffff8801d468b688 by task syz-executor0/6757

CPU: 1 PID: 6757 Comm: syz-executor0 Not tainted 4.13.0-rc6-next-20170825+ #9
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:16 [inline]
 dump_stack+0x194/0x257 lib/dump_stack.c:52
 print_address_description+0x73/0x250 mm/kasan/report.c:252
 kasan_report_error mm/kasan/report.c:351 [inline]
 kasan_report+0x24e/0x340 mm/kasan/report.c:409
 __asan_report_store8_noabort+0x17/0x20 mm/kasan/report.c:435
 hlist_add_head include/linux/list.h:673 [inline]
 enqueue_timer kernel/time/timer.c:519 [inline]
 __internal_add_timer+0x275/0x2d0 kernel/time/timer.c:530
 internal_add_timer kernel/time/timer.c:572 [inline]
 __mod_timer kernel/time/timer.c:1012 [inline]
 mod_timer+0x6e8/0xf40 kernel/time/timer.c:1059
 tun_flow_init drivers/net/tun.c:1098 [inline]
 tun_set_iff drivers/net/tun.c:2044 [inline]
 __tun_chr_ioctl+0x1b2c/0x3d40 drivers/net/tun.c:2262
 tun_chr_ioctl+0x2a/0x40 drivers/net/tun.c:2507
 vfs_ioctl fs/ioctl.c:45 [inline]
 do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:685
 SYSC_ioctl fs/ioctl.c:700 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
 entry_SYSCALL_64_fastpath+0x1f/0xbe
RIP: 0033:0x4512e9
RSP: 002b:00007fa68df1dc08 EFLAGS: 00000216 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000718000 RCX: 00000000004512e9
RDX: 0000000020511fd8 RSI: 00000000400454ca RDI: 0000000000000022
RBP: 0000000000003ca0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000216 R12: 00000000004b70fc
R13: 00000000ffffffff R14: 0000000020000000 R15: 0000000000fff000

Allocated by task 6757:
 save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
 save_stack+0x43/0xd0 mm/kasan/kasan.c:447
 set_track mm/kasan/kasan.c:459 [inline]
 kasan_kmalloc+0xad/0xe0 mm/kasan/kasan.c:551
 __do_kmalloc_node mm/slab.c:3689 [inline]
 __kmalloc_node+0x47/0x70 mm/slab.c:3696
 kmalloc_node include/linux/slab.h:535 [inline]
 kvmalloc_node+0x64/0xd0 mm/util.c:397
 kvmalloc include/linux/mm.h:529 [inline]
 kvzalloc include/linux/mm.h:537 [inline]
 alloc_netdev_mqs+0x16e/0xed0 net/core/dev.c:8009
 tun_set_iff drivers/net/tun.c:2008 [inline]
 __tun_chr_ioctl+0x12be/0x3d40 drivers/net/tun.c:2262
 tun_chr_ioctl+0x2a/0x40 drivers/net/tun.c:2507
 vfs_ioctl fs/ioctl.c:45 [inline]
 do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:685
 SYSC_ioctl fs/ioctl.c:700 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
 entry_SYSCALL_64_fastpath+0x1f/0xbe

Freed by task 6757:
 save_stack_trace+0x16/0x20 arch/x86/kernel/stacktrace.c:59
 save_stack+0x43/0xd0 mm/kasan/kasan.c:447
 set_track mm/kasan/kasan.c:459 [inline]
 kasan_slab_free+0x71/0xc0 mm/kasan/kasan.c:524
 __cache_free mm/slab.c:3503 [inline]
 kfree+0xca/0x250 mm/slab.c:3820
 kvfree+0x36/0x60 mm/util.c:416
 netdev_freemem net/core/dev.c:7961 [inline]
 free_netdev+0x2cf/0x360 net/core/dev.c:8123
 tun_set_iff drivers/net/tun.c:2091 [inline]
 __tun_chr_ioctl+0x2cff/0x3d40 drivers/net/tun.c:2262
 tun_chr_ioctl+0x2a/0x40 drivers/net/tun.c:2507
 vfs_ioctl fs/ioctl.c:45 [inline]
 do_vfs_ioctl+0x1b1/0x1530 fs/ioctl.c:685
 SYSC_ioctl fs/ioctl.c:700 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:691
 entry_SYSCALL_64_fastpath+0x1f/0xbe

The buggy address belongs to the object at ffff8801d4688280
 which belongs to the cache kmalloc-16384 of size 16384
The buggy address is located 13320 bytes inside of
 16384-byte region [ffff8801d4688280, ffff8801d468c280)
The buggy address belongs to the page:
page:ffffea000751a200 count:1 mapcount:0 mapping:ffff8801d4688280 index:0x0 compound_mapcount: 0
flags: 0x200000000008100(slab|head)
raw: 0200000000008100 ffff8801d4688280 0000000000000000 0000000100000001
raw: ffffea00072b7a20 ffffea0007234220 ffff8801dac02200 0000000000000000
page dumped because: kasan: bad access detected

Memory state around the buggy address:
 ffff8801d468b580: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff8801d468b600: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff8801d468b680: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                      ^
 ffff8801d468b700: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff8801d468b780: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================

Crashes (6):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2017/08/28 03:39 linux-next 7159188b70e3 4074aed7 .config console log report ci-upstream-next-kasan-gce
2017/08/27 05:54 linux-next 7159188b70e3 4074aed7 .config console log report ci-upstream-next-kasan-gce
2017/08/26 21:25 linux-next 7159188b70e3 4074aed7 .config console log report ci-upstream-next-kasan-gce
2017/08/25 11:24 linux-next 7159188b70e3 4074aed7 .config console log report skylake-linux-next-kasan-qemu
2017/08/24 21:05 linux-next 9506597de2cd 3f1aca48 .config console log report ci-upstream-next-kasan-gce
2017/08/24 19:23 linux-next 9506597de2cd 3f1aca48 .config console log report ci-upstream-next-kasan-gce
* Struck through repros no longer work on HEAD.