syzbot


possible deadlock in pie_timer

Status: fixed on 2019/10/15 23:40
Subsystems: net
[Documentation on labels]
Fix commit: e3ae1f96accd net: sched: sch_sfb: don't call qdisc_put() while holding tree lock
First crash: 1722d, last: 1721d
Cause bisection: the cause commit could be any of (bisect log):
  9211bfbff80a netfilter: add missing IS_ENABLED(CONFIG_BRIDGE_NETFILTER) checks to header-file.
  47e640af2e49 netfilter: add missing IS_ENABLED(CONFIG_NF_TABLES) check to header-file.
  a1b2f04ea527 netfilter: add missing includes to a number of header-files.
  0abc8bf4f284 netfilter: add missing IS_ENABLED(CONFIG_NF_CONNTRACK) checks to some header-files.
  bd96b4c75675 netfilter: inline four headers files into another one.
  43dd16efc7f2 netfilter: nf_tables: store data in offload context registers
  78458e3e08cd netfilter: add missing IS_ENABLED(CONFIG_NETFILTER) checks to some header-files.
  20a9379d9a03 netfilter: remove "#ifdef __KERNEL__" guards from some headers.
  bd8699e9e292 netfilter: nft_bitwise: add offload support
  2a475c409fe8 kbuild: remove all netfilter headers from header-test blacklist.
  7e59b3fea2a2 netfilter: remove unnecessary spaces
  1b90af292e71 ipvs: Improve robustness to the ipvs sysctl
  5785cf15fd74 netfilter: nf_tables: add missing prototypes.
  0a30ba509fde netfilter: nf_nat_proto: make tables static
  e84fb4b3666d netfilter: conntrack: use shared sysctl constants
  105333435b4f netfilter: connlabels: prefer static lock initialiser
  8c0bb7873815 netfilter: synproxy: rename mss synproxy_options field
  c162610c7db2 Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next
  
Similar bugs (4)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream possible deadlock in pie_timer (2) net 2 358d 358d 23/27 fixed on 2023/09/28 17:51
linux-6.1 possible deadlock in pie_timer 1 318d 318d 0/3 auto-obsoleted due to no activity on 2023/11/10 02:26
linux-4.19 possible deadlock in pie_timer C error 1 862d 862d 0/1 upstream: reported C repro on 2022/02/03 09:38
linux-4.14 possible deadlock in pie_timer C inconclusive 3 852d 1713d 0/1 upstream: reported C repro on 2019/10/06 16:17

Sample crash report:
======================================================
WARNING: possible circular locking dependency detected
5.3.0+ #0 Not tainted
------------------------------------------------------
sshd/8762 is trying to acquire lock:
ffff888098218200 (&(&sch->q.lock)->rlock){+.-.}, at: spin_lock include/linux/spinlock.h:338 [inline]
ffff888098218200 (&(&sch->q.lock)->rlock){+.-.}, at: pie_timer+0x9e/0x830 net/sched/sch_pie.c:449

but task is already holding lock:
ffff8880ae809da0 ((&q->adapt_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:172 [inline]
ffff8880ae809da0 ((&q->adapt_timer)){+.-.}, at: call_timer_fn+0xe0/0x780 kernel/time/timer.c:1394

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 ((&q->adapt_timer)){+.-.}:
       del_timer_sync+0xb0/0x2a0 kernel/time/timer.c:1354
       pie_destroy+0x47/0x60 net/sched/sch_pie.c:554
       qdisc_destroy+0x11f/0x630 net/sched/sch_generic.c:968
       qdisc_put+0x85/0xa0 net/sched/sch_generic.c:995
       sfb_change+0x3d8/0xe90 net/sched/sch_sfb.c:522
       qdisc_change net/sched/sch_api.c:1321 [inline]
       tc_modify_qdisc+0xfcf/0x1c50 net/sched/sch_api.c:1623
       rtnetlink_rcv_msg+0x463/0xb00 net/core/rtnetlink.c:5223
       netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
       rtnetlink_rcv+0x1d/0x30 net/core/rtnetlink.c:5241
       netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
       netlink_unicast+0x531/0x710 net/netlink/af_netlink.c:1328
       netlink_sendmsg+0x8a5/0xd60 net/netlink/af_netlink.c:1917
       sock_sendmsg_nosec net/socket.c:637 [inline]
       sock_sendmsg+0xd7/0x130 net/socket.c:657
       ___sys_sendmsg+0x803/0x920 net/socket.c:2311
       __sys_sendmsg+0x105/0x1d0 net/socket.c:2356
       __do_sys_sendmsg net/socket.c:2365 [inline]
       __se_sys_sendmsg net/socket.c:2363 [inline]
       __x64_sys_sendmsg+0x78/0xb0 net/socket.c:2363
       do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

-> #0 (&(&sch->q.lock)->rlock){+.-.}:
       check_prev_add kernel/locking/lockdep.c:2476 [inline]
       check_prevs_add kernel/locking/lockdep.c:2581 [inline]
       validate_chain kernel/locking/lockdep.c:2971 [inline]
       __lock_acquire+0x2596/0x4a00 kernel/locking/lockdep.c:3955
       lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
       spin_lock include/linux/spinlock.h:338 [inline]
       pie_timer+0x9e/0x830 net/sched/sch_pie.c:449
       call_timer_fn+0x1ac/0x780 kernel/time/timer.c:1404
       expire_timers kernel/time/timer.c:1449 [inline]
       __run_timers kernel/time/timer.c:1773 [inline]
       __run_timers kernel/time/timer.c:1740 [inline]
       run_timer_softirq+0x6c3/0x1790 kernel/time/timer.c:1786
       __do_softirq+0x262/0x98c kernel/softirq.c:292
       do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1082
       do_softirq.part.0+0x11a/0x170 kernel/softirq.c:337
       do_softirq kernel/softirq.c:329 [inline]
       __local_bh_enable_ip+0x211/0x270 kernel/softirq.c:189
       local_bh_enable include/linux/bottom_half.h:32 [inline]
       rcu_read_unlock_bh include/linux/rcupdate.h:688 [inline]
       ip_finish_output2+0x8f8/0x2590 net/ipv4/ip_output.c:229
       __ip_finish_output net/ipv4/ip_output.c:308 [inline]
       __ip_finish_output+0x5fc/0xbc0 net/ipv4/ip_output.c:290
       ip_finish_output+0x38/0x1f0 net/ipv4/ip_output.c:318
       NF_HOOK_COND include/linux/netfilter.h:294 [inline]
       ip_output+0x21f/0x670 net/ipv4/ip_output.c:432
       dst_output include/net/dst.h:436 [inline]
       ip_local_out+0xbb/0x1b0 net/ipv4/ip_output.c:125
       __ip_queue_xmit+0x86f/0x1c00 net/ipv4/ip_output.c:532
       ip_queue_xmit+0x5a/0x70 include/net/ip.h:237
       __tcp_transmit_skb+0x1a6b/0x3820 net/ipv4/tcp_output.c:1169
       tcp_transmit_skb net/ipv4/tcp_output.c:1185 [inline]
       tcp_write_xmit+0xf0f/0x5910 net/ipv4/tcp_output.c:2440
       __tcp_push_pending_frames+0xb4/0x350 net/ipv4/tcp_output.c:2616
       tcp_push+0x488/0x700 net/ipv4/tcp.c:724
       tcp_sendmsg_locked+0x2876/0x3220 net/ipv4/tcp.c:1403
       tcp_sendmsg+0x30/0x50 net/ipv4/tcp.c:1434
       inet_sendmsg+0x9e/0xe0 net/ipv4/af_inet.c:807
       sock_sendmsg_nosec net/socket.c:637 [inline]
       sock_sendmsg+0xd7/0x130 net/socket.c:657
       sock_write_iter+0x27c/0x3e0 net/socket.c:989
       call_write_iter include/linux/fs.h:1895 [inline]
       new_sync_write+0x4d3/0x770 fs/read_write.c:483
       __vfs_write+0xe1/0x110 fs/read_write.c:496
       vfs_write+0x268/0x5d0 fs/read_write.c:558
       ksys_write+0x14f/0x290 fs/read_write.c:611
       __do_sys_write fs/read_write.c:623 [inline]
       __se_sys_write fs/read_write.c:620 [inline]
       __x64_sys_write+0x73/0xb0 fs/read_write.c:620
       do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
       entry_SYSCALL_64_after_hwframe+0x49/0xbe

other info that might help us debug this:

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock((&q->adapt_timer));
                               lock(&(&sch->q.lock)->rlock);
                               lock((&q->adapt_timer));
  lock(&(&sch->q.lock)->rlock);

 *** DEADLOCK ***

3 locks held by sshd/8762:
 #0: ffff88808a7dc150 (sk_lock-AF_INET){+.+.}, at: lock_sock include/net/sock.h:1522 [inline]
 #0: ffff88808a7dc150 (sk_lock-AF_INET){+.+.}, at: tcp_sendmsg+0x22/0x50 net/ipv4/tcp.c:1433
 #1: ffffffff88faacc0 (rcu_read_lock){....}, at: sock_net include/net/sock.h:2454 [inline]
 #1: ffffffff88faacc0 (rcu_read_lock){....}, at: __ip_queue_xmit+0x42/0x1c00 net/ipv4/ip_output.c:457
 #2: ffff8880ae809da0 ((&q->adapt_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:172 [inline]
 #2: ffff8880ae809da0 ((&q->adapt_timer)){+.-.}, at: call_timer_fn+0xe0/0x780 kernel/time/timer.c:1394

stack backtrace:
CPU: 0 PID: 8762 Comm: sshd Not tainted 5.3.0+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <IRQ>
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 lib/dump_stack.c:113
 print_circular_bug.isra.0.cold+0x163/0x172 kernel/locking/lockdep.c:1685
 check_noncircular+0x32e/0x3e0 kernel/locking/lockdep.c:1809
 check_prev_add kernel/locking/lockdep.c:2476 [inline]
 check_prevs_add kernel/locking/lockdep.c:2581 [inline]
 validate_chain kernel/locking/lockdep.c:2971 [inline]
 __lock_acquire+0x2596/0x4a00 kernel/locking/lockdep.c:3955
 lock_acquire+0x190/0x410 kernel/locking/lockdep.c:4487
 __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
 _raw_spin_lock+0x2f/0x40 kernel/locking/spinlock.c:151
 spin_lock include/linux/spinlock.h:338 [inline]
 pie_timer+0x9e/0x830 net/sched/sch_pie.c:449
 call_timer_fn+0x1ac/0x780 kernel/time/timer.c:1404
 expire_timers kernel/time/timer.c:1449 [inline]
 __run_timers kernel/time/timer.c:1773 [inline]
 __run_timers kernel/time/timer.c:1740 [inline]
 run_timer_softirq+0x6c3/0x1790 kernel/time/timer.c:1786
 __do_softirq+0x262/0x98c kernel/softirq.c:292
 do_softirq_own_stack+0x2a/0x40 arch/x86/entry/entry_64.S:1082
 </IRQ>
 do_softirq.part.0+0x11a/0x170 kernel/softirq.c:337
 do_softirq kernel/softirq.c:329 [inline]
 __local_bh_enable_ip+0x211/0x270 kernel/softirq.c:189
 local_bh_enable include/linux/bottom_half.h:32 [inline]
 rcu_read_unlock_bh include/linux/rcupdate.h:688 [inline]
 ip_finish_output2+0x8f8/0x2590 net/ipv4/ip_output.c:229
 __ip_finish_output net/ipv4/ip_output.c:308 [inline]
 __ip_finish_output+0x5fc/0xbc0 net/ipv4/ip_output.c:290
 ip_finish_output+0x38/0x1f0 net/ipv4/ip_output.c:318
 NF_HOOK_COND include/linux/netfilter.h:294 [inline]
 ip_output+0x21f/0x670 net/ipv4/ip_output.c:432
 dst_output include/net/dst.h:436 [inline]
 ip_local_out+0xbb/0x1b0 net/ipv4/ip_output.c:125
 __ip_queue_xmit+0x86f/0x1c00 net/ipv4/ip_output.c:532
 ip_queue_xmit+0x5a/0x70 include/net/ip.h:237
 __tcp_transmit_skb+0x1a6b/0x3820 net/ipv4/tcp_output.c:1169
 tcp_transmit_skb net/ipv4/tcp_output.c:1185 [inline]
 tcp_write_xmit+0xf0f/0x5910 net/ipv4/tcp_output.c:2440
 __tcp_push_pending_frames+0xb4/0x350 net/ipv4/tcp_output.c:2616
 tcp_push+0x488/0x700 net/ipv4/tcp.c:724
 tcp_sendmsg_locked+0x2876/0x3220 net/ipv4/tcp.c:1403
 tcp_sendmsg+0x30/0x50 net/ipv4/tcp.c:1434
 inet_sendmsg+0x9e/0xe0 net/ipv4/af_inet.c:807
 sock_sendmsg_nosec net/socket.c:637 [inline]
 sock_sendmsg+0xd7/0x130 net/socket.c:657
 sock_write_iter+0x27c/0x3e0 net/socket.c:989
 call_write_iter include/linux/fs.h:1895 [inline]
 new_sync_write+0x4d3/0x770 fs/read_write.c:483
 __vfs_write+0xe1/0x110 fs/read_write.c:496
 vfs_write+0x268/0x5d0 fs/read_write.c:558
 ksys_write+0x14f/0x290 fs/read_write.c:611
 __do_sys_write fs/read_write.c:623 [inline]
 __se_sys_write fs/read_write.c:620 [inline]
 __x64_sys_write+0x73/0xb0 fs/read_write.c:620
 do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x7f6c76f9d370
Code: 73 01 c3 48 8b 0d c8 4a 2b 00 31 d2 48 29 c2 64 89 11 48 83 c8 ff eb ea 90 90 83 3d 85 a2 2b 00 00 75 10 b8 01 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 0e 8a 01 00 48 89 04 24
RSP: 002b:00007ffec05a4428 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000038 RCX: 00007f6c76f9d370
RDX: 0000000000000038 RSI: 000055602f514460 RDI: 0000000000000003
RBP: 000055602f514460 R08: 0000000000000001 R09: 0101010101010101
R10: 0000000000000008 R11: 0000000000000246 R12: 00007ffec05a448c
R13: 000055602e43ffb4 R14: 0000000000000028 R15: 000055602e441ca0

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/09/28 20:31 upstream f1f2f614d535 eb6b9855 .config console log report syz C ci-upstream-kasan-gce-root
2019/09/28 01:30 upstream da05b5ea12c1 d8074e0b .config console log report syz C ci-upstream-kasan-gce-smack-root
* Struck through repros no longer work on HEAD.