GRED: Unable to relocate VQ 0x0 after dequeue, screwing up backlog ====================================================== WARNING: possible circular locking dependency detected 6.10.0-rc4-syzkaller-00869-g185d72112b95 #0 Not tainted ------------------------------------------------------ ksoftirqd/0/16 is trying to acquire lock: ffff88802e799218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline] ffff88802e799218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+.-.}-{2:2}, at: __dev_xmit_skb net/core/dev.c:3841 [inline] ffff88802e799218 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+.-.}-{2:2}, at: __dev_queue_xmit+0x22f7/0x3d30 net/core/dev.c:4359 but task is already holding lock: ffff8880586990d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline] ffff8880586990d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4345 [inline] ffff8880586990d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:341 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}: lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline] _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154 spin_lock include/linux/spinlock.h:351 [inline] __netif_tx_lock include/linux/netdevice.h:4345 [inline] sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:341 __dev_xmit_skb net/core/dev.c:3807 [inline] __dev_queue_xmit+0x1a24/0x3d30 net/core/dev.c:4359 neigh_output include/net/neighbour.h:542 [inline] ip6_finish_output2+0xffa/0x1680 net/ipv6/ip6_output.c:137 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222 NF_HOOK+0x9e/0x430 include/linux/netfilter.h:314 mld_sendpack+0x843/0xdb0 net/ipv6/mcast.c:1818 mld_send_cr net/ipv6/mcast.c:2119 [inline] mld_ifc_work+0x7d6/0xd90 net/ipv6/mcast.c:2650 process_one_work kernel/workqueue.c:3231 [inline] process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312 worker_thread+0x86d/0xd70 kernel/workqueue.c:3393 kthread+0x2f0/0x390 kernel/kthread.c:389 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 -> #0 (dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2){+.-.}-{2:2}: check_prev_add kernel/locking/lockdep.c:3134 [inline] check_prevs_add kernel/locking/lockdep.c:3253 [inline] validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3869 __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline] _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154 spin_lock include/linux/spinlock.h:351 [inline] __dev_xmit_skb net/core/dev.c:3841 [inline] __dev_queue_xmit+0x22f7/0x3d30 net/core/dev.c:4359 neigh_output include/net/neighbour.h:542 [inline] ip6_finish_output2+0xffa/0x1680 net/ipv6/ip6_output.c:137 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222 NF_HOOK include/linux/netfilter.h:314 [inline] ndisc_send_skb+0xab2/0x1380 net/ipv6/ndisc.c:509 ndisc_solicit+0x493/0x6a0 neigh_probe net/core/neighbour.c:1077 [inline] __neigh_event_send+0xece/0x15b0 net/core/neighbour.c:1244 neigh_event_send_probe include/net/neighbour.h:466 [inline] neigh_event_send include/net/neighbour.h:472 [inline] neigh_resolve_output+0x1b5/0x740 net/core/neighbour.c:1549 neigh_output include/net/neighbour.h:542 [inline] ip6_finish_output2+0xffa/0x1680 net/ipv6/ip6_output.c:137 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222 ip6_send_skb+0x112/0x230 net/ipv6/ip6_output.c:1958 icmp6_send+0x15fc/0x2070 net/ipv6/icmp.c:624 __icmpv6_send include/linux/icmpv6.h:28 [inline] icmpv6_send include/linux/icmpv6.h:49 [inline] ip6_link_failure+0x3c/0x4f0 net/ipv6/route.c:2799 ip_tunnel_xmit+0x1653/0x2950 net/ipv4/ip_tunnel.c:863 __gre_xmit+0x1cf/0x260 net/ipv4/ip_gre.c:483 erspan_xmit+0xabe/0x1310 net/ipv4/ip_gre.c:742 __netdev_start_xmit include/linux/netdevice.h:4880 [inline] netdev_start_xmit include/linux/netdevice.h:4894 [inline] xmit_one net/core/dev.c:3578 [inline] dev_hard_start_xmit+0x27a/0x7e0 net/core/dev.c:3594 sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:343 qdisc_restart net/sched/sch_generic.c:408 [inline] __qdisc_run+0xbfd/0x2170 net/sched/sch_generic.c:416 qdisc_run+0xda/0x270 include/net/pkt_sched.h:127 net_tx_action+0x89c/0xa50 net/core/dev.c:5287 handle_softirqs+0x2c4/0x970 kernel/softirq.c:554 run_ksoftirqd+0xca/0x130 kernel/softirq.c:928 smpboot_thread_fn+0x544/0xa30 kernel/smpboot.c:164 kthread+0x2f0/0x390 kernel/kthread.c:389 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 other info that might help us debug this: Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&qdisc_xmit_lock_key#3); lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2); lock(&qdisc_xmit_lock_key#3); lock(dev->qdisc_tx_busylock ?: &qdisc_tx_busylock#2); *** DEADLOCK *** 8 locks held by ksoftirqd/0/16: #0: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline] #0: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline] #0: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: net_tx_action+0x708/0xa50 net/core/dev.c:5256 #1: ffff8880586990d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: spin_lock include/linux/spinlock.h:351 [inline] #1: ffff8880586990d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: __netif_tx_lock include/linux/netdevice.h:4345 [inline] #1: ffff8880586990d8 (&qdisc_xmit_lock_key#3){+.-.}-{2:2}, at: sch_direct_xmit+0x1c4/0x5f0 net/sched/sch_generic.c:341 #2: ffff88802a9c81d8 (k-slock-AF_INET6){+.-.}-{2:2}, at: spin_trylock include/linux/spinlock.h:361 [inline] #2: ffff88802a9c81d8 (k-slock-AF_INET6){+.-.}-{2:2}, at: icmpv6_xmit_lock net/ipv6/icmp.c:108 [inline] #2: ffff88802a9c81d8 (k-slock-AF_INET6){+.-.}-{2:2}, at: icmp6_send+0xc45/0x2070 net/ipv6/icmp.c:545 #3: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline] #3: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: icmp6_send+0xba4/0x2070 net/ipv6/icmp.c:537 #4: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline] #4: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline] #4: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x712/0x1680 net/ipv6/ip6_output.c:122 #5: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline] #5: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline] #5: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: ip6_nd_hdr net/ipv6/ndisc.c:452 [inline] #5: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: ndisc_send_skb+0x574/0x1380 net/ipv6/ndisc.c:503 #6: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline] #6: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline] #6: ffffffff8e333fa0 (rcu_read_lock){....}-{1:2}, at: ip6_finish_output2+0x712/0x1680 net/ipv6/ip6_output.c:122 #7: ffffffff8e334000 (rcu_read_lock_bh){....}-{1:2}, at: local_bh_disable include/linux/bottom_half.h:20 [inline] #7: ffffffff8e334000 (rcu_read_lock_bh){....}-{1:2}, at: rcu_read_lock_bh include/linux/rcupdate.h:833 [inline] #7: ffffffff8e334000 (rcu_read_lock_bh){....}-{1:2}, at: __dev_queue_xmit+0x2d2/0x3d30 net/core/dev.c:4318 stack backtrace: CPU: 0 PID: 16 Comm: ksoftirqd/0 Not tainted 6.10.0-rc4-syzkaller-00869-g185d72112b95 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/07/2024 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114 check_noncircular+0x36a/0x4a0 kernel/locking/lockdep.c:2187 check_prev_add kernel/locking/lockdep.c:3134 [inline] check_prevs_add kernel/locking/lockdep.c:3253 [inline] validate_chain+0x18e0/0x5900 kernel/locking/lockdep.c:3869 __lock_acquire+0x1346/0x1fd0 kernel/locking/lockdep.c:5137 lock_acquire+0x1ed/0x550 kernel/locking/lockdep.c:5754 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline] _raw_spin_lock+0x2e/0x40 kernel/locking/spinlock.c:154 spin_lock include/linux/spinlock.h:351 [inline] __dev_xmit_skb net/core/dev.c:3841 [inline] __dev_queue_xmit+0x22f7/0x3d30 net/core/dev.c:4359 neigh_output include/net/neighbour.h:542 [inline] ip6_finish_output2+0xffa/0x1680 net/ipv6/ip6_output.c:137 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222 NF_HOOK include/linux/netfilter.h:314 [inline] ndisc_send_skb+0xab2/0x1380 net/ipv6/ndisc.c:509 ndisc_solicit+0x493/0x6a0 neigh_probe net/core/neighbour.c:1077 [inline] __neigh_event_send+0xece/0x15b0 net/core/neighbour.c:1244 neigh_event_send_probe include/net/neighbour.h:466 [inline] neigh_event_send include/net/neighbour.h:472 [inline] neigh_resolve_output+0x1b5/0x740 net/core/neighbour.c:1549 neigh_output include/net/neighbour.h:542 [inline] ip6_finish_output2+0xffa/0x1680 net/ipv6/ip6_output.c:137 ip6_finish_output+0x41e/0x810 net/ipv6/ip6_output.c:222 ip6_send_skb+0x112/0x230 net/ipv6/ip6_output.c:1958 icmp6_send+0x15fc/0x2070 net/ipv6/icmp.c:624 __icmpv6_send include/linux/icmpv6.h:28 [inline] icmpv6_send include/linux/icmpv6.h:49 [inline] ip6_link_failure+0x3c/0x4f0 net/ipv6/route.c:2799 ip_tunnel_xmit+0x1653/0x2950 net/ipv4/ip_tunnel.c:863 __gre_xmit+0x1cf/0x260 net/ipv4/ip_gre.c:483 erspan_xmit+0xabe/0x1310 net/ipv4/ip_gre.c:742 __netdev_start_xmit include/linux/netdevice.h:4880 [inline] netdev_start_xmit include/linux/netdevice.h:4894 [inline] xmit_one net/core/dev.c:3578 [inline] dev_hard_start_xmit+0x27a/0x7e0 net/core/dev.c:3594 sch_direct_xmit+0x2b6/0x5f0 net/sched/sch_generic.c:343 qdisc_restart net/sched/sch_generic.c:408 [inline] __qdisc_run+0xbfd/0x2170 net/sched/sch_generic.c:416 qdisc_run+0xda/0x270 include/net/pkt_sched.h:127 net_tx_action+0x89c/0xa50 net/core/dev.c:5287 handle_softirqs+0x2c4/0x970 kernel/softirq.c:554 run_ksoftirqd+0xca/0x130 kernel/softirq.c:928 smpboot_thread_fn+0x544/0xa30 kernel/smpboot.c:164 kthread+0x2f0/0x390 kernel/kthread.c:389 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244 GRED: Unable to relocate VQ 0x0 after dequeue, screwing up backlog