bisecting cause commit starting from f616447034a120b18f6e612814641e7d8f5d7f0a building syzkaller on 24dc29dba51a8ae7ae86ebc04521a0b2223c531f testing commit f616447034a120b18f6e612814641e7d8f5d7f0a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8c36399573979793e94d3130547c859d96826990aea31317c12e63113dab5306 all runs: crashed: WARNING in gnet_stats_add_basic testing release v5.14 testing commit 7d2a07b769330c34b4deabeed939325c77a7ec2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: de5d0644639cc5c360c5e4129b2c3253ccb03fa29846cb1eb6936ee1da8e68ac all runs: OK # git bisect start f616447034a120b18f6e612814641e7d8f5d7f0a 7d2a07b769330c34b4deabeed939325c77a7ec2f Bisecting: 6951 revisions left to test after this (roughly 13 steps) [634135a07b887a8ad8904da8c147407650747a38] Merge tag 'soc-5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit 634135a07b887a8ad8904da8c147407650747a38 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: de751451a9e39ff9addd9e94e226be6b9559699d9dc27b0c10472041bc55c088 run #0: OK run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: boot failed: possible deadlock in blktrans_open # git bisect good 634135a07b887a8ad8904da8c147407650747a38 Bisecting: 3521 revisions left to test after this (roughly 12 steps) [cc09ee80c3b18ae1a897a30a17fe710b2b2f620a] Merge tag 'mm-slub-5.15-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux testing commit cc09ee80c3b18ae1a897a30a17fe710b2b2f620a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5c829bd97e812a8176c0eaed92bcfa244848a49c25ba3d4d0848aec3ab270fa1 all runs: OK # git bisect good cc09ee80c3b18ae1a897a30a17fe710b2b2f620a Bisecting: 1759 revisions left to test after this (roughly 11 steps) [78764f450bd9649e03b7a3012ab52075c0b60a5e] Merge tag 'mlx5-fixes-2021-09-30' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux testing commit 78764f450bd9649e03b7a3012ab52075c0b60a5e compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: b635d040a2966f44cce1012ac2977dfcea9c014fe08e20eeb2b8b9fbb3b0e347 all runs: OK # git bisect good 78764f450bd9649e03b7a3012ab52075c0b60a5e Bisecting: 888 revisions left to test after this (roughly 10 steps) [ffdd33dd9c12a8c263f78d778066709ef94671f9] netfilter: core: Fix clang warnings about unused static inlines testing commit ffdd33dd9c12a8c263f78d778066709ef94671f9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e4628bbe102911f3cb57b90c3753d7f011873250d0097eaa79511b2636d7699b all runs: OK # git bisect good ffdd33dd9c12a8c263f78d778066709ef94671f9 Bisecting: 443 revisions left to test after this (roughly 9 steps) [efb52a7d9511df818391f1afa459507425833438] Merge tag 'powerpc-5.15-3' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux testing commit efb52a7d9511df818391f1afa459507425833438 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 205cd3f937f64ae14c9a5302186234222e1ef5c26a9a21aa7da404a5a8dcb2a3 all runs: OK # git bisect good efb52a7d9511df818391f1afa459507425833438 Bisecting: 259 revisions left to test after this (roughly 8 steps) [8b017fbe0bbb98dd71fb4850f6b9cc0e136a26b8] net: of: fix stub of_net helpers for CONFIG_NET=n testing commit 8b017fbe0bbb98dd71fb4850f6b9cc0e136a26b8 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2265ddef3141f61506f6792c1f5fc48403da58d60d2cbda5e9d9ce2660519604 all runs: OK # git bisect good 8b017fbe0bbb98dd71fb4850f6b9cc0e136a26b8 Bisecting: 129 revisions left to test after this (roughly 7 steps) [4ece1ae440151a2e5af441d2c9b1d8eb3a700670] net: microchip: lan743x: add support for PTP pulse width (duty cycle) testing commit 4ece1ae440151a2e5af441d2c9b1d8eb3a700670 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7aeb80adac12cbff8d3ea5a4e8df67801a2354c8e9077799596240835c5852ff all runs: OK # git bisect good 4ece1ae440151a2e5af441d2c9b1d8eb3a700670 Bisecting: 71 revisions left to test after this (roughly 6 steps) [803a4344c7907e929dc155b6c958c972c0f20c21] Merge branch '100GbE' of git://git.kernel.org/pub/scm/linux/kernel/git/tnguy/next-queue testing commit 803a4344c7907e929dc155b6c958c972c0f20c21 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 01999188029156f0deb1bef15523dbeb10c85bb68e0a0f17418a4f22fc212876 all runs: OK # git bisect good 803a4344c7907e929dc155b6c958c972c0f20c21 Bisecting: 35 revisions left to test after this (roughly 5 steps) [24fb68111d4509524b483b2577f1b20a24f5fdfd] net/smc: retrieve v2 gid from IB device testing commit 24fb68111d4509524b483b2577f1b20a24f5fdfd compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d30b99642443f45325ac7c8035aa138239ad3297eeca3966b384cb6d19eac2ef all runs: OK # git bisect good 24fb68111d4509524b483b2577f1b20a24f5fdfd Bisecting: 17 revisions left to test after this (roughly 4 steps) [d40dfa0cebd8197aaca2fcac4b9fa61da6e1c9fd] net: w5100: Make w5100_remove() return void testing commit d40dfa0cebd8197aaca2fcac4b9fa61da6e1c9fd compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 51916346f9f46e6586c2831a82644a2f670fa0561dc3f33c4caf23b0d93ddc50 all runs: crashed: WARNING in gnet_stats_add_basic # git bisect bad d40dfa0cebd8197aaca2fcac4b9fa61da6e1c9fd Bisecting: 8 revisions left to test after this (roughly 3 steps) [7361df4606ba5ab6b998f4467496b4bbf4e5526b] mq, mqprio: Use gnet_stats_add_queue(). testing commit 7361df4606ba5ab6b998f4467496b4bbf4e5526b compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 47740de90f639be4a359d35205b003d57007868360a3b1a6d1ed7c8dec3d9d83 all runs: OK # git bisect good 7361df4606ba5ab6b998f4467496b4bbf4e5526b Bisecting: 4 revisions left to test after this (roughly 2 steps) [f56940daa5a74fb20b5f5487535549949f2d8d0c] net: sched: Use _bstats_update/set() instead of raw writes testing commit f56940daa5a74fb20b5f5487535549949f2d8d0c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 91640ded8dc47e9f92c2da904eb5099138917db3e44a1a60735fddd34eb2db0a all runs: OK # git bisect good f56940daa5a74fb20b5f5487535549949f2d8d0c Bisecting: 2 revisions left to test after this (roughly 1 step) [29cbcd85828372333aa87542c51f2b2b0fd4380c] net: sched: Remove Qdisc::running sequence counter testing commit 29cbcd85828372333aa87542c51f2b2b0fd4380c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 51916346f9f46e6586c2831a82644a2f670fa0561dc3f33c4caf23b0d93ddc50 all runs: crashed: WARNING in gnet_stats_add_basic # git bisect bad 29cbcd85828372333aa87542c51f2b2b0fd4380c Bisecting: 0 revisions left to test after this (roughly 0 steps) [50dc9a8572aa4d7cdc56670228fcde40289ed289] net: sched: Merge Qdisc::bstats and Qdisc::cpu_bstats data types testing commit 50dc9a8572aa4d7cdc56670228fcde40289ed289 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 732ecbbba2aa9621b3f43ecf2da012c2c6923eb842048ee21155f533e56ce0f3 all runs: OK # git bisect good 50dc9a8572aa4d7cdc56670228fcde40289ed289 29cbcd85828372333aa87542c51f2b2b0fd4380c is the first bad commit commit 29cbcd85828372333aa87542c51f2b2b0fd4380c Author: Ahmed S. Darwish Date: Sat Oct 16 10:49:10 2021 +0200 net: sched: Remove Qdisc::running sequence counter The Qdisc::running sequence counter has two uses: 1. Reliably reading qdisc's tc statistics while the qdisc is running (a seqcount read/retry loop at gnet_stats_add_basic()). 2. As a flag, indicating whether the qdisc in question is running (without any retry loops). For the first usage, the Qdisc::running sequence counter write section, qdisc_run_begin() => qdisc_run_end(), covers a much wider area than what is actually needed: the raw qdisc's bstats update. A u64_stats sync point was thus introduced (in previous commits) inside the bstats structure itself. A local u64_stats write section is then started and stopped for the bstats updates. Use that u64_stats sync point mechanism for the bstats read/retry loop at gnet_stats_add_basic(). For the second qdisc->running usage, a __QDISC_STATE_RUNNING bit flag, accessed with atomic bitops, is sufficient. Using a bit flag instead of a sequence counter at qdisc_run_begin/end() and qdisc_is_running() leads to the SMP barriers implicitly added through raw_read_seqcount() and write_seqcount_begin/end() getting removed. All call sites have been surveyed though, and no required ordering was identified. Now that the qdisc->running sequence counter is no longer used, remove it. Note, using u64_stats implies no sequence counter protection for 64-bit architectures. This can lead to the qdisc tc statistics "packets" vs. "bytes" values getting out of sync on rare occasions. The individual values will still be valid. Signed-off-by: Ahmed S. Darwish Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: David S. Miller include/linux/netdevice.h | 4 ---- include/net/gen_stats.h | 19 ++++++++---------- include/net/sch_generic.h | 33 +++++++++++++------------------ net/core/gen_estimator.c | 16 +++++++++------ net/core/gen_stats.c | 50 ++++++++++++++++++++++++++--------------------- net/sched/act_api.c | 9 +++++---- net/sched/act_police.c | 2 +- net/sched/sch_api.c | 16 +++------------ net/sched/sch_atm.c | 3 +-- net/sched/sch_cbq.c | 9 +++------ net/sched/sch_drr.c | 10 +++------- net/sched/sch_ets.c | 3 +-- net/sched/sch_generic.c | 10 ++-------- net/sched/sch_hfsc.c | 8 +++----- net/sched/sch_htb.c | 7 +++---- net/sched/sch_mq.c | 7 +++---- net/sched/sch_mqprio.c | 14 ++++++------- net/sched/sch_multiq.c | 3 +-- net/sched/sch_prio.c | 4 ++-- net/sched/sch_qfq.c | 7 +++---- net/sched/sch_taprio.c | 2 +- 21 files changed, 102 insertions(+), 134 deletions(-) culprit signature: 51916346f9f46e6586c2831a82644a2f670fa0561dc3f33c4caf23b0d93ddc50 parent signature: 732ecbbba2aa9621b3f43ecf2da012c2c6923eb842048ee21155f533e56ce0f3 revisions tested: 16, total time: 3h57m0.112361803s (build: 1h45m21.220259325s, test: 2h10m10.772345527s) first bad commit: 29cbcd85828372333aa87542c51f2b2b0fd4380c net: sched: Remove Qdisc::running sequence counter recipients (to): ["a.darwish@linutronix.de" "bigeasy@linutronix.de" "davem@davemloft.net"] recipients (cc): [] crash: WARNING in gnet_stats_add_basic ------------[ cut here ]------------ WARNING: CPU: 1 PID: 7052 at net/core/gen_stats.c:157 gnet_stats_add_basic+0x1d1/0x240 net/core/gen_stats.c:171 Modules linked in: CPU: 1 PID: 7052 Comm: syz-executor.3 Not tainted 5.15.0-rc5-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:gnet_stats_add_basic+0x1d1/0x240 net/core/gen_stats.c:157 Code: 25 78 b6 fc 05 0f 82 4d ff ff ff 48 8b 44 24 20 4c 01 28 44 8b 24 24 4c 01 60 08 48 83 c4 28 5b 5d 41 5c 41 5d 41 5e 41 5f c3 <0f> 0b e9 d4 fe ff ff 48 89 df e8 20 0e 26 fb e9 65 fe ff ff 48 89 RSP: 0018:ffffc90000dc0ba0 EFLAGS: 00010206 RAX: 0000000000000100 RBX: ffff88806f74a420 RCX: 0000000000000001 RDX: ffff88806f74a420 RSI: 0000000000000000 RDI: ffffc90000dc0cb0 RBP: 0000000000000000 R08: 0000000000000001 R09: ffffc90000dc0cbf R10: fffff520001b8197 R11: 0000000000000016 R12: ffff88806ff02608 R13: ffff88806ff02680 R14: ffff88806ff02680 R15: ffff88806ff02621 FS: 0000555556f5a400(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f15315f61b8 CR3: 0000000078a49000 CR4: 00000000003506e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: est_fetch_counters+0xab/0x130 net/core/gen_estimator.c:69 est_timer+0x90/0x6f0 net/core/gen_estimator.c:83 call_timer_fn+0x163/0x4a0 kernel/time/timer.c:1421 expire_timers kernel/time/timer.c:1466 [inline] __run_timers.part.0+0x524/0x890 kernel/time/timer.c:1734 __run_timers kernel/time/timer.c:1715 [inline] run_timer_softirq+0x9c/0x190 kernel/time/timer.c:1747 __do_softirq+0x29b/0x9c2 kernel/softirq.c:558 invoke_softirq kernel/softirq.c:432 [inline] __irq_exit_rcu+0x123/0x180 kernel/softirq.c:636 irq_exit_rcu+0x5/0x20 kernel/softirq.c:648 sysvec_apic_timer_interrupt+0x93/0xc0 arch/x86/kernel/apic/apic.c:1097 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638 RIP: 0010:__syscall_enter_from_user_work kernel/entry/common.c:87 [inline] RIP: 0010:syscall_enter_from_user_mode+0x2b/0x70 kernel/entry/common.c:108 Code: 54 49 89 f4 55 48 89 fd 48 8b 7c 24 10 e8 bd f5 ff ff eb 27 eb 2b e8 e4 c1 04 f9 e8 9f c0 04 f9 fb 65 48 8b 04 25 40 f0 01 00 <48> 8b 70 08 40 f6 c6 3f 75 19 4c 89 e0 5d 41 5c c3 eb 1b 0f 0b eb RSP: 0018:ffffc90002aaff28 EFLAGS: 00000206 RAX: ffff888073a60000 RBX: 0000000000000000 RCX: 1ffffffff18fe0d1 RDX: 0000000000000000 RSI: ffffffff88cb4a00 RDI: ffffffff89206ee0 RBP: ffffc90002aaff58 R08: 0000000000000001 R09: 0000000000000001 R10: ffffed10173e6539 R11: 0000000000000001 R12: 0000000000000037 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 do_syscall_64+0x16/0xb0 arch/x86/entry/common.c:76 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f153151905a Code: 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 37 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007ffe53bed408 EFLAGS: 00000246 ORIG_RAX: 0000000000000037 RAX: ffffffffffffffda RBX: 0000000000000029 RCX: 00007f153151905a RDX: 0000000000000040 RSI: 0000000000000029 RDI: 0000000000000003 RBP: 00007ffe53bed430 R08: 00007ffe53bed42c R09: ffff000000000000 R10: 00007ffe53bed430 R11: 0000000000000246 R12: 00007ffe53bed490 R13: 0000000000000003 R14: 00007ffe53bed42c R15: 00007f15315f8220 ---------------- Code disassembly (best guess): 0: 54 push %rsp 1: 49 89 f4 mov %rsi,%r12 4: 55 push %rbp 5: 48 89 fd mov %rdi,%rbp 8: 48 8b 7c 24 10 mov 0x10(%rsp),%rdi d: e8 bd f5 ff ff callq 0xfffff5cf 12: eb 27 jmp 0x3b 14: eb 2b jmp 0x41 16: e8 e4 c1 04 f9 callq 0xf904c1ff 1b: e8 9f c0 04 f9 callq 0xf904c0bf 20: fb sti 21: 65 48 8b 04 25 40 f0 mov %gs:0x1f040,%rax 28: 01 00 * 2a: 48 8b 70 08 mov 0x8(%rax),%rsi <-- trapping instruction 2e: 40 f6 c6 3f test $0x3f,%sil 32: 75 19 jne 0x4d 34: 4c 89 e0 mov %r12,%rax 37: 5d pop %rbp 38: 41 5c pop %r12 3a: c3 retq 3b: eb 1b jmp 0x58 3d: 0f 0b ud2 3f: eb .byte 0xeb