ci starts bisection 2022-12-27 11:20:47.896843896 +0000 UTC m=+395256.220026070 bisecting fixing commit since 3f667b5d4053ad54aee13dab5c94f04ff75ddfdf building syzkaller on 44068e196185e2f5a7c94629b6245cdde008b140 ensuring issue is reproducible on original commit 3f667b5d4053ad54aee13dab5c94f04ff75ddfdf testing commit 3f667b5d4053ad54aee13dab5c94f04ff75ddfdf gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: fd19f54fdde3b8f525009148b45739489095b11ac230fc523db560e680f5e68b all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch testing current HEAD 1b929c02afd37871d5afb9d498426f83432e71c2 testing commit 1b929c02afd37871d5afb9d498426f83432e71c2 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0b7b1ebf1c80ce97b2080ed3d38c7c2f17780222ab872419a91b0b0d3eb32d26 all runs: OK # git bisect start 1b929c02afd37871d5afb9d498426f83432e71c2 3f667b5d4053ad54aee13dab5c94f04ff75ddfdf Bisecting: 46810 revisions left to test after this (roughly 16 steps) [a48e789dd2633bdeb6552dfdfedd0435f9c2f897] Merge tag 'linux-can-next-for-5.20-20220703' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can-next testing commit a48e789dd2633bdeb6552dfdfedd0435f9c2f897 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2f21be07dede74a4811f08aabb888cfa9de31848b331edcaf1dada751e9acf11 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #6: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #7: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #8: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #9: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good a48e789dd2633bdeb6552dfdfedd0435f9c2f897 Bisecting: 23465 revisions left to test after this (roughly 15 steps) [4c0ed7d8d6e3dc013c4599a837de84794baa5b62] Merge tag 'pull-path' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs testing commit 4c0ed7d8d6e3dc013c4599a837de84794baa5b62 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cbf4676e51c22029a588470e24448a3b19413f34faa529b2a8a84e79ecdfd5ea all runs: OK # git bisect bad 4c0ed7d8d6e3dc013c4599a837de84794baa5b62 Bisecting: 11694 revisions left to test after this (roughly 14 steps) [f20c95b46b8fa3ad34b3ea2e134337f88591468b] Merge tag 'tpmdd-next-v5.20' of git://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-tpmdd testing commit f20c95b46b8fa3ad34b3ea2e134337f88591468b gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 12b32a4ac1c07454c154537152b5170834b562fab6987ed3780c16f8ec2c8804 all runs: basic kernel testing failed: WARNING: ODEBUG bug in mgmt_index_removed # git bisect skip f20c95b46b8fa3ad34b3ea2e134337f88591468b Bisecting: 11694 revisions left to test after this (roughly 14 steps) [41021f728a91035997f1ad3d6d4e983711f3c483] habanalabs: fix race between hl_get_compute_ctx() and hl_ctx_put() testing commit 41021f728a91035997f1ad3d6d4e983711f3c483 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 33ad56f58e0eec089e2fff161b95eb51c84c374f502d46e41dba945483aaf271 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good 41021f728a91035997f1ad3d6d4e983711f3c483 Bisecting: 11694 revisions left to test after this (roughly 14 steps) [07db88f11e63d78c5062447faf942745ce8959ff] wifi: mt7601u: eeprom: fix clang -Wformat warning testing commit 07db88f11e63d78c5062447faf942745ce8959ff gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d6836fecb2c876b38b5b99c13bf1e470fee191f572e339779d7660b809a9a3ce run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #6: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #7: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #8: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #9: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good 07db88f11e63d78c5062447faf942745ce8959ff Bisecting: 10838 revisions left to test after this (roughly 14 steps) [668c3c237f5ddc2889879b08f26d2374231f3287] Merge tag 'sound-6.0-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit 668c3c237f5ddc2889879b08f26d2374231f3287 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 85dbf17794a0df4eca1f0a38fcb612240cdd606557df5d42d48dfba7c4c914c9 all runs: basic kernel testing failed: WARNING: ODEBUG bug in mgmt_index_removed # git bisect skip 668c3c237f5ddc2889879b08f26d2374231f3287 Bisecting: 10838 revisions left to test after this (roughly 14 steps) [aae59bdf2585dafffc0bf71af729fe641fc2a771] iio:accel:mc3230: Remove duplicated error reporting in .remove() testing commit aae59bdf2585dafffc0bf71af729fe641fc2a771 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1fe0c4ad48b53cb11dfe485bc5393303690db95dab0ebc2908d79d809c7bbd45 run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch run #8: crashed: INFO: rcu detected stall in corrupted run #9: boot failed: INFO: task hung in add_early_randomness # git bisect good aae59bdf2585dafffc0bf71af729fe641fc2a771 Bisecting: 10838 revisions left to test after this (roughly 14 steps) [019805fea91599b22dfa62ffb29c022f35abeb06] NFSD: fix use-after-free on source server when doing inter-server copy testing commit 019805fea91599b22dfa62ffb29c022f35abeb06 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8cd3876bffe53f6c42875e89d0368d854f2092b55a7b2703cdf74a578d61d1d4 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good 019805fea91599b22dfa62ffb29c022f35abeb06 Bisecting: 3376 revisions left to test after this (roughly 12 steps) [0baf6dcc02c130a69fb21088ec31a0ba7a896f22] Merge tag 'hwmon-for-v6.1' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging testing commit 0baf6dcc02c130a69fb21088ec31a0ba7a896f22 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f77a7fb8049d85a48032d7d5c3b427262cf8bb426809fcef7eb52f1828fbf258 all runs: OK # git bisect bad 0baf6dcc02c130a69fb21088ec31a0ba7a896f22 Bisecting: 1564 revisions left to test after this (roughly 11 steps) [accc3b4a572bba903a801a393532272727f83f5b] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net testing commit accc3b4a572bba903a801a393532272727f83f5b gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: fa93a5fbf167fa4891ea17ec199884d8b6305aa1c2f7af88c8a2c528a6175004 all runs: OK # git bisect bad accc3b4a572bba903a801a393532272727f83f5b Bisecting: 920 revisions left to test after this (roughly 10 steps) [13eccc1bbb2e98df5d616398cd9215d9edab522f] net: dsa: suppress device links to LAG DSA masters testing commit 13eccc1bbb2e98df5d616398cd9215d9edab522f gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 84aa7986bd2f0e0c883f8b3a976c72fcca79e0787ab5f746b1ecc70b788e61c6 all runs: OK # git bisect bad 13eccc1bbb2e98df5d616398cd9215d9edab522f Bisecting: 471 revisions left to test after this (roughly 9 steps) [aa3fab0110580aaeb9422e73b031822990651552] Merge branch 'net_sched-redundant-resource-cleanups' testing commit aa3fab0110580aaeb9422e73b031822990651552 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0103d7364873eb1edbeaee5da36167852ab902776330d3f77d2961f984311596 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good aa3fab0110580aaeb9422e73b031822990651552 Bisecting: 235 revisions left to test after this (roughly 8 steps) [13248b975038241be329388a9a707dd12fdd5466] net: dsa: LAN9303: Add basic support for LAN9354 testing commit 13248b975038241be329388a9a707dd12fdd5466 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 96cdd94211c89e895d72ba888739dd5d3d279180c1390f7619f639bc3b7143bc all runs: OK # git bisect bad 13248b975038241be329388a9a707dd12fdd5466 Bisecting: 120 revisions left to test after this (roughly 7 steps) [03fdb11da92fde0bdc0b6e9c1c642b7414d49e8d] net: moxa: fix endianness-related issues from 'sparse' testing commit 03fdb11da92fde0bdc0b6e9c1c642b7414d49e8d gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0cb2aabc9a574ef8a393e5816eb5c95effd6061fc1184a34752df0f733942da5 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good 03fdb11da92fde0bdc0b6e9c1c642b7414d49e8d Bisecting: 60 revisions left to test after this (roughly 6 steps) [1c636b6277a2b2bf504df490b8dbadd2bd34ccd4] selftests/bpf: Add test cases for htab update testing commit 1c636b6277a2b2bf504df490b8dbadd2bd34ccd4 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e27bf11704f2b286fea1d55eec29f229bbe83d4b1d63d286f165da9fe661cae6 all runs: OK # git bisect bad 1c636b6277a2b2bf504df490b8dbadd2bd34ccd4 Bisecting: 29 revisions left to test after this (roughly 5 steps) [f52c8947347d43546581ab1bf8fa19867b664a8c] Merge branch 'bpf: expose bpf_{g,s}et_retval to more cgroup hooks' testing commit f52c8947347d43546581ab1bf8fa19867b664a8c gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 61e701109b79d0c67508e3c2949d1b60279ddd6f19a2b0d2d910f3a91e9e2ad6 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good f52c8947347d43546581ab1bf8fa19867b664a8c Bisecting: 14 revisions left to test after this (roughly 4 steps) [d4ffb6f39f1a1b260966b43a4ffdb64779c650dd] bpf: Add CGROUP prefix to cgroup_iter_order testing commit d4ffb6f39f1a1b260966b43a4ffdb64779c650dd gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: dd02403ac3ede844ab6d4d0e8b05b1a47dd4c42b5b2af805b85bd78e79cee025 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good d4ffb6f39f1a1b260966b43a4ffdb64779c650dd Bisecting: 6 revisions left to test after this (roughly 3 steps) [2eb680401df62c035ff50a7faf1296565b030df7] selftests/bpf: Fix connect4_prog tcp/socket header type conflict testing commit 2eb680401df62c035ff50a7faf1296565b030df7 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bc6ad2ecbc28bbb4d56741e8c6ed855334567b280d83fddc492b17fa0eb684a4 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good 2eb680401df62c035ff50a7faf1296565b030df7 Bisecting: 2 revisions left to test after this (roughly 2 steps) [197072945a708d62181895409effdfcda80c7798] selftest/bpf: Ensure no module loading in bpf_setsockopt(TCP_CONGESTION) testing commit 197072945a708d62181895409effdfcda80c7798 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2845a04e77f0828f5d44e610a319f3211eca99c4c1261a093e918c7f4a9aa334 all runs: crashed: KASAN: slab-out-of-bounds Read in __htab_map_lookup_and_delete_batch # git bisect good 197072945a708d62181895409effdfcda80c7798 Bisecting: 1 revision left to test after this (roughly 1 step) [2775da21628738ce073a3a6a806adcbaada0f091] bpf: Disable preemption when increasing per-cpu map_locked testing commit 2775da21628738ce073a3a6a806adcbaada0f091 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: eb27a4c41c1eea48aaca4e4b793df3ed1f4d0e63bc8813e5f1a21adc42e71c6e all runs: OK # git bisect bad 2775da21628738ce073a3a6a806adcbaada0f091 2775da21628738ce073a3a6a806adcbaada0f091 is the first bad commit commit 2775da21628738ce073a3a6a806adcbaada0f091 Author: Hou Tao Date: Wed Aug 31 12:26:27 2022 +0800 bpf: Disable preemption when increasing per-cpu map_locked Per-cpu htab->map_locked is used to prohibit the concurrent accesses from both NMI and non-NMI contexts. But since commit 74d862b682f5 ("sched: Make migrate_disable/enable() independent of RT"), migrate_disable() is also preemptible under CONFIG_PREEMPT case, so now map_locked also disallows concurrent updates from normal contexts (e.g. userspace processes) unexpectedly as shown below: process A process B htab_map_update_elem() htab_lock_bucket() migrate_disable() /* return 1 */ __this_cpu_inc_return() /* preempted by B */ htab_map_update_elem() /* the same bucket as A */ htab_lock_bucket() migrate_disable() /* return 2, so lock fails */ __this_cpu_inc_return() return -EBUSY A fix that seems feasible is using in_nmi() in htab_lock_bucket() and only checking the value of map_locked for nmi context. But it will re-introduce dead-lock on bucket lock if htab_lock_bucket() is re-entered through non-tracing program (e.g. fentry program). One cannot use preempt_disable() to fix this issue as htab_use_raw_lock being false causes the bucket lock to be a spin lock which can sleep and does not work with preempt_disable(). Therefore, use migrate_disable() when using the spinlock instead of preempt_disable() and defer fixing concurrent updates to when the kernel has its own BPF memory allocator. Fixes: 74d862b682f5 ("sched: Make migrate_disable/enable() independent of RT") Reviewed-by: Hao Luo Signed-off-by: Hou Tao Link: https://lore.kernel.org/r/20220831042629.130006-2-houtao@huaweicloud.com Signed-off-by: Martin KaFai Lau kernel/bpf/hashtab.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) culprit signature: eb27a4c41c1eea48aaca4e4b793df3ed1f4d0e63bc8813e5f1a21adc42e71c6e parent signature: 2845a04e77f0828f5d44e610a319f3211eca99c4c1261a093e918c7f4a9aa334 revisions tested: 22, total time: 5h14m30.92000562s (build: 2h50m42.880157414s, test: 2h18m14.441133587s) first good commit: 2775da21628738ce073a3a6a806adcbaada0f091 bpf: Disable preemption when increasing per-cpu map_locked recipients (to): ["haoluo@google.com" "houtao1@huawei.com" "martin.lau@kernel.org"] recipients (cc): []