bisecting fixing commit since 911e3a46fb38669560021537e00222591231f456 building syzkaller on be531bb42381b245eed805e49fd889d1c2118c76 testing commit 911e3a46fb38669560021537e00222591231f456 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5f4c040d9cbf85a30ba4ad270e0dc94f34d371a46e469a85c30b2cb1b12fd59b run #0: crashed: BUG: corrupted list in netif_napi_add run #1: crashed: BUG: corrupted list in netif_napi_add run #2: crashed: BUG: corrupted list in netif_napi_add run #3: crashed: BUG: corrupted list in netif_napi_add run #4: crashed: BUG: corrupted list in netif_napi_add run #5: crashed: BUG: corrupted list in netif_napi_add run #6: crashed: BUG: corrupted list in netif_napi_add run #7: crashed: BUG: corrupted list in netif_napi_add run #8: crashed: WARNING in napi_complete_done run #9: crashed: BUG: corrupted list in netif_napi_add run #10: crashed: BUG: corrupted list in netif_napi_add run #11: crashed: BUG: corrupted list in netif_napi_add run #12: crashed: BUG: corrupted list in netif_napi_add run #13: crashed: BUG: corrupted list in netif_napi_add run #14: crashed: BUG: corrupted list in netif_napi_add run #15: crashed: WARNING in napi_complete_done run #16: crashed: BUG: corrupted list in netif_napi_add run #17: crashed: BUG: corrupted list in netif_napi_add run #18: crashed: BUG: corrupted list in netif_napi_add run #19: crashed: BUG: corrupted list in netif_napi_add testing current HEAD a3c62a042237d1adeb0290dcb768e17edd6dcd25 testing commit a3c62a042237d1adeb0290dcb768e17edd6dcd25 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8878818e0c51d9ef32d9e35d955077653d00de8e68acd019b7a1df7642448635 all runs: OK # git bisect start a3c62a042237d1adeb0290dcb768e17edd6dcd25 911e3a46fb38669560021537e00222591231f456 Bisecting: 6991 revisions left to test after this (roughly 13 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [d461e96cd22b5aeb1df448536b92e8d8e88c4a05] Merge tag 'drivers-5.16' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit d461e96cd22b5aeb1df448536b92e8d8e88c4a05 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cd912c415fa9e31a7c929ec985a7eacfa11751bc8b930eccc21fabd04ae6d86d all runs: crashed: BUG: corrupted list in netif_napi_add # git bisect good d461e96cd22b5aeb1df448536b92e8d8e88c4a05 Bisecting: 3498 revisions left to test after this (roughly 12 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [89fa0be0a09c8edb5e028e7368ae87c8f6cbc462] Merge tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux testing commit 89fa0be0a09c8edb5e028e7368ae87c8f6cbc462 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 866500fc30ccb63e10b321bb1adc5a61c45b8b532042d882fa2847cb7c843bb4 all runs: crashed: BUG: corrupted list in netif_napi_add # git bisect good 89fa0be0a09c8edb5e028e7368ae87c8f6cbc462 Bisecting: 1749 revisions left to test after this (roughly 11 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [e968b1b3e9b86c4751faea019a5d340fee9e9142] arp: Remove #ifdef CONFIG_PROC_FS testing commit e968b1b3e9b86c4751faea019a5d340fee9e9142 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 95bbb9bc2832a87ed6465ca5f51ec53e2fe6d1ecb2d7e68ca350f2eaba4648aa all runs: OK # git bisect bad e968b1b3e9b86c4751faea019a5d340fee9e9142 Bisecting: 870 revisions left to test after this (roughly 10 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [9311ccef2782488ab58676c8282f9f71d01cebf5] Merge tag 'mlx5-fixes-2021-11-16' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux testing commit 9311ccef2782488ab58676c8282f9f71d01cebf5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 58598bce78a523fad2759af8a7cc499efa1ab469707bec829f0da02f82035842 all runs: OK # git bisect bad 9311ccef2782488ab58676c8282f9f71d01cebf5 Bisecting: 437 revisions left to test after this (roughly 9 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [dbf49896187fd58c577fa1574a338e4f3672b4b2] Merge branch 'akpm' (patches from Andrew) testing commit dbf49896187fd58c577fa1574a338e4f3672b4b2 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2ac1a8a003648decba4cae8a7e86afd5c459973a3472b50d484b67873e576fc8 all runs: OK # git bisect bad dbf49896187fd58c577fa1574a338e4f3672b4b2 Bisecting: 202 revisions left to test after this (roughly 8 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [2ec20f489591962db8ff1718aa6055c08d88d0cc] Merge tag 'nfs-for-5.16-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs testing commit 2ec20f489591962db8ff1718aa6055c08d88d0cc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 04907dc53a0efbc68a10d22289ccbe5fb5245b37a5498ead645bd544052f09d6 run #0: crashed: WARNING in napi_complete_done run #1: crashed: BUG: corrupted list in netif_napi_add run #2: crashed: BUG: corrupted list in netif_napi_add run #3: crashed: BUG: corrupted list in netif_napi_add run #4: crashed: BUG: corrupted list in netif_napi_add run #5: crashed: BUG: corrupted list in netif_napi_add run #6: crashed: BUG: corrupted list in netif_napi_add run #7: crashed: BUG: corrupted list in netif_napi_add run #8: crashed: BUG: corrupted list in netif_napi_add run #9: crashed: BUG: corrupted list in netif_napi_add # git bisect good 2ec20f489591962db8ff1718aa6055c08d88d0cc Bisecting: 105 revisions left to test after this (roughly 7 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [9758aba8542bb43029d077303d05df1d00a8dbb5] amt: add IPV6 Kconfig dependency testing commit 9758aba8542bb43029d077303d05df1d00a8dbb5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 19764316c543add8d8dd362652a2350fa965712fe1c09623cf86eee277068570 all runs: crashed: BUG: corrupted list in netif_napi_add # git bisect good 9758aba8542bb43029d077303d05df1d00a8dbb5 Bisecting: 52 revisions left to test after this (roughly 6 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [913ffbdd99856184b4e8004f984d7432dcb990cd] mm: unexport folio_memcg_{,un}lock testing commit 913ffbdd99856184b4e8004f984d7432dcb990cd compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6c3a368b1a08a4d3c2f80a91f626631394f3bbbac25500548d6bc1fd9dd19a80 run #0: crashed: BUG: corrupted list in netif_napi_add run #1: crashed: BUG: corrupted list in netif_napi_add run #2: crashed: WARNING in napi_complete_done run #3: crashed: BUG: corrupted list in netif_napi_add run #4: crashed: BUG: corrupted list in netif_napi_add run #5: crashed: BUG: corrupted list in netif_napi_add run #6: crashed: BUG: corrupted list in netif_napi_add run #7: crashed: BUG: corrupted list in netif_napi_add run #8: crashed: BUG: corrupted list in netif_napi_add run #9: crashed: BUG: corrupted list in netif_napi_add # git bisect good 913ffbdd99856184b4e8004f984d7432dcb990cd Bisecting: 26 revisions left to test after this (roughly 5 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [bb7bbb6e36474933540c24ae1f1ad651b843981f] net: marvell: mvpp2: Fix wrong SerDes reconfiguration order testing commit bb7bbb6e36474933540c24ae1f1ad651b843981f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ce80f18f885e720aff1ebbf66ad506188515cc62c6457a3cbab6a05a15ac2183 all runs: crashed: BUG: corrupted list in netif_napi_add # git bisect good bb7bbb6e36474933540c24ae1f1ad651b843981f Bisecting: 16 revisions left to test after this (roughly 4 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [d336509cb9d03970911878bb77f0497f64fda061] selftests/net: udpgso_bench_rx: fix port argument testing commit d336509cb9d03970911878bb77f0497f64fda061 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c32d5abc31c303cf3ada1f8c7278214522f6c5e6cf3111a33dbdd6b36873f01b all runs: OK # git bisect bad d336509cb9d03970911878bb77f0497f64fda061 Bisecting: 4 revisions left to test after this (roughly 2 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [c7ebe23cee350fb187ee00ff445b01e11de0bfe9] net/mlx5: Lag, fix a potential Oops with mlx5_lag_create_definer() testing commit c7ebe23cee350fb187ee00ff445b01e11de0bfe9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 98a43dfb0859535aa5b3f46e03c294bb3b43509b6484407364ecbd8b5ff08499 run #0: crashed: BUG: corrupted list in netif_napi_add run #1: crashed: WARNING in napi_complete_done run #2: crashed: BUG: corrupted list in netif_napi_add run #3: crashed: BUG: corrupted list in netif_napi_add run #4: crashed: BUG: corrupted list in netif_napi_add run #5: crashed: BUG: corrupted list in netif_napi_add run #6: crashed: BUG: corrupted list in netif_napi_add run #7: crashed: WARNING in napi_complete_done run #8: crashed: BUG: corrupted list in netif_napi_add run #9: crashed: BUG: corrupted list in netif_napi_add # git bisect good c7ebe23cee350fb187ee00ff445b01e11de0bfe9 Bisecting: 2 revisions left to test after this (roughly 1 step) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [0315a075f1343966ea2d9a085666a88a69ea6a3d] net: fix premature exit from NAPI state polling in napi_disable() testing commit 0315a075f1343966ea2d9a085666a88a69ea6a3d compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c32d5abc31c303cf3ada1f8c7278214522f6c5e6cf3111a33dbdd6b36873f01b all runs: OK # git bisect bad 0315a075f1343966ea2d9a085666a88a69ea6a3d Bisecting: 0 revisions left to test after this (roughly 0 steps) warning: unable to access '/syzkaller/.config/git/ignore': Permission denied warning: unable to access '/syzkaller/.config/git/attributes': Permission denied [e5d5aadcf3cd59949316df49c27cb21788d7efe4] net/smc: fix sk_refcnt underflow on linkdown and fallback testing commit e5d5aadcf3cd59949316df49c27cb21788d7efe4 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5418dadac3132b26d007e5e7820921ffc0d3e5b15216b2e2a2c2672e2fc2b013 all runs: crashed: BUG: corrupted list in netif_napi_add # git bisect good e5d5aadcf3cd59949316df49c27cb21788d7efe4 warning: unable to access '/syzkaller/.config/git/attributes': Permission denied 0315a075f1343966ea2d9a085666a88a69ea6a3d is the first bad commit commit 0315a075f1343966ea2d9a085666a88a69ea6a3d Author: Alexander Lobakin Date: Wed Nov 10 20:56:05 2021 +0100 net: fix premature exit from NAPI state polling in napi_disable() Commit 719c57197010 ("net: make napi_disable() symmetric with enable") accidentally introduced a bug sometimes leading to a kernel BUG when bringing an iface up/down under heavy traffic load. Prior to this commit, napi_disable() was polling n->state until none of (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC) is set and then always flip them. Now there's a possibility to get away with the NAPIF_STATE_SCHE unset as 'continue' drops us to the cmpxchg() call with an uninitialized variable, rather than straight to another round of the state check. Error path looks like: napi_disable(): unsigned long val, new; /* new is uninitialized */ do { val = READ_ONCE(n->state); /* NAPIF_STATE_NPSVC and/or NAPIF_STATE_SCHED is set */ if (val & (NAPIF_STATE_SCHED | NAPIF_STATE_NPSVC)) { /* true */ usleep_range(20, 200); continue; /* go straight to the condition check */ } new = val | <...> } while (cmpxchg(&n->state, val, new) != val); /* state == val, cmpxchg() writes garbage */ napi_enable(): do { val = READ_ONCE(n->state); BUG_ON(!test_bit(NAPI_STATE_SCHED, &val)); /* 50/50 boom */ <...> while the typical BUG splat is like: [ 172.652461] ------------[ cut here ]------------ [ 172.652462] kernel BUG at net/core/dev.c:6937! [ 172.656914] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 172.661966] CPU: 36 PID: 2829 Comm: xdp_redirect_cp Tainted: G I 5.15.0 #42 [ 172.670222] Hardware name: Intel Corporation S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0014.082620210524 08/26/2021 [ 172.680646] RIP: 0010:napi_enable+0x5a/0xd0 [ 172.684832] Code: 07 49 81 cc 00 01 00 00 4c 89 e2 48 89 d8 80 e6 fb f0 48 0f b1 55 10 48 39 c3 74 10 48 8b 5d 10 f6 c7 04 75 3d f6 c3 01 75 b4 <0f> 0b 5b 5d 41 5c c3 65 ff 05 b8 e5 61 53 48 c7 c6 c0 f3 34 ad 48 [ 172.703578] RSP: 0018:ffffa3c9497477a8 EFLAGS: 00010246 [ 172.708803] RAX: ffffa3c96615a014 RBX: 0000000000000000 RCX: ffff8a4b575301a0 < snip > [ 172.782403] Call Trace: [ 172.784857] [ 172.786963] ice_up_complete+0x6f/0x210 [ice] [ 172.791349] ice_xdp+0x136/0x320 [ice] [ 172.795108] ? ice_change_mtu+0x180/0x180 [ice] [ 172.799648] dev_xdp_install+0x61/0xe0 [ 172.803401] dev_xdp_attach+0x1e0/0x550 [ 172.807240] dev_change_xdp_fd+0x1e6/0x220 [ 172.811338] do_setlink+0xee8/0x1010 [ 172.814917] rtnl_setlink+0xe5/0x170 [ 172.818499] ? bpf_lsm_binder_set_context_mgr+0x10/0x10 [ 172.823732] ? security_capable+0x36/0x50 < snip > Fix this by replacing 'do { } while (cmpxchg())' with an "infinite" for-loop with an explicit break. From v1 [0]: - just use a for-loop to simplify both the fix and the existing code (Eric). [0] https://lore.kernel.org/netdev/20211110191126.1214-1-alexandr.lobakin@intel.com Fixes: 719c57197010 ("net: make napi_disable() symmetric with enable") Suggested-by: Eric Dumazet # for-loop Signed-off-by: Alexander Lobakin Reviewed-by: Jesse Brandeburg Reviewed-by: Eric Dumazet Link: https://lore.kernel.org/r/20211110195605.1304-1-alexandr.lobakin@intel.com Signed-off-by: Jakub Kicinski net/core/dev.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) culprit signature: c32d5abc31c303cf3ada1f8c7278214522f6c5e6cf3111a33dbdd6b36873f01b parent signature: 5418dadac3132b26d007e5e7820921ffc0d3e5b15216b2e2a2c2672e2fc2b013 revisions tested: 15, total time: 3h22m13.318956962s (build: 1h38m52.183530161s, test: 1h41m43.922743874s) first good commit: 0315a075f1343966ea2d9a085666a88a69ea6a3d net: fix premature exit from NAPI state polling in napi_disable() recipients (to): ["alexandr.lobakin@intel.com" "edumazet@google.com" "jesse.brandeburg@intel.com" "kuba@kernel.org"] recipients (cc): []