bisecting fixing commit since c84e1efae022071a4fcf9f1899bf71777c49943a building syzkaller on 76b4dcc777ca842ee42225411c5c9d5ac3e91b8a testing commit c84e1efae022071a4fcf9f1899bf71777c49943a compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1e475877e101ebe1ecf845bd5ab0de0b6cc042e5364316c2c3bbaacc19e0a506 all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb testing current HEAD 64222515138e43da1fcf288f0289ef1020427b87 testing commit 64222515138e43da1fcf288f0289ef1020427b87 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bd48dde3eac81b50cca1fa2b820933812f6f289e197cf700a544ff262d45e347 all runs: OK # git bisect start 64222515138e43da1fcf288f0289ef1020427b87 c84e1efae022071a4fcf9f1899bf71777c49943a Bisecting: 38079 revisions left to test after this (roughly 15 steps) [d72cd4ad4174cfd2257c426ad51e4f53bcfde9c9] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi testing commit d72cd4ad4174cfd2257c426ad51e4f53bcfde9c9 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8880f8e0b49ef4a0bf7c1a7c49f960b4bd8b662904e57e91078000aa61da16da all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good d72cd4ad4174cfd2257c426ad51e4f53bcfde9c9 Bisecting: 19015 revisions left to test after this (roughly 14 steps) [da85e7ed6993144a9ca43a1106c7f898626390a3] Merge tag 'mtd/for-5.14' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux testing commit da85e7ed6993144a9ca43a1106c7f898626390a3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e69f096b6229ea4c062fef545690bb14b358b1a9e06733fe106f5706a62179e4 all runs: boot failed: kernel panic: VFS: Unable to mount root fs on unknown-block(NUM,NUM) # git bisect skip da85e7ed6993144a9ca43a1106c7f898626390a3 Bisecting: 19015 revisions left to test after this (roughly 14 steps) [f6533121696b2126a33b436f433a048b4427944f] doc: hwpoison: correct the support for hugepage testing commit f6533121696b2126a33b436f433a048b4427944f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 36b9e37527cdf471cb4409d766cb02fefd41fcfdcbce5294dd1ca22061ef1245 all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good f6533121696b2126a33b436f433a048b4427944f Bisecting: 6254 revisions left to test after this (roughly 13 steps) [477f70cd2a67904e04c2c2b9bd0fa2e95222f2f6] Merge tag 'drm-next-2021-08-31-1' of git://anongit.freedesktop.org/drm/drm testing commit 477f70cd2a67904e04c2c2b9bd0fa2e95222f2f6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 957df592e76b118128bd753e22ed20a2be82c435ff6321520d55d73d6047c00d run #0: crashed: WARNING in port100_send_cmd_async/usb_submit_urb run #1: crashed: WARNING in port100_send_cmd_async/usb_submit_urb run #2: crashed: WARNING in port100_send_cmd_async/usb_submit_urb run #3: crashed: WARNING in port100_send_cmd_async/usb_submit_urb run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK reproducer seems to be flaky # git bisect good 477f70cd2a67904e04c2c2b9bd0fa2e95222f2f6 Bisecting: 3104 revisions left to test after this (roughly 12 steps) [192ad3c27a4895ee4b2fa31c5b54a932f5bb08c1] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit 192ad3c27a4895ee4b2fa31c5b54a932f5bb08c1 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a4874e13c2d986443f9546fcdab1aae88ad6d2ab0d9680021e2de728e7002743 all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good 192ad3c27a4895ee4b2fa31c5b54a932f5bb08c1 Bisecting: 1545 revisions left to test after this (roughly 11 steps) [9bc62afe03afdf33904f5e784e1ad68c50ff00bb] Merge tag 'net-5.15-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net testing commit 9bc62afe03afdf33904f5e784e1ad68c50ff00bb compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d4d6d41e15fbf712e6d927c6a19cbd4eb37c9ff0df7e7c25f1f765a241d55f6a all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good 9bc62afe03afdf33904f5e784e1ad68c50ff00bb Bisecting: 748 revisions left to test after this (roughly 10 steps) [3e899c7209dd8f7afca59518c5ace0f03385dbc3] Merge tag 'armsoc-fixes-5.15' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc testing commit 3e899c7209dd8f7afca59518c5ace0f03385dbc3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 6ab26b861142f4c97d46192afb63c633f2b542a8cba63eea68bb3df732ae2ec6 all runs: OK # git bisect bad 3e899c7209dd8f7afca59518c5ace0f03385dbc3 Bisecting: 400 revisions left to test after this (roughly 9 steps) [115f6134a050bb098414f38555a5ab780ebbfef0] Merge tag 'gpio-fixes-for-v5.15-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux testing commit 115f6134a050bb098414f38555a5ab780ebbfef0 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ce307576fda6eb08757bef150c83f4863208c066de5aa35b1475d9be5cdad7cb all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good 115f6134a050bb098414f38555a5ab780ebbfef0 Bisecting: 203 revisions left to test after this (roughly 8 steps) [7fab1c12bde926c5a8c7d5984c551d0854d7e0b3] objtool: print out the symbol type when complaining about it testing commit 7fab1c12bde926c5a8c7d5984c551d0854d7e0b3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1a15efe56dbf538622649420b9a564cb85027e91150dd5bd6238ac10c67d3c57 all runs: OK # git bisect bad 7fab1c12bde926c5a8c7d5984c551d0854d7e0b3 Bisecting: 107 revisions left to test after this (roughly 7 steps) [89e503592385fbed872c7ea1fb89931ece3409a5] Merge tag 'iommu-fixes-v5.15-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/joro/iommu testing commit 89e503592385fbed872c7ea1fb89931ece3409a5 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: e4d52136a6cda088ea974abb4c0f6bf9db4fffb195e8e5ed69ce58664cb5b7b8 all runs: OK # git bisect bad 89e503592385fbed872c7ea1fb89931ece3409a5 Bisecting: 44 revisions left to test after this (roughly 6 steps) [3b1b6e82fb5e08e2cb355d7b2ee8644ec289de66] net: phy: enhance GPY115 loopback disable function testing commit 3b1b6e82fb5e08e2cb355d7b2ee8644ec289de66 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 22a8712caf2b2db86a56e466d7bc6c52a941bb0ca7be4ddb48d6e61a9c566f01 all runs: OK # git bisect bad 3b1b6e82fb5e08e2cb355d7b2ee8644ec289de66 Bisecting: 19 revisions left to test after this (roughly 5 steps) [7fe7f3182a0dd8f9bad463598ed103b3d8cfa739] Merge git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf testing commit 7fe7f3182a0dd8f9bad463598ed103b3d8cfa739 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 32f33f55ae7e3aa3b0f8daa1aca909e3a55a70f3ebfc6363e06e0b9b936843b2 all runs: OK # git bisect bad 7fe7f3182a0dd8f9bad463598ed103b3d8cfa739 Bisecting: 11 revisions left to test after this (roughly 4 steps) [7970a19b71044bf4dc2c1becc200275bdf1884d4] netfilter: nf_nat_masquerade: defer conntrack walk to work queue testing commit 7970a19b71044bf4dc2c1becc200275bdf1884d4 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 35d1472440ba5cd5f2450d3ed0224ba7773a4e9c753a3f0dd7377fd575ce375f all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good 7970a19b71044bf4dc2c1becc200275bdf1884d4 Bisecting: 5 revisions left to test after this (roughly 3 steps) [597aa16c782496bf74c5dc3b45ff472ade6cee64] net: ipv4: Fix rtnexthop len when RTA_FLOW is present testing commit 597aa16c782496bf74c5dc3b45ff472ade6cee64 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: cd77db1be3dfade81bb06298ce74b2d2a5e670f9fa7d54bd8e0f9d9c6c5708e9 all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good 597aa16c782496bf74c5dc3b45ff472ade6cee64 Bisecting: 2 revisions left to test after this (roughly 2 steps) [e9edc188fc76499b0b9bd60364084037f6d03773] netfilter: conntrack: serialize hash resizes and cleanups testing commit e9edc188fc76499b0b9bd60364084037f6d03773 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 110d3619d8d3984bda2347b20c2871ac331de897237918dad92478238bdf2554 all runs: OK # git bisect bad e9edc188fc76499b0b9bd60364084037f6d03773 Bisecting: 0 revisions left to test after this (roughly 1 step) [b53deef054e58fe4f37c66211b8ece9f8fc1aa13] netfilter: log: work around missing softdep backend module testing commit b53deef054e58fe4f37c66211b8ece9f8fc1aa13 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 353bc981a68517a3c2fe8194cec562c9ca10328e68a3c41819149896fcbfe689 all runs: crashed: WARNING in port100_send_cmd_async/usb_submit_urb # git bisect good b53deef054e58fe4f37c66211b8ece9f8fc1aa13 e9edc188fc76499b0b9bd60364084037f6d03773 is the first bad commit commit e9edc188fc76499b0b9bd60364084037f6d03773 Author: Eric Dumazet Date: Fri Sep 17 15:15:56 2021 -0700 netfilter: conntrack: serialize hash resizes and cleanups Syzbot was able to trigger the following warning [1] No repro found by syzbot yet but I was able to trigger similar issue by having 2 scripts running in parallel, changing conntrack hash sizes, and: for j in `seq 1 1000` ; do unshare -n /bin/true >/dev/null ; done It would take more than 5 minutes for net_namespace structures to be cleaned up. This is because nf_ct_iterate_cleanup() has to restart everytime a resize happened. By adding a mutex, we can serialize hash resizes and cleanups and also make get_next_corpse() faster by skipping over empty buckets. Even without resizes in the picture, this patch considerably speeds up network namespace dismantles. [1] INFO: task syz-executor.0:8312 can't die for more than 144 seconds. task:syz-executor.0 state:R running task stack:25672 pid: 8312 ppid: 6573 flags:0x00004006 Call Trace: context_switch kernel/sched/core.c:4955 [inline] __schedule+0x940/0x26f0 kernel/sched/core.c:6236 preempt_schedule_common+0x45/0xc0 kernel/sched/core.c:6408 preempt_schedule_thunk+0x16/0x18 arch/x86/entry/thunk_64.S:35 __local_bh_enable_ip+0x109/0x120 kernel/softirq.c:390 local_bh_enable include/linux/bottom_half.h:32 [inline] get_next_corpse net/netfilter/nf_conntrack_core.c:2252 [inline] nf_ct_iterate_cleanup+0x15a/0x450 net/netfilter/nf_conntrack_core.c:2275 nf_conntrack_cleanup_net_list+0x14c/0x4f0 net/netfilter/nf_conntrack_core.c:2469 ops_exit_list+0x10d/0x160 net/core/net_namespace.c:171 setup_net+0x639/0xa30 net/core/net_namespace.c:349 copy_net_ns+0x319/0x760 net/core/net_namespace.c:470 create_new_namespaces+0x3f6/0xb20 kernel/nsproxy.c:110 unshare_nsproxy_namespaces+0xc1/0x1f0 kernel/nsproxy.c:226 ksys_unshare+0x445/0x920 kernel/fork.c:3128 __do_sys_unshare kernel/fork.c:3202 [inline] __se_sys_unshare kernel/fork.c:3200 [inline] __x64_sys_unshare+0x2d/0x40 kernel/fork.c:3200 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7f63da68e739 RSP: 002b:00007f63d7c05188 EFLAGS: 00000246 ORIG_RAX: 0000000000000110 RAX: ffffffffffffffda RBX: 00007f63da792f80 RCX: 00007f63da68e739 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000040000000 RBP: 00007f63da6e8cc4 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007f63da792f80 R13: 00007fff50b75d3f R14: 00007f63d7c05300 R15: 0000000000022000 Showing all locks held in the system: 1 lock held by khungtaskd/27: #0: ffffffff8b980020 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x53/0x260 kernel/locking/lockdep.c:6446 2 locks held by kworker/u4:2/153: #0: ffff888010c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline] #0: ffff888010c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: arch_atomic_long_set include/linux/atomic/atomic-long.h:41 [inline] #0: ffff888010c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: atomic_long_set include/linux/atomic/atomic-instrumented.h:1198 [inline] #0: ffff888010c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_data kernel/workqueue.c:634 [inline] #0: ffff888010c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: set_work_pool_and_clear_pending kernel/workqueue.c:661 [inline] #0: ffff888010c69138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x896/0x1690 kernel/workqueue.c:2268 #1: ffffc9000140fdb0 ((kfence_timer).work){+.+.}-{0:0}, at: process_one_work+0x8ca/0x1690 kernel/workqueue.c:2272 1 lock held by systemd-udevd/2970: 1 lock held by in:imklog/6258: #0: ffff88807f970ff0 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0xe9/0x100 fs/file.c:990 3 locks held by kworker/1:6/8158: 1 lock held by syz-executor.0/8312: 2 locks held by kworker/u4:13/9320: 1 lock held by syz-executor.5/10178: 1 lock held by syz-executor.4/10217: Signed-off-by: Eric Dumazet Reported-by: syzbot Signed-off-by: Pablo Neira Ayuso net/netfilter/nf_conntrack_core.c | 70 +++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 33 deletions(-) culprit signature: 110d3619d8d3984bda2347b20c2871ac331de897237918dad92478238bdf2554 parent signature: 353bc981a68517a3c2fe8194cec562c9ca10328e68a3c41819149896fcbfe689 Reproducer flagged being flaky revisions tested: 18, total time: 3h43m31.633024775s (build: 1h43m20.925002754s, test: 1h57m42.129950609s) first good commit: e9edc188fc76499b0b9bd60364084037f6d03773 netfilter: conntrack: serialize hash resizes and cleanups recipients (to): ["coreteam@netfilter.org" "davem@davemloft.net" "edumazet@google.com" "fw@strlen.de" "kadlec@netfilter.org" "kuba@kernel.org" "netdev@vger.kernel.org" "netfilter-devel@vger.kernel.org" "pablo@netfilter.org" "pablo@netfilter.org"] recipients (cc): ["linux-kernel@vger.kernel.org"]