bisecting cause commit starting from 304e024216a802a7dc8ba75d36de82fa136bbf3e building syzkaller on a34e2c332411388ed2b3f6f1a3acdc062feceb79 testing commit 304e024216a802a7dc8ba75d36de82fa136bbf3e with gcc (GCC) 8.1.0 kernel signature: 14a861c4e34a518d8436b10c2a3346748502d250ad7acc1d217477c2f7f22fb5 all runs: crashed: WARNING: refcount bug in process_one_work testing release v5.6 testing commit 7111951b8d4973bda27ff663f2cf18b663d15b48 with gcc (GCC) 8.1.0 kernel signature: 782c70f31829739ec1ac49821fa4ed4b2a6abe4cf56cc330707a8a76458e6ef8 all runs: OK # git bisect start 304e024216a802a7dc8ba75d36de82fa136bbf3e 7111951b8d4973bda27ff663f2cf18b663d15b48 Bisecting: 2816 revisions left to test after this (roughly 12 steps) [15c981d16d70e8a5be297fa4af07a64ab7e080ed] Merge tag 'for-5.7-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux testing commit 15c981d16d70e8a5be297fa4af07a64ab7e080ed with gcc (GCC) 8.1.0 kernel signature: 84e80aa45aba6e9bb38baf422cfe22eb40d8d1a76d612f82fd6e2ba637b8b3df all runs: OK # git bisect good 15c981d16d70e8a5be297fa4af07a64ab7e080ed Bisecting: 1408 revisions left to test after this (roughly 11 steps) [4094445229760d0d31a4190dfe88fe815c9fc34e] netfilter: nf_tables: add elements with stateful expressions testing commit 4094445229760d0d31a4190dfe88fe815c9fc34e with gcc (GCC) 8.1.0 kernel signature: 1e738c8e009b3b5279600d93c3db3a0c6807440636f81db6026ea9ff1874e19e all runs: OK # git bisect good 4094445229760d0d31a4190dfe88fe815c9fc34e Bisecting: 701 revisions left to test after this (roughly 10 steps) [aba6d497c8214b81d298f8d5d752a3cd97e8056b] Merge tag 'mlx5-updates-2020-03-29' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux testing commit aba6d497c8214b81d298f8d5d752a3cd97e8056b with gcc (GCC) 8.1.0 kernel signature: 50ff55edca2aca1f17fbe670321ee7d899f39a140c3ba7e2ef98d331858395e8 all runs: OK # git bisect good aba6d497c8214b81d298f8d5d752a3cd97e8056b Bisecting: 317 revisions left to test after this (roughly 9 steps) [1f944f976d7ef8a29d1ad296253d3a9387c58e62] Merge tag 'tty-5.7-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty testing commit 1f944f976d7ef8a29d1ad296253d3a9387c58e62 with gcc (GCC) 8.1.0 kernel signature: 1b5db503ac09704f5a36320cb45f281b39218d0a23e0cb87dac1364548a795c2 all runs: OK # git bisect good 1f944f976d7ef8a29d1ad296253d3a9387c58e62 Bisecting: 135 revisions left to test after this (roughly 7 steps) [ed52f2c608c9451fa2bad298b2ab927416105d65] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next testing commit ed52f2c608c9451fa2bad298b2ab927416105d65 with gcc (GCC) 8.1.0 kernel signature: 70838d7914c011df1d6255ec7fdffb24a47a1a95856e8c125c75f48fd6e8e912 all runs: OK # git bisect good ed52f2c608c9451fa2bad298b2ab927416105d65 Bisecting: 67 revisions left to test after this (roughly 6 steps) [74bef188eac103ecfeaf9cd48b86e12f2a2492af] platform/x86: surface3_power: Use dev_err() instead of pr_err() testing commit 74bef188eac103ecfeaf9cd48b86e12f2a2492af with gcc (GCC) 8.1.0 kernel signature: cae2cc1a461838b1fcb9fdf63844a762d897284842206f79f60b7513d15ef600 all runs: OK # git bisect good 74bef188eac103ecfeaf9cd48b86e12f2a2492af Bisecting: 33 revisions left to test after this (roughly 5 steps) [7f80ccfe996871ca69648efee74a60ae7ad0dcd9] net: ipv6: rpl_iptunnel: Fix potential memory leak in rpl_do_srh_inline testing commit 7f80ccfe996871ca69648efee74a60ae7ad0dcd9 with gcc (GCC) 8.1.0 kernel signature: 902205c2f0842d8a6ff47bc00b142da16159517c5986f2d4b20751707f0674d1 all runs: OK # git bisect good 7f80ccfe996871ca69648efee74a60ae7ad0dcd9 Bisecting: 16 revisions left to test after this (roughly 4 steps) [ae5f4bdccf030d524f995dfc364ea6a96c22882c] NTB: add helper functions to set and clear sideinfo testing commit ae5f4bdccf030d524f995dfc364ea6a96c22882c with gcc (GCC) 8.1.0 kernel signature: 4b3396a9032015703f681501f9deb879bb71a954d29159fb91e69c4bc9e0e8b1 all runs: OK # git bisect good ae5f4bdccf030d524f995dfc364ea6a96c22882c Bisecting: 8 revisions left to test after this (roughly 3 steps) [dba43fc4ba2fed63e898867fa973c69c37623939] Merge tag 'platform-drivers-x86-v5.7-1' of git://git.infradead.org/linux-platform-drivers-x86 testing commit dba43fc4ba2fed63e898867fa973c69c37623939 with gcc (GCC) 8.1.0 kernel signature: 2f64a61c149488f3cc8187229df1cf8a602285a9c35203e11af642a955f7567a all runs: OK # git bisect good dba43fc4ba2fed63e898867fa973c69c37623939 Bisecting: 4 revisions left to test after this (roughly 2 steps) [b350f0a3eb264962caefeb892af56c1b727ee03f] NTB: add pci shutdown handler for AMD NTB testing commit b350f0a3eb264962caefeb892af56c1b727ee03f with gcc (GCC) 8.1.0 kernel signature: 44dc80b3924e5060cc20f8819e494213675cc50593eb9a91f2fdd4a95efbf9a5 all runs: OK # git bisect good b350f0a3eb264962caefeb892af56c1b727ee03f Bisecting: 2 revisions left to test after this (roughly 1 step) [29d9f30d4ce6c7a38745a54a8cddface10013490] Merge git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next testing commit 29d9f30d4ce6c7a38745a54a8cddface10013490 with gcc (GCC) 8.1.0 kernel signature: 758b5b25ed0193e5bc00840c4c13e3b3a031ee1255c36c6a5179cb4556581282 all runs: OK # git bisect good 29d9f30d4ce6c7a38745a54a8cddface10013490 Bisecting: 0 revisions left to test after this (roughly 1 step) [1a323ea5356edbb3073dc59d51b9e6b86908857d] x86: get rid of 'errret' argument to __get_user_xyz() macross testing commit 1a323ea5356edbb3073dc59d51b9e6b86908857d with gcc (GCC) 8.1.0 kernel signature: e003baaa0456f87a821798d7244810cf1977aa780d94c933c52365731dbafeac all runs: OK # git bisect good 1a323ea5356edbb3073dc59d51b9e6b86908857d 304e024216a802a7dc8ba75d36de82fa136bbf3e is the first bad commit commit 304e024216a802a7dc8ba75d36de82fa136bbf3e Author: Cong Wang Date: Sat Mar 28 12:12:59 2020 -0700 net_sched: add a temporary refcnt for struct tcindex_data Although we intentionally use an ordered workqueue for all tc filter works, the ordering is not guaranteed by RCU work, given that tcf_queue_work() is esstenially a call_rcu(). This problem is demostrated by Thomas: CPU 0: tcf_queue_work() tcf_queue_work(&r->rwork, tcindex_destroy_rexts_work); -> Migration to CPU 1 CPU 1: tcf_queue_work(&p->rwork, tcindex_destroy_work); so the 2nd work could be queued before the 1st one, which leads to a free-after-free. Enforcing this order in RCU work is hard as it requires to change RCU code too. Fortunately we can workaround this problem in tcindex filter by taking a temporary refcnt, we only refcnt it right before we begin to destroy it. This simplifies the code a lot as a full refcnt requires much more changes in tcindex_set_parms(). Reported-by: syzbot+46f513c3033d592409d2@syzkaller.appspotmail.com Fixes: 3d210534cc93 ("net_sched: fix a race condition in tcindex_destroy()") Cc: Thomas Gleixner Cc: Paul E. McKenney Cc: Jamal Hadi Salim Cc: Jiri Pirko Signed-off-by: Cong Wang Reviewed-by: Paul E. McKenney Signed-off-by: David S. Miller net/sched/cls_tcindex.c | 44 ++++++++++++++++++++++++++++++++++++++------ 1 file changed, 38 insertions(+), 6 deletions(-) culprit signature: 14a861c4e34a518d8436b10c2a3346748502d250ad7acc1d217477c2f7f22fb5 parent signature: e003baaa0456f87a821798d7244810cf1977aa780d94c933c52365731dbafeac revisions tested: 14, total time: 3h48m54.775218865s (build: 1h24m54.01870597s, test: 2h22m48.99169283s) first bad commit: 304e024216a802a7dc8ba75d36de82fa136bbf3e net_sched: add a temporary refcnt for struct tcindex_data cc: ["davem@davemloft.net" "paulmck@kernel.org" "xiyou.wangcong@gmail.com"] crash: WARNING: refcount bug in process_one_work ------------[ cut here ]------------ refcount_t: underflow; use-after-free. WARNING: CPU: 1 PID: 848 at lib/refcount.c:28 refcount_warn_saturate+0x12b/0x140 lib/refcount.c:28 Kernel panic - not syncing: panic_on_warn set ... CPU: 1 PID: 848 Comm: kworker/u4:4 Not tainted 5.6.0-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Workqueue: tc_filter_workqueue tcindex_destroy_work Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 panic+0x22a/0x4e3 kernel/panic.c:221 __warn.cold.10+0x25/0x26 kernel/panic.c:582 report_bug+0x1ad/0x270 lib/bug.c:195 fixup_bug arch/x86/kernel/traps.c:175 [inline] do_error_trap+0x123/0x210 arch/x86/kernel/traps.c:267 do_invalid_op+0x31/0x40 arch/x86/kernel/traps.c:286 invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027 RIP: 0010:refcount_warn_saturate+0x12b/0x140 lib/refcount.c:28 Code: 0a f7 fd 0f 0b e9 53 ff ff ff 48 89 df e8 3d 3c 55 fe e9 23 ff ff ff 48 c7 c7 40 ef ad 87 c6 05 47 07 51 06 01 e8 71 0a f7 fd <0f> 0b e9 2c ff ff ff 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 41 RSP: 0018:ffffc90003497d68 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ffff88809b83342c RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000008 RDI: ffffffff8b6993a0 RBP: 0000000000000003 R08: ffffed1015d266a1 R09: ffffed1015d266a1 R10: ffffed1015d266a0 R11: ffff8880ae933507 R12: ffffc90003497e00 R13: ffff88809b833430 R14: 0000000000000001 R15: ffff8880aa434858 process_one_work+0x908/0x15d0 kernel/workqueue.c:2266 worker_thread+0x82/0xb50 kernel/workqueue.c:2412 kthread+0x340/0x410 kernel/kthread.c:268 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352 Kernel Offset: disabled Rebooting in 86400 seconds..