ci starts bisection 2023-06-24 07:04:35.560148956 +0000 UTC m=+346941.221635373 bisecting fixing commit since 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 building syzkaller on 9e2ebb3c174f1e168bc1fbadd5f02f2e25e314fc ensuring issue is reproducible on original commit 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 testing commit 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2098d1f918030f38a96b947f1a3936c6ff9f6d8e18c51260ddafdb42338a3d78 run #0: crashed: INFO: task hung in devlink_free run #1: crashed: INFO: task hung in devlink_free run #2: crashed: INFO: task hung in devlink_free run #3: crashed: INFO: task hung in devlink_free run #4: crashed: INFO: task hung in devlink_free run #5: crashed: INFO: task hung in devlink_free run #6: crashed: INFO: task hung in devlink_free run #7: crashed: INFO: task hung in devlink_free run #8: crashed: INFO: task hung in devlink_free run #9: crashed: INFO: task hung in del_device_store run #10: crashed: INFO: task hung in devlink_free run #11: crashed: INFO: task hung in devlink_free run #12: crashed: INFO: task hung in devlink_free run #13: crashed: INFO: task hung in devlink_free run #14: crashed: INFO: task hung in devlink_free run #15: crashed: INFO: task hung in devlink_free run #16: crashed: INFO: task hung in devlink_free run #17: crashed: INFO: task hung in devlink_free run #18: crashed: INFO: task hung in devlink_free run #19: crashed: INFO: task hung in devlink_free testing current HEAD d1d29a42f7acde2fe618cf66b6bfbe047dc51b6b testing commit d1d29a42f7acde2fe618cf66b6bfbe047dc51b6b gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7df0363c2defaae93cae340d5e94f9850606c983a796418d3cbedc2d10dd138f all runs: OK too many neither good nor bad results, skipping this commit # git bisect start d1d29a42f7acde2fe618cf66b6bfbe047dc51b6b 5b7c4cabbb65f5c469464da6c5f614cbd7f730f2 Bisecting: 14181 revisions left to test after this (roughly 14 steps) [c8cc58e289ed3b5bc50258f52776cf3dfa3bad66] Merge tag 'drm-next-2023-04-24' of git://anongit.freedesktop.org/drm/drm testing commit c8cc58e289ed3b5bc50258f52776cf3dfa3bad66 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 7f7ff3f45f041b33856883aa99b462796dbcd5bf7c3c8e16e40621bc3d3ef49a run #0: OK run #1: crashed: INFO: task hung in devlink_free run #2: crashed: INFO: task hung in devlink_free run #3: crashed: INFO: task hung in devlink_free run #4: crashed: INFO: task hung in devlink_free run #5: crashed: INFO: task hung in devlink_free run #6: crashed: INFO: task hung in devlink_free run #7: crashed: INFO: task hung in devlink_free run #8: crashed: INFO: task hung in devlink_free run #9: crashed: INFO: task hung in devlink_free # git bisect good c8cc58e289ed3b5bc50258f52776cf3dfa3bad66 Bisecting: 6955 revisions left to test after this (roughly 13 steps) [7fa8a8ee9400fe8ec188426e40e481717bc5e924] Merge tag 'mm-stable-2023-04-27-15-30' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm testing commit 7fa8a8ee9400fe8ec188426e40e481717bc5e924 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bdc1682139867c0b6117f3d078d68b461b901cdeb9a3a296d82fc05c0d011bd1 all runs: crashed: INFO: task hung in devlink_free # git bisect good 7fa8a8ee9400fe8ec188426e40e481717bc5e924 Bisecting: 3476 revisions left to test after this (roughly 12 steps) [bb7c241fae6228e89c0286ffd6f249b3b0dea225] Merge tag 'ext4_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4 testing commit bb7c241fae6228e89c0286ffd6f249b3b0dea225 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 35ff503c3577ab399c1f10a4733b5bccd0776bce8aa722e9a393ef3134131dd9 all runs: crashed: INFO: task hung in devlink_free # git bisect good bb7c241fae6228e89c0286ffd6f249b3b0dea225 Bisecting: 1738 revisions left to test after this (roughly 11 steps) [692b7dc87ca6d55ab254f8259e6f970171dc9d01] Merge tag 'hyperv-fixes-signed-20230619' of git://git.kernel.org/pub/scm/linux/kernel/git/hyperv/linux testing commit 692b7dc87ca6d55ab254f8259e6f970171dc9d01 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d9ca7addd05c8f47cce777b739cfd19132f4c39f7d892d124d1284ac30fc055c all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad 692b7dc87ca6d55ab254f8259e6f970171dc9d01 Bisecting: 879 revisions left to test after this (roughly 10 steps) [f9010dbdce911ee1f1af1398a24b1f9f992e0080] fork, vhost: Use CLONE_THREAD to fix freezer/ps regression testing commit f9010dbdce911ee1f1af1398a24b1f9f992e0080 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: eb51f0c127333dba56402c2f171a04248dadb9b8a56f20f024cc4d539783e6aa all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad f9010dbdce911ee1f1af1398a24b1f9f992e0080 Bisecting: 420 revisions left to test after this (roughly 9 steps) [a35747c3107ebb8ef2749d4dabaf71c205e0d0fe] Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm testing commit a35747c3107ebb8ef2749d4dabaf71c205e0d0fe gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 545ea8364696e1f9fe39b3e169a0798212f902f4885c2387eb559420b3920127 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad a35747c3107ebb8ef2749d4dabaf71c205e0d0fe Bisecting: 198 revisions left to test after this (roughly 8 steps) [46be92e58fa8868fc10854de94f270e1d58ec434] Merge tag 'sound-6.4-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit 46be92e58fa8868fc10854de94f270e1d58ec434 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f13739339a638c2680ad7909e3b5f9bc998f22aaf1aba923990f18caa0ee8fc8 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad 46be92e58fa8868fc10854de94f270e1d58ec434 Bisecting: 118 revisions left to test after this (roughly 7 steps) [6e42fae0a15519393d3cc5500dc8d84b8549a337] Merge tag 'linux-can-fixes-for-6.4-20230518' of git://git.kernel.org/pub/scm/linux/kernel/git/mkl/linux-can testing commit 6e42fae0a15519393d3cc5500dc8d84b8549a337 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: d8836d855a721e23c18cbb491e6aa88183ea29f941a5691d434ec5b7f28e0564 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad 6e42fae0a15519393d3cc5500dc8d84b8549a337 Bisecting: 59 revisions left to test after this (roughly 6 steps) [c589557dd1426f5adf90c7a919d4fde5a3e4ef64] can: kvaser_pciefd: Empty SRB buffer in probe testing commit c589557dd1426f5adf90c7a919d4fde5a3e4ef64 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ff0f3651479efd4c15d576569d876a92199fd8a0dc5f888b83f8d03a737f0ab5 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad c589557dd1426f5adf90c7a919d4fde5a3e4ef64 Bisecting: 29 revisions left to test after this (roughly 5 steps) [b48a18644046c9bc0667493a147dfa01e8241eab] net: mdio: i2c: fix rollball accessors testing commit b48a18644046c9bc0667493a147dfa01e8241eab gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5187fb81a9e15c8f20b7dd41feab1c27d06dba75a8a943c0c1a2be7e3bb8964a all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad b48a18644046c9bc0667493a147dfa01e8241eab Bisecting: 14 revisions left to test after this (roughly 4 steps) [6d4486efe9c69626cab423456169e250a5cd3af5] vsock: avoid to close connected socket after the timeout testing commit 6d4486efe9c69626cab423456169e250a5cd3af5 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 50359061f84021ab4d4dfc664525425fa10177fa41e49a332e13a73664425f75 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad 6d4486efe9c69626cab423456169e250a5cd3af5 Bisecting: 6 revisions left to test after this (roughly 3 steps) [0b01db274028f5acd207332686ffc92ac77491ac] net: phy: dp83867: add w/a for packet errors seen with short cables testing commit 0b01db274028f5acd207332686ffc92ac77491ac gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2e676eeaa09d7a9aa69e22a45745224d91ddbdbacb087c1ba3cc5538e6d936c3 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad 0b01db274028f5acd207332686ffc92ac77491ac Bisecting: 3 revisions left to test after this (roughly 2 steps) [7ce93d6f91d8871a609f55bce91a507c392805e6] Merge branch 'selftests-seg6-make-srv6_end_dt4_l3vpn_test-more-robust' testing commit 7ce93d6f91d8871a609f55bce91a507c392805e6 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: bfec2cb0fc822269ea4c55c396acc715b2f4c488fa932f8e7f8d1aa828b83acf all runs: crashed: INFO: task hung in devlink_free # git bisect good 7ce93d6f91d8871a609f55bce91a507c392805e6 Bisecting: 1 revision left to test after this (roughly 1 step) [ef1148d4487438a3408d6face2a8360d91b4af70] ipv6: remove nexthop_fib6_nh_bh() testing commit ef1148d4487438a3408d6face2a8360d91b4af70 gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 2929f18eb965a1812ab09578a8fe4cf8f3f6c744db28f43cd2d0f4f225a60f60 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad ef1148d4487438a3408d6face2a8360d91b4af70 Bisecting: 0 revisions left to test after this (roughly 0 steps) [e93c9378e33f68b61ea9318580d841caa22fb9ea] devlink: change per-devlink netdev notifier to static one testing commit e93c9378e33f68b61ea9318580d841caa22fb9ea gcc compiler: gcc (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 49ba6b69d5238f9695652055eea313b8e74c52ac95b153f159a7a12bd7bc49d7 all runs: OK too many neither good nor bad results, skipping this commit # git bisect bad e93c9378e33f68b61ea9318580d841caa22fb9ea e93c9378e33f68b61ea9318580d841caa22fb9ea is the first bad commit commit e93c9378e33f68b61ea9318580d841caa22fb9ea Author: Jiri Pirko Date: Wed May 10 16:46:21 2023 +0200 devlink: change per-devlink netdev notifier to static one The commit 565b4824c39f ("devlink: change port event netdev notifier from per-net to global") changed original per-net notifier to be per-devlink instance. That fixed the issue of non-receiving events of netdev uninit if that moved to a different namespace. That worked fine in -net tree. However, later on when commit ee75f1fc44dd ("net/mlx5e: Create separate devlink instance for ethernet auxiliary device") and commit 72ed5d5624af ("net/mlx5: Suspend auxiliary devices only in case of PCI device suspend") were merged, a deadlock was introduced when removing a namespace with devlink instance with another nested instance. Here there is the bad flow example resulting in deadlock with mlx5: net_cleanup_work -> cleanup_net (takes down_read(&pernet_ops_rwsem) -> devlink_pernet_pre_exit() -> devlink_reload() -> mlx5_devlink_reload_down() -> mlx5_unload_one_devl_locked() -> mlx5_detach_device() -> del_adev() -> mlx5e_remove() -> mlx5e_destroy_devlink() -> devlink_free() -> unregister_netdevice_notifier() (takes down_write(&pernet_ops_rwsem) Steps to reproduce: $ modprobe mlx5_core $ ip netns add ns1 $ devlink dev reload pci/0000:08:00.0 netns ns1 $ ip netns del ns1 Resolve this by converting the notifier from per-devlink instance to a static one registered during init phase and leaving it registered forever. Use this notifier for all devlink port instances created later on. Note what a tree needs this fix only in case all of the cited fixes commits are present. Reported-by: Moshe Shemesh Fixes: 565b4824c39f ("devlink: change port event netdev notifier from per-net to global") Fixes: ee75f1fc44dd ("net/mlx5e: Create separate devlink instance for ethernet auxiliary device") Fixes: 72ed5d5624af ("net/mlx5: Suspend auxiliary devices only in case of PCI device suspend") Signed-off-by: Jiri Pirko Reviewed-by: Simon Horman Link: https://lore.kernel.org/r/20230510144621.932017-1-jiri@resnulli.us Signed-off-by: Jakub Kicinski net/devlink/core.c | 16 +++++++--------- net/devlink/devl_internal.h | 1 - net/devlink/leftover.c | 5 ++--- 3 files changed, 9 insertions(+), 13 deletions(-) culprit signature: 49ba6b69d5238f9695652055eea313b8e74c52ac95b153f159a7a12bd7bc49d7 parent signature: bfec2cb0fc822269ea4c55c396acc715b2f4c488fa932f8e7f8d1aa828b83acf revisions tested: 17, total time: 8h8m40.918417228s (build: 5h42m37.362614874s, test: 2h17m35.674673146s) first good commit: e93c9378e33f68b61ea9318580d841caa22fb9ea devlink: change per-devlink netdev notifier to static one recipients (to): ["jiri@nvidia.com" "kuba@kernel.org" "simon.horman@corigine.com"] recipients (cc): []