bisecting cause commit starting from cd607737f3b84432b8a00cd31ac753dab1c38678 building syzkaller on 2c31c529a9a44be5d99e769204b7a4b84b93eec1 testing commit cd607737f3b84432b8a00cd31ac753dab1c38678 with gcc (GCC) 8.1.0 kernel signature: 7cb18d7675b09086b1a283de8ad191a0ae63e6d9da4bfb30ce099844e8f68992 run #0: crashed: INFO: task hung in io_queue_file_removal run #1: crashed: INFO: task hung in io_queue_file_removal run #2: crashed: INFO: task hung in io_queue_file_removal run #3: crashed: INFO: task hung in io_queue_file_removal run #4: crashed: INFO: task hung in io_queue_file_removal run #5: crashed: INFO: task hung in io_queue_file_removal run #6: crashed: INFO: task hung in io_queue_file_removal run #7: crashed: INFO: task hung in io_queue_file_removal run #8: crashed: INFO: task hung in io_queue_file_removal run #9: boot failed: can't ssh into the instance testing release v5.5 testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0 kernel signature: e9b7dc3efc24ad651452e2dc3dd7eb6258d8b2587cbc26e495f366ace1efea22 all runs: OK # git bisect start cd607737f3b84432b8a00cd31ac753dab1c38678 d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 Bisecting: 5653 revisions left to test after this (roughly 13 steps) [9f68e3655aae6d49d6ba05dd263f99f33c2567af] Merge tag 'drm-next-2020-01-30' of git://anongit.freedesktop.org/drm/drm testing commit 9f68e3655aae6d49d6ba05dd263f99f33c2567af with gcc (GCC) 8.1.0 kernel signature: 472f1a0c010447b1b7383f9f9c2f7358549a091800880ac264fb82b8ea0e3f7f run #0: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #1: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #2: crashed: WARNING in percpu_ref_exit run #3: crashed: WARNING in percpu_ref_exit run #4: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #5: crashed: WARNING in percpu_ref_exit run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 9f68e3655aae6d49d6ba05dd263f99f33c2567af Bisecting: 3686 revisions left to test after this (roughly 12 steps) [fb95aae6e67c4e319a24b3eea32032d4246a5335] Merge tag 'sound-5.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit fb95aae6e67c4e319a24b3eea32032d4246a5335 with gcc (GCC) 8.1.0 kernel signature: 95a660c02af09450f8467e56afef5beeac3dcd669b41f4a6b07e1aac453c8588 all runs: OK # git bisect good fb95aae6e67c4e319a24b3eea32032d4246a5335 Bisecting: 1843 revisions left to test after this (roughly 11 steps) [8815a94f27d2f30fe1216ce10c7da0f6ae69ca0f] drm/vmwgfx: move the require_exist handling together testing commit 8815a94f27d2f30fe1216ce10c7da0f6ae69ca0f with gcc (GCC) 8.1.0 kernel signature: 574ce1a00fec2981fa2007434a2896fe5a3a18a6bce3fd67cc7ce8ffb79f4eb7 all runs: boot failed: general protection fault in do_mount_root # git bisect skip 8815a94f27d2f30fe1216ce10c7da0f6ae69ca0f Bisecting: 1842 revisions left to test after this (roughly 11 steps) [4872e6aa217fbb475ffa0ad7bda0d9acff543f2c] drm/vmwgfx: check master authentication in surface_ref ioctls testing commit 4872e6aa217fbb475ffa0ad7bda0d9acff543f2c with gcc (GCC) 8.1.0 kernel signature: d168e67e16158ae4923876127531227323857ceb81cdc04a6b3f1936f80100f3 all runs: boot failed: general protection fault in do_mount_root # git bisect skip 4872e6aa217fbb475ffa0ad7bda0d9acff543f2c Bisecting: 1842 revisions left to test after this (roughly 11 steps) [fa889d85551e0bd962fdefe1cc113f9ba1d04a36] Merge tag 'gpio-v5.6-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-gpio testing commit fa889d85551e0bd962fdefe1cc113f9ba1d04a36 with gcc (GCC) 8.1.0 kernel signature: 24709ce3f5baeec97ef49ecd8567d9897a824eb24943d556519b4d636973e953 all runs: OK # git bisect good fa889d85551e0bd962fdefe1cc113f9ba1d04a36 Bisecting: 1827 revisions left to test after this (roughly 11 steps) [dd22dfa62c9cb2669ed4b181e359645108c69578] Merge branch 'linux-5.6' of git://github.com/skeggsb/linux into drm-next testing commit dd22dfa62c9cb2669ed4b181e359645108c69578 with gcc (GCC) 8.1.0 kernel signature: 15f216313339c97daa4f7dda0fd530c47116e7915973b2e9b87d8ab226b5f365 all runs: boot failed: general protection fault in do_mount_root # git bisect skip dd22dfa62c9cb2669ed4b181e359645108c69578 Bisecting: 1827 revisions left to test after this (roughly 11 steps) [1d47d0bb72895e754ffbdc410314ddb9c790c6fa] fbdev: omapfb: use devm_platform_ioremap_resource() to simplify code testing commit 1d47d0bb72895e754ffbdc410314ddb9c790c6fa with gcc (GCC) 8.1.0 kernel signature: 638229a437608f73ffb0a24d392e9458ef5df4422b0692d14c48013f63c13052 all runs: OK # git bisect good 1d47d0bb72895e754ffbdc410314ddb9c790c6fa Bisecting: 1573 revisions left to test after this (roughly 11 steps) [3d4743131b8de970faa4b979ead0fadfe5d2de9d] Backmerge v5.5-rc7 into drm-next testing commit 3d4743131b8de970faa4b979ead0fadfe5d2de9d with gcc (GCC) 8.1.0 kernel signature: 74ae4ac3784b64d74fdc5bcb57f53a691f74ad9c3a41eebae59ad86d99467536 all runs: OK # git bisect good 3d4743131b8de970faa4b979ead0fadfe5d2de9d Bisecting: 751 revisions left to test after this (roughly 10 steps) [7ba31c3f2f1ee095d8126f4d3757fc3b2bc3c838] Merge tag 'staging-5.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging testing commit 7ba31c3f2f1ee095d8126f4d3757fc3b2bc3c838 with gcc (GCC) 8.1.0 kernel signature: f4dcd6a4f27b3a7b232fd9c8a65820fbdd1b149641c1a446bd31d8862fb27998 all runs: OK # git bisect good 7ba31c3f2f1ee095d8126f4d3757fc3b2bc3c838 Bisecting: 303 revisions left to test after this (roughly 9 steps) [33c84e89abe4a92ab699c33029bd54269d574782] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi testing commit 33c84e89abe4a92ab699c33029bd54269d574782 with gcc (GCC) 8.1.0 kernel signature: cde5fad18946aa2327da81bec9bb755573a55e1196d44629a0664a5dc10ee465 all runs: OK # git bisect good 33c84e89abe4a92ab699c33029bd54269d574782 Bisecting: 119 revisions left to test after this (roughly 7 steps) [893e591b59036f9bc629f55bce715d67bdd266a2] Merge tag 'devicetree-for-5.6' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux testing commit 893e591b59036f9bc629f55bce715d67bdd266a2 with gcc (GCC) 8.1.0 kernel signature: 4aca313fb7b4d9ad6f15c7c9867a32cc868509e73ad00ddbeab19ab0ba7218f4 run #0: crashed: WARNING in percpu_ref_exit run #1: crashed: WARNING in percpu_ref_exit run #2: crashed: WARNING in percpu_ref_exit run #3: crashed: WARNING in percpu_ref_exit run #4: OK run #5: crashed: WARNING in percpu_ref_exit run #6: OK run #7: OK run #8: OK run #9: OK # git bisect bad 893e591b59036f9bc629f55bce715d67bdd266a2 Bisecting: 95 revisions left to test after this (roughly 7 steps) [9ca4c6429f92598a84e4c3292ea7d187c9d7b033] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc testing commit 9ca4c6429f92598a84e4c3292ea7d187c9d7b033 with gcc (GCC) 8.1.0 kernel signature: 0f7d0d258cef929e2a1b85c18cfe1f2e557e4a8c9fe49ebbb287763a61133a2d run #0: crashed: WARNING in percpu_ref_exit run #1: crashed: WARNING in percpu_ref_exit run #2: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #3: crashed: WARNING in percpu_ref_exit run #4: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #5: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #6: crashed: WARNING in percpu_ref_exit run #7: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister run #8: crashed: WARNING in percpu_ref_exit run #9: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister # git bisect bad 9ca4c6429f92598a84e4c3292ea7d187c9d7b033 Bisecting: 43 revisions left to test after this (roughly 6 steps) [66f4af93da5761d2fa05c0dc673a47003cdb9cfe] io_uring: add support for probing opcodes testing commit 66f4af93da5761d2fa05c0dc673a47003cdb9cfe with gcc (GCC) 8.1.0 kernel signature: cfc480731f5f472221baf11be4cc0f53be11ad5e38e49ca5db55e1ee5249a9d8 all runs: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister # git bisect bad 66f4af93da5761d2fa05c0dc673a47003cdb9cfe Bisecting: 21 revisions left to test after this (roughly 5 steps) [4840e418c2fc533d55ff6caa5b9313eed1d26cfd] io_uring: add IORING_OP_FADVISE testing commit 4840e418c2fc533d55ff6caa5b9313eed1d26cfd with gcc (GCC) 8.1.0 kernel signature: 03b1fe50fce9a420e1bda50f366fa1c5b7f18cdab26fd49e0590a8398c4dd7ea all runs: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister # git bisect bad 4840e418c2fc533d55ff6caa5b9313eed1d26cfd Bisecting: 10 revisions left to test after this (roughly 4 steps) [eddc7ef52a6b37b7ba3d1c8a8fbb63d5d9914f8a] io_uring: add support for IORING_OP_STATX testing commit eddc7ef52a6b37b7ba3d1c8a8fbb63d5d9914f8a with gcc (GCC) 8.1.0 kernel signature: c8912d8e1cee190f074d8cfe3749537b130052b472f981189357c3a589596cf3 all runs: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister # git bisect bad eddc7ef52a6b37b7ba3d1c8a8fbb63d5d9914f8a Bisecting: 5 revisions left to test after this (roughly 3 steps) [15b71abe7b52df214785dde0de9f581cc0216d17] io_uring: add support for IORING_OP_OPENAT testing commit 15b71abe7b52df214785dde0de9f581cc0216d17 with gcc (GCC) 8.1.0 kernel signature: 435f0c84276db077db1edb9ea3b9833665c9d0e5c712ef0087ffe5b67aa65839 all runs: OK # git bisect good 15b71abe7b52df214785dde0de9f581cc0216d17 Bisecting: 2 revisions left to test after this (roughly 2 steps) [b5dba59e0cf7e2cc4d3b3b1ac5fe81ddf21959eb] io_uring: add support for IORING_OP_CLOSE testing commit b5dba59e0cf7e2cc4d3b3b1ac5fe81ddf21959eb with gcc (GCC) 8.1.0 kernel signature: b1e072bdf5fe88dd79fafcc4c5ab139b13d5dffe54b4a1e2c497f29661788a09 all runs: OK # git bisect good b5dba59e0cf7e2cc4d3b3b1ac5fe81ddf21959eb Bisecting: 1 revision left to test after this (roughly 1 step) [05f3fb3c5397524feae2e73ee8e150a9090a7da2] io_uring: avoid ring quiesce for fixed file set unregister and update testing commit 05f3fb3c5397524feae2e73ee8e150a9090a7da2 with gcc (GCC) 8.1.0 kernel signature: 3d5abe3cf1765349fb52ee188d073e61c500f9b3d00cbacc2936f26932e60889 all runs: crashed: WARNING: ODEBUG bug in io_sqe_files_unregister # git bisect bad 05f3fb3c5397524feae2e73ee8e150a9090a7da2 05f3fb3c5397524feae2e73ee8e150a9090a7da2 is the first bad commit commit 05f3fb3c5397524feae2e73ee8e150a9090a7da2 Author: Jens Axboe Date: Mon Dec 9 11:22:50 2019 -0700 io_uring: avoid ring quiesce for fixed file set unregister and update We currently fully quiesce the ring before an unregister or update of the fixed fileset. This is very expensive, and we can be a bit smarter about this. Add a percpu refcount for the file tables as a whole. Grab a percpu ref when we use a registered file, and put it on completion. This is cheap to do. Upon removal of a file from a set, switch the ref count to atomic mode. When we hit zero ref on the completion side, then we know we can drop the previously registered files. When the old files have been dropped, switch the ref back to percpu mode for normal operation. Since there's a period between doing the update and the kernel being done with it, add a IORING_OP_FILES_UPDATE opcode that can perform the same action. The application knows the update has completed when it gets the CQE for it. Between doing the update and receiving this completion, the application must continue to use the unregistered fd if submitting IO on this particular file. This takes the runtime of test/file-register from liburing from 14s to about 0.7s. Signed-off-by: Jens Axboe fs/io_uring.c | 485 ++++++++++++++++++++++++++++++------------ include/uapi/linux/io_uring.h | 1 + 2 files changed, 351 insertions(+), 135 deletions(-) culprit signature: 3d5abe3cf1765349fb52ee188d073e61c500f9b3d00cbacc2936f26932e60889 parent signature: b1e072bdf5fe88dd79fafcc4c5ab139b13d5dffe54b4a1e2c497f29661788a09 revisions tested: 20, total time: 5h25m17.096228923s (build: 2h13m53.233709894s, test: 3h9m45.342326322s) first bad commit: 05f3fb3c5397524feae2e73ee8e150a9090a7da2 io_uring: avoid ring quiesce for fixed file set unregister and update cc: ["axboe@kernel.dk" "io-uring@vger.kernel.org" "linux-fsdevel@vger.kernel.org" "linux-kernel@vger.kernel.org" "viro@zeniv.linux.org.uk"] crash: WARNING: ODEBUG bug in io_sqe_files_unregister ------------[ cut here ]------------ ODEBUG: free active (active state 0) object type: work_struct hint: io_ring_file_ref_switch+0x0/0xbc0 include/linux/refcount.h:191 WARNING: CPU: 0 PID: 8343 at lib/debugobjects.c:484 debug_print_object+0x160/0x210 lib/debugobjects.c:481 Kernel panic - not syncing: panic_on_warn set ... CPU: 0 PID: 8343 Comm: syz-executor.1 Not tainted 5.5.0-rc7-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x128/0x182 lib/dump_stack.c:118 panic+0x22a/0x4e3 kernel/panic.c:221 __warn.cold.10+0x25/0x26 kernel/panic.c:582 report_bug+0x1ad/0x270 lib/bug.c:195 fixup_bug arch/x86/kernel/traps.c:174 [inline] do_error_trap+0x123/0x210 arch/x86/kernel/traps.c:267 do_invalid_op+0x31/0x40 arch/x86/kernel/traps.c:286 invalid_op+0x23/0x30 arch/x86/entry/entry_64.S:1027 RIP: 0010:debug_print_object+0x160/0x210 lib/debugobjects.c:481 Code: 8d 87 48 89 fa 48 c1 ea 03 80 3c 02 00 0f 85 96 00 00 00 48 8b 14 dd a0 5a 8d 87 4c 89 f6 48 c7 c7 00 50 8d 87 e8 3c 7e fb fd <0f> 0b 83 05 2b d4 2d 06 01 48 83 c4 18 5b 5d 41 5c 41 5d 41 5e c3 RSP: 0000:ffffc90002d37c80 EFLAGS: 00010086 RAX: 0000000000000000 RBX: 0000000000000003 RCX: 0000000000000000 RDX: 0000000000000001 RSI: 0000000000000007 RDI: ffffffff8b3d6360 RBP: 0000000000000001 R08: ffffed1015d045c9 R09: ffffed1015d045c9 R10: ffffed1015d045c8 R11: ffff8880ae822e43 R12: ffffffff88969c00 R13: ffffffff814242c0 R14: ffffffff878d5700 R15: ffff88809e34ee58 __debug_check_no_obj_freed lib/debugobjects.c:963 [inline] debug_check_no_obj_freed+0x2de/0x454 lib/debugobjects.c:994 kfree+0xf4/0x2b0 mm/slab.c:3756 io_sqe_files_unregister+0x2ca/0x520 fs/io_uring.c:4452 io_ring_ctx_free fs/io_uring.c:5397 [inline] io_ring_ctx_wait_and_kill+0x391/0x8b0 fs/io_uring.c:5466 io_uring_release+0x39/0x50 fs/io_uring.c:5474 __fput+0x256/0x780 fs/file_table.c:280 task_work_run+0x103/0x180 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:188 [inline] exit_to_usermode_loop+0x23d/0x2d0 arch/x86/entry/common.c:164 prepare_exit_to_usermode arch/x86/entry/common.c:195 [inline] syscall_return_slowpath arch/x86/entry/common.c:278 [inline] do_syscall_64+0x4f8/0x5e0 arch/x86/entry/common.c:304 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x4163e1 Code: 75 14 b8 03 00 00 00 0f 05 48 3d 01 f0 ff ff 0f 83 04 1b 00 00 c3 48 83 ec 08 e8 0a fc ff ff 48 89 04 24 b8 03 00 00 00 0f 05 <48> 8b 3c 24 48 89 c2 e8 53 fc ff ff 48 89 d0 48 83 c4 08 48 3d 01 RSP: 002b:00007ffc9df63c80 EFLAGS: 00000293 ORIG_RAX: 0000000000000003 RAX: 0000000000000000 RBX: 0000000000000004 RCX: 00000000004163e1 RDX: 0000001b2ee20000 RSI: 0000000000000001 RDI: 0000000000000003 RBP: 0000000000000001 R08: 00ffffffffffffff R09: 00ffffffffffffff R10: 00007ffc9df63d60 R11: 0000000000000293 R12: 000000000076c900 R13: 000000000076c900 R14: 0000000000012d83 R15: 000000000076bf0c Kernel Offset: disabled Rebooting in 86400 seconds..