bisecting cause commit starting from e21a712a9685488f5ce80495b37b9fdbe96c230d building syzkaller on 6affd8e838ce8a0c7d72445a7f67fe3bde8bbe04 testing commit e21a712a9685488f5ce80495b37b9fdbe96c230d with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in blk_mq_free_request run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: WARNING: refcount bug in blk_mq_free_request run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: general protection fault in dd_insert_requests run #9: crashed: WARNING: refcount bug in blk_mq_free_request testing release v5.2 testing commit 0ecfebd2b52404ae0c54a878c872bb93363ada36 with gcc (GCC) 8.1.0 all runs: crashed: WARNING: refcount bug in blk_mq_free_request testing release v5.1 testing commit e93c9c99a629c61837d5a7fc2120cd2b6c70dbdd with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in blk_mq_free_request run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: general protection fault in dd_insert_requests run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING: refcount bug in blk_mq_free_request testing release v5.0 testing commit 1c163f4c7b3f621efff9b28a47abb36f7378d783 with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in blk_mq_free_request run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: general protection fault in dd_insert_requests run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING: refcount bug in blk_mq_free_request testing release v4.20 testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING in corrupted run #2: crashed: WARNING in corrupted run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING in corrupted run #5: crashed: WARNING: refcount bug in corrupted run #6: crashed: WARNING: refcount bug in corrupted run #7: crashed: BUG: corrupted list in blk_mq_add_to_requeue_list run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING: refcount bug in blk_mq_free_request testing release v4.19 testing commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING in blk_mq_bio_to_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING in blk_mq_bio_to_request run #5: crashed: WARNING: refcount bug in corrupted run #6: crashed: WARNING: refcount bug in blk_mq_free_request run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: BUG: corrupted list in blk_mq_add_to_requeue_list testing release v4.18 testing commit 94710cac0ef4ee177a63b5227664b38c95bbf703 with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in corrupted run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING in blk_mq_bio_to_request run #4: crashed: WARNING: refcount bug in corrupted run #5: crashed: WARNING in corrupted run #6: crashed: WARNING: refcount bug in blk_mq_free_request run #7: crashed: WARNING in blk_mq_bio_to_request run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING: refcount bug in blk_mq_free_request testing release v4.17 testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0 all runs: OK # git bisect start v4.18 v4.17 Bisecting: 7032 revisions left to test after this (roughly 13 steps) [3036bc45364f98515a2c446d7fac2c34dcfbeff4] Merge tag 'media/v4.18-2' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media testing commit 3036bc45364f98515a2c446d7fac2c34dcfbeff4 with gcc (GCC) 8.1.0 run #0: crashed: WARNING in corrupted run #1: crashed: WARNING in corrupted run #2: crashed: WARNING in corrupted run #3: crashed: WARNING: refcount bug in corrupted run #4: crashed: WARNING: refcount bug in corrupted run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: boot failed: KASAN: use-after-free Write in call_usermodehelper_exec_work run #7: boot failed: KASAN: use-after-free Write in call_usermodehelper_exec_work run #8: boot failed: KASAN: use-after-free Write in call_usermodehelper_exec_work run #9: boot failed: KASAN: use-after-free Write in call_usermodehelper_exec_work # git bisect bad 3036bc45364f98515a2c446d7fac2c34dcfbeff4 Bisecting: 3644 revisions left to test after this (roughly 12 steps) [135c5504a600ff9b06e321694fbcac78a9530cd4] Merge tag 'drm-next-2018-06-06-1' of git://anongit.freedesktop.org/drm/drm testing commit 135c5504a600ff9b06e321694fbcac78a9530cd4 with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in corrupted run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in blk_mq_free_request run #5: crashed: WARNING in blk_mq_sched_insert_request run #6: crashed: WARNING: refcount bug in blk_mq_free_request run #7: crashed: WARNING in corrupted run #8: crashed: WARNING in corrupted run #9: crashed: WARNING: refcount bug in blk_mq_free_request # git bisect bad 135c5504a600ff9b06e321694fbcac78a9530cd4 Bisecting: 1901 revisions left to test after this (roughly 11 steps) [5231804cf9e584f3e7e763a0d6d2fffe011c1bce] Merge tag 'leds_for_4.18-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/j.anaszewski/linux-leds testing commit 5231804cf9e584f3e7e763a0d6d2fffe011c1bce with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in corrupted run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in corrupted run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: WARNING in blk_mq_bio_to_request run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING in corrupted run #9: crashed: WARNING in corrupted # git bisect bad 5231804cf9e584f3e7e763a0d6d2fffe011c1bce Bisecting: 914 revisions left to test after this (roughly 10 steps) [4057adafb395204af4ff93f3669ecb49eb45b3cf] Merge branch 'core-rcu-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip testing commit 4057adafb395204af4ff93f3669ecb49eb45b3cf with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in blk_mq_free_request run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: WARNING: refcount bug in blk_mq_free_request run #7: crashed: WARNING in corrupted run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING in corrupted # git bisect bad 4057adafb395204af4ff93f3669ecb49eb45b3cf Bisecting: 433 revisions left to test after this (roughly 9 steps) [eeee3149aaa022145b2659e3b0601dc705d69402] Merge tag 'docs-4.18' of git://git.lwn.net/linux testing commit eeee3149aaa022145b2659e3b0601dc705d69402 with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in corrupted run #1: crashed: WARNING in corrupted run #2: crashed: WARNING in blk_mq_bio_to_request run #3: crashed: WARNING in corrupted run #4: crashed: WARNING in corrupted run #5: crashed: WARNING in blk_mq_bio_to_request run #6: crashed: WARNING in corrupted run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING: refcount bug in corrupted run #9: crashed: WARNING: refcount bug in corrupted # git bisect bad eeee3149aaa022145b2659e3b0601dc705d69402 Bisecting: 249 revisions left to test after this (roughly 8 steps) [cf626b0da78df6669c6b5f51ddd9a70a0702e579] Merge branch 'hch.procfs' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs testing commit cf626b0da78df6669c6b5f51ddd9a70a0702e579 with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_free_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in blk_mq_free_request run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: WARNING: refcount bug in corrupted run #7: crashed: WARNING in blk_mq_bio_to_request run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING: refcount bug in blk_mq_free_request # git bisect bad cf626b0da78df6669c6b5f51ddd9a70a0702e579 Bisecting: 126 revisions left to test after this (roughly 7 steps) [ab4f47a9f4a12603a1806230d44ead2e54158f85] nvme: allow duplicate controller if prior controller being deleted testing commit ab4f47a9f4a12603a1806230d44ead2e54158f85 with gcc (GCC) 8.1.0 run #0: crashed: WARNING: refcount bug in blk_mq_check_expired run #1: crashed: WARNING in blk_mq_bio_to_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING in corrupted run #4: crashed: WARNING: refcount bug in corrupted run #5: crashed: WARNING: refcount bug in blk_mq_free_request run #6: crashed: WARNING in corrupted run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING: refcount bug in corrupted run #9: crashed: WARNING: refcount bug in blk_mq_free_request # git bisect bad ab4f47a9f4a12603a1806230d44ead2e54158f85 Bisecting: 64 revisions left to test after this (roughly 6 steps) [d416c92c5d6229b33f37f0f75e52194081ccbcc4] blk-mq: clear hctx->dispatch_from when mappings change testing commit d416c92c5d6229b33f37f0f75e52194081ccbcc4 with gcc (GCC) 8.1.0 all runs: OK # git bisect good d416c92c5d6229b33f37f0f75e52194081ccbcc4 Bisecting: 32 revisions left to test after this (roughly 5 steps) [d5eff33ee6f80864d889d64fd3f6ea7b78dd1b24] nvmet: add simple file backed ns support testing commit d5eff33ee6f80864d889d64fd3f6ea7b78dd1b24 with gcc (GCC) 8.1.0 all runs: OK # git bisect good d5eff33ee6f80864d889d64fd3f6ea7b78dd1b24 Bisecting: 16 revisions left to test after this (roughly 4 steps) [c5fb85b7ff7648a8b51af2062b9242a83d513b68] mtip32xx: complete requests from ->timeout testing commit c5fb85b7ff7648a8b51af2062b9242a83d513b68 with gcc (GCC) 8.1.0 run #0: crashed: kernel BUG at block/blk-mq.c:LINE! run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING in corrupted run #3: crashed: WARNING: refcount bug in corrupted run #4: crashed: WARNING in corrupted run #5: crashed: WARNING in corrupted run #6: crashed: WARNING: refcount bug in corrupted run #7: crashed: WARNING: refcount bug in blk_mq_free_request run #8: crashed: WARNING: refcount bug in blk_mq_free_request run #9: crashed: WARNING: refcount bug in corrupted # git bisect bad c5fb85b7ff7648a8b51af2062b9242a83d513b68 Bisecting: 7 revisions left to test after this (roughly 3 steps) [ecb37ce9baac653cc09e2b631393dde3df82979f] bcache: Move couple of functions to sysfs.c testing commit ecb37ce9baac653cc09e2b631393dde3df82979f with gcc (GCC) 8.1.0 all runs: OK # git bisect good ecb37ce9baac653cc09e2b631393dde3df82979f Bisecting: 3 revisions left to test after this (roughly 2 steps) [12f5b93145450c750f315657ef239a314811aeeb] blk-mq: Remove generation seqeunce testing commit 12f5b93145450c750f315657ef239a314811aeeb with gcc (GCC) 8.1.0 run #0: crashed: WARNING in blk_mq_bio_to_request run #1: crashed: WARNING: refcount bug in blk_mq_free_request run #2: crashed: WARNING: refcount bug in blk_mq_free_request run #3: crashed: WARNING: refcount bug in blk_mq_free_request run #4: crashed: WARNING: refcount bug in corrupted run #5: crashed: WARNING in blk_mq_bio_to_request run #6: crashed: WARNING: refcount bug in corrupted run #7: crashed: WARNING in blk_mq_bio_to_request run #8: crashed: WARNING: refcount bug in corrupted run #9: crashed: WARNING in blk_mq_bio_to_request # git bisect bad 12f5b93145450c750f315657ef239a314811aeeb Bisecting: 1 revision left to test after this (roughly 1 step) [01fc27d96950149c3e6c0b8dfbe05e26725381cb] libata: remove ata_scsi_timed_out testing commit 01fc27d96950149c3e6c0b8dfbe05e26725381cb with gcc (GCC) 8.1.0 all runs: OK # git bisect good 01fc27d96950149c3e6c0b8dfbe05e26725381cb Bisecting: 0 revisions left to test after this (roughly 0 steps) [ad103e79838d4b4cd4d6dd0bdfaef937e8652ae9] blk-mq: Fix timeout and state order testing commit ad103e79838d4b4cd4d6dd0bdfaef937e8652ae9 with gcc (GCC) 8.1.0 all runs: OK # git bisect good ad103e79838d4b4cd4d6dd0bdfaef937e8652ae9 12f5b93145450c750f315657ef239a314811aeeb is the first bad commit commit 12f5b93145450c750f315657ef239a314811aeeb Author: Keith Busch Date: Tue May 29 15:52:28 2018 +0200 blk-mq: Remove generation seqeunce This patch simplifies the timeout handling by relying on the request reference counting to ensure the iterator is operating on an inflight and truly timed out request. Since the reference counting prevents the tag from being reallocated, the block layer no longer needs to prevent drivers from completing their requests while the timeout handler is operating on it: a driver completing a request is allowed to proceed to the next state without additional syncronization with the block layer. This also removes any need for generation sequence numbers since the request lifetime is prevented from being reallocated as a new sequence while timeout handling is operating on it. To enables this a refcount is added to struct request so that request users can be sure they're operating on the same request without it changing while they're processing it. The request's tag won't be released for reuse until both the timeout handler and the completion are done with it. Signed-off-by: Keith Busch [hch: slight cleanups, added back submission side hctx lock, use cmpxchg for completions] Signed-off-by: Christoph Hellwig Signed-off-by: Jens Axboe :040000 040000 a03c5a01a8c326797f47b51a84217e96a048a2fb 060c50c7ddfe297923b9d406f50e1cffdd1eaccd M block :040000 040000 dbbe09bc48d2e2b8cb96e9f1f82747c0aade5de6 8ec17df25eacdef236790e24cec3beace1ac94f9 M include revisions tested: 22, total time: 5h30m17.150798803s (build: 2h2m53.037488839s, test: 3h20m30.318910658s) first bad commit: 12f5b93145450c750f315657ef239a314811aeeb blk-mq: Remove generation seqeunce cc: ["axboe@kernel.dk" "hch@lst.de" "keith.busch@intel.com" "linux-block@vger.kernel.org" "linux-kernel@vger.kernel.org"] crash: WARNING in blk_mq_bio_to_request block nbd5: Connection timed out print_req_error: I/O error, dev nbd5, sector 0 block nbd5: Connection timed out ldm_validate_partition_table(): Disk read failed. ------------[ cut here ]------------ WARNING: CPU: 1 PID: 18552 at include/linux/blk-cgroup.h:354 blkg_lookup include/linux/blk-cgroup.h:291 [inline] WARNING: CPU: 1 PID: 18552 at include/linux/blk-cgroup.h:354 blk_get_rl include/linux/blk-cgroup.h:432 [inline] WARNING: CPU: 1 PID: 18552 at include/linux/blk-cgroup.h:354 blk_mq_bio_to_request+0x3b9/0x4f0 block/blk-mq.c:1621 Kernel panic - not syncing: panic_on_warn set ... CPU: 1 PID: 18552 Comm: syz-executor.5 Not tainted 4.17.0-rc4+ #1 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x109/0x15a lib/dump_stack.c:113 panic+0x1c6/0x36b kernel/panic.c:184 __warn.cold.8+0x120/0x168 kernel/panic.c:536 report_bug+0x1a4/0x200 lib/bug.c:186 fixup_bug arch/x86/kernel/traps.c:178 [inline] do_error_trap+0x1df/0x330 arch/x86/kernel/traps.c:296 do_invalid_op+0x1b/0x20 arch/x86/kernel/traps.c:315 invalid_op+0x14/0x20 arch/x86/entry/entry_64.S:992 RIP: 0010:blkg_get include/linux/blk-cgroup.h:354 [inline] RIP: 0010:blk_get_rl include/linux/blk-cgroup.h:436 [inline] RIP: 0010:blk_mq_bio_to_request+0x3b9/0x4f0 block/blk-mq.c:1621 RSP: 0018:ffff8800a59aecb8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8800a4081a40 RCX: ffffffff82d86cb7 RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff880099ff4d00 RBP: ffff8800a59aece0 R08: ffffed00133fe9a1 R09: ffffed00133fe9a0 R10: ffffed00133fe9a0 R11: ffff880099ff4d03 R12: ffff880099ff4c00 R13: ffff880099ff4d00 R14: ffff8800a40184c0 R15: 0000000000000000 blk_mq_make_request+0x518/0x2060 block/blk-mq.c:1852 generic_make_request+0x50d/0xc20 block/blk-core.c:2447 submit_bio+0x9f/0x400 block/blk-core.c:2555 submit_bh_wbc+0x4d1/0x700 fs/buffer.c:3076 submit_bh fs/buffer.c:3082 [inline] block_read_full_page+0x7b2/0xc30 fs/buffer.c:2287 blkdev_readpage+0x13/0x20 fs/block_dev.c:571 do_read_cache_page+0x5c6/0xf80 mm/filemap.c:2805 read_cache_page+0x4b/0x80 mm/filemap.c:2893 read_mapping_page include/linux/pagemap.h:402 [inline] read_dev_sector+0xc0/0x2e0 block/partition-generic.c:657 read_part_sector block/partitions/check.h:38 [inline] adfspart_check_EESOX+0x10d/0x8f0 block/partitions/acorn.c:522 check_partition+0x2ff/0x5c5 block/partitions/check.c:167 rescan_partitions+0x199/0x860 block/partition-generic.c:521 __blkdev_get+0x8e0/0x1190 fs/block_dev.c:1504 blkdev_get+0x26b/0x880 fs/block_dev.c:1611 blkdev_open+0x19f/0x200 fs/block_dev.c:1769 do_dentry_open+0x5c5/0xe50 fs/open.c:784 vfs_open+0xfc/0x240 fs/open.c:906 do_last fs/namei.c:3365 [inline] path_openat+0xc02/0x3830 fs/namei.c:3501 do_filp_open+0x177/0x250 fs/namei.c:3535 do_sys_open+0x1dd/0x360 fs/open.c:1093 __do_sys_open fs/open.c:1111 [inline] __se_sys_open fs/open.c:1106 [inline] __x64_sys_open+0x79/0xb0 fs/open.c:1106 do_syscall_64+0xd0/0x4d0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x413711 RSP: 002b:00007fb345f907a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000002 RAX: ffffffffffffffda RBX: 6666666666666667 RCX: 0000000000413711 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00007fb345f90850 RBP: 000000000075bf20 R08: 000000000000000f R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000293 R12: 00007fb345f916d4 R13: 00000000004c8c9e R14: 00000000004dfb40 R15: 00000000ffffffff Kernel Offset: disabled Rebooting in 86400 seconds..