bisecting cause commit starting from 8a61716ff2ab23eddd1f7a05a075a374e4d0c3d4
building syzkaller on 7ff74a98320355d2a8c097333827b6565dbf64b9
testing commit 8a61716ff2ab23eddd1f7a05a075a374e4d0c3d4 with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in sys_sendfile64
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in sys_sendfile64
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in sys_sendfile64
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in sys_sendfile64
run #8: crashed: INFO: rcu detected stall in sys_sendfile64
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.20
testing commit 8fe28cb58bcb235034b64cbbb7550a8a43fd88be with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in sys_sendfile64
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in rw_verify_area
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in sys_sendfile64
run #8: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #9: crashed: INFO: rcu detected stall in sys_sendfile64
testing release v4.19
testing commit 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #4: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in sys_sendfile64
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.18
testing commit 94710cac0ef4ee177a63b5227664b38c95bbf703 with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in sys_sendfile64
run #2: crashed: INFO: rcu detected stall in sys_sendfile64
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in sys_sendfile64
run #6: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in sys_sendfile64
run #9: crashed: INFO: rcu detected stall in sys_sendfile64
testing release v4.17
testing commit 29dcea88779c856c7dc92040a0c01233263101d4 with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in sys_sendfile64
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in sys_sendfile64
run #4: crashed: INFO: rcu detected stall in sys_sendfile64
run #5: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #6: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.16
testing commit 0adb32858b0bddf4ada5f364a84ed60b196dbcda with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in sendfile64
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in sendfile64
run #5: crashed: INFO: rcu detected stall in sendfile64
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in sendfile64
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.15
testing commit d8a5b80568a9cb66810e75b182018e9edb68e8ff with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.14
testing commit bebc6082da0a9f5d47a1ea2edc099bf671058bd4 with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: BUG: workqueue lockup
testing release v4.13
testing commit 569dbb88e80deb68974ef6fdd6a13edb9d686261 with gcc (GCC) 8.1.0
all runs: crashed: INFO: rcu detected stall in corrupted
testing release v4.12
testing commit 6f7da290413ba713f0cdd9ff1a2a9bb129ef4f6c with gcc (GCC) 8.1.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #9: crashed: no output from test machine
testing release v4.11
testing commit a351e9b9fc24e982ec2f0e76379a49826036da12 with gcc (GCC) 7.3.0
run #0: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "45506" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/usr/local/google/home/dvyukov/syzkaller/ci-bisect3/jobs/linux/workdir/image/key" "/usr/local/google/home/dvyukov/syzkaller/ci-bisect3/jobs/linux/workdir/repro.prog" "root@localhost:/repro.prog"]: exit status 1
Warning: Permanently added '[localhost]:45506' (ECDSA) to the list of known hosts.
/usr/local/google/home/dvyukov/syzkaller/ci-bisect3/jobs/linux/workdir/repro.prog: Broken pipe

run #1: OK
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: BUG: workqueue lockup
run #9: crashed: BUG: workqueue lockup
testing release v4.10
testing commit c470abd4fde40ea6a0846a2beab642a578c0b8cd with gcc (GCC) 5.5.0
run #0: OK
run #1: OK
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: BUG: workqueue lockup
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: INFO: rcu detected stall in ext4_file_write_iter
testing release v4.9
testing commit 69973b830859bc6529a7a0468ba0d80ee5117826 with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: no output from test machine
testing release v4.8
testing commit c8d2bc9bc39ebea8437fd974fdbc21847bb897a3 with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: BUG: workqueue lockup
run #9: crashed: BUG: workqueue lockup
testing release v4.7
testing commit 523d939ef98fd712632d93a5a2b588e477a7565e with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: BUG: workqueue lockup
run #9: crashed: BUG: workqueue lockup
testing release v4.6
testing commit 2dcd0af568b0cf583645c8a317dd12e344b1c72a with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: crashed: BUG: workqueue lockup
testing release v4.5
testing commit b562e44f507e863c6792946e4e1b1449fbbac85d with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in corrupted
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "16539" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/usr/local/google/home/dvyukov/syzkaller/ci-bisect3/jobs/linux/workdir/image/key" "/tmp/syz-executor119972885" "root@localhost:/syz-executor119972885"]: exit status 1
ssh: connect to host localhost port 16539: Connection timed out
lost connection

testing release v4.4
testing commit afd2ff9b7e1b367172f18ba7f693dfb62bdcb2dc with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: crashed: INFO: rcu detected stall in corrupted
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: INFO: rcu detected stall in corrupted
run #7: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #8: crashed: INFO: rcu detected stall in corrupted
run #9: basic kernel testing failed: failed to copy test binary to VM: failed to run ["scp" "-P" "42172" "-F" "/dev/null" "-o" "UserKnownHostsFile=/dev/null" "-o" "BatchMode=yes" "-o" "IdentitiesOnly=yes" "-o" "StrictHostKeyChecking=no" "-o" "ConnectTimeout=10" "-i" "/usr/local/google/home/dvyukov/syzkaller/ci-bisect3/jobs/linux/workdir/image/key" "/tmp/syz-executor260412175" "root@localhost:/syz-executor260412175"]: exit status 1
ssh: connect to host localhost port 42172: Connection timed out
lost connection

testing release v4.3
testing commit 6a13feb9c82803e2b815eca72fa7a9f5561d7861 with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in corrupted
run #1: crashed: INFO: rcu detected stall in corrupted
run #2: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #3: crashed: INFO: rcu detected stall in corrupted
run #4: crashed: INFO: rcu detected stall in corrupted
run #5: crashed: INFO: rcu detected stall in corrupted
run #6: crashed: unregister_netdevice: waiting for DEV to become free
run #7: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #8: crashed: unregister_netdevice: waiting for DEV to become free
run #9: crashed: INFO: rcu detected stall in corrupted
testing release v4.2
testing commit 64291f7db5bd8150a74ad2036f1037e6a0428df2 with gcc (GCC) 5.5.0
crypto/rsa_helper.c:18:25: fatal error: rsakey-asn1.h: No such file or directory
testing release v4.1
testing commit b953c0d234bc72e8489d3bf51a276c5c4ec85345 with gcc (GCC) 5.5.0
run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter
run #2: OK
run #3: OK
run #4: OK
run #5: OK
run #6: OK
run #7: OK
run #8: OK
run #9: OK
revisions tested: 21, total time: 3h41m20.471267332s (build: 1h20m54.29662466s, test: 2h15m40.293826855s)
the crash already happened on the oldest tested release
crash: INFO: rcu detected stall in ext4_file_write_iter
bridge0: port 1(bridge_slave_0) entered forwarding state
bridge0: port 2(bridge_slave_1) entered forwarding state
bridge0: port 2(bridge_slave_1) entered forwarding state
bridge0: port 1(bridge_slave_0) entered forwarding state
bridge0: port 2(bridge_slave_1) entered forwarding state
INFO: rcu_preempt detected stalls on CPUs/tasks:
	(detected by 0, t=10502 jiffies, g=261, c=260, q=6)
All QSes seen, last rcu_preempt kthread activity 10502 (4294958734-4294948232), jiffies_till_next_fqs=1, root ->qsmask 0x0
syz-executor.0  R  running task    12712  5840   5403 0x00000000
 ffff88003b1e87d0 ffff88003ec03d98 ffffffff811a3fa2 0000000000002906
 ffffffff8325ce40 0000000000000105 0000000000000000 ffff88003ec03e28
 ffffffff811ea439 0000000000000000 ffff88003ec10820 ffff88003b1e87d0
Call Trace:
 <IRQ>  [<ffffffff811a3fa2>] sched_show_task+0xf2/0x1a0 kernel/sched/core.c:4545
 [<ffffffff811ea439>] print_other_cpu_stall kernel/rcu/tree.c:1226 [inline]
 [<ffffffff811ea439>] check_cpu_stall kernel/rcu/tree.c:1332 [inline]
 [<ffffffff811ea439>] __rcu_pending kernel/rcu/tree.c:3422 [inline]
 [<ffffffff811ea439>] rcu_pending kernel/rcu/tree.c:3486 [inline]
 [<ffffffff811ea439>] rcu_check_callbacks+0xd69/0xd70 kernel/rcu/tree.c:2703
 [<ffffffff811f0864>] update_process_times+0x34/0x60 kernel/time/timer.c:1396
 [<ffffffff81203140>] tick_sched_handle.isra.22+0x20/0x60 kernel/time/tick-sched.c:151
 [<ffffffff812033fc>] tick_sched_timer+0x3c/0x80 kernel/time/tick-sched.c:1124
 [<ffffffff811f1466>] __run_hrtimer+0xe6/0x760 kernel/time/hrtimer.c:1220
 [<ffffffff811f2733>] hrtimer_interrupt+0x103/0x230 kernel/time/hrtimer.c:1299
 [<ffffffff810be455>] local_apic_timer_interrupt+0x35/0x60 arch/x86/kernel/apic/apic.c:890
 [<ffffffff810be6ac>] smp_apic_timer_interrupt+0x3c/0x50 arch/x86/kernel/apic/apic.c:914
 [<ffffffff82653f73>] apic_timer_interrupt+0x73/0x80 arch/x86/kernel/entry_64.S:923
 <EOI>  [<ffffffff8264ebd8>] ? arch_local_irq_restore arch/x86/include/asm/paravirt.h:809 [inline]
 <EOI>  [<ffffffff8264ebd8>] ? __mutex_lock_common kernel/locking/mutex.c:599 [inline]
 <EOI>  [<ffffffff8264ebd8>] ? mutex_lock_nested+0x528/0x610 kernel/locking/mutex.c:617
 [<ffffffff813fea40>] ext4_file_write_iter+0x80/0x4a0 fs/ext4/file.c:115
 [<ffffffff812fde52>] vfs_iter_write+0x62/0xa0 fs/read_write.c:364
 [<ffffffff813365d4>] iter_file_splice_write+0x254/0x3a0 fs/splice.c:1011
 [<ffffffff81335f01>] do_splice_from fs/splice.c:1115 [inline]
 [<ffffffff81335f01>] direct_splice_actor+0x31/0x40 fs/splice.c:1281
 [<ffffffff8133687c>] splice_direct_to_actor+0xcc/0x210 fs/splice.c:1234
 [<ffffffff81336a4a>] do_splice_direct+0x8a/0xb0 fs/splice.c:1324
 [<ffffffff812fef7b>] do_sendfile+0x1ab/0x3a0 fs/read_write.c:1227
 [<ffffffff812ffb1f>] SYSC_sendfile64 fs/read_write.c:1282 [inline]
 [<ffffffff812ffb1f>] SyS_sendfile64+0x4f/0xb0 fs/read_write.c:1274
 [<ffffffff82653032>] system_call_fastpath+0x16/0x7a
rcu_preempt kthread starved for 10504 jiffies!