bisecting cause commit starting from bdcc9f6a568275aed4cc32fd2312432d2ff1b704 building syzkaller on 098b5d530648147c744a7c2eb8b78c1307f9d3ce testing commit bdcc9f6a568275aed4cc32fd2312432d2ff1b704 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 48aa9fa11f09c25fb5e2cc3d4865740e75a46554194fd385db06b63ea44c2bc2 run #0: crashed: INFO: task hung in process_one_work run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK reproducer seems to be flaky testing release v5.14 testing commit 7d2a07b769330c34b4deabeed939325c77a7ec2f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 188662a1d46039e68d7e80a1aac184133bb46fdd40eebe4a31907f7a00d06475 all runs: OK # git bisect start bdcc9f6a568275aed4cc32fd2312432d2ff1b704 7d2a07b769330c34b4deabeed939325c77a7ec2f Bisecting: 13044 revisions left to test after this (roughly 14 steps) [412a5feba414127a6c69452dfad454086867011f] Merge 5.15-rc6 into tty-next testing commit 412a5feba414127a6c69452dfad454086867011f compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 39c44d63e905fce923e37851996f6ee93d8fc08ea46f484c2cb70f7a12333269 all runs: OK # git bisect good 412a5feba414127a6c69452dfad454086867011f Bisecting: 6532 revisions left to test after this (roughly 13 steps) [0306023d64d510a92f6bae2c6759fd854ee5a1c3] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git testing commit 0306023d64d510a92f6bae2c6759fd854ee5a1c3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 29aeb77b276f31963723690075b88c022e169d7c82c90fb45d642cce9077aaff all runs: OK # git bisect good 0306023d64d510a92f6bae2c6759fd854ee5a1c3 Bisecting: 3150 revisions left to test after this (roughly 12 steps) [ed3cecd2f5acb59937ddf0609e8cde33555c31dd] Merge branch 'auto-latest' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git testing commit ed3cecd2f5acb59937ddf0609e8cde33555c31dd compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: fc80d02fb691b081866741f012c3fc9365a7db1950cd2243140da5e5c58c4d5d run #0: crashed: INFO: task hung in process_one_work run #1: crashed: INFO: task hung in process_one_work run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad ed3cecd2f5acb59937ddf0609e8cde33555c31dd Bisecting: 1813 revisions left to test after this (roughly 11 steps) [79e34df59d6c92616a1444f5eac072f796df6fee] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound.git testing commit 79e34df59d6c92616a1444f5eac072f796df6fee compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: fbb84da61b41681a9fb8f77f88b8f1631bbaea8d1a1d3196870ff1db5acb86e7 all runs: OK # git bisect good 79e34df59d6c92616a1444f5eac072f796df6fee Bisecting: 822 revisions left to test after this (roughly 10 steps) [14a490279a2ecacfc5a6e6f805c49a2cd9f1ccbc] Merge branch 'for-next' of git://git.kernel.dk/linux-block.git testing commit 14a490279a2ecacfc5a6e6f805c49a2cd9f1ccbc compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: a56cf884d63931ab66631ae08f06479d30c232a12fbead1fca21c97665dcd14b all runs: OK # git bisect good 14a490279a2ecacfc5a6e6f805c49a2cd9f1ccbc Bisecting: 399 revisions left to test after this (roughly 9 steps) [cf1e78907df6f5dec0eac7b65863ad112cc32f6c] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git testing commit cf1e78907df6f5dec0eac7b65863ad112cc32f6c compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 1cbb2e2ee998b0e0caed0c2bd2a232e249351fde598aa3e54bf085ba665965bc all runs: OK # git bisect good cf1e78907df6f5dec0eac7b65863ad112cc32f6c Bisecting: 192 revisions left to test after this (roughly 8 steps) [d7398e442f2e486b62298b6968ebf05466893873] Merge sched/core into tip/master testing commit d7398e442f2e486b62298b6968ebf05466893873 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: ff9b9c31416e5f758d3be229dabb27c7a7642705dc1b1435ec22a247533dc003 run #0: crashed: INFO: task hung in process_one_work run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad d7398e442f2e486b62298b6968ebf05466893873 Bisecting: 90 revisions left to test after this (roughly 7 steps) [e0d4656d2c025bcec31a247977c5298f3bf497e3] Merge locking/core into tip/master testing commit e0d4656d2c025bcec31a247977c5298f3bf497e3 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 0afc31952feae4f1b0ebb6993a6098869d2542861c9f2507b84a9604f3471720 all runs: OK # git bisect good e0d4656d2c025bcec31a247977c5298f3bf497e3 Bisecting: 45 revisions left to test after this (roughly 6 steps) [f9ec6fea201429b5a3f76319e943989f1a1e25ef] sched/topology: Remove unused numa_distance in cpu_attach_domain() testing commit f9ec6fea201429b5a3f76319e943989f1a1e25ef compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: c838fd300ba3286c2741e2bb9073645c523f9a5942a48fd75cec1e246997d9b8 run #0: crashed: INFO: task hung in process_one_work run #1: OK run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad f9ec6fea201429b5a3f76319e943989f1a1e25ef Bisecting: 22 revisions left to test after this (roughly 5 steps) [1a7243ca4074beed97b68d7235a6e34862fc2cd6] kthread: Move prio/affinite change into the newly created thread testing commit 1a7243ca4074beed97b68d7235a6e34862fc2cd6 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8ccb8f075d6577a7ee8279e3c5c158de307db1e497d4ecb406e9f814a0d93b85 run #0: crashed: INFO: task hung in process_one_work run #1: crashed: INFO: rcu detected stall in snd_seq_write run #2: OK run #3: OK run #4: OK run #5: OK run #6: OK run #7: OK run #8: OK run #9: OK run #10: OK run #11: OK run #12: OK run #13: OK run #14: OK run #15: OK run #16: OK run #17: OK run #18: OK run #19: OK # git bisect bad 1a7243ca4074beed97b68d7235a6e34862fc2cd6 Bisecting: 10 revisions left to test after this (roughly 4 steps) [1c36432b278cecf1499f21fae19836e614954309] kselftests/sched: cleanup the child processes testing commit 1c36432b278cecf1499f21fae19836e614954309 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 5fd5a9934448c3f9a7b1c51ec9b1e9ed9fa5cf88eef2b67976e910006cdac948 all runs: OK # git bisect good 1c36432b278cecf1499f21fae19836e614954309 Bisecting: 5 revisions left to test after this (roughly 3 steps) [ed7b564cfdd0668efbd739d0b4e2d67797293f32] sched/rt: Support sched_stat_runtime tracepoint for RT sched class testing commit ed7b564cfdd0668efbd739d0b4e2d67797293f32 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: 8141fca85bfd5cad9e3990fc99343748777de24a16751c6bc91bd9644208f640 all runs: OK # git bisect good ed7b564cfdd0668efbd739d0b4e2d67797293f32 Bisecting: 2 revisions left to test after this (roughly 2 steps) [b5eb4a5f6521d58d5564b8746701bd67a92a2b11] sched/dl: Support schedstats for deadline sched class testing commit b5eb4a5f6521d58d5564b8746701bd67a92a2b11 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f8e5d8279ebac1fcae6d2c536828f8b0ac3ffdc0afdb574aca6acec7f863bcf6 all runs: OK # git bisect good b5eb4a5f6521d58d5564b8746701bd67a92a2b11 Bisecting: 0 revisions left to test after this (roughly 1 step) [c597bfddc9e9e8a63817252b67c3ca0e544ace26] sched: Provide Kconfig support for default dynamic preempt mode testing commit c597bfddc9e9e8a63817252b67c3ca0e544ace26 compiler: gcc (GCC) 10.2.1 20210217, GNU ld (GNU Binutils for Debian) 2.35.2 kernel signature: f39d7718c9745f9b707522b2fe187e3b499c9432ea95a881aa94bc8bc8bcd839 all runs: OK # git bisect good c597bfddc9e9e8a63817252b67c3ca0e544ace26 1a7243ca4074beed97b68d7235a6e34862fc2cd6 is the first bad commit commit 1a7243ca4074beed97b68d7235a6e34862fc2cd6 Author: Sebastian Andrzej Siewior Date: Tue Nov 10 12:38:47 2020 +0100 kthread: Move prio/affinite change into the newly created thread With enabled threaded interrupts the nouveau driver reported the following: | Chain exists of: | &mm->mmap_lock#2 --> &device->mutex --> &cpuset_rwsem | | Possible unsafe locking scenario: | | CPU0 CPU1 | ---- ---- | lock(&cpuset_rwsem); | lock(&device->mutex); | lock(&cpuset_rwsem); | lock(&mm->mmap_lock#2); The device->mutex is nvkm_device::mutex. Unblocking the lockchain at `cpuset_rwsem' is probably the easiest thing to do. Move the priority reset to the start of the newly created thread. Fixes: 710da3c8ea7df ("sched/core: Prevent race condition between cpuset and __sched_setscheduler()") Reported-by: Mike Galbraith Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/a23a826af7c108ea5651e73b8fbae5e653f16e86.camel@gmx.de kernel/kthread.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) culprit signature: 8ccb8f075d6577a7ee8279e3c5c158de307db1e497d4ecb406e9f814a0d93b85 parent signature: f39d7718c9745f9b707522b2fe187e3b499c9432ea95a881aa94bc8bc8bcd839 Reproducer flagged being flaky revisions tested: 16, total time: 4h11m29.201647513s (build: 1h41m35.241824462s, test: 2h28m7.556503038s) first bad commit: 1a7243ca4074beed97b68d7235a6e34862fc2cd6 kthread: Move prio/affinite change into the newly created thread recipients (to): ["bigeasy@linutronix.de" "linux-kernel@vger.kernel.org" "peterz@infradead.org"] recipients (cc): ["akpm@linux-foundation.org" "peterz@infradead.org" "pmladek@suse.com" "valentin.schneider@arm.com"] crash: INFO: rcu detected stall in snd_seq_write rcu: INFO: rcu_preempt self-detected stall on CPU rcu: 1-....: (10499 ticks this GP) idle=747/1/0x4000000000000000 softirq=14573/14573 fqs=4993 (t=10500 jiffies g=12301 q=218361) NMI backtrace for cpu 1 CPU: 1 PID: 12482 Comm: syz-executor.2 Not tainted 5.15.0-rc4-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x57/0x7d lib/dump_stack.c:106 nmi_cpu_backtrace.cold+0x30/0xc0 lib/nmi_backtrace.c:105 nmi_trigger_cpumask_backtrace+0x11a/0x160 lib/nmi_backtrace.c:62 trigger_single_cpu_backtrace include/linux/nmi.h:164 [inline] rcu_dump_cpu_stacks+0x25e/0x3f0 kernel/rcu/tree_stall.h:343 print_cpu_stall kernel/rcu/tree_stall.h:627 [inline] check_cpu_stall kernel/rcu/tree_stall.h:711 [inline] rcu_pending kernel/rcu/tree.c:3880 [inline] rcu_sched_clock_irq.cold+0x9d/0x746 kernel/rcu/tree.c:2599 update_process_times+0x13b/0x1c0 kernel/time/timer.c:1785 tick_sched_handle+0x6f/0x130 kernel/time/tick-sched.c:226 tick_sched_timer+0x132/0x210 kernel/time/tick-sched.c:1421 __run_hrtimer kernel/time/hrtimer.c:1685 [inline] __hrtimer_run_queues+0x18a/0xb00 kernel/time/hrtimer.c:1749 hrtimer_interrupt+0x2f5/0x780 kernel/time/hrtimer.c:1811 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1086 [inline] __sysvec_apic_timer_interrupt+0x146/0x530 arch/x86/kernel/apic/apic.c:1103 sysvec_apic_timer_interrupt+0x8e/0xc0 arch/x86/kernel/apic/apic.c:1097 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:638 RIP: 0010:__raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:161 [inline] RIP: 0010:_raw_spin_unlock_irqrestore+0x38/0x70 kernel/locking/spinlock.c:194 Code: 74 24 10 e8 5a 47 18 f9 48 89 ef e8 a2 bd 18 f9 81 e3 00 02 00 00 75 25 9c 58 f6 c4 02 75 2d 48 85 db 74 01 fb bf 01 00 00 00 53 a1 0b f9 65 8b 05 5c 32 c6 77 85 c0 74 0a 5b 5d c3 e8 d0 f2 RSP: 0018:ffffc9000d837a88 EFLAGS: 00000206 RAX: 0000000000000002 RBX: 0000000000000200 RCX: 1ffffffff1d2d93e RDX: 0000000000000000 RSI: ffffffff888b4c60 RDI: 0000000000000001 RBP: ffff888058ad2488 R08: 0000000000000001 R09: ffffffff8e8f7b37 R10: 0000000000000001 R11: 0000000000000000 R12: ffff8880489b5d28 R13: 0000000000000000 R14: ffff8880489b5d00 R15: ffff888058ad2488 spin_unlock_irqrestore include/linux/spinlock.h:418 [inline] snd_seq_cell_alloc.constprop.0+0x2ab/0x560 sound/core/seq/seq_memory.c:271 snd_seq_event_dup+0x126/0x870 sound/core/seq/seq_memory.c:300 snd_seq_client_enqueue_event.constprop.0+0x19a/0x3c0 sound/core/seq/seq_clientmgr.c:970 snd_seq_write+0x2db/0x510 sound/core/seq/seq_clientmgr.c:1096 vfs_write+0x1c4/0x900 fs/read_write.c:592 ksys_write+0x171/0x1d0 fs/read_write.c:647 do_syscall_x64 arch/x86/entry/common.c:50 [inline] do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x7fae27d76ae9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fae274cb188 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 00007fae27e8a020 RCX: 00007fae27d76ae9 RDX: 00000000fffffee4 RSI: 00000000200000c0 RDI: 0000000000000003 RBP: 00007fae27dd0f25 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007ffceeb24baf R14: 00007fae274cb300 R15: 0000000000022000 ---------------- Code disassembly (best guess): 0: 74 24 je 0x26 2: 10 e8 adc %ch,%al 4: 5a pop %rdx 5: 47 18 f9 rex.RXB sbb %r15b,%r9b 8: 48 89 ef mov %rbp,%rdi b: e8 a2 bd 18 f9 callq 0xf918bdb2 10: 81 e3 00 02 00 00 and $0x200,%ebx 16: 75 25 jne 0x3d 18: 9c pushfq 19: 58 pop %rax 1a: f6 c4 02 test $0x2,%ah 1d: 75 2d jne 0x4c 1f: 48 85 db test %rbx,%rbx 22: 74 01 je 0x25 24: fb sti 25: bf 01 00 00 00 mov $0x1,%edi * 2a: e8 53 a1 0b f9 callq 0xf90ba182 <-- trapping instruction 2f: 65 8b 05 5c 32 c6 77 mov %gs:0x77c6325c(%rip),%eax # 0x77c63292 36: 85 c0 test %eax,%eax 38: 74 0a je 0x44 3a: 5b pop %rbx 3b: 5d pop %rbp 3c: c3 retq 3d: e8 .byte 0xe8 3e: d0 f2 shl %dl