syzbot


BUG: soft lockup in smp_call_function

Status: upstream: reported C repro on 2022/04/23 04:43
Reported-by: syzbot+9a524f9e9ff12d5eb182@syzkaller.appspotmail.com
First crash: 229d, last: 41d
similar bugs (5):
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-4.19 BUG: soft lockup in smp_call_function C error 2 100d 228d 0/1 upstream: reported C repro on 2022/04/23 05:26
upstream BUG: soft lockup in smp_call_function C done 807 1d18h 878d 0/24 upstream: reported C repro on 2020/07/12 23:02
android-54 BUG: soft lockup in smp_call_function 1 397d 397d 0/2 auto-closed as invalid on 2022/03/05 18:04
android-5-15 BUG: soft lockup in smp_call_function 1 12d 12d 0/2 premoderation: reported on 2022/11/25 06:05
upstream INFO: rcu detected stall in smp_call_function C 107 606d 861d 22/24 fixed on 2021/04/13 06:26

Sample crash report:
clocksource:                       'kvm-clock' wd_now: 4b0301c7f6 wd_last: 4ae534a60d mask: ffffffffffffffff
clocksource:                       'tsc' cs_now: 779ddfc2 cs_last: a4788f0a6e mask: ffffffffffffffff
tsc: Marking TSC unstable due to clocksource watchdog
[Firmware Bug]: TSC ADJUST differs: CPU0 0 --> -705490478624. Restoring
watchdog: BUG: soft lockup - CPU#1 stuck for 303s! [syz-executor402:7994]
Modules linked in:
irq event stamp: 229122
hardirqs last  enabled at (229121): [<ffffffff81228be9>] _flat_send_IPI_mask+0x109/0x1a0 arch/x86/kernel/apic/apic_flat_64.c:62
systemd[1]: systemd-journald.service: Main process exited, code=killed, status=6/ABRT
hardirqs last disabled at (229122): [<ffffffff874018ae>] apic_timer_interrupt+0x8e/0xa0 arch/x86/entry/entry_64.S:793
softirqs last  enabled at (226806): [<ffffffff8760068b>] __do_softirq+0x68b/0x9ff kernel/softirq.c:314
systemd[1]: systemd-journald.service: Unit entered failed state.
softirqs last disabled at (226795): [<ffffffff81321cf3>] invoke_softirq kernel/softirq.c:368 [inline]
softirqs last disabled at (226795): [<ffffffff81321cf3>] irq_exit+0x193/0x240 kernel/softirq.c:409
CPU: 1 PID: 7994 Comm: syz-executor402 Not tainted 4.14.276-syzkaller #0
systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
task: ffff88809f6183c0 task.stack: ffff888095bf8000
RIP: 0010:__read_once_size include/linux/compiler.h:185 [inline]
RIP: 0010:csd_lock_wait kernel/smp.c:108 [inline]
RIP: 0010:smp_call_function_single+0x181/0x370 kernel/smp.c:302
systemd[1]: systemd-journald.service: Service has no hold-off time, scheduling restart.
RSP: 0018:ffff888095bffaa0 EFLAGS: 00000297 ORIG_RAX: ffffffffffffff10
RAX: ffff88809f6183c0 RBX: 1ffff11012b7ff58 RCX: 0000000000000000
systemd[1]: Stopped Flush Journal to Persistent Storage.
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000286
RBP: ffff888095bffb48 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
systemd[1]: Stopping Flush Journal to Persistent Storage...
R13: ffff888095bffb78 R14: 0000000000000001 R15: 0000000000000010
FS:  0000555556ab0300(0000) GS:ffff8880ba500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
systemd[1]: Stopped Journal Service.
CR2: 00007fc592fe7130 CR3: 00000000ac072000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
systemd[1]: Starting Journal Service...
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 wrmsr_safe_on_cpu+0x8f/0xd0 arch/x86/lib/msr-smp.c:188
 msr_write+0xfc/0x190 arch/x86/kernel/msr.c:95
 do_loop_readv_writev fs/read_write.c:698 [inline]
 do_loop_readv_writev fs/read_write.c:682 [inline]
 do_iter_write+0x3da/0x550 fs/read_write.c:956
 vfs_writev+0x125/0x290 fs/read_write.c:999
 do_pwritev fs/read_write.c:1088 [inline]
 SYSC_pwritev fs/read_write.c:1135 [inline]
 SyS_pwritev+0x15a/0x200 fs/read_write.c:1130
 do_syscall_64+0x1d5/0x640 arch/x86/entry/common.c:292
 entry_SYSCALL_64_after_hwframe+0x46/0xbb
RIP: 0033:0x7fc592f763c9
RSP: 002b:00007ffdf0adb548 EFLAGS: 00000246 ORIG_RAX: 0000000000000128
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc592f763c9
RDX: 0000000000000002 RSI: 0000000020000100 RDI: 0000000000000004
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000010 R11: 0000000000000246 R12: 00007ffdf0adb570
R13: 431bde82d7b634db R14: 0000000000000000 R15: 0000000000000000
Code: cd 08 00 48 8b 54 24 10 4c 89 e9 8b 7c 24 1c 48 8d 74 24 40 e8 71 fa ff ff 41 89 c4 8b 44 24 58 a8 01 74 0f e8 b1 cd 08 00 f3 90 <8b> 44 24 58 a8 01 75 f1 e8 a2 cd 08 00 e8 9d cd 08 00 bf 01 00 
----------------
Code disassembly (best guess):
   0:	cd 08                	int    $0x8
   2:	00 48 8b             	add    %cl,-0x75(%rax)
   5:	54                   	push   %rsp
   6:	24 10                	and    $0x10,%al
   8:	4c 89 e9             	mov    %r13,%rcx
   b:	8b 7c 24 1c          	mov    0x1c(%rsp),%edi
   f:	48 8d 74 24 40       	lea    0x40(%rsp),%rsi
  14:	e8 71 fa ff ff       	callq  0xfffffa8a
  19:	41 89 c4             	mov    %eax,%r12d
  1c:	8b 44 24 58          	mov    0x58(%rsp),%eax
  20:	a8 01                	test   $0x1,%al
  22:	74 0f                	je     0x33
  24:	e8 b1 cd 08 00       	callq  0x8cdda
  29:	f3 90                	pause
* 2b:	8b 44 24 58          	mov    0x58(%rsp),%eax <-- trapping instruction
  2f:	a8 01                	test   $0x1,%al
  31:	75 f1                	jne    0x24
  33:	e8 a2 cd 08 00       	callq  0x8cdda
  38:	e8 9d cd 08 00       	callq  0x8cdda
  3d:	bf                   	.byte 0xbf
  3e:	01 00                	add    %eax,(%rax)

Fix bisection attempts:
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci2-linux-4-14 2022/10/27 20:51 linux-4.14.y 41f36d7859a7 131df97d .config log report syz C
ci2-linux-4-14 2022/09/20 07:04 linux-4.14.y 5df8b4735177 131df97d .config log report syz C
ci2-linux-4-14 2022/08/21 06:38 linux-4.14.y b641242202ed 131df97d .config log report syz C
ci2-linux-4-14 2022/07/22 06:16 linux-4.14.y 9c3bf9cf362f 131df97d .config log report syz C
ci2-linux-4-14 2022/06/22 05:52 linux-4.14.y 84bae26850e3 131df97d .config log report syz C
ci2-linux-4-14 2022/05/23 05:06 linux-4.14.y dffb5c6ff09c 131df97d .config log report syz C
* Struck through repros no longer work on HEAD.
Crashes (1):
Manager Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Title
ci2-linux-4-14 2022/04/23 04:42 linux-4.14.y 15a1c6b6f516 131df97d .config log report syz C BUG: soft lockup in smp_call_function
* Struck through repros no longer work on HEAD.