BUG: soft lockup in __do_sys_clock

BUG: soft lockup in __do_sys_clock_adjtime
Status: upstream: reported C repro on 2020/07/07 01:42
Reported-by: syzbot+b63f85efcdedbba8b3be@syzkaller.appspotmail.com
First crash: 295d, last: 21d

Cause bisection: introduced by (bisect log) :
commit 5a781ccbd19e4664babcbe4b4ead7aa2b9283d22
Author: Vinicius Costa Gomes <vinicius.gomes@intel.com>
Date: Sat Sep 29 00:59:43 2018 +0000

tc: Add support for configuring the taprio scheduler

Crash: INFO: rcu detected stall in smp_call_function (log)
Repro: C syz .config

Sample crash report:

watchdog: BUG: soft lockup - CPU#1 stuck for 123s! [systemd-timesyn:4344]
Modules linked in:
irq event stamp: 102956
hardirqs last  enabled at (102955): [<ffffffff88000c42>] asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:596
hardirqs last disabled at (102956): [<ffffffff87e462cd>] idtentry_enter_cond_rcu+0x1d/0x50 arch/x86/entry/common.c:607
softirqs last  enabled at (76528): [<ffffffff862c5138>] sock_orphan include/net/sock.h:1872 [inline]
softirqs last  enabled at (76528): [<ffffffff862c5138>] sk_common_release+0x138/0x390 net/core/sock.c:3307
softirqs last disabled at (76526): [<ffffffff862c50c2>] sock_orphan include/net/sock.h:1868 [inline]
softirqs last disabled at (76526): [<ffffffff862c50c2>] sk_common_release+0xc2/0x390 net/core/sock.c:3307
CPU: 1 PID: 4344 Comm: systemd-timesyn Not tainted 5.8.0-rc3-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:csd_lock_wait kernel/smp.c:108 [inline]
RIP: 0010:smp_call_function_single+0x192/0x4f0 kernel/smp.c:382
Code: 10 8b 7c 24 1c 48 8d 74 24 40 48 89 44 24 50 48 8b 44 24 08 48 89 44 24 58 e8 fa f9 ff ff 41 89 c5 eb 07 e8 f0 ff 0a 00 f3 90 <44> 8b 64 24 48 31 ff 41 83 e4 01 44 89 e6 e8 5b fc 0a 00 45 85 e4
RSP: 0018:ffffc900018679e0 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 1ffff9200030cf40 RCX: ffffffff8168b785
RDX: ffff8880a6700340 RSI: ffffffff8168b770 RDI: 0000000000000005
RBP: ffffc90001867ac0 R08: 0000000000000001 R09: ffff8880ae636dc7
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000040
FS:  00007efe995408c0(0000) GS:ffff8880ae700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055eae694ed48 CR3: 00000000a6bc5000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 smp_call_function_many_cond+0x1a4/0x990 kernel/smp.c:518
 smp_call_function_many kernel/smp.c:577 [inline]
 smp_call_function kernel/smp.c:599 [inline]
 on_each_cpu+0x4a/0x240 kernel/smp.c:699
 clock_was_set+0x18/0x20 kernel/time/hrtimer.c:872
 timekeeping_inject_offset+0x3e9/0x4d0 kernel/time/timekeeping.c:1305
 do_adjtimex+0x28f/0x990 kernel/time/timekeeping.c:2332
 do_clock_adjtime kernel/time/posix-timers.c:1109 [inline]
 __do_sys_clock_adjtime+0x155/0x250 kernel/time/posix-timers.c:1121
 do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:359
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7efe989361b7
Code: Bad RIP value.
RSP: 002b:00007ffc4cfe68e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000131
RAX: ffffffffffffffda RBX: 0000562d621a2630 RCX: 00007efe989361b7
RDX: ffffffffe7ce3d82 RSI: 00007ffc4cfe69b0 RDI: 0000000000000000
RBP: 00007ffc4cfe69b0 R08: 0000000000000020 R09: 000000000000000d
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00007ffc4cfe6aa0 R14: 0000000000000005 R15: 0000000000000001
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 6820 Comm: syz-executor620 Not tainted 5.8.0-rc3-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:lock_acquire+0xa2/0xad0 kernel/locking/lockdep.c:4934
Code: 08 00 00 00 f3 c7 40 0c f3 f3 f3 f3 65 48 8b 04 25 28 00 00 00 48 89 84 24 b8 00 00 00 31 c0 48 89 f8 48 c1 e8 03 0f b6 0c 08 <48> 89 f8 83 e0 07 83 c0 03 38 c8 7c 08 84 c9 0f 85 61 07 00 00 45
RSP: 0018:ffffc90000007be8 EFLAGS: 00000807
RAX: 1ffff11014596d81 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8880a2cb6c0c
RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffffffff8cb15150 R14: 0000000000000000 R15: ffff8880a2cb6340
FS:  0000000002039880(0000) GS:ffff8880ae600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000020000610 CR3: 000000009fdd4000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0x8c/0xc0 kernel/locking/spinlock.c:159
 debug_object_deactivate lib/debugobjects.c:710 [inline]
 debug_object_deactivate+0x101/0x300 lib/debugobjects.c:698
 debug_hrtimer_deactivate kernel/time/hrtimer.c:421 [inline]
 debug_deactivate kernel/time/hrtimer.c:482 [inline]
 __run_hrtimer kernel/time/hrtimer.c:1488 [inline]
 __hrtimer_run_queues+0x3cb/0xfc0 kernel/time/hrtimer.c:1584
 hrtimer_interrupt+0x32a/0x930 kernel/time/hrtimer.c:1646
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1080 [inline]
 __sysvec_apic_timer_interrupt+0x142/0x5e0 arch/x86/kernel/apic/apic.c:1097
 asm_call_on_stack+0xf/0x20 arch/x86/entry/entry_64.S:711
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:22 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:48 [inline]
 sysvec_apic_timer_interrupt+0xe0/0x120 arch/x86/kernel/apic/apic.c:1091
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:596
RIP: 0010:arch_local_irq_restore arch/x86/include/asm/paravirt.h:765 [inline]
RIP: 0010:on_each_cpu+0x149/0x240 kernel/smp.c:702
Code: 00 fc ff df 48 c1 e8 03 80 3c 10 00 0f 85 e6 00 00 00 48 83 3d 57 12 4c 08 00 0f 84 af 00 00 00 e8 6c e9 0a 00 48 89 df 57 9d <0f> 1f 44 00 00 e8 5d e9 0a 00 bf 01 00 00 00 e8 83 a4 e6 ff 31 ff
RSP: 0018:ffffc90001747d70 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000293 RCX: 0000000000000000
RDX: ffff8880a2cb6340 RSI: ffffffff8168cdf4 RDI: 0000000000000293
RBP: 0000000000000200 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffc90001747ed0 R15: ffffffffa0fc31fd
 clock_was_set+0x18/0x20 kernel/time/hrtimer.c:872
 do_settimeofday64 kernel/time/timekeeping.c:1257 [inline]
 do_settimeofday64+0x350/0x4e0 kernel/time/timekeeping.c:1223
 do_sys_settimeofday64 kernel/time/time.c:195 [inline]
 do_sys_settimeofday64+0x1de/0x260 kernel/time/time.c:169
 __do_sys_clock_settime kernel/time/posix-timers.c:1079 [inline]
 __se_sys_clock_settime kernel/time/posix-timers.c:1067 [inline]
 __x64_sys_clock_settime+0x197/0x260 kernel/time/posix-timers.c:1067
 do_syscall_64+0x60/0xe0 arch/x86/entry/common.c:359
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x4437c9
Code: Bad RIP value.
RSP: 002b:00007ffca52e53d8 EFLAGS: 00000246 ORIG_RAX: 00000000000000e3
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00000000004437c9
RDX: 00000000004437c9 RSI: 0000000020000400 RDI: 0000000000000000
RBP: 00007ffca52e53e0 R08: 0000000001bbbbbb R09: 0000000001bbbbbb
R10: 0000000001bbbbbb R11: 0000000000000246 R12: 00007ffca52e53f0
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000

Fix bisection attempts:
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro
ci-upstream-kasan-gce-selinux-root	2021/04/02 02:30	upstream	1678e493	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2021/03/01 15:11	upstream	fe07bfda	51095195	.config	log		syz	C
ci-upstream-kasan-gce-selinux-root	2021/01/15 17:27	upstream	5ee88057	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/12/16 03:45	upstream	e87b0708	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/11/08 14:00	upstream	4429f14a	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/10/09 13:32	upstream	583090b1	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/09/06 12:49	upstream	dd9fb9bb	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/08/06 13:00	upstream	47ec5303	51095195	.config	log	report	syz	C

Crashes (3):
Manager	Time	Kernel	Commit	Syzkaller	Config	Log	Report	Syz repro	C repro
ci-upstream-kasan-gce-selinux-root	2020/07/07 01:42	upstream	7cc2a8ea	51095195	.config	log	report	syz	C
ci-upstream-kasan-gce-selinux-root	2020/07/02 03:53	upstream	cd77006e	bed10395	.config	log	report
ci-upstream-linux-next-kasan-gce-root	2020/07/03 19:57	linux-next	aab20039	bed10395	.config	log	report