syzbot


KASAN: stack-out-of-bounds Read in rcu_process_callbacks

Status: fixed on 2018/08/07 13:43
Subsystems: kernel
[Documentation on labels]
Fix commit: 99ba2b5aba24 bpf: sockhash, disallow bpf_tcp_close and update in parallel
First crash: 2340d, last: 2340d

Sample crash report:
IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
8021q: adding VLAN 0 to HW filter on device team0
==================================================================
BUG: KASAN: stack-out-of-bounds in lookup_object lib/debugobjects.c:157 [inline]
BUG: KASAN: stack-out-of-bounds in debug_object_active_state+0x471/0x4d0 lib/debugobjects.c:725
Read of size 8 at addr ffff8801ab88d558 by task ksoftirqd/0/9

CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.18.0-rc4+ #147
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1c9/0x2b4 lib/dump_stack.c:113
 print_address_description+0x6c/0x20b mm/kasan/report.c:256
 kasan_report_error mm/kasan/report.c:354 [inline]
 kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412
 __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
 lookup_object lib/debugobjects.c:157 [inline]
 debug_object_active_state+0x471/0x4d0 lib/debugobjects.c:725
 debug_rcu_head_unqueue kernel/rcu/rcu.h:144 [inline]
 rcu_do_batch kernel/rcu/tree.c:2557 [inline]
 invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline]
 __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline]
 rcu_process_callbacks+0xfa5/0x1850 kernel/rcu/tree.c:2802
 __do_softirq+0x2e8/0xb17 kernel/softirq.c:288
 run_ksoftirqd+0x86/0x100 kernel/softirq.c:649
 smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
 kthread+0x345/0x410 kernel/kthread.c:246
 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412

Allocated by task 1102416563:
(stack is not available)

Freed by task 2294223768:
BUG: unable to handle kernel paging request at ffffffff8c431978
PGD 8e6d067 P4D 8e6d067 PUD 8e6e063 PMD 0 
Oops: 0000 [#1] SMP KASAN
CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.18.0-rc4+ #147
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:depot_fetch_stack+0x10/0x30 lib/stackdepot.c:201
Code: e8 45 1b 47 fe e9 b3 fd ff ff e8 3b 1b 47 fe e9 55 fd ff ff 90 90 90 90 90 90 89 f8 c1 ef 11 25 ff ff 1f 00 81 e7 f0 3f 00 00 <48> 03 3c c5 80 19 43 8b 8b 47 0c 48 83 c7 18 c7 46 10 00 00 00 00 
RSP: 0018:ffff8801d9ab77a0 EFLAGS: 00010006
RAX: 00000000001fffff RBX: ffff8801ab88d574 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801d9ab77a8 RDI: 0000000000003ff0
RBP: ffff8801d9ab77d0 R08: ffff8801d9aa2240 R09: ffffed003b5c3ec2
R10: ffffed003b5c3ec2 R11: ffff8801dae1f617 R12: ffff8801ab88d540
R13: ffff8801ab88d558 R14: ffff8801da810dc0 R15: ffff8801ab88d568
FS:  0000000000000000(0000) GS:ffff8801dae00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffff8c431978 CR3: 00000001ad7af000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 describe_object mm/kasan/report.c:245 [inline]
 print_address_description+0x11b/0x20b mm/kasan/report.c:263
 kasan_report_error mm/kasan/report.c:354 [inline]
 kasan_report.cold.7+0x242/0x2fe mm/kasan/report.c:412
 __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
 lookup_object lib/debugobjects.c:157 [inline]
 debug_object_active_state+0x471/0x4d0 lib/debugobjects.c:725
 debug_rcu_head_unqueue kernel/rcu/rcu.h:144 [inline]
 rcu_do_batch kernel/rcu/tree.c:2557 [inline]
 invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline]
 __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline]
 rcu_process_callbacks+0xfa5/0x1850 kernel/rcu/tree.c:2802
 __do_softirq+0x2e8/0xb17 kernel/softirq.c:288
 run_ksoftirqd+0x86/0x100 kernel/softirq.c:649
 smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
 kthread+0x345/0x410 kernel/kthread.c:246
 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
Modules linked in:
Dumping ftrace buffer:
   (ftrace buffer empty)
CR2: ffffffff8c431978

======================================================
WARNING: possible circular locking dependency detected
4.18.0-rc4+ #147 Not tainted
------------------------------------------------------
ksoftirqd/0/9 is trying to acquire lock:
(____ptrval____) ((console_sem).lock){-.-.}, at: down_trylock+0x13/0x70 kernel/locking/semaphore.c:136

but task is already holding lock:
(____ptrval____) (report_lock){..-.}, at: kasan_start_report mm/kasan/report.c:170 [inline]
(____ptrval____) (report_lock){..-.}, at: kasan_report_error mm/kasan/report.c:346 [inline]
(____ptrval____) (report_lock){..-.}, at: kasan_report+0x8e/0x110 mm/kasan/report.c:412

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #4 (report_lock){..-.}:
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
       kasan_start_report mm/kasan/report.c:170 [inline]
       kasan_report_error mm/kasan/report.c:346 [inline]
       kasan_report+0x8e/0x110 mm/kasan/report.c:412
       __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
       lookup_object lib/debugobjects.c:157 [inline]
       debug_object_active_state+0x471/0x4d0 lib/debugobjects.c:725
       debug_rcu_head_unqueue kernel/rcu/rcu.h:144 [inline]
       rcu_do_batch kernel/rcu/tree.c:2557 [inline]
       invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline]
       __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline]
       rcu_process_callbacks+0xfa5/0x1850 kernel/rcu/tree.c:2802
       __do_softirq+0x2e8/0xb17 kernel/softirq.c:288
       run_ksoftirqd+0x86/0x100 kernel/softirq.c:649
       smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
       kthread+0x345/0x410 kernel/kthread.c:246
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412

-> #3 (&obj_hash[i].lock){-.-.}:
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
       __debug_object_init+0x127/0x12e0 lib/debugobjects.c:381
       debug_object_init+0x16/0x20 lib/debugobjects.c:429
       debug_hrtimer_init kernel/time/hrtimer.c:410 [inline]
       debug_init kernel/time/hrtimer.c:458 [inline]
       hrtimer_init+0x97/0x480 kernel/time/hrtimer.c:1308
       init_dl_task_timer+0x1b/0x50 kernel/sched/deadline.c:1056
       __sched_fork+0x2ae/0x590 kernel/sched/core.c:2185
       init_idle+0x75/0x7a0 kernel/sched/core.c:5405
       sched_init+0xbf3/0xd2c kernel/sched/core.c:6103
       start_kernel+0x47d/0x949 init/main.c:602
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:452
       x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:433
       secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:242

-> #2 (&rq->lock){-.-.}:
       __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline]
       _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:144
       rq_lock kernel/sched/sched.h:1812 [inline]
       task_fork_fair+0x93/0x680 kernel/sched/fair.c:9952
       sched_fork+0x446/0xb40 kernel/sched/core.c:2381
       copy_process.part.39+0x1c09/0x7220 kernel/fork.c:1773
       copy_process kernel/fork.c:1616 [inline]
       _do_fork+0x291/0x12a0 kernel/fork.c:2099
       kernel_thread+0x34/0x40 kernel/fork.c:2158
       rest_init+0x22/0xe4 init/main.c:408
       start_kernel+0x90e/0x949 init/main.c:738
       x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:452
       x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:433
       secondary_startup_64+0xa5/0xb0 arch/x86/kernel/head_64.S:242

-> #1 (&p->pi_lock){-.-.}:
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
       try_to_wake_up+0xd2/0x12b0 kernel/sched/core.c:1985
       wake_up_process+0x10/0x20 kernel/sched/core.c:2148
       __up.isra.1+0x1c0/0x2a0 kernel/locking/semaphore.c:262
       up+0x13c/0x1c0 kernel/locking/semaphore.c:187
       __up_console_sem+0xbe/0x1b0 kernel/printk/printk.c:242
       console_unlock+0x7a2/0x10b0 kernel/printk/printk.c:2411
       vprintk_emit+0x6c6/0xdf0 kernel/printk/printk.c:1907
       vprintk_default+0x28/0x30 kernel/printk/printk.c:1948
       vprintk_func+0x7a/0xe7 kernel/printk/printk_safe.c:382
       printk+0xa7/0xcf kernel/printk/printk.c:1981
       load_umh+0x51/0xbd net/bpfilter/bpfilter_kern.c:98
       do_one_initcall+0x127/0x913 init/main.c:884
       do_initcall_level init/main.c:952 [inline]
       do_initcalls init/main.c:960 [inline]
       do_basic_setup init/main.c:978 [inline]
       kernel_init_freeable+0x49b/0x58e init/main.c:1135
       kernel_init+0x11/0x1b3 init/main.c:1061
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412

-> #0 ((console_sem).lock){-.-.}:
       lock_acquire+0x1e4/0x540 kernel/locking/lockdep.c:3924
       __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
       _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
       down_trylock+0x13/0x70 kernel/locking/semaphore.c:136
       __down_trylock_console_sem+0xae/0x200 kernel/printk/printk.c:225
       console_trylock+0x15/0xa0 kernel/printk/printk.c:2230
       console_trylock_spinning kernel/printk/printk.c:1643 [inline]
       vprintk_emit+0x6ad/0xdf0 kernel/printk/printk.c:1906
       vprintk_default+0x28/0x30 kernel/printk/printk.c:1948
       vprintk_func+0x7a/0xe7 kernel/printk/printk_safe.c:382
       printk+0xa7/0xcf kernel/printk/printk.c:1981
       kasan_start_report mm/kasan/report.c:171 [inline]
       kasan_report_error mm/kasan/report.c:346 [inline]
       kasan_report+0x9e/0x110 mm/kasan/report.c:412
       __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
       lookup_object lib/debugobjects.c:157 [inline]
       debug_object_active_state+0x471/0x4d0 lib/debugobjects.c:725
       debug_rcu_head_unqueue kernel/rcu/rcu.h:144 [inline]
       rcu_do_batch kernel/rcu/tree.c:2557 [inline]
       invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline]
       __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline]
       rcu_process_callbacks+0xfa5/0x1850 kernel/rcu/tree.c:2802
       __do_softirq+0x2e8/0xb17 kernel/softirq.c:288
       run_ksoftirqd+0x86/0x100 kernel/softirq.c:649
       smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
       kthread+0x345/0x410 kernel/kthread.c:246
       ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412

other info that might help us debug this:

Chain exists of:
  (console_sem).lock --> &obj_hash[i].lock --> report_lock

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(report_lock);
                               lock(&obj_hash[i].lock);
                               lock(report_lock);
  lock((console_sem).lock);

 *** DEADLOCK ***

2 locks held by ksoftirqd/0/9:
 #0: (____ptrval____) (&obj_hash[i].lock){-.-.}, at: debug_object_active_state+0xff/0x4d0 lib/debugobjects.c:723
 #1: (____ptrval____) (report_lock){..-.}, at: kasan_start_report mm/kasan/report.c:170 [inline]
 #1: (____ptrval____) (report_lock){..-.}, at: kasan_report_error mm/kasan/report.c:346 [inline]
 #1: (____ptrval____) (report_lock){..-.}, at: kasan_report+0x8e/0x110 mm/kasan/report.c:412

stack backtrace:
CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 4.18.0-rc4+ #147
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x1c9/0x2b4 lib/dump_stack.c:113
 print_circular_bug.isra.36.cold.57+0x1bd/0x27d kernel/locking/lockdep.c:1227
 check_prev_add kernel/locking/lockdep.c:1867 [inline]
 check_prevs_add kernel/locking/lockdep.c:1980 [inline]
 validate_chain kernel/locking/lockdep.c:2421 [inline]
 __lock_acquire+0x3449/0x5020 kernel/locking/lockdep.c:3435
 lock_acquire+0x1e4/0x540 kernel/locking/lockdep.c:3924
 __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline]
 _raw_spin_lock_irqsave+0x96/0xc0 kernel/locking/spinlock.c:152
 down_trylock+0x13/0x70 kernel/locking/semaphore.c:136
 __down_trylock_console_sem+0xae/0x200 kernel/printk/printk.c:225
 console_trylock+0x15/0xa0 kernel/printk/printk.c:2230
 console_trylock_spinning kernel/printk/printk.c:1643 [inline]
 vprintk_emit+0x6ad/0xdf0 kernel/printk/printk.c:1906
 vprintk_default+0x28/0x30 kernel/printk/printk.c:1948
 vprintk_func+0x7a/0xe7 kernel/printk/printk_safe.c:382
 printk+0xa7/0xcf kernel/printk/printk.c:1981
 kasan_start_report mm/kasan/report.c:171 [inline]
 kasan_report_error mm/kasan/report.c:346 [inline]
 kasan_report+0x9e/0x110 mm/kasan/report.c:412
 __asan_report_load8_noabort+0x14/0x20 mm/kasan/report.c:433
 lookup_object lib/debugobjects.c:157 [inline]
 debug_object_active_state+0x471/0x4d0 lib/debugobjects.c:725
 debug_rcu_head_unqueue kernel/rcu/rcu.h:144 [inline]
 rcu_do_batch kernel/rcu/tree.c:2557 [inline]
 invoke_rcu_callbacks kernel/rcu/tree.c:2818 [inline]
 __rcu_process_callbacks kernel/rcu/tree.c:2785 [inline]
 rcu_process_callbacks+0xfa5/0x1850 kernel/rcu/tree.c:2802
 __do_softirq+0x2e8/0xb17 kernel/softirq.c:288
 run_ksoftirqd+0x86/0x100 kernel/softirq.c:649
 smpboot_thread_fn+0x417/0x870 kernel/smpboot.c:164
 kthread+0x345/0x410 kernel/kthread.c:246
 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:412
---[ end trace 9d5d7b7db15a4262 ]---
RIP: 0010:depot_fetch_stack+0x10/0x30 lib/stackdepot.c:201
Code: e8 45 1b 47 fe e9 b3 fd ff ff e8 3b 1b 47 fe e9 55 fd ff ff 90 90 90 90 90 90 89 f8 c1 ef 11 25 ff ff 1f 00 81 e7 f0 3f 00 00 <48> 03 3c c5 80 19 43 8b 8b 47 0c 48 83 c7 18 c7 46 10 00 00 00 00 
RSP: 0018:ffff8801d9ab77a0 EFLAGS: 00010006
RAX: 00000000001fffff RBX: ffff8801ab88d574 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff8801d9ab77a8 RDI: 0000000000003ff0
RBP: ffff8801d9ab77d0 R08: ffff8801d9aa2240 R09: ffffed003b5c3ec2
R10: ffffed003b5c3ec2 R11: ffff8801dae1f617 R12: ffff8801ab88d540
R13: ffff8801ab88d558 R14: ffff8801da810dc0 R15: ffff8801ab88d568
FS:  0000000000000000(0000) GS:ffff8801dae00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffff8c431978 CR3: 00000001ad7af000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400

Crashes (1):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2018/07/15 02:48 upstream c31496dbacc2 92a49505 .config console log report syz C ci-upstream-kasan-gce-root
* Struck through repros no longer work on HEAD.