syzbot


BUG: soft lockup in debug_check_no_obj_freed

Status: upstream: reported syz repro on 2024/10/19 15:37
Subsystems: serial
[Documentation on labels]
Reported-by: syzbot+a234c2d63e0c171ca10e@syzkaller.appspotmail.com
First crash: 128d, last: 62d
Discussions (1)
Title Replies (including bot) Last reply
[syzbot] [serial?] BUG: soft lockup in debug_check_no_obj_freed 4 (7) 2024/10/27 09:33
Last patch testing requests (4)
Created Duration User Patch Repo Result
2024/11/03 10:09 13m retest repro https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing OK log
2024/10/22 08:58 6m eadavis@qq.com patch https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing error
2024/10/22 08:14 7m eadavis@qq.com patch https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing error
2024/10/22 07:51 14m eadavis@qq.com patch https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing error

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 143s! [syz.1.14:5352]
Modules linked in:
irq event stamp: 10723757
hardirqs last  enabled at (10723756): [<ffffffff8bb65a13>] irqentry_exit+0x63/0x90 kernel/entry/common.c:357
hardirqs last disabled at (10723757): [<ffffffff8bb635de>] sysvec_apic_timer_interrupt+0xe/0xc0 arch/x86/kernel/apic/apic.c:1043
softirqs last  enabled at (8470396): [<ffffffff81575bd4>] __do_softirq kernel/softirq.c:588 [inline]
softirqs last  enabled at (8470396): [<ffffffff81575bd4>] invoke_softirq kernel/softirq.c:428 [inline]
softirqs last  enabled at (8470396): [<ffffffff81575bd4>] __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
softirqs last disabled at (8470399): [<ffffffff81575bd4>] __do_softirq kernel/softirq.c:588 [inline]
softirqs last disabled at (8470399): [<ffffffff81575bd4>] invoke_softirq kernel/softirq.c:428 [inline]
softirqs last disabled at (8470399): [<ffffffff81575bd4>] __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
CPU: 0 UID: 0 PID: 5352 Comm: syz.1.14 Not tainted 6.11.0-rc3-syzkaller-00036-g9d5906799f7d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024
RIP: 0010:__debug_check_no_obj_freed lib/debugobjects.c:979 [inline]
RIP: 0010:debug_check_no_obj_freed+0x22f/0x580 lib/debugobjects.c:1020
Code: 48 b9 eb 83 b5 80 46 86 c8 61 48 0f af c1 48 c1 e8 2f 83 e0 f8 4c 8d bc c0 00 7f 0d 95 48 8d bc c0 08 7f 0d 95 48 89 7c 24 20 <e8> 6c ce 07 07 48 89 44 24 18 4c 89 fb 48 c1 eb 03 48 b8 00 00 00
RSP: 0018:ffffc90000007480 EFLAGS: 00000206
RAX: 000000000000c828 RBX: ffff888079702640 RCX: 61c8864680b583eb
RDX: 0000000000000001 RSI: ffffc90000007540 RDI: ffffffff95148870
RBP: ffffc900000075c8 R08: ffffc90000007547 R09: 0000000000000000
R10: ffffc90000007520 R11: fffff52000000ea9 R12: ffff888079702730
R13: ffff888079702640 R14: 0000000000000000 R15: ffffffff95148868
FS:  00007f273fe756c0(0000) GS:ffff8880b9200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b32713ff8 CR3: 0000000072172000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ>
 slab_free_hook mm/slub.c:2219 [inline]
 slab_free mm/slub.c:4473 [inline]
 kmem_cache_free+0x10f/0x350 mm/slub.c:4548
 kfree_skb_reason include/linux/skbuff.h:1260 [inline]
 __netif_receive_skb_core+0x3edd/0x4570 net/core/dev.c:5635
 __netif_receive_skb_one_core net/core/dev.c:5659 [inline]
 __netif_receive_skb+0x12f/0x650 net/core/dev.c:5775
 process_backlog+0x662/0x15b0 net/core/dev.c:6108
 __napi_poll+0xcb/0x490 net/core/dev.c:6772
 napi_poll net/core/dev.c:6841 [inline]
 net_rx_action+0x89b/0x1240 net/core/dev.c:6963
 handle_softirqs+0x2c4/0x970 kernel/softirq.c:554
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 5352 Comm: syz.1.14 Not tainted 6.11.0-rc3-syzkaller-00036-g9d5906799f7d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024
RIP: 0010:io_serial_out+0x7e/0xc0 drivers/tty/serial/8250/8250_port.c:413
Code: fc 89 e9 41 d3 e7 48 83 c3 40 48 89 d8 48 c1 e8 03 42 80 3c 20 00 74 08 48 89 df e8 ec db bb fc 44 03 3b 44 89 f0 44 89 fa ee <5b> 41 5c 41 5e 41 5f 5d c3 cc cc cc cc 89 e9 80 e1 07 38 c1 7c a7
RSP: 0018:ffffc90000006810 EFLAGS: 00000006
RAX: 0000000000000000 RBX: ffffffff95254f80 RCX: 0000000000000000
RDX: 00000000000003f9 RSI: 0000000000000000 RDI: 0000000000000020
RBP: 0000000000000000 R08: ffffffff853ee2ab R09: fffff52000000ce4
R10: dffffc0000000000 R11: ffffffff853ee260 R12: dffffc0000000000
R13: ffffffff95254f98 R14: 0000000000000000 R15: 00000000000003f9
FS:  00007f273fe756c0(0000) GS:ffff8880b9200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001b32713ff8 CR3: 0000000072172000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <IRQ>
 serial8250_console_write+0x332/0x1770 drivers/tty/serial/8250/8250_port.c:3358
 console_emit_next_record kernel/printk/printk.c:2983 [inline]
 console_flush_all+0x865/0xfd0 kernel/printk/printk.c:3049
 console_unlock+0x13b/0x4d0 kernel/printk/printk.c:3118
 vprintk_emit+0x5a6/0x770 kernel/printk/printk.c:2348
 _printk+0xd5/0x120 kernel/printk/printk.c:2373
 printk_stack_address arch/x86/kernel/dumpstack.c:72 [inline]
 show_trace_log_lvl+0x43a/0x520 arch/x86/kernel/dumpstack.c:285
 watchdog_timer_fn+0x75b/0x960 kernel/watchdog.c:759
 __run_hrtimer kernel/time/hrtimer.c:1689 [inline]
 __hrtimer_run_queues+0x551/0xd50 kernel/time/hrtimer.c:1753
 hrtimer_interrupt+0x396/0x990 kernel/time/hrtimer.c:1815
 local_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1032 [inline]
 __sysvec_apic_timer_interrupt+0x110/0x3f0 arch/x86/kernel/apic/apic.c:1049
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
 sysvec_apic_timer_interrupt+0x52/0xc0 arch/x86/kernel/apic/apic.c:1043
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:__debug_check_no_obj_freed lib/debugobjects.c:979 [inline]
RIP: 0010:debug_check_no_obj_freed+0x22f/0x580 lib/debugobjects.c:1020
Code: 48 b9 eb 83 b5 80 46 86 c8 61 48 0f af c1 48 c1 e8 2f 83 e0 f8 4c 8d bc c0 00 7f 0d 95 48 8d bc c0 08 7f 0d 95 48 89 7c 24 20 <e8> 6c ce 07 07 48 89 44 24 18 4c 89 fb 48 c1 eb 03 48 b8 00 00 00
RSP: 0018:ffffc90000007480 EFLAGS: 00000206
RAX: 000000000000c828 RBX: ffff888079702640 RCX: 61c8864680b583eb
RDX: 0000000000000001 RSI: ffffc90000007540 RDI: ffffffff95148870
RBP: ffffc900000075c8 R08: ffffc90000007547 R09: 0000000000000000
R10: ffffc90000007520 R11: fffff52000000ea9 R12: ffff888079702730
R13: ffff888079702640 R14: 0000000000000000 R15: ffffffff95148868
 slab_free_hook mm/slub.c:2219 [inline]
 slab_free mm/slub.c:4473 [inline]
 kmem_cache_free+0x10f/0x350 mm/slub.c:4548
 kfree_skb_reason include/linux/skbuff.h:1260 [inline]
 __netif_receive_skb_core+0x3edd/0x4570 net/core/dev.c:5635
 __netif_receive_skb_one_core net/core/dev.c:5659 [inline]
 __netif_receive_skb+0x12f/0x650 net/core/dev.c:5775
 process_backlog+0x662/0x15b0 net/core/dev.c:6108
 __napi_poll+0xcb/0x490 net/core/dev.c:6772
 napi_poll net/core/dev.c:6841 [inline]
 net_rx_action+0x89b/0x1240 net/core/dev.c:6963
 handle_softirqs+0x2c4/0x970 kernel/softirq.c:554
 __do_softirq kernel/softirq.c:588 [inline]
 invoke_softirq kernel/softirq.c:428 [inline]
 __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:649
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
 sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:__text_poke+0xa4a/0xd30 arch/x86/kernel/alternative.c:1960
Code: 7c 24 50 00 75 19 e8 25 45 5e 00 eb 18 e8 1e 45 5e 00 e8 09 16 81 0a 48 83 7c 24 50 00 74 e7 e8 0c 45 5e 00 fb 48 8b 44 24 78 <42> 80 3c 28 00 74 0d 48 8d bc 24 60 01 00 00 e8 42 8a c5 00 48 8b
RSP: 0018:ffffc900044af5c0 EFLAGS: 00000287
RAX: 1ffff92000895ee4 RBX: 0000000000000000 RCX: 0000000000040000
RDX: ffffc90004551000 RSI: 0000000000002805 RDI: 0000000000002806
RBP: ffffc900044af790 R08: ffffffff81353434 R09: 1ffffffff26e5f08
R10: dffffc0000000000 R11: fffffbfff26e5f09 R12: 1ffff92000895ec8
R13: dffffc0000000000 R14: 0000000000000046 R15: ffffffff816fe490
 text_poke arch/x86/kernel/alternative.c:1984 [inline]
 text_poke_bp_batch+0x265/0xb30 arch/x86/kernel/alternative.c:2292
 text_poke_flush arch/x86/kernel/alternative.c:2486 [inline]
 text_poke_finish+0x30/0x50 arch/x86/kernel/alternative.c:2493
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 static_key_enable_cpuslocked+0x136/0x260 kernel/jump_label.c:210
 static_key_enable+0x1a/0x20 kernel/jump_label.c:223
 tracepoint_add_func+0x953/0x9e0 kernel/tracepoint.c:361
 tracepoint_probe_register_prio kernel/tracepoint.c:511 [inline]
 tracepoint_probe_register+0x105/0x160 kernel/tracepoint.c:531
 perf_trace_event_reg kernel/trace/trace_event_perf.c:129 [inline]
 perf_trace_event_init+0x478/0x930 kernel/trace/trace_event_perf.c:202
 perf_trace_init+0x243/0x2e0 kernel/trace/trace_event_perf.c:226
 perf_tp_event_init+0x8d/0x110 kernel/events/core.c:10245
 perf_try_init_event+0x139/0x3f0 kernel/events/core.c:11720
 perf_init_event kernel/events/core.c:11790 [inline]
 perf_event_alloc+0x131a/0x22d0 kernel/events/core.c:12070
 __do_sys_perf_event_open kernel/events/core.c:12577 [inline]
 __se_sys_perf_event_open+0xb43/0x38d0 kernel/events/core.c:12468
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f273f1799b9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f273fe75038 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: ffffffffffffffda RBX: 00007f273f315f80 RCX: 00007f273f1799b9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000040
RBP: 00007f273f1e78d8 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f273f315f80 R15: 00007ffc29607bf8
 </TASK>
 __do_softirq kernel/softirq.c:588 [inline]
 invoke_softirq kernel/softirq.c:428 [inline]
 __irq_exit_rcu+0xf4/0x1c0 kernel/softirq.c:637
 irq_exit_rcu+0x9/0x30 kernel/softirq.c:649
 instr_sysvec_apic_timer_interrupt arch/x86/kernel/apic/apic.c:1043 [inline]
 sysvec_apic_timer_interrupt+0xa6/0xc0 arch/x86/kernel/apic/apic.c:1043
 </IRQ>
 <TASK>
 asm_sysvec_apic_timer_interrupt+0x1a/0x20 arch/x86/include/asm/idtentry.h:702
RIP: 0010:__text_poke+0xa4a/0xd30 arch/x86/kernel/alternative.c:1960
Code: 7c 24 50 00 75 19 e8 25 45 5e 00 eb 18 e8 1e 45 5e 00 e8 09 16 81 0a 48 83 7c 24 50 00 74 e7 e8 0c 45 5e 00 fb 48 8b 44 24 78 <42> 80 3c 28 00 74 0d 48 8d bc 24 60 01 00 00 e8 42 8a c5 00 48 8b
RSP: 0018:ffffc900044af5c0 EFLAGS: 00000287
RAX: 1ffff92000895ee4 RBX: 0000000000000000 RCX: 0000000000040000
RDX: ffffc90004551000 RSI: 0000000000002805 RDI: 0000000000002806
RBP: ffffc900044af790 R08: ffffffff81353434 R09: 1ffffffff26e5f08
R10: dffffc0000000000 R11: fffffbfff26e5f09 R12: 1ffff92000895ec8
R13: dffffc0000000000 R14: 0000000000000046 R15: ffffffff816fe490
 text_poke arch/x86/kernel/alternative.c:1984 [inline]
 text_poke_bp_batch+0x265/0xb30 arch/x86/kernel/alternative.c:2292
 text_poke_flush arch/x86/kernel/alternative.c:2486 [inline]
 text_poke_finish+0x30/0x50 arch/x86/kernel/alternative.c:2493
 arch_jump_label_transform_apply+0x1c/0x30 arch/x86/kernel/jump_label.c:146
 static_key_enable_cpuslocked+0x136/0x260 kernel/jump_label.c:210
 static_key_enable+0x1a/0x20 kernel/jump_label.c:223
 tracepoint_add_func+0x953/0x9e0 kernel/tracepoint.c:361
 tracepoint_probe_register_prio kernel/tracepoint.c:511 [inline]
 tracepoint_probe_register+0x105/0x160 kernel/tracepoint.c:531
 perf_trace_event_reg kernel/trace/trace_event_perf.c:129 [inline]
 perf_trace_event_init+0x478/0x930 kernel/trace/trace_event_perf.c:202
 perf_trace_init+0x243/0x2e0 kernel/trace/trace_event_perf.c:226
 perf_tp_event_init+0x8d/0x110 kernel/events/core.c:10245
 perf_try_init_event+0x139/0x3f0 kernel/events/core.c:11720
 perf_init_event kernel/events/core.c:11790 [inline]
 perf_event_alloc+0x131a/0x22d0 kernel/events/core.c:12070
 __do_sys_perf_event_open kernel/events/core.c:12577 [inline]
 __se_sys_perf_event_open+0xb43/0x38d0 kernel/events/core.c:12468
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f273f1799b9
Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 a8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f273fe75038 EFLAGS: 00000246 ORIG_RAX: 000000000000012a
RAX: ffffffffffffffda RBX: 00007f273f315f80 RCX: 00007f273f1799b9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000040
RBP: 00007f273f1e78d8 R08: 0000000000000000 R09: 0000000000000000
R10: ffffffffffffffff R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f273f315f80 R15: 00007ffc29607bf8
 </TASK>
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 UID: 0 PID: 2500 Comm: kworker/u8:8 Not tainted 6.11.0-rc3-syzkaller-00036-g9d5906799f7d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 06/27/2024
Workqueue: writeback wb_workfn (flush-8:0)
RIP: 0010:csd_lock_wait kernel/smp.c:312 [inline]
RIP: 0010:smp_call_function_many_cond+0x1865/0x29d0 kernel/smp.c:856
Code: 89 e6 83 e6 01 31 ff e8 99 14 0c 00 41 83 e4 01 49 bc 00 00 00 00 00 fc ff df 75 07 e8 44 10 0c 00 eb 38 f3 90 42 0f b6 04 23 <84> c0 75 11 41 f7 45 00 01 00 00 00 74 1e e8 28 10 0c 00 eb e4 44
RSP: 0018:ffffc900094fe220 EFLAGS: 00000293
RAX: 0000000000000000 RBX: 1ffff11017248c61 RCX: ffff888029f51e00
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
RBP: ffffc900094fe400 R08: ffffffff81876917 R09: 1ffffffff202fa55
R10: dffffc0000000000 R11: ffffffff8142d5e0 R12: dffffc0000000000
R13: ffff8880b9246308 R14: ffff8880b933fb80 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8880b9300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f36738a2020 CR3: 00000000235a0000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 on_each_cpu_cond_mask+0x3f/0x80 kernel/smp.c:1023
 __flush_tlb_multi arch/x86/include/asm/paravirt.h:91 [inline]
 flush_tlb_multi arch/x86/mm/tlb.c:941 [inline]
 flush_tlb_mm_range+0x329/0x5c0 arch/x86/mm/tlb.c:1027
 flush_tlb_page arch/x86/include/asm/tlbflush.h:254 [inline]
 ptep_clear_flush+0x11a/0x170 mm/pgtable-generic.c:101
 page_vma_mkclean_one+0x58a/0x9a0 mm/rmap.c:1029
 page_mkclean_one+0x280/0x420 mm/rmap.c:1070
 rmap_walk_file+0x52f/0x9f0 mm/rmap.c:2687
 rmap_walk mm/rmap.c:2705 [inline]
 folio_mkclean+0x262/0x440 mm/rmap.c:1102
 folio_clear_dirty_for_io+0x22b/0xcf0 mm/page-writeback.c:3024
 mpage_submit_folio+0x88/0x230 fs/ext4/inode.c:1924
 mpage_map_and_submit_buffers fs/ext4/inode.c:2189 [inline]
 mpage_map_and_submit_extent fs/ext4/inode.c:2328 [inline]
 ext4_do_writepages+0x1db0/0x3d40 fs/ext4/inode.c:2753
 ext4_writepages+0x213/0x3c0 fs/ext4/inode.c:2842
 do_writepages+0x35d/0x870 mm/page-writeback.c:2683
 __writeback_single_inode+0x165/0x10b0 fs/fs-writeback.c:1651
 writeback_sb_inodes+0x99c/0x1380 fs/fs-writeback.c:1947
 __writeback_inodes_wb+0x11b/0x260 fs/fs-writeback.c:2018
 wb_writeback+0x495/0xd40 fs/fs-writeback.c:2129
 wb_check_old_data_flush fs/fs-writeback.c:2233 [inline]
 wb_do_writeback fs/fs-writeback.c:2286 [inline]
 wb_workfn+0xba1/0x1090 fs/fs-writeback.c:2314
 process_one_work kernel/workqueue.c:3231 [inline]
 process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
 worker_thread+0x86d/0xd40 kernel/workqueue.c:3390
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Crashes (2):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/08/15 05:28 bpf 9d5906799f7d e4bacdaf .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-bpf-kasan-gce BUG: soft lockup in debug_check_no_obj_freed
2024/10/19 15:36 https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb.git usb-testing 07b887f8236e cd6fc0a3 .config console log report syz / log [disk image] [vmlinux] [kernel image] ci2-upstream-usb INFO: rcu detected stall in debug_check_no_obj_freed
* Struck through repros no longer work on HEAD.