syzbot


INFO: task hung in blkdev_get_by_dev

Status: fixed on 2025/01/02 10:34
Bug presence: origin:lts-only
[Documentation on labels]
Reported-by: syzbot+769a3aefabbe0951666b@syzkaller.appspotmail.com
Fix commit: eff23e59b787 devlink: bump the instance index directly when iterating
First crash: 365d, last: 78d
Fix bisection: fixed by (bisect log) :
commit eff23e59b7871eba63de916a3c9f091cae2253a0
Author: Jakub Kicinski <kuba@kernel.org>
Date: Fri Jan 6 06:33:54 2023 +0000

  devlink: bump the instance index directly when iterating

  
Bug presence (2)
Date Name Commit Repro Result
2024/04/30 linux-6.1.y (ToT) dcbc050cb0d3 C [report] INFO: rcu detected stall in corrupted
2024/04/30 upstream (ToT) 98369dccd2f8 C Didn't crash
Similar bugs (8)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
linux-5.15 INFO: task hung in blkdev_get_by_dev (2) 1 507d 507d 0/3 auto-obsoleted due to no activity on 2023/12/18 22:46
upstream INFO: task hung in blkdev_get_by_dev (3) block 1 1056d 1056d 0/28 auto-closed as invalid on 2022/05/08 09:47
linux-5.15 INFO: task hung in blkdev_get_by_dev 1 638d 638d 0/3 auto-obsoleted due to no activity on 2023/08/23 09:06
linux-5.15 INFO: task hung in blkdev_get_by_dev (3) 17 142d 362d 0/3 auto-obsoleted due to no activity on 2024/12/17 17:06
upstream INFO: task hung in blkdev_get_by_dev fs 1 1436d 1436d 0/28 auto-closed as invalid on 2021/05/23 22:12
upstream INFO: task hung in blkdev_get_by_dev (4) block 1 885d 885d 0/28 auto-obsoleted due to no activity on 2022/11/25 11:03
upstream INFO: task hung in blkdev_get_by_dev (2) block 1727 1088d 1167d 0/28 closed as dup on 2021/11/21 14:02
upstream INFO: task hung in blkdev_get_by_dev (5) block C error 32 23d 698d 0/28 upstream: reported C repro on 2023/03/02 18:11
Fix bisection attempts (4)
Created Duration User Patch Repo Result
2024/12/28 17:09 8h24m bisect fix linux-6.1.y OK (1) job log
2024/11/11 06:54 2h28m bisect fix linux-6.1.y OK (0) job log log
2024/05/08 15:17 2h30m fix candidate upstream OK (0) job log log
2024/04/30 16:24 2h54m bisect fix linux-6.1.y OK (0) job log log

Sample crash report:
INFO: task syz.0.445:5187 blocked for more than 143 seconds.
      Not tainted 6.1.108-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.0.445       state:D stack:26656 pid:5187  ppid:3637   flags:0x00004004
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5241 [inline]
 __schedule+0x143f/0x4570 kernel/sched/core.c:6558
 schedule+0xbf/0x180 kernel/sched/core.c:6634
 schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:6693
 __mutex_lock_common kernel/locking/mutex.c:679 [inline]
 __mutex_lock+0x6b9/0xd80 kernel/locking/mutex.c:747
 blkdev_get_by_dev+0x148/0xa10 block/bdev.c:815
 blkdev_open+0x12e/0x2e0 block/fops.c:500
 do_dentry_open+0x7f9/0x10f0 fs/open.c:882
 do_open fs/namei.c:3628 [inline]
 path_openat+0x2644/0x2e60 fs/namei.c:3785
 do_filp_open+0x230/0x480 fs/namei.c:3812
 do_sys_openat2+0x13b/0x4f0 fs/open.c:1318
 do_sys_open fs/open.c:1334 [inline]
 __do_sys_openat fs/open.c:1350 [inline]
 __se_sys_openat fs/open.c:1345 [inline]
 __x64_sys_openat+0x243/0x290 fs/open.c:1345
 do_syscall_x64 arch/x86/entry/common.c:51 [inline]
 do_syscall_64+0x3b/0xb0 arch/x86/entry/common.c:81
 entry_SYSCALL_64_after_hwframe+0x68/0xd2
RIP: 0033:0x7f1a2d77b890
RSP: 002b:00007f1a2e539b70 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1a2d77b890
RDX: 0000000000000000 RSI: 00007f1a2e539c10 RDI: 00000000ffffff9c
RBP: 00007f1a2e539c10 R08: 0000000000000000 R09: 002364626e2f7665
R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f1a2d935f80 R15: 00007ffc9bf36488
 </TASK>
INFO: task syz.0.445:5190 blocked for more than 143 seconds.
      Not tainted 6.1.108-syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.0.445       state:D stack:27744 pid:5190  ppid:3637   flags:0x00004004
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5241 [inline]
 __schedule+0x143f/0x4570 kernel/sched/core.c:6558
 schedule+0xbf/0x180 kernel/sched/core.c:6634
 schedule_preempt_disabled+0xf/0x20 kernel/sched/core.c:6693
 __mutex_lock_common kernel/locking/mutex.c:679 [inline]
 __mutex_lock+0x6b9/0xd80 kernel/locking/mutex.c:747
 blkdev_put+0x100/0x750 block/bdev.c:913
 blkdev_close+0x56/0x80 block/fops.c:514
 __fput+0x3f6/0x8d0 fs/file_table.c:320
 task_work_run+0x246/0x300 kernel/task_work.c:203
 resume_user_mode_work include/linux/resume_user_mode.h:49 [inline]
 exit_to_user_mode_loop+0xde/0x100 kernel/entry/common.c:177
 exit_to_user_mode_prepare+0xb1/0x140 kernel/entry/common.c:210
 __syscall_exit_to_user_mode_work kernel/entry/common.c:292 [inline]
 syscall_exit_to_user_mode+0x60/0x270 kernel/entry/common.c:303
 do_syscall_64+0x47/0xb0 arch/x86/entry/common.c:87
 entry_SYSCALL_64_after_hwframe+0x68/0xd2
RIP: 0033:0x7f1a2d77cef9
RSP: 002b:00007f1a2e519038 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: 0000000000000000 RBX: 00007f1a2d936058 RCX: 00007f1a2d77cef9
RDX: 0000000000000000 RSI: 000000000000ab03 RDI: 0000000000000003
RBP: 00007f1a2d7ef01e R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000001 R14: 00007f1a2d936058 R15: 00007ffc9bf36488
 </TASK>

Showing all locks held in the system:
5 locks held by kworker/u4:0/9:
6 locks held by kworker/u4:1/11:
1 lock held by rcu_tasks_kthre/12:
 #0: ffffffff8d32b190 (rcu_tasks.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x29/0xe30 kernel/rcu/tasks.h:517
1 lock held by rcu_tasks_trace/13:
 #0: ffffffff8d32b990 (rcu_tasks_trace.tasks_gp_mutex){+.+.}-{3:3}, at: rcu_tasks_one_gp+0x29/0xe30 kernel/rcu/tasks.h:517
1 lock held by khungtaskd/28:
 #0: ffffffff8d32afc0 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:350 [inline]
 #0: ffffffff8d32afc0 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:791 [inline]
 #0: ffffffff8d32afc0 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x51/0x290 kernel/locking/lockdep.c:6494
1 lock held by dhcpcd/3305:
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: __netlink_dump_start+0x12e/0x6c0 net/netlink/af_netlink.c:2301
2 locks held by getty/3397:
 #0: ffff88802e1c5098 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x21/0x70 drivers/tty/tty_ldisc.c:244
 #1: ffffc900031262f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6a7/0x1db0 drivers/tty/n_tty.c:2198
3 locks held by kworker/1:3/3646:
 #0: ffff88814b874138 ((wq_completion)ipv6_addrconf){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #1: ffffc90003dafd20 ((work_completion)(&(&ifa->dad_work)->work)){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #2: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: addrconf_dad_work+0xcc/0x16b0 net/ipv6/addrconf.c:4115
3 locks held by kworker/1:5/3687:
 #0: ffff888017c70938 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #1: ffffc90003fbfd20 (free_ipc_work){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #2: ffffffff8d3305b8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: exp_funnel_lock kernel/rcu/tree_exp.h:323 [inline]
 #2: ffffffff8d3305b8 (rcu_state.exp_mutex){+.+.}-{3:3}, at: synchronize_rcu_expedited+0x360/0x930 kernel/rcu/tree_exp.h:962
3 locks held by kworker/1:6/3690:
 #0: ffff8880b8f3a9d8 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x26/0x140 kernel/sched/core.c:537
 #1: ffff8880b8f27788 (&per_cpu_ptr(group->pcpu, cpu)->seq){-.-.}-{0:0}, at: psi_task_switch+0x43d/0x770 kernel/sched/psi.c:1001
 #2: ffff8880b8f28358 (&base->lock){-.-.}-{2:2}, at: lock_timer_base+0x120/0x260 kernel/time/timer.c:999
1 lock held by udevd/3718:
 #0: ffff888143f2a4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0x148/0xa10 block/bdev.c:815
2 locks held by kworker/1:7/3719:
 #0: ffff888017c72138 ((wq_completion)rcu_gp){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #1: ffffc9000431fd20 ((work_completion)(&rew->rew_work)){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
1 lock held by syz.0.445/5187:
 #0: ffff888143f2a4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_get_by_dev+0x148/0xa10 block/bdev.c:815
1 lock held by syz.0.445/5190:
 #0: ffff888143f2a4c8 (&disk->open_mutex){+.+.}-{3:3}, at: blkdev_put+0x100/0x750 block/bdev.c:913
3 locks held by kworker/u4:15/5720:
 #0: ffff888017c79138 ((wq_completion)events_unbound){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #1: ffffc90019a2fd20 ((linkwatch_work).work){+.+.}-{0:0}, at: process_one_work+0x7a9/0x11d0 kernel/workqueue.c:2267
 #2: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: linkwatch_event+0xa/0x50 net/core/link_watch.c:263
3 locks held by syz-executor/6478:
 #0: ffff88805d6a50b8 (&hdev->req_lock){+.+.}-{3:3}, at: hci_dev_do_close net/bluetooth/hci_core.c:508 [inline]
 #0: ffff88805d6a50b8 (&hdev->req_lock){+.+.}-{3:3}, at: hci_unregister_dev+0x1ff/0x500 net/bluetooth/hci_core.c:2710
 #1: ffff88805d6a4078 (&hdev->lock){+.+.}-{3:3}, at: hci_dev_close_sync+0x48d/0x1020 net/bluetooth/hci_sync.c:5053
 #2: ffffffff8e650248 (hci_cb_list_lock){+.+.}-{3:3}, at: hci_disconn_cfm include/net/bluetooth/hci_core.h:1805 [inline]
 #2: ffffffff8e650248 (hci_cb_list_lock){+.+.}-{3:3}, at: hci_conn_hash_flush+0xb8/0x2a0 net/bluetooth/hci_conn.c:2446
1 lock held by syz-executor/6818:
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:74 [inline]
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x7c1/0xff0 net/core/rtnetlink.c:6118
1 lock held by syz-executor/6910:
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:74 [inline]
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x7c1/0xff0 net/core/rtnetlink.c:6118
1 lock held by syz-executor/6929:
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: rtnl_lock net/core/rtnetlink.c:74 [inline]
 #0: ffffffff8e4f8ea8 (rtnl_mutex){+.+.}-{3:3}, at: rtnetlink_rcv_msg+0x7c1/0xff0 net/core/rtnetlink.c:6118

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 28 Comm: khungtaskd Not tainted 6.1.108-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x1e3/0x2cb lib/dump_stack.c:106
 nmi_cpu_backtrace+0x4e1/0x560 lib/nmi_backtrace.c:111
 nmi_trigger_cpumask_backtrace+0x1ae/0x3f0 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:148 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:220 [inline]
 watchdog+0xf88/0xfd0 kernel/hung_task.c:377
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 3685 Comm: kworker/0:3 Not tainted 6.1.108-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 08/06/2024
Workqueue: events_power_efficient wg_ratelimiter_gc_entries
RIP: 0010:kasan_check_range+0x1ac/0x290 mm/kasan/generic.c:189
Code: 4c 89 d5 48 8d 5d 07 48 85 ed 48 0f 49 dd 48 83 e3 f8 48 29 dd 74 12 41 80 39 00 0f 85 a2 00 00 00 49 ff c1 48 ff cd 75 ee 5b <41> 5e 41 5f 5d c3 45 84 f6 75 61 41 f7 c6 00 ff 00 00 75 5d 41 f7
RSP: 0018:ffffc90003f9fa60 EFLAGS: 00000256
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffffffff816a82fb
RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffffff8e99db28
RBP: 0000000000000001 R08: dffffc0000000000 R09: fffffbfff1d33b66
R10: 0000000000000000 R11: dffffc0000000001 R12: 1ffff920007f3f58
R13: dffffc0000000000 R14: fffffbfff1d33b66 R15: ffffffff8dcffb20
FS:  0000000000000000(0000) GS:ffff8880b8e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3da4279440 CR3: 000000000d08e000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 instrument_atomic_read include/linux/instrumented.h:72 [inline]
 _test_bit include/asm-generic/bitops/instrumented-non-atomic.h:141 [inline]
 cpumask_test_cpu include/linux/cpumask.h:444 [inline]
 cpu_online include/linux/cpumask.h:1030 [inline]
 trace_lock_acquire include/trace/events/lock.h:24 [inline]
 lock_acquire+0xeb/0x5a0 kernel/locking/lockdep.c:5633
 __raw_spin_lock include/linux/spinlock_api_smp.h:133 [inline]
 _raw_spin_lock+0x2a/0x40 kernel/locking/spinlock.c:154
 spin_lock include/linux/spinlock.h:351 [inline]
 wg_ratelimiter_gc_entries+0x5c/0x470 drivers/net/wireguard/ratelimiter.c:63
 process_one_work+0x8a9/0x11d0 kernel/workqueue.c:2292
 worker_thread+0xa47/0x1200 kernel/workqueue.c:2439
 kthread+0x28d/0x320 kernel/kthread.c:376
 ret_from_fork+0x1f/0x30 arch/x86/entry/entry_64.S:295
 </TASK>

Crashes (25):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/09/06 01:15 linux-6.1.y 699506173494 464ac2ed .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/08/16 23:50 linux-6.1.y 117ac406ba90 e1c76ab2 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/08/15 18:42 linux-6.1.y 117ac406ba90 e4bacdaf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/08/02 05:56 linux-6.1.y c1cec4dad96b 1e9c4cf3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/07/28 01:13 linux-6.1.y c1cec4dad96b 46eb10b7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/07/26 16:19 linux-6.1.y c18e82d3ee44 46eb10b7 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/07/20 08:40 linux-6.1.y 9b3f9a5b12dc b88348e9 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/07/15 21:53 linux-6.1.y cac15753b8ce e8709b21 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/07/15 13:01 linux-6.1.y cac15753b8ce c605e6a2 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/07/06 10:25 linux-6.1.y 7753af06eebf 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/06/20 00:39 linux-6.1.y eb44d83053d6 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/06/12 23:23 linux-6.1.y ae9f2a70d69e f815599d .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/06/08 08:47 linux-6.1.y 88690811da69 82c05ab8 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/05/31 18:49 linux-6.1.y 88690811da69 0c378259 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/05/26 16:30 linux-6.1.y 88690811da69 a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/05/26 00:39 linux-6.1.y 88690811da69 a10a183e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/05/24 08:33 linux-6.1.y 4078fa637fcd 8f98448e .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/03/31 03:25 linux-6.1.y e5cd595e23c1 6baf5069 .config console log report syz C [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/03/30 16:51 linux-6.1.y e5cd595e23c1 6baf5069 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/03/26 12:25 linux-6.1.y d7543167affd bcd9b39f .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/03/25 10:15 linux-6.1.y d7543167affd 0ea90952 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/01/29 00:38 linux-6.1.y 883d1a956208 cc4a4020 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan INFO: task hung in blkdev_get_by_dev
2024/10/08 03:26 linux-6.1.y aa4cd140bba5 d7906eff .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 INFO: task hung in blkdev_get_by_dev
2024/08/15 15:33 linux-6.1.y 117ac406ba90 e4bacdaf .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 INFO: task hung in blkdev_get_by_dev
2024/06/29 15:00 linux-6.1.y 99e6a620de00 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-linux-6-1-kasan-arm64 INFO: task hung in blkdev_get_by_dev
* Struck through repros no longer work on HEAD.