syzbot


INFO: task hung in bch2_fs_read_only_work

Status: upstream: reported C repro on 2024/05/11 22:46
Subsystems: bcachefs
[Documentation on labels]
Reported-by: syzbot+8996d8f176cf946ef641@syzkaller.appspotmail.com
First crash: 80d, last: 5d01h
Cause bisection: introduced by (bisect log) :
commit 03ef80b469d5d83530ce1ce15be78a40e5300f9b
Author: Kent Overstreet <kent.overstreet@linux.dev>
Date: Sat Sep 23 22:41:51 2023 +0000

  bcachefs: Ignore unknown mount options

Crash: BUG: soft lockup in sys_mount (log)
Repro: C syz .config
  
Discussions (2)
Title Replies (including bot) Last reply
[PATCH] bcachefs: fix hung in bch2_fs_read_only_work 2 (2) 2024/07/05 01:18
[syzbot] [bcachefs?] INFO: task hung in bch2_fs_read_only_work 2 (8) 2024/07/01 08:45
Last patch testing requests (3)
Created Duration User Patch Repo Result
2024/07/01 08:45 1h08m lizhi.xu@windriver.com patch git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git f06ce441457d OK log
2024/06/30 11:39 56m hdanton@sina.com patch https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master report log
2024/06/30 09:24 1h03m hdanton@sina.com https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master report log

Sample crash report:
INFO: task kworker/1:0:25 blocked for more than 143 seconds.
      Not tainted 6.10.0-rc2-syzkaller-00007-gf06ce441457d #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/1:0     state:D
 stack:26248 pid:25    tgid:25    ppid:2      flags:0x00004000
Workqueue: events_long bch2_fs_read_only_work
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5408 [inline]
 __schedule+0x1796/0x49d0 kernel/sched/core.c:6745
 __schedule_loop kernel/sched/core.c:6822 [inline]
 schedule+0x14b/0x320 kernel/sched/core.c:6837
 schedule_preempt_disabled+0x13/0x30 kernel/sched/core.c:6894
 rwsem_down_write_slowpath+0xeeb/0x13b0 kernel/locking/rwsem.c:1178
 __down_write_common+0x1af/0x200 kernel/locking/rwsem.c:1306
 bch2_fs_read_only_work+0x25/0x40 fs/bcachefs/super.c:392
 process_one_work kernel/workqueue.c:3231 [inline]
 process_scheduled_works+0xa2c/0x1830 kernel/workqueue.c:3312
 worker_thread+0x86d/0xd70 kernel/workqueue.c:3393
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>

Showing all locks held in the system:
3 locks held by kworker/1:0/25:
 #0: ffff888015081148 ((wq_completion)events_long){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3206 [inline]
 #0: ffff888015081148 ((wq_completion)events_long){+.+.}-{0:0}, at: process_scheduled_works+0x90a/0x1830 kernel/workqueue.c:3312
 #1: ffffc900001f7d00 ((work_completion)(&c->read_only_work)){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3207 [inline]
 #1: ffffc900001f7d00 ((work_completion)(&c->read_only_work)){+.+.}-{0:0}, at: process_scheduled_works+0x945/0x1830 kernel/workqueue.c:3312
 #2: ffff888075980278 (&c->state_lock){+.+.}-{3:3}, at: bch2_fs_read_only_work+0x25/0x40 fs/bcachefs/super.c:392
1 lock held by khungtaskd/30:
 #0: ffffffff8e333f60 (rcu_read_lock){....}-{1:2}, at: rcu_lock_acquire include/linux/rcupdate.h:329 [inline]
 #0: ffffffff8e333f60 (rcu_read_lock){....}-{1:2}, at: rcu_read_lock include/linux/rcupdate.h:781 [inline]
 #0: ffffffff8e333f60 (rcu_read_lock){....}-{1:2}, at: debug_show_all_locks+0x55/0x2a0 kernel/locking/lockdep.c:6614
5 locks held by kworker/u8:2/35:
1 lock held by klogd/4522:
2 locks held by getty/4833:
 #0: ffff88802fb4a0a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
 #1: ffffc90002f062f0 (&ldata->atomic_read_lock){+.+.}-{3:3}, at: n_tty_read+0x6b5/0x1e10 drivers/tty/n_tty.c:2201
3 locks held by syz-executor131/5078:

=============================================

NMI backtrace for cpu 1
CPU: 1 PID: 30 Comm: khungtaskd Not tainted 6.10.0-rc2-syzkaller-00007-gf06ce441457d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0x241/0x360 lib/dump_stack.c:114
 nmi_cpu_backtrace+0x49c/0x4d0 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x198/0x320 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:162 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:223 [inline]
 watchdog+0xfde/0x1020 kernel/hung_task.c:379
 kthread+0x2f0/0x390 kernel/kthread.c:389
 ret_from_fork+0x4b/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:244
 </TASK>
Sending NMI from CPU 1 to CPUs 0:
NMI backtrace for cpu 0
CPU: 0 PID: 4515 Comm: syslogd Not tainted 6.10.0-rc2-syzkaller-00007-gf06ce441457d #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/02/2024
RIP: 0010:skb_ext_put include/linux/skbuff.h:4688 [inline]
RIP: 0010:skb_release_head_state+0x19d/0x250 net/core/skbuff.c:1165
Code: 4e f8 eb 16 31 ff 89 ee e8 80 50 4e f8 85 ed 0f 8e 8a 00 00 00 e8 33 4c 4e f8 49 8d 5e 7f 48 89 d8 48 c1 e8 03 42 0f b6 04 20 <84> c0 0f 85 84 00 00 00 0f b6 1b 31 ff 89 de e8 0f 4f 4e f8 85 db
RSP: 0018:ffffc90004907908 EFLAGS: 00000a07
RAX: 0000000000000000 RBX: ffff88802e28ecff RCX: ffff88807d280000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffffffff8947c904 R09: 1ffff11003ca5853
R10: dffffc0000000000 R11: ffffed1003ca5854 R12: dffffc0000000000
R13: ffff88802e28ec80 R14: ffff88802e28ec80 R15: ffff88802e28ece0
FS:  00007fa63915c380(0000) GS:ffff8880b9400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055e2897e7600 CR3: 000000002dc2c000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <NMI>
 </NMI>
 <TASK>
 skb_release_all net/core/skbuff.c:1171 [inline]
 __kfree_skb net/core/skbuff.c:1187 [inline]
 consume_skb+0x7b/0x160 net/core/skbuff.c:1409
 __unix_dgram_recvmsg+0xcea/0x12f0 net/unix/af_unix.c:2474
 sock_recvmsg_nosec net/socket.c:1046 [inline]
 sock_recvmsg+0x22f/0x280 net/socket.c:1068
 sock_read_iter+0x2ca/0x3e0 net/socket.c:1138
 new_sync_read fs/read_write.c:395 [inline]
 vfs_read+0x9c4/0xbd0 fs/read_write.c:476
 ksys_read+0x1a0/0x2c0 fs/read_write.c:619
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xf3/0x230 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fa6392b0b6a
Code: 00 3d 00 00 41 00 75 0d 50 48 8d 3d 2d 08 0a 00 e8 ea 7d 01 00 31 c0 e9 07 ff ff ff 64 8b 04 25 18 00 00 00 85 c0 75 1b 0f 05 <48> 3d 00 f0 ff ff 76 6c 48 8b 15 8f a2 0d 00 f7 d8 64 89 02 48 83
RSP: 002b:00007ffec9bf54e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fa6392b0b6a
RDX: 00000000000000ff RSI: 000055cf5b545950 RDI: 0000000000000000
RBP: 000055cf5b545910 R08: 0000000000000001 R09: 0000000000000000
R10: 00007fa63944f3a3 R11: 0000000000000246 R12: 000055cf5b54598a
R13: 000055cf5b545950 R14: 0000000000000000 R15: 00007fa639493a80
 </TASK>
INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.151 msecs

Crashes (200):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2024/06/04 03:28 upstream f06ce441457d a1feae05 .config strace log report syz / log C [disk image] [vmlinux] [kernel image] [mounted in repro] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/17 20:09 upstream 51835949dda3 0f902625 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/17 03:26 upstream 408323581b72 215bec2d .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/07/17 00:05 upstream 408323581b72 215bec2d .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/07/16 17:17 upstream d67978318827 b66b37bd .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/15 14:19 upstream 0c3836482481 c605e6a2 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/14 07:57 upstream 4d145e3f830b eaeb5c15 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/13 21:56 upstream 528dd46d0fc3 eaeb5c15 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/10 22:31 upstream a19ea421490d 6f7fa383 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/10 20:11 upstream a19ea421490d 6f7fa383 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/10 05:23 upstream 34afb82a3c67 79d68ada .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/07/10 02:55 upstream 34afb82a3c67 bc144f9a .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/09 22:40 upstream 34afb82a3c67 79d68ada .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/09 03:17 upstream 4376e966ecb7 bc23a442 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/09 00:40 upstream 4376e966ecb7 bc23a442 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/08 23:17 upstream 4376e966ecb7 cde64f7d .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/08 12:42 upstream 256abd8e550c cde64f7d .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/08 07:18 upstream 256abd8e550c 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/07 22:55 upstream c6653f49e4fd 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/07 14:49 upstream c6653f49e4fd 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/06 08:25 upstream d270dd21bee0 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/06 00:39 upstream d270dd21bee0 2a40360c .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/04 16:08 upstream 795c58e4c7fc 3f2748a3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/04 03:01 upstream 8a9c6c40432e f76a75f3 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/02 22:46 upstream 1dfe225e9af5 07f0a0a0 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/07/01 17:51 upstream 22a40d14b572 b294e901 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/30 20:30 upstream e0b668b07034 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/30 16:57 upstream 8282d5af7be8 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/29 05:27 upstream de0a9f448633 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/29 01:13 upstream 5bbd9b249880 757f06b1 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/26 07:47 upstream 55027e689933 dec8bc94 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/25 23:03 upstream 55027e689933 da0bd5ca .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/06/25 02:42 upstream 626737a5791b 215eef4a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/25 00:09 upstream 626737a5791b 215eef4a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/24 21:04 upstream 626737a5791b 215eef4a .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/24 03:34 upstream 7c16f0a4ed1c edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/23 22:52 upstream 7c16f0a4ed1c edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/23 09:10 upstream 5f583a3162ff edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/23 03:28 upstream 5f583a3162ff edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/06/23 00:42 upstream 35bb670d65fc edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/22 01:40 upstream 66cc544fd75c edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/21 14:14 upstream 50736169ecc8 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/21 11:46 upstream 50736169ecc8 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/21 11:33 upstream 50736169ecc8 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/21 09:59 upstream 50736169ecc8 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/21 04:10 upstream 50736169ecc8 dac2aa43 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/20 09:37 upstream 2ccbdf43d5e7 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/20 07:59 upstream 2ccbdf43d5e7 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/19 17:33 upstream 2ccbdf43d5e7 41b7e219 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/18 21:56 upstream 2ccbdf43d5e7 639d6cdf .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/16 22:37 upstream 2ccbdf43d5e7 f429ab00 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/06/16 07:41 upstream 2ccbdf43d5e7 f429ab00 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/15 04:57 upstream 2ccbdf43d5e7 8d849073 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/15 01:08 upstream 2ccbdf43d5e7 8d849073 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/14 05:28 upstream 2ccbdf43d5e7 a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-smack-root INFO: task hung in bch2_fs_read_only_work
2024/06/14 01:37 upstream 2ccbdf43d5e7 a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-selinux-root INFO: task hung in bch2_fs_read_only_work
2024/06/13 20:07 upstream 2ccbdf43d5e7 a9616ff5 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/13 12:54 upstream 2ccbdf43d5e7 2aa5052f .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/06/12 13:30 upstream 2ef5971ff345 4d75f4f7 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/05/11 18:48 upstream cf87f46fd34d 9026e142 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/05/06 04:07 upstream b9158815de52 610f2a54 .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/05/03 18:51 upstream f03359bca01b dd26401e .config console log report info [disk image] [vmlinux] [kernel image] ci2-upstream-fs INFO: task hung in bch2_fs_read_only_work
2024/06/25 04:00 linux-next 62c97045b8f7 215eef4a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-linux-next-kasan-gce-root INFO: task hung in bch2_fs_read_only_work
2024/07/11 12:20 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci ef445d1539dd c699c2eb .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 INFO: task hung in bch2_fs_read_only_work
2024/06/28 02:48 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci b4a3f9b4863a b62c7d46 .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 INFO: task hung in bch2_fs_read_only_work
2024/06/24 17:44 git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci ac2193b4b460 edc5149a .config console log report info [disk image] [vmlinux] [kernel image] ci-upstream-gce-arm64 INFO: task hung in bch2_fs_read_only_work
* Struck through repros no longer work on HEAD.