syzbot


INFO: task hung in ovs_dp_cmd_dump

Status: auto-closed as invalid on 2020/01/14 06:18
Subsystems: openvswitch
[Documentation on labels]
Reported-by: syzbot+c53ee2ad43c048dfd390@syzkaller.appspotmail.com
First crash: 1693d, last: 1654d

Sample crash report:
INFO: task syz-executor.2:1246 blocked for more than 143 seconds.
      Not tainted 5.4.0-rc1+ #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
syz-executor.2  D27936  1246   1225 0x00000004
Call Trace:
 context_switch kernel/sched/core.c:3384 [inline]
 __schedule+0x94f/0x1e70 kernel/sched/core.c:4069
 schedule+0xd9/0x260 kernel/sched/core.c:4136
 schedule_preempt_disabled+0x13/0x20 kernel/sched/core.c:4195
 __mutex_lock_common kernel/locking/mutex.c:1033 [inline]
 __mutex_lock+0x7b0/0x13c0 kernel/locking/mutex.c:1103
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1118
 ovs_lock net/openvswitch/datapath.c:105 [inline]
 ovs_dp_cmd_dump+0x1f3/0x540 net/openvswitch/datapath.c:1820
 netlink_dump+0x558/0xfb0 net/netlink/af_netlink.c:2244
 __netlink_dump_start+0x5b1/0x7d0 net/netlink/af_netlink.c:2352
 genl_family_rcv_msg+0xbf3/0xf90 net/netlink/genetlink.c:583
 genl_rcv_msg+0xca/0x170 net/netlink/genetlink.c:654
 netlink_rcv_skb+0x177/0x450 net/netlink/af_netlink.c:2477
 genl_rcv+0x29/0x40 net/netlink/genetlink.c:665
 netlink_unicast_kernel net/netlink/af_netlink.c:1302 [inline]
 netlink_unicast+0x531/0x710 net/netlink/af_netlink.c:1328
 netlink_sendmsg+0x8a5/0xd60 net/netlink/af_netlink.c:1917
 sock_sendmsg_nosec net/socket.c:637 [inline]
 sock_sendmsg+0xd7/0x130 net/socket.c:657
 ___sys_sendmsg+0x803/0x920 net/socket.c:2311
 __sys_sendmsg+0x105/0x1d0 net/socket.c:2356
 __do_sys_sendmsg net/socket.c:2365 [inline]
 __se_sys_sendmsg net/socket.c:2363 [inline]
 __x64_sys_sendmsg+0x78/0xb0 net/socket.c:2363
 do_syscall_64+0xfa/0x760 arch/x86/entry/common.c:290
 entry_SYSCALL_64_after_hwframe+0x49/0xbe
RIP: 0033:0x459a59
Code: Bad RIP value.
RSP: 002b:00007f6a021c3c78 EFLAGS: 00000246 ORIG_RAX: 000000000000002e
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 0000000000459a59
RDX: 0000000000000000 RSI: 0000000020000040 RDI: 0000000000000003
RBP: 000000000075bfc8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f6a021c46d4
R13: 00000000004c7b88 R14: 00000000004dda18 R15: 00000000ffffffff

Showing all locks held in the system:
3 locks held by kworker/u4:0/7:
1 lock held by khungtaskd/1070:
 #0: ffffffff88faadc0 (rcu_read_lock){....}, at: debug_show_all_locks+0x5f/0x27e kernel/locking/lockdep.c:5337
3 locks held by rs:main Q:Reg/8847:
 #0: ffff8880ae934d18 (&rq->lock){-.-.}, at: newidle_balance+0xa32/0xe90 kernel/sched/fair.c:9817
 #1: ffffffff88faadc0 (rcu_read_lock){....}, at: trace_sched_stat_runtime include/trace/events/sched.h:435 [inline]
 #1: ffffffff88faadc0 (rcu_read_lock){....}, at: update_curr+0x2ea/0x8d0 kernel/sched/fair.c:860
 #2: ffff88808943e508 (&sb->s_type->i_mutex_key#11){+.+.}, at: inode_trylock include/linux/fs.h:811 [inline]
 #2: ffff88808943e508 (&sb->s_type->i_mutex_key#11){+.+.}, at: ext4_file_write_iter+0x220/0x13c0 fs/ext4/file.c:234
1 lock held by rsyslogd/8849:
2 locks held by getty/8939:
 #0: ffff8880a31e71d0 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f352e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/8940:
 #0: ffff8880a4193210 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f392e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/8941:
 #0: ffff88808b48e5d0 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f2d2e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/8942:
 #0: ffff88808bf66090 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f312e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/8943:
 #0: ffff88808b6cd290 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f412e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/8944:
 #0: ffff88808bf66910 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f292e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
2 locks held by getty/8945:
 #0: ffff88808b48f6d0 (&tty->ldisc_sem){++++}, at: ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:340
 #1: ffffc90005f112e0 (&ldata->atomic_read_lock){+.+.}, at: n_tty_read+0x232/0x1c10 drivers/tty/n_tty.c:2156
6 locks held by kworker/u4:2/27152:
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: set_work_data kernel/workqueue.c:620 [inline]
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:647 [inline]
 #0: ffff8880a999ce68 ((wq_completion)netns){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2240
 #1: ffff888058cdfdc0 (net_cleanup_work){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2244
 #2: ffffffff89995f48 (pernet_ops_rwsem){++++}, at: cleanup_net+0xae/0xa70 net/core/net_namespace.c:557
 #3: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_lock net/openvswitch/datapath.c:105 [inline]
 #3: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_exit_net+0x206/0xc30 net/openvswitch/datapath.c:2421
 #4: ffffffff899a31a0 (rtnl_mutex){+.+.}, at: rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72
 #5: ffffffff88faeb78 (rcu_state.exp_mutex){+.+.}, at: exp_funnel_lock kernel/rcu/tree_exp.h:285 [inline]
 #5: ffffffff88faeb78 (rcu_state.exp_mutex){+.+.}, at: synchronize_rcu_expedited+0x4d6/0x5f0 kernel/rcu/tree_exp.h:817
2 locks held by kworker/1:4/9188:
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: __write_once_size include/linux/compiler.h:226 [inline]
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: arch_atomic64_set arch/x86/include/asm/atomic64_64.h:34 [inline]
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: atomic64_set include/asm-generic/atomic-instrumented.h:855 [inline]
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: atomic_long_set include/asm-generic/atomic-long.h:40 [inline]
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: set_work_data kernel/workqueue.c:620 [inline]
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: set_work_pool_and_clear_pending kernel/workqueue.c:647 [inline]
 #0: ffff8880aa435ba8 ((wq_completion)rcu_gp){+.+.}, at: process_one_work+0x88b/0x1740 kernel/workqueue.c:2240
 #1: ffff88805c54fdc0 ((work_completion)(&rew.rew_work)){+.+.}, at: process_one_work+0x8c1/0x1740 kernel/workqueue.c:2244
3 locks held by syz-executor.2/1246:
 #0: ffffffff899f4088 (cb_lock){++++}, at: genl_rcv+0x1a/0x40 net/netlink/genetlink.c:664
 #1: ffff88809acae918 (nlk_cb_mutex-GENERIC){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_lock net/openvswitch/datapath.c:105 [inline]
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_dp_cmd_dump+0x1f3/0x540 net/openvswitch/datapath.c:1820
3 locks held by syz-executor.5/1264:
 #0: ffffffff899f4088 (cb_lock){++++}, at: genl_rcv+0x1a/0x40 net/netlink/genetlink.c:664
 #1: ffff888091246618 (nlk_cb_mutex-GENERIC){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_lock net/openvswitch/datapath.c:105 [inline]
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_dp_cmd_dump+0x1f3/0x540 net/openvswitch/datapath.c:1820
2 locks held by syz-executor.1/1273:
 #0: ffffffff899f4088 (cb_lock){++++}, at: genl_rcv+0x1a/0x40 net/netlink/genetlink.c:664
 #1: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_lock net/openvswitch/datapath.c:105 [inline]
 #1: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_dp_cmd_new+0x5d9/0xe30 net/openvswitch/datapath.c:1642
3 locks held by syz-executor.3/1278:
 #0: ffffffff899f4088 (cb_lock){++++}, at: genl_rcv+0x1a/0x40 net/netlink/genetlink.c:664
 #1: ffff88808ecd52d8 (nlk_cb_mutex-GENERIC){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_lock net/openvswitch/datapath.c:105 [inline]
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_dp_cmd_dump+0x1f3/0x540 net/openvswitch/datapath.c:1820
3 locks held by syz-executor.4/1280:
 #0: ffffffff899f4088 (cb_lock){++++}, at: genl_rcv+0x1a/0x40 net/netlink/genetlink.c:664
 #1: ffff888046baea98 (nlk_cb_mutex-GENERIC){+.+.}, at: netlink_dump+0xe7/0xfb0 net/netlink/af_netlink.c:2199
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_lock net/openvswitch/datapath.c:105 [inline]
 #2: ffffffff89bf0e20 (ovs_mutex){+.+.}, at: ovs_dp_cmd_dump+0x1f3/0x540 net/openvswitch/datapath.c:1820

=============================================

NMI backtrace for cpu 0
CPU: 0 PID: 1070 Comm: khungtaskd Not tainted 5.4.0-rc1+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 __dump_stack lib/dump_stack.c:77 [inline]
 dump_stack+0x172/0x1f0 lib/dump_stack.c:113
 nmi_cpu_backtrace.cold+0x70/0xb2 lib/nmi_backtrace.c:101
 nmi_trigger_cpumask_backtrace+0x23b/0x28b lib/nmi_backtrace.c:62
 arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:38
 trigger_all_cpu_backtrace include/linux/nmi.h:146 [inline]
 check_hung_uninterruptible_tasks kernel/hung_task.c:205 [inline]
 watchdog+0x9d0/0xef0 kernel/hung_task.c:289
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 27152 Comm: kworker/u4:2 Not tainted 5.4.0-rc1+ #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
RIP: 0010:io_serial_out+0x73/0x90 drivers/tty/serial/8250/8250_port.c:457
Code: 00 49 8d 7c 24 38 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 ea 03 d3 e3 80 3c 02 00 75 19 41 03 5c 24 38 44 89 e8 89 da ee <5b> 41 5c 41 5d 5d c3 e8 81 34 1e fe eb c0 e8 da 34 1e fe eb e0 0f
RSP: 0018:ffff888058cdf2f0 EFLAGS: 00000002
RAX: 000000000000005b RBX: 00000000000003f8 RCX: 0000000000000000
RDX: 00000000000003f8 RSI: ffffffff83903866 RDI: ffffffff8b57da78
RBP: ffff888058cdf308 R08: ffff88807b5aa180 R09: ffffed100b19be5c
R10: ffffed100b19be5b R11: 0000000000000003 R12: ffffffff8b57da40
R13: 000000000000005b R14: ffffffff8b57da40 R15: 000000000000005b
FS:  0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffff600400 CR3: 00000000a4834000 CR4: 00000000001406e0
DR0: 0000000020000080 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
Call Trace:
 serial_port_out include/linux/serial_core.h:265 [inline]
 serial8250_console_putchar+0x45/0x60 drivers/tty/serial/8250/8250_port.c:3111
 uart_console_write+0x54/0x110 drivers/tty/serial/serial_core.c:1928
 serial8250_console_write+0x612/0x9a0 drivers/tty/serial/8250/8250_port.c:3174
 univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:587
 call_console_drivers kernel/printk/printk.c:1791 [inline]
 console_unlock+0xb89/0xf10 kernel/printk/printk.c:2473
 vprintk_emit+0x2a0/0x700 kernel/printk/printk.c:1996
 vprintk_default+0x28/0x30 kernel/printk/printk.c:2023
 vprintk_func+0x7e/0x189 kernel/printk/printk_safe.c:386
 printk+0xba/0xed kernel/printk/printk.c:2056
 kobject_uevent_env+0x1f5/0x101d lib/kobject_uevent.c:496
 kobject_uevent+0x20/0x26 lib/kobject_uevent.c:641
 kobject_cleanup lib/kobject.c:680 [inline]
 kobject_release lib/kobject.c:722 [inline]
 kref_put include/linux/kref.h:65 [inline]
 kobject_put.cold+0x177/0x2e6 lib/kobject.c:739
 net_rx_queue_update_kobjects+0x359/0x470 net/core/net-sysfs.c:970
 remove_queue_kobjects net/core/net-sysfs.c:1557 [inline]
 netdev_unregister_kobject+0x151/0x1f0 net/core/net-sysfs.c:1708
 rollback_registered_many+0x7c6/0xfc0 net/core/dev.c:8278
 rollback_registered+0x109/0x1d0 net/core/dev.c:8296
 unregister_netdevice_queue net/core/dev.c:9345 [inline]
 unregister_netdevice_queue+0x1ee/0x2c0 net/core/dev.c:9338
 unregister_netdevice include/linux/netdevice.h:2644 [inline]
 internal_dev_destroy+0xc2/0x150 net/openvswitch/vport-internal_dev.c:223
 ovs_vport_del+0x15e/0x1d0 net/openvswitch/vport.c:253
 ovs_dp_detach_port+0x126/0x1b0 net/openvswitch/datapath.c:212
 __dp_destroy+0x351/0x450 net/openvswitch/datapath.c:1710
 ovs_exit_net+0x287/0xc30 net/openvswitch/datapath.c:2423
 ops_exit_list.isra.0+0xaa/0x150 net/core/net_namespace.c:172
 cleanup_net+0x4e7/0xa70 net/core/net_namespace.c:595
 process_one_work+0x9af/0x1740 kernel/workqueue.c:2269
 worker_thread+0x98/0xe40 kernel/workqueue.c:2415
 kthread+0x361/0x430 kernel/kthread.c:255
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352

Crashes (4):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2019/10/16 06:17 bpf 9e8acd9c44a0 d4ea592f .config console log report ci-upstream-bpf-kasan-gce
2019/09/07 06:27 bpf 2339cd6cd0b5 a60cb4cd .config console log report ci-upstream-bpf-kasan-gce
2019/10/13 12:38 bpf-next b8fc345d6b5d 2f661ec4 .config console log report ci-upstream-bpf-next-kasan-gce
2019/09/28 00:05 bpf-next b41dae061bbd d8074e0b .config console log report ci-upstream-bpf-next-kasan-gce
* Struck through repros no longer work on HEAD.