BUG: workqueue lockup - pool[ 285.916979] INFO: task kworker/u4:0:5 blocked for more than 140 seconds. Not tainted 4.9.194+ #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. kworker/u4:0 D 26016 5 2 0x80000000 Workqueue: events_unbound fsnotify_mark_destroy_workfn 0000000000000087 ffff8801da605f00 0000000000000000 ffff8801db621000 ffff8801d5ccdf00 ffff8801db621018 ffff8801da67f730 ffffffff8281af8e ffff8801da6067d8 dffffc0000000000 00ff8801da6067f8 ffff8801db6218f0Call Trace: [<00000000e9689410>] schedule+0x92/0x1c0 kernel/sched/core.c:3546 [<0000000098c5e9b4>] schedule_timeout+0x766/0xe50 kernel/time/timer.c:1771 [<000000008594680c>] do_wait_for_common kernel/sched/completion.c:75 [inline] [<000000008594680c>] __wait_for_common kernel/sched/completion.c:93 [inline] [<000000008594680c>] wait_for_common+0x2c6/0x4d0 kernel/sched/completion.c:101 [<000000004990f1bd>] wait_for_completion+0x18/0x20 kernel/sched/completion.c:122 [<00000000b6de60ad>] __synchronize_srcu+0x26d/0x3d0 kernel/rcu/srcu.c:448 [<00000000830f97d6>] synchronize_srcu+0x1f/0x40 kernel/rcu/srcu.c:492 [<0000000060c2ffe2>] fsnotify_mark_destroy_list+0x110/0x390 fs/notify/mark.c:551 [<00000000399cb367>] fsnotify_mark_destroy_workfn+0xe/0x10 fs/notify/mark.c:561 [<00000000a35fbbf2>] process_one_work+0x88b/0x1600 kernel/workqueue.c:2114 [<000000002bc43147>] worker_thread+0x5df/0x11d0 kernel/workqueue.c:2251 [<00000000a28e4d29>] kthread+0x278/0x310 kernel/kthread.c:211 [<00000000166d87d5>] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:375 Showing all locks held in the system: 2 locks held by kworker/u4:0/5: #0: ("events_unbound"){.+.+.+}, at: [<000000006cd4e232>] process_one_work+0x790/0x1600 kernel/workqueue.c:2107 ((reaper_work).work){+.+...}, at: [<00000000e48dda5b>] process_one_work+0x7ce/0x1600 kernel/workqueue.c:2111 #0: (rcu_read_lock){......}, at: [<000000005786bdbd>] check_hung_uninterruptible_tasks kernel/hung_task.c:169 [inline] (rcu_read_lock){......}, at: [<000000005786bdbd>] watchdog+0x14b/0xaf0 kernel/hung_task.c:263 (tasklist_lock){.+.+..}, at: [<0000000016cf10b5>] debug_show_all_locks+0x7f/0x21f kernel/locking/lockdep.c:4336 #0: (&f->f_pos_lock){+.+.+.}, at: [<000000001a55a40e>] __fdget_pos+0xa8/0xd0 fs/file.c:782 #0: (&tty->ldisc_sem){++++++}, at: [<00000000c53beb50>] ldsem_down_read+0x33/0x40 drivers/tty/tty_ldsem.c:376 (&ldata->atomic_read_lock){+.+...}, at: [<00000000ba09f6e9>] n_tty_read+0x1fe/0x1820 drivers/tty/n_tty.c:2156 #0: (&lo->lo_ctl_mutex/1){+.+.+.}, at: [<00000000472f6e17>] lo_ioctl+0x8e/0x1b10 drivers/block/loop.c:1404 (lock#2){+.+...}, at: [<000000002a14188c>] lru_add_drain_all+0x34/0x3e0 mm/swap.c:696 (cpu_hotplug.dep_map){++++++}, at: [<00000000dd06f9ca>] get_online_cpus+0x32/0x90 kernel/cpu.c:254 #0: (&lo->lo_ctl_mutex/1){+.+.+.}, at: [<00000000472f6e17>] lo_ioctl+0x8e/0x1b10 drivers/block/loop.c:1404 ============================================= NMI backtrace for cpu 1 CPU: 1 PID: 24 Comm: khungtaskd Not tainted 4.9.194+ #0 ffff8801d98d7cc8 ffffffff81b67001 0000000000000001 0000000000000000 0000000000000001 ffffffff81099d01 dffffc0000000000 ffff8801d98d7d00 ffffffff81b7229c 0000000000000001 0000000000000000 0000000000000001Call Trace: [<0000000013f68dce>] __dump_stack lib/dump_stack.c:15 [inline] [<0000000013f68dce>] dump_stack+0xc1/0x120 lib/dump_stack.c:51 [<00000000d14dac69>] nmi_cpu_backtrace.cold+0x47/0x87 lib/nmi_backtrace.c:99 [<00000000cc1497dd>] nmi_trigger_cpumask_backtrace+0x124/0x155 lib/nmi_backtrace.c:60 [<00000000c0d89ccc>] arch_trigger_cpumask_backtrace+0x14/0x20 arch/x86/kernel/apic/hw_nmi.c:37 [<00000000150bbd7f>] trigger_all_cpu_backtrace include/linux/nmi.h:58 [inline] [<00000000150bbd7f>] check_hung_task kernel/hung_task.c:126 [inline] [<00000000150bbd7f>] check_hung_uninterruptible_tasks kernel/hung_task.c:183 [inline] [<00000000150bbd7f>] watchdog+0x670/0xaf0 kernel/hung_task.c:263 [<00000000a28e4d29>] kthread+0x278/0x310 kernel/kthread.c:211 [<00000000166d87d5>] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:375 Sending NMI from CPU 1 to CPUs 0: NMI backtrace for cpu 0 CPU: 0 PID: 351 Comm: kworker/0:2 Not tainted 4.9.194+ #0 Workqueue: events rtc_timer_do_workc task: 000000000f8e677f task.stack: 00000000a0b07971 RIP: 0010:[] c [<0000000044186611>] io_serial_in+0x6b/0x90 drivers/tty/serial/8250/8250_port.c:414 RSP: 0000:ffff8801db6077b0 EFLAGS: 00000002 RAX: dffffc0000000000 RBX: 00000000000003fd RCX: 0000000000000000 RDX: 00000000000003fd RSI: ffffffff81d86c81 RDI: ffffffff84b68e58 RBP: ffff8801db6077c0 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: ffffffff841d75ea R12: ffffffff84b68e20 R13: 0000000000000020 R14: fffffbfff096d20b R15: fffffbfff096d1cd FS: 0000000000000000(0000) GS:ffff8801db600000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f19be9d8000 CR3: 00000001c6245000 CR4: 00000000001606b0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffffffff84b68e20c 000000000000270ec ffff8801db607810c ffffffff81d88f24c ffff8801d5ccdf00c ffffffff84b68e68c ffffffff84b6905ac ffffffff841d75ccc ffffffff84b68e20c 0000000000000037c dffffc0000000000c 0000000000000037c Call Trace: d [<0000000052892d65>] serial_in drivers/tty/serial/8250/8250.h:111 [inline] d [<0000000052892d65>] wait_for_xmitr+0x94/0x1e0 drivers/tty/serial/8250/8250_port.c:1997 [<00000000ff69ff68>] serial8250_console_putchar+0x20/0x60 drivers/tty/serial/8250/8250_port.c:3103 [<00000000a7d40afd>] uart_console_write+0x56/0xe0 drivers/tty/serial/serial_core.c:1885 [<0000000000309b11>] serial8250_console_write+0x2fb/0x860 drivers/tty/serial/8250/8250_port.c:3169 [<000000004dec6766>] univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:594 [<0000000077a11f60>] call_console_drivers.isra.0.constprop.0+0x1ac/0x360 kernel/printk/printk.c:1594 [<000000006d98ac95>] console_cont_flush kernel/printk/printk.c:2321 [inline] [<000000006d98ac95>] console_unlock+0x17b/0xb20 kernel/printk/printk.c:2387 [<000000002e75943e>] vprintk_emit+0x43f/0x6f0 kernel/printk/printk.c:1908 [<00000000905bc2d5>] vprintk+0x28/0x30 kernel/printk/printk.c:1918 [<000000005d3578f2>] vprintk_func kernel/printk/internal.h:36 [inline] [<000000005d3578f2>] printk+0xc2/0xf5 kernel/printk/printk.c:1980 [<0000000048fbf75c>] wq_watchdog_timer_fn.cold+0x11/0x66 kernel/workqueue.c:5432 [<00000000fc537a08>] call_timer_fn+0x167/0x6d0 kernel/time/timer.c:1319 [<000000009c081f4b>] expire_timers+0x25b/0x5c0 kernel/time/timer.c:1359 [<0000000058953c4b>] __run_timers kernel/time/timer.c:1674 [inline] [<0000000058953c4b>] run_timer_softirq+0x404/0x620 kernel/time/timer.c:1689 [<0000000093b4d291>] __do_softirq+0x22d/0x964 kernel/softirq.c:288 [<000000008ea21244>] invoke_softirq kernel/softirq.c:368 [inline] [<000000008ea21244>] irq_exit+0x119/0x160 kernel/softirq.c:409 [<000000009cfc6f58>] exiting_irq arch/x86/include/asm/apic.h:669 [inline] [<000000009cfc6f58>] smp_apic_timer_interrupt+0x7e/0xb0 arch/x86/kernel/apic/apic.c:1000 [<00000000330c020f>] apic_timer_interrupt+0xa5/0xb0 arch/x86/entry/entry_64.S:653 d [<00000000fe217761>] ? arch_local_irq_restore arch/x86/include/asm/paravirt.h:768 [inline] d [<00000000fe217761>] ? __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:162 [inline] d [<00000000fe217761>] ? _raw_spin_unlock_irqrestore+0x5f/0x70 kernel/locking/spinlock.c:191 [<00000000cdd94e32>] spin_unlock_irqrestore include/linux/spinlock.h:362 [inline] [<00000000cdd94e32>] __wake_up+0x3f/0x50 kernel/sched/wait.c:96 [<0000000057056b9c>] rtc_handle_legacy_irq+0x12c/0x190 drivers/rtc/interface.c:523 [<00000000c6c165be>] rtc_uie_update_irq+0x20/0x30 drivers/rtc/interface.c:550 [<00000000dcc9a327>] rtc_timer_do_work+0x1fe/0x600 drivers/rtc/interface.c:881 [<00000000a35fbbf2>] process_one_work+0x88b/0x1600 kernel/workqueue.c:2114 [<000000002bc43147>] worker_thread+0x5df/0x11d0 kernel/workqueue.c:2251 [<00000000a28e4d29>] kthread+0x278/0x310 kernel/kthread.c:211 [<00000000166d87d5>] ret_from_fork+0x5c/0x70 arch/x86/entry/entry_64.S:375 Code: c24 cc9 c00 c00 c00 c49 c8d c7c c24 c38 c48 cb8 c00 c00 c00 c00 c00 cfc cff cdf c48 c89 cfa c48 cc1 cea c03 cd3 ce3 c80 c3c c02 c00 c75 c17 c41 c03 c5c c24 c38 c89 cda cec c<5b> c0f cb6 cc0 c41 c5c c5d cc3 ce8 c48 c7b c77 cff ceb cc2 ce8 ca1 c7b c77 cff ceb c