BUG: workqueue lockup - pool cpus=1 node=0 flags=0x0 nice=0 stuck for 222s! Showing busy workqueues and worker pools: workqueue events: flags=0x0 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=10/256 pending: defense_work_handler, defense_work_handler, defense_work_handler, defense_work_handler, psi_update_work, cache_reap, psi_update_work, xfrm_state_gc_task, kvmclock_sync_fn, switchdev_deferred_process_work ====================================================== WARNING: possible circular locking dependency detected 4.20.0+ #169 Not tainted ------------------------------------------------------ syz-executor0/29433 is trying to acquire lock: 00000000474edcd4 (console_owner){-.-.}, at: log_next kernel/printk/printk.c:491 [inline] 00000000474edcd4 (console_owner){-.-.}, at: console_unlock+0x570/0x1180 kernel/printk/printk.c:2401 but task is already holding lock: 000000005c61fd28 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold.50+0xad0/0x159a kernel/workqueue.c:4557 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #4 (&(&pool->lock)->rlock){-.-.}: __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] __queue_work+0x2ff/0x1440 kernel/workqueue.c:1417 queue_work_on+0x19a/0x1e0 kernel/workqueue.c:1486 queue_work include/linux/workqueue.h:512 [inline] schedule_work include/linux/workqueue.h:570 [inline] put_pwq+0x175/0x1c0 kernel/workqueue.c:1090 put_pwq_unlocked.part.28+0x34/0x70 kernel/workqueue.c:1107 put_pwq_unlocked kernel/workqueue.c:1101 [inline] destroy_workqueue+0x868/0x9c0 kernel/workqueue.c:4217 do_floppy_init drivers/block/floppy.c:4708 [inline] floppy_async_init+0x1fed/0x2144 drivers/block/floppy.c:4726 async_run_entry_fn+0x1c4/0x7f0 kernel/async.c:127 process_one_work+0xc90/0x1c40 kernel/workqueue.c:2153 worker_thread+0x17f/0x1390 kernel/workqueue.c:2296 kthread+0x35a/0x440 kernel/kthread.c:246 ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352 -> #3 (&pool->lock/1){..-.}: __raw_spin_lock include/linux/spinlock_api_smp.h:142 [inline] _raw_spin_lock+0x2d/0x40 kernel/locking/spinlock.c:144 spin_lock include/linux/spinlock.h:329 [inline] __queue_work+0x2ff/0x1440 kernel/workqueue.c:1417 queue_work_on+0x19a/0x1e0 kernel/workqueue.c:1486 queue_work include/linux/workqueue.h:512 [inline] tty_schedule_flip+0x14c/0x1d0 drivers/tty/tty_buffer.c:413 tty_flip_buffer_push+0x15/0x20 drivers/tty/tty_buffer.c:556 pty_write+0x19d/0x1f0 drivers/tty/pty.c:125 n_tty_write+0xc5b/0x11a0 drivers/tty/n_tty.c:2354 do_tty_write drivers/tty/tty_io.c:959 [inline] tty_write+0x3f1/0x880 drivers/tty/tty_io.c:1043 __vfs_write+0x119/0x9f0 fs/read_write.c:485 vfs_write+0x1fc/0x560 fs/read_write.c:549 ksys_write+0x101/0x260 fs/read_write.c:598 __do_sys_write fs/read_write.c:610 [inline] __se_sys_write fs/read_write.c:607 [inline] __x64_sys_write+0x73/0xb0 fs/read_write.c:607 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe -> #2 (&(&port->lock)->rlock){-.-.}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x99/0xd0 kernel/locking/spinlock.c:152 tty_port_tty_get+0x20/0x80 drivers/tty/tty_port.c:287 tty_port_default_wakeup+0x15/0x40 drivers/tty/tty_port.c:47 tty_port_tty_wakeup+0x5d/0x70 drivers/tty/tty_port.c:387 uart_write_wakeup+0x44/0x60 drivers/tty/serial/serial_core.c:103 serial8250_tx_chars+0x4be/0xb60 drivers/tty/serial/8250/8250_port.c:1806 serial8250_handle_irq.part.23+0x1ee/0x280 drivers/tty/serial/8250/8250_port.c:1879 serial8250_handle_irq drivers/tty/serial/8250/8250_port.c:1865 [inline] serial8250_default_handle_irq+0xc8/0x150 drivers/tty/serial/8250/8250_port.c:1895 serial8250_interrupt+0xef/0x190 drivers/tty/serial/8250/8250_core.c:125 __handle_irq_event_percpu+0x1e6/0xb00 kernel/irq/handle.c:149 handle_irq_event_percpu+0xa0/0x1d0 kernel/irq/handle.c:189 handle_irq_event+0xa7/0x135 kernel/irq/handle.c:206 handle_edge_irq+0x227/0x880 kernel/irq/chip.c:791 generic_handle_irq_desc include/linux/irqdesc.h:154 [inline] handle_irq+0x252/0x3d8 arch/x86/kernel/irq_64.c:78 do_IRQ+0x98/0x1c0 arch/x86/kernel/irq.c:246 ret_from_intr+0x0/0x1e native_safe_halt+0x6/0x10 arch/x86/include/asm/irqflags.h:57 arch_safe_halt arch/x86/include/asm/paravirt.h:151 [inline] default_idle+0xbf/0x490 arch/x86/kernel/process.c:561 arch_cpu_idle+0x10/0x20 arch/x86/kernel/process.c:552 default_idle_call+0x6d/0x90 kernel/sched/idle.c:93 cpuidle_idle_call kernel/sched/idle.c:153 [inline] do_idle+0x3db/0x5b0 kernel/sched/idle.c:262 cpu_startup_entry+0x18/0x20 kernel/sched/idle.c:353 start_secondary+0x447/0x5f0 arch/x86/kernel/smpboot.c:271 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243 -> #1 (&port_lock_key){-.-.}: __raw_spin_lock_irqsave include/linux/spinlock_api_smp.h:110 [inline] _raw_spin_lock_irqsave+0x99/0xd0 kernel/locking/spinlock.c:152 serial8250_console_write+0x8dd/0xb10 drivers/tty/serial/8250/8250_port.c:3247 univ8250_console_write+0x5f/0x70 drivers/tty/serial/8250/8250_core.c:586 call_console_drivers kernel/printk/printk.c:1728 [inline] console_unlock+0xcd5/0x1180 kernel/printk/printk.c:2414 vprintk_emit+0x39c/0x990 kernel/printk/printk.c:1922 vprintk_default+0x28/0x30 kernel/printk/printk.c:1964 vprintk_func+0x7e/0x181 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:1997 register_console+0x7a5/0xbb0 kernel/printk/printk.c:2729 univ8250_console_init+0x3f/0x4b drivers/tty/serial/8250/8250_core.c:681 console_init+0x6b4/0x9e7 kernel/printk/printk.c:2815 start_kernel+0x5de/0x8fd init/main.c:667 x86_64_start_reservations+0x29/0x2b arch/x86/kernel/head64.c:470 x86_64_start_kernel+0x76/0x79 arch/x86/kernel/head64.c:451 secondary_startup_64+0xa4/0xb0 arch/x86/kernel/head_64.S:243 -> #0 (console_owner){-.-.}: lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 console_lock_spinning_enable kernel/printk/printk.c:1591 [inline] console_unlock+0x5dd/0x1180 kernel/printk/printk.c:2411 vprintk_emit+0x39c/0x990 kernel/printk/printk.c:1922 vprintk_default+0x28/0x30 kernel/printk/printk.c:1964 vprintk_func+0x7e/0x181 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:1997 show_pwq kernel/workqueue.c:4464 [inline] show_workqueue_state.cold.50+0xc6c/0x159a kernel/workqueue.c:4559 wq_watchdog_timer_fn+0x6ea/0x810 kernel/workqueue.c:5571 call_timer_fn+0x272/0x920 kernel/time/timer.c:1325 expire_timers kernel/time/timer.c:1362 [inline] __run_timers+0x7e5/0xc70 kernel/time/timer.c:1681 run_timer_softirq+0x88/0xb0 kernel/time/timer.c:1696 __do_softirq+0x30c/0xb2e kernel/softirq.c:292 invoke_softirq kernel/softirq.c:373 [inline] irq_exit+0x17f/0x1c0 kernel/softirq.c:413 exiting_irq arch/x86/include/asm/apic.h:536 [inline] smp_apic_timer_interrupt+0x1cb/0x760 arch/x86/kernel/apic/apic.c:1061 apic_timer_interrupt+0xf/0x20 arch/x86/entry/entry_64.S:807 rcu_dynticks_curr_cpu_in_eqs+0x4e/0x170 kernel/rcu/tree.c:289 rcu_is_watching+0x10/0x30 kernel/rcu/tree.c:906 rcu_read_unlock include/linux/rcupdate.h:657 [inline] find_get_entry+0xd39/0x1120 mm/filemap.c:1445 pagecache_get_page+0x12f/0xf00 mm/filemap.c:1518 find_or_create_page include/linux/pagemap.h:322 [inline] grow_dev_page fs/buffer.c:947 [inline] grow_buffers fs/buffer.c:1016 [inline] __getblk_slow fs/buffer.c:1043 [inline] __getblk_gfp+0x3aa/0xd50 fs/buffer.c:1320 __bread_gfp+0x2d/0x300 fs/buffer.c:1354 sb_bread include/linux/buffer_head.h:307 [inline] fat__get_entry+0x5a6/0xa40 fs/fat/dir.c:101 fat_get_entry fs/fat/dir.c:129 [inline] fat_search_long+0x33b/0x1570 fs/fat/dir.c:479 vfat_find+0x16d/0x1a0 fs/fat/namei_vfat.c:697 vfat_lookup+0x107/0x640 fs/fat/namei_vfat.c:711 __lookup_hash+0x12e/0x190 fs/namei.c:1546 filename_create+0x1e5/0x5b0 fs/namei.c:3636 user_path_create fs/namei.c:3693 [inline] do_mkdirat+0xda/0x310 fs/namei.c:3831 __do_sys_mkdir fs/namei.c:3855 [inline] __se_sys_mkdir fs/namei.c:3853 [inline] __x64_sys_mkdir+0x5c/0x80 fs/namei.c:3853 do_syscall_64+0x1b9/0x820 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe other info that might help us debug this: Chain exists of: console_owner --> &pool->lock/1 --> &(&pool->lock)->rlock Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&(&pool->lock)->rlock); lock(&pool->lock/1); lock(&(&pool->lock)->rlock); lock(console_owner); *** DEADLOCK *** 8 locks held by syz-executor0/29433: #0: 000000004f5aea56 (sb_writers#14){.+.+}, at: sb_start_write include/linux/fs.h:1597 [inline] #0: 000000004f5aea56 (sb_writers#14){.+.+}, at: mnt_want_write+0x3f/0xc0 fs/namespace.c:360 #1: 000000005eebf7d8 (&sb->s_type->i_mutex_key#20/1){+.+.}, at: inode_lock_nested include/linux/fs.h:792 [inline] #1: 000000005eebf7d8 (&sb->s_type->i_mutex_key#20/1){+.+.}, at: filename_create+0x1b2/0x5b0 fs/namei.c:3635 #2: 00000000eb4e7e9a (&sbi->s_lock){+.+.}, at: vfat_lookup+0xf1/0x640 fs/fat/namei_vfat.c:709 #3: 0000000034b12537 (rcu_read_lock){....}, at: find_get_entry+0x2e3/0x1120 mm/filemap.c:1409 #4: 00000000e88b0e6a ((&wq_watchdog_timer)){+.-.}, at: lockdep_copy_map include/linux/lockdep.h:173 [inline] #4: 00000000e88b0e6a ((&wq_watchdog_timer)){+.-.}, at: call_timer_fn+0x1db/0x920 kernel/time/timer.c:1315 #5: 000000003b0b4daf (rcu_read_lock_sched){....}, at: show_workqueue_state+0x0/0x1d0 kernel/workqueue.c:4423 #6: 000000005c61fd28 (&(&pool->lock)->rlock){-.-.}, at: show_workqueue_state.cold.50+0xad0/0x159a kernel/workqueue.c:4557 #7: 0000000084fa799f (console_lock){+.+.}, at: console_trylock_spinning kernel/printk/printk.c:1653 [inline] #7: 0000000084fa799f (console_lock){+.+.}, at: vprintk_emit+0x37d/0x990 kernel/printk/printk.c:1921 stack backtrace: CPU: 0 PID: 29433 Comm: syz-executor0 Not tainted 4.20.0+ #169 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x1d3/0x2c6 lib/dump_stack.c:113 print_circular_bug.isra.33.cold.54+0x1bd/0x27d kernel/locking/lockdep.c:1221 check_prev_add kernel/locking/lockdep.c:1863 [inline] check_prevs_add kernel/locking/lockdep.c:1976 [inline] validate_chain kernel/locking/lockdep.c:2347 [inline] __lock_acquire+0x3360/0x4c20 kernel/locking/lockdep.c:3341 lock_acquire+0x1ed/0x520 kernel/locking/lockdep.c:3844 console_lock_spinning_enable kernel/printk/printk.c:1591 [inline] console_unlock+0x5dd/0x1180 kernel/printk/printk.c:2411 vprintk_emit+0x39c/0x990 kernel/printk/printk.c:1922 vprintk_default+0x28/0x30 kernel/printk/printk.c:1964 vprintk_func+0x7e/0x181 kernel/printk/printk_safe.c:398 printk+0xa7/0xcf kernel/printk/printk.c:1997 show_pwq kernel/workqueue.c:4464 [inline] show_workqueue_state.cold.50+0xc6c/0x159a kernel/workqueue.c:4559 wq_watchdog_timer_fn+0x6ea/0x810 kernel/workqueue.c:5571 call_timer_fn+0x272/0x920 kernel/time/timer.c:1325 ? _raw_spin_unlock_irq+0x27 Lost 160 message(s)! workqueue events_power_efficient: flags=0x80 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=3/256 pending: do_cache_clean, neigh_periodic_work, check_lifetime workqueue mm_percpu_wq: flags=0x8 pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=1/256 pending: vmstat_update workqueue writeback: flags=0x4e pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/256 in-flight: 11911:wb_workfn workqueue bat_events: flags=0xe000a pwq 4: cpus=0-1 flags=0x4 nice=0 active=1/1 pending: batadv_purge_orig delayed: batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker, batadv_nc_worker pool 4: cpus=0-1 flags=0x4 nice=0 hung=1s workers=17 idle: 27191 27163 27141 27157 21 44 133 27181 27176 27172 12273 27187 11757 11143 27170 27168