syzbot


possible deadlock in nfnl_lock

Status: upstream: reported C repro on 2020/01/07 11:25
Reported-by: syzbot+980e9895df7189834175@syzkaller.appspotmail.com
First crash: 1815d, last: 1675d
Fix bisection: failed (error log, bisect log)
  
Last patch testing requests (10)
Created Duration User Patch Repo Result
2022/12/18 16:31 13m retest repro linux-4.14.y report log
2022/12/18 07:31 9m retest repro linux-4.14.y report log
2022/12/17 21:31 10m retest repro linux-4.14.y report log
2022/12/17 12:31 13m retest repro linux-4.14.y report log
2022/12/17 02:31 13m retest repro linux-4.14.y report log
2022/12/16 17:31 10m retest repro linux-4.14.y report log
2022/08/30 01:27 11m retest repro linux-4.14.y report log
2022/08/30 00:27 11m retest repro linux-4.14.y report log
2022/08/29 23:27 11m retest repro linux-4.14.y report log
2022/08/29 22:27 15m retest repro linux-4.14.y report log
Fix bisection attempts (4)
Created Duration User Patch Repo Result
2020/06/24 14:50 0m (2) bisect fix linux-4.14.y error job log
2020/05/25 14:16 23m bisect fix linux-4.14.y OK (0) job log log
2020/04/25 13:50 24m bisect fix linux-4.14.y OK (0) job log log
2020/03/26 13:27 23m bisect fix linux-4.14.y OK (0) job log log

Sample crash report:
audit: type=1400 audit(1582391367.887:38): avc:  denied  { write } for  pid=7282 comm="syz-executor810" scontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tcontext=unconfined_u:system_r:insmod_t:s0-s0:c0.c1023 tclass=netlink_netfilter_socket permissive=1
IPVS: ftp: loaded support on port[0] = 21
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
======================================================
WARNING: possible circular locking dependency detected
4.14.171-syzkaller #0 Not tainted
------------------------------------------------------
kworker/u4:1/22 is trying to acquire lock:
 (&table[i].mutex){+.+.}, at: [<ffffffff853f0884>] nfnl_lock+0x24/0x30 net/netfilter/nfnetlink.c:61

but task is already holding lock:
 (rtnl_mutex){+.+.}, at: [<ffffffff85214c07>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #2 (rtnl_mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72
       unregister_netdevice_notifier+0x5f/0x2c0 net/core/dev.c:1634
       tee_tg_destroy+0x61/0xc0 net/netfilter/xt_TEE.c:123
       cleanup_entry+0x17d/0x230 net/ipv4/netfilter/ip_tables.c:666
       __do_replace+0x3c5/0x5b0 net/ipv4/netfilter/ip_tables.c:1086
       do_replace net/ipv4/netfilter/ip_tables.c:1142 [inline]
       do_ipt_set_ctl+0x296/0x3ee net/ipv4/netfilter/ip_tables.c:1674
       nf_sockopt net/netfilter/nf_sockopt.c:106 [inline]
       nf_setsockopt+0x67/0xc0 net/netfilter/nf_sockopt.c:115
       ip_setsockopt net/ipv4/ip_sockglue.c:1255 [inline]
       ip_setsockopt+0x9b/0xb0 net/ipv4/ip_sockglue.c:1240
       udp_setsockopt+0x4e/0x90 net/ipv4/udp.c:2451
       sock_common_setsockopt+0x94/0xd0 net/core/sock.c:2968
       SYSC_setsockopt net/socket.c:1865 [inline]
       SyS_setsockopt+0x13c/0x210 net/socket.c:1844
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #1 (&xt[i].mutex){+.+.}:
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       xt_find_revision+0x82/0x200 net/netfilter/x_tables.c:373
       nfnl_compat_get+0x229/0x950 net/netfilter/nft_compat.c:678
       nfnetlink_rcv_msg+0xa08/0xc00 net/netfilter/nfnetlink.c:214
       netlink_rcv_skb+0x14f/0x3c0 net/netlink/af_netlink.c:2432
       nfnetlink_rcv+0x1ab/0x1650 net/netfilter/nfnetlink.c:515
       netlink_unicast_kernel net/netlink/af_netlink.c:1286 [inline]
       netlink_unicast+0x44d/0x650 net/netlink/af_netlink.c:1312
       netlink_sendmsg+0x7c4/0xc60 net/netlink/af_netlink.c:1877
       sock_sendmsg_nosec net/socket.c:646 [inline]
       sock_sendmsg+0xce/0x110 net/socket.c:656
       ___sys_sendmsg+0x70a/0x840 net/socket.c:2062
       __sys_sendmsg+0xb9/0x140 net/socket.c:2096
       SYSC_sendmsg net/socket.c:2107 [inline]
       SyS_sendmsg+0x2d/0x50 net/socket.c:2103
       do_syscall_64+0x1e8/0x640 arch/x86/entry/common.c:292
       entry_SYSCALL_64_after_hwframe+0x42/0xb7

-> #0 (&table[i].mutex){+.+.}:
       check_prev_add kernel/locking/lockdep.c:1901 [inline]
       check_prevs_add kernel/locking/lockdep.c:2018 [inline]
       validate_chain kernel/locking/lockdep.c:2460 [inline]
       __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
       lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
       __mutex_lock_common kernel/locking/mutex.c:756 [inline]
       __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
       mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
       nfnl_lock+0x24/0x30 net/netfilter/nfnetlink.c:61
       nf_tables_netdev_event+0x13f/0x580 net/netfilter/nf_tables_netdev.c:122
       notifier_call_chain+0x111/0x1b0 kernel/notifier.c:93
       __raw_notifier_call_chain kernel/notifier.c:394 [inline]
       raw_notifier_call_chain+0x2e/0x40 kernel/notifier.c:401
       call_netdevice_notifiers_info+0x56/0x70 net/core/dev.c:1671
       call_netdevice_notifiers net/core/dev.c:1687 [inline]
       rollback_registered_many+0x70d/0xb60 net/core/dev.c:7205
       unregister_netdevice_many.part.0+0x1b/0x1e0 net/core/dev.c:8278
       unregister_netdevice_many+0x3b/0x50 net/core/dev.c:8277
       ip6gre_exit_net+0x3fe/0x5a0 net/ipv6/ip6_gre.c:1208
       ops_exit_list.isra.0+0xaa/0x150 net/core/net_namespace.c:142
       cleanup_net+0x3ba/0x870 net/core/net_namespace.c:484
       process_one_work+0x863/0x1600 kernel/workqueue.c:2114
       worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
       kthread+0x319/0x430 kernel/kthread.c:232
       ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404

other info that might help us debug this:

Chain exists of:
  &table[i].mutex --> &xt[i].mutex --> rtnl_mutex

 Possible unsafe locking scenario:

       CPU0                    CPU1
       ----                    ----
  lock(rtnl_mutex);
                               lock(&xt[i].mutex);
                               lock(rtnl_mutex);
  lock(&table[i].mutex);

 *** DEADLOCK ***

4 locks held by kworker/u4:1/22:
 #0:  ("%s""netns"){+.+.}, at: [<ffffffff813cd58e>] work_static include/linux/workqueue.h:199 [inline]
 #0:  ("%s""netns"){+.+.}, at: [<ffffffff813cd58e>] set_work_data kernel/workqueue.c:619 [inline]
 #0:  ("%s""netns"){+.+.}, at: [<ffffffff813cd58e>] set_work_pool_and_clear_pending kernel/workqueue.c:646 [inline]
 #0:  ("%s""netns"){+.+.}, at: [<ffffffff813cd58e>] process_one_work+0x76e/0x1600 kernel/workqueue.c:2085
 #1:  (net_cleanup_work){+.+.}, at: [<ffffffff813cd5cb>] process_one_work+0x7ab/0x1600 kernel/workqueue.c:2089
 #2:  (net_mutex){+.+.}, at: [<ffffffff851b828c>] cleanup_net+0x11c/0x870 net/core/net_namespace.c:450
 #3:  (rtnl_mutex){+.+.}, at: [<ffffffff85214c07>] rtnl_lock+0x17/0x20 net/core/rtnetlink.c:72

stack backtrace:
CPU: 1 PID: 22 Comm: kworker/u4:1 Not tainted 4.14.171-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Workqueue: netns cleanup_net
Call Trace:
 __dump_stack lib/dump_stack.c:17 [inline]
 dump_stack+0x142/0x197 lib/dump_stack.c:58
 print_circular_bug.isra.0.cold+0x1cc/0x28f kernel/locking/lockdep.c:1258
 check_prev_add kernel/locking/lockdep.c:1901 [inline]
 check_prevs_add kernel/locking/lockdep.c:2018 [inline]
 validate_chain kernel/locking/lockdep.c:2460 [inline]
 __lock_acquire+0x2cb3/0x4620 kernel/locking/lockdep.c:3487
 lock_acquire+0x16f/0x430 kernel/locking/lockdep.c:3994
 __mutex_lock_common kernel/locking/mutex.c:756 [inline]
 __mutex_lock+0xe8/0x1470 kernel/locking/mutex.c:893
 mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
 nfnl_lock+0x24/0x30 net/netfilter/nfnetlink.c:61
 nf_tables_netdev_event+0x13f/0x580 net/netfilter/nf_tables_netdev.c:122
 notifier_call_chain+0x111/0x1b0 kernel/notifier.c:93
 __raw_notifier_call_chain kernel/notifier.c:394 [inline]
 raw_notifier_call_chain+0x2e/0x40 kernel/notifier.c:401
 call_netdevice_notifiers_info+0x56/0x70 net/core/dev.c:1671
 call_netdevice_notifiers net/core/dev.c:1687 [inline]
 rollback_registered_many+0x70d/0xb60 net/core/dev.c:7205
 unregister_netdevice_many.part.0+0x1b/0x1e0 net/core/dev.c:8278
 unregister_netdevice_many+0x3b/0x50 net/core/dev.c:8277
 ip6gre_exit_net+0x3fe/0x5a0 net/ipv6/ip6_gre.c:1208
 ops_exit_list.isra.0+0xaa/0x150 net/core/net_namespace.c:142
 cleanup_net+0x3ba/0x870 net/core/net_namespace.c:484
 process_one_work+0x863/0x1600 kernel/workqueue.c:2114
 worker_thread+0x5d9/0x1050 kernel/workqueue.c:2248
 kthread+0x319/0x430 kernel/kthread.c:232
 ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:404
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
IPVS: ftp: loaded support on port[0] = 21
ip_tables: iptables: counters copy to user failed while replacing table
net_ratelimit: 5 callbacks suppressed
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
IPVS: ftp: loaded support on port[0] = 21
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
ip_tables: iptables: counters copy to user failed while replacing table
IPVS: ftp: loaded support on port[0] = 21
IPVS: ftp: loaded support on port[0] = 21
net_ratelimit: 8 callbacks suppressed
ip_tables: iptables: counters copy to user failed while replacing table

Crashes (18):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/02/22 17:11 linux-4.14.y 98db2bf27b9e 2c36e7a7 .config console log report syz C ci2-linux-4-14
2020/01/19 17:53 linux-4.14.y c1141b3aab36 0342f8c7 .config console log report syz C ci2-linux-4-14
2020/01/16 13:21 linux-4.14.y c04fc6fa5c96 3de7aabb .config console log report syz C ci2-linux-4-14
2020/01/09 13:25 linux-4.14.y b0cdffaa546e ddc3e859 .config console log report syz C ci2-linux-4-14
2020/01/08 16:42 linux-4.14.y 84f5ad468100 ddc3e859 .config console log report syz C ci2-linux-4-14
2020/01/07 11:24 linux-4.14.y 84f5ad468100 1bcd407e .config console log report syz C ci2-linux-4-14
2020/02/25 11:51 linux-4.14.y 98db2bf27b9e 59b57593 .config console log report ci2-linux-4-14
2020/02/25 02:50 linux-4.14.y 98db2bf27b9e 59b57593 .config console log report ci2-linux-4-14
2020/02/23 07:03 linux-4.14.y 98db2bf27b9e 2c36e7a7 .config console log report ci2-linux-4-14
2020/02/22 16:31 linux-4.14.y 98db2bf27b9e 2c36e7a7 .config console log report ci2-linux-4-14
2020/02/21 14:17 linux-4.14.y 98db2bf27b9e bd2a74a3 .config console log report ci2-linux-4-14
2020/02/17 01:25 linux-4.14.y 98db2bf27b9e 1f448cd6 .config console log report ci2-linux-4-14
2020/02/17 01:20 linux-4.14.y 98db2bf27b9e 1f448cd6 .config console log report ci2-linux-4-14
2020/02/16 21:39 linux-4.14.y 98db2bf27b9e cf914200 .config console log report ci2-linux-4-14
2020/02/16 11:02 linux-4.14.y 98db2bf27b9e cf914200 .config console log report ci2-linux-4-14
2020/01/24 08:09 linux-4.14.y 8bac50406cca 2e95ab33 .config console log report ci2-linux-4-14
2020/01/20 16:12 linux-4.14.y c1141b3aab36 c40da18c .config console log report ci2-linux-4-14
2020/01/20 11:53 linux-4.14.y c1141b3aab36 c40da18c .config console log report ci2-linux-4-14
* Struck through repros no longer work on HEAD.