syzbot


BUG: soft lockup in mac80211_hwsim_beacon

Status: auto-obsoleted due to no activity on 2023/01/21 14:41
Subsystems: wireless
[Documentation on labels]
Reported-by: syzbot+d6219cf21f26bdfcc22e@syzkaller.appspotmail.com
First crash: 1266d, last: 692d
Cause bisection: failed (error log, bisect log)
  
Fix bisection: fixed by (bisect log) [no-op commit]:
commit 3c5c67ec29a918dfb2ffc94429437794ddd225e8
Author: Andreas Gruenbacher <agruenba@redhat.com>
Date: Mon Nov 29 20:56:16 2021 +0000

  gfs2: Fix gfs2_instantiate description

  
Duplicate bugs (1)
duplicates (1):
Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
INFO: rcu detected stall in sys_epoll_wait (5) fs input usb syz error 2 512d 1030d 0/26 closed as dup on 2021/05/25 09:57
Discussions (2)
Title Replies (including bot) Last reply
Re: BUG: soft lockup in ieee80211_tasklet_handler 3 (3) 2021/03/04 08:30
BUG: soft lockup in mac80211_hwsim_beacon 0 (2) 2020/12/15 04:42
Similar bugs (4)
Kernel Title Repro Cause bisect Fix bisect Count Last Reported Patched Status
upstream BUG: soft lockup in mac80211_hwsim_beacon (2) wireless 1 8d19h 7d13h 0/26 upstream: reported on 2024/03/11 14:28
upstream INFO: rcu detected stall in mac80211_hwsim_beacon kernel C error inconclusive 1904 692d 1272d 0/26 closed as invalid on 2022/10/12 18:37
linux-5.15 INFO: rcu detected stall in mac80211_hwsim_beacon 1 150d 150d 0/3 auto-obsoleted due to no activity on 2024/01/28 07:17
linux-5.15 INFO: rcu detected stall in mac80211_hwsim_beacon (2) 1 27d 27d 0/3 upstream: reported on 2024/02/20 17:39
Last patch testing requests (5)
Created Duration User Patch Repo Result
2023/01/21 09:32 20m retest repro upstream OK log
2023/01/21 02:32 20m retest repro upstream OK log
2022/10/13 03:30 0m retest repro upstream error OK
2022/10/13 02:30 0m retest repro upstream error OK
2021/11/21 06:48 14m phind.uet@gmail.com linux-next OK
Fix bisection attempts (3)
Created Duration User Patch Repo Result
2022/01/13 09:00 4h53m bisect fix upstream job log (1)
2021/12/14 08:31 28m bisect fix upstream job log (0) log
2021/10/08 13:40 28m bisect fix upstream job log (0) log

Sample crash report:
watchdog: BUG: soft lockup - CPU#0 stuck for 134s! [syz-executor.0:16504]
Modules linked in:
irq event stamp: 17598823
hardirqs last  enabled at (17598822): [<ffffffff89000d42>] asm_sysvec_irq_work+0x12/0x20 arch/x86/include/asm/idtentry.h:654
hardirqs last disabled at (17598823): [<ffffffff88eb155c>] sysvec_apic_timer_interrupt+0xc/0x100 arch/x86/kernel/apic/apic.c:1096
softirqs last  enabled at (15331510): [<ffffffff89000eaf>] asm_call_irq_on_stack+0xf/0x20
softirqs last disabled at (15331513): [<ffffffff89000eaf>] asm_call_irq_on_stack+0xf/0x20
CPU: 0 PID: 16504 Comm: syz-executor.0 Not tainted 5.11.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:deref_stack_reg+0xee/0x150 arch/x86/kernel/unwind_orc.c:355
Code: 4c 89 e2 48 89 ef e8 81 dc f5 ff 41 89 c0 31 c0 45 85 c0 75 27 48 ba 00 00 00 00 00 fc ff df 48 89 d9 48 89 ef e8 12 f9 ff ff <48> c1 e9 03 80 3c 11 00 75 31 48 89 03 b8 01 00 00 00 48 83 c4 08
RSP: 0018:ffffc90000007640 EFLAGS: 00000293
RAX: ffff8880147d1a00 RBX: ffffc90000007788 RCX: ffffc90000007788
RDX: dffffc0000000000 RSI: ffffc9000af57c90 RDI: ffffc9000af57c90
RBP: ffffc9000af57c90 R08: ffffffff8d65eb8e R09: 0000000000000001
R10: 0000000000082083 R11: 0000000000000001 R12: ffffc90000007748
R13: ffffc9000af50000 R14: ffffc90000007798 R15: ffffc90000007748
FS:  0000000000000000(0000) GS:ffff8880b9e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000052e8ac CR3: 000000003c7b2000 CR4: 0000000000350ef0
Call Trace:
 <IRQ>
 unwind_next_frame+0x12e5/0x1f90 arch/x86/kernel/unwind_orc.c:584
 arch_stack_walk+0x7d/0xe0 arch/x86/kernel/stacktrace.c:25
 stack_trace_save+0x8c/0xc0 kernel/stacktrace.c:121
 kasan_save_stack+0x1b/0x40 mm/kasan/common.c:38
 kasan_set_track mm/kasan/common.c:46 [inline]
 set_alloc_info mm/kasan/common.c:401 [inline]
 ____kasan_kmalloc.constprop.0+0x82/0xa0 mm/kasan/common.c:429
 kasan_slab_alloc include/linux/kasan.h:205 [inline]
 slab_post_alloc_hook mm/slab.h:512 [inline]
 slab_alloc_node mm/slub.c:2892 [inline]
 __kmalloc_node_track_caller+0x1e0/0x3e0 mm/slub.c:4496
 __kmalloc_reserve net/core/skbuff.c:142 [inline]
 __alloc_skb+0xae/0x5a0 net/core/skbuff.c:210
 skb_copy+0x137/0x2f0 net/core/skbuff.c:1514
 mac80211_hwsim_tx_frame_no_nl.isra.0+0xb17/0x1330 drivers/net/wireless/mac80211_hwsim.c:1493
 mac80211_hwsim_tx_frame+0x14f/0x1e0 drivers/net/wireless/mac80211_hwsim.c:1705
 mac80211_hwsim_beacon_tx+0x4ba/0x910 drivers/net/wireless/mac80211_hwsim.c:1759
 __iterate_interfaces+0x1e5/0x520 net/mac80211/util.c:793
 ieee80211_iterate_active_interfaces_atomic+0x8d/0x170 net/mac80211/util.c:829
 mac80211_hwsim_beacon+0xd5/0x1a0 drivers/net/wireless/mac80211_hwsim.c:1782
 __run_hrtimer kernel/time/hrtimer.c:1519 [inline]
 __hrtimer_run_queues+0x693/0xea0 kernel/time/hrtimer.c:1583
 hrtimer_run_softirq+0x17b/0x360 kernel/time/hrtimer.c:1600
 __do_softirq+0x2a5/0x9f7 kernel/softirq.c:343
 asm_call_irq_on_stack+0xf/0x20
 </IRQ>
 __run_on_irqstack arch/x86/include/asm/irq_stack.h:26 [inline]
 run_on_irqstack_cond arch/x86/include/asm/irq_stack.h:77 [inline]
 do_softirq_own_stack+0xaa/0xd0 arch/x86/kernel/irq_64.c:77
 invoke_softirq kernel/softirq.c:226 [inline]
 __irq_exit_rcu kernel/softirq.c:420 [inline]
 irq_exit_rcu+0x134/0x200 kernel/softirq.c:432
 sysvec_apic_timer_interrupt+0x4d/0x100 arch/x86/kernel/apic/apic.c:1096
 asm_sysvec_apic_timer_interrupt+0x12/0x20 arch/x86/include/asm/idtentry.h:628
RIP: 0010:__raw_read_unlock include/linux/rwlock_api_smp.h:226 [inline]
RIP: 0010:_raw_read_unlock+0x15/0x40 kernel/locking/spinlock.c:255
Code: e8 00 04 6b f8 58 44 89 e0 5b 41 5c c3 e8 7a bd 12 f8 eb c5 90 55 48 8b 74 24 08 48 89 fd 48 83 c7 18 e8 ce fc 6a f8 48 89 ef <e8> a6 b4 6b f8 bf 01 00 00 00 e8 4c c1 5f f8 65 8b 05 05 68 14 77
RSP: 0018:ffffc9000af57b08 EFLAGS: 00000282
RAX: 0000000000000000 RBX: dffffc0000000000 RCX: 0000000000000000
RDX: 1ffff1100450f4a8 RSI: 0000000000000002 RDI: ffffffff8b00a080
RBP: ffffffff8b00a080 R08: 0000000000000001 R09: ffffffff8ebd57b7
R10: fffffbfff1d7aaf6 R11: 0000000000000000 R12: ffff888025508e00
R13: ffff88801f9f04a0 R14: ffff88801f9f0868 R15: ffff88801f9f0028
 mm_update_next_owner+0x226/0x7a0 kernel/exit.c:417
 exit_mm kernel/exit.c:499 [inline]
 do_exit+0xa8f/0x29e0 kernel/exit.c:810
 do_group_exit+0x125/0x310 kernel/exit.c:920
 get_signal+0x3ec/0x2010 kernel/signal.c:2770
 arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
 handle_signal_work kernel/entry/common.c:147 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
 exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:201
 __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
 syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:302
 entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x45e229
Code: Unable to access opcode bytes at RIP 0x45e1ff.
RSP: 002b:00007f2c5a6a9cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000000 RBX: 000000000119bf88 RCX: 000000000045e229
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000000119bf88
RBP: 000000000119bf80 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000000000119bf8c
R13: 00007fff6bbe298f R14: 00007f2c5a6aa9c0 R15: 000000000119bf8c
Sending NMI from CPU 0 to CPUs 1:
NMI backtrace for cpu 1
CPU: 1 PID: 16513 Comm: syz-executor.0 Not tainted 5.11.0-rc1-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
RIP: 0010:queued_write_lock_slowpath+0x131/0x270 kernel/locking/qrwlock.c:77
Code: 00 00 00 00 fc ff df 49 01 c7 41 83 c6 03 41 0f b6 07 41 38 c6 7c 08 84 c0 0f 85 fe 00 00 00 8b 03 3d 00 01 00 00 74 19 f3 90 <41> 0f b6 07 41 38 c6 7c ec 84 c0 74 e8 48 89 df e8 4a b8 5b 00 eb
RSP: 0018:ffffc9000afa7a60 EFLAGS: 00000006
RAX: 0000000000000300 RBX: ffffffff8b00a080 RCX: ffffffff8159462a
RDX: fffffbfff1601411 RSI: 0000000000000004 RDI: ffffffff8b00a080
RBP: 00000000000000ff R08: 0000000000000001 R09: ffffffff8b00a083
R10: fffffbfff1601410 R11: 0000000000000000 R12: 1ffff920015f4f4d
R13: ffffffff8b00a084 R14: 0000000000000003 R15: fffffbfff1601410
FS:  0000000000000000(0000) GS:ffff8880b9f00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000016b9e60 CR3: 000000003c7b2000 CR4: 0000000000350ee0
Call Trace:
 queued_write_lock include/asm-generic/qrwlock.h:95 [inline]
 do_raw_write_lock+0x1ce/0x280 kernel/locking/spinlock_debug.c:207
 exit_notify kernel/exit.c:666 [inline]
 do_exit+0xbd7/0x29e0 kernel/exit.c:843
 do_group_exit+0x125/0x310 kernel/exit.c:920
 get_signal+0x3ec/0x2010 kernel/signal.c:2770
 arch_do_signal_or_restart+0x2a8/0x1eb0 arch/x86/kernel/signal.c:811
 handle_signal_work kernel/entry/common.c:147 [inline]
 exit_to_user_mode_loop kernel/entry/common.c:171 [inline]
 exit_to_user_mode_prepare+0x148/0x250 kernel/entry/common.c:201
 __syscall_exit_to_user_mode_work kernel/entry/common.c:291 [inline]
 syscall_exit_to_user_mode+0x19/0x50 kernel/entry/common.c:302
 ret_from_fork+0x15/0x30 arch/x86/entry/entry_64.S:289
RIP: 0033:0x460bf9
Code: Unable to access opcode bytes at RIP 0x460bcf.
RSP: 002b:00007f2c5a688db0 EFLAGS: 00000202 ORIG_RAX: 0000000000000038
RAX: 0000000000000000 RBX: 00007f2c5a689700 RCX: 0000000000460bf9
RDX: 00007f2c5a6899d0 RSI: 00007f2c5a688db0 RDI: 00000000003d0f00
RBP: 00007fff6bbe2af0 R08: 00007f2c5a689700 R09: 00007f2c5a689700
R10: 00007f2c5a6899d0 R11: 0000000000000202 R12: 0000000000000000
R13: 00007fff6bbe298f R14: 00007f2c5a6899c0 R15: 000000000119c034

Crashes (180):
Time Kernel Commit Syzkaller Config Log Report Syz repro C repro VM info Assets (help?) Manager Title
2020/12/28 11:55 upstream 5c8fe583cce5 2242f77f .config console log report syz ci-upstream-kasan-gce-root
2020/12/15 04:41 upstream 2c85ebc57b3e 97183ed7 .config console log report syz ci-upstream-kasan-gce-root
2021/11/14 07:48 upstream ccfff0a2bd2a 83f5c9b5 .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/11/11 01:17 upstream 89d714ab6043 75b04091 .config console log report info ci-upstream-kasan-gce-selinux-root BUG: soft lockup in mac80211_hwsim_beacon
2021/11/06 07:35 upstream fe91c4725aee 4c1be0be .config console log report info ci-upstream-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/08/31 13:43 upstream b91db6a0b52e 8f58a0ef .config console log report info ci-upstream-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/08/18 10:26 upstream 614cb2751d31 a2fe1cb5 .config console log report info ci-upstream-kasan-gce-selinux-root BUG: soft lockup in mac80211_hwsim_beacon
2021/08/15 16:10 upstream 0aa78d17099b 2489ab88 .config console log report info ci-upstream-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/08/13 13:10 upstream f8e6dfc64f61 3fd2ea69 .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/06/09 13:11 upstream 368094df48e6 84fe5d96 .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/06/08 21:21 upstream 4c8684fe555e b718257f .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/05/15 17:54 upstream f36edc5533b2 93f844de .config console log report info ci-upstream-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/05/15 04:32 upstream bd3c9cdb21a2 8bdd5343 .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/05/10 08:54 upstream 6efb943b8616 bc5434be .config console log report info ci-upstream-kasan-gce-selinux-root BUG: soft lockup in mac80211_hwsim_beacon
2021/04/14 11:11 upstream 50987beca096 3134b37f .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/03/30 21:47 upstream 2bb25b3a748a 6a81331a .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/03/27 08:16 upstream 0f4498cef9f5 a8529b82 .config console log report info ci-upstream-kasan-gce-smack-root BUG: soft lockup in mac80211_hwsim_beacon
2021/09/05 20:23 bpf 57f780f1c433 d236a457 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/08/13 02:32 bpf 2d3a1e3615c5 3fd2ea69 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/21 08:44 bpf d6371c76e20d 1b201b48 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/15 18:05 bpf 5acc7d3e8d34 b9a2f64e .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/07 03:13 bpf bc832065b60f cca78469 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/03 03:55 bpf 0fc4dcc13f09 55aa55c2 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/24 06:52 bpf c2f5c57d99de fe4ab389 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/23 13:49 bpf 7506d211b932 aba2b2fb .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/10 01:10 bpf 11fc79fc9f2e 84fe5d96 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/01 05:54 bpf ff2e6efda0d5 032639db .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/05/30 07:13 bpf ff2e6efda0d5 325a8dab .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/05/13 04:11 bpf c87db2405fe8 ed7d41c5 .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/04/19 03:01 bpf b02265429681 7e2b734b .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/04/18 05:09 bpf b02265429681 7e2b734b .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/04/02 21:27 bpf 6dcc4e383869 6a81331a .config console log report info ci-upstream-bpf-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/09/08 13:39 bpf-next 006a5099fc18 e2776ee4 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/09/06 20:37 bpf-next 27151f177827 6ca60148 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/09/04 03:39 bpf-next a16ef91aa61a d236a457 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/08/25 10:59 bpf-next 8c0bb89e8e4d b599f2fc .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/08/03 01:32 bpf-next b61a28cf11d6 6c236867 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/20 00:03 bpf-next 08f71a1e39a1 bc48c9ab .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/13 18:51 bpf-next 97eb31384af9 fa0594c3 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/07/03 01:11 bpf-next dbe69e433722 55aa55c2 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/26 00:03 bpf-next a196fa78a265 ae6bf8dd .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/11 21:19 bpf-next 380afe720896 1ba81399 .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2021/06/07 19:21 bpf-next cf68fa431d5d e59537be .config console log report info ci-upstream-bpf-next-kasan-gce BUG: soft lockup in mac80211_hwsim_beacon
2022/04/26 08:55 linux-next e7d6987e09a3 1fa34c1b .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/26 07:23 linux-next e7d6987e09a3 1fa34c1b .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/26 05:46 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/26 04:11 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/26 03:06 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/26 02:05 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/25 23:28 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2022/04/25 19:47 linux-next e7d6987e09a3 152baedd .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/11/07 21:52 linux-next 6a37ebbe07bf 4c1be0be .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/11/07 12:20 linux-next 6a37ebbe07bf 4c1be0be .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/10/09 22:42 linux-next 683f29b781ae 838e7e2c .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/08/10 08:17 linux-next da454ebf578f 6972b106 .config console log report info ci-upstream-linux-next-kasan-gce-root BUG: soft lockup in mac80211_hwsim_beacon
2021/01/16 22:12 upstream 1d94330a437a 65a7a854 .config console log report info ci-upstream-kasan-gce-root
2020/09/29 21:30 bpf-next 00e8c44a147a 5abc3f1a .config console log report info ci-upstream-bpf-next-kasan-gce
* Struck through repros no longer work on HEAD.