bisecting cause commit starting from 90568ecf561540fa330511e21fcd823b0c3829c6 building syzkaller on 06150bf1b39b70e521560bc943ac19b281903ebc testing commit 90568ecf561540fa330511e21fcd823b0c3829c6 with gcc (GCC) 8.1.0 kernel signature: b9c1dc902060d7476c7bdceea4c0224be545b0b87a638c5a1de542482c07d81d all runs: crashed: BUG: sleeping function called from invalid context in __kmalloc testing release v5.5 testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0 kernel signature: b97d9756a141759b36030059d883bb30d1a23cb0e07eb43fc78f7d7383d7e260 all runs: OK # git bisect start 90568ecf561540fa330511e21fcd823b0c3829c6 d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 Bisecting: 4927 revisions left to test after this (roughly 12 steps) [975f9ce9a067a82b89d49e63938e01b2773ac9d4] Merge tag 'driver-core-5.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core testing commit 975f9ce9a067a82b89d49e63938e01b2773ac9d4 with gcc (GCC) 8.1.0 kernel signature: 3474c52f8cc389fa67bb7efb8af2612c94a23d3f7718876003694f39111b8748 all runs: OK # git bisect good 975f9ce9a067a82b89d49e63938e01b2773ac9d4 Bisecting: 2246 revisions left to test after this (roughly 11 steps) [9f68e3655aae6d49d6ba05dd263f99f33c2567af] Merge tag 'drm-next-2020-01-30' of git://anongit.freedesktop.org/drm/drm testing commit 9f68e3655aae6d49d6ba05dd263f99f33c2567af with gcc (GCC) 8.1.0 kernel signature: 79ca9ecd0762495594f67aeadfbbe6576d4c62997ec07f71cb3840956c095754 all runs: OK # git bisect good 9f68e3655aae6d49d6ba05dd263f99f33c2567af Bisecting: 1122 revisions left to test after this (roughly 10 steps) [15f8e73355df9ec48902d128a0ef01a6b8bff453] Merge branch 'pcmcia-next' of git://git.kernel.org/pub/scm/linux/kernel/git/brodo/linux testing commit 15f8e73355df9ec48902d128a0ef01a6b8bff453 with gcc (GCC) 8.1.0 kernel signature: a53193d471be9c2a0ff48b18c923f26d3b348c9edbaebce53924835f83cc14eb all runs: OK # git bisect good 15f8e73355df9ec48902d128a0ef01a6b8bff453 Bisecting: 499 revisions left to test after this (roughly 9 steps) [71c3a888cbcaf453aecf8d2f8fb003271d28073f] Merge tag 'powerpc-5.6-1' of git://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux testing commit 71c3a888cbcaf453aecf8d2f8fb003271d28073f with gcc (GCC) 8.1.0 kernel signature: 19076f04ecbaf930e63ede6b5a58c4be771b77f27817691cc43cd8bbbb23cfb3 all runs: OK # git bisect good 71c3a888cbcaf453aecf8d2f8fb003271d28073f Bisecting: 259 revisions left to test after this (roughly 8 steps) [0384066381ed5572cf1f57f8d01eaccd3f6d4785] Merge tag 'libata-5.6-2020-02-05' of git://git.kernel.dk/linux-block testing commit 0384066381ed5572cf1f57f8d01eaccd3f6d4785 with gcc (GCC) 8.1.0 kernel signature: 2aead40a112896abe358dac3706e606c43c975852105ed9b428ccde3821401e7 all runs: OK # git bisect good 0384066381ed5572cf1f57f8d01eaccd3f6d4785 Bisecting: 130 revisions left to test after this (roughly 7 steps) [e310396bb8d7db977a0e10ef7b5040e98b89c34c] Merge tag 'trace-v5.6-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace testing commit e310396bb8d7db977a0e10ef7b5040e98b89c34c with gcc (GCC) 8.1.0 kernel signature: b899558d70a14a36989629896a2cab59c9277b08941746a8a5862f5a99929afd all runs: OK # git bisect good e310396bb8d7db977a0e10ef7b5040e98b89c34c Bisecting: 48 revisions left to test after this (roughly 6 steps) [750ce8ccd8a875ed9410fab01a3f468dab692eb4] Merge tag 'sound-fix-5.6-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound testing commit 750ce8ccd8a875ed9410fab01a3f468dab692eb4 with gcc (GCC) 8.1.0 kernel signature: 609fa5d6a4c06cf6a547f71fffd94b250e6fce99ce58da575571bc45c497fc85 run #0: crashed: BUG: sleeping function called from invalid context in __kmalloc run #1: crashed: BUG: sleeping function called from invalid context in __kmalloc run #2: crashed: BUG: sleeping function called from invalid context in __kmalloc run #3: crashed: BUG: sleeping function called from invalid context in __kmalloc run #4: crashed: BUG: sleeping function called from invalid context in __kmalloc run #5: crashed: BUG: sleeping function called from invalid context in __kmalloc run #6: crashed: BUG: sleeping function called from invalid context in __kmalloc run #7: crashed: BUG: sleeping function called from invalid context in __kmalloc run #8: crashed: BUG: sleeping function called from invalid context in __kmalloc run #9: boot failed: can't ssh into the instance # git bisect bad 750ce8ccd8a875ed9410fab01a3f468dab692eb4 Bisecting: 37 revisions left to test after this (roughly 5 steps) [4c46bef2e96a92df0f40fc91848e56889ef7c15e] Merge tag 'ceph-for-5.6-rc1' of https://github.com/ceph/ceph-client testing commit 4c46bef2e96a92df0f40fc91848e56889ef7c15e with gcc (GCC) 8.1.0 kernel signature: 9f682bf74fa44100b7c3599eadc192252a2e437cb8c0436fd12fda3261b88624 run #0: crashed: BUG: sleeping function called from invalid context in __kmalloc run #1: crashed: BUG: sleeping function called from invalid context in __kmalloc run #2: crashed: BUG: sleeping function called from invalid context in __kmalloc run #3: crashed: BUG: sleeping function called from invalid context in __kmalloc run #4: crashed: BUG: sleeping function called from invalid context in __kmalloc run #5: crashed: BUG: sleeping function called from invalid context in __kmalloc run #6: crashed: BUG: sleeping function called from invalid context in __kmalloc run #7: crashed: BUG: sleeping function called from invalid context in __kmalloc run #8: crashed: BUG: sleeping function called from invalid context in __kmalloc run #9: boot failed: can't ssh into the instance # git bisect bad 4c46bef2e96a92df0f40fc91848e56889ef7c15e Bisecting: 21 revisions left to test after this (roughly 5 steps) [3c802092dab69351b1c2e52a2250f47d5bf60253] ceph: print r_direct_hash in hex in __choose_mds() dout testing commit 3c802092dab69351b1c2e52a2250f47d5bf60253 with gcc (GCC) 8.1.0 kernel signature: 3f77114688c9ee72f7afdf2ddaad7888d23a1d8ade1de108c62caf136efbf76f all runs: crashed: BUG: sleeping function called from invalid context in __kmalloc # git bisect bad 3c802092dab69351b1c2e52a2250f47d5bf60253 Bisecting: 10 revisions left to test after this (roughly 4 steps) [c4853e9776caefbd2f59739ce1a75798a2b4b7a5] ceph: retry the same mds later after the new session is opened testing commit c4853e9776caefbd2f59739ce1a75798a2b4b7a5 with gcc (GCC) 8.1.0 kernel signature: 28c7ce68e92845f6c4010bf6e5d0844f04286e21947ad2b4fccc15e1a07337b6 all runs: OK # git bisect good c4853e9776caefbd2f59739ce1a75798a2b4b7a5 Bisecting: 5 revisions left to test after this (roughly 3 steps) [4fbc0c711b2464ee1551850b85002faae0b775d5] ceph: remove the extra slashes in the server path testing commit 4fbc0c711b2464ee1551850b85002faae0b775d5 with gcc (GCC) 8.1.0 kernel signature: e91eed9ebe1943d8458c65430dde482bd51827a6975d3bad99ced438dca21259 all runs: crashed: BUG: sleeping function called from invalid context in __kmalloc # git bisect bad 4fbc0c711b2464ee1551850b85002faae0b775d5 Bisecting: 2 revisions left to test after this (roughly 1 step) [9f8b72b3a9485d659410989c6daf5467ebe264ea] ceph: only touch the caps which have the subset mask requested testing commit 9f8b72b3a9485d659410989c6daf5467ebe264ea with gcc (GCC) 8.1.0 kernel signature: 5ed210afcdf5db26d8d795a943e615dda50f4818dd5ca7f5b7126df8fc956f98 all runs: OK # git bisect good 9f8b72b3a9485d659410989c6daf5467ebe264ea Bisecting: 0 revisions left to test after this (roughly 1 step) [b38c9eb4757d5bac1eb8634a9516ef918fca2525] ceph: add possible_max_rank and make the code more readable testing commit b38c9eb4757d5bac1eb8634a9516ef918fca2525 with gcc (GCC) 8.1.0 kernel signature: f9b2261798f199d9bfcf9688e6c2e6a174328538eb4ce92eaa5ed4e11f43e106 all runs: OK # git bisect good b38c9eb4757d5bac1eb8634a9516ef918fca2525 4fbc0c711b2464ee1551850b85002faae0b775d5 is the first bad commit commit 4fbc0c711b2464ee1551850b85002faae0b775d5 Author: Xiubo Li Date: Fri Dec 20 09:34:04 2019 -0500 ceph: remove the extra slashes in the server path It's possible to pass the mount helper a server path that has more than one contiguous slash character. For example: $ mount -t ceph 192.168.195.165:40176:/// /mnt/cephfs/ In the MDS server side the extra slashes of the server path will be treated as snap dir, and then we can get the following debug logs: ceph: mount opening path // ceph: open_root_inode opening '//' ceph: fill_trace 0000000059b8a3bc is_dentry 0 is_target 1 ceph: alloc_inode 00000000dc4ca00b ceph: get_inode created new inode 00000000dc4ca00b 1.ffffffffffffffff ino 1 ceph: get_inode on 1=1.ffffffffffffffff got 00000000dc4ca00b And then when creating any new file or directory under the mount point, we can hit the following BUG_ON in ceph_fill_trace(): BUG_ON(ceph_snap(dir) != dvino.snap); Have the client ignore the extra slashes in the server path when mounting. This will also canonicalize the path, so that identical mounts can be consilidated. 1) "//mydir1///mydir//" 2) "/mydir1/mydir" 3) "/mydir1/mydir/" Regardless of the internal treatment of these paths, the kernel still stores the original string including the leading '/' for presentation to userland. URL: https://tracker.ceph.com/issues/42771 Signed-off-by: Xiubo Li Reviewed-by: Jeff Layton Signed-off-by: Ilya Dryomov fs/ceph/super.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 102 insertions(+), 20 deletions(-) culprit signature: e91eed9ebe1943d8458c65430dde482bd51827a6975d3bad99ced438dca21259 parent signature: f9b2261798f199d9bfcf9688e6c2e6a174328538eb4ce92eaa5ed4e11f43e106 revisions tested: 15, total time: 4h11m39.477078922s (build: 1h41m53.769951278s, test: 2h28m45.692491905s) first bad commit: 4fbc0c711b2464ee1551850b85002faae0b775d5 ceph: remove the extra slashes in the server path cc: ["idryomov@gmail.com" "jlayton@kernel.org" "xiubli@redhat.com"] crash: BUG: sleeping function called from invalid context in __kmalloc BUG: sleeping function called from invalid context at mm/slab.h:565 in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 8641, name: syz-executor.3 1 lock held by syz-executor.3/8641: #0: ffffffff88e558b8 (sb_lock){+.+.}, at: spin_lock include/linux/spinlock.h:338 [inline] #0: ffffffff88e558b8 (sb_lock){+.+.}, at: sget_fc+0xba/0x700 fs/super.c:521 Preemption disabled at: [] spin_lock include/linux/spinlock.h:338 [inline] [] sget_fc+0xba/0x700 fs/super.c:521 CPU: 0 PID: 8641 Comm: syz-executor.3 Not tainted 5.5.0-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x12d/0x187 lib/dump_stack.c:118 ___might_sleep.cold.99+0x1f5/0x238 kernel/sched/core.c:6800 __might_sleep+0x95/0x190 kernel/sched/core.c:6753 slab_pre_alloc_hook mm/slab.h:565 [inline] slab_alloc mm/slab.c:3306 [inline] __do_kmalloc mm/slab.c:3654 [inline] __kmalloc+0x2de/0x790 mm/slab.c:3665 kmalloc include/linux/slab.h:561 [inline] path_remove_extra_slash.part.7+0x99/0x260 fs/ceph/super.c:495 path_remove_extra_slash fs/ceph/super.c:482 [inline] compare_mount_options fs/ceph/super.c:553 [inline] ceph_compare_super+0x1f0/0x510 fs/ceph/super.c:1051 sget_fc+0x1bd/0x700 fs/super.c:524 ceph_get_tree+0x59a/0x1370 fs/ceph/super.c:1126 vfs_get_tree+0x86/0x2d0 fs/super.c:1547 do_new_mount fs/namespace.c:2822 [inline] do_mount+0x1270/0x1b60 fs/namespace.c:3142 __do_sys_mount fs/namespace.c:3351 [inline] __se_sys_mount fs/namespace.c:3328 [inline] __x64_sys_mount+0x169/0x1c0 fs/namespace.c:3328 do_syscall_64+0xca/0x5f0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45b399 Code: ad b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 7b b6 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007fcca437dc78 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 RAX: ffffffffffffffda RBX: 00007fcca437e6d4 RCX: 000000000045b399 RDX: 0000000020000140 RSI: 00000000200000c0 RDI: 0000000020000040 RBP: 000000000075bf20 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffffff R13: 0000000000000745 R14: 00000000004c8c38 R15: 000000000075bf2c ceph: No mds server is up or the cluster is laggy