bisecting cause commit starting from 770fbb32d34e5d6298cc2be590c9d2fd6069aa17 building syzkaller on fd2a5f28eb5e2b7c83b5e814f53e44e2a5dde24c testing commit 770fbb32d34e5d6298cc2be590c9d2fd6069aa17 with gcc (GCC) 8.1.0 kernel signature: e27250ded0053673d4557d66b60875399be918e5a61be214dc1729cc5e69683e run #0: crashed: general protection fault in __queue_work run #1: crashed: general protection fault in __loop_clr_fd run #2: crashed: general protection fault in __queue_work run #3: crashed: general protection fault in __queue_work run #4: crashed: general protection fault in __queue_work run #5: crashed: general protection fault in __queue_work run #6: crashed: general protection fault in __queue_work run #7: crashed: general protection fault in __queue_work run #8: crashed: general protection fault in __loop_clr_fd run #9: crashed: general protection fault in __queue_work testing release v5.5 testing commit d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 with gcc (GCC) 8.1.0 kernel signature: 86eedfa0e4698c1dec598d4f30d8015d62c6b295f993decb5c4f0466628a9234 all runs: OK # git bisect start 770fbb32d34e5d6298cc2be590c9d2fd6069aa17 d5226fa6dbae0569ee43ecfc08bdcd6770fc4755 Bisecting: 8851 revisions left to test after this (roughly 13 steps) [f8f0d0b6fa203bfa363d30f34f6fecce9e5cc2f7] mm: ptdump: reduce level numbers by 1 in note_page() testing commit f8f0d0b6fa203bfa363d30f34f6fecce9e5cc2f7 with gcc (GCC) 8.1.0 kernel signature: 61c3fa5eec1f0af2e2b19303aafa6200ed013d5a1863a6704899d56ec23bb5a8 all runs: OK # git bisect good f8f0d0b6fa203bfa363d30f34f6fecce9e5cc2f7 Bisecting: 4440 revisions left to test after this (roughly 12 steps) [f3c8aeba659078b44a0a5080c3690ac07f2433bc] Merge remote-tracking branch 'sh/sh-next' testing commit f3c8aeba659078b44a0a5080c3690ac07f2433bc with gcc (GCC) 8.1.0 kernel signature: e34b230a30eacc6e43fa629f33aba36d8bfffdaf9e46a8c845be4afdd2d82042 all runs: OK # git bisect good f3c8aeba659078b44a0a5080c3690ac07f2433bc Bisecting: 2049 revisions left to test after this (roughly 11 steps) [37cc022471948f45fc5364e5fae43034281c9b31] Merge remote-tracking branch 'drm/drm-next' testing commit 37cc022471948f45fc5364e5fae43034281c9b31 with gcc (GCC) 8.1.0 kernel signature: 8e2db56872bc23b623e78844129bdf32467ed84dd3c9659eb71844c4f3796e88 all runs: OK # git bisect good 37cc022471948f45fc5364e5fae43034281c9b31 Bisecting: 1091 revisions left to test after this (roughly 10 steps) [16ec05acd66e5c80698c2f5e50723e580d270b73] Merge remote-tracking branch 'edac/edac-for-next' testing commit 16ec05acd66e5c80698c2f5e50723e580d270b73 with gcc (GCC) 8.1.0 kernel signature: 30cad2bae465f48e857afb1774a0a57b02d7d857ef161b1b8c32a312a7e74201 all runs: OK # git bisect good 16ec05acd66e5c80698c2f5e50723e580d270b73 Bisecting: 543 revisions left to test after this (roughly 9 steps) [3a79df366d6fe0842cede099a5a0018e6dfbba3e] Merge remote-tracking branch 'staging/staging-next' testing commit 3a79df366d6fe0842cede099a5a0018e6dfbba3e with gcc (GCC) 8.1.0 kernel signature: 56863246ad010bf11c3b94f3bf641f8e77f0ad86b367421fd2a0130522f39873 all runs: OK # git bisect good 3a79df366d6fe0842cede099a5a0018e6dfbba3e Bisecting: 263 revisions left to test after this (roughly 8 steps) [e5a270da0559739b8843c023c9a00d4dc734498c] Merge remote-tracking branch 'livepatching/for-next' testing commit e5a270da0559739b8843c023c9a00d4dc734498c with gcc (GCC) 8.1.0 kernel signature: b1a29043b437d7adc154c371d7a70fc61c2a78523ef52309c8dc09d29197d051 all runs: OK # git bisect good e5a270da0559739b8843c023c9a00d4dc734498c Bisecting: 131 revisions left to test after this (roughly 7 steps) [69f9c7f07535c068ba118bf6b07dee8d0ecffa6d] lib/test_lockup: test module to generate lockups testing commit 69f9c7f07535c068ba118bf6b07dee8d0ecffa6d with gcc (GCC) 8.1.0 kernel signature: ce907d0c2eb123231d4b632ff67008749c85de0076ef3ef301b94384fc431191 all runs: OK # git bisect good 69f9c7f07535c068ba118bf6b07dee8d0ecffa6d Bisecting: 65 revisions left to test after this (roughly 6 steps) [249f562d176bc651048bf7f07e490e031e30f469] Merge remote-tracking branch 'devfreq/devfreq-next' testing commit 249f562d176bc651048bf7f07e490e031e30f469 with gcc (GCC) 8.1.0 kernel signature: c45ed6b7d4e10218f00dafd1713ba2448f084c93b9b7a85a1ba740f7b03b13a8 all runs: OK # git bisect good 249f562d176bc651048bf7f07e490e031e30f469 Bisecting: 32 revisions left to test after this (roughly 5 steps) [c9e1feb96bd90a4b51d440a015ba2f1c0562de59] loop: charge i/o to mem and blk cg testing commit c9e1feb96bd90a4b51d440a015ba2f1c0562de59 with gcc (GCC) 8.1.0 kernel signature: 8a86aa60b2bfe5b87db384206f1eb7f89aaea784cc0a09c7e01ce549ff0d8741 run #0: crashed: general protection fault in __queue_work run #1: crashed: general protection fault in __queue_work run #2: crashed: general protection fault in __queue_work run #3: crashed: general protection fault in __queue_work run #4: crashed: general protection fault in __queue_work run #5: crashed: general protection fault in __queue_work run #6: crashed: general protection fault in __queue_work run #7: crashed: general protection fault in __loop_clr_fd run #8: crashed: general protection fault in __queue_work run #9: crashed: general protection fault in __loop_clr_fd # git bisect bad c9e1feb96bd90a4b51d440a015ba2f1c0562de59 Bisecting: 16 revisions left to test after this (roughly 4 steps) [24dae0e8ef60f08b11046ec77981719a9e70b1b1] checkpatch: check SPDX tags in YAML files testing commit 24dae0e8ef60f08b11046ec77981719a9e70b1b1 with gcc (GCC) 8.1.0 kernel signature: a0a0ed4c8fd6e436293d09cca978a044d3cce7e660595b0278dca934d62cf650 all runs: OK # git bisect good 24dae0e8ef60f08b11046ec77981719a9e70b1b1 Bisecting: 8 revisions left to test after this (roughly 3 steps) [3066eb0bdbb165dc83d95b04f30a04ff639c020d] fs/binfmt_elf.c: delete "loc" variable testing commit 3066eb0bdbb165dc83d95b04f30a04ff639c020d with gcc (GCC) 8.1.0 kernel signature: bec6b66ffa0e414866e6beae89b14d88de56385167903d3f437c86b1dd257ebc all runs: OK # git bisect good 3066eb0bdbb165dc83d95b04f30a04ff639c020d Bisecting: 4 revisions left to test after this (roughly 2 steps) [b3ec4e6372e378c428e508a08b69a44af54c3c8f] samples/hw_breakpoint: drop use of kallsyms_lookup_name() testing commit b3ec4e6372e378c428e508a08b69a44af54c3c8f with gcc (GCC) 8.1.0 kernel signature: c39c7927ae9dc922e4609a62af36cf849e01d253a328c2456bb6d6b3dc584dde all runs: OK # git bisect good b3ec4e6372e378c428e508a08b69a44af54c3c8f Bisecting: 2 revisions left to test after this (roughly 1 step) [38422114414048cbcf86acc28d0ae2c9e445cdf9] init/main.c: mark boot_config_checksum static testing commit 38422114414048cbcf86acc28d0ae2c9e445cdf9 with gcc (GCC) 8.1.0 kernel signature: ad13c30c0c6cf8da38a3cbe844fc7cef9c68dfdd5735608e5433cb2d9b9cb529 all runs: OK # git bisect good 38422114414048cbcf86acc28d0ae2c9e445cdf9 Bisecting: 0 revisions left to test after this (roughly 1 step) [e64cc074981343d74017f83fefabdfb1ea65d88c] mm: charge active memcg when no mm is set testing commit e64cc074981343d74017f83fefabdfb1ea65d88c with gcc (GCC) 8.1.0 kernel signature: fca8ede0283305f044498e89a9d6704d89c3e36b739b3a39e32b10c311c8b14c run #0: crashed: general protection fault in __queue_work run #1: crashed: general protection fault in __queue_work run #2: crashed: general protection fault in __queue_work run #3: crashed: general protection fault in __queue_work run #4: crashed: general protection fault in __queue_work run #5: crashed: general protection fault in __queue_work run #6: crashed: general protection fault in __queue_work run #7: crashed: general protection fault in __queue_work run #8: crashed: general protection fault in __loop_clr_fd run #9: crashed: general protection fault in __loop_clr_fd # git bisect bad e64cc074981343d74017f83fefabdfb1ea65d88c Bisecting: 0 revisions left to test after this (roughly 0 steps) [29dab2122492f6dbc0b895ca5bd047e166684d1a] loop: use worker per cgroup instead of kworker testing commit 29dab2122492f6dbc0b895ca5bd047e166684d1a with gcc (GCC) 8.1.0 kernel signature: 54304e586b1f0ee1eb46c4f509d2e9c651dea5e906f9292c320ef416293f6716 run #0: crashed: general protection fault in __queue_work run #1: crashed: general protection fault in __queue_work run #2: crashed: general protection fault in __queue_work run #3: crashed: general protection fault in __queue_work run #4: crashed: general protection fault in __queue_work run #5: crashed: general protection fault in __loop_clr_fd run #6: crashed: general protection fault in __queue_work run #7: crashed: general protection fault in __loop_clr_fd run #8: crashed: general protection fault in __queue_work run #9: crashed: general protection fault in __loop_clr_fd # git bisect bad 29dab2122492f6dbc0b895ca5bd047e166684d1a 29dab2122492f6dbc0b895ca5bd047e166684d1a is the first bad commit commit 29dab2122492f6dbc0b895ca5bd047e166684d1a Author: Dan Schatzberg Date: Tue Feb 25 15:14:07 2020 +1100 loop: use worker per cgroup instead of kworker Patch series "Charge loop device i/o to issuing cgroup", v3. The loop device runs all i/o to the backing file on a separate kworker thread which results in all i/o being charged to the root cgroup. This allows a loop device to be used to trivially bypass resource limits and other policy. This patch series fixes this gap in accounting. A simple script to demonstrate this behavior on cgroupv2 machine: ''' #!/bin/bash set -e CGROUP=/sys/fs/cgroup/test.slice LOOP_DEV=/dev/loop0 if [[ ! -d $CGROUP ]] then sudo mkdir $CGROUP fi grep oom_kill $CGROUP/memory.events # Set a memory limit, write more than that limit to tmpfs -> OOM kill sudo unshare -m bash -c " echo \$\$ > $CGROUP/cgroup.procs; echo 0 > $CGROUP/memory.swap.max; echo 64M > $CGROUP/memory.max; mount -t tmpfs -o size=512m tmpfs /tmp; dd if=/dev/zero of=/tmp/file bs=1M count=256" || true grep oom_kill $CGROUP/memory.events # Set a memory limit, write more than that limit through loopback # device -> no OOM kill sudo unshare -m bash -c " echo \$\$ > $CGROUP/cgroup.procs; echo 0 > $CGROUP/memory.swap.max; echo 64M > $CGROUP/memory.max; mount -t tmpfs -o size=512m tmpfs /tmp; truncate -s 512m /tmp/backing_file losetup $LOOP_DEV /tmp/backing_file dd if=/dev/zero of=$LOOP_DEV bs=1M count=256; losetup -D $LOOP_DEV" || true grep oom_kill $CGROUP/memory.events ''' Naively charging cgroups could result in priority inversions through the single kworker thread in the case where multiple cgroups are reading/writing to the same loop device. This patch series does some minor modification to the loop driver so that each cgroup can make forward progress independently to avoid this inversion. With this patch series applied, the above script triggers OOM kills when writing through the loop device as expected. This patch (of 3): Existing uses of loop device may have multiple cgroups reading/writing to the same device. Simply charging resources for I/O to the backing file could result in priority inversion where one cgroup gets synchronously blocked, holding up all other I/O to the loop device. In order to avoid this priority inversion, we use a single workqueue where each work item is a "struct loop_worker" which contains a queue of struct loop_cmds to issue. The loop device maintains a tree mapping blk css_id -> loop_worker. This allows each cgroup to independently make forward progress issuing I/O to the backing file. There is also a single queue for I/O associated with the rootcg which can be used in cases of extreme memory shortage where we cannot allocate a loop_worker. The locking for the tree and queues is fairly heavy handed - we acquire the per-loop-device spinlock any time either is accessed. The existing implementation serializes all I/O through a single thread anyways, so I don't believe this is any worse. Link: http://lkml.kernel.org/r/eab018412a0c9feb573d757b1bbd5f58b33e2a8d.1582581887.git.schatzberg.dan@gmail.com Signed-off-by: Dan Schatzberg Acked-by: Johannes Weiner Cc: Jens Axboe Cc: Tejun Heo Cc: Li Zefan Cc: Michal Hocko Cc: Vladimir Davydov Cc: Hugh Dickins Cc: Roman Gushchin Cc: Shakeel Butt Cc: Chris Down Cc: Yang Shi Cc: Thomas Gleixner , Signed-off-by: Andrew Morton Signed-off-by: Stephen Rothwell drivers/block/loop.c | 207 ++++++++++++++++++++++++++++++++++++++++++++------- drivers/block/loop.h | 11 ++- 2 files changed, 188 insertions(+), 30 deletions(-) culprit signature: 54304e586b1f0ee1eb46c4f509d2e9c651dea5e906f9292c320ef416293f6716 parent signature: ad13c30c0c6cf8da38a3cbe844fc7cef9c68dfdd5735608e5433cb2d9b9cb529 revisions tested: 17, total time: 4h39m38.304447071s (build: 1h56m4.410137936s, test: 2h42m16.001539484s) first bad commit: 29dab2122492f6dbc0b895ca5bd047e166684d1a loop: use worker per cgroup instead of kworker cc: ["akpm@linux-foundation.org" "hannes@cmpxchg.org" "schatzberg.dan@gmail.com" "sfr@canb.auug.org.au"] crash: general protection fault in __loop_clr_fd general protection fault, probably for non-canonical address 0xdffffc0000000021: 0000 [#1] PREEMPT SMP KASAN KASAN: null-ptr-deref in range [0x0000000000000108-0x000000000000010f] CPU: 1 PID: 8983 Comm: syz-executor.3 Not tainted 5.6.0-rc3-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:workqueue_sysfs_unregister kernel/workqueue.c:5689 [inline] RIP: 0010:destroy_workqueue+0x25/0x5f0 kernel/workqueue.c:4349 Code: 80 00 00 00 00 48 b8 00 00 00 00 00 fc ff df 41 57 41 56 41 55 41 54 55 48 89 fd 48 81 c7 08 01 00 00 48 89 fa 53 48 c1 ea 03 <80> 3c 02 00 0f 85 e8 04 00 00 48 8b 85 08 01 00 00 48 85 c0 74 14 RSP: 0018:ffffc90002af7d38 EFLAGS: 00010206 RAX: dffffc0000000000 RBX: ffff888218aab8f0 RCX: 0000000000000000 RDX: 0000000000000021 RSI: 0000000000000008 RDI: 0000000000000108 RBP: 0000000000000000 R08: fffffbfff165d57e R09: fffffbfff165d57e R10: fffffbfff165d57d R11: ffffffff8b2eabef R12: ffffffff87cc1060 R13: 0000000000000000 R14: ffff888091b487c0 R15: 00000000080e001f FS: 0000000002bff940(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000625208 CR3: 000000009b966000 CR4: 00000000001406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: loop_unprepare_queue drivers/block/loop.c:895 [inline] __loop_clr_fd+0x15a/0x1070 drivers/block/loop.c:1211 __blkdev_driver_ioctl block/ioctl.c:321 [inline] blkdev_ioctl+0x4ab/0x580 block/ioctl.c:717 block_ioctl+0xcc/0x120 fs/block_dev.c:1983 vfs_ioctl fs/ioctl.c:47 [inline] ksys_ioctl+0xb8/0x110 fs/ioctl.c:763 __do_sys_ioctl fs/ioctl.c:772 [inline] __se_sys_ioctl fs/ioctl.c:770 [inline] __x64_sys_ioctl+0x6a/0xb0 fs/ioctl.c:770 do_syscall_64+0xc6/0x5e0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x45c317 Code: 48 83 c4 08 48 89 d8 5b 5d c3 66 0f 1f 84 00 00 00 00 00 48 89 e8 48 f7 d8 48 39 c3 0f 92 c0 eb 92 66 90 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 0f 83 0d b8 fb ff c3 66 2e 0f 1f 84 00 00 00 00 RSP: 002b:00007fff85fe1d58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 000000000045c317 RDX: 0000000000000000 RSI: 0000000000004c01 RDI: 0000000000000003 RBP: 0000000000000002 R08: 0000000000000000 R09: 000000000000000a R10: 0000000000000075 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fff85fe1d90 R14: 0000000000014eab R15: 00007fff85fe1da0 Modules linked in: ---[ end trace 38a2823d53a6182b ]--- RIP: 0010:workqueue_sysfs_unregister kernel/workqueue.c:5689 [inline] RIP: 0010:destroy_workqueue+0x25/0x5f0 kernel/workqueue.c:4349 Code: 80 00 00 00 00 48 b8 00 00 00 00 00 fc ff df 41 57 41 56 41 55 41 54 55 48 89 fd 48 81 c7 08 01 00 00 48 89 fa 53 48 c1 ea 03 <80> 3c 02 00 0f 85 e8 04 00 00 48 8b 85 08 01 00 00 48 85 c0 74 14 RSP: 0018:ffffc90002af7d38 EFLAGS: 00010206 RAX: dffffc0000000000 RBX: ffff888218aab8f0 RCX: 0000000000000000 RDX: 0000000000000021 RSI: 0000000000000008 RDI: 0000000000000108 RBP: 0000000000000000 R08: fffffbfff165d57e R09: fffffbfff165d57e R10: fffffbfff165d57d R11: ffffffff8b2eabef R12: ffffffff87cc1060 R13: 0000000000000000 R14: ffff888091b487c0 R15: 00000000080e001f FS: 0000000002bff940(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000076c000 CR3: 000000009b966000 CR4: 00000000001406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400