march=atom compiletime issues #1483

arndb · 2021-10-19T07:56:17Z

Building x86 with CONFIG_MATOM has multiple problems on clang-14 and earlier versions:

make[4]: *** [/git/arm-soc/scripts/Makefile.build:288: arch/x86/hyperv/hv_init.o] Error 1
error: ran out of registers during register allocation
error: ran out of registers during register allocation

make[6]: *** [/git/arm-soc/scripts/Makefile.build:288: drivers/gpu/drm/vmwgfx/vmwgfx_msg.o] Error 1
error: ran out of registers during register allocation
error: ran out of registers during register allocation
2 errors generated.

arch/x86/crypto/curve25519-x86_64.c:610:3: error: inline assembly requires more registers than available

and hundreds of:
drivers/hwtracing/stm/core.o: warning: objtool: stm_char_write()+0x18c: unreachable instruction
drivers/android/binder.o: warning: objtool: binder_transaction()+0x1aed: unreachable instruction
drivers/hwtracing/intel_th/gth.o: warning: objtool: intel_th_gth_enable()+0x41: unreachable instruction
drivers/hwtracing/intel_th/msu.o: warning: objtool: intel_th_msc_probe()+0x120: unreachable instruction
drivers/mux/core.o: warning: objtool: mux_chip_alloc()+0x1d4: unreachable instruction
drivers/interconnect/core.o: warning: objtool: icc_set_bw()+0x2a: unreachable instruction
drivers/rapidio/devices/tsi721.o: warning: objtool: tsi721_rio_map_inb_mem()+0x47a: unreachable instruction
drivers/dax/bus.o: warning: objtool: size_store()+0x635: unreachable instruction
drivers/input/serio/libps2.o: warning: objtool: ps2_drain()+0x2e: unreachable instruction
drivers/greybus/core.o: warning: objtool: greybus_uevent()+0x86: unreachable instruction
drivers/hid/usbhid/hiddev.o: warning: objtool: hiddev_ioctl_string()+0xae: unreachable instruction
drivers/bcma/driver_chipcommon.o: warning: objtool: bcma_chipco_watchdog_timer_set()+0x4c: unreachable instruction
drivers/input/serio/userio.o: warning: objtool: userio_char_read()+0x256: unreachable instruction
drivers/comedi/comedi_fops.o: warning: objtool: comedi_read()+0x2d8: unreachable instruction
drivers/ntb/hw/epf/ntb_hw_epf.o: warning: objtool: ntb_epf_send_command()+0x9b: unreachable instruction

These were hidden from my randconfig builds as I had disabled the option to work around https://llvm.org/pr42565. However, that bug is now fixed, and enabling MATOM again now runs into the other issues.

arndb · 2021-10-19T08:00:20Z

Attaching four randconfig files and their output:
config-matom.tar.gz

arndb · 2021-10-19T17:13:52Z

Submitted a workaround: https://lore.kernel.org/all/[email protected]/T/#u

nickdesaulniers · 2022-01-14T19:59:10Z

Easily observable with defconfig + CONFIG_MATOM=y.

I thought this might be related to #1566, but the patch I have for that doesn't change anything here AFAICT.

A reduction produced:

// clang -O2 -march=atom foo.c
__register_chrdev() { asm(".byte 15, 0x0b"); }

char_dev.o: warning: objtool: __register_chrdev()+0x8: unreachable instruction

$ llvm-objdump -dr char_dev.o

char_dev.o:     file format elf64-x86-64

Disassembly of section .text:

0000000000000000 <__register_chrdev>:
       0: 0f 0b                         ud2
       2: 90                            nop
       3: 90                            nop
       4: 90                            nop
       5: 90                            nop
       6: 90                            nop
       7: 90                            nop
       8: c3                            retq

Not sure about that though. That must have been over-reduced because asm(".byte 15, 0x0b"); is adding the trapping ud2 which makes the retq at __register_chrdev()+0x8 unreachable.

nickdesaulniers · 2022-01-14T20:09:13Z

I thought this might be related to #1566, but the patch I have for that doesn't change anything here AFAICT.

Hmm...extracting the function in IR, I do see .section .discard.reachable,"",@progbits, so maybe it is related.

nickdesaulniers · 2022-01-14T22:11:49Z

diff --git a/arch/x86/include/asm/bug.h b/arch/x86/include/asm/bug.h
index 84b87538a15d..20201f1aedeb 100644
--- a/arch/x86/include/asm/bug.h
+++ b/arch/x86/include/asm/bug.h
@@ -76,7 +76,6 @@ do {                                                          \
 #define __WARN_FLAGS(flags)                                    \
 do {                                                           \
        instrumentation_begin();                                \
-       _BUG_FLAGS(ASM_UD2, BUGFLAG_WARNING|(flags));           \
        annotate_reachable();                                   \
        instrumentation_end();                                  \
 } while (0)

is enough to eliminate all of these but one. (As does deleting HAVE_ARCH_BUG, BUG, and __WARN_FLAGS from that same file). I suspect that the annotate_reachable()/annotate_unreachable()/instrumentation_begin()/instrumentation_end() stuff is subtly broken. It assumes that instructions can't be reordered across an asm statement, and thus that a label created in inline asm will always refer to all of the instructions from the following C statement, which simply isn't true. If we used asm goto somehow and passed in the address of a label, then the compiler would know that the statement pointed to by the label was an important control flow barrier.

nickdesaulniers · 2022-01-14T23:50:39Z

is enough to eliminate all of these but one.

Which is #1563. Heh, I'm getting a sinking feeling that #1566, #1563, and #1483 are all interrelated.

nickdesaulniers · 2022-01-28T21:56:49Z

FWIW: https://godbolt.org/z/cfsc4vdGK

I have a patch, but testing is tricky since there are so many related objtool warnings.

In __WARN_FLAGS, we had two asm statements (abbreviated): asm("ud2"); asm volatile (".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier and no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards versus inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Link: #1483 Signed-off-by: Nick Desaulniers <[email protected]>

nickdesaulniers · 2022-02-02T20:58:14Z

submitted: https://lore.kernel.org/lkml/[email protected]/

In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux/linux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected]

nickdesaulniers · 2022-02-10T18:16:47Z

accepted: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=objtool/urgent&id=bfb1a7c91fb7758273b4a8d735313d9cc388b502

nickdesaulniers · 2022-02-14T20:09:25Z

commit bfb1a7c ("x86/bug: Merge annotate_reachable() into _BUG_FLAGS() asm")
available in v5.17-rc4.

I did a scrub in the upstream llvm issue tracker for ran out of registers during register allocation. It looks like there are quite a few open issues specific to i386:

"ran out of registers" while building i386 linux kernel llvm/llvm-project#41914
error: ran out of registers during register allocation llvm/llvm-project#11767
Regression(r178606): Clang runs out of registers when building libyuv's row_posix for 32bit with -fPIC llvm/llvm-project#16170

Trying to reproduce the reported error in arch/x86/hyperv/hv_init.o.
I tried ARCH=i386 defconfig plus
HYPERVISOR_GUEST=y
CONFIG_HYPERV=y
CONFIG_MATOM=y

but was unable to reproduce on mainline w/ clang-15.

Trying to reproduce the reported error in drivers/gpu/drm/vmwgfx/vmwgfx_msg.o.

I tried ARCH=i386 defconfig plus
CONFIG_DRM_VMWGFX=y
CONFIG_MATOM=y

but was unable to reproduce on mainline w/ clang-15.

Trying to reproduce the failure in arch/x86/crypto/curve25519-x86_64.c

I tried ARCH=i386 defconfig plus
CONFIG_WIREGUARD=y
CONFIG_CRYPTO_CURVE25519_X86=y
CONFIG_MATOM=y

but CONFIG_CRYPTO_CURVE25519_X86 is not selectable if !64BIT.

Since there are open issues upstream, I'm not convinced this is the last we'll see of register exhaustion related errors on i386 (ran out of registers during register allocation or inline assembly requires more registers than available).

But this bug is a mess of multiple different reports for different things, so closing this out since the objtool warnings are fixed by

commit bfb1a7c ("x86/bug: Merge annotate_reachable() into _BUG_FLAGS() asm")
available in v5.17-rc4.

I assume the rest of the issues are coming from randconfigs. Let's get bugs filed for individual cases when we find them again. It would be helpful to isolate which configs are necessary to reproduce from defconfig.

nickdesaulniers · 2022-02-14T20:12:59Z

I tried ARCH=i386 defconfig plus
but CONFIG_CRYPTO_CURVE25519_X86 is not selectable if !64BIT.

oh, are there 64b atom?

[ Upstream commit bfb1a7c ] In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux/linux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sasha Levin <[email protected]>

[ Upstream commit bfb1a7c ] In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sasha Levin <[email protected]>

[ Upstream commit bfb1a7c ] In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux/linux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sasha Levin <[email protected]>

In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected]

BugLink: https://bugs.launchpad.net/bugs/1963890 [ Upstream commit bfb1a7c ] In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux/linux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Sasha Levin <[email protected]> Signed-off-by: Paolo Pisati <[email protected]>

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2121207 commit bfb1a7c91fb7758273b4a8d735313d9cc388b502 Author: Nick Desaulniers <[email protected]> Date: Wed Feb 2 12:55:53 2022 -0800 x86/bug: Merge annotate_reachable() into _BUG_FLAGS() asm In __WARN_FLAGS(), we had two asm statements (abbreviated): asm volatile("ud2"); asm volatile(".pushsection .discard.reachable"); These pair of statements are used to trigger an exception, but then help objtool understand that for warnings, control flow will be restored immediately afterwards. The problem is that volatile is not a compiler barrier. GCC explicitly documents this: > Note that the compiler can move even volatile asm instructions > relative to other code, including across jump instructions. Also, no clobbers are specified to prevent instructions from subsequent statements from being scheduled by compiler before the second asm statement. This can lead to instructions from subsequent statements being emitted by the compiler before the second asm statement. Providing a scheduling model such as via -march= options enables the compiler to better schedule instructions with known latencies to hide latencies from data hazards compared to inline asm statements in which latencies are not estimated. If an instruction gets scheduled by the compiler between the two asm statements, then objtool will think that it is not reachable, producing a warning. To prevent instructions from being scheduled in between the two asm statements, merge them. Also remove an unnecessary unreachable() asm annotation from BUG() in favor of __builtin_unreachable(). objtool is able to track that the ud2 from BUG() terminates control flow within the function. Link: https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html#Volatile Link: ClangBuiltLinux/linux#1483 Signed-off-by: Nick Desaulniers <[email protected]> Signed-off-by: Josh Poimboeuf <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Joe Lawrence <[email protected]>

nickdesaulniers added [ARCH] x86 This bug impacts ARCH=i386 [BUG] llvm A bug that should be fixed in upstream LLVM labels Nov 17, 2021

nickdesaulniers added the [TOOL] objtool warning is produced by the kernel's objtool label Jan 14, 2022

nickdesaulniers self-assigned this Jan 28, 2022

nickdesaulniers added the [PATCH] Exists There is a patch that fixes this issue label Jan 28, 2022

nickdesaulniers added [PATCH] Submitted A patch has been submitted for review and removed [PATCH] Exists There is a patch that fixes this issue labels Feb 2, 2022

nickdesaulniers mentioned this issue Feb 3, 2022

net/core/skbuff.o: unreachable instruction #1563

Closed

nickdesaulniers added [PATCH] Accepted A submitted patch has been accepted upstream and removed [PATCH] Submitted A patch has been submitted for review labels Feb 10, 2022

nickdesaulniers closed this as completed Feb 14, 2022

nickdesaulniers added [BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 5.17 This bug was fixed in Linux 5.17 and removed [BUG] llvm A bug that should be fixed in upstream LLVM [PATCH] Accepted A submitted patch has been accepted upstream labels Feb 14, 2022

alavalas1 mentioned this issue Jul 30, 2024

CONFIG-MATOM=y (arch/x86/crypto/curve25519-x86_64.c: error: inline assembly requires more registers than available ) #2046

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

march=atom compiletime issues #1483

march=atom compiletime issues #1483

arndb commented Oct 19, 2021 •

edited by nathanchance

Loading

arndb commented Oct 19, 2021

Uh oh!

arndb commented Oct 19, 2021

Uh oh!

nickdesaulniers commented Jan 14, 2022

Uh oh!

nickdesaulniers commented Jan 14, 2022

Uh oh!

nickdesaulniers commented Jan 14, 2022 •

edited

Loading

Uh oh!

nickdesaulniers commented Jan 14, 2022

Uh oh!

nickdesaulniers commented Jan 28, 2022

Uh oh!

nickdesaulniers commented Feb 2, 2022

Uh oh!

nickdesaulniers commented Feb 10, 2022

Uh oh!

nickdesaulniers commented Feb 14, 2022 •

edited

Loading

Uh oh!

nickdesaulniers commented Feb 14, 2022

Uh oh!

march=atom compiletime issues #1483

march=atom compiletime issues #1483

Comments

arndb commented Oct 19, 2021 • edited by nathanchance Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

arndb commented Oct 19, 2021

Uh oh!

arndb commented Oct 19, 2021

Uh oh!

nickdesaulniers commented Jan 14, 2022

Uh oh!

nickdesaulniers commented Jan 14, 2022

Uh oh!

nickdesaulniers commented Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickdesaulniers commented Jan 14, 2022

Uh oh!

nickdesaulniers commented Jan 28, 2022

Uh oh!

nickdesaulniers commented Feb 2, 2022

Uh oh!

nickdesaulniers commented Feb 10, 2022

Uh oh!

nickdesaulniers commented Feb 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nickdesaulniers commented Feb 14, 2022

Uh oh!

arndb commented Oct 19, 2021 •

edited by nathanchance

Loading

nickdesaulniers commented Jan 14, 2022 •

edited

Loading

nickdesaulniers commented Feb 14, 2022 •

edited

Loading