error: write on a pipe with no reader

[nathanchance]

After llvm/llvm-project@51b557a, I am seeing that error messages in several of my builds. It appears to be more pronounced on stable trees vs. mainline and -next.

On 5.18, I can reproduce it with:

$ make -skj"$(nproc)" ARCH=x86_64 LLVM=1 mrproper allmodconfig all
...
error: write on a pipe with no reader
error: write on a pipe with no reader
...

[nickdesaulniers]

Does V=1 -j1 make it more obvious how clang was invoked when this error is printed?

[nathanchance]

It appears this is related to genksyms, as it only occurs with CONFIG_MODVERSIONS, which causes clang -E to be piped to genksyms. I am trying the -j1 V=1 trick to see if I can try and reproduce this outside of the build system but it seems like this is a bug in genksyms based on the commit message of the LLVM change?

[nathanchance]

I cannot reproduce this outside of the build system unfortunately :( running the commands that I see directly above the message when running with -j1 V=1 shows no message but it is 100% reproducible with CONFIG_MODVERSIONS enabled with at least x86_64_defconfig.

[nickdesaulniers]

running the commands that I see directly above the message when running with -j1 V=1

Can you share some of those commands?

[nathanchance]

Can you share some of those commands?

Sure; pardon the formatting dump:

$ rg -B 1 "error: write" build.log &| cat
  if /mnt/nvme/tmp/build/llvm-testing/stage1/bin/llvm-objdump -h sound/pci/hda/hda_codec.o | grep -q __ksymtab; then /mnt/nvme/tmp/build/llvm-testing/stage1/bin/clang -E -D__GENKSYMS__ -Wp,-MMD,sound/pci/hda/.hda_codec.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Werror -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access    -DKBUILD_MODFILE='"sound/pci/hda/snd-hda-codec"' -DKBUILD_BASENAME='"hda_codec"' -DKBUILD_MODNAME='"snd_hda_codec"' -D__KBUILD_MODNAME=kmod_snd_hda_codec sound/pci/hda/hda_codec.c | scripts/genksyms/genksyms    -r /dev/null > sound/pci/hda/.tmp_hda_codec.ver; /mnt/nvme/tmp/build/llvm-testing/stage1/bin/ld.lld -m elf_x86_64 -r -o sound/pci/hda/.tmp_hda_codec.o sound/pci/hda/hda_codec.o -T sound/pci/hda/.tmp_hda_codec.ver; mv -f sound/pci/hda/.tmp_hda_codec.o sound/pci/hda/hda_codec.o; rm -f sound/pci/hda/.tmp_hda_codec.ver; fi
error: write on a pipe with no reader
--
  if /mnt/nvme/tmp/build/llvm-testing/stage1/bin/llvm-objdump -h drivers/clk/clk.o | grep -q __ksymtab; then /mnt/nvme/tmp/build/llvm-testing/stage1/bin/clang -E -D__GENKSYMS__ -Wp,-MMD,drivers/clk/.clk.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Werror -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access    -DKBUILD_MODFILE='"drivers/clk/clk"' -DKBUILD_BASENAME='"clk"' -DKBUILD_MODNAME='"clk"' -D__KBUILD_MODNAME=kmod_clk drivers/clk/clk.c | scripts/genksyms/genksyms    -r /dev/null > drivers/clk/.tmp_clk.ver; /mnt/nvme/tmp/build/llvm-testing/stage1/bin/ld.lld -m elf_x86_64 -r -o drivers/clk/.tmp_clk.o drivers/clk/clk.o -T drivers/clk/.tmp_clk.ver; mv -f drivers/clk/.tmp_clk.o drivers/clk/clk.o; rm -f drivers/clk/.tmp_clk.ver; fi
error: write on a pipe with no reader
--
  if /mnt/nvme/tmp/build/llvm-testing/stage1/bin/llvm-objdump -h net/core/skbuff.o | grep -q __ksymtab; then /mnt/nvme/tmp/build/llvm-testing/stage1/bin/clang -E -D__GENKSYMS__ -Wp,-MMD,net/core/.skbuff.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Werror -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access    -DKBUILD_MODFILE='"net/core/skbuff"' -DKBUILD_BASENAME='"skbuff"' -DKBUILD_MODNAME='"skbuff"' -D__KBUILD_MODNAME=kmod_skbuff net/core/skbuff.c | scripts/genksyms/genksyms    -r /dev/null > net/core/.tmp_skbuff.ver; /mnt/nvme/tmp/build/llvm-testing/stage1/bin/ld.lld -m elf_x86_64 -r -o net/core/.tmp_skbuff.o net/core/skbuff.o -T net/core/.tmp_skbuff.ver; mv -f net/core/.tmp_skbuff.o net/core/skbuff.o; rm -f net/core/.tmp_skbuff.ver; fi
error: write on a pipe with no reader
--
  if /mnt/nvme/tmp/build/llvm-testing/stage1/bin/llvm-objdump -h net/core/dev.o | grep -q __ksymtab; then /mnt/nvme/tmp/build/llvm-testing/stage1/bin/clang -E -D__GENKSYMS__ -Wp,-MMD,net/core/.dev.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Werror -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access    -DKBUILD_MODFILE='"net/core/dev"' -DKBUILD_BASENAME='"dev"' -DKBUILD_MODNAME='"dev"' -D__KBUILD_MODNAME=kmod_dev net/core/dev.c | scripts/genksyms/genksyms    -r /dev/null > net/core/.tmp_dev.ver; /mnt/nvme/tmp/build/llvm-testing/stage1/bin/ld.lld -m elf_x86_64 -r -o net/core/.tmp_dev.o net/core/dev.o -T net/core/.tmp_dev.ver; mv -f net/core/.tmp_dev.o net/core/dev.o; rm -f net/core/.tmp_dev.ver; fi
error: write on a pipe with no reader
--
  if /mnt/nvme/tmp/build/llvm-testing/stage1/bin/llvm-objdump -h drivers/net/phy/phy_device.o | grep -q __ksymtab; then /mnt/nvme/tmp/build/llvm-testing/stage1/bin/clang -E -D__GENKSYMS__ -Wp,-MMD,drivers/net/phy/.phy_device.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -include ./include/linux/compiler_types.h -D__KERNEL__ -Qunused-arguments -fmacro-prefix-map=./= -Wall -Wundef -Werror=strict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -fno-PIE -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Wno-format-security -std=gnu11 --target=x86_64-linux-gnu -fintegrated-as -Werror=unknown-warning-option -Werror=ignored-optimization-argument -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mretpoline-external-thunk -fno-delete-null-pointer-checks -Wno-frame-address -Wno-address-of-packed-member -O2 -Wframe-larger-than=2048 -fstack-protector-strong -Werror -Wimplicit-fallthrough -Wno-gnu -Wno-unused-but-set-variable -Wno-unused-const-variable -fomit-frame-pointer -ftrivial-auto-var-init=zero -enable-trivial-auto-var-init-zero-knowing-it-will-be-removed-from-clang -fno-stack-clash-protection -Wdeclaration-after-statement -Wvla -Wno-pointer-sign -Wcast-function-type -fno-strict-overflow -fno-stack-check -Werror=date-time -Werror=incompatible-pointer-types -Wno-initializer-overrides -Wno-format -Wno-sign-compare -Wno-format-zero-length -Wno-pointer-to-enum-cast -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access    -DKBUILD_MODFILE='"drivers/net/phy/libphy"' -DKBUILD_BASENAME='"phy_device"' -DKBUILD_MODNAME='"libphy"' -D__KBUILD_MODNAME=kmod_libphy drivers/net/phy/phy_device.c | scripts/genksyms/genksyms    -r /dev/null > drivers/net/phy/.tmp_phy_device.ver; /mnt/nvme/tmp/build/llvm-testing/stage1/bin/ld.lld -m elf_x86_64 -r -o drivers/net/phy/.tmp_phy_device.o drivers/net/phy/phy_device.o -T drivers/net/phy/.tmp_phy_device.ver; mv -f drivers/net/phy/.tmp_phy_device.o drivers/net/phy/phy_device.o; rm -f drivers/net/phy/.tmp_phy_device.ver; fi
error: write on a pipe with no reader

I did notice more errors with a higher level of parallelism with make and the other odd thing is I don't see the same number of errors each time... smells like a race condition somewhere.

[nathanchance]

So this appears to be related to how the kernel calls llvm-objdump? I noticed that I get significantly fewer of these messages on mainline vs. stable (not zero though) so I did a reverse bisect and landed on commit 8017ce50641c ("kbuild: refactor cmd_modversions_c"), which makes sense because we are getting rid of calling $(OBJDUMP) with CONFIG_MODVERSIONS. I further confirmed this by supplying LLVM=1 OBJDUMP=objdump on stable and saw all of the messages disappear.

[nickdesaulniers]

https://stackoverflow.com/questions/19120263/why-exit-code-141-with-grep-q

[nickdesaulniers]

Perhaps redirecting stderr of $(OBJDUMP) to /dev/null is the solution when piping the output to grep -q? Or process substitution (see one of the answers in the link above).

[nickdesaulniers]

https://lore.kernel.org/linux-kbuild/[email protected]/