runtime: deadlock at _cgo_wait_runtime_init_done

[tungnh28]

What version of Go are you using (go version)?

$ go version
go version go1.15.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\nguyenha\AppData\Local\go-build
set GOENV=C:\Users\nguyenha\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\nguyenha\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\nguyenha\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\work\projects\mimic\mimic-core\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\nguyenha\AppData\Local\Temp\go-build126925810=/tmp/go-build -gno-record-gcc-switches

What did you do?

set CGO_ENABLED=1
set CC=clang
go build -buildmode=c-archive -o bin/libcrawler.lib -v core/export

Go side:
package main
import "C"

//export Init
func Init() {
print("something")
}
// Required by CGO
//
func main() {

}

From C++ side:
#include
#include "libcrawler.h"
#include

using namespace std;

int main() {
cout << "Hello, world1!" << endl;

// x_cgo_notify_runtime_init_done(NULL);
Init();
cout << "Hello, world!" << endl;
return 0;
}

I use clang-cl to compile c++ project

What did you expect to see?

Can call Go function (Init()) from C++

What did you see instead?

A deadlock at _cgo_wait_runtime_init_done. I tried to test by calling x_cgo_notify_runtime_init_done(NULL); then I could pass the loop but crash later.

[ianlancetaylor]

Is the code you are showing us the complete test case? How exactly are you building and running the program? Please try to show us exactly how we can reproduce the problem ourselves. Thanks.

[tungnh28]

The build command and export.go and client.cpp are exactly the same as I posted.
From c++, I will create a simple project and send you.

[tungnh28]

Test.zip.
Here you go, visual studio project, compile using clang-cl. If you find anything or any hint that I could try from my side, please let me know :). @ianlancetaylor

[ianlancetaylor]

Thanks, but, sorry, I can't use a Visual Studio project. I'm not a Windows user at all.

Is there a way that you can show me the exact clang commands that you use to build the executable? Thanks.

[tungnh28]

The command in windows:
-cd your_project_root
-clang -Xlinker /libpath:D:\work\projects\test\Test\Test\lib -Xlinker libcrawler.a Test.cpp libcrawler.h

It looks like the problem is windows-only, buildmode=c-archive only. I have tried with buildmode=c-shared and it is ok

[dzonerzy]

Same issue here, works fine with c-shared but with c-archive _cgo_wait_runtime_init_done just hand due to an infinite loop 'cause runtime_init_done is never 1 and thus the event won't be triggered, still trying to understand the root cause.

[james-li]

same issue with c-shared in linux.

I've compiled a so file to use LD_PRELOAD. Deadlock occurs when using the LD_PRELOAD so file

[golib]# LD_PRELOAD=`pwd`/libshellhook.so bash  grepconf.sh -c
<hang here>

When I use gdb to debug this, it's stack show:

(gdb) bt
#0  0x00007fdf8d79a995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007fdf8e313723 in _cgo_wait_runtime_init_done () at gcc_libinit.c:40
#2  0x00007fdf8e313446 in execve (filename=0x1909350 "/usr/libexec/grepconf.sh", argv=0x1908b00, envp=0x1907d10) at _cgo_export.c:26
#3  0x000000000042fc02 in shell_execve ()

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 0x7fdf8e856740 (LWP 25485) "bash" 0x00007fdf8d79a995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

go version

$ go version
go version go1.14.9 linux/amd64

It works fine in fedora 32 and centos 6, but hangs in ubuntu 1804 and centos 7(3.10.0-862.11.6.el7.x86_64.debug )

[james-li]

I've found some clues. I put some debug code in /usr/lib/golang/src/runtime/cgo/gcc_libinit.c to trace the x_cgo_notify_runtime_init_done and _cgo_wait_runtime_init_done.

$ diff -u  ./src/runtime/cgo/gcc_libinit.c /usr/lib/golang/src/runtime/cgo/gcc_libinit.c
--- ./src/runtime/cgo/gcc_libinit.c	2021-01-21 13:31:03.192576533 +0800
+++ /usr/lib/golang/src/runtime/cgo/gcc_libinit.c	2021-01-21 16:08:13.426097949 +0800
@@ -11,6 +11,8 @@
 #include <stdlib.h>
 #include <string.h> // strerror
 #include <time.h>
+#include <unistd.h>
+#include <sys/types.h>
 #include "libcgo.h"
 #include "libcgo_unix.h"
 
@@ -28,7 +30,9 @@
 	if (err != 0) {
 		fprintf(stderr, "pthread_create failed: %s", strerror(err));
 		abort();
-	}
+	}else{
+		fprintf(stderr, "pthread_create success: %d:%x\n", getpid(), (unsigned)p);
+        }
 }
 
 uintptr_t
@@ -37,7 +41,15 @@
 
 	pthread_mutex_lock(&runtime_init_mu);
 	while (runtime_init_done == 0) {
-		pthread_cond_wait(&runtime_init_cond, &runtime_init_mu);
+                struct timespec ts;
+                clock_gettime(CLOCK_REALTIME, &ts);
+                ts.tv_sec += 1;
+		if(pthread_cond_timedwait(&runtime_init_cond, &runtime_init_mu, &ts)){
+                        fprintf(stderr, "wait runtime init done at thread %d:%u, value address %p\n", 
+                                (int)getpid(), (unsigned)pthread_self(), &runtime_init_done);
+                        continue;
+                }
+//                pthread_cond_wait(&runtime_init_cond, &runtime_init_mu);
 	}
 
 	// TODO(iant): For the case of a new C thread calling into Go, such
@@ -49,7 +61,10 @@
 	// initialization to be complete anyhow, later, by waiting for
 	// main_init_done to be closed in cgocallbackg1. We should wait here
 	// instead. See also issue #15943.
-	pfn = cgo_context_function;
+        if(runtime_init_done)
+                pfn = cgo_context_function;
+        else
+                pfn = nil;
 
 	pthread_mutex_unlock(&runtime_init_mu);
 	if (pfn != nil) {
@@ -64,10 +79,14 @@
 
 void
 x_cgo_notify_runtime_init_done(void* dummy __attribute__ ((unused))) {
-	pthread_mutex_lock(&runtime_init_mu);
-	runtime_init_done = 1;
-	pthread_cond_broadcast(&runtime_init_cond);
-	pthread_mutex_unlock(&runtime_init_mu);
+        int tries = 0;
+        while(tries ++ < 3){
+                fprintf(stderr, "notify runtime init done at thread %d:%u, value address %p\n", (int)getpid(), (unsigned)pthread_self(), &runtime_init_done);
+                pthread_mutex_lock(&runtime_init_mu);
+                runtime_init_done = 1;
+                pthread_cond_broadcast(&runtime_init_cond);
+                pthread_mutex_unlock(&runtime_init_mu);
+        }
 }
 
 // Sets the context function to call to record the traceback context

Here is the log

notify runtime init done at thread 32626:1276888832, value address 0x7ff74cea5268
wait runtime init done at thread 32630:1292597056, value address 0x7ff74cea5268

It's clear that the notify and wait action occur in two different processes(parent process and child process). So deadlock is unavoidable.

[ianlancetaylor]

@james-li Using LD_PRELOAD is interesting, but I think it is completely different problem than the rest of this issue, which is about Windows and does not use LD_PRELOAD. I encourage you to take this topic to a different issue, or to a forum. Thanks. (I don't really expect LD_PRELOAD to work, but perhaps there is some way that it could.)

[james-li]

@ianlancetaylor Thanks. I post another issue
#43836