Skip to content

fix: concurrent map iteration and map write panic #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 4, 2025
Merged

fix: concurrent map iteration and map write panic #26

merged 2 commits into from
Feb 4, 2025

Conversation

nickpetrovic
Copy link
Contributor

@nickpetrovic nickpetrovic commented Feb 4, 2025 •
edited
Loading

I observed this in staging. I believe this change will fix it because at the current moment, we loop through this map at an interval in a goroutine while other functions are actively modifying this map.

worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 fatal error: concurrent map iteration and map write
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 183 [running]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 github.com/beam-cloud/blobcache-v2/pkg.(*BlobCacheClient).manageLocalClientCache.func1()
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/github.com/beam-cloud/[email protected]/pkg/client.go:334 +0x172
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by github.com/beam-cloud/blobcache-v2/pkg.(*BlobCacheClient).manageLocalClientCache in goroutine 80
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/github.com/beam-cloud/[email protected]/pkg/client.go:325 +0x6b
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 1 [select]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/transport.(*recvBufferReader).readHeaderClient(0xc00030c870, {0xc003019cc0, 0x5, 0x5})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:199 +0xa7
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/transport.(*recvBufferReader).ReadHeader(0xc00030c870, {0xc003019cc0?, 0xc00015e488?, 0xc00003cea0?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:145 +0x45
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/transport.(*transportReader).ReadHeader(0xc0004275c0, {0xc003019cc0?, 0xc00d3f32bd?, 0x2?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:614 +0x25
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/transport.(*Stream).ReadHeader(0xc00003cea0, {0xc003019cc0, 0x5, 0x5})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:557 +0xa6
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.(*parser).recvMsg(0xc003019cb0, 0x400000)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:659 +0x3e
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.recvAndDecompress(0xc003019cb0, 0xc00003cea0, {0x0, 0x0}, 0x400000, 0x0, {0x0, 0x0}, 0x0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:823 +0x9a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.recv(0xc00426fe80?, {0x710a9b7ca018, 0x4cd2ba0}, 0x4c3bd40?, {0x0?, 0x0?}, {0x2cb4820, 0xc00f3b76d0}, 0x2f3ca01?, 0x0, ...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/rpc_util.go:916 +0x91
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.(*csAttempt).recvMsg(0xc003017ad0, {0x2cb4820, 0xc00f3b76d0}, 0xc003579140?)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/stream.go:1129 +0x2ee
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x0?)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/stream.go:969 +0x1f
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.(*clientStream).withRetry(0xc00003cc60, 0xc00a924110, 0xc00a924100)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/stream.go:773 +0x3ae
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc.(*clientStream).RecvMsg(0xc00003cc60, {0x2cb4820?, 0xc00f3b76d0?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/stream.go:968 +0x17f
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 github.com/beam-cloud/beta9/proto.(*workerRepositoryServiceGetNextContainerRequestClient).Recv(0xc000135a50)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/workspace/proto/worker_repo_grpc.pb.go:96 +0x46
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 github.com/beam-cloud/beta9/pkg/worker.(*Worker).Run(0xc003077008)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/workspace/pkg/worker/worker.go:294 +0x23e
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 main.main()
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/workspace/cmd/worker/main.go:35 +0x316
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 166 [chan receive, 115 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 tailscale.com/control/controlclient.(*Auto).authRoutine(0xc000832140)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/[email protected]/control/controlclient/auto.go:319 +0x898
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by tailscale.com/control/controlclient.(*Auto).Start in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/[email protected]/control/controlclient/auto.go:227 +0x56
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 79 [select, 2 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 github.com/beam-cloud/blobcache-v2/pkg.(*DiscoveryClient).StartInBackground(0xc000562f00, {0x3400720, 0xc00077b040})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/github.com/beam-cloud/[email protected]/pkg/discovery.go:70 +0x1ee
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by github.com/beam-cloud/blobcache-v2/pkg.NewBlobCacheClient in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/github.com/beam-cloud/[email protected]/pkg/client.go:111 +0x81f
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 27361 [select]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 github.com/mdlayher/socket.rwT[...].func2()
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/github.com/mdlayher/[email protected]/conn.go:778 +0xbb
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by github.com/mdlayher/socket.rwT[...] in goroutine 131
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/github.com/mdlayher/[email protected]/conn.go:775 +0x579
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 13 [select, 115 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000822380, {0x3400720, 0xc000216190})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 14 [select, 55 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0008223b0, {0x3400720, 0xc0002161e0})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 15 [select, 55 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0008223e0, {0x3400720, 0xc000216230})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 62 [IO wait, 2 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.runtime_pollWait(0x710ae3855bf8, 0x72)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/runtime/netpoll.go:345 +0x85
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.(*pollDesc).wait(0xc0002a6480?, 0xc0006acc00?, 0x0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.(*pollDesc).waitRead(...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.(*FD).Read(0xc0002a6480, {0xc0006acc00, 0xc00, 0xc00})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 net.(*netFD).Read(0xc0002a6480, {0xc0006acc00?, 0x710a9b7db308?, 0xc00fb8a8a0?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/fd_posix.go:55 +0x25
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 net.(*conn).Read(0xc0001364e8, {0xc0006acc00?, 0xc0000cb930?, 0x413c3b?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/net.go:185 +0x45
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*atLeastReader).Read(0xc00fb8a8a0, {0xc0006acc00?, 0x0?, 0xc00fb8a8a0?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:806 +0x3b
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 bytes.(*Buffer).ReadFrom(0xc000205430, {0x33c8960, 0xc00fb8a8a0})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/bytes/buffer.go:211 +0x98
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).readFromUntil(0xc000205188, {0x33c9360, 0xc0001364e8}, 0xc0000cb978?)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:828 +0xde
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).readRecordOrCCS(0xc000205188, 0x0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:626 +0x3cf
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).readRecord(...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:588
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).Read(0xc000205188, {0xc0006f4000, 0x1000, 0x2?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:1370 +0x156
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 net/http.(*persistConn).Read(0xc0006806c0, {0xc0006f4000?, 0x456280?, 0xc0000cbf28?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/http/transport.go:1977 +0x4a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 bufio.(*Reader).fill(0xc000764f60)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/bufio/bufio.go:110 +0x103
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 bufio.(*Reader).Peek(0xc000764f60, 0x1)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/bufio/bufio.go:148 +0x53
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 net/http.(*persistConn).readLoop(0xc0006806c0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/http/transport.go:2141 +0x1b9
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by net/http.(*Transport).dialConn in goroutine 30
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/http/transport.go:1799 +0x152f
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 50 [select, 115 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000822b40, {0x3400720, 0xc000216500})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 51 [select, 21 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000822b70, {0x3400720, 0xc000216550})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 52 [select, 21 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc000822ba0, {0x3400720, 0xc0002165a0})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 91 [select, 2 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 tailscale.com/logtail.(*Logger).drainBlock(...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/[email protected]/logtail/logtail.go:304
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 tailscale.com/logtail.(*Logger).drainPending(0xc0001ae608)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/[email protected]/logtail/logtail.go:359 +0x635
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 tailscale.com/logtail.(*Logger).uploading(0xc0001ae608, {0x3400720, 0xc0002b45f0})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/[email protected]/logtail/logtail.go:399 +0x91
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by tailscale.com/logtail.NewLogger in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/[email protected]/logtail/logtail.go:179 +0x80a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 3006 [chan receive, 103 minutes]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 github.com/beam-cloud/beta9/pkg/common.(*LogBuffer).processWrites(0xc01069a870)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/workspace/pkg/common/logbuf.go:25 +0x46
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by github.com/beam-cloud/beta9/pkg/common.NewLogBuffer in goroutine 1
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/workspace/pkg/common/logbuf.go:20 +0x98
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 goroutine 28 [IO wait]:
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.runtime_pollWait(0x710ae38565a8, 0x72)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/runtime/netpoll.go:345 +0x85
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.(*pollDesc).wait(0xc0002a6380?, 0xc000260000?, 0x0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.(*pollDesc).waitRead(...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 internal/poll.(*FD).Read(0xc0002a6380, {0xc000260000, 0x1000, 0x1000})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/internal/poll/fd_unix.go:164 +0x27a
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 net.(*netFD).Read(0xc0002a6380, {0xc000260000?, 0x710a9a25f948?, 0xc010a3b7b8?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/fd_posix.go:55 +0x25
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 net.(*conn).Read(0xc000418030, {0xc000260000?, 0xc003db1970?, 0x413c3b?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/net/net.go:185 +0x45
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*atLeastReader).Read(0xc010a3b7b8, {0xc000260000?, 0x0?, 0xc010a3b7b8?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:806 +0x3b
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 bytes.(*Buffer).ReadFrom(0xc000005430, {0x33c8960, 0xc010a3b7b8})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/bytes/buffer.go:211 +0x98
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).readFromUntil(0xc000005188, {0x33c9360, 0xc000418030}, 0xc003db19b8?)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:828 +0xde
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).readRecordOrCCS(0xc000005188, 0x0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:626 +0x3cf
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).readRecord(...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:588
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 crypto/tls.(*Conn).Read(0xc000005188, {0xc0005a6000, 0x8000, 0x0?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/crypto/tls/conn.go:1370 +0x156
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 bufio.(*Reader).Read(0xc000400540, {0xc0001b8ba0, 0x9, 0xc000100008?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/bufio/bufio.go:241 +0x197
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 io.ReadAtLeast({0x33cc3e0, 0xc000400540}, {0xc0001b8ba0, 0x9, 0x9}, 0x9)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/io/io.go:335 +0x90
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 io.ReadFull(...)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/usr/local/go/src/io/io.go:354
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 golang.org/x/net/http2.readFrameHeader({0xc0001b8ba0, 0x9, 0xc01026d398?}, {0x33cc3e0?, 0xc000400540?})
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237 +0x65
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 golang.org/x/net/http2.(*Framer).ReadFrame(0xc0001b8b60)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501 +0x85
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00015e488, 0xc0004005a0)
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1645 +0x1d0
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 22
worker-default-ubuntu-7663613f-qp6bf worker 02-04 10:00:02 	/go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:412 +0x1e79

@nickpetrovic nickpetrovic merged commit 80d61da into master Feb 4, 2025
@luke-lombardi luke-lombardi deleted the np/fix-concurrent-map-iteration-and-write branch February 4, 2025 20:26
@nickpetrovic nickpetrovic mentioned this pull request Feb 4, 2025
Merged
nickpetrovic added a commit to beam-cloud/beta9 that referenced this pull request Feb 4, 2025
@nickpetrovic
fix: worker blobcache panic (#930)
877503d 
Blobcache Changes

* Fix contention issues, hotfix for GC by @luke-lombardi in
beam-cloud/blobcache-v2#24
* readahead tuning by @luke-lombardi in
beam-cloud/blobcache-v2#25
* fix: concurrent map iteration and map write panic by @nickpetrovic in
beam-cloud/blobcache-v2#26

Resolve BE-2332
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@luke-lombardi luke-lombardi luke-lombardi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nickpetrovic @luke-lombardi