runtime,cmd/compile: frequent memory corruption on NetBSD and OpenBSD since 2021-10-11

[bcmills]

greplogs --dashboard -md -l -e 'freeIndex is not valid' --since=2021-05-01

2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0

[bcmills]

CC @bsiegert @coypoop

[bcmills]

This appears to be a regression in Go 1.18.

Since NetBSD is not a first-class port, this doesn't necessarily block the 1.18 release — however, if the regression remains at the time of the release it at least needs a clear writeup in the release notes. (That part, at least, is a release-blocker.)

[bcmills]

@jeremyfaller, any idea who on the Go team might have insight into what broke on the 18th specifically?

[coypoop]

I'm not sure this error is entirely a regression, as I've seen this with pre-Go-1.18 on NetBSD. But perhaps something is making it much more frequent.

[jeremyfaller]

Did a little digging on Oct17-18, and didn't see much that looked suspicious.

pinging @mknyszek

[bcmills]

There have been a lot of other failures since then that look like arbitrary memory corruption in the runtime.
Whatever this is, it looks like it's got all of the NetBSD builders in pretty bad shape.
(Silver lining: the failure rate is high enough that this should be feasible to bisect!)

greplogs --dashboard -md -l -e '(?ms)\Anetbsd.*fatal error:' --since=2021-10-01

2021-11-02T15:15:09-4ff95d6-da7173a/netbsd-amd64-9_0
2021-11-02T06:25:39-30b2efe-088bb4b/netbsd-amd64-9_0
2021-11-02T06:19:33-4633b2d/netbsd-amd64-9_0
2021-11-02T05:43:05-a97c527/netbsd-amd64-9_0
2021-11-02T05:43:05-835dae6-a97c527/netbsd-386-9_0
2021-11-02T05:35:11-6f327f7/netbsd-386-9_0
2021-11-02T03:09:01-a45457d/netbsd-amd64-9_0
2021-11-02T00:12:17-4ff95d6-81fea0b/netbsd-386-9_0
2021-11-01T22:55:50-02e5913/netbsd-amd64-9_0
2021-11-01T21:27:26-4ff95d6-631b567/netbsd-386-9_0
2021-11-01T21:27:26-2c98350-631b567/netbsd-amd64-9_0
2021-11-01T19:56:50-ee08195-5f1bcd1/netbsd-386-9_0
2021-11-01T18:53:06-9967326-b5a5b7b/netbsd-amd64-9_0
2021-11-01T16:31:02-2bcf1c0/netbsd-386-9_0
2021-11-01T15:55:25-e2e910e/netbsd-386-9_0
2021-11-01T15:20:08-b7d65fb-82f902a/netbsd-amd64-9_0
2021-11-01T14:32:47-30b2efe-4a84298/netbsd-386-9_0
2021-11-01T13:54:03-42daa65-4a84298/netbsd-386-9_0
2021-11-01T13:12:37-611d5d6-4056934/netbsd-amd64-9_0
2021-11-01T13:12:37-4ff95d6-4056934/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-386-9_0
2021-11-01T02:47:30-fde4cc2/netbsd-amd64-9_0
2021-10-31T18:39:05-12ab535-89c5270/netbsd-amd64-9_0
2021-10-31T17:52:41-4ff95d6-3fa9ee4/netbsd-386-9_0
2021-10-30T18:30:34-ba495a6-6113dac/netbsd-386-9_0
2021-10-30T18:30:34-6113dac/netbsd-amd64-9_0
2021-10-30T18:30:34-351c04c-6113dac/netbsd-386-9_0
2021-10-30T16:47:17-30b5d63/netbsd-386-9_0
2021-10-30T16:46:47-d1dceaf/netbsd-386-9_0
2021-10-30T16:45:25-e39b854/netbsd-386-9_0
2021-10-30T16:08:13-b3129d9-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-d19c5bd/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-386-9_0
2021-10-30T00:47:26-a6c6f4b-5d6d9f5/netbsd-amd64-9_0
2021-10-30T00:47:26-a6c6f4b-4a84298/netbsd-386-9_0
2021-10-29T23:15:00-a2be0cd-8dfb447/netbsd-amd64-9_0
2021-10-29T22:29:31-c96bc14-c812b97/netbsd-386-9_0
2021-10-29T22:27:54-71e6ab8/netbsd-amd64-9_0
2021-10-29T22:27:26-3571ab5/netbsd-386-9_0
2021-10-29T21:19:39-a2be0cd-d19c5bd/netbsd-amd64-9_0
2021-10-29T21:19:39-a2be0cd-994049a/netbsd-amd64-9_0
2021-10-29T19:42:51-540bb53-4a84298/netbsd-amd64-9_0
2021-10-29T19:42:43-4ff95d6-d19c5bd/netbsd-386-9_0
2021-10-29T19:30:03-4592933/netbsd-386-9_0
2021-10-29T18:35:01-8e112a7/netbsd-amd64-9_0
2021-10-29T18:23:03-413672f/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-386-9_0
2021-10-29T17:44:15-d8fc7f7/netbsd-amd64-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-386-9_0
2021-10-29T04:21:07-d6a9af8-f288526/netbsd-amd64-9_0
2021-10-29T02:12:05-d6a9af8-d3d8852/netbsd-amd64-9_0
2021-10-28T22:38:00-af05d8b/netbsd-amd64-9_0
2021-10-28T20:43:39-b954024/netbsd-386-9_0
2021-10-28T20:43:39-6944b10-b954024/netbsd-amd64-9_0
2021-10-28T20:27:44-5e21728/netbsd-386-9_0
2021-10-28T19:04:41-4e1c44d-18b9702/netbsd-386-9_0
2021-10-28T18:48:25-e7eb6f6-9004433/netbsd-386-9_0
2021-10-28T18:17:57-f229e70/netbsd-386-9_0
2021-10-28T18:01:34-03971e3-18b9702/netbsd-386-9_0
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/netbsd-amd64-9_0
2021-10-28T15:31:34-8de2a7f-8c9c148/netbsd-amd64-9_0
2021-10-28T03:44:12-103d89b-2ff1074/netbsd-amd64-9_0
2021-10-28T03:35:34-b8f928b/netbsd-amd64-9_0
2021-10-28T01:15:26-b2fe2eb/netbsd-386-9_0
2021-10-28T01:15:26-103d89b-b2fe2eb/netbsd-386-9_0
2021-10-27T21:37:54-7b0b504-749f6e9/netbsd-amd64-9_0
2021-10-27T21:34:46-de1abf7/netbsd-386-9_0
2021-10-27T21:34:46-de1abf7/netbsd-amd64-9_0
2021-10-27T20:29:07-51be206/netbsd-386-9_0
2021-10-27T20:25:06-bbc0595/netbsd-amd64-9_0
2021-10-27T20:24:34-56dcf97/netbsd-386-9_0
2021-10-27T20:03:17-7b0b504-68bd512/netbsd-386-9_0
2021-10-27T19:33:43-7b0b504-11b64b4/netbsd-amd64-9_0
2021-10-27T19:33:07-1b02aa2-514ebae/netbsd-386-9_0
2021-10-27T18:08:49-08f2e89-bb49eb3/netbsd-amd64-9_0
2021-10-27T17:22:43-714a425-11b64b4/netbsd-386-9_0
2021-10-27T16:59:43-c0ac39c/netbsd-amd64-9_0
2021-10-27T16:39:27-94870a3-4f73fd0/netbsd-386-9_0
2021-10-27T13:12:49-e1e2965-cfb5321/netbsd-amd64-9_0
2021-10-27T13:12:49-d418f37-cfb5321/netbsd-386-9_0
2021-10-27T06:23:35-5786a54/netbsd-386-9_0
2021-10-27T05:33:58-ca5f65d/netbsd-386-9_0
2021-10-26T22:24:36-591e12a-80be4a4/netbsd-386-9_0
2021-10-26T22:05:53-80be4a4/netbsd-amd64-9_0
2021-10-26T22:00:36-26dbf47-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:32:57-86f6bf1/netbsd-amd64-9_0
2021-10-26T21:17:38-03971e3-091db63/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-386-9_0
2021-10-26T18:40:06-9626607-11b64b4/netbsd-amd64-9_0
2021-10-26T18:33:39-244f92e-11b64b4/netbsd-amd64-9_0
2021-10-26T17:10:45-2c98350-76cef81/netbsd-386-9_0
2021-10-26T15:46:18-c4ead46-1b2362b/netbsd-386-9_0
2021-10-26T15:20:53-903c757-1b2362b/netbsd-amd64-9_0
2021-10-26T11:58:05-1e2820a/netbsd-386-9_0
2021-10-26T11:58:05-1e2820a/netbsd-amd64-9_0
2021-10-26T02:02:46-903c757-11b64b4/netbsd-amd64-9_0
2021-10-26T01:22:47-adfb85b/netbsd-386-9_0
2021-10-26T01:22:47-adfb85b/netbsd-amd64-9_0
2021-10-26T00:38:52-22652b4-0ec1b62/netbsd-386-9_0
2021-10-26T00:30:24-22652b4-b36b001/netbsd-amd64-9_0
2021-10-25T22:17:34-fd2f4b5/netbsd-386-9_0
2021-10-25T20:40:31-5a40697-2ac3bdf/netbsd-amd64-9_0
2021-10-19T18:56:08-07e5527/netbsd-386-9_0
2021-10-19T08:44:22-98f6e03-1e49210/netbsd-386-9_0
2021-10-19T07:45:46-ee92daa/netbsd-amd64-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-386-9_0
2021-10-19T07:45:46-98f6e03-ee92daa/netbsd-amd64-9_0
2021-10-18T21:57:36-98f6e03-3befaf0/netbsd-amd64-9_0
2021-10-18T21:52:05-98f6e03-425db64/netbsd-386-9_0
2021-10-18T21:52:05-425db64/netbsd-amd64-9_0
2021-10-18T20:39:22-267abbe/netbsd-amd64-9_0
2021-10-18T19:46:27-394a1ad/netbsd-386-9_0
2021-10-14T07:18:59-1349c6e/netbsd-amd64-9_0
2021-10-12T21:14:34-03971e3-c580180/netbsd-amd64-9_0
2021-10-11T15:48:29-3b03d30-2478b6f/netbsd-amd64-9_0

[mknyszek]

On it.