runtime: performance and diagnostics meeting notes

[mknyszek]

Since 2022, members of the Go team and the Go community have met every two weeks to discuss Go runtime topics. Until 2025 the focus was mainly on diagnostics, but we've also delved into plenty of other topics related to the runtime as well. To reflect this, I'm renaming this group to the "performance and diagnostics biweekly meeting."

We currently have a virtual sync every 2 weeks, Thursdays at 11 AM NYC time. Please ping me at mknyszek -- at -- golang.org for an invite if you're interested in attending. This issue will be updated regularly with meeting notes from those meetings.

The reasons for starting this meeting in 2022 are still relevant today. So for the sake of some nostalgia, and also as a reminder of the importance of diagnostics to making Go an effective tool for our users, I leave behind some of the original text from this GitHub issue below.

---

As the Go user base grows, more and more Go developers are seeking to understand the performance of their programs and reduce resource costs. However, they are locked into the relatively limited diagnostic tools we provide today. Some teams build their own tools, but right now that requires a large investment. This issue extends to the Go team as well, where we often put significant effort into ad-hoc performance tooling to analyze the performance of Go itself.

This issue is a tracking issue for improving the state of Go runtime diagnostics and its tooling, focusing primarily on runtime/trace traces and heap analysis tooling.

[mknyszek]

2022-12-07 Sync

Attendees: @mknyszek @aclements @prattmic @felixge @nsrip-dd @rhysh

Notes:

• pprof labels in execution traces
  • Michael K: I need to follow up on the CL and issue.
  • Michael P: Have you considered runtime/trace regions?
  • Nick: Yes, but it doesn't quite hit our use-cases.
  • Rhys: Could use logs instead. Is the point to follow goroutines?
  • Felix: Yeah.
  • Rhys: Started parsing the trace format for some internal tooling.
  • Felix: Prefer labels because inheritance. Mostly like it for profile tools on top, but maybe doesn't matter for tracing tools. The parent goroutine gets recorded so things like request IDs can be tracked in post-processing.
    • Also useful for attributing CPU time.
  • Michael P: Tasks are inherited, but inheritance is processed downstream.
    • Would be nice to have pprof labels just so you don't have to think about which one to use.
  • Michael K: Useful even just to bridge pprof and runtime/trace.
  • Austin: Agreed. We can one day get to the point of deprecating the old APIs as well.
  • Rhys: RE: Attributing CPU time, can see at better than 10 ms of granularity already (even if Ps aren't CPU time, it's still time the rest of the app couldn't run).
  • Michael P: There's an issue about measure CPU time on-line. (TODO: Find it.)
• Trace parsing API
  • Michael K: How important is this? Priority?
  • Felix: Important for the community, but we can't make use of it in our case. Will the trace format change in between releases?
  • Michael K: I think we can always guarantee a trace format for a release.
  • Michael P: How high level should this API be?
    • cmd/trace has two levels:
      • Low-level that understands format and events.
      • Higher-level that understands relationships between goroutines, etc.
  • Michael K: Page trace splits this into "parser" and "simulator." The latter is more stateful.
  • Felix: Intuitive feeling toward lower level API.
  • Rhys: +1 to low level.
  • Austin: Scalability of processing traces.
    • Currently not in a good state in low or high level format (currently requires the whole trace).
    • Can fix trace wire format for low-level parsing scalability issues, but it's much less clear how to do this for the high-level format.
  • Austin: Flight recorder idea.
    • Interacts interestingly with simulation. Current trace snapshots everything.
    • Solved this in debuglog; reads its own tail and keeps local state updated.
    • Complicated trade-offs in this space.
  • Felix: We use a lot of JFR, one thing that's nice is it's broken down into self-contained chunks.
• Michael K sent out a very half-baked trace format revamp. (Thanks for the comments! Far from ready to share more widely.)
  • The next step is to measure the actual current overhead.
    • Maybe add a mode to Sweet?
    • Rhys: Have been collecting CPU profiles and execution traces. 20% of CPU time during execution trace is for execution trace itself. 95% of overhead is collecting stack traces.
      • Collect 1 second every 5000 seconds and no one complains. People do complain about goroutine profiles every 2 minutes.
    • Michael K: Shooting for KUTrace overhead, so making stack traces optional/faster is just step 1.
    • Felix: Trace effect on tail latency.
      • Rhys: Traces are my view of tail latency.
    • Felix: Benchmark for pathological cases and worst case.
    • Austin: Linked to trace overhead issue, Dmitry proposed switching to frame pointer unwinding.
    • Felix: At some point implemented frame pointer unwinding in userland and it was 50x faster (link).
    • Rhys: Not sure what kind of tool you could build without stack traces in an app that doesn't set pprof labels, tasks, regions, trace logs, etc.
    • Michael K: Integration with structured logging?
    • Michael P: It does add yet another way to add things to runtime/trace.
    • Rhys: The standard library (e.g. database/sql) doesn't currently use runtime/trace at all, maybe it should.
    • Michael K: This connects to deciding what goes into a trace. I think this is a very good idea.
    • Felix: +1. Java world and JFR does this.
• Deciding what goes into a trace
  • Disabling stack tracing / reduce stack trace depth
  • Filtering by pprof labels
  • Specific event classes
  • Standard library events
  • Rhys: I've made this decision for my organization. Expected that you do profiling for a long running service. No opportunity for app owners to express opinions. People who complained forked the package, turned it off, and now coming back. I kind of want everything.
  • Felix: I would love to be in a place where we can do that, but we get pushback from users when the overhead is too high.
  • Rhys: The question is how close we get within 1% overhead. My choice was to get everything, but less often.
  • Felix: Desire to get more of everything is in conflict with adding more kinds of things in the trace.
  • Michael P: Agreed. Ideally we have tracing that's sufficiently fast that we have on all the time, but if libraries are allowed to add new traces then it could be a problem. It would be nice to turn that off without forking a library.
• Before next sync:
  • Michael K: Unblock pprof labels patch and benchmarking trace overhead.
  • Felix: I can contribute a worst case benchmark.
    • Currently blocked on pprof labels in trace.
  • Felix: Started to work on gentraceback. Might work on it over the holidays.
    • Trying for bug-for-bug compatibility.
  • Michael P: Austin has been working on this too.

[felixge]

I'll miss the Dec 22nd meetup because I'm traveling for the holidays. That being said, if I find time I might also look into #57159 . Getting a proof of concept for Perfetto UI integration (ideally using their protocol buffer format) is probably more important than the gentraceback refactoring at this point. I just tried to work with a 300 MB (15s of prod activity) yesterday, and it was a real eye opener to the way the current UI struggles.

[tbg]

I don't know if it's relevant (probably nothing new for the folks on this thread), but I had similar problems with the go tool trace viewer where it would freeze on me all the time, esp. in the per-goroutine view (/trace?goid=N). I figured out you can download perfetto-compatible JSON data from /jsontrace?goid=N. (/jsontrace gives the default view). This can then be uploaded to ui.perfetto.dev. This doesn't show all the information in the trace so it's not as great, but I was glad to have something that worked.

[thediveo]

would the pprof labels also show up in goroutine traces?

[qmuntal]

I'm working on a PoC that improves native stack unwinding on Windows by adding additional information to the PE file. This will help debugging with WinDbg and profiling with Windows Performance Analyzer. Would this work fit into the effort tracked by this issue?

[mknyszek]

@thediveo I think that might be a good question for #56295, or you could file another issue. Off the top of my head, that doesn't sound like it would be too difficult to do.

@qmuntal Oh neat! That's awesome. I think it's a little tangential to the work we're proposing here, unless you also plan to do anything with the runtime's unwinder (i.e. gentraceback). Then again, if one of the goals is better integration with the Windows Performance Analyzer that's certainly more in the same spirit. Do you have an issue for tracking that already?

[qmuntal]

Do you have an issue for tracking that already?

I still have to prepare the proposal, I plan to submit it next week.

unless you also plan to do anything with the runtime's unwinder (i.e. gentraceback).

Not for now, but once I finish this I want to investigate how feasible is too unwind native code and merge it with the Go unwinding, in case the exception happens in a non-Go module.

[qmuntal]

Do you have an issue for tracking that already?

I do now #57302 😄