Very Long JIT Times During ETW Rundown

[brianrob]

Description

When a process does ETW rundown, a substantial portion of the rundown operation involves iterating over all jitted methods in the process and emitting events that support symbol and source-line resolution.

In programs with lots of jitted methods, this process can take a long time - 10s of seconds or even minutes. During this process, it is not possible to jit any methods, because CodeHeapIterator acquires and holds m_CodeHeapCritSec for the lifetime of the iterator.

Historically, we saw this on only the largest of workloads, and were able to address one of the code paths that lead to contention on m_CodeHeapCritSec, which took some of the pressure off. Unfortunately, though, we're seeing it more and more now, and it can have a material impact on service availability.

In looking at the code, I see that m_CodeHeapCritSec is protecting a few different data structures and scenarios, but I think I can break it down into two categories:

1. Creation and destruction of code heaps, as well as manipulation of the code heap list.
2. Allocation and free of memory into a code heap.

I'm wondering if it might be possible to separate these two categories of operations into two separate locks. By doing so, we could continue to allow jitting of methods concurrent with iteration of methods, and only block when a new code heap is needed, or manipulation of the code heap list is required. My suspicion is that manipulation of the list is much less common than allocation/free of memory within the code heap, and so this would (ideally) significantly reduce the affect of rundown on the running process.

[brianrob]

@jkotas, would love to get your opinion on this.

[jkotas]

My suspicion is that manipulation of the list is much less common than allocation/free of memory within the code heap

That depends on the workload. If a large workload creates collectible assemblies (e.g. using ref.emit), it is going to be creating and destroying codeheaps pretty often.

would love to get your opinion on this.

My solution would be:

• Take a quick snapshot (e.g. list of all codeheaps to process) under a global lock
• Delay destruction of any items in the snapshot until they got processed by the rundown
• Process the individual items in the snapshot without taking any locks for a long time

This design should avoid the process from being blocked for a long time by the rundown in all circumstances. The only potential problem is higher memory consumption during rundown since the items that need to be processed by the rundown may be kept in memory longer.

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[brianrob]

Thanks @jkotas. That makes sense. Is there an issue iterating over the methods in a code heap that is part of such a snapshot? From a quick look at MethodSectionIterator::Next, I am thinking that it we'd need to protect against a race between the iterator and a thread allocating a new method on the codeheap.

[jkotas]

I am thinking that it we'd need to protect against a race between the iterator and a thread allocating a new method on the codeheap.

Yes. For regular codeheaps, we should not need to take any locks. We just need to remember the last method to iterate in the codeheap as part of the snapshot. For dynamic method heaps, we would need to be more careful and take some locks. These can be very short duration locks.

[brianrob]

I'm going to take a stab at implementing this as time allows.