feat: Support CUDA graphs for EAGLE3

[mikeiovine]

Support CUDA graphs for the EAGLE-3 spec decode. Also contains fixes for loading eagle 3 weights for llama3 70B models (previously only tested for 8b).

The graphs significantly improve the performance. However, we still have a lot of work to do to eliminate the host overheads.

[mikeiovine]

Putting this out to get early feedback and ideas. I'm not really happy with the design right now. Specifically, passing the states between the target and draft models gets pretty complicated when CUDA graphs enter the picture.

[hlu1]

Mostly nits.

[mikeiovine]

Had some discussion offline with @hlu1 about how to make it cleaner:

1. extra_model_inputs is hard to extend. A single class ModelInput that all models consume would allow us to easily add new features to other models.
2. At the same time, spec_decode_extra_input_info can be generalized. We can rename it to get_input_shapes, and it can return a dict (str -> (shape, dtype)). It can have some sensible defaults to avoid the burden of having to implement this for more standard models. Decided to just rename it get_warmup_extra_inputs.

I think (2) should definitely be done now. (1) is a pretty big refactor and would probably have to be done in followups.

[mikeiovine]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2196 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2196 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1586 completed with status: 'FAILURE'

[mikeiovine]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2212 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2212 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1601 completed with status: 'FAILURE'

[mikeiovine]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2348 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2348 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1688 completed with status: 'FAILURE'

[mikeiovine]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2357 [ run ] triggered by Bot

[mikeiovine]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2363 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2357 [ run ] completed with state ABORTED

[hlu1]

Approve to unblock.

[tensorrt-cicd]

PR_Github #2363 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1699 completed with status: 'SUCCESS'

[mikeiovine]

/bot run --disable-fail-fast

[mikeiovine]

Running CI one more time before merging as it has been a while since my last rebase.

[tensorrt-cicd]

PR_Github #2497 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2497 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1791 completed with status: 'SUCCESS'

[mikeiovine]

/bot skip --comment "Pipeline passed before last rebase"

[tensorrt-cicd]

PR_Github #2510 [ skip ] triggered by Bot

[tensorrt-cicd]

PR_Github #2510 [ skip ] completed with state SUCCESS
Skipping testing for commit efba97d