feat: [Deepseek] Add trtllm-gen MOE FP4 MOE backend

[hlu1]

TRT-LLM gen cut-off commits:

• routing kernel : 8339307c73fd8e30eea9ee0d540b2b9af2425708
• fp8 MOE cubins: 121b8bcbbade35125763e20a4845c79d8a2c7175
• fp4 MOE cubins: a9f6863d2257f06b6955a29cde4ee4e139e0e129 (most likely)

Added features:

• A new FusedMOE backend, "TRTLLM" for fp8 Deepseek recipe and nvfp4 which invoke the trtllm-gen MOE kernels.
• E2E enablement with fp8 Deepseek checkpoint. The bmms in MLA are in bf16. The other gemm/MOE are in fp8.

Remaining tasks:

• Routing kernel and cubins currently correspond to different trtllmgen commits. They need to be updated to top of tree.
• Change type of scoreBias from float to bfloat16 in the routing kernel and verify the accuracy: auto scoreBias = float{scoreSigmoid + float{biasVal}};. This must be done before updating the routing kernel to top of tree.
• Enable TRT-LLM Gen MOE for Deepseek model in CI
• Remove w1, w3 swap in fp4FC1 when the cubins are updated next time (Done internally already)
• Read tile config from kernel instead of hardcoding it to 128
• FP8 e2e with R1 is currently broken. There are runtime errors during warming up.

nvfp4 accuracy results:
MMLU: 86.91
gpqa_diamond (3 runs): 0.6717171717171717, 0.7373737373737373, 0.6919191919191919

[hlu1]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #1523 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #1523 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #1137 completed with status: 'FAILURE'

[nekorobov]

Thanks a lot! Only few nits. The API of the MoE cubins is a subject for change in the future in any case

[hlu1]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #1639 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #1639 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #1226 completed with status: 'FAILURE'

[hlu1]

/bot run --disable-fail-fast

[zongfeijing]

/bot run --multi-gpu-test

[tensorrt-cicd]

PR_Github #2833 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2833 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2000 completed with status: 'SUCCESS'

[zongfeijing]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2838 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2838 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #2004 completed with status: 'FAILURE'

[zongfeijing]

/bot run

[tensorrt-cicd]

PR_Github #2844 [ run ] triggered by Bot

[tensorrt-cicd]

PR_Github #2844 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2009 completed with status: 'FAILURE'

[zongfeijing]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #2857 [ run ] triggered by Bot

[meowcoder22]

hi again @zongfeijing

In runner.cu, there is option called options.mUseCustomLowLatencyImpl = false;

when set to true, it require mTileN size of 32, 64, 128

however in gemmList.h, there is only kernel that support mTileN sisze of 8

@hlu1 is there any plan to include appropriate kernel here?

Otherwise what is the use of this option

[tensorrt-cicd]

PR_Github #2857 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #2018 completed with status: 'SUCCESS'

[zongfeijing]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #2873 [ reuse-pipeline ] triggered by Bot

[tensorrt-cicd]

PR_Github #2873 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #2857 for commit 4bd7319