Skip to content

Doc: update steps of using Draft-Target-Model (DTM) in the documents. #3366

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 9, 2025

Conversation

wili-65535
Copy link
Collaborator

In this PR, we update the steps of using Draft-Target-Model (DTM) speculative decoding in TensorRT-LLM and TensorRT-LLm-backend.

Based on TRTLLM-backend: commit 86542b637bbccdd708ab892337c8ad3a95932131 (tag: v0.18.0) and docker image nvcr.io/nvidia/tritonserver:25.03-trtllm-python-py3.

@wili-65535 wili-65535 force-pushed the doc/Draft-Target-Model branch 2 times, most recently from 29bb508 to 9d67cb7 Compare April 9, 2025 00:24
wili-65535 added 4 commits April 9, 2025 12:40
@wili-65535 wili-65535 force-pushed the doc/Draft-Target-Model branch from 9d67cb7 to f3624c7 Compare April 9, 2025 04:40
@tensorrt-cicd
Copy link
Collaborator

PR_Github #1554 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #1161 completed with status: 'FAILURE'

Signed-off-by: wili-65535 <[email protected]>
@NVIDIA NVIDIA deleted a comment from tensorrt-cicd Apr 9, 2025
@wili-65535
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #1559 [ run ] triggered by Bot

@kaiyux kaiyux requested review from lfr-0531 and yweng0828 April 9, 2025 05:43
@kaiyux
Copy link
Member

kaiyux commented Apr 9, 2025

@lfr-0531 @yweng0828 Can you help review this PR?

@tensorrt-cicd
Copy link
Collaborator

PR_Github #1559 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1165 completed with status: 'SUCCESS'

Copy link
Collaborator

@lfr-0531 lfr-0531 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~

@yweng0828
Copy link
Collaborator

Hi @wili-65535 , thanks for the update. The documentation is much better organized now. : )

@kaiyux kaiyux requested a review from Copilot April 9, 2025 09:11
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

examples/draft_target_model/README.md:89

  • The configuration explanation above uses '[4,[0],[1],False]' as an example, but the code sample uses '[4,[0],[1],True]'. Please update one of them to ensure consistency.
    --draft_target_model_config="[4,[0],[1],True]" \

wili-65535 added 2 commits April 9, 2025 17:18
Signed-off-by: wili-65535 <[email protected]>
Signed-off-by: wili-65535 <[email protected]>
@kaiyux
Copy link
Member

kaiyux commented Apr 9, 2025

/bot reuse-pipeline

@kaiyux kaiyux enabled auto-merge (squash) April 9, 2025 09:23
@wili-65535
Copy link
Collaborator Author

Thank you @lfr-0531, @yweng0828 and @kaiyux ! And I fix the comments Copilot comes up with, so could we continue to merge?

@kaiyux
Copy link
Member

kaiyux commented Apr 9, 2025

Thank you @lfr-0531, @yweng0828 and @kaiyux ! And I fix the comments Copilot comes up with, so could we continue to merge?

I've already set auto-merge. Thanks.

@tensorrt-cicd
Copy link
Collaborator

PR_Github #1590 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #1590 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #1559 for commit ec270fc

@kaiyux kaiyux merged commit 6f1b2cd into NVIDIA:main Apr 9, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants