Skip to content

feat: allocate minimal blocks per window size #3028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 45 commits into from
Apr 17, 2025
Merged

feat: allocate minimal blocks per window size #3028

merged 45 commits into from
Apr 17, 2025

Conversation

netanel-haber
Copy link
Collaborator

No description provided.

@juney-nvidia juney-nvidia changed the title Feature/allocate minimal blocks per window size feat: Feature/allocate minimal blocks per window size Mar 24, 2025
@netanel-haber netanel-haber marked this pull request as draft March 24, 2025 12:22
@netanel-haber netanel-haber changed the title feat: Feature/allocate minimal blocks per window size feat: \allocate minimal blocks per window size Mar 24, 2025
@netanel-haber netanel-haber changed the title feat: \allocate minimal blocks per window size feat: allocate minimal blocks per window size Mar 24, 2025
@netanel-haber netanel-haber force-pushed the feature/allocate_minimal_blocks_per_window_size branch from b13fd0b to 8ee8036 Compare March 24, 2025 15:04
@netanel-haber
implement variable window attention by breaking the block manager int…
efa1996 
…o window block managers per window size

Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber netanel-haber force-pushed the feature/allocate_minimal_blocks_per_window_size branch from 8ee8036 to efa1996 Compare March 24, 2025 15:31
@netanel-haber netanel-haber mentioned this pull request Mar 24, 2025
Open
@netanel-haber
revert isCyclic to be true if the min attention window is reached, no…
bfdcb5a 
…t per window size

Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber
add explanatory comment to mCyclicThreshold
8b87594 
Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber
load correct gemma config
8b8a5cb 
Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber
don't shadow inputLength in addSequence - it should remain the functi…
1b68b16 
…on scope input length between window size loop iterations

Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber
fix KVCacheManagerVariableWindowAttentionWithReuseTest for multiple w…
8c7e8e4 
…indow block managers

Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber
Copy link
Collaborator Author

/bot run

1 similar comment
@Naveassaf
Copy link
Collaborator

/bot run

@Naveassaf
Copy link
Collaborator

/bot run

@niukuo
Copy link
Collaborator

niukuo commented Mar 26, 2025

PR_Github #587 [ run ] triggered by Bot

@niukuo
Copy link
Collaborator

niukuo commented Mar 26, 2025

PR_Github #587 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #499 completed with status: 'FAILURE'

@Naveassaf
Copy link
Collaborator

/bot run --disable-fail-fast

@niukuo
Copy link
Collaborator

niukuo commented Mar 26, 2025

PR_Github #601 [ run ] triggered by Bot

@niukuo
Copy link
Collaborator

niukuo commented Mar 26, 2025

PR_Github #601 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #509 completed with status: 'FAILURE'

@Naveassaf
Copy link
Collaborator

/bot run

@niukuo
Copy link
Collaborator

niukuo commented Mar 26, 2025

PR_Github #607 [ run ] triggered by Bot

@niukuo
Copy link
Collaborator

niukuo commented Mar 26, 2025

PR_Github #607 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #514 completed with status: 'FAILURE'

@netanel-haber
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2183 [ run ] completed with state ABORTED

@netanel-haber
Copy link
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2207 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2199 [ run ] completed with state ABORTED

@netanel-haber
only quantize gemma2 VSWA
bb4095e 
Signed-off-by: Netanel Haber <[email protected]>

remove misleading comment

Signed-off-by: Netanel Haber <[email protected]>

fix test_gemma

Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber netanel-haber force-pushed the feature/allocate_minimal_blocks_per_window_size branch from de24687 to bb4095e Compare April 14, 2025 22:23
@netanel-haber
fix test_gemma
dfddbe5 
Signed-off-by: Netanel Haber <[email protected]>
@netanel-haber
fix test_gemma
cf86e35 
Signed-off-by: Netanel Haber <[email protected]>
@tensorrt-cicd
Copy link
Collaborator

PR_Github #2207 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1597 completed with status: 'FAILURE'

@netanel-haber
Copy link
Collaborator Author

/bot --help

@github-actions GitHub Actions
Copy link

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]

Launch build/test pipelines. All previously running jobs will be killed.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-1, xxx" (OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-[Post-Merge]-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

netanel-haber and others added 2 commits April 16, 2025 16:20
@netanel-haber
in sendRequestInfo, fromOldAllocatedBlockIds->fromOldAllocatedBlockId…
4947552 
…s, like in main

Signed-off-by: Netanel Haber <[email protected]>
@Funatiq @netanel-haber
fix: disable KV cache reuse if using attention sink (NVIDIA#3021)
9a60e91 
* fix: disable KV cache reuse if using attention sink

Signed-off-by: Robin Kobus <[email protected]>

* fix: disable KV cache reuse if sink bubble

Signed-off-by: Robin Kobus <[email protected]>

* add comment

Signed-off-by: Robin Kobus <[email protected]>

---------

Signed-off-by: Robin Kobus <[email protected]>
@netanel-haber
Copy link
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2494 [ run ] triggered by Bot

@netanel-haber
Copy link
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2501 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2494 [ run ] completed with state ABORTED

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2501 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1795 completed with status: 'FAILURE'

@netanel-haber
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2525 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2525 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #1809 completed with status: 'SUCCESS'

@Funatiq
Copy link
Collaborator

Funatiq commented Apr 17, 2025

/bot reuse-pipeline

@Funatiq Funatiq enabled auto-merge (squash) April 17, 2025 07:50
@tensorrt-cicd
Copy link
Collaborator

PR_Github #2612 [ reuse-pipeline ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #2612 [ reuse-pipeline ] completed with state SUCCESS
Reusing PR_Github #2525 for commit 9f6be5d

@Funatiq Funatiq merged commit 3c52ac0 into NVIDIA:main Apr 17, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Funatiq Funatiq Funatiq approved these changes

@thorjohnsen thorjohnsen Awaiting requested review from thorjohnsen

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@netanel-haber @Naveassaf @niukuo @tensorrt-cicd @Funatiq