Skip to content

Add support for GPTNeo for KV cache injection #1720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 14, 2023
Merged

Add support for GPTNeo for KV cache injection #1720

merged 6 commits into from
Sep 14, 2023

Conversation

mgoin
Copy link
Member

@mgoin mgoin commented Sep 12, 2023

Thanks to @dbogunowicz for building up the injection config. Tested with TinyStories models on HF and then in deepsparse pipeline

https://huggingface.co/mgoin/TinyStories-1M-deepsparse
https://huggingface.co/mgoin/TinyStories-33M-deepsparse

optimum-cli export onnx --task text-generation -m roneneldan/TinyStories-1M TinyStories-1M 
python ~/onnx_kv_inject.py --input-file TinyStories-1M/decoder_model.onnx --output-file TinyStories-1M/model.onnx
numactl -N0 -m0 python deepsparse/examples/chat-compare/app.py TinyStories-1M --sequence_length 2048 --max_new_tokens 2000
Welcome to the interactive text generation interface!
Type 'exit' to end the session.
> Once upon a time

<DeepSparse output>

Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, shiny rock in the sky. She wanted to touch it, but it was too high.

Lily's mommy told her that it was important to be careful and not touch things. Lily didn't want to touch it, so she asked her mommy if she could touch it. Her mommy said yes and they went to the park.

Lily was so happy and excited to see the rock. She picked it up and started to play with it. She was so happy that she forgot about the rock. She was happy that she had made a new friend.
<|endoftext|>

<ORT output>

Once upon a time, there was a little girl named Lily. She loved to play outside in the sunshine. One day, she saw a big, shiny rock in the sky. She wanted to touch it, but it was too high.

Lily's mommy told her that it was important to be careful and not touch things. Lily didn't want to touch it, so she asked her mommy if she could touch it. Her mommy said yes and they went to the park.

Lily was so happy and excited to see the rock. She picked it up and started to play with it. She was so happy that she forgot about the rock. She was happy that she had made a new friend.
<|endoftext|>

@mgoin mgoin requested review from dbogunowicz, a team, natuan and abhinavnmagic and removed request for a team September 12, 2023 16:56
dbogunowicz
dbogunowicz previously approved these changes Sep 13, 2023
@mgoin mgoin merged commit d8f015b into main Sep 14, 2023
@mgoin mgoin deleted the gpt-neo-kv-inject branch September 14, 2023 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants