Bug: CUDA error: an illegal memory access was encountered

[nux]

What happened?

Not sure if this is a problem with me or ik_llama - but getting this while starting prompt processing (ubergarm's deepseek-v3)

May 15 08:57:29 red llama-swap[80783]: INFO [ launch_slot_with_task] slot is processing task | tid="139638925832192" timestamp=1747317449 id_slot=0 id_task=3
May 15 08:57:29 red llama-swap[80783]: INFO [ update_slots] kv cache rm [p0, end) | tid="139638925832192" timestamp=1747317449 id_slot=0 id_task=3 p0=0
May 15 08:57:36 red kernel: NVRM: Xid (PCI:0000:01:00): 31, pid=80798, name=llama-server, Ch 00000008, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC1 GPCCLIENT_T1_3 faulted @ 0x7e9f_4f200000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
May 15 08:57:36 red llama-swap[80783]: CUDA error: an illegal memory access was encountered
May 15 08:57:36 red llama-swap[80783]: current device: 0, in function ggml_backend_cuda_synchronize at /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:3067
May 15 08:57:36 red llama-swap[80783]: cudaStreamSynchronize(cuda_ctx->stream())
May 15 08:57:36 red llama-swap[80783]: /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:110: CUDA error
May 15 08:57:36 red kernel: llama-server[80906]: segfault at 204803fe0 ip 00007f00399189d7 sp 00007ffc4a6104f0 error 4 in libcuda.so.575.51.03[7f00395c5000+e97000] likely on CPU 11 (core 11, socket 0)
May 15 08:57:36 red kernel: Code: ef e8 9d c9 ca ff 83 3d 7e 57 2f 05 01 49 8b 1c 24 76 0a 8b 05 86 57 2f 05 85 c0 74 56 49 8b 44 24 10 41 8b 4c 24 24 48 8b 13 <8b> 00 41 39 c6 74 52 8b b3 40 40 00 00 48 89 f0 89 8c b3 44 40 00

Name and Version

./build/bin/llama-server --version
version: 3697 (34ae71c)
built with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

No response

Relevant log output

[ikawrakow]

What was the command line?
Are you running this model for the first time? If not, did you experience this error on an earlier ik_llama.cpp version?

[nux]

Here is the command I am running:
/home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf --ctx-size 32768 -mla 2 -fa -amb 512 -fmoe --temp 0.3 --min-p 0.05 --n-gpu-layers 63 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8081

This is the model I use primarily - been working well for a while now. I pulled it out of my normal llama-swap setup and running manually....

It worked when I sent a random benchmark to it (solobench). When I attempt to redo a prompt sent from open-webui it crashed again:
INFO [ update_slots] kv cache rm [p0, end) | tid="139648547147776" timestamp=1747318222 id_slot=0 id_task=0 p0=0
CUDA error: an illegal memory access was encountered
current device: 0, in function ggml_backend_cuda_synchronize at /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:3067
cudaStreamSynchronize(cuda_ctx->stream())
/home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:110: CUDA error

I will attempt to send the same prompt that I am sending in open webui, with the cli client I used when it worked...

Odd. The prompt was pasting some php code and a warning it was throwing.

Do you want me to try and get the prompt posted for you? Would try and remove parts of prompt I don't really want to post on github, and see if it still crashes.

[Panchovix]

If you try without -fmoe, does it works?

[nux]

Nope:

INFO [ update_slots] kv cache rm [p0, end) | tid="140102707097600" timestamp=1747318723 id_slot=0 id_task=0 p0=0
CUDA error: an illegal memory access was encountered
current device: 0, in function ggml_backend_cuda_synchronize at /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:3067
cudaStreamSynchronize(cuda_ctx->stream())
/home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:110: CUDA error
Aborted

Command I ran was:

nux@red ~/dev/ik_llama.cpp $ /home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf --alias ubergarm/DeepSeek-R1-V3-0324-IQ4_K_R4 --alias "deepseek-v3" --ctx-size 32768 -mla 2 -fa -amb 512 --temp 0.3 --min-p 0.05 --n-gpu-layers 63 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8080

[nux]

Would you like me to try with llama.cpp vanilla? Err...I'm not sure that model loads there. Perhaps I could try other models if you think it would be useful

[Panchovix]

I think R4 doesn't work on llamacpp, yeah. You can try with unsloth quants there https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF-UD.

[ikawrakow]

There is a place in the log that looks like this:

llama_model_loader: - type  f32:   66 tensors
llama_model_loader: - type q6_K:    1 tensors
llama_model_loader: - type iq5_ks:  225 tensors

Seeing this will be helpful.

[nux]

llama_model_loader: - type f32: 361 tensors
llama_model_loader: - type q8_0: 612 tensors
llama_model_loader: - type iq4_k_r4: 116 tensors
llama_model_loader: - type iq5_k_r4: 58 tensors

I ran the same prompt through: unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf with ik_llama /home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/nvme/models/unsloth/Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf --ctx-size 32768 -mla 2 -fa -amb 512 --temp 0.3 --min-p 0.05 --n-gpu-layers 63 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8080
and it worked fine.

It looks like I do have unsloth/DeepSeek-V3-0324-GGUF/UD-Q4_K_XL on a network storage. If you want me to test that I can.

Edit: I will throw another prompt at the model I had a problem with for some other php stuff and see how it goes. At this point if it's that one prompt and everything else works fine...we could close this for now

[nux]

Worked with another php related prompt (first one had a ~80 line function pasted in, this one was only 5 lines). Odd...

[ikawrakow]

It looks like I do have unsloth/DeepSeek-V3-0324-GGUF/UD-Q4_K_XL on a network storage. If you want me to test that I can.

If you have time, yes, this can be helpful.

But based on the described symptoms

• It was working before with the same model where we get illegal memory access now
• There are no tensors that were computed on the CPU before and are now computed on the GPU

I have no hypothesis what changed. You can try using -mla 3 instead of -mla 2 as this is now supported on CUDA. It may make your TG speed better (especially for long context), but it also eliminates two matrix multiplications that are done in the FA kernel.

[nux]

Interesting...I've been trying various combinations of models/parameters, and so far here's what I have:

ik_llama crashes with ds v3 with unsloth or ubergarms variant.

If I run it fully on CPU it doesnt crash:
/home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf --alias "deepseek-v3" --ctx-size 32768 -fa --temp 0.3 --min-p 0.05 --n-gpu-layers 0 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8080

If I put a single layer on GPU it does crash:
/home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf --alias "deepseek-v3" --ctx-size 32768 -fa --temp 0.3 --min-p 0.05 --n-gpu-layers 1 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8080

I left off all the mla amb fmoe stuff. Going to see if it crashes with vanilla llama.cpp

ik_llama.cpp:
CUDA error: an illegal memory access was encountered
current device: 0, in function ggml_backend_cuda_synchronize at /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:3067
cudaStreamSynchronize(cuda_ctx->stream())
/home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:110: CUDA error
Aborted
nux@red ~/dev/ik_llama.cpp $ /home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/amp/models/unsloth/DeepSeek-V3-0324-GGUF/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00009.gguf --alias "deepseek-v3" --ctx-size 32768 -fa --temp 0.3 --min-p 0.05 --n-gpu-layers 1 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8080

llama.cpp:
/home/nux/dev/llama.cpp/build/bin/llama-server --model /mnt/amp/models/unsloth/DeepSeek-V3-0324-GGUF/UD-Q4_K_XL/DeepSeek-V3-0324-UD-Q4_K_XL-00001-of-00009.gguf --alias deepseek-v3 --ctx-size 32768 -fa --temp 0.3 --min-p 0.05 --n-gpu-layers 1 --override-tensor exps=CPU --parallel 1 --threads 32 --host 0.0.0.0 --port 8080

and it works.
prompt eval time = 21301.26 ms / 515 tokens ( 41.36 ms per token, 24.18 tokens per second)
eval time = 98803.52 ms / 626 tokens ( 157.83 ms per token, 6.34 tokens per second)
total time = 120104.78 ms / 1141 tokens

[ciprianveg]

Here is the command I am running: /home/nux/dev/ik_llama.cpp/build/bin/llama-server --model /mnt/nvme/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4/DeepSeek-V3-0324-IQ4_K_R4-00001-of-00010.gguf --ctx-size 32768 -mla 2 -fa -amb 512 -fmoe --temp 0.3 --min-p 0.05 --n-gpu-layers 63 --override-tensor "exps=CPU" --parallel 1 --threads 32 --host 0.0.0.0 --port 8081

This is the model I use primarily - been working well for a while now. I pulled it out of my normal llama-swap setup and running manually....

It worked when I sent a random benchmark to it (solobench). When I attempt to redo a prompt sent from open-webui it crashed again: INFO [ update_slots] kv cache rm [p0, end) | tid="139648547147776" timestamp=1747318222 id_slot=0 id_task=0 p0=0 CUDA error: an illegal memory access was encountered current device: 0, in function ggml_backend_cuda_synchronize at /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:3067 cudaStreamSynchronize(cuda_ctx->stream()) /home/nux/dev/ik_llama.cpp/ggml/src/ggml-cuda.cu:110: CUDA error

I will attempt to send the same prompt that I am sending in open webui, with the cli client I used when it worked...

Odd. The prompt was pasting some php code and a warning it was throwing.

Do you want me to try and get the prompt posted for you? Would try and remove parts of prompt I don't really want to post on github, and see if it still crashes.

Similar thing happens to me, it worked 2 days ago, i rebuilt it with latest sources yesterday. I am using qwen3 235b q4 ud xl. Llama-sweep-bench works fine but when i send a real prompt via open web ui, it crashes.. On the last days since it worked I changed nvidia driver from 535 to 550, cuda version from 12.2 to 12.6, i pulled latest changes yesterday and rebuilt..

[ikawrakow]

@ciprianveg Can you also give the build for the last version that worked, tell us if the crash happens during PP or during TG, and post the line from the log where it says where the illegal memory access was encountered? Thanks. Also, is it a single GPU or a multi-GPU setup?

[ciprianveg]

Hello, it was built from main 20 h ago, now i rebuilt from main 30m ago with latest changes (from 2h ago) and same error:
INFO [ update_slots] kv cache rm [p0, end) | tid="136731577430016" timestamp=1747469764 id_slot=0 id_task=0 p0=0
VERB [ update_slots] prompt processing progress | tid="136731577430016" timestamp=1747469764 id_slot=0 n_past=33 n_ctx=20480 n_tokens=33 progress=1.0
VERB [ update_slots] prompt done | tid="136731577430016" timestamp=1747469764 id_slot=0 n_past=33 n_ctx=20480 n_tokens=33
VERB [ update_slots] decoding batch | tid="136731577430016" timestamp=1747469764 n_tokens=33
CUDA error: an illegal memory access was encountered
current device: 2, in function ggml_backend_cuda_synchronize at /home/ciprian/ai/ik_llama.cpp/ggml/src/ggml-cuda.cu:3067
cudaStreamSynchronize(cuda_ctx->stream())
/home/ciprian/ai/ik_llama.cpp/ggml/src/ggml-cuda.cu:110: CUDA error
Could not attach to process. If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
No stack.
The program is not being run.
Aborted (core dumped)

This was the test query: Tell me a random fun fact about the Roman Empire

what is strange is the with exact same command th llama-sweeb-bench works ok:
main: n_kv_max = 20480, n_batch = 2048, n_ubatch = 2048, flash_attn = 1, n_gpu_layers = 99, n_threads = 16, n_threads_batch = 16

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
2048	512	0	12.843	159.47	51.788	9.89
2048	512	2048	13.000	157.54	51.361	9.97

last main pull done, that worked was 3 days ago..

[ciprianveg]

it happens with both Qwen3-235B-A22B-UD-Q3_K_XL and Qwen3-235B-A22B-UD-Q4_K_XL. I am using 2 3090 gpus and 2 A4000, built with 1 copy of cache parameter. I think the multiple gpus can be the issue.. but iti is very strange that llama sweep bench works
..