[Bug]: guided_grammar example syntax does not work

[Famondir]

Your current environment

I'm using vllm hosted on a K8s instance and was not able to execute the environment collection python file there. But this is the error I get:

Error message

INFO 04-21 00:33:45 [loggers.py:87] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 66.7%                                                                                  
INFO 04-21 00:33:55 [logger.py:39] Received request chatcmpl-5a7a5fbaada34f3a88b577a238ddd279: prompt: "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nGenerate an SQL query to show the 'username' and 'email'from the 'users' table.<|im_end|>\n<|im_start|>assistant\ 
INFO 04-21 00:33:55 [async_llm.py:228] Added request chatcmpl-5a7a5fbaada34f3a88b577a238ddd279.                                                                                                                                                                                                         
ERROR 04-21 00:33:55 [core.py:387] EngineCore hit an exception: Traceback (most recent call last):                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 380, in run_engine_core                                                                                                                                                                
ERROR 04-21 00:33:55 [core.py:387]     engine_core.run_busy_loop()                                                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 402, in run_busy_loop                                                                                                                                                                  
ERROR 04-21 00:33:55 [core.py:387]     self._process_engine_step()                                                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 431, in _process_engine_step                                                                                                                                                           
ERROR 04-21 00:33:55 [core.py:387]     outputs = self.step_fn()                                                                                                                                                                                                                                         
ERROR 04-21 00:33:55 [core.py:387]               ^^^^^^^^^^^^^^                                                                                                                                                                                                                                         
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 206, in step                                                                                                                                                                           
ERROR 04-21 00:33:55 [core.py:387]     scheduler_output = self.scheduler.schedule()                                                                                                                                                                                                                     
ERROR 04-21 00:33:55 [core.py:387]                        ^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                     
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 286, in schedule                                                                                                                                                              
ERROR 04-21 00:33:55 [core.py:387]     if structured_output_req and structured_output_req.grammar:                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                       
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/structured_output/request.py", line 43, in grammar                                                                                                                                                           
ERROR 04-21 00:33:55 [core.py:387]     completed = self._check_grammar_completion()                                                                                                                                                                                                                     
ERROR 04-21 00:33:55 [core.py:387]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                     
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/structured_output/request.py", line 31, in _check_grammar_completion                                                                                                                                         
ERROR 04-21 00:33:55 [core.py:387]     self._grammar = self._grammar.result(timeout=0.0001)                                                                                                                                                                                                             
ERROR 04-21 00:33:55 [core.py:387]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                             
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result                                                                                                                                                                                        
ERROR 04-21 00:33:55 [core.py:387]     return self.__get_result()                                                                                                                                                                                                                                       
ERROR 04-21 00:33:55 [core.py:387]            ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                       
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result                                                                                                                                                                                  
ERROR 04-21 00:33:55 [core.py:387]     raise self._exception                                                                                                                                                                                                                                            
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run                                                                                                                                                                                           
ERROR 04-21 00:33:55 [core.py:387]     result = self.fn(*self.args, **self.kwargs)                                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/structured_output/__init__.py", line 77, in _async_create_grammar                                                                                                                                            
ERROR 04-21 00:33:55 [core.py:387]     return self.backend.compile_grammar(request_type, grammar_spec)                                                                                                                                                                                                  
ERROR 04-21 00:33:55 [core.py:387]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                  
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/structured_output/backend_xgrammar.py", line 97, in compile_grammar                                                                                                                                          
ERROR 04-21 00:33:55 [core.py:387]     ctx = self.compiler.compile_grammar(grammar_spec)                                                                                                                                                                                                                
ERROR 04-21 00:33:55 [core.py:387]           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/xgrammar/compiler.py", line 211, in compile_grammar                                                                                                                                                                  
ERROR 04-21 00:33:55 [core.py:387]     grammar = Grammar.from_ebnf(grammar, root_rule_name=root_rule_name)                                                                                                                                                                                              
ERROR 04-21 00:33:55 [core.py:387]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                              
ERROR 04-21 00:33:55 [core.py:387]   File "/usr/local/lib/python3.12/dist-packages/xgrammar/grammar.py", line 110, in from_ebnf                                                                                                                                                                         
ERROR 04-21 00:33:55 [core.py:387]     return Grammar._create_from_handle(_core.Grammar.from_ebnf(ebnf_string, root_rule_name))                                                                                                                                                                         
ERROR 04-21 00:33:55 [core.py:387]                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                          
ERROR 04-21 00:33:55 [core.py:387] RuntimeError: [00:33:55] /project/cpp/grammar_parser.cc:78: EBNF parse error at line 2, column 5: Expect rule name                                                                                                                                                   
ERROR 04-21 00:33:55 [core.py:387]                                                                                                                                                                                                                                                                      
ERROR 04-21 00:33:55 [core.py:387]                                                                                                                                                                                                                                                                      
CRITICAL 04-21 00:33:55 [core_client.py:359] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.                                                                                                                                                         
Killed                                                                                                                                                                                                                                                                                                  
stream closed EOF for s92818/vllm-deployment-openai-768966b6df-fmrwx (vllm-deployment-openai)  

🐛 Describe the bug

I'm trying to use guided grammar as found in documentation. All examples about structured outputs worked as presented but the one with EBNF. I tried to remove the tabstops in front of the lines, to rename the start node to root but it did not help.

wrong_simplified_sql_grammar = """
?start: select_statement
?select_statement: "SELECT " column_list " FROM " table_name
?column_list: column_name ("," column_name)*
?table_name: identifier
?column_name: identifier
?identifier: /[a-zA-Z_][a-zA-Z0-9_]*/
"""

prompt = ("Generate an SQL query to show the 'username' and 'email'"
          "from the 'users' table.")
completion = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B",
    messages=[{
        "role": "user",
        "content": prompt,
    }],
    extra_body={
        "guided_grammar": wrong_simplified_sql_grammar
        },
)
print(completion.choices[0].message.content)

Because I got some messages about wrong escape sequences in other examples I added an r in front of the string. But then the ? at the line start get formatted. Looking at the EBNF documentation of xgrammar made me reformulate the grammar as following and it worked:

simplified_sql_grammar = r"""
root ::= select_statement
select_statement ::= "SELECT " column_list " FROM " table_name
column_list ::= column_name (", " column_name)*
table_name ::= identifier
column_name ::= identifier
identifier ::= [a-zA-Z_][a-zA-Z_]*
"""

Naming the first node root seems to be obligatory. The / in /[a-zA-Z_][a-zA-Z_]*/ will kill the deployment as well.

I was not able to try it with the transformers locally yet. But even though it would not work their either I think the documentation for vllm might have to be adjusted.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.