Python: Ability to get rendered template or token count estimate

[nmccann]

While trying to support dynamically trimming chat history or other input parameters, I've found it useful to be able to render a template for a given function, so that it can then be passed to a tokenizer. The tokenizer's output can then be used to determine if the input should be trimmed, and then the process is repeated until it fits within the model's token limit or some other criteria is met.

However, with the Python library the only way I've found to do this ends up being quite hacky, for example:

# prompt_template is a PromptTemplateConfig used to create a function that is registered with the kernel
prompt_template = TEMPLATE_FORMAT_MAP[prompt_template_config.template_format](
        prompt_template_config=prompt_template_config
)

result = await prompt_template.render(
        kernel,
        arguments=KernelArguments(input=user_prompt, history=chat_history),
)

# Tokenize the result, and trim chat_history if necessary

I suspect the reason that such a feature doesn't exist yet is because more complex uses of SemanticKernel make it impossible to preemptively calculate the token count like this, but perhaps it would be possible to at least provide an estimate?

Note that as per the title, the feature could either return just a rendered template or a token count estimate. There are cases where the rendered template itself might be useful, but I think most of the time users would want the token count. However that may be more difficult to generalize as different users may want to use different tokenizers, so returning the template allows for that.

Thank you!

[eavanvalkenburg]

This would be a perfect scenario for the prompt rendering filter, have a look at that, and let us know! Cc @matthewbolanos

[nmccann]

@eavanvalkenburg that does sound like exactly what I'm looking for, and helped me to search the samples to find this. Thank you - I'll try that out :)

[nmccann]

For reference, I was able to get this working and it's quite a bit cleaner then my previous approach. Thank you for the suggestion!

# Other Kernel Setup...
kernel.add_filter(
            FilterTypes.PROMPT_RENDERING,
            prompt_rendering_filter(max_in_tokens, below_token_func),
        )

def prompt_rendering_filter(max_tokens: int, is_below_token_limit_func):
    async def inner(context: PromptRenderContext, next):
        await next(context)
        history: ChatHistory = context.arguments["history"]

        def _should_trim():
            input = context.rendered_prompt or ""
            return not is_below_token_limit_func(input, max_tokens) and len(history) > 0

        while _should_trim():
            history.messages.pop(0)
            await next(context)

    return inner