[backport -> release/3.6.x] feat(plugins): ai-transformer plugins

[team-gateway-bot]

Automated backport to release/3.6.x, triggered by a label in #12341.

Original description

Summary

Adds two plugins that introspects AI requests and responses against the same, or separate, LLM services.

## AI Request Transformer

Uses a configured LLM service to introspect / transform the consumer's request body, before proxying upstream.

Works like so:

• Kong admin sets up an "llm:" configuration block, following the exact same schema format as "ai-proxy" plugin, and the same "driver" capabilities
• Kong admin sets up a "prompt" for the request introspection, that becomes the "system" message in the LLM chat request
• User makes HTTP(S) call - before proxying to the backend, the entire request body is set as the "user" message in the LLM chat request, and then sent to the configured LLM service
• The response "assistant" message returned from the LLM, is subsequently set as the upstream request body

This ai-request-transformer plugin runs AFTER all of the "ai-prompt-*" plugins, allowing it to also introspect LLM requests against... a different LLM!

Here is a sample deck config:

_format_version: "3.0"

services:
- name: httpbin
  host: httpbin.org
  path: "/anything"
  port: 443
  protocol: https
  routes:
  - name: httpbin-GET
    paths:
    - "~/httpbin$"
    methods:
    - POST

plugins:
- name: "ai-request-transformer"
  config:
    prompt: "You will respond with the same JSON format, but with any credit card numbers masked out."
    llm:
      auth:
        header_name: "Authorization"
        header_value: "{vault://env/OPENAI_AUTH_HEADER}"  # this will be "Bearer XYZ"
      route_type: "llm/v1/chat"
      logging:
        log_statistics: true
        log_payloads: false
      model:
        provider: "openai"
        name: "gpt-4"
        options:
          max_tokens: 1024
          temperature: 1.0

AI Response Transformer

Uses a configured LLM to introspect the upstream's HTTP(S) response, before sending back to the client.

Works like so:

• Kong admin sets up an "llm:" configuration block, following the exact same schema format as "ai-proxy" plugin, and the same "driver" capabilities
• Kong admin sets up a "prompt" for the response introspection, that becomes the "system" message in the LLM chat request
• User makes HTTP(S) call - after proxying to the backend, the entire response body is set as the "user" message in the LLM chat request, and then sent to the configured LLM service
• The response "assistant" message returned from the LLM, is subsequently set as the client response body
• The plugin returns early (kong.response.exit) and can handle gzip / chunked requests. The code for this is copied from Kong plugin "forward-proxy"

The is also an extra feature in ai-response, whereby if you ask the LLM to respond in this format:

{
  "headers":
    {
      "new-header": "new-value"
    },
  "status": 201,
  "body": "new response body"
}

then Kong will parse these instructions and set all response headers, set response status code, and set replacement response body, based on this output. This allows changing e.g. Content-Type, or throwing errors (/ stop words) from the LLM.

This ai-response-transformer plugin runs AFTER the "ai-proxy" plugin, allowing it to also introspect LLM responses against a different LLM!

Here is a sample deck config:

plugins:
- name: "ai-request-transformer"
  config:
    prompt: "You will convert my message to XML. Respond ONLY with the XML conversion output."
    llm:
      auth:
        header_name: "Authorization"
        header_value: "{vault://env/OPENAI_AUTH_HEADER}"
      route_type: "llm/v1/chat"
      logging:
        log_statistics: true
        log_payloads: false
      model:
        provider: "openai"
        name: "gpt-4"
        options:
          max_tokens: 1024  # You need WAY more tokens for response parsing, based on its Content-Length
          temperature: 1.0

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - Docs being worked on internally.

Issue reference

Internal project.

[team-gateway-bot]

Cherry-pick failed for release/3.6.x: couldn't find remote ref release/3.6.x.
Please ensure that this Github repo has a branch named release/3.6.x.

[team-gateway-bot]

Cherry-pick failed for release/3.6.x, because it was unable to create a new branch.

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream release/3.6.x
git worktree add -d .worktree/cherry-pick-12426-to-release/3.6.x-to-upstream upstream/release/3.6.x
cd .worktree/cherry-pick-12426-to-release/3.6.x-to-upstream
git checkout -b cherry-pick-12426-to-release/3.6.x-to-upstream
ancref=$(git merge-base afae9d0b8be35ac90256015f0a6a607ff28ba75d 35ae4ba3b3ff6190e54d4cf76392b8b13ef12c3e)
git cherry-pick -x $ancref..35ae4ba3b3ff6190e54d4cf76392b8b13ef12c3e