Add Remote LLM Support for Perturbation-Based Attribution via RemoteLLMAttribution and VLLMProvider

[saichandrapandraju]

This PR introduces support for applying Captum's perturbation-based attribution algorithms to remotely hosted large language models (LLMs). It enables users to perform interpretability analyses on models served via APIs, such as those using vLLM, without requiring access to model internals.

Motivation:

Captum’s current LLM attribution framework requires access to local models, limiting its usability in production and hosted environments. With the rise of scalable remote inference backends and OpenAI-compatible APIs, this PR allows Captum to be used for black-box interpretability with hosted models, as long as they return token-level log probabilities.

This integration also aligns with ongoing efforts like llama-stack, which aims to provide a unified API layer for inference (and also for RAG, Agents, Tools, Safety, Evals, and Telemetry) across multiple backends—further expanding Captum’s reach for model explainability.

Key Additions:

• RemoteLLMProvider Interface:
  A generic interface for fetching log probabilities from remote LLMs, making it easy to plug in various inference backends.
• VLLMProvider Implementation:
  A concrete subclass of RemoteLLMProvider tailored for models served using vLLM, handling the specifics of communicating with vLLM endpoints to retrieve necessary data for attribution.
• RemoteLLMAttribution class:
  A subclass of LLMAttribution that overrides internal methods to work with remote providers. It enables all perturbation-based algorithms (e.g., Feature Ablation, Shapley Values, KernelSHAP) using only the output logprobs from a remote LLM.
• OpenAI-Compatible API Support:
  Used openai client under the hood for querying remote models, as many LLM serving solutions now support the OpenAI-compatible API format (e.g., vLLM OpenAI server and projects like llama-stack(see here for ongoing work related to this).

Issue(s) related to this:

• Captum with vLLM inference engine #1529

[facebook-github-bot]

Hi @saichandrapandraju!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at [email protected]. Thanks!

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[saichandrapandraju]

Hi @vivekmig @yucu @aobo-y could you check this PR please. Let me know if I have to change anything

[aobo-y]

Thank you @saichandrapandraju for the great effort! Generally, i agree this idea makes a lot of sense.
But our team may need some time to look into the code changes and get back to you.

@craymichael can you take a look at it? since you have studied the integration with llama-stack before

[saichandrapandraju]

Thank you for the positive feedback @aobo-y ! Happy to hear the direction makes sense. Please take your time reviewing — I’ll be around to clarify or iterate on anything as needed. Looking forward to it!