Newshot

AI Gateway now supports Private Inference

Summary

Private Inference connects your own model endpoints to AI Gateway . Enterprises running fine-tuned, self-hosted, or prerelease models used to need separate keys, client code, and logs for them. Those models can now run through the same authentication, SDKs and APIs , and observability as catalog models , so every model your team uses is managed in one place. Use Private Inference to: Switch between private and catalog models: Call private models through Chat Completions, Responses, or Messages. AI Gateway translates each request to your endpoint's format, so switching only changes the model ID. Keep models scoped to your team: Only your Vercel team can view or call private models, and they aren't listed in the public catalog. Credentials are stored encrypted, and you can enable Zero Data Retention per provider. Connect coding agents: Use private models in Claude Code, Codex, Cursor, and other coding agents connected to AI Gateway. Monitor traffic together: View private model requests a…

Original text(via Vercel Blog)

Private Inference connects your own model endpoints to AI Gateway.

Enterprises running fine-tuned, self-hosted, or prerelease models used to need separate keys, client code, and logs for them. Those models can now run through the same authentication, SDKs and APIs, and observability as catalog models, so every model your team uses is managed in one place.

Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.
Private providers and their models on the Private Inference page.

Use Private Inference to:

  • Switch between private and catalog models: Call private models through Chat Completions, Responses, or Messages. AI Gateway translates each request to your endpoint's format, so switching only changes the model ID.
  • Keep models scoped to your team: Only your Vercel team can view or call private models, and they aren't listed in the public catalog. Credentials are stored encrypted, and you can enable Zero Data Retention per provider.
  • Connect coding agents: Use private models in Claude Code, Codex, Cursor, and other coding agents connected to AI Gateway.
  • Monitor traffic together: View private model requests alongside catalog traffic in logs, trace drains, and custom reporting. Observability charts break down requests, tokens, and time to first token by private model.
  • Configure fallbacks: Optionally fall back to a catalog model if your private endpoint fails.

To get started, open the Private Inference page and add your endpoint as a private provider. Enter its base URL, select its API format, and add its credentials. Then add the models it serves.

Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.
Connecting a model endpoint to AI Gateway.

You can also register the provider and model with the Vercel CLI, version 65.16.0 or later:


        # Connect your endpoint as a private providervercel ai-gateway private-inference providers create acme-vllm \  --base-url https://vllm.example.com/v1
# Expose the endpoint's model as private/support-fine-tunevercel ai-gateway private-inference models create support-fine-tune \  --provider acme-vllm \  --api-model acme/support-llama-ft
      

Call the model using its private/<slug> ID:


        import { streamText } from 'ai';
const result = streamText({  model: 'private/support-fine-tune',  prompt: 'Draft a reply to a customer who was charged twice.',  providerOptions: {    gateway: {      // Optional fallback to a catalog model      models: ['anthropic/claude-sonnet-5.5'],    },  },});
for await (const text of result.textStream) {  process.stdout.write(text);}
      

Private model requests appear in AI Gateway logs alongside requests to catalog models.

Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.
Private and catalog model requests in AI Gateway logs, along with the detailed metadata logging the private model request.

Read the Private Inference documentation for supported model types, requirements, and billing details.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by Vercel Blog
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗