Connecting DocsGPT to Cloud LLM Providers

DocsGPT is designed to seamlessly integrate with a variety of Cloud Large Language Model (LLM) providers, giving you access to state-of-the-art AI models for document question answering.

Configuration via .env file

The primary method for configuring your LLM provider in DocsGPT is through the .env file. For a comprehensive understanding of all available settings, please refer to the detailed DocsGPT Settings Guide.

To connect to a cloud LLM provider, you will typically need to configure the following basic settings in your .env file:

• LLM_PROVIDER: The provider whose models answer by default (for example openai, google, anthropic).
• API_KEY: Your API key for that provider. You can use the provider’s own variable instead, such as OPENAI_API_KEY (see Provider API keys).
• LLM_NAME (optional): The default model. It must be a model id from DocsGPT’s model catalog , such as gpt-5.5, gemini-3.5-flash or claude-sonnet-4-6, or from your own custom model YAML. A name that is not in the catalog is ignored, and DocsGPT logs a warning and uses the provider’s first model. Leave it unset to use that first model.

For example:

LLM_PROVIDER=openai
API_KEY=YOUR_OPENAI_API_KEY
LLM_NAME=gpt-5.5

Explicitly Supported Cloud Providers

DocsGPT offers direct, streamlined support for the following cloud LLM providers. The table below outlines the LLM_PROVIDER value and an example LLM_NAME (a catalog id) for each provider.

The Google provider uses a Gemini API key from Google AI Studio. Vertex AI credentials (a project, a location and a service account) are not supported.

DocsGPT also ships a model catalog (docsgpt/core/models/*.yaml) that the in-app model picker reads. It includes DeepSeek, Alibaba Qwen and Z.ai GLM models, which appear once their key is set; see the table below.

Provider API keys

Each provider’s models are registered when its key is set. API_KEY counts as the key of the provider named in LLM_PROVIDER; the other variables can be set together to offer several providers in the model picker.

LLM_PROVIDER still decides the default. With LLM_PROVIDER=openai, OPENAI_API_KEY and ANTHROPIC_API_KEY both set, both providers appear in the picker and gpt-5.5 is the default. To make a DeepSeek, Qwen or GLM model the default, set LLM_NAME to its id (for example LLM_NAME=deepseek-v4-flash).

To add a provider or model that is not in the catalog, write a model YAML and load it with MODELS_CONFIG_DIR.

Connecting to OpenAI-Compatible Cloud APIs

DocsGPT can also connect to any cloud provider that offers an OpenAI-compatible API through OPENAI_BASE_URL. Set LLM_PROVIDER=openai, the provider’s endpoint as OPENAI_BASE_URL, its key as API_KEY, and the model name as LLM_NAME. LLM_NAME is required here, because DocsGPT registers exactly the model(s) it names; separate several with commas.

Setting OPENAI_BASE_URL hides the DocsGPT and OpenAI catalogs, so only the models in LLM_NAME are offered, plus any whose provider-specific key is set.

Example for DeepSeek (OpenAI-Compatible API):

LLM_PROVIDER=openai
API_KEY=YOUR_API_KEY # Your DeepSeek API key
LLM_NAME=deepseek-v4-flash # The model name DeepSeek's API expects
OPENAI_BASE_URL=https://api.deepseek.com/v1 # DeepSeek's OpenAI API URL

DeepSeek is also in the catalog, so setting DEEPSEEK_API_KEY is usually simpler: it keeps the rest of the catalog available.

Remember to consult the documentation of your chosen OpenAI-compatible cloud provider for their specific API endpoint, required model names, and authentication methods.

The openai_compatible provider

openai_compatible is not a value for LLM_PROVIDER on its own: setting it registers nothing unless a model YAML with its key is loaded. It is the provider behind models that carry their own base_url and API key:

• catalog entries such as DeepSeek, Alibaba and Z.ai, which name their key variable (api_key_env) and endpoint in their YAML;
• your own model YAMLs in MODELS_CONFIG_DIR;
• per-user custom models that users add in Settings → Custom Models (bring your own model). Outbound requests for these use an SSRF-pinned HTTP client, so their base URL must be a public address: a server on localhost or your LAN (such as a local Ollama) is refused. Configure those as the operator, with OPENAI_BASE_URL or a model YAML, which is not checked. See Outbound network access.

OpenAI Responses API and reasoning

For OpenAI models that support it, DocsGPT can call the newer Responses API (/v1/responses) instead of Chat Completions. This is selected per model in the catalog via an api_flavor: responses capability and enables features like server-side reasoning. Related settings:

• reasoning_effort — per-model reasoning effort hint (for example medium) declared in the model catalog.
• OPENAI_RESPONSES_STORE (default false) — when true, lets OpenAI persist Responses API state server-side and chains calls with previous_response_id. When false, DocsGPT requests encrypted reasoning items and persists those ciphertext items itself so reasoning survives tool calls and later conversation turns without OpenAI retaining the response.
• OPENAI_RESPONSES_CHAIN_ACROSS_TURNS (default true) and OPENAI_RESPONSES_CHAIN_BUDGET_TOKENS (default: the model’s context_window) — in store mode a new user turn chains onto the previous response, which keeps the provider’s stored transcript and its prompt cache warm. The chain is bounded: once the previous turn’s reported prompt reaches the budget, or after the conversation history has been compressed, the next turn starts from DocsGPT’s own saved history instead. Chained tool rounds do not re-send an unchanged system message.
• OPENAI_RESPONSES_TRUNCATION_AUTO (default false) — send truncation: "auto" so the provider drops the oldest conversation items instead of failing a request that exceeds the model’s window.
• OPENAI_PROMPT_CACHE_KEY (default true) and OPENAI_PROMPT_CACHE_RETENTION (default unset) — prompt-cache hints on Responses API calls: an opaque per-user prompt_cache_key (a hash, never the user id itself), and a prompt_cache_retention value such as 24h where the provider supports it.
• V1_SESSION_TTL_SECONDS (default 86400) — lifetime of the hashed Redis mapping used to associate OpenAI-compatible client session headers with hidden DocsGPT conversations. Explicit DocsGPT conversation IDs always take precedence.

See App Configuration for the full settings reference.

Adding Support for Other Cloud Providers

If you wish to connect to a cloud provider that is not explicitly listed above or doesn’t offer OpenAI API compatibility, you can extend DocsGPT to support it. Write an LLM class in docsgpt/llm/ (the existing modules are examples) and a provider plugin in docsgpt/llm/providers/ that points at it and says how to find its API key, then add the plugin to ALL_PROVIDERS in docsgpt/llm/providers/__init__.py and list its models in a YAML under docsgpt/core/models/. See Adding Support for Other Local Engines for the two parts.