Skip to main content

Troubleshooting common Prompts errors (LLM auto-labeling)

Abubakar Saad
Updated

Prompts lets you connect an LLM (OpenAI, Azure OpenAI, Azure AI Foundry, Gemini, Vertex AI, Anthropic, or a custom OpenAI-compatible endpoint) and use it to draft, evaluate, and generate predictions on your project tasks. Behind the scenes, Prompts uses an inference broker (Adala) to send your tasks and prompt to the model and map the response back into your labeling configuration.

Most support issues fall into a few buckets: the feature isn't available, a project can't be selected, a model connection won't validate, or a run produces errors/empty output. This guide walks through the errors we see most often and how to resolve them.

Prompts is available in Label Studio Enterprise and Starter Cloud only. It is not available in open source Label Studio.

For background, see Prompts overview and use cases, Model provider API keys, and Draft and run prompts.

Before you start: prerequisites and constraints

Confirm the basics before troubleshooting a specific error:

  • Plan: Prompts requires Label Studio Enterprise or Starter Cloud.
  • Role: Only Owners, Administrators, and Managers can create Prompts and apply auto-annotations. Managers can only apply models to projects they are a member of. Reviewers and Annotators cannot access the Prompts tool (they can only see the predictions it produces).
  • A configured model API key and a project with a supported labeling configuration.
  • Task size / encoding: Each task must be UTF-8 compatible and no more than 1MB (roughly 200-500 pages of text).
  • Do not connect an ML backend to a project you use with Prompts. This can affect how evaluation metrics are calculated.
  • Firewall/allowlist: If you restrict outbound network access to your models, you may need to allow HumanSignal's IP ranges. See the network access row in the Prompts overview.

Prompts doesn't appear in the navigation

If you don't see Prompts in the navigation or an auto-label action on the project home page:

  1. Confirm your plan is Enterprise or Starter Cloud. Prompts is not in open source Label Studio.
  2. Confirm your role. If you are a Reviewer or Annotator, you will not see the Prompts tool at all.
  3. On SaaS, Prompts may not yet be enabled for your organization. Contact your CSM or submit a support ticket with your login URL and organization name to have it enabled.
  4. For on-prem/self-hosted Enterprise, Prompts must be installed and enabled for your deployment. See the on-prem release notes and the Install Prompts documentation.

A project isn't listed in the Target Project dropdown

A very common report: "my project doesn't appear in the Target Project dropdown" when creating a Prompt. This is almost always expected behavior, not a bug. The dropdown only lists projects whose labeling configuration is built from tags that Prompts supports.

Supported tags and data types:

  • Data types: Text and Image (Image is only supported when the file is served from cloud storage, not uploaded through the import UI).
  • Object tags: Text, HyperText, Image.
  • Control tags: Choices, Labels, TextArea, Pairwise, Number, Rating.

If your project is built around unsupported tags, it is filtered out of the dropdown. Common culprits:

  • OCRLabels and Pdf (Extraction / OCR projects): These tags are not supported. A project that uses only OCRLabels cannot be used with Prompts. If the project also includes a supported control tag (for example Choices or Labels), it can be selected, but the OCRLabels output is ignored when generating predictions.
  • ReactCode without a data attribute: A ReactCode-based config must include the data attribute to be eligible, for example <ReactCode name="react-app" toName="react-app" data="$image">. Without it, the project will not appear in the dropdown.

Resolution: Restructure the project so its config includes at least one supported object tag and one supported control tag. For documents, extract the text into a <Text> tag (or <HyperText> for HTML) and pair it with a supported control tag:

<View>
  <Text name="text" value="$text"/>
  <Choices name="category" toName="text" choice="multiple">
    <Choice value="Invoice"/>
    <Choice value="Contract"/>
    <Choice value="Other"/>
  </Choices>
</View>

Also confirm the project is not in a Personal Sandbox workspace, that you are a member of the project (required for Managers), and that it is not connected to an ML backend.

"Unable to validate connection: Invalid connection" when adding a model

When adding a model under API Keys, you may see:

Unable to validate connection: [ErrorDetail(string='Invalid connection', code='invalid')]

This means the validation request did not receive the values it expected. When using the Custom provider, all required fields must be populated, even if some are placeholders:

  1. Name / Deployment: This is the model name, not the endpoint. For an OpenAI-compatible server it must match the model the server exposes (for example, the Ollama model name, or deepseek-ai/DeepSeek-R1).
  2. Endpoint: The full endpoint URL. For OpenAI-compatible servers the v1 suffix is typically required, for example https://my.openai.endpoint.com/v1.
  3. API Key: Your actual API key. Put your credential here.
  4. Auth Token: Leave empty (or set an empty string) unless your server requires a separate server-level Authorization token.

The single most frequent cause of a failed Custom connection is putting the credential in the Auth Token field instead of the API Key field. If your validation payload shows "api_key": "", move your key into the API Key field and retry.

See Model provider API keys for the full field reference and Ollama/Hugging Face examples.

Self-hosted or OpenAI-compatible endpoint won't connect

If you host your own OpenAI-compatible model (Ollama, sglang, vLLM, a gateway, etc.), keep these requirements in mind:

  • Use the Custom provider, not the OpenAI provider. The built-in OpenAI provider expects OpenAI's own service; a self-hosted endpoint must be added as Custom.
  • The endpoint must support JSON / structured output mode. Prompts relies on OpenAI-style structured output: the API must accept response_format with type: json_object (or a JSON schema) and return schema-conformant JSON. If your server can't honor structured output, validation and runs will fail.
  • You do not need an sk--prefixed key. Earlier versions assumed OpenAI-style sk-... keys; this requirement has been relaxed, so keys in other formats are accepted.

A common self-hosted error looks like:

Unable to validate connection: ['Invalid connection: 400 {"detail":"Failed to check availability of requested model 'gpt-oss-120b': Connection error."}']

Here Label Studio reached the validation flow but the backend could not complete the connection to your model. Check the following:

  1. Verify from the backend, not just curl. Validation runs from Python inside the app pod, so a working curl doesn't guarantee success. Test with a Python requests call from inside the app pod against your endpoint (and its /models endpoint if available).
  2. TLS / certificate trust (on-prem). Python/OpenSSL is stricter than curl. If your endpoint presents a custom CA or an incomplete certificate chain, add the CA to the trust store. Inspect the chain with openssl s_client -connect YOUR-HOST:443 -servername YOUR-HOST -showcerts.
  3. Confirm the model name. The Name/Deployment value must match a model your server actually serves.
  4. Watch for provider-added path prefixes. Some OpenAI-compatible providers namespace models based on the endpoint path (for example, a /v1/openai/... path can cause the provider to look for openai/<model>). If you see Requested model 'openai/<your-model>' is not available with your api_key and settings, verify the endpoint path and that your key is authorized for that model.

ConstrainedGenerationError / "could not generate a properly-formatted response"

During a run, a task may fail with a tooltip like:

Could not evaluate: ConstrainedGenerationError — "The selected provider model could not generate a properly-formatted response."

This is raised by the inference broker when the model's raw output can't be parsed back into the structured schema derived from your labeling config (label names, spans, JSON keys). It is usually a model-side limitation in honoring structured/constrained output, not a configuration defect in Label Studio. It tends to show up on complex extraction/NER prompts and varies significantly by provider.

Things to try, in order:

  1. Switch to a provider/model with strong structured-output support. In practice, some providers struggle to consistently honor constrained schemas on complex NER/JSON even when the same prompt works in that vendor's own playground. If the errors disappear on a different model (for example an OpenAI structured-output model), that confirms the constraint is the model's structured-output handling. See Using Structured Outputs with Label Studio.
  2. Confirm the model is actually receiving your input (see the next section on 0 tokens).
  3. Reduce per-run complexity. Fewer entity types per prompt, or shorter/chunked documents. Long documents combined with many entity types is the highest-failure combination.
  4. Align your requested output with your interface. Make sure the labels/values you ask the model to produce match the control tag's classes exactly (asking for ISSUE/NON-ISSUE while the interface expects Yes/No will fail consistently).
  5. Keep output-format instructions strict and separate from your extraction/classification logic.

Empty predictions, 0 tokens, or input not reaching the model

If runs "succeed" but produce no output, or you see Total Tokens = 0 on a task, the model likely received little or no usable input. This frequently happens when a text/document variable resolves to a URL string rather than the actual content.

  1. Check what your input variable resolves to. If a variable like {document} passes a blob/storage URL string (for example https://…blob…/file.pdf) instead of inline text, the model has nothing to read. A Total Tokens = 0 on the task is the strongest signal of this.
  2. Serve real text (or a supported image from cloud storage). Extract document text into a <Text>/<HyperText> field, or for images make sure they are served from cloud storage (image input is not supported when uploaded through the import UI).
  3. Spot-check a few tasks manually to confirm the content the model saw matches the source document.
  4. Note that PDF/Extraction-style configs are not directly supported (see the Target Project dropdown section above) and are a common source of empty output.

A run hangs, freezes, or completes with 0 tasks on large datasets

For large projects, a full run may appear to hang or complete immediately with 0 completed tasks.

  1. Start small. Use the Sample Tasks (first 20 tasks) or a Ground Truths baseline to validate your prompt and connection before running against All Project Tasks.
  2. Check task size. Each task must be ≤ 1MB. Oversized tasks (very large documents) can cause failures.
  3. Re-run only what's needed. Enabling only run on tasks without predictions lets you re-run to clear transient failures without reprocessing everything.
  4. If a run stays in "running" indefinitely with no output, or completes with 0 tasks and no error, capture the Prompt ID and Project ID and submit a ticket so we can investigate the run server-side.

Predictions/credentials that work, then intermittently fail

  • Intermittent 400s from the provider. We've seen provider-side credential/quota errors (for example, transient 400 responses from OpenAI) that resolve on their own even when the key is valid and unexpired. Confirm the key is active in the provider's console, then retry. If it persists, rotate the key.
  • Multiple predictions per task. Each time you click Evaluate or Run, Prompts creates predictions for the selected baseline and overwrites previous predictions from that prompt. If you have multiple Prompts on one project, or mix Run with "Get predictions for all tasks," you will see multiple predictions per task in the Data Manager. This is expected.

Temperature and other model settings can't be changed

The Prompts UI does not currently expose LLM parameters such as temperature. Supported models are run at temperature=0 to produce deterministic, structured output. There is no UI setting to change this on SaaS. If deterministic output at a different temperature is essential to your workflow, submit a ticket as a feature request describing your provider, model, and desired value.

Submit a ticket

If none of the above resolves your issue, submit a ticket with:

  • Your plan (Enterprise SaaS, Starter Cloud, or on-prem) and login URL / organization name.
  • The Prompt ID and Project ID involved.
  • The exact error message or tooltip text, and a screenshot of the run/results if possible.
  • The model provider and model you selected (for example OpenAI gpt-5, Azure OpenAI deployment, or a Custom endpoint).
  • For Custom/self-hosted endpoints: whether you used the Custom provider, the endpoint format (you can redact the host but keep the path), and whether the credential is in the API Key or Auth Token field.

Related to