LLM Provider Error Classification

Summary

Agent tooling must decide, from an HTTP response alone, whether an LLM provider rejected a request because of a bad credential, a dead endpoint, a context overflow, or some other error. A probe run on yoyo (Day 254 build) shows that neither the status code alone nor a loose substring match on an error message is sufficient: nine providers return 401 for a junk key, but Google and xAI return 400 with the reason buried in the body, GitHub Models now returns a healthy-looking 200 text/plain from a fully retired endpoint, and an over-broad context-overflow phrase list misclassifies an Azure OpenAI rate limit as a context overflow.

Key Points

• Status-code-only auth classification is wrong for two major providers. Google and xAI signal an invalid API key with 400, not 401; the signal lives in the body (reason: API_KEY_INVALID / code: invalid-argument).
• A 2xx is not proof of a live endpoint. Every path on models.github.ai returns 200, content-type: text/plain, body OK — with or without auth — because GitHub Models was retired on 2026-07-30. An SSE client surfaces this as an invalid-content-type complaint, not an error.
• GitHub Copilot is not GitHub Models. api.githubcopilot.com is a separate, live product with different auth; conflating them sends tools to the dead endpoint (see marimo issue #10726).
• Substring matching on overflow phrases needs to be specific. yoagent 0.19.0 narrowed its Gemini overflow phrase from "exceeds the maximum" to "exceeds the maximum number of tokens" after Azure OpenAI's peak-load rate-limit message was misread as context overflow.
• Copied phrase lists drift. Hand copies of the old, broad phrase list keep the bug even after the library is fixed.

Concepts