Handle streams, retries and usage without duplicate work

Persist intent before opening the connection

Create a unique application operation record for each intended paid request and store its Idempotency-Key before sending the POST. Keep the authenticated account, selected model, request settings and relevant conversation reference with it. Do not log credential values or sensitive prompt contents. Your user interface should be able to reconnect to that operation record.

Send the key in the Idempotency-Key header to the documented Chat Completions endpoint. A duplicate chat request can return HTTP 409 rather than generate another answer; do not assume replaying POST is how you retrieve its result. SDK retries and proxy retries need deliberate configuration: the documentation’s Python example disables automatic SDK retries with max_retries=0. [1]

Separate deltas, tool progress and completion

With stream: true, consume server-sent events using a proper incremental parser. Network reads can split a single event or combine multiple events; a read is not an event boundary. Accumulate text deltas without treating the last received character as a completed response. Treat protocol errors and connection loss as separate states from a completed answer.

The documented stream can contain choices: [] with an execution object for hosted-tool progress. A final usage receipt can also have empty choices, so a client that skips every empty-choice event loses meaningful state. Preserve the final usage and gateway request information, including confirmed cost when supplied. Do not display an estimated cost as a settled charge. [1]

For client-owned tools, assemble tool-call argument fragments by call identity and wait for complete arguments before validating or executing. Retain the assistant call and match each result with its tool_call_id in the next conversation request. Your client owns those effects; recovery of a model request is not proof that your local tool action completed.

Inspect the original operation after interruption

Read GET /v1/requests with the original Idempotency-Key and the same account’s valid authorisation. The route requires that header and authenticates the caller; the public model catalogue does not serve this purpose. Keep inspecting the existing operation according to its returned state rather than inventing success from a timeout or retrying under a fresh paid identity. [1]

When hosted tools are involved, preserve their original run IDs. The documented GET /v1/executions/RUN_ID reads the run, and DELETE requests Stop. A Stop is confirmed when the original run reports stopped; sending the request alone does not prove that state. Inspect confirmed results and usage before reporting completion or deciding what work remains. [1]

curl https://getruthless.io/v1/requests \
  -H "Authorization: Bearer $RUTHLESS_API_KEY" \
  -H "Idempotency-Key: $ORIGINAL_REQUEST_KEY"

Test reconciliation before adding retry convenience

Use a local test server or controlled fixtures to break a stream after text, after a tool announcement and before the usage receipt. Verify that the application retains the operation identity, marks the result unconfirmed and follows its recovery path. Then test complete streams with empty-choice progress and usage events. These exercises need no paid model calls.

For a client-owned mutation, separately simulate a crash after the effect but before the tool result is stored. The executor must reconcile its own operation identity rather than execute the mutation blindly. Return the known result to the conversation only when its status is established. A newly authorised follow-up model turn is distinct from retrying an unresolved original request.

Finally compare your stored receipt with the account’s Usage & balance view when exercising the real integration under an authorised test budget. Account credit, reserved amounts and confirmed charges are different observations. If the outcome remains unavailable, state that plainly and retain the original identity for investigation; do not fabricate a zero charge or launch another paid attempt.

Review this stream consumer and request state machine. Find paths where connection loss becomes false failure, empty-choice usage is discarded, a tool executes twice or a fresh paid retry hides an unresolved original request. Propose fixtures and expected persisted state for each path. Context: [redacted consumer and executor].