Streaming responses still need a final accounting outcome
Keep first-token latency separate from the evidence needed to settle a model request.
Streaming makes a model response useful before it is complete. The first tokens can arrive while the provider is still generating output and the final usage is still unknown. A UI that has started rendering is therefore not evidence that the ledger can already close the request.
The proxy needs to preserve the streaming experience while tracking enough state to account for the call afterward.
Read the provider’s event contract
Anthropic's streaming documentation describes a sequence of message and content events, including usage updates and possible errors. A parser needs to understand those event boundaries rather than assume that each network chunk is a complete JSON object or a complete usage record.
Network chunks can split a frame or contain several frames. An accounting parser should retain only the incomplete frame and required usage state when possible. Buffering an entire response merely to find a final usage value can add memory pressure without helping the user receive the stream sooner.
Reference: Anthropic: streaming messages
Reserve while the outcome is pending
A metered request keeps its hold during generation. Other requests should not reuse that room simply because the client has already displayed part of the response. Settlement follows the supported usage evidence or the proxy's defined fallback for an incomplete result.
Consider an illustrative response that streams for thirty seconds. Its reservation protects the budget during those thirty seconds, while its final usage may be smaller or larger than the estimate. The exact behavior for missing usage belongs in the accounting contract, not in an undocumented browser assumption.
Treat disconnects as uncertainty
Closing a browser or terminal does not prove that the provider performed no work. The request may already be running, and the proxy may need a bounded period to obtain usage or determine the final outcome. Likewise, a timeout must release resources without simply erasing the request from the ledger.
FlockTab's proxies have upstream deadlines and a documented estimate-settlement fallback for successful responses without usable usage. Known non-successful responses refund their holds. Preserve the distinction when investigating a call whose output was interrupted.
Test the stream at awkward boundaries
Build fixtures that split a usage event across chunks, send several events together, return an error after streaming begins, and disconnect before the final frame. Check that each request reaches one accounting outcome and that the parser does not block forwarding normal chunks.
Measure first-byte latency, total duration, and memory independently. A proxy can be fast at forwarding and still wrong at settlement; it can also account correctly while buffering unnecessarily. A reliable streaming path needs both timely delivery and an explainable final record.
Sources checked 23 September 2026. Numerical scenarios are illustrative unless explicitly identified as provider data. Vendor limits and interfaces can change; consult the linked documentation for current details.