Test AI Stream Failure Before Saving the Answer

A streaming AI answer is complete only when the application has evidence of successful completion and the result passes its own contract checks. Receiving text is insufficient. For QA, the useful failure test is to interrupt the stream after plausible content arrives and verify that neither the interface nor the database labels it a finished answer.

This extends Test AI Latency from First Token to Finished Answer. That lesson separates responsiveness from completion time. This one tests the transition from provisional text to accepted output. The workflow, identifiers, and expected outcomes below are synthetic test fixtures, not production measurements or employer data.

Where the false success enters

Consider a release-note assistant. A browser sends a request to an application gateway, which calls a model provider. The gateway forwards incremental text to the browser and stores the final note when generation succeeds. That path has several observers: the provider adapter, gateway, browser, and persistence layer.

The proposed defect is a handler that marks success whenever its read loop ends. It treats an exhausted stream like a completed document. If the connection closes after the first paragraph, the user sees a polished fragment with a finished badge. The stored note contains the same fragment, and a dashboard records a successful request.

This is a realistic protocol boundary to test. Anthropic's streaming documentation describes errors delivered as events inside an already established stream. OpenAI's Node SDK streaming guide distinguishes completed, failed, and incomplete response events, and discusses a stream ending without a terminal event. Neither a transport connection nor visible text settles the application outcome.

Do not copy event names between providers. Implement the adapter against the specific API and SDK version under test, then translate its evidence into application states with defined meanings.

Define the acceptance boundary

For this exercise, use five application states: receiving, accepted, failed, incomplete, and unconfirmed. These are proposed product states, not a universal provider vocabulary.

Receiving allows provisional text to appear. Accepted requires both a successful terminal result and a complete release note that satisfies the fixture contract. Failed records an explicit provider or application failure. Incomplete records an explicitly unfinished generation. Unconfirmed means delivery stopped without enough evidence to classify the provider outcome.

A lost connection does not establish that generation stopped upstream. If the integration supports retrieving the response later, reconciliation may resolve the unconfirmed state. Until then, retain uncertainty. Do not silently convert it into acceptance.

The synthetic contract requires three sections: Changes, Risks, and Verification. All three must be present and nonempty before persistence as an accepted note. A completed provider response with only Changes therefore fails application validation. Completion and content validity remain separate assertions, as discussed in A Valid JSON Response Can Still Be a Bad Contract.

Build one controlled stream

Create a local provider stub and send requests through the actual gateway and browser integration. Use request R41 and attempt A1. The stub emits a valid opening sequence, then text for Changes, then pauses at a test-controlled barrier. Assert that the browser displays provisional content while the accepted-note store remains empty.

Release the barrier into one of the endings below. Reset the fixture between cases. The body before the barrier must remain identical, so the ending is the variable under test.

These rows are assertions to implement, not results from a completed experiment. Use the provider-specific wire format in the stub and exercise the same parser used in deployment. A mock that returns an already classified application result bypasses the adapter where this defect could live.

Observe more than the badge

For every row, inspect the rendered status, persisted record, and completion metric. In this fixture, only the first row may increment accepted-answer count. Keep attempt identifiers alongside the logical request identifier so an eventual recovery does not erase the failed attempt.

Make the broken behavior explicit as a negative control: temporarily change the handler so any end of stream promotes accumulated text to accepted. The abrupt-close row must fail because it creates an accepted record. Restore the correct handler and require that row to pass. If both versions pass, the test is observing the wrong boundary or never delivered the intended interruption.

Also run the success row through the same barrier. A handler that rejects every stream can pass all the failure assertions while making the product unusable. The successful baseline proves that the acceptance path still works.

For a retry exercise, create attempt A2 under R41 with visibly different fixture text. Verify that A1 fragments cannot appear inside A2's accepted note. Partial text may remain available for diagnosis, but the proposed contract never concatenates separate generations into a single accepted result.

What this test can establish

A deterministic stub tests how your integration handles specified events. It does not establish how often a provider will disconnect, whether a real proxy buffers traffic, or whether every SDK failure takes the same path. Follow it with a controlled integration check through the deployed gateway when that environment is available, and keep those observations separate from synthetic expectations.

The release evidence should include the fixture sequence, adapter version, observed terminal state, validation result, and accepted-record count. Preserve the interruption point so another engineer can reproduce the transition.

In your client, what exact event allows provisional text to become a saved, completed answer?