Inspect the evidence. Report what happened.

BA flag: The support answer says an activated Starter customer can receive a 30-day refund. BA suspects the answer used the wrong policy.

1. Opik
   Which short token best describes what the agent answered?Choose one: eligible-30-day-refund, service-credit-only, not-refundable.
2. Opik
   Which Opik span shows the policy documents retrieved for the answer?
3. Opik
   Which retrieved document should not have been used for an HK Starter customer?
4. Code
   In shared/fixtures.py, which token describes the correct HK Starter policy for activated seats?Choose one: activated-refundable, activated-not-refundable, pro-policy-applies.

Case: 001
A:
B:
C:
D:

BA flag: This refund run was much slower than nearby runs. BA wants to know whether the answer is safe or whether the fallback path changed the evidence.

1. Opik
   Which span took the longest in this trace?
2. Opik
   What fallback path ran after the primary retrieval problem?
3. Opik
   Did the fallback use stronger or weaker evidence than the primary intended filter?Choose one: stronger-evidence, weaker-evidence.
4. Opik
   Should the manager approve this run without review?Choose one: approve, manual-review, reject.

Case: 002
A:
B:
C:
D:

BA flag: The refund calculator returned a confident "eligible" result, but the customer complained that they are on Starter, not Pro.

1. Opik
   Which short token names the spans showing the customer/request product before the tool call?Choose one: 003.request-product-spans, 003.tool-argument-span, 003.final-answer-span.
2. Opik
   Which span shows the actual tool arguments sent to the refund calculator?
3. Code
   In cases/case_003_tool_argument.py, what short token names the hard-coded argument or mapping causing the tool to receive product="pro"?Choose one: product=pro, product=starter, region=hk.
4. Opik
   What should the manager conclude: tool failure, mapping failure, or model failure?Choose one: tool-failure, mapping-failure, model-failure.

Case: 003
A:
B:
C:
D:

BA flag: The ticket summary looks incomplete. BA wants to know whether this was a model quality issue or an application persistence issue.

1. Opik
   Which span shows the streamed model output and finish reason?
2. Opik
   What finish reason did the model return?
3. Opik
   Which span persisted the answer despite the incomplete stream?
4. Code
   In cases/case_004_stream_cutoff.py, which missing check should block saving?Choose one: finish-reason-check, ticket-id-check, region-check.

Case: 004
A:
B:
C:
D:

BA flag: The GitHub issue summary includes an internal debug token and tells support to bypass approval.

1. Opik
   Which retrieved item contains the unsafe instruction?
2. Opik
   Which span shows that untrusted text was included in the prompt without a boundary?
3. Opik
   Which span shows the guardrail result?
4. Opik
   What should the manager decide for this summary?Choose one: approve, block-or-rewrite, manual-review-only.

Case: 005
A:
B:
C:
D:

BA flag: The app selected a prompt version for release, but BA believes another candidate is safer.

1. Opik
   In Logs, which prompt version did the app select?
2. Experiments
   In Experiments, which prompt version should ship based on faithfulness and release-gate scores?
3. Experiments
   Which metric made the selected candidate look attractive?
4. Code
   In cases/case_006_release_gate.py, what short token names the missing release selection condition?Choose one: faithfulness-expert-gate, cost-only-gate, answer-relevance-gate.

Case: 006
A:
B:
C:
D: