Trace debugging lab

Inspect the evidence. Report what happened.

Six support-agent runs were flagged before release. Use Opik traces, Experiments, and a small Python codebase to prepare a concise report for your manager.

Enter your assigned group from group-00 to group-19. Opens in a new tab.
View questions How to submit
Debugging brief 6 cases / 24 answers
  1. 1
    Choose a caseOpen its trace in Opik.
  2. 2
    Follow the spansFind the named evidence.
  3. 3
    Check marked codeOnly four answers need Python.
  4. 4
    Submit a commentEach comment updates your issue score.

Investigation queue

Workshop questions

Most answers are visible in Opik. Open Python only when a question carries the Code label.

Before Case 001

Start in the workshop traces

Opik may open a default project or a different Logs tab. Make these two selections before investigating a case.

  1. Opik project menu with HKPUG Mini Workshop selected above Default Project
    1
    Choose HKPUG Mini Workshop Open the Project menu and select the workshop project. Do not use Default Project.
  2. Opik Logs page with the Traces tab selected between Threads and Spans
    2
    Open Logs, then Traces Select Traces, not Threads or Spans, to find the six workshop cases.
001 Policy evidence does not match customerWrong policy retrieval / 4 questions

BA flag: The support answer says an activated Starter customer can receive a 30-day refund. BA suspects the answer used the wrong policy.

  1. Opik
    Which short token best describes what the agent answered?Choose one: eligible-30-day-refund, service-credit-only, not-refundable.
  2. Opik
    Which Opik span shows the policy documents retrieved for the answer?
  3. Opik
    Which retrieved document should not have been used for an HK Starter customer?
  4. Code
    In shared/fixtures.py, which token describes the correct HK Starter policy for activated seats?Choose one: activated-refundable, activated-not-refundable, pro-policy-applies.
Case: 001
A:
B:
C:
D:
002 Slow run with broad retrieval fallbackLatency and evidence quality / 4 questions

BA flag: This refund run was much slower than nearby runs. BA wants to know whether the answer is safe or whether the fallback path changed the evidence.

  1. Opik
    Ignoring the root orchestration span, which retrieval span took the longest?
  2. Opik
    Which fallback retrieval span ran after the primary retrieval problem?
  3. Opik
    Did the fallback use stronger or weaker evidence than the primary intended filter?Choose one: stronger-evidence, weaker-evidence.
  4. Opik
    Should the manager approve this run without review?Choose one: approve, manual-review, reject.
Case: 002
A:
B:
C:
D:
003 Tool result is confident but input is wrongTool-call argument mapping / 4 questions

BA flag: The refund calculator returned a confident "eligible" result, but the customer complained that they are on Starter, not Pro.

  1. Opik
    Which span first shows the parsed request product as starter before the tool call?Choose one: 003.parse_refund_request, 003.map_tool_arguments, 003.compose_tool_answer.
  2. Opik
    Which calculator span receives product: pro in its input?
  3. Code
    In cases/case_003_tool_argument.py, what short token names the hard-coded argument or mapping causing the tool to receive product="pro"?Choose one: product=pro, product=starter, region=hk.
  4. Opik
    What should the manager conclude: tool failure, mapping failure, or model failure?Choose one: tool-failure, mapping-failure, model-failure.
Case: 003
A:
B:
C:
D:
004 Streamed draft was persisted after cutoffIncomplete generation and persistence / 4 questions

BA flag: The ticket summary looks incomplete. BA wants to know whether this was a model quality issue or an application persistence issue.

  1. Opik
    Which span shows the streamed model output and finish reason?
  2. Opik
    What finish reason did the model return?
  3. Opik
    Which span persisted the answer despite the incomplete stream?
  4. Code
    In cases/case_004_stream_cutoff.py, which missing check should block saving?Choose one: finish-reason-check, ticket-id-check, region-check.
Case: 004
A:
B:
C:
D:
005 Issue summary contains unsafe comment textUntrusted retrieved context / 4 questions

BA flag: The GitHub issue summary includes an internal debug token and tells support to bypass approval.

  1. Opik
    Which retrieved item contains the unsafe instruction?
  2. Opik
    Which span shows that untrusted text was included in the prompt without a boundary?
  3. Opik
    Which span shows the guardrail result?
  4. Opik
    What should the manager decide for this summary?Choose one: approve, block-or-rewrite, manual-review-only.
Case: 005
A:
B:
C:
D:
006 Release candidate looks good but gate is weakExperiment metrics and selection / 4 questions

BA flag: The app selected a prompt version for release, but BA believes another candidate is safer.

  1. Opik
    In Logs, which prompt version did the app select?
  2. Experiments
    In Experiments, which prompt version should ship based on faithfulness and release-gate scores?
  3. Experiments
    Which metric made the selected candidate look attractive?
  4. Code
    In cases/case_006_release_gate.py, what short token names the missing release selection condition?Choose one: faithfulness-expert-gate, cost-only-gate, answer-relevance-gate.
Case: 006
A:
B:
C:
D:

Manager report

One issue. Answers go in comments.

Create the group issue once. After that, every answer and retry is a new comment on the same issue.

  1. 1
    Create the group issue

    Choose Mini workshop answers, enter your group, and create one issue.

  2. 2
    Post an answer comment

    Paste one case template into Add a comment. Partial submissions are allowed.

  3. 3
    Read the cumulative result

    The bot marks each item pass, check again, or not submitted.

GitHub Mini workshop answers form showing the title, group field, answer format, and Create button
Enter your assigned group and click Create. The issue body is setup only; grading starts when you post a comment.

Comment format

Submit only what you have answered

Use exact tokens from the questions. Leave unknown fields blank or omit them. Do not add explanations inside answer fields. 

Case: 001
A: <short-exact-answer>
B:
C:
D:
  • One answer is enough. You can submit only A and return to the other fields later.
  • Progress accumulates. Correct answers from earlier comments remain in your issue score.
  • Retry with a new comment. Do not edit an old comment; edited comments are not graded again.
  • Latest non-empty answer wins. A new value replaces the previous value for that item.

Ready to start? Keep the new issue open for your group for the whole workshop.

Open submission issue