fix(gooddata-eval): persist skill_routing credit across conversation turns

[Tomkess]

Summary

_activated_skills() only looked at the current turn's tool calls, so skill_routing was recomputed from scratch every turn. set_skills keeps a skill active until it's replaced, so an agent correctly omits a redundant call on a later turn that reuses the same skill — but the scorer forced skill_routing=False whenever no skill was activated this specific turn, regardless of whether it was already active.

Found via scripts/authoring/debug_conversation.py (gdc-mic-ai-evaluation repo) replaying analyst-explores-dynamic-currency-conversion: turns t4/t5 both ran create_adhoc_visualization/create_metric successfully against an already-active skill, yet scored FAIL solely because of this.

Fix: replace-with-carry-over, not accumulate

set_skills REPLACES the active set rather than adding to it — verified against the gen-ai service's skill registry, and stated in the tool's own description. So:

if activated:
    active_skills = set(activated)
skill_routing = turn.expected_skill in active_skills

A turn that declares nothing inherits the previous turn's set; a turn that declares something drops whatever it left out.

Note: an earlier revision of this PR used a running union instead. @hkad98 caught that this only moves the bug: a skill dropped by a later set_skills call stayed credited, so a turn expecting it and declaring nothing scored PASS against a skill that was no longer active — strictly worse than the false FAIL being fixed, since it reports a broken conversation as working. Now corrected.

Reporting

skill_routing is conversation-scoped while activated_skills stays per-turn, so a reused-skill turn shows skill_routing=True alongside activated_skills=[]. That reads as a scoring bug to anyone triaging a report, so TurnResult.active_skills (also in detail["turns"][]) now carries the set the credit was actually drawn from. Documented on TurnResult.

Test plan

• test_..._skill_routing_persists_across_turns — a skill activated in turn 1 and reused without re-declaring in turn 2 is credited on both.
• test_..._skill_routing_false_when_skill_never_activated — guards against being too lenient; a skill no turn ever activates still fails routing.
• test_..._skill_routing_false_after_a_later_call_deactivates_it — the case the union version got wrong: t1 activates metric, t2 replaces it with visualization, t3 expects metric and declares nothing → correctly FAILs. Verified meaningful by re-injecting the union logic and watching this test fail, then restoring.
• Full package suite — 476 passed, 0 failed.
• ruff check / ruff format --check clean.

Known follow-up (not in this PR)

_activated_skills() reads the requested skill names from the tool call's arguments, but the service drops names it doesn't recognise and pulls in declared dependencies — and the tool's result echoes back the authoritative post-replacement set. Reading the result instead would be truer to what actually became active. Pre-existing and a behaviour change, so left separable.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 14a78872-7aa1-42e3-89da-9fa1af677a5a

📥 Commits

Reviewing files that changed from the base of the PR and between 8a7cf06 and e18efdc.

📒 Files selected for processing (2)

• packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py
• packages/gooddata-eval/tests/test_agentic_conversation.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The conversation now preserves activated skills across turns. Skill routing checks all skills activated so far. Tests cover both previously activated skills and skills that were never activated.

Changes

Persistent skill routing

Layer / File(s)	Summary
Cumulative skill tracking and validation packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py, packages/gooddata-eval/tests/test_agentic_conversation.py	The conversation accumulates activated skills. Later turns can reuse earlier activations. Tests confirm persistent routing and rejection of never-activated skills.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to e18ef

The PR persists skill activation across conversation turns while retaining the false case for skills that were never activated. No actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: hkad98, lupko, pcerny

Poem

A rabbit watched the skills take flight

One turn lit sparks, the next stayed bright
The ledger kept each skill in view
Unused skills stayed false and true
“Hop,” said the rabbit, “the route is right!”

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: persisting skill_routing credit across conversation turns.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.58%. Comparing base (8a7cf06) to head (c5215eb).
⚠️ Report is 13 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1762      +/-   ##
==========================================
+ Coverage   80.65%   81.58%   +0.93%     
==========================================
  Files         272      275       +3     
  Lines       19369    19867     +498     
==========================================
+ Hits        15622    16209     +587     
+ Misses       3747     3658      -89     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hkad98]

The bug is real and the debug provenance in the test docstring (t4/t5 of analyst-explores-dynamic-currency-conversion) is exactly the right thing to record. One question I'd want answered before this merges, plus one reporting consequence.

Is set_skills additive or replacing?

activated_skills_so_far is monotonic — it only ever grows, never forgets:

activated_skills_so_far |= set(activated)
skill_routing = turn.expected_skill in activated_skills_so_far

That's correct if the platform treats set_skills as "add these to the active set". But the name and the list-valued argument both suggest it replaces the active set. If it does, then a turn calling set_skills(["metric"]) deactivates visualization, and a later turn expecting visualization that issues no call of its own still gets credited for a skill that is no longer active. The false-FAIL being fixed here becomes a false-PASS in the mirror case.

test_run_agentic_conversation_skill_routing_false_when_skill_never_activated guards the never-activated case, which is good, but not the deactivated one — that's the case the cumulative union can't distinguish.

If replace semantics apply, the state to track is "last set wins, carried over when a turn issues no call of its own":

if activated:
    active_skills = set(activated)
skill_routing = turn.expected_skill in active_skills

The two implementations disagree on real multi-skill fixtures, so it's worth confirming with the platform side rather than inferring. If additive is confirmed, the current code is right as-is and it'd be good to say so in the comment — that's the assumption the whole fix rests on.

Reporting now self-contradicts.

TurnResult.activated_skills stays per-turn (conversation.py:402) while skill_routing became cumulative, so detail["turns"] will contain entries like:

{"turn_id": "t5", "skill_routing": true, "activated_skills": []}

which reads as a bug to anyone triaging a report — the credited skill is nowhere in the list the same record shows. Either report the set the credit was drawn from, or add a field distinguishing "activated this turn" from "active during this turn". (This is also the payload #1757 serializes, so the two PRs meet here.)

full_skill_coverage (conversation.py:415) is unaffected — it already unions activated_skills across all turns.

[Tomkess]

@hkad98 good catch, and I don't want to guess on this one. The additive-vs-replacing question determines whether the current fix is correct or just moves the bug — I can't verify set_skills's real semantics from this repo alone, that needs confirmation from someone who knows the platform/gen-ai side. Holding off on changing the accumulation logic until that's answered, rather than picking one interpretation and hoping.

Once that's settled: if additive is confirmed, I'll add a comment here stating that assumption explicitly (agreed, the code should say why it's safe, not just that it works). If it's replacing, I'll switch to the "last set wins, carried over when a turn issues no call" version you sketched.

The reporting self-contradiction (skill_routing: true alongside activated_skills: []) is real either way, but I'd rather fix it once, after the semantics question lands — the right shape (report the cumulative set the credit was drawn from, vs. a separate "active during this turn" field) depends on which model turns out to be correct, and it also touches #1757's serialization, so I don't want to patch it twice.

[hkad98]

Agreed — holding is the right call here, and I'd rather have this open pending an answer than merged on a guess. Nothing further from me on the accumulation logic until the semantics land.

Two things to make the answer easier to get, since whoever knows set_skills can settle both in one pass.

1. The actual question to ask. Concretely: after set_skills(["metric"]) on turn 3, is visualization (set on turn 1) still active on turn 4? Additive → this PR is already correct. Replacing → the "last set wins, carried over" version. There's also a third possibility worth ruling out explicitly: that activation is per-request and the platform re-derives it each turn, in which case neither model is right and the original per-turn check was closer than it looked.

2. A related inconsistency to confirm at the same time — the argument key. _activated_skills hedges across two spellings:

skills.extend(args.get("skill_names") or args.get("skills") or [])

but the visualization evaluator only reads one:

# evaluators/visualization.py:78
skill_names = args.get("skill_names", [])

Our own tests use both — test_visualization_evaluator.py:96 and test_agentic_visualization.py:42 use skill_names, test_agentic_conversation.py:381 uses skills. So one of these is wrong about the real payload, and if the live key is skills, then _check_visualization_skill_activated silently returns False for every visualization item and that evaluator has been under-reporting skill activation. Pre-existing and out of scope here, but it's the same question to the same person, and it'd be good to know which spelling is real before more code hedges across both.

Also fine to fix the skill_routing: true / activated_skills: [] reporting contradiction once afterwards rather than twice — the right shape does depend on which model wins, and #1757 serializes it.

[Tomkess]

@hkad98 addressed the reporting half in 4507fcb5.

The two fields measure different scopes on purpose, and that wasn't written down anywhere — which is why the output reads as a bug:

• activated_skills — what this turn declared. Empty whenever the agent reused an already-active skill.
• skill_routing — whether expected_skill was active by this point in the conversation.

So skill_routing=True + activated_skills=[] is the reused-skill case, not a scoring error. Documented on TurnResult and at the computation site, plus an assertion on the existing persistence test (activated_skills == ["metric"] on t1, == [] on t2) so the combination is pinned as intended output rather than something a future reader "fixes".

I deliberately did not change what activated_skills contains. Whether the cumulative set is the right thing to report depends on the additive-vs-replacing answer: if additive, it genuinely is "active during this turn"; if replacing, it's a superset including deactivated skills and reporting it would be worse than the current confusion. This makes the output legible under either answer without pre-committing.

Still need that answer before touching the accumulation logic or the field's contents — that's the one thing here I can't settle from the repo. 475 passed, lint clean.

[Tomkess]

@hkad98 checked the gen-ai service implementation — it replaces, it does not accumulate. Your suspicion was right and the original fix here was wrong.

So the running union produced exactly the false PASS you described. Concretely, with the old code: t1 set_skills(["metric"]), t2 set_skills(["visualization"]) — metric now off — t3 expects metric and declares nothing → union still contained metric → PASS on a skill that wasn't active. Worse than the false FAIL this PR set out to fix, since it reports a broken conversation as working.

Fixed in 287e1de3 with your sketch:

if activated:
    active_skills = set(activated)
skill_routing = turn.expected_skill in active_skills

The original false-FAIL fix still holds — a turn reusing an already-active skill without re-declaring keeps its credit; only turns whose skill was actively replaced lose it.

Also took the second option you offered on reporting: added TurnResult.active_skills (and detail["turns"][].active_skills) carrying the set the credit was drawn from, so skill_routing=True beside activated_skills=[] is self-explanatory.

New test ..._skill_routing_false_after_a_later_call_deactivates_it covers the deactivation case. I verified it's meaningful by re-injecting the union logic and watching it fail, then restoring. 476 passed, lint clean.

One thing I noticed while checking, not fixed: _activated_skills() reads the requested skill names from the tool call's arguments, but the service drops names it doesn't recognise and pulls in declared dependencies — and the tool's own result echoes back the authoritative post-replacement set. Reading the result instead would be truer to what actually became active. Pre-existing, separable, and a behaviour change, so I left it — happy to file it if you think it's worth its own PR.

(Ping me directly if you want the specific implementation references I checked — keeping them out of this repo's comments.)