Skip to content

test: cover greeting to deferred email tool continuation - #218

Merged
senamakel merged 2 commits into
tinyhumansai:mainfrom
senamakel:transcript-hi-e2e-live
Sep 25, 2026
Merged

senamakel merged 2 commits into
tinyhumansai:mainfrom
senamakel:transcript-hi-e2e-live

Conversation

@senamakel

Copy link
Copy Markdown
Member

Summary

  • Add an end-to-end agent-loop regression for a greeting followed by an email request, deferred tool discovery, Gmail execution, and a final answer.
  • Make the scripted model require the discovered tool schema and active request in the same final user turn, reproducing the prompt split found during live validation.
  • Pin tinyinference with the prompt continuation correction.

Dependency

Depends on tinyhumansai/tinyinference#30. Keep this PR draft until that gitlink is available upstream.

Live validation

A scratch-profile OpenHuman run against managed DeepSeek V4 Flash still sometimes returned the earlier greeting even with the correct email request on the backend wire. The end-to-end test proves the local loop behavior, not provider reliability. A supported Qwen3.5 Flash run completed the same two-turn scenario and invoked GMAIL_FETCH_EMAILS. The production model/routing choice is being handled separately.

Verification

  • cargo test --manifest-path Cargo.toml -p tinyagents-integration-tests --test e2e_tool_dialects (23 passed)
  • cargo test --manifest-path Cargo.toml -p tinyagents-harness agent_loop::dialect::test --lib (6 passed)
  • cargo fmt --manifest-path Cargo.toml --all --check

@coderabbitai

coderabbitai Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

  • Run on-demand review

This review includes 3 billable files and costs up to $0.75.

Or wait 29 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: b9827a76-2aca-45c6-989c-d016e447f760

📥 Commits

Reviewing files that changed from the base of the PR and between 270fb82 and 081355f.

📒 Files selected for processing (3)
  • crates/tinyagents-harness/src/agent_loop/dialect/test.rs
  • crates/tinyagents-integration-tests/tests/e2e_tool_dialects.rs
  • vendor/tinyinference

Comment @coderabbitai help to get the list of available commands.

@senamakel
senamakel marked this pull request as ready for review September 25, 2026 12:49
@tinysweeper

tinysweeper Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

Tiny Sweeper review

This pull request adds an end-to-end regression test for a greeting followed by a deferred email tool invocation, and adjusts an existing dialect test to use `last()` instead of an indexed access. The new test validates that a deferred email tool is invoked after a tool search when preceded by a greeting, but it is missing a search tool registration which may leave the first tool call unhandled.

State: Changes requested
Priority: high
Reviewed head: 081355fb5262
Updated: 1790340767 (Unix time)

Review snapshot

Change surface Files Review signal Count
Production 0 Active findings 1
Tests 2 Noted findings 0
Documentation 0 Resolved findings 0
Configuration 0 Pending checks/questions 0

Completeness: Complete
Test assessment: No supported feature-to-test mapping was available; this does not mean tests are absent or passed.

What changed

Modified `crates/tinyagents-harness/src/agent_loop/dialect/test.rs` to use `last()` instead of indexing for assertion. Added new test `a_greeting_does_not_replace_the_email_request_after_tool_search` in `crates/tinyagents-integration-tests/tests/e2e_tool_dialects.rs` that simulates a conversation with greeting and deferred email tool.

Features

None identified with supported citations.

Tests

No supported feature-to-test mapping was produced. Test execution is not inferred.

Findings

  • high · description · Register a search tool to handle the mock model's first tool call — The `EmailContinuationModel` returns `<tool_call>tool_search(query="fetch emails from Gmail")</tool_call>` on its first invocation, but no tool with that name is registered on the (\(pull request description\))

Before merge

  • Address Register a search tool to handle the mock model's first tool call (\(pull request description\)).

How this fits together

flowchart LR
  n0["...latest_request_after_a_tool_search_result<br/>changed"]:::changed
  n1["harness_with<br/>changed"]:::changed
  n2["apply_to_request"]:::impacted
  n3["new"]:::impacted
  n4["CaptureMiddleware"]:::impacted
  n0 -->|calls| n2
  n0 -->|tests| n2
  n1 -->|calls| n3
  n1 -->|uses| n4
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading
Agent review details

critique

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: Reviewed 2 files; 0 findings. _The code index is behind this pull request (indexed at `981e9fb9e238`), so retrieved context may be out of date._ _Memory was unavailable (model: cortex: v1/recall: timed out after 10s), so this review ran without it._

security

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: Reviewed 2 files; 0 findings. _The code index is behind this pull request (indexed at `981e9fb9e238`), so retrieved context may be out of date._ _Memory was unavailable (model: cortex: v1/recall: timed out after 10s), so this review ran without it._

tests

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Lane summary: No behavioural change: nothing outside documentation, configuration and tests.

commits

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Lane summary: Nothing sensitive found in what this pull request commits.

description

  • Conclusion: Failure
  • Scope reviewed: all assigned evidence
  • Lane summary: The pull request adds an end-to-end regression test for a greeting followed by a deferred email tool invocation, and adjusts an existing dialect test to use `last()` instead of an indexed access. The new test is missing a search-tool registration, so the mock model's first tool call for `tool_search` will be unhandled, making the test scenario incomplete and potentially giving false confidence. The change in the existing dialect test alters the assertion target (from second-to-last to last message) without a clear justification; it may be intentional but the risk is low. The remaining changes appear sound. _The code index is behind this pull request (indexed at `981e9fb9e238`), so retrieved context may be out of date._ _Memory was unavailable (model: cortex: v1/recall: timed out after 10s), so this review ran without it._
  • Evidence: \(pull request description\) — Register a search tool to handle the mock model's first tool call

e2e

  • Conclusion: Neutral
  • Scope reviewed: all assigned evidence
  • Lane summary: No behavioural change: nothing outside documentation, configuration and tests.
Evidence and run details
  • Models: ladder/vectors, gpt-5.6-luna, deepseek/deepseek-v4-flash
  • Spend: $0.005586
  • Tokens: 158211 input · 11359 output · 9692 cached · 370 embedding
Head State Pass summary
081355fb5262 changes requested 1 active finding(s), 0 resolved finding(s) (at 1790340767)

tinysweeper 0.1.0

@senamakel
senamakel merged commit 2d5f835 into tinyhumansai:main Sep 25, 2026
10 checks passed

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: 1 lane(s) blocking, worst finding is high.

Fix or reply to the findings below and push. The next review clears this automatically once they are gone — you should not need to dismiss anything by hand.

             $0.0056 · 158,211 in / 11,359 out · 9,692 cached (6%) · ladder/vectors, gpt-5.6-luna, deepseek/deepseek-v4-flash · 370 embedded
critique:    $0.0024 · 92,633 in  / 3,723 out  · 6,108 cached (7%) · gpt-5.6-luna
security:    $0.0013 · 58,041 in  / 1,287 out  · 3,584 cached (6%) · gpt-5.6-luna
description: $0.0011 · 4,557 in   / 3,578 out  · 0 cached (0%)     · deepseek/deepseek-v4-flash

@tinysweeper tinysweeper Bot added the priority: p1 Next. Wrong behaviour a user will hit, or a security weakness behind a condition. label Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p1 Next. Wrong behaviour a user will hit, or a security weakness behind a condition.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant