Skip to content

fix(controller): delete a ToolServer created moments ago instead of 404ing - #2991

Open
AmirF194 wants to merge 2 commits into
kagent-dev:mainfrom
AmirF194:fix/2849-delete-toolserver-before-reconcile
Open

AmirF194 wants to merge 2 commits into
kagent-dev:mainfrom
AmirF194:fix/2849-delete-toolserver-before-reconcile

Conversation

@AmirF194

Copy link
Copy Markdown
Contributor

DeleteToolServer resolves a server's kind by looking its name up in the
discovery projection (the toolserver table). That table's only writer is
the reconciler, and the reconciler runs only after it has already tried to
reach the server over the network. So a RemoteMCPServer or MCPServer
created moments ago is live in Kubernetes with no row in the projection yet:
the lookup returns an empty groupKind, and DeleteToolServer answers
NotFound for an object that is genuinely there and never gets removed.

This adds a fallback: when the projection has no row for the name, look the
object up live in Kubernetes across the kinds CreateToolServer can produce,
before answering NotFound. A name that is absent from both the projection
and Kubernetes still 404s.

This covers the "undeletable" half of #2849 only. The "invisible in the
list" half needs the bigger change the issue itself proposes, making
ListToolServers authoritative about existence by listing Kubernetes
directly and left-joining the projection for discovered tools, which is a
proto change plus a UI column, so I left it for a separate PR.

How I verified this

  • Added TestServiceDeleteToolServerBeforeReconcile, which fails on main
    (the pre-reconcile object 404s and is left undeleted) and passes on this
    branch, for both RemoteMCPServer and MCPServer; a third subtest checks
    a name absent from Kubernetes still 404s.
  • go test -race -v ./core/internal/service/tool/... passes, including the
    existing TestServiceDeleteToolServer.
  • go vet and golangci-lint run report no issues on the changed files.
  • I have not touched the UI symptom described in the issue (a newly created
    server missing from the list for a few seconds); it still needs the
    redesign above.

Refs #2849.

…04ing

DeleteToolServer resolves a server's kind by name lookup in the discovery
projection, whose only writer is the reconciler and which only runs after
it has tried to reach the server. A RemoteMCPServer or MCPServer created
moments ago is live in Kubernetes with no row in the projection yet, so
the lookup returns an empty groupKind and DeleteToolServer answers
NotFound for an object that exists and is never actually removed.

Fall back to a live Kubernetes lookup across the kinds CreateToolServer
can produce before treating an empty groupKind as nonexistence. This
covers the "undeletable" half of kagent-dev#2849 only; the "invisible in the list"
half needs the larger List-authoritative-over-Kubernetes redesign the
issue itself proposes (a proto change plus a UI column), left for a
separate change.

Refs kagent-dev#2849

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
@AmirF194
AmirF194 requested a review from a team as a code owner September 28, 2026 12:47
@github-actions github-actions Bot added the bug Something isn't working label Sep 28, 2026
if err := s.kubeClient.Get(ctx, ref, &kmcp.MCPServer{}); err == nil {
return mcpServerGVK.GroupKind().String()
}
if err := s.kubeClient.Get(ctx, ref, &corev1.Service{}); err == nil {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

createtoolserver never makes services, so this deletes any service

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, dropped the Service check entirely. CreateToolServer never creates one, so it was only ever going to match an unrelated Service sharing the name and delete that instead. Pushed in d0418ac.

if err := s.kubeClient.Get(ctx, ref, &corev1.Service{}); err == nil {
return "Service"
}
return ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rbac or timeout errors on get become a 404

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, an RBAC or timeout error here isn't the same as not found. Both Get calls now check IsNotFound explicitly and propagate anything else as an internal error. Same commit.

…t errors

liveToolServerGroupKind() had two gaps moezdil caught on review:

- It matched a bare corev1.Service by name/namespace as a fallback kind.
  CreateToolServer never creates a Service for a ToolServer, so this only
  ever matches an unrelated Service that happens to share the ref, and
  DeleteToolServer would go on to delete it. Dropped the Service check
  entirely; there is no creation-lag race to fix for a kind this path
  never creates.
- Every Get error, not just NotFound, was folded into "keep trying the
  next kind" and ultimately into a bare 404. An RBAC or timeout error now
  propagates as an internal error instead of reading as ToolServer not
  found.

Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants