test: integration tests for the tier 1 release fixes - #703
Conversation
`--deployment-target "ABC,XYZ"` was sent to the server as a single target name because the flag is a pflag StringArray, while its legacy aliases (`--target`, `--specificMachines`) are StringSlice and already split on commas. Expand comma-separated values for the environment, tenant, tenant-tag and target flags on `release deploy` and `runbook run`, so the comma form matches the repeat-the-flag form. Values that can legitimately contain a comma (--variable, --skip, package/git-resource specs) are left alone. Fixes #556 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`release create --no-prompt` sends the create request straight to the server without resolving package versions first. When a package has no version in its feed the server raises a null reference exception, which surfaces as "Octopus API error: Object reference not set to an instance of an object. []". On a 5xx failure the CLI now repeats the package version resolution the server does, and reports the packages, steps and feeds that have no version available. Where it can't identify a specific package, an unhandled server error now carries a hint about the likely causes. Fixes #426 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`release deploy` passed --version straight to the executions API, which answers an unknown version with "Object reference not set to an instance of an object". Resolve the release before deploying so a version that doesn't exist is reported by name, and call out `latest` explicitly since it is not a supported alias. Refs #294 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…enant The executions API only matches channels, environments and tenants by name, so `release create`, `release deploy` and `runbook run` passed whatever the caller typed straight through and the server rejected IDs. `--project` already worked because the server accepts a project ID or name. Resolve those identifiers client side through the shared selectors package before handing them to the executor, preferring an ID match over a name match so it behaves the same way as `--project`. Fixes #250 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| }) | ||
| if err != nil { | ||
| return err | ||
| return DiagnoseCreateReleaseFailure(octopus, options, err) |
There was a problem hiding this comment.
Nil pointer dereference on the post-create lookup failure path (pre-existing, but this function is being touched here and the new diagnosis flow makes create failures more visible): a few lines below at the options.Response handling, when octopus.Releases.GetByID(options.Response.ReleaseID) fails, the error branch still dereferences the nil result:
newlyCreatedRelease, lookupErr := octopus.Releases.GetByID(options.Response.ReleaseID)
if lookupErr != nil {
cmd.PrintErrf("Warning: cannot fetch release details: %v\n", lookupErr)
printReleaseVersion(options.Response.ReleaseVersion, newlyCreatedRelease.Assembled, newlyCreatedRelease.ReleaseNotes, nil)ReleaseService.GetByID returns nil, err on failure, so a transient server error right after a successful create panics the CLI instead of printing the warning.
|
|
||
| // diagnosis is best-effort; if any part of it fails we must not mask the original failure | ||
| if octopus != nil && options != nil { | ||
| if missingPackages, findErr := findPackagesWithoutVersions(octopus, options); findErr == nil && len(missingPackages) > 0 { |
There was a problem hiding this comment.
The package diagnosis can misattribute an unrelated 5xx and hide the real cause. This branch runs for any 5xx, not just the null-reference case, and MissingPackageVersionsError.Error() does not include the original server message (it is only reachable via Unwrap). If the server 500s for an unrelated reason (timeout, genuine server bug) while the project happens to contain a package with no version in its feed — or the CLI's re-derived baseline disagrees with the server (e.g. a --package override the CLI silently failed to parse but the server accepted) — the user is told to push packages instead of seeing the actual failure.
Consider gating the missing-package diagnosis on strings.Contains(apiError.ErrorMessage, serverNullReferenceMessage) as well, and/or including the wrapped cause text in the error output.
| // `--flag A --flag B`. Whitespace around each entry is trimmed and blank entries are dropped. | ||
| // Only apply this to flags whose values cannot legitimately contain a comma; notably NOT to | ||
| // --variable, --skip or the package/git-resource specs. | ||
| func ExpandCommaSeparated(values []string) []string { |
There was a problem hiding this comment.
Regression for names that legitimately contain commas, with no escape hatch. Octopus allows commas in environment/tenant/machine names. Before this change --environment "Dev, East" (one env named Dev, East) worked; now it is unconditionally split into Dev + East and the deploy fails with "cannot find an environment...". The doc comment acknowledges the constraint but there is no way for a user to opt out (quoting doesn't help — the split happens after shell parsing).
The old octo CLI had the same splitting behaviour, so this may be an accepted trade-off — but worth an explicit decision and a mention in the flag help/changelog, since previously-working invocations now break silently.
|
|
||
| // FindTenant looks a tenant up by either its ID or its name. | ||
| func FindTenant(octopus *octopusApiClient.Client, tenantIdentifier string) (*tenants.Tenant, error) { | ||
| tenant, err := octopus.Tenants.GetByIdentifier(tenantIdentifier) |
There was a problem hiding this comment.
False "cannot find a tenant" for tenants beyond the first page of a partial-name search. The SDK's Tenants.GetByIdentifier name fallback (GetByName) issues Get(TenantsQuery{PartialName: name}) and scans only the first page of results for an exact match — it never pages. On a space with many tenants whose names share a common substring (e.g. dozens of "Store ..." tenants), a tenant whose exact name lands beyond page 1 returns ErrItemNotFound, and this new resolution step fails a deploy/runbook run that previously worked (the raw name used to be passed straight to the executions API, which matched it fine).
Also note GetByIdentifier uses a direct type assertion on the GetByID error, so a non-APIError failure (network blip) silently falls through to the name lookup rather than being reported.
| // resolveEnvironmentNames maps environment names or IDs onto canonical environment names, because | ||
| // the executions API only matches environments by name. Ephemeral environments aren't part of the | ||
| // regular environment list, so they're looked up separately when the regular lookup comes up empty. | ||
| func resolveEnvironmentNames(octopus *octopusApiClient.Client, space *spaces.Space, environmentIdentifiers []string) ([]string, error) { |
There was a problem hiding this comment.
A mixed list of regular + ephemeral environment identifiers can never resolve. selectors.FindEnvironments errors on the first identifier that isn't a regular environment, and the fallback findEphemeralEnvironments errors on the first identifier that isn't ephemeral — so --environment regularEnv,ephemeralEnv now fails client-side with "cannot find an environment ..." even though before this change both names were passed through verbatim for the server to judge. Probably invalid server-side anyway (one channel type per deployment), but if so the current error message points at the wrong thing: it claims the regular env doesn't exist when it does. Resolving each identifier individually (regular-then-ephemeral per item) would handle both this and give a precise error.
| idLookup := make(map[string]*environments.Environment, len(allEnvs)) | ||
| nameLookup := make(map[string]*environments.Environment, len(allEnvs)) | ||
| for _, env := range allEnvs { | ||
| idLookup[strings.ToLower(env.GetID())] = env |
There was a problem hiding this comment.
Silent precedence flip for existing callers. The old executionscommon.FindEnvironments checked the name lookup before the ID lookup; this new implementation checks ID first. Since executionscommon.FindEnvironments is now an alias to this, the change reaches all its existing callers (tenant connect, five target ... create commands, runbook, deploy): in a space where an environment is named the same as another environment's ID, those commands now resolve to a different environment than before. The tests show this is deliberate ("consistent with how projects and tenants resolve") — flagging it because it's an observable behaviour change to commands this PR doesn't otherwise touch, and may deserve a changelog note.
|
|
||
| // the executions API only matches tenants by name, so resolve any IDs we were given | ||
| if len(options.Tenants) > 0 { | ||
| selectedTenants, err := selectors.FindTenants(octopus, options.Tenants) |
There was a problem hiding this comment.
Every automation-mode invocation now pays extra HTTP round trips even when plain names were given. For each tenant supplied by name this is two requests (a guaranteed 404 on GET /tenants/<name>, then the partial-name search), plus GET /environments/all, plus the release lookup — sequentially, on every CI deploy. The same pattern is in runbook run (and there the environment/tenant resolution also runs in interactive mode, where AskQuestions resolves environments again — duplicate /environments/all calls).
A cheap win: only hit the ID lookup when the identifier actually looks like an ID (^Tenants-\d+$ / ^Environments-\d+$), or resolve tenants with a single query instead of per-tenant round trips. Related: selectors.FindEnvironment (singular) previously used a paged partial-name server query and now loads the entire space's environment list to find one environment.
| } | ||
|
|
||
| // the executions API only matches environments by name, so resolve any IDs we were given | ||
| if len(options.Environments) > 0 { |
There was a problem hiding this comment.
Altitude: the ID-to-name resolution is scattered per command, at differing depths. release create resolves the channel only in the automation branch; release deploy resolves tenants before both modes but environments only in the automation branch; runbook run resolves both unconditionally. The invariant they all enforce ("the executions API only matches by name") belongs to the layer that builds the executions-API commands (pkg/executor/release.go / the runbook executor), where one implementation would cover all three commands, both modes, and any future execution flag — instead of a pattern that has to be remembered (and is already applied inconsistently) in each command. Fine to land as-is for the tier-1 fixes, but worth a follow-up.
The package diagnosis ran for any APIError with a 5xx status. On an unrelated server error that had the side effect of (a) replacing a real server message with MissingPackageVersionsError, whose Error() doesn't include the cause, and (b) firing ~6 extra requests at a server that is already failing. Require the null reference message before diagnosing, which is the only failure this code knows how to explain. The fallback hint no longer needs its own check, since reaching it now implies the message matched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two ways the replay could diverge from what the server actually did: - With --ignore-channel-rules the server resolves package versions without applying the channel's version rules, but the replay always applied them. A package with versions in its feed, none satisfying the rules, would be reported as "no version could be found", misdiagnosing the real failure. Build the baseline without the rule filter in that case. - --channel reaches the server as ChannelIDOrName, but the lookup matched on name only, so passing a channel ID silently dropped the diagnosis to the generic hint. Match on either. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SDK's DoRawJsonRequest short-circuits on `resp.ContentLength == 0` and returns `(resp, nil)` for any status code, so DoRequest hands back a zero-valued Release with a nil error. A 404 with no body lands there, but so does a 403 with an empty body or a 502 from a proxy, and the status code is not recoverable at this layer. Reporting all of those as "cannot find a release with version X" is misleading during an outage or a permissions failure. Introduce selectors.ReleaseNotFoundError, which records whether the server confirmed the answer with a 404 carrying an APIError body, and hedge the wording when it did not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Tenants.GetByIdentifier`'s name fallback (`GetByName`) issues a single `tenants?partialName=<name>` query and scans only the first page of the result. `partialName` is a contains filter, so an exact name that sorts past a page's worth of other tenants containing the same substring - e.g. `--tenant Smith` in a space full of `... Smith` tenants - came back as `ErrItemNotFound` and failed the deploy, even though the same name worked before this branch, when it was passed through and matched server side. `selectors.FindTenant` now does the ID lookup itself and walks every page of the partial name search looking for an exact match, keeping the same ID-beats-name precedence. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback: the five-line expansion block at the top of deployRun was duplicated verbatim in runbookRun, so any new multi-value flag has to be added to two hand-maintained lists. ExpandCommaSeparatedFlags takes the flags themselves and expands them in place, leaving one call per command. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… them
Review feedback: dropping blanks let an explicitly-provided flag expand to
nothing. Because pkg/executor/release.go routes on
`len(params.Tenants) > 0 || len(params.TenantTags) > 0`, `--tenant "$A,$B"`
with both variables unset expanded to nil and the CLI silently submitted an
*untenanted* deployment to the environment. Before this branch the literal ","
was sent as a tenant name and the server rejected it. The same class of change
applied to `--exclude-deployment-target "$X"` with $X empty, where the
exclusion list quietly became empty.
A blank component always means a caller-side substitution produced nothing, so
ExpandCommaSeparated now returns an error naming the flag and quoting the
offending value. This also covers the partial case ("$A,$B" with only $B
empty), which would otherwise have silently narrowed the deployment scope.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review feedback: the split was unconditional, so a tenant/target/environment named e.g. "Foo, Inc" could no longer be passed through the primary flags at all. The sharper edge was the interactive echo — a value chosen from a picker is backfilled into resolvedFlags and flag.GenerateAutomationCmd emits it verbatim, so the printed "Automation Command" was not re-runnable: pasting it into CI would split "Foo, Inc" back into two names, erroring if they don't exist or deploying to the wrong tenants if they do. `\,` now means a literal comma. A backslash anywhere else is preserved verbatim, so names such as DOMAIN\host are unaffected. Interactive selections are escaped with executionscommon.EscapeCommas on the way into the automation command, so the echoed command round-trips. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With the release resolved before the deploy, `options.ReleaseID` is always set on both paths that reach the link: AskQuestions in interactive mode, the pre-flight lookup in automation mode (the executor rejects the deploy unless both ProjectName and ReleaseVersion are set, which is exactly when the pre-flight runs). The FindProject + GetReleaseInProject fallback can only run when the pre-flight lookup already failed, where repeating it would fail too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pre-flight lookup is new to `release deploy`; before it, the automation path never read the release and the executions API only ever saw the version string. Failing the whole deploy on any lookup error would break a CI service account scoped to deploy but not to ReleaseView, and would turn a transient 5xx on that GET into an aborted deployment that previously succeeded. Fail only on a ReleaseNotFoundError, which is the case issue #294 is about. For anything else, carry on without the release ID and let the server remain the authority on permissions and availability. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
shared.FindRelease was left as a one-line passthrough, so remove it. Doing so also puts GetReleaseID's spaceID parameter to use — it was accepted and then ignored in favour of octopus.GetSpaceID(). Both callers already pass opts.Client.GetSpaceID(), so the resolved space is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…emeral fallback with runbook run The ephemeral fallback was all-or-nothing over the whole `--environment` list: a list mixing a regular and an ephemeral environment could never resolve, because the regular lookup errored on the ephemeral name and the ephemeral lookup then errored on the regular one, leaving the user with `cannot find an environment with the ID or name of '<ephemeral env>'` - blaming an environment that exists. It also fell back on *any* error from the regular lookup, including a transport failure. `selectors.ResolveEnvironmentNames` now resolves each identifier in turn against the regular environment list, consulting the ephemeral list only for identifiers that list doesn't have (fetched once, lazily). Single-type lists behave exactly as before; mixed lists resolve, and a genuine miss names the identifier that actually went missing. `runbook run` uses the same resolver, so an ephemeral environment name that used to be passed through to the server no longer fails client side. Also flips ephemeral name/ID indexing in `findEphemeralEnvironments` so an ID match wins a collision, matching the precedence everywhere else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ccb5646 to
97602a7
Compare
ea972e2 narrowed the diagnosis to failures carrying the server's null reference message, to stop an unrelated 5xx being reported as a package problem. That works, but it also switches the fix off on current servers: the #426 path there fails with "There are no viable release plans in any channels", not a null reference, so the message the server sends for this is version-dependent and can't be relied on as the trigger. Address the underlying complaint instead. MissingPackageVersionsError now prints what the server actually said, so a misattributed diagnosis costs the user a misleading paragraph rather than the real cause, which was previously reachable only via Unwrap and never printed (main.go prints err.Error() alone). With nothing hidden, the trigger widens back to any 5xx and keeps working across server versions. The null reference message itself is still suppressed from that output -- it says nothing the diagnosis doesn't say better -- so the integration test's guard against it resurfacing stays valid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers behaviour that only a real server exercises: unknown release versions, packages with no version in their feed, channel and environment IDs on the executions API, and comma-separated deployment targets. Refs #294, #426, #250, #556 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… fixes Each of the four fixes is green on its own branch, but merged they change each other's request sequences, and the unit tests that merged cleanly are the ones that break. Nothing here is a defect in an individual PR; it is ordinary merge fallout, recorded because whichever lands last will hit it. - #294 adds a release pre-flight lookup and #250 an environments/all lookup ahead of the deployment POST. The tests added by #556, and the --priority tests that arrived on main in #708, merged without conflict and went looking for the POST, finding a GET. For the tenanted comma test the result was a hang rather than a failure: MockHttpServer blocks waiting for a request the CLI no longer makes in that order. - #250's "specifying project, environment and tenant by ID" still expected the two post-deploy web-URL lookups that #294 drops; the pre-flight now supplies the release ID, so no request follows the POST. - #556's runbook comma test needed #250's environments/all lookup, which runbook run performs unconditionally. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
97602a7 to
54072b6
Compare
Restacked onto current
|
1 participant
Adds end-to-end integration tests for the four "Tier 1" fixes — the ones whose correctness depends on how the real Octopus Server behaves, which
testutil.MockHttpServercannot validate by construction.Refs #294, #426, #250, #556.
Why this branch stacks the four fixes
The tests assert post-fix behaviour, so they need the fixes present to pass. This branch merges the four feature branches and adds the tests on top:
nj/issue-294nj/issue-426nj/issue-250nj/issue-556This branch is not for merging as-is. It exists to prove the four compose and to carry the new tests. Once the four land on
main, the last commit here rebases ontomainon its own.Tests added
All in
test/integration/release_test.go, following the existing harness (integration.RunCli,CreateCommonProject,t.Cleanupteardown).TestReleaseDeployUnknownVersion— release deploy returned error when using latest as input param for version #294. Asserts the API premise directly (an unknown version yields no usable release), then thatrelease deploynames the version it could not find, and thatlatestis explained rather than passed through.TestReleaseCreateMissingPackageVersion— Unhelpful output when attempting to create a release with a package that doesn't exist #426. Project with a package step whose package has no version in the built-in feed; asserts the failure names the package and the step.TestReleaseCreateAndDeployByID— Some arguments do not accept the ID instead of the name #250.release create --channel <Channels-N>lands on that channel;release deploy --environment <Environments-N>produces a deployment in that environment.TestReleaseDeployCommaSeparatedTargets— Support comma-delimited values on octopus release deploy --deployment-target command #556. Two cloud-region targets,--deployment-target "A,B", asserts the resulting deployment'sSpecificMachineIdscontains both.Verification
Run against a real Octopus Server (local dev instance, server
main):mainwith the same test file: all 4 fail. They are genuine regression tests, not tests that pass either way.go build ./...clean;go test ./pkg/...green (63 packages).Finding: the null-reference symptom no longer reproduces
#294 and #426 both describe
Octopus API error: Object reference not set to an instance of an object. []. On a current server that is not what happens:Octopus API error: Release 9.9.9 for project <name> was not found. []Octopus API error: There are no viable release plans in any channels using the provided arguments... Cannot resolveThe server-side defect appears to have been fixed since those issues were filed (2022.3 and 2024.4 respectively). Both CLI fixes still improve the message materially, and older servers still exhibit the original behaviour — but the premise that the server null-refs is no longer true on current versions. Two consequences:
assert.NotContains(..., "Object reference not set")lines in these tests are not load-bearing on a current server. They are kept as regression guards for older ones; the positive assertions are what carry the tests.Finding: the four fixes do not compose without test changes
Each of the four is green on its own branch, but merged they break each other's unit tests — invisible on the individual branches by construction. Fixed in the first commit here:
pkg/cmd/release/deploy/deploy_test.go, 8 conflict hunks: release deploy returned error when using latest as input param for version #294 adds a release pre-flight lookup and Some arguments do not accept the ID instead of the name #250 adds an environment lookup to the same request sequences. Resolved by expecting both, release first, matching the merged resolution order indeployRun.MockHttpServerdeadlocked waiting for a request that never came — the full unit suite hung rather than failed.release deploy specifying project, environment and tenant by IDstill expected the two post-deploy web-URL lookups that release deploy returned error when using latest as input param for version #294 removes.environments/alllookup.None of these are defects in the individual PRs; they are ordinary merge fallout. Flagging them because whichever of the four merges last will hit exactly this, and the failure mode for two of them is a hang, not a red test.
Notes
test/integrationhas no build tag andGetApiClientcallsos.Exit(999)whenOCTOPUS_TEST_URL/OCTOPUS_TEST_APIKEYare unset, so a barego test ./...from the repo root hard-exits. Run the suite fromtest/integration.allowDeploymentsTorestores the fixture lifecycle's phases on cleanup, otherwise the environment cannot be deleted.🤖 Generated with Claude Code