CLI: checkout direct from project.yaml

[josephjclark]

Short Description

This PR enables users to run a deploy from a file: either an exported project.yaml (from the app) or from A single v2 state file (the thing created when fetching a project).

openfn project deploy .projects/main@app.openfn.org.yaml --name my-new-project

Note that if you try and deploy a project to the same instance and the same UUID, it'll just override. In this case you have to pass --new or set a new name.

A side-effect of this work is that you can now edit a local project.yaml file and deploy it directly to lightning - previously the CLI didn't allow this because a deploy would re-build the project.yaml from the local filesystem. Being able to modify the yaml directly is probably a useful escape hatch. But one I'll leave undocumented for now.

Fixes #1264

Bonus fix for #1523

QA Notes

To test this ( I recommend running against local or at least staging)

You can set OPENFN_ENDPOINT and OPENFN_API_KEY env vars to configure deploy targets and access and stuff. .env files work.

Scenario 1 - deploy an exported project

• Go to any project in the app and export it as yaml
• copy the path to the file
• With a superuser API token, deploy that file to an instance: openfn project deploy downlaoded-project.yaml --name my-duplicate
• The project should exist on the target server
• Should work with whatever name you pass, or no name
• Should work on the same instance as you exported from, or a different one
• Should work with a new project, or an existing one

Scenario 2 - re-deploy an existing project

• Fetch a project: openfn project fetch <uuid> --alias og
• Note that you'll now have a file like og@app.openfn.org.yaml
• Now, with a superuser API token, deploy that file back: openfn project deploy ./projects/og@app.openfn.org.yaml --name my-duplicate
• Should work with whatever name you pass, or no name
• Should work on the same instance as you exported from, or a different one
• • Should work with a new project, or an existing one

Test Cases

This PR has crept into some sensitive areas so I want to run a really thorough suite of QA against staging:

• Do a regular change and deploy
• deploy checked out as new
• deploy checked as an update to an existing project
• export a project spec (v1 and v2?) and deploy as new
• export a project spec (v1 and v2?) and deploy to existing
• Take a project state and re-deploy as new
• Take a project state and re-deploy to existing

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

• I have used Claude Code
• I have used another model
• I have not used AI

You can read more details in our
Responsible AI Policy

[josephjclark]

ok, this is merged and caught up. Just need the integration tests and a round of manual testing and I think we're good.

[josephjclark]

It looks like if a workflow is removed, and you try to deploy to a different project, we get a "nothing to deploy" error. That might need raising as a new bug because it's too far removed from this work

Ie:

• Checkout a project
• Post it as new
• Checkout the OG
• Delete a workflow
• Post it to the new openfn project deploy new
• should be told nothing has changed

btw deleting a workflow on the OG project DID work, so not sure why this isn't stable

[josephjclark]

Ok I think this might finally be done- it raised some really tricky questions which I hope I've navigated reasonably well.

Tomorrow I'll re-do the manual test suite again before merging

[josephjclark]

Just hit some drama trying to deploy an exported project.yaml:

1. "extraneous parameters: credentials". The provisioner is rejecting any credentials being uploaded. That doens't normally happen does it? Oh - the credential key needed to be converted into project_credential. I won't why that didn't happen?

2. One edge is like x->x - it's an edge pointing to itself. That doesn't exist in the base project yaml. How did that get created?

The more changes I have to make on this PR the more nervous I get. It's been way, way more work than I antipicated.

EDIT: Hang on hang on, that's because the exported project is using the v1 format. All bets are off.

That's because the portability PR hasn't been merged in lightning. And is full of conflicts. Damn. Now I have a conundrum: do I fix support for v1 exports (which tbh shoulldn't be super hard) or do I merge and run? I think at this stage I'll raise an issue, merge and run.

Opened as #1540. I'll skip the export testing for now. I might come back.

[josephjclark]

Yet another problem. If you try and deploy a project file directly without --new, the cli will try and fetch the latest version. Which means if you edit the project file locally and try to deploy, changes will be overwritten.

We need to disable that extra fetch in load-from-file mode.

[josephjclark]

Ok I think this is ready - I'm going to merge it to the release and give the release a test on Monday.

I'm thinking about some quick fixes for credentials too, because if you have a credential this basically completely breaks