Apollo Server

Apollo is OpenFn's knowledge, AI and data platform, providing services to support the OpenFn toolchain.

Apollo is known as the God of (among other things) truth, prophecy and oracles.

This repo contains:

• A bunjs-based webserver
• A number of python-based AI services
• A number of Typescript-based data services

Documentation

This README covers running, debugging and deploying the server. Deeper docs live alongside the code:

• Architecture - see Server Architecture below, and each service's own README for service-specific detail.
• Contributing & service conventions - CONTRIBUTING.md explains how to add and structure a Python service (entry.py, imports, logging, code quality).
• Services - every service has its own README with payload specs and examples. See the Services index below.
• Instance auth - platform/src/auth/README.md covers authenticating /services/* per client and managing per-client Anthropic keys.
• Testing - services/testing/README.md documents the shared acceptance-test harness (YAML specs + LLM-as-judge); per-service guides live in each service's tests/.

Requirements

To run this server locally, you'll need the following dependencies to be installed:

• python 3.11 (yes, 3.11 exactly, see Python Setup)
• poetry
• bunjs

We recommend using asdf with the python plugin installed.

Getting Started

To run the server locally, you need to install the python dependencies

poetry install

Then start the server (note that bun install is not needed, see below)

bun start

To start a hot-reloading development server which watches your typescript, run:

bun dev

To see an index of the available language services, head to localhost:3000.

Python Setup

This repo uses poetry to manage dependencies.

We use an "in-project" venv , which means a .venv folder will be created when you run poetry install.

All python is invoked through entry.py, which loads the environment properly so that relative imports work.

You can invoke entry.py directly (ie, without HTTP or any intermediate js) through bun from the root:

bun py echo --input tmp/payload.json

Bun installation

Bun does not require an installation, like npm does. You can run bun start right after cloning the repo.

Bun will then install dependencies against the global cache on your machine. This still uses the lockfile (bun.lockb).

To update a module version, run bun add <module>@<version>, which will update your lockfile.

One drawback of this is that there is no intellisense, because IDEs rely on node_modules to load d.ts files. You are welcome to run bun install to run from a node_modules. None of this affects python.

See bun's install docs for more details.

CLI

To communicate with and test the server, you can use @openfn/cli.

Use the apollo command with your service name and pass a json file:

openfn apollo echo tmp/payload.json

Pass --staging, --production or --local to call different deployments of apollo.

To default to using your local server, you can set an env var:

export OPENFN_APOLLO_DEFAULT_ENV=local

Or pass an explicit URL if you're not running on the default port:

export OPENFN_APOLLO_DEFAULT_ENV=http://localhost:6666

Output will be shown in stdout by default. Pass -o path/to/output/.json to save the output to disk.

You can get more help with:

openfn apollo help

Note that if a service returns a { files: {} } object in the payload, and you pass -o with a folder, those files will be written to disk.

API Keys & Env vars

Some services require API keys.

Rather than coding these into your JSON payloads directly, keys can be loaded from the .env file at the root. See .env.example for the full list of keys and env vars Apollo reads.

Also note that tmp dirs are untracked, so if you do want to store credentials in your json, keep it inside a tmp dir and it'll remain safe and secret.

APOLLO_UNICODE_STEP_NAMES

Controls which characters a workflow step name may contain. Apollo sanitises step names on the way out and Lightning validates them on the way in, so the two rules have to agree. The default is false.

Value	Rule
false (default)	ASCII only. Letters, digits, spaces, hyphens, underscores. Accents are folded (Café becomes Cafe) and anything else is dropped. This is the rule Lightning enforces today.
true	Anything except control characters. Letters and marks from any script, all punctuation and symbols, emoji, /, :, >, &, quotes and apostrophes. Vérifier l'état and 患者確認 survive exactly as typed.

Leave it off until Lightning ships Unicode step names (Lightning#4577), then turn it on. Turning it on first means Apollo emits names Lightning rejects. Leaving it off afterwards means Apollo renames steps people typed deliberately, across the whole workflow, on every turn that returns YAML.

The permissive rule is deliberately maximal. Apollo being stricter than Lightning is the worse of the two failures: Lightning rejecting a name is loud and recoverable, whereas Apollo quietly renaming a valid name is the silent vandalism this flag exists to prevent.

The rejected control set is the same in both modes: C0 (U+0000-U+001F, NUL included), DEL (U+007F), C1 (U+0080-U+009F), the noncharacters U+FFFE and U+FFFF, the surrogates U+D800-U+DFFF, and the line and paragraph separators U+2028 and U+2029. A NUL byte in a name crashes the Postgres insert on Lightning's side. Names are NFC-normalised in both modes so that Apollo and Lightning agree on how to spell an accent, which is what step lookup matches on, and capped at 100 graphemes because that is what Ecto's validate_length counts.

The rule lives in services/name_rules.py, and everything that states or enforces it is derived from there: the sanitiser, the workflow-generation prompt (describe_rule_for_prompt), the acceptance-test judges (describe_rule_for_judge, substituted into the rubric markdown by judges.load_judge), and the assert_no_special_chars test assertion. Change the rule in that one file and all four follow.

The 100-character cap is counted in graphemes, because that is what Ecto's validate_length counts on Lightning's side. The clustering is hand-written in name_rules, with no third-party dependency, and it targets Elixir's String.length/1 rather than UAX #29 — Elixir deviates from the spec in two places (it does not implement the Unicode 15.1 Indic conjunct rule, and it ends an emoji ZWJ run at the joiner unless a pictograph follows) and the whole point is to agree with Elixir, not with the spec.

tools/unicode_parity is the harness that checks it. Run python3 edges.py, then elixir probe.exs, then python3 check.py with the Elixir version Lightning runs; --tables prints the literals to paste back into name_rules. It checks five things: every codepoint's break class, the Extended_Pictographic set, the trim set, what a GB11 emoji run may be separated from its joiner by, and cluster boundaries over a generated corpus. Normalisation is not among them: normalize_nfc is the standard library's, so there is no table of ours to check against Elixir.

Extended_Pictographic needs its own check because it is not a break class, so a per-codepoint sweep cannot see it — an over-broad set there silently changes clustering either side of a ZWJ and nothing else notices. That is exactly how a hand-written table with 531 wrong codepoints survived two rounds of review.

Re-run the harness whenever Python's or Elixir's Unicode version moves. name_rules.PARITY_SOURCE records what the committed tables were generated from, and a unit test pins it, so a silent regeneration fails loudly. Python moving ahead only makes Apollo overcount, which truncates early; Elixir moving ahead is the direction that reintroduces undercounting, and an undercount ships a name Lightning rejects.

Debugging

The server defaults to port 3000. You can test any service directly with curl to confirm Apollo is working independently of Lightning (or any other client).

For example, to trigger a workflow_chat stream:

curl -N -X POST http://localhost:3000/services/workflow_chat/stream \
    -H "Content-Type: application/json" \
    -d '{"content":"make a simple http workflow","history":[],"api_key":"<your-anthropic-api-key>"}'

The api_key field is your Anthropic API key. If ANTHROPIC_API_KEY is already set in your .env, you can omit it. In Lightning, this is configured via the ANTHROPIC_API_KEY environment variable and passed through to Apollo on each request.

The -N flag disables buffering so SSE events appear as they arrive. You should see a stream of event: log lines followed by event: complete. An event: error response means the issue is inside Apollo.

If the stream returns successfully here but Lightning isn't receiving it, the issue is on the Lightning side -- check that APOLLO_ENDPOINT=http://localhost:3000 is set correctly in Lightning's environment (no trailing slash).

To check API key connectivity (Anthropic, OpenAI, Pinecone), hit the status service:

curl http://localhost:3000/services/status

Troubleshooting

If you get errors like poetry: command not found (error code 127), and poetry is set up on your machine, you may need to add these env vars to your .bashrc (or whatever you use):

export BUN_INSTALL="$HOME/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"

Server Architecture

The Apollo server uses bunjs with the Elysia framework.

It is a very lightweight server. By default it includes no authentication, but instance auth can be enabled (see Database below).

Python services are hosted at /services/<name>. Each service expects a POST request with a JSON body, and will return JSON.

There is very little standard for formality in the JSON structures to date. The server may soon establish some conventions for better interopability with the CLI.

Python scripts are invoked through a child process. Each call to a service runs in its own context.

Python modules are pretty free-form but must adhere to a minimal structure. See the Contribution Guide for details.

Services

Each service lives in services/<name>/ and is auto-mounted by service discovery.

Start the server with bun start and head to http://localhost:3000 to see an overview of active services. Or go to https://apollo.openfn.org to see the production services.

Database

Apollo uses Postgres for two tables: adaptor_function_docs (parsed adaptor docs, used by load_adaptor_docs / search_adaptor_docs) and lightning_clients (the instance-auth allow-list).

There is no migration framework. The schema is just two schema.sql files you apply with psql, both written with CREATE TABLE IF NOT EXISTS so re-running them is safe:

• services/load_adaptor_docs/schema.sql
  • adaptor_function_docs. This table is also created lazily the first time load_adaptor_docs runs, so applying it by hand is optional.
• lightning_clients - created and kept current by the migration runner (platform/src/db/migrate.ts, migrations under platform/migrations/). It is applied automatically at Apollo startup when POSTGRES_URL is set; no manual psql step is needed.

First, make sure you've configured your desired POSTGRES_URL in your .env file.

Create the DB

Create a Postgres DB matching your POSTGRES_URL from the .env file

To reset the DB

set -a; . ./.env; set +a; psql "$POSTGRES_URL" -c "DROP TABLE IF EXISTS lightning_clients, adaptor_function_docs CASCADE;"

Run the migrations

lightning_clients is migrated automatically at Apollo startup (see platform/src/db/migrate.ts). Apply the adaptor-docs schema separately:

set -a; . ./.env; set +a; psql "$POSTGRES_URL" -f services/load_adaptor_docs/schema.sql

Instance authentication (optional)

/services/* is authenticated so that only known clients (e.g. specific Lightning instances) may call it, with Apollo using each client's own Anthropic API key for that client's requests.

The auth hook is always active. A request that carries an api_key must resolve to a known Lightning client or it is rejected. Clients are looked up in the lightning_clients table via APOLLO_CLIENTS_DB_URL (falling back to POSTGRES_URL when unset).

To enable it and provision clients, see platform/src/auth/.

Deploying: pin APOLLO_INTERNAL_TOKEN in production

APOLLO_INTERNAL_TOKEN is the mechanism that lets internal apollo() self-calls through the auth hook: a self-call carries it in the x-apollo-internal header and the hook matches it. Because the global ANTHROPIC_API_KEY is dev-only, a token that fails to match is a dead end (a 401), not a soft fallback to the global key - so the match has to work in every topology.

• Always set APOLLO_INTERNAL_TOKEN to a shared value across the deployment in production. When it is set, the per-process minting path never runs.
• The per-process random mint is a dev-only convenience. Apollo assumes one Bun process per host; the apollo() self-call relies on loopback calls landing on the same process, so a minted token only works single-process-per-host.
• If reusePort clustering is ever enabled, a shared APOLLO_INTERNAL_TOKEN is required, not optional: a self-call can otherwise be routed to a sibling process that minted a different token and will 401. Startup logs the token's provenance (env vs minted) and warns when this dangerous combination is detected; a mismatch at the hook is logged as a distinct, greppable warning.

Websockets

Every service can receive connections in one of two ways:

• HTTP POST method
• Websocket connection

The same URL is used for both connections, clients must request upgrade to a websocket.

Websocket connections will receive a live log stream.

Websockets use the following events:

start: sent by the client with a JSON payload in the data key.

complete: sent by the server when the python script has completed. The result is a JSON payload in the data key.

log: sent by the server whenever the python process logs a line through a logger object.

Note that print() statements do not get sent out to the web socket, as these are intended for local debugging. Only logs from a logger object are diverted.

Docker

To build the docker image:

docker build .  -t openfn-apollo

To run it on port 3000

docker run -p 3000:3000 openfn-apollo

Contributing

See the Contribution Guide for more details about how and where to contribute to the Apollo platform.

Release

See How To Release Wiki page.