Skip to content

federation: verification process and registry publication on sync backend - #364

Open
klpoland wants to merge 6 commits into
masterfrom
feature-kpoland-federation-registry-sign-publish
Open

klpoland wants to merge 6 commits into
masterfrom
feature-kpoland-federation-registry-sign-publish

Conversation

@klpoland

@klpoland klpoland commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Note

High Risk
Changes authentication boundaries (Bearer admin API), crypto signing, and SSRF-sensitive verify fetch on the sync path that Traefik exposes publicly.

Overview
Moves federation registry ownership from Django to federation sync: signed manifest publish, applicant domain verify (SSRF-safe well-known fetch), and public well-known routes now live on sync; the gateway keeps intake UI and DB state and delegates verify via FederationSyncClient → POST /internal/registry/verify with FEDERATION_SYNC_ADMIN_TOKEN.

Sync gains [registry.publish] (Ed25519 signing, Docker volumes for manifest/cache), /.well-known/sds-federation.json and /.well-known/sds-registry/*, plus CLI sds-federation-registry. Applicant FEDERATION_REGISTRY_CHALLENGE_TOKEN is set on whoever serves well-known (sync .env.peer / .env.local, not mothership django.env). Gateway drops the well-known URL route and django-side verify/fetch/sign settings; staff Record publish only updates DB after sync publish. Compose, Traefik well-known routing, federation-doctor registry checks, and docs are updated accordingly.

Reviewed by Cursor Bugbot for commit fb80b54. Bugbot is set up for automated code reviews on this repo. Configure here.

@klpoland klpoland self-assigned this Nov 4, 2026
@klpoland klpoland added feature New feature or request refactoring General code improvements gateway Gateway component federation Federation related work (sync service, document storage, peer configuration, etc.) labels Nov 4, 2026
@semanticdiff-com

semanticdiff-com Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Review changes with  SemanticDiff

Changed Files
File Status
  federation/tests/test_registry_verify.py  31% smaller
  gateway/config/settings/base.py  29% smaller
  federation/sds_federation/services/registry_verify.py  27% smaller
  gateway/sds_gateway/users/views/registry_application.py  23% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_verify.py  22% smaller
  gateway/config/settings/utils.py  22% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_application_verify.py  21% smaller
  gateway/sds_gateway/api_methods/tests/test_file_endpoints.py  15% smaller
  gateway/sds_gateway/visualizations/tests/test_views.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_asset_access_control.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_authenticate.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_composite_capture_serialization.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_permission_updates.py  15% smaller
  gateway/sds_gateway/users/tests/test_check_superuser_exists_command.py  15% smaller
  gateway/sds_gateway/users/tests/test_create_ci_superuser_command.py  15% smaller
  gateway/sds_gateway/api_methods/tests/test_celery_tasks.py  15% smaller
  gateway/sds_gateway/users/tests/test_dataset_authors.py  15% smaller
  gateway/sds_gateway/users/tests/test_managers.py  15% smaller
  gateway/sds_gateway/users/tests/test_registry_application_staff_review.py  15% smaller
  federation/sds_federation/services/site_registry.py  13% smaller
  gateway/sds_gateway/api_methods/tests/test_permissions.py  12% smaller
  gateway/sds_gateway/api_methods/tests/test_federation_hardening.py  12% smaller
  gateway/sds_gateway/api_methods/tests/test_capture_endpoints.py  12% smaller
  gateway/sds_gateway/api_methods/federation/registry_challenge.py  12% smaller
  federation/sds_federation/main.py  12% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_challenge.py  2% smaller
  federation-shared.example.env Unsupported file format
  federation/.gitignore Unsupported file format
  federation/README.md Unsupported file format
  federation/compose.local.yaml  0% smaller
  federation/compose.peer.local.yaml  0% smaller
  federation/compose.peer.remote.yaml  0% smaller
  federation/compose.production.yaml  0% smaller
  federation/docs/peer-remote-test.md Unsupported file format
  federation/env.local.example Unsupported file format
  federation/env.peer.example Unsupported file format
  federation/federation.example.toml Unsupported file format
  federation/pyproject.toml Unsupported file format
  federation/scripts/federation-doctor.sh Unsupported file format
  federation/scripts/render-site-config.sh Unsupported file format
  federation/scripts/write_site_env.py  0% smaller
  federation/sds_federation/cli/__init__.py  0% smaller
  federation/sds_federation/cli/registry.py  0% smaller
  federation/sds_federation/models.py  0% smaller
  federation/sds_federation/routes/internal_registry.py  0% smaller
  federation/sds_federation/routes/well_known.py  0% smaller
  federation/sds_federation/services/registry_challenge.py  0% smaller
  federation/sds_federation/services/registry_publish.py  0% smaller
  federation/sds_federation/services/registry_sign.py  0% smaller
  federation/sds_federation/services/well_known.py  0% smaller
  federation/secrets/registry/.gitkeep Unsupported file format
  federation/secrets/registry/README.md Unsupported file format
  federation/site.example.env Unsupported file format
  federation/templates/federation.toml.tmpl Unsupported file format
  federation/templates/gateway-federation.env.tmpl Unsupported file format
  federation/templates/traefik-sync.toml.tmpl Unsupported file format
  federation/tests/conftest.py  0% smaller
  federation/tests/test_internal_auth.py  0% smaller
  federation/tests/test_internal_mount.py  0% smaller
  federation/tests/test_registry_challenge.py  0% smaller
  federation/tests/test_registry_publish.py  0% smaller
  federation/tests/test_registry_serve.py  0% smaller
  federation/tests/test_registry_sign.py  0% smaller
  federation/tests/test_site_registry_accept.py  0% smaller
  federation/tests/test_well_known_route.py  0% smaller
  gateway/.envs/example/django.env Unsupported file format
  gateway/.envs/example/django.prod-example.env Unsupported file format
  gateway/.gitignore Unsupported file format
  gateway/config/urls.py  0% smaller
  gateway/docs/federation-registry-verify.md Unsupported file format
  gateway/justfile Unsupported file format
  gateway/pyproject.toml Unsupported file format
  gateway/scripts/generate-secrets.sh Unsupported file format
  gateway/sds_gateway/api_methods/federation/registry_application_helpers.py  0% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_review.py  0% smaller
  gateway/sds_gateway/api_methods/federation/registry_application_staff_ops.py  0% smaller
  gateway/sds_gateway/api_methods/federation/sync_client.py  0% smaller
  gateway/sds_gateway/api_methods/federation/well_known.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_federation_sync_client.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_federation_well_known.py  0% smaller
  gateway/sds_gateway/api_methods/tests/test_registry_application_staff_ops.py  0% smaller
  gateway/sds_gateway/api_methods/views/federation_well_known.py  0% smaller
  gateway/sds_gateway/templates/users/registry_application_detail.html Unsupported file format
  gateway/sds_gateway/templates/users/registry_application_list.html Unsupported file format
  gateway/sds_gateway/users/tests/test_federation_registry_intake_settings.py  0% smaller
  gateway/sds_gateway/users/tests/test_registry_application_verify_view.py  0% smaller
  gateway/sds_gateway/users/urls.py  0% smaller
  gateway/sds_gateway/users/views/__init__.py  0% smaller
  gateway/uv.lock Unsupported file format
  network/traefik/conf.d/00-federation-sync.toml Unsupported file format
  sdk/pyproject.toml Unsupported file format

@klpoland
klpoland marked this pull request as ready for review November 4, 2026 20:49

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit fb80b54. Configure here.

- ../gateway/.envs/local/opensearch.env
- ../federation-shared.env
# Optional: federation/.env.local — FEDERATION_REGISTRY_CHALLENGE_TOKEN when sync serves
# well-known (see env.local.example). Domain verify/publish use sync internal API, not django.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local sync never receives challenge env

High Severity

compose.local.yaml never injects FEDERATION_REGISTRY_CHALLENGE_TOKEN (or the other .env.local registry vars) into the flagship sync container. Peer compose files interpolate that variable; local compose only comments that .env.local is optional. Compose --env-file interpolates the YAML and does not become container env unless listed in env_file or environment.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit fb80b54. Configure here.

*)
export FEDERATION_REGISTRY_PUBLISH_TOML=""
;;
esac

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Onboarding omits sync internal URL

High Severity

render-site-config.sh writes FEDERATION_SYNC_INTERNAL_URL into gateway django.env but never assigns a default (unlike FEDERATION_SYNC_HEALTH_URL). envsubst therefore emits an empty value, so FederationSyncClient.from_settings() raises and every Verify domain action fails as unavailable.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit fb80b54. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New feature or request federation Federation related work (sync service, document storage, peer configuration, etc.) gateway Gateway component refactoring General code improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant