Repository navigation
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
| Layer / File(s) | Summary |
|---|---|
Template assets and writing-system construction backend/FwHeadless/*, backend/FwLite/*, backend/FwLite/FwLiteProjectSync.Tests/* |
The headless project packages LCModel templates and configures their runtime path. Project loading now accepts UI, analysis, and vernacular writing-system lists and restores their requested order. Tests cover the generated writing-system collections. |
Headless creation workflow
| Layer / File(s) | Summary |
|---|---|
Headless repository creation workflow backend/FwHeadless/Services/*, backend/Testing/FwHeadless/* |
The creation service generates the project, validates the data-model version, writes genesis metadata, initializes and branches the repository, pushes it, and cleans up failures. Creation reservations now block conflicting sync jobs. Mercurial helper methods and reservation tests were added. |
Headless API entry point
| Layer / File(s) | Summary |
|---|---|
Headless project creation endpoint backend/FwHeadless/Program.cs, backend/FwHeadless/Routes/ProjectRoutes.cs |
The application maps POST /api/project/initFwDataProject. The endpoint validates writing-system and language-tag input, resolves the project code, and invokes ProjectCreationService. |
LexBox API orchestration
| Layer / File(s) | Summary |
|---|---|
LexBox API orchestration and validation backend/LexBoxApi/*, backend/LexCore/Entities/*, backend/Testing/LexBoxApi/*, backend/Testing/SyncReverseProxy/*, frontend/schema.graphql |
The admin endpoint creates the project, calls FwHeadless, rolls back failed creation, and supports the LanguageForgeNonSR migration status. The client uses typed JSON and a five-minute timeout. Tests cover validation, permissions, repository commits, and writing-system order. |
Estimated code review effort: 4 (Complex) | ~45 minutes
Change: Feature
Merge Risk: 
Moderate · previous review a7e20
-
Creation failures or disconnects can leave inconsistent project state, while reachable internal workloads may bypass the administrator-facing API. Resolve these risks before merging.
🚥 Pre-merge checks | ✅ 4 | ❌ 1
-
❌ Failed checks (1 warning)
Check name Status Explanation Resolution Docstring Coverage ⚠️ WarningDocstring coverage is 18.68% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 91 functions across 24 files. (2 skipped:… Write docstrings for the functions missing them to satisfy the coverage threshold. ✅ Passed checks (4 passed)
-
Check name Status Explanation Title check ✅ Passed The title clearly and concisely describes the main change: adding a Lexbox API to create an empty FLEx project. Description check ✅ Passed The description directly explains the new API, repository initialization behavior, Mercurial commits, migration status, and linked issue. Linked Issues check ✅ Passed Issue #2589requires a migration status value, an initFwData project-creation parameter, and a separate admin permission check. The PR addsProjectMigrationStatus.LanguageForgeNonSRinProject.cs…Out of Scope Changes check ✅ Passed The reviewed changes support the stated API feature. Headless project creation, repository initialization, writing-system handling, rollback, concurrency protection, client wiring, and automated tests…
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 18.68% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 91 functions across 24 files. (2 skipped: 2 unsupported.)
-
Finishing Touches
-
Generate docstrings
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
Fix failing CI checks
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Tip
🎯 Customize your risk assessments
-
Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.
✅ Automatic approvals, with your rules
-
Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
A rabbit reads each line,
The patch grows clear beneath the moon,
Small changes hop in place,
Tests guard the garden path,
Reviews bloom before the dawn.
CodeRabbit Verify
- Run CodeRabbit Verify to test the changes in this pull request.
| } | ||
| catch (Exception ex) | ||
| { | ||
| logger.LogError(ex, "Failed to roll back project {ProjectId} ({Code}) after a failed template creation", projectId, code); |
There was a problem hiding this comment.
Ah, I see: the security risk is forging new log entries via inserting newlines (or CRLF). We should be sanitizing the project code anyway, at an earlier point than this.
There was a problem hiding this comment.
Yes, we are sanitizing the project code, and can't hit this code path with unsanitized user data. False positive.
|
The latest updates on your projects. Learn more about Argos notifications ↗︎ Awaiting the start of a new Argos build… |
|
Rebasing on top of develop before I push my new work from today. |
11e808d to
707f704
Compare
|
@coderabbitai fullreview |
|
✅ Action performedFull review finished. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
backend/LexBoxApi/Controllers/ProjectController.cs (1)
53-56: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDead
code is nullcheck.
codeis a non-nullablestringbound implicitly from the query string. With nullable reference types enabled, ASP.NET Core's[ApiController]model binding treats such parameters as required and returns 400 before the action runs, makingcode is nullunreachable. Based on learnings, "avoid adding manual string.IsNullOrWhiteSpace guards for missing values on non-nullable query params."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/LexBoxApi/Controllers/ProjectController.cs` around lines 53 - 56, Remove the unreachable `code is null` condition from the validation in `ProjectController`, while preserving the minimum-length and `Project.ProjectCodeRegex` checks and existing 400 response.Source: Learnings
backend/FwHeadless/Services/ProjectCreationService.cs (1)
64-64: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winParameter order flips between
CreateFromTemplate,BuildFromTemplate, andNewProject.
CreateFromTemplatetakes(vernacular, analysis, ui),BuildFromTemplatealso takes(vernacular, analysis, ui), butNewProjecttakes(analysis, vernacular, ui). All three currently thread the right list to the right slot (verified by parameter-name binding), but with two identically-typedIReadOnlyList<string>positional arguments and no compiler check, this is an easy spot for a future edit to silently swap analysis/vernacular writing systems in every newly-created project.♻️ Suggested consistency fix
- private void BuildFromTemplate( - FwDataProject fwDataProject, - IReadOnlyList<string> vernacularWritingSystems, - IReadOnlyList<string> analysisWritingSystems, - string uiWs) + private void BuildFromTemplate( + FwDataProject fwDataProject, + IReadOnlyList<string> analysisWritingSystems, + IReadOnlyList<string> vernacularWritingSystems, + string uiWs) { - using var cache = projectLoader.NewProject(fwDataProject, analysisWritingSystems, vernacularWritingSystems, uiWs); + using var cache = projectLoader.NewProject(fwDataProject, analysisWritingSystems, vernacularWritingSystems, uiWs); }and update the call site to
BuildFromTemplate(fwDataProject, analysisWritingSystems, vernacularWritingSystems, uiWritingSystem)to match.Also applies to: 95-105
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/FwHeadless/Services/ProjectCreationService.cs` at line 64, Standardize the writing-system parameter order across CreateFromTemplate, BuildFromTemplate, and NewProject to use vernacular, analysis, then UI. Update NewProject’s signature and internal binding as needed, and change the BuildFromTemplate call site to pass vernacularWritingSystems before analysisWritingSystems while preserving each list’s intended destination.backend/FwLite/FwDataMiniLcmBridge.Tests/Fixtures/MockFwProjectLoader.cs (1)
22-38: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winMock doesn't override the new list-based
NewProjectoverload.
MockFwProjectLoaderonly overrides the single-pair overload. The newNewProject(project, IReadOnlyList<string>, IReadOnlyList<string>, string)overload is virtual on the baseProjectLoaderbut unoverridden here, so any test/consumer that calls it through this mock will silently fall through to the real liblcm-backed implementation instead of the lightweight in-memory blank project — defeating the point of using the mock.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@backend/FwLite/FwDataMiniLcmBridge.Tests/Fixtures/MockFwProjectLoader.cs` around lines 22 - 38, The MockFwProjectLoader must override the new list-based NewProject overload so calls using IReadOnlyList<string> work through the lightweight in-memory path. Add the override alongside the existing NewProject method, preserving the same initialization, cache creation, project registration, and return behavior while adapting the list-based workspace arguments as required by the base API.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/LexBoxApi/Services/ProjectService.cs`:
- Around line 241-254: Update CleanupFailedProjectCreation to invoke the
FwHeadless deletion/guard operation before DeleteRepoIfExists and any database
removal, mirroring the ordering in DeleteProjectPermanently. Ensure an
in-progress sync raises ProjectSyncInProgressException and prevents cleanup from
deleting an active or completed creation; preserve the existing repository,
database, and cache cleanup after the guard succeeds.
---
Nitpick comments:
In `@backend/FwHeadless/Services/ProjectCreationService.cs`:
- Line 64: Standardize the writing-system parameter order across
CreateFromTemplate, BuildFromTemplate, and NewProject to use vernacular,
analysis, then UI. Update NewProject’s signature and internal binding as needed,
and change the BuildFromTemplate call site to pass vernacularWritingSystems
before analysisWritingSystems while preserving each list’s intended destination.
In `@backend/FwLite/FwDataMiniLcmBridge.Tests/Fixtures/MockFwProjectLoader.cs`:
- Around line 22-38: The MockFwProjectLoader must override the new list-based
NewProject overload so calls using IReadOnlyList<string> work through the
lightweight in-memory path. Add the override alongside the existing NewProject
method, preserving the same initialization, cache creation, project
registration, and return behavior while adapting the list-based workspace
arguments as required by the base API.
In `@backend/LexBoxApi/Controllers/ProjectController.cs`:
- Around line 53-56: Remove the unreachable `code is null` condition from the
validation in `ProjectController`, while preserving the minimum-length and
`Project.ProjectCodeRegex` checks and existing 400 response.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e44e80e4-8af0-4b09-bccb-de1a96b9adf5
📒 Files selected for processing (21)
backend/FwHeadless/FwHeadless.csprojbackend/FwHeadless/FwHeadlessConfig.csbackend/FwHeadless/FwHeadlessKernel.csbackend/FwHeadless/Program.csbackend/FwHeadless/Routes/ProjectRoutes.csbackend/FwHeadless/Services/ISendReceiveService.csbackend/FwHeadless/Services/ProjectCreationService.csbackend/FwHeadless/Services/SendReceiveHelpers.csbackend/FwHeadless/Services/SendReceiveService.csbackend/FwHeadless/Services/SyncHostedService.csbackend/FwLite/FwDataMiniLcmBridge.Tests/Fixtures/MockFwProjectLoader.csbackend/FwLite/FwDataMiniLcmBridge/LcmUtils/ProjectLoader.csbackend/FwLite/FwLiteProjectSync.Tests/ProjectTemplateTests.csbackend/LexBoxApi/Controllers/ProjectController.csbackend/LexBoxApi/LexBoxKernel.csbackend/LexBoxApi/Services/FwHeadlessClient.csbackend/LexBoxApi/Services/ProjectService.csbackend/LexCore/Entities/CreateProjectFromTemplateInput.csbackend/Testing/FwHeadless/SyncHostedServiceCreationReservationTests.csbackend/Testing/LexBoxApi/CreateFromTemplateValidationTests.csbackend/Testing/SyncReverseProxy/CreateProjectFromTemplateTests.cs
rmunn
left a comment
There was a problem hiding this comment.
Ready for review, but not yet ready to merge: there's a subtle writing-system-related bug I'm trying to pin down (see comment below).
| return LoadCache(project); | ||
| } | ||
|
|
||
| private static CoreWritingSystemDefinition CreateWritingSystemDefinition(string ws) => new(ws) { Id = ws }; |
There was a problem hiding this comment.
This is causing a bug: when a writing system ID would be canonicalized to a different value (e.g., one test project had qaa-x-qaa-v which SIL.Core canonicalizes to qaa-x-v), and then LfMerge produced the error "Unable to set writing system 'qaa-x-v' because this id already exists." I'm still working on a solution.
There was a problem hiding this comment.
Solution implemented in an LfMerge PR; I'll merge that first, then this won't be a problem any more.
|
The Claude-generated code in 02c41f5 makes ProjectService.CreateProject and ProjectService.CreateDraftProject inconsistent with each other, and also with the GraphQL CreateProject mutation. Claude only added the optional Is that what we want? Pro: no change to graphQL inputs (apart from the new enum value), so no need for the GraphQL mutation handlers to implement the "admin-only" rule on setting ProjectOrigin in a new project. Con: inconsistency, no way for this to be set via GraphQL. I'm inclined to modify the CreateProjectInput to include that enum as an optional parameter (i.e. a nullable enum), so that the GraphQL mutations are consistent with the ProjectService methods. But that does mean a wider area where we need to get the admin checks right. EDIT: No, wait, that doesn't make sense. Our rule is that only admins can set ProjectOrigin, and admins will never be creating draft projects, so anyone creating a draft project would never be allowed to set ProjectOrigin anyway, so it makes no sense for CreateDraftProject to take that parameter. The inconsistency beteen CreateProject and CreateDraftProject is just fine here. |
rmunn
left a comment
There was a problem hiding this comment.
Did a self-review. Lots of places where a comment needs to be edited or corrected, and a couple minor code changes. But this is close to being ready to merge. I do want to merge an LfMerge bugfix first, though.
Shared by LexBoxApi (query param) and FwHeadless (creation handler); None is the default. Values: None, Standard, Enhanced. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds a new FieldWorks project from the SIL.LCModel template and pushes it into the empty hg repo LexBox already created: clone the empty remote → build fw.fwdata (first vernacular + first analysis WS) → add the remaining writing systems via the existing CreateWritingSystem surface (dedup + save-on-dispose, then release LCM file locks) → hg add/commit → push. Guards the fdoDataModelVersion, cleans up the local folder on failure, and stubs anthropologyCategories (// TODO: Implement). Adds a per-project creation reservation to SyncHostedService so a sync can't be queued for a project mid-creation (and two creations can't race). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /api/project/create-from-template (projectId query param + JSON body of vernacular/analysis writing systems and anthropologyCategories). Validates the WS tags with IetfLanguageTag at the boundary, resolves the code via IProjectLookupService, and delegates to ProjectCreationService. Registers the service and maps the route. Threads AnthropologyCategories through to the stub. Internal endpoint (cluster-only, no auth) — LexBoxApi does the admin check and the project/repo creation before calling this. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Posts the vernacular/analysis writing systems and anthropologyCategories to the FwHeadless create-from-template endpoint for a given project id. Runs inline; returns null on success or the error body on failure (for the controller's saga rollback). Threads a CancellationToken like the sibling methods. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /api/project/createFromTemplate (admin only): validates the code and that at least one wsVernacular is given (400), rejects a taken code (409), defaults wsAnalysis to ["en"], then creates the FLEx project (Postgres row + empty repo) and asks FwHeadless to populate the repo with the template .fwdata. On FwHeadless failure it compensates by deleting the repo + project row; on success it refreshes LastCommit and returns the new project id. wsVernacular/wsAnalysis are repeatable query params; anthropologyCategories (enhanced|standard|none) defaults to none. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Unit tests (no infrastructure): ProjectController.CreateFromTemplate rejects a missing vernacular WS and an invalid code with 400; SyncHostedService's creation reservation blocks a second concurrent creation and refuses to queue a sync while a project is being created, and is reusable after release. The end-to-end (template build + empty-repo first push) is integration-level and runs in CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Close a TOCTOU race between TryStartProjectCreation and QueueJob (they checked each other's dictionary without a shared lock, so a sync could be queued for a project mid-creation and race on the repo). Both check-then-add paths now run under one lock. - AssertModelVersionMatches now fails closed: an unreadable version is treated as a mismatch rather than silently allowing the push. - Log a warning when a not-yet-implemented anthropologyCategories value is requested so the no-op stub isn't silent. - Route: null-safe writing-system checks and reject an empty analysis list with 400 (rather than a 500 from the service guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Propagate FwHeadless's status code: an invalid writing-system tag now surfaces as 400 instead of being flattened to 500 (FwHeadlessClient returns the status + body). - Give the FwHeadless HTTP client a 5-minute timeout so inline creation (cold LCM load + push) doesn't hit the 100s default. - Route the failure compensation through a new ProjectService.CleanupFailedProjectCreation so repo+row deletion and cache invalidation stay in one place (the controller no longer deletes the row directly and skip cache invalidation). - Extract the analysis-default (["en"]) into a testable helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Assert the ProblemDetails.Detail so each validation test proves its own branch (the two 400s were previously indistinguishable). - Add the mirror race case: a queued sync blocks a creation. - Cover the analysis-default helper (null/empty -> ["en"], supplied kept). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end test (Category=Integration, CI-only — needs the lexbox stack): calls the admin createFromTemplate endpoint, then asserts the empty repo received the first commit (GetProjectLastCommit non-null; hgweb tip is no longer the all-zero hash) and that the pushed fw.fwdata carries the requested vernacular + analysis writing systems. Also asserts a missing vernacular is rejected with 400. Cleans up via softDeleteProject. This exercises the one path that can't be unit-tested: LfMergeBridge/Chorus pushing a locally-committed brand-new project into an empty remote repo. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ProjectCreationService needs NewLangProj.fwdata (and its sibling template files) to build a new project, but FwDataBridgeConfig.TemplatesFolder defaults to the FieldWorks user-data dir (~/.local/share/fieldworks/Templates), which doesn't exist in the container — creation failed with DirectoryNotFoundException on NewLangProj.fwdata. Copy the SIL.LCModel package's contentFiles/Templates/* into FwHeadless's output and point TemplatesFolder at that shipped folder (AppContext.BaseDirectory/Templates) unless overridden in config. Mirrors how FwLiteWeb and the bridge tests ship the templates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Creating a project from a template pushes into a repo LexBox has only `hg init`'d server-side, so it has zero changesets. Cloning that empty remote is a no-op that returns Success=true but leaves no local `.hg`, so the subsequent `hg add` failed with "no repository found ... (.hg not found)". Replace the clone with LfMerge's from-scratch genesis recipe (verified against the lfmerge reference implementation): `hg init` the local fw/ folder, set the branch to the FDO model version BEFORE the first commit, build the .fwdata, then commit + push. Chorus's Language_Forge_Send_Receive refuses to make the first-ever commit itself, so the manual CommitFile stays. The branch step is the data-correctness fix: FLEx clients clone looking for a branch named after the model version, so a first commit landing on `default` would be invisible to them. Numeric branch names work via the fixutf8 hg extension already wired into Mercurial/mercurial.ini. Adds InitRepo and SetBranch primitives to SendReceiveHelpers / ISendReceiveService. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
| { | ||
| if (nextLineIsValue) | ||
| { | ||
| line = line.Replace(forwardSlashValue, backslashValue); |
There was a problem hiding this comment.
if this does nothing then we should just return as there's no point rewriting the whole file.
| private string WriteInitialCustomPropertiesFile(FwDataProject fwDataProject) | ||
| { | ||
| var path = Path.Join(fwDataProject.ProjectFolder, CustomPropertiesFilename); | ||
| using (var writer = XmlWriter.Create(path, CanonicalXmlSettings.CreateXmlWriterSettings())) |
There was a problem hiding this comment.
this seems like overkill. Just File.WriteAllText with text:
<?xml version="1.0" encoding="utf-8"?>
<AdditionalFields />1. No need to use a full XmlWriter for a two-line file 2. When replacing forward slash with backslash in LinkedFilesDirectory in a newly-created .fwdata file, exit early if we can detect that we made no changes, avoiding the need to copy the rest of the file.
A project may be queued/running a sync or being created, never both. Tracking both in a single ConcurrentDictionary lets one atomic TryAdd/GetOrAdd decide who gets the project, so the separate lock that serialised check-then-add across two dictionaries is no longer needed. Removals are conditional on the exact entry, so ending one kind of job can never clear the other kind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
QueueJob returned a bool that conflated "already queued", "project is still being created" and "channel write failed", and the execute route ignored it, answering 200 "Sync job queued" even when no sync would run. Return a QueueJobResult instead, and have the route answer 409 while the project is still being created and 503 if the job couldn't be queued. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A sync request that arrived while the project's sync was already running got a 200, as if a sync covering the caller's changes had been queued. But the running sync may have read the repo before those changes were pushed, so the caller could wrongly conclude its changes were synced. Track whether a queued sync has started. A request that finds the sync still waiting in the queue is covered by it and still gets a 200; one that finds it already running now gets 429 Too Many Requests, telling the caller to try again once the running sync finishes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The creation TaskCompletionSource already used RunContinuationsAsynchronously, but the sync one didn't. So when ExecuteAsync completed a sync, a long-polling await-finished request could run its response handling inline on the sync loop, delaying the next project's sync until that handler hit a real await. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
If a sync is already queued, we don't return an error but let it fall through to the "Sync job queued" HTTP 200 result. This might look like a bug, so add a comment explaining that it's intentional.
Neither LexBoxApi nor FwHeadless limited how many writing systems a project-creation request could list, how long each tag could be, or whether a tag was repeated. LexBoxApi also created the project row and repo before any tag was checked, so a bad request was only rejected by FwHeadless after there was already a project to tear down. Add a shared shape check on InitFwDataProjectInput: at most 100 writing systems per list, at most 128 characters per tag, no empty tags, and no case-insensitive duplicates within a list. The limits are deliberately generous so that metadata in private-use (-x-) subtags isn't forbidden. LexBoxApi runs the check before creating anything, and FwHeadless runs it again before its BCP 47 check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Devin is concerned that FwHeadless is blindly trusting its API endpoints but has no NetworkPolicy, so any compromised container anywhere in our Kubernetes cluster could execute API endpoints in FwHeadless. Claude Opus 5.5 suggested the following NetworkPolicy addition to the # FwHeadless has no authentication of its own: LexBoxApi is the only intended caller and does the
# authorization, so only the lexbox pod may reach it. Kubelet health probes are always allowed.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: fw-headless-network-policy
namespace: languagedepot
labels:
app: fw-headless
spec:
podSelector:
matchLabels:
app: fw-headless
policyTypes:
- Ingress
ingress:
- from:
# A podSelector with no namespaceSelector only matches pods in this policy's own namespace, so this
# works unchanged in both languagedepot and languagedepot-dev without letting one reach the other
- podSelector:
matchLabels:
app: lexbox
ports:
- protocol: TCP
port: 8081I'm not going to add that yet, because I want to discuss it first (and that's kind of out-of-scope for this PR and should probably be a separate PR, because this is not an issue introduced by this PR's new API). But I wanted to mention it. @hahn-kev, any thoughts? |
|
@hahn-kev - Devin is concerned that SyncHarmonyProject, which doesn't check the same ConcurrentDictionary that ExecuteMergeRequest does, could be called at the same time that project creation is called. Do you think that's a realistic concern? Devin's comments below. backend/FwHeadless/Services/SyncHostedService.cs:R75-83 Direct sync bypasses creation reservation During creation, The reservation coordinates only Example: An admin starts template creation. Before its initial push completes, another client calls Recommended fix: Route every I asked Claude to implement a fix, and it seemed to me that the fix was not worth the extra complexity that it added to SyncHarmonyProject, because how is anyone going to be trying to run a Harmony sync on a project they don't yet have on their FW Lite client? Unless there's a way for FW Lite to download a project whose repo is still being initialized, I think this concern can't happen. BUT there's also the possibility of a FwHeadless sync running at the same time as a SyncHarmonyProject, and that one is realistic. You know the Harmony code better than I do; is that something we need to guard against, or would it not be a problem? |
CleanupFailedProjectCreation deleted the project row first, so if deleting the repo then failed, the repo was left with no row pointing at it: invisible to admins, and it made the next creation attempt with that code fail. Deleting the repo first means any failure leaves a project row an admin can see and delete, and a retry gets a clear "already exists" conflict. The cache invalidations now run in a finally, so they're no longer skipped when a step fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
FWIW, here's Devin's opinion of the proposed NetworkPolicy fix (my question was "Would this be sufficient?"): Mostly, yes. The selectors and port match the real manifests, so the policy blocks the unauthenticated access the finding describes. A few caveats: What matches
Caveats
Also add |
InitFwDataProject validated the vernacular and analysis lists before creating anything, but not wsUi. An empty or over-long UI tag created the project row and repo, was then rejected by FwHeadless, and had to be torn down again. ValidateUiWritingSystem does the same shape checks as the list validator (present, at most MaxWritingSystemTagLength characters) and is shared by LexBoxApi and FwHeadless, so both reject the same tags. FwHeadless still checks BCP 47 validity itself, since LexBoxApi doesn't reference SIL.WritingSystems. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
InitFwDataProject's ProjectExists check is only a read, so two requests for the same new code can both pass it. The unique index on Project.Code then makes the second insert fail, and that DbUpdateException surfaced as a 500 instead of the documented 409. Catch the duplicate-project-code violation around CreateProject using the existing DbError mapping (as the GraphQL mutations already do) and return the same 409 as the early check. The losing request has created nothing: its insert fails before the repo is initialised, and its transaction rolls back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
All Devin review comments addressed, except for the flags which are very low-severity. I haven't commented on everything @hahn-kev mentioned, but if I missed anything it's only one or two items. I think it's time for another human review; this is closing in on good. |
InitFwDataProject used to hold the request open while FwHeadless built and pushed the project, which can take longer than the ingress's 60-second proxy timeout. A caller could then get a 504 for a creation that went on to succeed, and a LexBox restart mid-request skipped both the rollback on failure and recording the commit on success. The endpoint now creates the project row (marked FwDataInitStatus InProgress in the same insert) and the empty repo, queues InitFwDataProjectJob, and returns 202 with a Location header for the new fwDataProjectCreationStatus endpoint, which reports Creating, Created or CreationFailed. The job runs on Quartz's persistent store with RequestRecovery, so a run interrupted by LexBox stopping is rerun, and every run is written to be safe to repeat: - A non-empty repo means an earlier attempt's push landed, so the project is marked ready without asking FwHeadless. - A creation FwHeadless is still running is waited for, not restarted. - Otherwise FwHeadless's local project state is cleared (an interrupted attempt may have left some) and the creation started. - A failure FwHeadless reports is rolled back with CleanupFailedProjectCreation. - An unknown outcome (FwHeadless unreachable or busy, our request timing out, or a 409 because something else holds the project) is retried a minute later, up to 10 attempts, after which the project stays marked InProgress for an admin. The reason for a failure is kept in memory for an hour after the rollback has deleted the row, so the status endpoint can report it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
InitFwDataProject used to fall back to the project code when no name was given. Make name a required parameter instead, and reject a blank one with a 400 before anything is created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@hahn-kev I've addressed all your review comments. What do you think? |
#2959) ProjectLoader.NewProject now takes lists of analysis and vernacular writing systems plus a UI writing system. The first of each list is the default; the rest are passed to CreateNewLangProj as additional writing systems, and the new project keeps them in list order. The single-writing-system overload now goes through the list overload. Split out of #2439. Co-authored-by: Kevin Hahn <kevin_hahn@sil.org> Co-authored-by: Robin Munn <rmunn@pobox.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A sync request that arrived while the project's sync was already running got a 200, as if a sync covering the caller's changes had been queued. But the running sync may have read the repo before those changes were pushed, so the caller could wrongly conclude its changes were synced. QueueJob now returns a QueueJobResult and tracks whether a queued sync has started. A request that finds the sync still waiting in the queue is covered by it and still gets a 200; one that finds it already running gets 429, and a full queue gets 503 instead of a false 200. Split out of #2439. Co-authored-by: Kevin Hahn <kevin_hahn@sil.org> Co-authored-by: Robin Munn <rmunn@pobox.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
replaced by #2961 |
POST api/project/initFwDataProject creates the project row (marked
FwDataInitStatus.InProgress) and an empty repo, then returns 202 and
queues InitFwDataProjectJob. The job asks FwHeadless to build the project
from the SIL.LCModel template with the requested writing systems and push
it, then marks the project ready, or rolls it back if FwHeadless reports
a failure. The job is safe to rerun: it uses the repo's tip as the source
of truth and retries while the outcome is unknown.
GET api/project/fwDataProjectCreationStatus/{projectId} reports progress.
In FwHeadless, ProjectCreationService does hg init, builds the .fwdata,
commits FLExProject.CustomProperties on the default branch as rev 0,
switches to the send/receive branch with an empty commit (LfMergeBridge
won't create a branch itself), then sends/receives. A new project
therefore starts with three commits instead of FieldWorks' two.
SyncHostedService now also tracks creations, so a sync can't race a
creation of the same project (409 on the merge route).
Also adds the LanguageForgeNonSR project origin, which admins can pass as
projectOrigin.
Split out of #2439. Fixes #2589.
Co-authored-by: Robin Munn <rmunn@pobox.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This is a new Lexbox API to create a new repo populated with a FLEx project, rather than requiring users to create an empty repo and then push into it from FW Classic. Due to our using the LfMergeBridge code (which refuses to create a new Mercurial branch), projects created with this code end up with three initial commits in Mercurial, rather than the two initial commits that projects created in FLEx end up with. But one of them is an empty commit that simply changes branches to
750000.7000072(and creates the branch in the process), and that doesn't cause any problems for Chorus or FLExBridge.Self-review done, have identified a few issues to work on. Mostly comment editing, but a couple unit tests I want to add (and one method I want to get rid of).
Fixes #2589.