feat(user-sync): sync OSW User items from MediaWiki and ORCID

docs/tools/index.md

@@ -2,8 +2,10 @@
 
 Besides the Python API, osw ships two adapters that talk to a live instance:
 the `osw` command line client, and an MCP server for agent clients such as
-Claude Code. Both run the same operations from one shared, SDK-free core
-(`osw.service`), so a command and its matching tool behave identically. They
+Claude Code. This section also covers standalone scripts built on the Python
+API. The current one is the [user sync](user-sync.md). The two adapters run
+the same operations from one shared, SDK-free core (`osw.service`), so a
+command and its matching tool behave identically. They
 differ in exactly one way: only the CLI accepts filesystem paths.
 
 ## Setup
@@ -49,4 +51,5 @@ extras are listed in the [Get Started guide](../get-started.md#optional-extras).
 | ---- | -------- |
 | [CLI](cli.md) | The `.env` quick start, the full command reference, and the global flags |
 | [MCP server](mcp.md) | The tool surface, the no-filesystem-access and one-server-per-instance rules, and how to register the server with a client such as Claude Code |
+| [User Sync (script)](user-sync.md) | A Python script that creates or updates `User` items from the MediaWiki accounts of an instance, with the options, the data it writes and how it reconciles existing items |
 | [Configuration](configuration.md) | What both adapters share: where credentials and settings come from, and the full environment-variable reference |

docs/tools/user-sync.md

@@ -0,0 +1,112 @@
+# User Item Sync
+
+Create or update OSW `User` items from the MediaWiki accounts of an OSL
+instance, enriching ORCID users from the public ORCID API. The run is
+idempotent and previews every change before writing.
+
+This is a Python script built on the osw library. It is not a command of the
+`osw` CLI and not an MCP tool, see [Tools](index.md). It runs from a source
+checkout, because `examples/` is not part of the installed package.
+
+## When to use it
+
+OSL instances gain accounts from manual creation and from whitelisted ORCID
+single sign-on (whose username is the ORCID iD). Those accounts have no
+semantic `User` item until this tool reconciles them.
+
+## Run it
+
+```bash
+uv run python examples/user_sync.py --domain your-instance.example.org --dry-run
+```
+
+The script reads the credential file given by `--cred-filepath`, otherwise
+`accounts.pwd.yaml` in the working directory, otherwise the file named by the
+`OSW_CRED_FILEPATH` environment variable. It does not read a `.env` file.
+`--domain` falls back to the `OSW_DOMAIN` environment variable. See
+[Credentials](configuration.md#credentials) for the file format. Start with
+`--dry-run` to preview, then run without it to write after confirming at the
+prompt.
+
+To also clean up existing items, add `--prune`. Without it, the sync never
+writes `employment_contract_status` and never removes anything. With it,
+`employment_contract_status` is removed from the stored jsondata of existing
+items. `website` and `organization` are overwritten with an empty list, unless
+`--with-websites` or `--with-organizations` enables them. This happens in a
+second edit of the page's jsondata after the store. The sync decides from the
+stored jsondata, not from the loaded item, which shows the model default for
+`employment_contract_status`. A second `--prune` run therefore changes nothing:
+
+```bash
+uv run python examples/user_sync.py --domain your-instance.example.org --prune --dry-run
+```
+
+Users whose ORCID record exposes no email are listed in a warning at the end of
+the run; a missing email never fails the sync.
+
+### Options
+
+By default the tool syncs both ORCID and MediaWiki-native accounts and writes
+only core identity plus email.
+
+| Flag | Effect |
+| --- | --- |
+| `--domain` | Target OSL domain. |
+| `--cred-filepath` | Path to `accounts.pwd.yaml`. |
+| `--dry-run` | Preview only; never writes. |
+| `--auto-apply` | Non-interactive: apply creates, gap-fills and removals; keep existing on conflicts. |
+| `--limit N` | Process at most N accounts (testing). |
+| `--orcid-only` | Only ORCID-username accounts (excludes `--mw-only`). |
+| `--mw-only` | Only non-ORCID (MediaWiki-native) accounts (excludes `--orcid-only`). |
+| `--include-system` | Do not skip MediaWiki reserved system accounts. |
+| `--no-redirects` | Do not create `User:` redirect pages. |
+| `--with-websites` | Also store ORCID researcher URLs (opt-in). |
+| `--with-organizations` | Also create and link Organization items from ORCID affiliations (opt-in). |
+| `--with-extras` | Enable both websites and organizations. |
+| `--include-blocked` | Do not skip blocked MediaWiki accounts. |
+| `--prune` | Remove `employment_contract_status` from the stored jsondata of existing items, and overwrite `website` or `organization` with an empty list when their opt-in flag is not given, in a second edit of the page's jsondata after the store (default off; a normal run never writes `employment_contract_status` and never removes anything). |
+
+## What it does
+
+1. Enumerate MediaWiki accounts, skipping blocked accounts (unless
+   `--include-blocked` is set; the User item of a skipped blocked account is
+   left unchanged), bots and privileged groups
+   (`bot`, `sysop`, `bureaucrat`, `interface-admin`) and MediaWiki reserved
+   system usernames. Accounts whose username is an exact ORCID iD are exempt
+   from the group filter (so a person's own admin account is synced); other
+   accounts, such as `<orcid>@bot`, are still excluded by their groups.
+2. Enrich ORCID users from `https://pub.orcid.org/v3.0/<id>`: names and email
+   always; researcher URLs and affiliations only when their flag is enabled.
+3. Map each account to a proposed `User` item with a deterministic id
+   (uuid5 on the ORCID iD, else the username), so re-runs are idempotent.
+4. Reconcile against existing items. Existing items are found by `username`
+   across all `User` items. An item whose deterministic page already exists is
+   updated and not created again. Each user gets one of four categories: NEW,
+   GAP_FILL, CONFLICT or UNCHANGED. Each changed field has one of three
+   statuses: GAP_FILL, CONFLICT or REMOVE. The preview shows a removed field as
+   `-field`. A user with only gap-fills or removals is GAP_FILL. Gap-fills apply
+   automatically and conflicts are resolved interactively. Removals appear only
+   with `--prune`; a normal run never removes anything.
+5. Store items, create `User:<username>` redirects to each item, ensure any
+   linked Organization items, verify by reloading, and warn about users with no
+   email.
+
+## Data written
+
+| Field | Policy |
+| --- | --- |
+| `username`, `first_name`, `surname`, `label`, `orcid` | Always (core identity). |
+| `email` | Standard: always attempted; missing email is warned, not fatal; never removed. |
+| `website` | Opt-in (`--with-websites`); without that flag, on existing items overwritten with an empty list only with `--prune`, in a second jsondata edit after the store. |
+| `organization` | Opt-in (`--with-organizations`); without that flag, on existing items overwritten with an empty list only with `--prune`, in a second jsondata edit after the store. |
+| `employment_contract_status` | Never written without `--prune`; with `--prune`, removed from the stored jsondata of existing items in a second jsondata edit after the store. |
+
+## Notes
+
+- MediaWiki does not expose other users' email or real name, so ORCID is the
+  only source of rich data. Non-ORCID accounts get username-based placeholder
+  names, flagged in the preview. Placeholder names are written only for new
+  items and are never proposed for an existing item.
+- In a source checkout (for developers), the first accepted run autofetches
+  schemas and may regenerate `src/osw/model/entity.py`. Restore it if you do not
+  want that change.

examples/user_sync.py

@@ -0,0 +1,30 @@
+"""Create or update OSW User items from MediaWiki accounts and ORCID.
+
+Usage:
+    uv run python examples/user_sync.py --domain llm4eln.semos.dev --dry-run
+"""
+
+from osw.defaults import paths
+from osw.express import OswExpress
+from osw.tools.user_sync import config_from_args, run_user_sync
+
+
+def main() -> None:
+    config = config_from_args()
+    cred_filepath = config.cred_filepath
+    if cred_filepath is None and paths.cred_filepath.is_file():
+        # OswExpress only reads OSW_CRED_FILEPATH; use the default accounts file.
+        cred_filepath = paths.cred_filepath
+    osw = OswExpress(domain=config.domain, cred_filepath=cred_filepath)
+    report = run_user_sync(config, osw=osw)
+    print(report.summary())
+    for title in report.created:
+        print(f"  created: {title}")
+    for title in report.updated:
+        print(f"  updated: {title}")
+    for key, error in report.failed.items():
+        print(f"  FAILED {key}: {error}")
+
+
+if __name__ == "__main__":
+    main()

src/osw/tools/__init__.py

@@ -0,0 +1 @@
+"""Maintenance and automation tools built on top of the osw core library."""

src/osw/tools/user_sync/__init__.py

@@ -0,0 +1,22 @@
+"""Sync OSW User items from MediaWiki accounts and public ORCID data."""
+
+from __future__ import annotations
+
+from .config import (
+    ORGANIZATION_CATEGORY,
+    USER_CATEGORY,
+    SyncConfig,
+    build_arg_parser,
+    config_from_args,
+)
+from .sync import SyncReport, run_user_sync
+
+__all__ = [
+    "ORGANIZATION_CATEGORY",
+    "USER_CATEGORY",
+    "SyncConfig",
+    "SyncReport",
+    "build_arg_parser",
+    "config_from_args",
+    "run_user_sync",
+]

src/osw/tools/user_sync/build.py

@@ -0,0 +1,91 @@
+"""Build pydantic User/Organization entities from proposals.
+
+Must run within an open OSW connection (the models resolve linked entities on
+construction). Organization references are passed as a list of page-title IRIs;
+the reference attribute is never read back here, as that would trigger a
+network resolve.
+
+An empty field is set to None, so its key is absent from the serialized item.
+The store merges with the stored page, so an absent key keeps the stored value.
+Clearing a stored value is done by ``prune.clear_jsondata_keys`` instead.
+"""
+
+from __future__ import annotations
+
+from typing import Any, List, Set
+
+from opensemantic.base.v1 import Organization, User
+
+from .config import PROTECTED_FIELDS
+from .mapping import ProposedOrganization, ProposedUser
+
+# The concrete DisplayName type used by the label field.
+_DISPLAY_NAME = User.__fields__["label"].type_
+
+
+def _labels(text: str) -> List[Any]:
+    return [_DISPLAY_NAME(text=text)]
+
+
+def _strip_protected(entity: Any) -> None:
+    """Unset protected relations (e.g. employment_contract_status).
+
+    The attribute is set to None, so the key is absent from the serialized item
+    and the model default is not written. A stored value is left unchanged.
+    """
+    for name in PROTECTED_FIELDS:
+        setattr(entity, name, None)
+
+
+def build_organization(proposed: ProposedOrganization) -> Organization:
+    return Organization(uuid=proposed.uuid, label=_labels(proposed.name))
+
+
+def build_user(proposed: ProposedUser) -> User:
+    """Build a full User entity for creation (never carries protected fields)."""
+    args: dict = {
+        "uuid": proposed.uuid,
+        "username": proposed.username,
+        "first_name": proposed.first_name,
+        "surname": proposed.surname,
+        "label": _labels(proposed.label),
+    }
+    if proposed.orcid:
+        args["orcid"] = proposed.orcid
+    if proposed.emails:
+        args["email"] = set(proposed.emails)
+    if proposed.websites:
+        args["website"] = set(proposed.websites)
+    if proposed.organizations:
+        args["organization"] = list(proposed.organizations)
+    entity = User(**args)
+    _strip_protected(entity)
+    return entity
+
+
+def apply_update(entity: Any, proposed: ProposedUser, apply_fields: Set[str]) -> Any:
+    """Apply the accepted fields onto a loaded existing entity, in place.
+
+    An empty organization or website value is set to None, which leaves the
+    stored value unchanged. Protected fields are always unset.
+    """
+    if "username" in apply_fields:
+        entity.username = proposed.username
+    if "label" in apply_fields:
+        entity.label = _labels(proposed.label)
+    if "first_name" in apply_fields:
+        entity.first_name = proposed.first_name
+    if "surname" in apply_fields:
+        entity.surname = proposed.surname
+    if "orcid" in apply_fields:
+        entity.orcid = proposed.orcid
+    if "emails" in apply_fields:
+        entity.email = set(proposed.emails)
+    if "websites" in apply_fields:
+        entity.website = set(proposed.websites) if proposed.websites else None
+    if "organizations" in apply_fields:
+        entity.organization = (
+            list(proposed.organizations) if proposed.organizations else None
+        )
+    _strip_protected(entity)
+    return entity