Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions .github/workflows/architecture-pages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Architecture GitHub Pages

on:
push:
branches: [dev, vrdmr-docs-architecture-pages]
paths:
- "workers/**"
- "runtimes/**"
- "eng/docs/**"
- "docs/**"
- "mkdocs.yml"
- ".github/workflows/architecture-pages.yml"
pull_request:
branches: [dev]
paths:
- "workers/**"
- "runtimes/**"
- "eng/docs/**"
- "docs/**"
- "mkdocs.yml"
- ".github/workflows/architecture-pages.yml"
repository_dispatch:
types: [sibling-repo-updated]
schedule:
# Daily cross-repo sync with python-library, python-extensions, and language-worker-protobuf
- cron: "0 8 * * *"
workflow_dispatch:

permissions:
contents: read

concurrency:
group: "pages-architecture-${{ github.ref }}"
cancel-in-progress: true

jobs:
build:
name: Build & Validate Architecture Site
runs-on: ubuntu-latest
steps:
- name: Checkout python-worker
uses: actions/checkout@v4

- name: Checkout azure-functions-host (gRPC channel metadata)
uses: actions/checkout@v4
with:
repository: Azure/azure-functions-host
path: _sibling_repos/functions-host
fetch-depth: 1
sparse-checkout: src/WebJobs.Script.Grpc

- name: Checkout azure-functions-python-library
uses: actions/checkout@v4
with:
repository: Azure/azure-functions-python-library
path: _sibling_repos/python-library
fetch-depth: 1

- name: Checkout azure-functions-python-extensions
uses: actions/checkout@v4
with:
repository: Azure/azure-functions-python-extensions
path: _sibling_repos/python-extensions
fetch-depth: 1

- name: Checkout azure-functions-language-worker-protobuf
uses: actions/checkout@v4
with:
repository: Azure/azure-functions-language-worker-protobuf
path: _sibling_repos/language-worker-protobuf
fetch-depth: 1

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.13"
cache: "pip"
cache-dependency-path: eng/docs/requirements.txt

- name: Install MkDocs Material
run: python3 -m pip install -r eng/docs/requirements.txt

- name: Run architecture extractor & Markdown sync unit tests
run: python3 -m unittest discover -s eng/docs/tests -v

- name: Extract multi-repo architecture graph & check Markdown blocks
run: python3 eng/docs/extract_arch.py --sibling-root _sibling_repos --check-markdown

- name: Build MkDocs Material documentation
run: mkdocs build --strict

- name: Upload preview artifact
uses: actions/upload-artifact@v4
with:
name: architecture-site
path: ./site

- name: Upload GitHub Pages artifact
if: github.event_name != 'pull_request'
uses: actions/upload-pages-artifact@v3
with:
path: ./site

deploy:
name: Deploy to GitHub Pages
if: github.event_name != 'pull_request'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Configure Pages
uses: actions/configure-pages@v5
with:
enablement: true
continue-on-error: true

- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v4
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,8 @@ tests/**/extensions.csproj
__blobstorage__/*
__queuestorage__/*
__azurite*

# Generated architecture site artifacts
docs/assets/architecture-graph.json
docs/assets/architecture-data.js
_sibling_repos/
47 changes: 47 additions & 0 deletions docs/architecture/dependency-isolation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Dependency Isolation & Placeholder Specialization

In Linux Consumption and Flex Consumption, warm standby ("placeholder") containers boot the worker process **before** any customer function app or `.python_packages` directory is mounted. When a container is specialized to a customer site via `FunctionEnvironmentReloadRequest` (or during `WorkerInitRequest` on Dedicated/Premium plans), the worker must promote customer dependencies ahead of bundled worker packages.

---

## `sys.path` Search Priority After Specialization

Across both `proxy_worker` (`3.13–3.14`) and `r2p2` (`3.15+`), specialized `sys.path` ordering guarantees customer-pinned packages take precedence:

```text
1. Customer packages (<app_dir>/.python_packages/lib/site-packages)
2. Bundled worker runtime (workers_dir — fallback azure_functions_runtime v2/v1)
3. Customer app directory (function_app_directory — function_app.py & user modules)
```

---

## Evolution of Isolation Across Worker Generations

### Python ≤ 3.14 (`DependencyManager`)
In `workers/azure_functions_worker/utils/dependency.py` and `workers/proxy_worker/utils/dependency.py`:

1. **Cold Placeholder Boot (`use_worker_dependencies`)**: Forces worker dependencies first on `sys.path` so `grpcio` and `google.protobuf` C-extensions load from the worker's verified bundle rather than unverified paths.
2. **Specialization (`prioritize_customer_dependencies`)**: Reorders `sys.path` so customer `.python_packages` come first and clears previously imported worker copies of customer-overridable libraries.
3. **Structural Limitation**: Because `google.protobuf` uses a process-global C descriptor pool (`upb` / C++ extension), loading `protobuf` in the worker process at boot could still clash with incompatible `protobuf` versions pinned by customer apps.

### Python 3.15+ R2P2 (`bridge.py` + Namespace-Safe Eviction)
R2P2 eliminates transport collisions completely and hardens `sys.modules` eviction in [`workers/r2p2/bridge/bridge.py`](https://github.com/Azure/azure-functions-python-worker/blob/dev/workers/r2p2/bridge/bridge.py):

1. **Zero Transport Pollution**: Rust (`tonic` + `prost`) owns gRPC and Protobuf on the native side, while `protos_adapter.py` is pure stdlib Python. Neither `grpcio` nor `google.protobuf` is ever imported into `sys.modules`.
2. **PEP 420 Namespace-Preserving Module Eviction (`_clear_modules_from_path`)**:
- **Retains shared namespace parents**: Namespace packages without `__file__` (such as `azure` and `azurefunctions`) have no module initializer to re-run and dynamically follow `sys.path`; keeping them preserves already-attached customer child packages.
- **Evicts file-backed worker packages & descendants**: Any module whose `__file__` resolves under `workers_dir` is evicted together with all cached submodule descendants.
- **Detaches stale parent attributes**: When evicting `parent.child` from a retained `parent` namespace, `parent.__dict__.pop("child", None)` removes the stale attribute so `from parent import child` cleanly re-imports from the customer's `.python_packages`.
- **Protected bridge modules**: `bridge`, `protos_adapter`, and active runtime prefixes remain protected across specialization, followed by `importlib.invalidate_caches()`.

```mermaid
flowchart LR
Reload["FunctionEnvironmentReloadRequest"] --> Clear["_clear_modules_from_path(workers_dir)"]
Clear --> NS{"Module has __file__<br/>under workers_dir?"}
NS -->|"No (PEP 420 namespace:<br/>azure, azurefunctions)"| Keep["Keep namespace parent in sys.modules<br/>(search path tracks new sys.path)"]
NS -->|"Yes (file-backed worker pkg)"| Evict["Evict module + cached descendants<br/>& pop stale attr on retained parent"]
Keep --> Reorder["_prioritize_customer_dependencies(app_dir)"]
Evict --> Reorder
Reorder --> Invalidate["importlib.invalidate_caches()<br/>+ _select_runtime(app_dir)"]
```
109 changes: 109 additions & 0 deletions docs/architecture/ecosystem.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# 5-Repository Ecosystem Architecture

The Azure Functions Python stack is cleanly partitioned across **five GitHub repositories**, separating the `.NET` Script Host runtime, the language-neutral gRPC wire contract, the Python/Rust worker shells & runtimes, the customer-facing SDK, and optional framework/SDK extensions.

<div class="arch-atlas-mount" data-era="ecosystem"></div>

---

## Repository Boundaries & Cross-Repo Topology

<!-- BEGIN AUTO-GENERATED: ecosystem-reference -->

| Repository | PyPI Packages | Architectural Role |
| --- | --- | --- |
| [`Azure/azure-functions-host`](https://github.com/Azure/azure-functions-host) | — | .NET Functions Script Host runtime: event/trigger listeners, WorkerChannel gRPC server, Consumption specialization, scale probes, HTTP v2 proxy, and AppInsights/Kusto telemetry. |
| [`Azure/azure-functions-language-worker-protobuf`](https://github.com/Azure/azure-functions-language-worker-protobuf) | — | Canonical gRPC wire contract (`FunctionRpc.proto`, `settlement.proto`, `ClaimsIdentityRpc.proto`) shared between `azure-functions-host` and all language workers. |
| [`Azure/azure-functions-python-worker`](https://github.com/Azure/azure-functions-python-worker) | `azure-functions-runtime`, `azure-functions-runtime-v1` | Worker transports (monolith, proxy_worker, R2P2 Rust) and v1/v2 execution runtimes. |
| [`Azure/azure-functions-python-library`](https://github.com/Azure/azure-functions-python-library) | `azure-functions` | Customer-facing Python SDK: V2 @app decorators, Blueprints, Datum converters, Trigger/Binding types, Out[T], Context. |
| [`Azure/azure-functions-python-extensions`](https://github.com/Azure/azure-functions-python-extensions) | `azurefunctions-extensions-base`, `azurefunctions-extensions-http-fastapi`, `azurefunctions-extensions-bindings-blob`, `azurefunctions-extensions-bindings-servicebus`, `azurefunctions-agents-extensions-*` | Extension contracts: WebApp HTTP v2 streaming, SdkType deferred bindings, Settlement gRPC client, and Custom/Agent Runtime hooks. |

### Full 5-Repository Topology

```mermaid
flowchart TB
subgraph ce_host["Azure/azure-functions-host (.NET Script Host)"]
ne_host_chan["Script Host & WorkerChannel<br/>(functions-host)"]
ne_host_duplex["ServerDuplexChannel & ShMem<br/>(functions-host)"]
end
subgraph ce_proto["Azure/azure-functions-language-worker-protobuf (Contract)"]
ne_funcrpc["FunctionRpc.proto<br/>(worker-protobuf)"]
ne_settle["settlement.proto & ClaimsIdentity<br/>(worker-protobuf)"]
end
subgraph ce_worker["Azure/azure-functions-python-worker"]
subgraph ce_w_shell["Worker Transport Shells (workers/)"]
ne_shells["Transports: Monolith / Proxy / R2P2<br/>(python-worker)"]
end
subgraph ce_w_rt["Execution Runtimes (runtimes/v1 & v2)"]
ne_runtimes["azure-functions-runtime (v2 & v1)<br/>(python-worker)"]
end
end
subgraph ce_lib["Azure/azure-functions-python-library"]
ne_lib_dec["Decorators & Blueprints<br/>(python-library)"]
ne_lib_datum["meta.py Datum & Converters<br/>(python-library)"]
end
subgraph ce_ext["Azure/azure-functions-python-extensions"]
ne_ext_base["extensions-base (WebApp, SdkType, Runtime)<br/>(python-extensions)"]
ne_ext_pkgs["FastAPI, SDK Bindings & Agent Runtimes<br/>(python-extensions)"]
end
ne_host_chan -->|"dispatch / RpcLog sink"| ne_host_duplex
ne_host_duplex -->|"EventStream server"| ne_funcrpc
ne_host_chan -.->|"Settlement server"| ne_settle
ne_funcrpc -->|"prost / grpcio client"| ne_shells
ne_shells -->|"WorkerRequest / PyO3 FFI"| ne_runtimes
ne_runtimes -->|"index_function_app()"| ne_lib_dec
ne_runtimes -->|"Datum <-> Binding decode"| ne_lib_datum
ne_runtimes -->|"http_v2 / deferred SdkType"| ne_ext_base
ne_shells -.->|"entry_points(azurefunctions.runtimes)"| ne_ext_base
ne_settle -.->|"grpcClientType settlement"| ne_ext_base
ne_ext_base -->|"metaclass registration"| ne_ext_pkgs
```

### End-to-End 5-Repo Request Lifecycle

```mermaid
sequenceDiagram
participant host as functions-host
participant proto as worker-protobuf
participant shell as python-worker (Shell)
participant rt as python-worker (Runtime)
participant lib as library & extensions
host->>proto: WorkerChannel serializes event onto ServerDuplexChannel
proto->>shell: FunctionRpc.proto StreamingMessage frame over HTTP/2
shell->>rt: Dispatch WorkerInit / Invocation across seam
rt->>lib: Decode Datum (python-library) or SdkType/WebApp (python-extensions)
lib-->>host: InvocationResponse / RpcLog / Settlement back to Host
```

### Key Cross-Repository Components

| Component | Repository | Source Anchor | Responsibility |
| --- | --- | --- | --- |
| **Script Host & WorkerChannel** | `functions-host` | [`src/WebJobs.Script.Grpc/Channel/WorkerChannel.cs`](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.Grpc/Channel/WorkerChannel.cs) | .NET Script Host orchestrating triggers, worker process lifecycle, placeholder specialization, status probes, and RpcLog telemetry. |
| **ServerDuplexChannel & ShMem** | `functions-host` | [`src/WebJobs.Script.Grpc/Channel/ServerDuplexChannel.cs`](https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script.Grpc/Channel/ServerDuplexChannel.cs) | Host-side bidirectional gRPC `EventStream` endpoint and POSIX/Windows shared-memory data transfer. |
| **FunctionRpc.proto** | `worker-protobuf` | [`src/proto/FunctionRpc.proto`](https://github.com/Azure/azure-functions-language-worker-protobuf/blob/dev/src/proto/FunctionRpc.proto) | Canonical bidirectional gRPC protocol contract shared between `azure-functions-host` and all out-of-proc language workers. |
| **settlement.proto & ClaimsIdentity** | `worker-protobuf` | [`src/proto/servicebus/settlement.proto`](https://github.com/Azure/azure-functions-language-worker-protobuf/blob/dev/src/proto/servicebus/settlement.proto) | Protobuf service schema enabling Python extensions to settle ServiceBus messages directly against the Host. |
| **Transports: Monolith / Proxy / R2P2** | `python-worker` | [`workers/r2p2/src/main.rs:L231`](https://github.com/Azure/azure-functions-python-worker/blob/dev/workers/r2p2/src/main.rs#L231) | Version-selected worker shell managing Host gRPC stream, logging, and sys.path dependency isolation. |
| **azure-functions-runtime (v2 & v1)** | `python-worker` | [`runtimes/v2/azure_functions_runtime/handle_event.py:L69`](https://github.com/Azure/azure-functions-python-worker/blob/dev/runtimes/v2/azure_functions_runtime/handle_event.py#L69) | PyPI-published execution engines that index apps, decode/encode bindings, and run sync/async handlers. |
| **Decorators & Blueprints** | `python-library` | [`azure/functions/__init__.py`](https://github.com/Azure/azure-functions-python-library/blob/dev/azure/functions/__init__.py) | Defines V2 Python programming model (`@app.route`, `@app.queue_trigger`, `Blueprint`, MCP & Connector triggers). |
| **meta.py Datum & Converters** | `python-library` | [`azure/functions/meta.py`](https://github.com/Azure/azure-functions-python-library/blob/dev/azure/functions/meta.py) | Protobuf-agnostic `Datum` value container and trigger/binding converters (`HttpRequest`, `QueueMessage`, `Blob`). |
| **extensions-base (WebApp, SdkType, Runtime)** | `python-extensions` | [`azurefunctions-extensions-base/azurefunctions/extensions/base/runtime.py`](https://github.com/Azure/azure-functions-python-extensions/blob/dev/azurefunctions-extensions-base/azurefunctions/extensions/base/runtime.py) | Metaclass registries bridging HTTP v2 ASGI servers, SDK-type deferred bindings, and custom Agent Runtimes. |
| **FastAPI, SDK Bindings & Agent Runtimes** | `python-extensions` | [`azurefunctions-extensions-http-fastapi/README.md`](https://github.com/Azure/azure-functions-python-extensions/blob/dev/azurefunctions-extensions-http-fastapi/README.md) | Concrete extension packages for FastAPI HTTP streaming, Blob/Cosmos/EventHub/ServiceBus SDK types, and Agents. |

<!-- END AUTO-GENERATED: ecosystem-reference -->

---

## How Contracts Connect the 5 Repositories

1. **`azure-functions-host` ↔ `azure-functions-language-worker-protobuf` ↔ `azure-functions-python-worker`**:
- `WorkerChannel.cs` and `ServerDuplexChannel.cs` in `azure-functions-host` host the gRPC `FunctionRpc.EventStream` server defined by `FunctionRpc.proto`.
- Neither `azure-functions-host` nor `FunctionRpc.proto` needed protocol changes when Python moved from the `≤ 3.12` monolith to `3.13` `proxy_worker` or `3.15+` Rust `r2p2`.
2. **`azure-functions-python-worker` ↔ `azure-functions-python-library`**:
- `azure-functions` (`python-library`) defines `@app.*` decorators, `Blueprint`s, `Out[T]`, `Context`, and the protobuf-free `Datum(value, type)` intermediate representation (`azure/functions/meta.py`).
- R2P2's `native_invocation.py` constructs `Datum` objects directly from Rust `PyO3` tuples—meaning standard triggers and bindings execute with **zero Python protobuf dependencies**.
3. **`azure-functions-python-worker` ↔ `azure-functions-python-extensions`**:
- `azurefunctions-extensions-base` defines three metaclass extension seams:
- `WebApp` (`web.py`) — local loopback ASGI/WSGI HTTP v2 streaming (`azurefunctions-extensions-http-fastapi`).
- `SdkType` (`sdkType.py`) — deferred bindings hydrating rich Azure SDK clients (`BlobClient`, `ContainerClient`, Service Bus `ServiceBusReceivedMessage` + `settlement.proto` gRPC client).
- `RuntimeTrackerMeta` (`runtime.py`) — custom execution engines discovered via `entry_points(group='azurefunctions.runtimes')` when `PYTHON_ENABLE_AGENT_RUNTIME=1`.
Loading
Loading