Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 47 additions & 4 deletions docs/develop/dotnet/workers/serverless-workers/cloud-run.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import { ReleaseNoteHeader } from '@site/src/components';
On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other .NET Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific package.
A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).
Expand Down Expand Up @@ -133,7 +133,50 @@ public static string Process(IReadOnlyList<string> items)

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability {/* #add-observability */}
## Configure OpenTelemetry {/* #opentelemetry */}

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - .NET SDK](/develop/dotnet/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).
Run an OpenTelemetry Collector as a sidecar in the Worker Pool. Call `ApplyGoogleCloudRunOpenTelemetryDefaults()`
to configure tracing and Temporal Core metrics for OTLP export to the Collector at `localhost:4317`. Load the Client
connection settings with `ClientEnvConfig.LoadClientConnectOptions()`, then configure both plugins before connecting:

<!--SNIPSTART dotnet-cloud-run-->
[src/Gcp/CloudRun/Program.cs](https://github.com/temporalio/samples-dotnet/blob/main/src/Gcp/CloudRun/Program.cs)
```cs
// The Cloud Run Id plugin sets the client identity to "{instanceId}@{revision}" from Cloud Run
// metadata at connect time; every Worker created from the client inherits it.
connectOptions.Plugins = new ITemporalClientPlugin[] { new CloudRunIdPlugin() };

// ApplyGoogleCloudRunOpenTelemetryDefaults adds the tracing interceptor and a runtime exporting Core
// metrics + traces over OTLP to the collector sidecar; the returned handle owns the tracer provider.
// Both plugins configure the same connect options.
using var telemetry = connectOptions.ApplyGoogleCloudRunOpenTelemetryDefaults();

var client = await TemporalClient.ConnectAsync(connectOptions);
```
<!--SNIPEND-->

After the Worker stops, call `telemetry.FlushAsync(TimeSpan.FromSeconds(2))` to flush traces within the Cloud Run
termination window. For the Collector sidecar configuration and deployment steps, see the
[.NET Cloud Run sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Gcp/CloudRun).

## Set a Worker identity {/* #worker-identity */}

Use `CloudRunIdPlugin` to identify each Worker instance as `<instance-id>@<revision>`. The plugin reads Cloud Run environment variables and instance metadata when the Client connects, then Workers created from that Client inherit the identity.

Register the plugin on the Client options before connecting:

```csharp
using Temporalio.Client;
using Temporalio.Common.EnvConfig;
using Temporalio.Extensions.Gcp.CloudRun.Id;

// ...
var connectOptions = ClientEnvConfig.LoadClientConnectOptions();
connectOptions.Plugins = new ITemporalClientPlugin[] { new CloudRunIdPlugin() };
var client = await TemporalClient.ConnectAsync(connectOptions);
// ...
```

The [preceding example](#opentelemetry) registers `CloudRunIdPlugin` alongside the OpenTelemetry defaults.

The plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete example, see the [.NET Cloud Run sample](https://github.com/temporalio/samples-dotnet/tree/main/src/Gcp/CloudRun).
153 changes: 149 additions & 4 deletions docs/develop/go/workers/serverless-workers/cloud-run.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import { ReleaseNoteHeader } from '@site/src/components';
On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Go Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific package.
A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).
Expand Down Expand Up @@ -110,7 +110,152 @@ func MyActivity(ctx context.Context, input MyInput) (string, error) {

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability {/* #add-observability */}
## Configure OpenTelemetry {/* #opentelemetry */}

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Go SDK](/develop/go/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).
Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at `localhost:4317`. By default, it derives the service name from `OTEL_SERVICE_NAME`, `CLOUD_RUN_WORKER_POOL`, or `K_SERVICE`.

Create both Cloud Run plugins before connecting, then add them to the Client options. Client plugins that implement
`worker.Plugin` also apply to Workers created from that Client:

<!--SNIPSTART go-cloud-run {"selectedLines": ["24-43"]}-->
[gcp/cloudrun/worker/main.go](https://github.com/temporalio/samples-go/blob/main/gcp/cloudrun/worker/main.go)
```go
// ...
// Exports OTLP metrics and traces to the collector sidecar (defaults to localhost:4317).
otelPlugin, err := otel.NewPlugin(ctx, otel.PluginOptions{})
if err != nil {
log.Fatalln("Unable to create OpenTelemetry plugin", err)
}
// Sets the client identity to "<instanceID>@<revision>" from Cloud Run instance metadata.
idPlugin := id.NewCloudRunIDPlugin()

// A client plugin that also implements worker.Plugin is applied to workers automatically.
clientOptions, err := envconfig.LoadDefaultClientOptions()
if err != nil {
log.Fatalln("Unable to load Temporal client options", err)
}
clientOptions.Plugins = append(clientOptions.Plugins, otelPlugin, idPlugin)

c, err := client.Dial(clientOptions)
if err != nil {
log.Fatalln("Unable to create Temporal client", err)
}
log.Println("Client identity:", idPlugin.Metadata().Identity())
```
<!--SNIPEND-->

Configure the Collector sidecar to receive OTLP gRPC on `localhost:4317`, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

<!--SNIPSTART go-cloud-run-otel-collector-config-->
[gcp/cloudrun/otel-collector-config.yaml](https://github.com/temporalio/samples-go/blob/main/gcp/cloudrun/otel-collector-config.yaml)
```yaml
# Google-Built OpenTelemetry Collector sidecar: metrics -> Managed Service for Prometheus, traces -> Cloud Trace.
# Auth uses the worker-pool service account (ADC).
receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317

processors:
memory_limiter:
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
resource_detection:
detectors: [gcp]
timeout: 10s
# Batch traces for throughput. Do NOT batch the cumulative-metrics pipeline: a shutdown flush
# could be merged with a recent periodic export of the same series and rejected as a duplicate.
batch/traces:
send_batch_size: 200
timeout: 5s
# Rename Temporal datapoint labels that collide with the target labels Managed Service for Prometheus injects (e.g. namespace).
transform/collision:
metric_statements:
- context: datapoint
statements:
- set(attributes["exported_location"], attributes["location"])
- delete_key(attributes, "location")
- set(attributes["exported_cluster"], attributes["cluster"])
- delete_key(attributes, "cluster")
- set(attributes["exported_namespace"], attributes["namespace"])
- delete_key(attributes, "namespace")
- set(attributes["exported_job"], attributes["job"])
- delete_key(attributes, "job")
- set(attributes["exported_instance"], attributes["instance"])
- delete_key(attributes, "instance")
- set(attributes["exported_project_id"], attributes["project_id"])
- delete_key(attributes, "project_id")
# The Telemetry API expects the Google Cloud project in gcp.project_id.
transform/set_project_id:
error_mode: ignore
trace_statements:
- set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
- set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
googlemanagedprometheus:
otlp_grpc:
endpoint: telemetry.googleapis.com:443
compression: none
balancer_name: pick_first
auth:
authenticator: googleclientauth

extensions:
googleclientauth:
health_check:
endpoint: 0.0.0.0:13133

service:
extensions: [googleclientauth, health_check]
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, resource_detection, transform/collision]
exporters: [googlemanagedprometheus]
traces:
receivers: [otlp]
processors: [memory_limiter, resource_detection, transform/set_project_id, batch/traces]
exporters: [otlp_grpc]
telemetry:
logs:
level: info
```
<!--SNIPEND-->

On shutdown, stop the Worker and call `otelPlugin.Shutdown` with a deadline shorter than Cloud Run's termination window so telemetry can flush. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the [Go Cloud Run sample](https://github.com/temporalio/samples-go/tree/main/gcp/cloudrun).

## Set a Worker identity {/* #worker-identity */}

Use the Cloud Run ID plugin to identify each Worker instance as `<instance-id>@<revision>`. The plugin reads Cloud Run environment variables and instance metadata once when the Client connects, then Workers created from that Client inherit the identity.

Register the plugin on the Client options before connecting:

```go
import (
"log"

"go.temporal.io/sdk/client"
"go.temporal.io/sdk/contrib/envconfig"
"go.temporal.io/sdk/contrib/gcp/cloudrun/id"
)

// ...
clientOptions, err := envconfig.LoadDefaultClientOptions()
if err != nil {
log.Fatalln("Unable to load Temporal client options", err)
}
clientOptions.Plugins = append(clientOptions.Plugins, id.NewCloudRunIDPlugin())
c, err := client.Dial(clientOptions)
if err != nil {
log.Fatalln("Unable to create Temporal client", err)
}
defer c.Close()
// ...
```

The [preceding example](#opentelemetry) registers `id.NewCloudRunIDPlugin()` alongside the OpenTelemetry plugin.

The plugin requires the Cloud Run metadata server, so it fails when the Worker runs outside Cloud Run. For a complete example, see the [Go Cloud Run sample](https://github.com/temporalio/samples-go/tree/main/gcp/cloudrun).
147 changes: 143 additions & 4 deletions docs/develop/java/workers/serverless-workers/cloud-run.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import { ReleaseNoteHeader } from '@site/src/components';
On a [GCP Cloud Run worker pool](https://cloud.google.com/run/docs/resource-model#worker-pools), you run a standard long-lived Temporal Worker.
Register Workflows and Activities the same way you would with any other Java Worker, and Temporal Cloud scales the pool up and down as work arrives and drains.

A Cloud Run Worker needs no Cloud Run-specific package.
A Cloud Run Worker needs no Cloud Run-specific runtime or handler.
The one addition to a standard Worker is [Worker Versioning](/worker-versioning), which is required for Serverless Workers.

For the end-to-end deployment guide covering the Worker Pool, IAM, and compute configuration, see [Deploy a Serverless Worker on GCP Cloud Run](/production-deployment/worker-deployments/serverless-workers/cloud-run).
Expand Down Expand Up @@ -146,7 +146,146 @@ public class GreetingActivitiesImpl implements GreetingActivities {

For how scale-in decisions are made, see [Serverless Workers on GCP Cloud Run](/serverless-workers/cloud-run#lifecycle).

## Add observability {/* #add-observability */}
## Configure OpenTelemetry {/* #opentelemetry */}

A Cloud Run Worker emits the same traces and metrics as a Worker anywhere else.
For how to configure metrics export and OpenTelemetry tracing interceptors, see [Observability - Java SDK](/develop/java/platform/observability) and the [SDK metrics reference](/references/sdk-metrics).
Run an OpenTelemetry Collector as a sidecar in the Worker Pool. The Cloud Run OpenTelemetry plugin exports metrics and traces by OTLP gRPC to the Collector at `localhost:4317`. It configures the endpoint and service name from Cloud Run defaults.

Create both Cloud Run plugins and register them on `WorkflowServiceStubsOptions` before you create the Client:

<!--SNIPSTART java-cloud-run-worker-->
[gcp/cloud-run/src/main/java/io/temporal/samples/gcp/cloudrun/CloudRunWorker.java](https://github.com/temporalio/samples-java/blob/main/gcp/cloud-run/src/main/java/io/temporal/samples/gcp/cloudrun/CloudRunWorker.java)
```java
ClientConfigProfile profile = ClientConfigProfile.load();
CloudRunOpenTelemetryPlugin otelPlugin = CloudRunOpenTelemetryPlugin.newBuilder().build();
CloudRunIdPlugin idPlugin = new CloudRunIdPlugin();

WorkflowServiceStubsOptions serviceOptions =
WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions())
.setPlugins(otelPlugin, idPlugin)
.build();
WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceOptions);
```
<!--SNIPEND-->

Configure the Collector sidecar to receive OTLP gRPC on `localhost:4317`, export traces to Google Cloud, and export metrics to Google Managed Service for Prometheus:

<!--SNIPSTART java-cloud-run-collector-config-->
[gcp/cloud-run/collector-config.yaml](https://github.com/temporalio/samples-java/blob/main/gcp/cloud-run/collector-config.yaml)
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: localhost:4317

processors:
# Batch traces for throughput. Do not add this processor to the cumulative metrics pipeline:
# a shutdown flush can otherwise be batched with a recent periodic export of the same series.
batch/traces:
send_batch_max_size: 200
send_batch_size: 200
timeout: 5s
memory_limiter:
# This is the collector's memory polling cadence, not the SDK metric export interval.
check_interval: 1s
limit_percentage: 65
spike_limit_percentage: 20
resource_detection:
detectors: [gcp]
timeout: 10s
# Avoid collisions with labels that Google Managed Service for Prometheus adds.
transform/collision:
metric_statements:
- context: datapoint
statements:
- set(attributes["exported_location"], attributes["location"])
- delete_key(attributes, "location")
- set(attributes["exported_cluster"], attributes["cluster"])
- delete_key(attributes, "cluster")
- set(attributes["exported_namespace"], attributes["namespace"])
- delete_key(attributes, "namespace")
- set(attributes["exported_job"], attributes["job"])
- delete_key(attributes, "job")
- set(attributes["exported_instance"], attributes["instance"])
- delete_key(attributes, "instance")
- set(attributes["exported_project_id"], attributes["project_id"])
- delete_key(attributes, "project_id")
# The Telemetry API expects the Google Cloud project in gcp.project_id.
transform/set_project_id:
error_mode: ignore
trace_statements:
- set(resource.attributes["gcp.project_id"], resource.attributes["gcp.project.id"]) where resource.attributes["gcp.project.id"] != nil
- set(resource.attributes["gcp.project_id"], resource.attributes["cloud.account.id"]) where resource.attributes["gcp.project_id"] == nil and resource.attributes["cloud.account.id"] != nil

exporters:
googlemanagedprometheus:
# Google Cloud's supported OTLP path for traces is the Telemetry API.
otlp_grpc:
endpoint: telemetry.googleapis.com:443
compression: none
balancer_name: pick_first
auth:
authenticator: googleclientauth

extensions:
# Cloud Run container dependencies require a startup probe. This endpoint is also used for the
# collector liveness probe in worker-pool.yaml.
health_check:
endpoint: 0.0.0.0:13133
googleclientauth:

service:
extensions:
- health_check
- googleclientauth
pipelines:
metrics/otlp:
receivers: [otlp]
processors: [memory_limiter, resource_detection, transform/collision]
exporters: [googlemanagedprometheus]
traces:
receivers: [otlp]
processors: [memory_limiter, resource_detection, transform/set_project_id, batch/traces]
exporters: [otlp_grpc]
# Feed collector self-metrics back through the metrics pipeline.
telemetry:
metrics:
readers:
- periodic:
exporter:
otlp:
protocol: grpc
endpoint: http://localhost:4317
insecure: true
```
<!--SNIPEND-->

On shutdown, stop the `WorkerFactory`, then run `otelPlugin.newFlushHook()` before the Cloud Run termination window ends. For a Collector configuration that exports traces to Google Cloud and metrics to Google Managed Service for Prometheus, see the [Java Cloud Run sample](https://github.com/temporalio/samples-java/tree/main/gcp/cloud-run).

## Set a Worker identity {/* #worker-identity */}

Use `CloudRunIdPlugin` to identify each Worker instance as `<instance-id>@<revision>`. Register it on
`WorkflowServiceStubsOptions` before creating the service stubs. Clients and Workers created from those service stubs
inherit the identity:

```java
import io.temporal.client.WorkflowClient;
import io.temporal.envconfig.ClientConfigProfile;
import io.temporal.gcp.cloudrun.id.CloudRunIdPlugin;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;

// ...
ClientConfigProfile profile = ClientConfigProfile.load();
WorkflowServiceStubsOptions serviceOptions =
WorkflowServiceStubsOptions.newBuilder(profile.toWorkflowServiceStubsOptions())
.setPlugins(new CloudRunIdPlugin())
.build();
WorkflowServiceStubs service = WorkflowServiceStubs.newServiceStubs(serviceOptions);
WorkflowClient client = WorkflowClient.newInstance(service, profile.toWorkflowClientOptions());
// ...
```

The [preceding example](#opentelemetry) registers `CloudRunIdPlugin` alongside the OpenTelemetry plugin.

The plugin requires Cloud Run metadata, so Client creation fails when the Worker runs outside Cloud Run. For a complete example, see the [Java Cloud Run sample](https://github.com/temporalio/samples-java/tree/main/gcp/cloud-run).
Loading
Loading