Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@ to docs, or any other relevant information.
- Experimental Event Groups associate related workflow commands so their history events can be
viewed together. Create a group with `Workflow.newEventGroup` and attach it by calling
`addEventGroups` on the command's options.
- Added the experimental `temporal-aws-secrets-manager` module, a `temporal-envconfig` extension that loads the client
API key and mTLS material from AWS Secrets Manager secret IDs.
- `temporal-aws-lambda` now loads Temporal client credentials from AWS Secrets Manager when secret IDs are configured
with `TEMPORAL_EXT_AWS_SECRETS_MANAGER_*` environment variables or in the TOML config.

### Changed
- Updated Cloud API bindings to v0.27.0, including opt-in user group memberships, projects,
Expand Down
3 changes: 3 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ ext {

jsonPathVersion = '2.9.0' // compileOnly

// api in temporal-payload-storage-s3driver-awssdkv2, compileOnly in temporal-aws-secrets-manager
awsSdkVersion = '2.31.0'

cronUtilsVersion = '9.2.1' // for test server only

// This is the version of Spring Boot we use to _build the SDK_. This has to be in the
Expand Down
15 changes: 15 additions & 0 deletions contrib/temporal-aws-lambda/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,21 @@ public final class Handler implements RequestHandler<Object, Void> {

Connection options are loaded with `temporal-envconfig` when the handler is constructed during Lambda cold start. The Lambda worker checks `TEMPORAL_CONFIG_FILE` first, then `$LAMBDA_TASK_ROOT/temporal.toml`, then `./temporal.toml`, then falls back to the envconfig defaults and Temporal environment variables. The `configure` callback, passed as the second parameter to `LambdaWorker.define`, also runs during handler construction, so non-invocation configuration is prepared once and reused.

### Credentials from AWS Secrets Manager

To keep credentials out of environment variables and the Lambda package, store them in AWS Secrets Manager and configure only the secret IDs (names or ARNs). The [`temporal-aws-secrets-manager`](../temporal-aws-secrets-manager) envconfig extension is always enabled, so no code changes are needed:

| Environment variable | TOML (under `profile.<profile>.ext.aws_secrets_manager`) |
|---|---|
| `TEMPORAL_EXT_AWS_SECRETS_MANAGER_API_KEY_ID` | `api_key_id` |
| `TEMPORAL_EXT_AWS_SECRETS_MANAGER_TLS_CLIENT_CERT_ID` | `tls.client_cert_id` |
| `TEMPORAL_EXT_AWS_SECRETS_MANAGER_TLS_CLIENT_KEY_ID` | `tls.client_key_id` |
| `TEMPORAL_EXT_AWS_SECRETS_MANAGER_TLS_SERVER_CA_CERT_ID` | `tls.server_ca_cert_id` |

Secrets are fetched through the [AWS Parameters and Secrets Lambda Extension](https://docs.aws.amazon.com/secretsmanager/latest/userguide/retrieving-secrets_lambda.html) if the function has it. Otherwise they are fetched directly with the AWS SDK for Java 2.x, an optional dependency that you add yourself: add `software.amazon.awssdk:secretsmanager` to your function's dependencies. Either way they are cached across warm invocations. The Lambda execution role needs `secretsmanager:GetSecretValue` on the configured secrets. Client keys must be PKCS#8, as for envconfig. See the [`temporal-aws-secrets-manager` README](../temporal-aws-secrets-manager) for the full set of options, validation rules, caching behavior, and observability metrics.

When secret IDs are configured, the extension adds a plugin to the service stubs options. To add your own service stubs plugins, keep the existing ones rather than replacing them with `setPlugins`.

Use the per-invocation configuration overload when final options depend on the Lambda `Context` or on resources opened for one invocation. The cold-start `configure` callback passed as the second parameter to `LambdaWorker.define` still runs once; the invocation callback runs before Temporal service stubs, client, and worker are created:

```java
Expand Down
3 changes: 3 additions & 0 deletions contrib/temporal-aws-lambda/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,16 @@ dependencies {
api project(':temporal-opentelemetry')

implementation project(':temporal-envconfig')
implementation project(':temporal-aws-secrets-manager')
api "io.opentelemetry:opentelemetry-api"
implementation "io.opentelemetry.contrib:opentelemetry-aws-xray:$otelVersion"
implementation "org.slf4j:slf4j-api:$slf4jVersion"

testImplementation project(':temporal-sdk')
testImplementation "io.opentelemetry:opentelemetry-sdk"
testImplementation "junit:junit:${junitVersion}"
testImplementation platform("software.amazon.awssdk:bom:$awsSdkVersion")
testImplementation "software.amazon.awssdk:secretsmanager"

testRuntimeOnly group: 'ch.qos.logback', name: 'logback-classic', version: "${logbackVersion}"
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package io.temporal.aws.lambda;

import io.temporal.activity.DynamicActivity;
import io.temporal.aws.secretsmanager.SecretsManagerExtension;
import io.temporal.client.WorkflowClientOptions;
import io.temporal.common.VersioningBehavior;
import io.temporal.common.WorkerDeploymentVersion;
Expand Down Expand Up @@ -112,6 +113,8 @@ static Builder newBuilderFromEnvironment(Map<String, String> env, File cwd) thro
LoadClientConfigProfileOptions.newBuilder()
.setConfigFilePath(resolveConfigFilePath(env, cwd))
.setEnvOverrides(env)
.setExtensions(
Collections.singletonList(SecretsManagerExtension.getDefaultInstance()))
.build());
return new Builder(profile, env);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,6 @@
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.serviceclient.WorkflowServiceStubsPlugin;
import java.time.Duration;
import java.util.Arrays;
import java.util.Map;
import java.util.Objects;
import java.util.function.Consumer;
Expand Down Expand Up @@ -53,6 +52,11 @@ public static void configure(@Nonnull LambdaWorkerOptions.Builder options) {
* <p>By default this method creates an OpenTelemetry SDK with OTLP trace and metric exporters and
* AWS X-Ray-compatible trace ID generation. If {@link Builder#setOpenTelemetry(OpenTelemetry)} is
* used, the provided instance is used instead and exporters are not created.
*
* <p>The OpenTelemetry plugin is added before any service stubs plugins already in the options,
* so that its metrics scope is set before they run. For example, the AWS Secrets Manager
* extension's plugin fetches TLS secrets while the service stubs are configured, and reports
* those fetches to the metrics scope set at that point.
*/
public static void configure(
@Nonnull LambdaWorkerOptions.Builder options, @Nonnull Consumer<Builder> configure) {
Expand Down Expand Up @@ -213,7 +217,7 @@ void apply(LambdaWorkerOptions.Builder options) {
Objects.requireNonNull(options, "options");
applyLambdaDefaults();
OpenTelemetryPlugin plugin = delegate.build();
appendServiceStubsPlugin(options.getWorkflowServiceStubsOptionsBuilder(), plugin);
prependServiceStubsPlugin(options.getWorkflowServiceStubsOptionsBuilder(), plugin);
options.addShutdownHook(plugin.newFlushHook());
}

Expand Down Expand Up @@ -254,15 +258,20 @@ OpenTelemetry create(
IdGenerator idGenerator);
}

private static void appendServiceStubsPlugin(
/**
* Adds the plugin before the existing ones. Plugins configure the service stubs in order, so its
* metrics scope is set before the others run, for example before the AWS Secrets Manager
* extension's plugin reports its TLS secret fetches.
*/
private static void prependServiceStubsPlugin(
WorkflowServiceStubsOptions.Builder options, WorkflowServiceStubsPlugin plugin) {
WorkflowServiceStubsPlugin[] existing = options.build().getPlugins();
int existingLength = existing == null ? 0 : existing.length;
WorkflowServiceStubsPlugin[] plugins =
existingLength == 0
? new WorkflowServiceStubsPlugin[1]
: Arrays.copyOf(existing, existingLength + 1);
plugins[existingLength] = Objects.requireNonNull(plugin, "plugin");
WorkflowServiceStubsPlugin[] plugins = new WorkflowServiceStubsPlugin[existingLength + 1];
plugins[0] = Objects.requireNonNull(plugin, "plugin");
if (existingLength > 0) {
System.arraycopy(existing, 0, plugins, 1, existingLength);
}
options.setPlugins(plugins);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import io.temporal.common.VersioningBehavior;
import io.temporal.common.WorkerDeploymentVersion;
import io.temporal.common.converter.EncodedValues;
import io.temporal.serviceclient.WorkflowServiceStubsPlugin;
import io.temporal.worker.WorkerDeploymentOptions;
import io.temporal.worker.WorkerFactoryOptions;
import io.temporal.worker.WorkerOptions;
Expand Down Expand Up @@ -310,6 +311,26 @@ public void dynamicRegistrationMethodsRejectNullInputs() throws IOException {
NullPointerException.class, () -> options.registerDynamicActivityImplementation(null));
}

@Test
public void secretsManagerPluginIsAddedWhenSecretIdsAreConfigured() throws IOException {
Map<String, String> env = baseEnv();
env.put("TEMPORAL_EXT_AWS_SECRETS_MANAGER_API_KEY_ID", "temporal/api-key");
WorkflowServiceStubsPlugin[] plugins =
LambdaWorkerOptions.newBuilderFromEnvironment(env)
.getWorkflowServiceStubsOptionsBuilder()
.build()
.getPlugins();
assertEquals(1, plugins.length);
assertEquals("io.temporal.aws.secretsmanager", plugins[0].getName());

WorkflowServiceStubsPlugin[] noPlugins =
LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv())
.getWorkflowServiceStubsOptionsBuilder()
.build()
.getPlugins();
assertTrue(noPlugins == null || noPlugins.length == 0);
}

private Map<String, String> baseEnv() {
Map<String, String> env = new HashMap<>();
env.put(LambdaWorkerOptions.TEMPORAL_CONFIG_FILE, "/nonexistent/temporal.toml");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.uber.m3.tally.NoopScope;
import com.uber.m3.tally.Scope;
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.metrics.MeterBuilder;
import io.opentelemetry.api.metrics.MeterProvider;
Expand All @@ -17,6 +19,8 @@
import io.temporal.common.WorkerDeploymentVersion;
import io.temporal.opentelemetry.OpenTelemetryPlugin;
import io.temporal.opentelemetry.TimedShutdownHook;
import io.temporal.serviceclient.WorkflowServiceStubs;
import io.temporal.serviceclient.WorkflowServiceStubsOptions;
import io.temporal.serviceclient.WorkflowServiceStubsPlugin;
import java.io.Closeable;
import java.time.Duration;
Expand All @@ -26,6 +30,9 @@
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;
import javax.annotation.Nonnull;
import org.junit.Test;

public class OtelLambdaWorkerConfigurationHelperTest {
Expand Down Expand Up @@ -129,6 +136,55 @@ public void pluginInstallsMetricsScopeAndTracingInterceptors() throws Exception
1, options.getWorkerFactoryOptionsBuilder().build().getWorkerInterceptors().length);
}

@Test
public void pluginComesBeforeExistingPluginsSoTheySeeItsMetricsScope() throws Exception {
LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv());
// Stands in for a plugin that reports to the metrics scope while configuring the stubs, like
// the AWS Secrets Manager extension's plugin from the envconfig profile
AtomicReference<Scope> seenScope = new AtomicReference<>();
WorkflowServiceStubsPlugin existing =
new WorkflowServiceStubsPlugin() {
@Nonnull
@Override
public String getName() {
return "test.existing";
}

@Override
public void configureServiceStubs(@Nonnull WorkflowServiceStubsOptions.Builder builder) {
seenScope.set(builder.build().getMetricsScope());
}

@Nonnull
@Override
public WorkflowServiceStubs connectServiceClient(
@Nonnull WorkflowServiceStubsOptions options,
@Nonnull Supplier<WorkflowServiceStubs> next) {
return next.get();
}
};
options.getWorkflowServiceStubsOptionsBuilder().setPlugins(existing);

OtelLambdaWorkerConfigurationHelper.configure(
options, builder -> builder.setOpenTelemetry(OpenTelemetry.noop()).setFlushHook(() -> {}));

WorkflowServiceStubsPlugin[] plugins =
options.getWorkflowServiceStubsOptionsBuilder().build().getPlugins();
assertEquals(2, plugins.length);
assertTrue(plugins[0] instanceof OpenTelemetryPlugin);
assertSame(existing, plugins[1]);

// Configure in plugin order, as WorkflowServiceStubs.newServiceStubs does
WorkflowServiceStubsOptions.Builder stubsOptions =
WorkflowServiceStubsOptions.newBuilder(
options.getWorkflowServiceStubsOptionsBuilder().build());
for (WorkflowServiceStubsPlugin plugin : plugins) {
plugin.configureServiceStubs(stubsOptions);
}
assertNotNull(seenScope.get());
assertFalse(seenScope.get() instanceof NoopScope);
}

@Test
public void configureRegistersPluginAndPerInvocationFlushHook() throws Exception {
LambdaWorkerOptions.Builder options = LambdaWorkerOptions.newBuilderFromEnvironment(baseEnv());
Expand Down
Loading
Loading