diff --git a/flash/apps/deploy-apps.mdx b/flash/apps/deploy-apps.mdx
index a2cf29154..4ae9de94f 100644
--- a/flash/apps/deploy-apps.mdx
+++ b/flash/apps/deploy-apps.mdx
@@ -468,16 +468,16 @@ You can also override the environment:
FLASH_APP=my-app FLASH_ENV=production python call_inference.py
```
-### Error without context
+### Endpoint not found
-If Flash cannot resolve the app context and you haven't set the environment variables, it raises an error:
+If no endpoint with that name is deployed in the resolved app and environment, Flash raises an error:
```text
-RuntimeError: no flash context for endpoint 'inference'. either:
- - use 'flash dev' for local development
- - set FLASH_APP and FLASH_ENV to target a deployed environment
+RuntimeError: endpoint 'inference' not found in app 'my-app' environment 'production'. deploy it first with 'flash deploy'.
```
+Deploy the app with `flash deploy`, or set `FLASH_APP` and `FLASH_ENV` to target an app you've already deployed.
+
### Automatic context in deployed workers
When Flash deploys your app, it automatically sets `FLASH_APP` and `FLASH_ENV` environment variables on each worker. This enables cross-endpoint communication within your deployed application without additional configuration.
diff --git a/flash/cli/build.mdx b/flash/cli/build.mdx
index e658d18bc..bddc66aca 100644
--- a/flash/cli/build.mdx
+++ b/flash/cli/build.mdx
@@ -50,14 +50,14 @@ Comma-separated list of packages to exclude from the build (e.g., `torch,torchvi
-Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). Overrides per-resource `python_version` declarations and local interpreter detection.
+Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). Defaults to the `python_version` declared on your resource configs, or 3.12 if none declares one. The build fails if the flag conflicts with a declared `python_version`.
## What happens during build
-1. **Python version resolution**: Resolves the target Python version from CLI flag, resource configs, or your local interpreter.
-2. **Function discovery**: Finds all `@Endpoint` decorated functions.
-3. **Grouping**: Groups functions by their endpoint configuration.
+1. **Function discovery**: Finds all `@Endpoint` decorated functions.
+2. **Grouping**: Groups functions by their endpoint configuration.
+3. **Python version resolution**: Resolves the target Python version from the CLI flag, resource configs, or the 3.12 default.
4. **Manifest generation**: Creates `.flash/flash_manifest.json` with endpoint definitions.
5. **Source fingerprinting**: Computes a SHA-256 fingerprint of your source files to detect code changes between deployments.
6. **Dependency installation**: Installs Python packages for Linux x86_64.
@@ -116,36 +116,32 @@ Flash automatically handles cross-platform builds:
### Python version in deployed workers
-Flash workers support Python 3.10, 3.11, 3.12, and 3.13. The target version is determined by:
+Flash workers support Python 3.10, 3.11, 3.12, and 3.13. Your local Python version doesn't change the worker version. Flash picks the worker version in this order:
-1. **CLI flag:** The `--python-version` flag takes precedence.
-2. **Resource config:** The `python_version` parameter on your endpoint configs.
-3. **Local interpreter:** Your local Python version (from `sys.version_info`) when neither is specified.
+1. **CLI flag:** `--python-version`.
+2. **Resource config:** the `python_version` field on a resource config class such as `LiveServerless`. The `@Endpoint` decorator has no `python_version` parameter.
+3. **Default:** Python 3.12.
-All resources in a Flash app must use the same Python version because Flash ships a single tarball for the entire app. If resources declare conflicting versions, the build fails.
-
-
-**Breaking change in Flash 0.x:** Flash now matches your local Python version by default instead of always defaulting to Python 3.12. If you upgrade Flash and your local Python differs from 3.12, your first deploy will trigger a rolling release. To maintain consistent behavior across team members, declare `python_version` explicitly on your resource configs or use `--python-version` on the CLI.
-
+All endpoints in an app use the same version because Flash ships a single tarball for the entire app. The build fails if resources declare different versions, or if `--python-version` conflicts with a version a resource declares.
| Version | Cold start | Notes |
|---------|------------|-------|
| 3.12 | No overhead | PyTorch pre-installed in base image |
-| 3.13 | +~7 GB on GPU | Alternative Python install required |
-| 3.11 | +~7 GB on GPU | Alternative Python install required |
+| 3.13 | +~7 GB on GPU | PyTorch reinstalled for this interpreter |
+| 3.11 | +~7 GB on GPU | PyTorch reinstalled for this interpreter |
| 3.10 | +~7 GB on GPU | EOL 2026-10-31; consider migrating to 3.11+ |
-If your local Python version is not supported (for example, 3.9 or 3.14), the build fails with an actionable error message listing the supported versions.
+You can run `flash build` from any local Python version from 3.10 to 3.14. Flash downloads dependency wheels for the worker's Python version, not your local one.
Image tags follow the pattern `py{version}-{tag}` (for example, `runpod/flash:py3.12-latest`).
## Managing deployment size
-Runpod Serverless has a **1.5GB deployment limit**. Flash automatically excludes packages that are pre-installed in the base image:
+Runpod Serverless has a **1.5GB deployment limit**. Flash automatically leaves these packages out of the archive because the GPU worker images provide them:
- `torch`, `torchvision`, `torchaudio`, `triton`
-These packages are excluded at archive time, so you don't need to specify them manually.
+You don't need to exclude them manually. CPU worker images don't include them, so code on CPU workers can't import these packages.
### Manual exclusions
diff --git a/flash/cli/deploy.mdx b/flash/cli/deploy.mdx
index 16546d83f..035a57540 100644
--- a/flash/cli/deploy.mdx
+++ b/flash/cli/deploy.mdx
@@ -68,7 +68,7 @@ Build and launch a local Docker-based preview environment instead of deploying t
-Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). Overrides per-resource `python_version` declarations and local interpreter detection.
+Target Python version for worker images (3.10, 3.11, 3.12, or 3.13). Defaults to the `python_version` declared on your resource configs, or 3.12 if none declares one. The build fails if the flag conflicts with a declared `python_version`.
## What happens during deployment
@@ -222,7 +222,7 @@ This builds your project and runs it in Docker containers locally:
## Managing deployment size
-Runpod Serverless has a **1.5GB limit**. Flash automatically excludes packages that are pre-installed in the base image (`torch`, `torchvision`, `torchaudio`, `numpy`, `triton`).
+Runpod Serverless has a **1.5GB limit**. Flash automatically leaves `torch`, `torchvision`, `torchaudio`, and `triton` out of the upload because the GPU worker images provide them. CPU workers don't include them, so CPU code can't import these packages.
If the deployment is still too large, use `--exclude` to skip additional packages:
diff --git a/flash/configuration/parameters.mdx b/flash/configuration/parameters.mdx
index 7ec4e84cb..289da73af 100644
--- a/flash/configuration/parameters.mdx
+++ b/flash/configuration/parameters.mdx
@@ -35,7 +35,6 @@ This page provides a complete reference for all parameters available on the `End
| `scaler_value` | `int` | Scaling threshold | `4` |
| `template` | `PodTemplate` | Pod template overrides | `None` |
| `min_cuda_version` | `str` or `CudaVersion` | Minimum CUDA version for GPU host selection | `"12.8"` (GPU) / `None` (CPU) |
-| `python_version` | `str` | Python version for the worker image | Local Python |
## Parameter details
@@ -603,52 +602,25 @@ This parameter has no effect on CPU endpoints.
Valid CUDA versions: `CudaVersion.V11_1`, `V11_4`, `V11_7`, `V11_8`, `V12_0`, `V12_1`, `V12_2`, `V12_3`, `V12_4`, `V12_6`, `V12_8` (or equivalent strings like `"12.4"`). Invalid values raise a `ValueError`.
-### python_version
+### Worker Python version
-**Type**: `str`
-**Default**: Local Python version
-
-Sets the Python version for the worker image. Supported values: `"3.10"`, `"3.11"`, `"3.12"`, and `"3.13"`.
-
-When you don't specify a Python version, Flash matches your local interpreter (the Python version you run Flash from). The resolution order is:
-
-1. `--python-version` CLI flag (highest priority)
-2. `python_version` declared on resource configs
-3. Your local Python version (`sys.version_info`)
-
-```python
-from runpod_flash import Endpoint, GpuGroup
-
-# Explicitly set Python 3.11 for this endpoint
-@Endpoint(
- name="legacy-model",
- gpu=GpuGroup.ANY,
- python_version="3.11"
-)
-async def process(data): ...
+`Endpoint` doesn't take a `python_version` parameter. Workers run Python 3.12 by default, whatever your local version. To target another version, pass `--python-version` to `flash build` or `flash deploy`:
-# Uses your local Python version (e.g., 3.12 if that's what you're running)
-@Endpoint(name="modern-model", gpu=GpuGroup.ANY)
-async def infer(data): ...
+```bash
+flash deploy --python-version 3.11
```
-All resources in a Flash app must use the same Python version because Flash ships a single tarball for the entire app. If resources declare conflicting versions, the build fails.
+Supported values are `3.10`, `3.11`, `3.12`, and `3.13`. The flag applies to every endpoint in the app because Flash ships a single tarball for the entire app.
-**Breaking change:** Flash now matches your local Python version by default instead of always defaulting to Python 3.12. If your local Python differs from 3.12, your first deploy after upgrading Flash will trigger a rolling release. For consistent behavior across team members, declare `python_version` explicitly or use the `--python-version` CLI flag.
-
-
-
-Python 3.10, 3.11, and 3.13 workers incur approximately 7 GB of additional cold-start overhead on GPU endpoints because the alternative Python interpreter must be installed alongside the base image's PyTorch environment.
+Python 3.10, 3.11, and 3.13 workers incur approximately 7 GB of additional cold-start overhead on GPU endpoints because the GPU image's PyTorch is built for Python 3.12, so PyTorch and its CUDA packages are installed again for the selected interpreter.
Python 3.10 reaches end-of-life on 2026-10-31. Consider migrating to Python 3.11 or later.
-If your local Python version is not supported (for example, 3.9 or 3.14), the build fails with an actionable error message listing the supported versions.
-
-The `--python-version` CLI flag on `flash build` and `flash deploy` overrides both per-resource declarations and local interpreter detection.
+You can run Flash from any local Python version from 3.10 to 3.14.
## EndpointJob
@@ -690,7 +662,7 @@ These changes restart all workers:
- Datacenter (`datacenter`)
- Flashboot setting (`flashboot`)
- CUDA version requirement (`min_cuda_version`)
-- Python version (`python_version`)
+- Worker Python version (`--python-version`)
Workers are temporarily unavailable during recreation (typically 30-90 seconds).
diff --git a/flash/overview.mdx b/flash/overview.mdx
index 74350205b..25a3efb08 100644
--- a/flash/overview.mdx
+++ b/flash/overview.mdx
@@ -55,7 +55,7 @@ Flash requires a Runpod account with a verified email address.
### Install Flash
-Flash requires [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/) and runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2).
+Flash requires [Python 3.10 to 3.14](https://www.python.org/downloads/) and runs natively on macOS and Linux. Windows users can run Flash through [WSL2](/flash/windows-wsl2).
Install Flash using `pip` or `uv`:
diff --git a/flash/quickstart.mdx b/flash/quickstart.mdx
index 9bce0f573..7a929c310 100644
--- a/flash/quickstart.mdx
+++ b/flash/quickstart.mdx
@@ -15,7 +15,7 @@ This quickstart gets you running GPU workloads on Runpod in minutes. You'll exec
- [Runpod account](/accounts-billing/manage-accounts) with a verified email address.
- [An API key](/get-started/credentials#api-keys) with **All** access permissions to your Runpod account.
-- [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/) installed.
+- [Python 3.10 to 3.14](https://www.python.org/downloads/) installed. Workers run Python 3.12 by default, whatever your local version.
- [uv](https://docs.astral.sh/uv/) installed.
## Step 1: Install Flash
@@ -78,26 +78,25 @@ from runpod_flash import Endpoint, GpuGroup
gpu=GpuGroup.ANY, # Use any available GPU
workers=3,
idle_timeout=300, # Keep worker running for 5 minutes
- dependencies=["numpy", "torch"]
+ dependencies=["torch"]
)
def gpu_matrix_multiply(size):
# IMPORTANT: Import packages INSIDE the function
- import numpy as np
import torch
# Get GPU name
device_name = torch.cuda.get_device_name(0)
- # Create random matrices
- A = np.random.rand(size, size)
- B = np.random.rand(size, size)
+ # Create random matrices on the GPU
+ A = torch.rand(size, size, device="cuda")
+ B = torch.rand(size, size, device="cuda")
- # Multiply matrices
- C = np.dot(A, B)
+ # Multiply matrices on the GPU
+ C = A @ B
return {
"matrix_size": size,
- "result_mean": float(np.mean(C)),
+ "result_mean": C.mean().item(),
"gpu": device_name
}
@@ -118,32 +117,71 @@ if __name__ == "__main__":
Make sure you activate your virtual environment in the same directory where you created the `gpu_demo.py` file. If you open a new terminal, run `source .venv/bin/activate` before executing the script.
-## Step 4: Run it
+## Step 4: Deploy the endpoint
-Execute the script:
+From the directory that contains `gpu_demo.py`, deploy the endpoint:
+
+```bash
+flash deploy
+
+# If using uv:
+uv run flash deploy
+```
+
+Flash builds your project on your machine, uploads it, and creates the `flash-quickstart` endpoint. The app is named after the current directory, and the endpoint is deployed to the `production` environment:
+
+```text
+✓ built quickstart 1 files, 0 deps, 0.2 MB
+No app 'quickstart' found. Creating app and 'production' environment...
+
+✓ uploaded 0.2 MB 1.3s
+✓ deployed to production 2.3s
+
+ flash-quickstart https://api.runpod.ai/v2/abc123xyz/runsync
+
+curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer $RUNPOD_API_KEY" \
+ -d '{"input": {}}'
+```
+
+The URL contains your endpoint ID (`abc123xyz` here). You'll use it in [Step 9](#step-9-call-the-endpoint-from-any-language) to call the endpoint over HTTP. Don't run the printed `curl` example as is in 1.21.0. Step 9 explains why.
+
+
+Every `flash` command and script prints a deprecation notice (a `FutureWarning` in scripts) because `runpod-flash` is moving into the [Runpod SDK](/sdk/overview). The notice doesn't affect your deployment.
+
+
+
+`flash deploy` installs your endpoint's dependencies into a local build directory before uploading. Flash leaves `torch`, `torchvision`, `torchaudio`, and `triton` out of the upload because the worker image already provides them, but a dependency that requires `torch` (such as `accelerate`) still downloads it during the local build. On Linux, that can be several GB. If the build runs out of disk space, remove that dependency, or exclude it with `--exclude `. `--no-deps` also skips transitive dependencies, but it skips them for every package, so use it only if you list every dependency your code needs yourself. See [`flash deploy`](/flash/cli/deploy) for details.
+
+
+## Step 5: Run it
+
+Execute the script from the same directory:
```bash
python gpu_demo.py
```
-You'll see Flash provision a GPU worker and execute your function:
+Flash sends the call to your deployed endpoint and prints the result:
```text
Running matrix multiplication on Runpod GPU...
-Creating endpoint: flash-quickstart
-Provisioning Serverless endpoint...
-Endpoint ready
-Executing function on Runpod endpoint ID: xvf32dan8rcilp
-Initial job status: IN_QUEUE
-Job completed, output received
✓ Matrix size: 1000x1000
✓ Result mean: 249.8286
✓ GPU used: NVIDIA RTX A5000
```
-The first run takes 30-60 seconds, while Runpod provisions the endpoint, installs dependencies, and starts a worker. Subsequent runs take 2-3 seconds (because the worker is already running).
+The first run takes about a minute while a worker starts. Later runs return in a few seconds while the worker is still warm.
+
+
+The first request on a new worker can take longer than the 90 seconds `runpod-flash` 1.21.0 waits for a result, especially if the worker has to wait for a free GPU. If the script fails with `timed out after 90s. the endpoint may not be deployed or the worker is still starting` or `KeyError: 'matrix_size'`, the endpoint is deployed and the worker is still starting. Run the script again. See [First run fails after 90 seconds](#first-run-fails-after-90-seconds).
+
+
+If you see `endpoint 'flash-quickstart' not found in app '...' environment 'production'`, the endpoint isn't deployed for this directory. Run `flash deploy` from the directory that contains `gpu_demo.py`, then run the script again from that same directory.
+
If you're having authorization issues, you can set your API key directly in your terminal:
@@ -152,12 +190,12 @@ If you're having authorization issues, you can set your API key directly in your
export RUNPOD_API_KEY="your_key"
```
-Replace `your_key` with your actual API key from the [Credentials page](https://console.runpod.io/user/credentials).
+Replace `your_key` with your actual API key from the [Runpod console](https://console.runpod.io/user/settings).
-## Step 5: Update and run again
+## Step 6: Update and run again
-With your endpoint running, make a change and run the script again:
+With your endpoint running, change an input and run the script again:
1. Open `gpu_demo.py` and change the matrix size from `1000` to `2000`:
@@ -171,11 +209,11 @@ With your endpoint running, make a change and run the script again:
python gpu_demo.py
```
-This time, the result should appear in 1-3 seconds instead of 30-60 seconds, injects the code into the running worker so code changes take effect immediately without reprovisioning.
+This time, the result should appear in a few seconds because the worker is already warm. The argument is sent with the request, so you don't need to redeploy.
-This instant iteration is one of Flash's key features. You can develop and test GPU code as quickly as local development, even though it runs on remote hardware.
+If you change the code inside `gpu_matrix_multiply` or its `@Endpoint` configuration, run `flash deploy` again so the endpoint picks up the new version. Workers that are already warm can keep serving the previous version for a few minutes while new workers replace them. If a result doesn't reflect your change, or fails with `unexpected keyword argument` after you added a parameter, wait a minute and run the script again.
-## Step 6: Understand what you just did
+## Step 7: Understand what you just did
Let's break down the code you just ran:
@@ -200,25 +238,24 @@ Flash automatically loads your credentials from `flash login` or the `RUNPOD_API
gpu=GpuGroup.ANY,
workers=3,
idle_timeout=300,
- dependencies=["numpy", "torch"]
+ dependencies=["torch"]
)
def gpu_matrix_multiply(size):
- import numpy as np
import torch
# Get GPU name
device_name = torch.cuda.get_device_name(0)
- # Create random matrices
- A = np.random.rand(size, size)
- B = np.random.rand(size, size)
+ # Create random matrices on the GPU
+ A = torch.rand(size, size, device="cuda")
+ B = torch.rand(size, size, device="cuda")
- # Multiply matrices
- C = np.dot(A, B)
+ # Multiply matrices on the GPU
+ C = A @ B
return {
"matrix_size": size,
- "result_mean": float(np.mean(C)),
+ "result_mean": C.mean().item(),
"gpu": device_name
}
```
@@ -229,8 +266,8 @@ The `@Endpoint` decorator configures everything in one place:
- **`gpu`**: Which GPU to use (`GpuGroup.ANY` accepts any available GPU for faster provisioning).
- **`workers`**: Maximum parallel workers (allows 3 concurrent executions).
- **`idle_timeout`**: Seconds a worker stays active after completing a request before scaling down. Setting this to 300 (5 minutes) gives you more time to iterate on your code while the worker remains warm.
-- **`dependencies`**: Python packages to install on the worker.
-- **Function body**: The matrix multiplication code runs on the remote GPU, not your local machine.
+- **`dependencies`**: Python packages your function needs. `torch` is already in the worker image, so Flash leaves it out of the upload.
+- **Function body**: Runs on the remote worker, not your local machine. `device="cuda"` creates the matrices in GPU memory, so the multiplication runs on the GPU.
- **Return value**: The result is returned to your local machine as a Python dictionary.
See [GPU types](/flash/configuration/gpu-types) for available GPUs or [endpoint functions](/flash/create-endpoints) for all configuration options.
@@ -256,18 +293,16 @@ if __name__ == "__main__":
Here's what happens when you call an `@Endpoint` decorated function:
-1. Flash checks if the endpoint specified in your decorator already exists.
- - If yes: It updates the endpoint if the configuration has changed.
- - If no: It creates a new endpoint, initializes a worker, and installs your dependencies.
-2. Flash sends your code to the GPU worker
-3. The GPU worker executes the function with the provided inputs.
+1. Flash looks up the endpoint by its `name` in your app (the current directory name) and environment (`production` by default). `flash deploy` must have created it first.
+2. Flash sends the function's arguments to the endpoint.
+3. A GPU worker runs the deployed function with those arguments.
4. The result is returned to your local machine as a Python dictionary, where it's printed in your terminal.
Everything outside the `@Endpoint` function (all the `print` statements, etc.) runs **locally on your machine**. Only the decorated function runs remotely.
-## Step 7: Run multiple operations in parallel
+## Step 8: Run multiple operations in parallel
-Flash makes it easy to run multiple GPU operations concurrently. Replace your `main()` function with the code below:
+Flash makes it easy to run multiple GPU operations concurrently. Replace your `main()` function with the code below, and keep the `if __name__ == "__main__":` block at the end of the file. Without it, the script exits without printing anything.
```python
async def main():
@@ -285,24 +320,21 @@ async def main():
print(f"\n{i}. Size: {result['matrix_size']}x{result['matrix_size']}")
print(f" Mean: {result['result_mean']:.4f}")
print(f" GPU: {result['gpu']}")
+
+if __name__ == "__main__":
+ asyncio.run(main())
```
-Run the script again:
+You only changed code outside the `@Endpoint` function, so you don't need to redeploy. Run the script again:
```bash
python gpu_demo.py
```
-All three operations execute simultaneously:
+All three requests are sent at the same time. They run in parallel when several workers are up (`workers=3`), or one after another on a single warm worker:
```text
Running 3 matrix operations in parallel...
-Initial job status: IN_QUEUE
-Initial job status: IN_QUEUE
-Initial job status: IN_QUEUE
-Job completed, output received
-Job completed, output received
-Job completed, output received
1. Size: 500x500
Mean: 125.3097
@@ -317,6 +349,52 @@ Job completed, output received
GPU: NVIDIA RTX A5000
```
+## Step 9: Call the endpoint from any language
+
+Your endpoint is a standard [Serverless endpoint](/serverless/endpoints/send-requests), so any language that can send HTTP requests can call it. The JSON `input` object holds the function's arguments by name.
+
+`flash login` stores your API key for Flash, but `curl` needs it in an environment variable. Export it first, replacing `your_key` with your API key from the [Runpod console](https://console.runpod.io/user/settings):
+
+```bash
+export RUNPOD_API_KEY="your_key"
+```
+
+Submit a job with `/run`, using the endpoint ID from Step 4:
+
+```bash
+curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/run \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer $RUNPOD_API_KEY" \
+ -d '{"input": {"size": 500}}'
+```
+
+```json
+{"id": "50e2248c-55c3-4f61-b633-35e447a4d82d-u2", "status": "IN_QUEUE"}
+```
+
+Then poll `/status` with the job ID until `status` is `COMPLETED`:
+
+```bash
+curl https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID \
+ -H "Authorization: Bearer $RUNPOD_API_KEY"
+```
+
+```json
+{
+ "id": "50e2248c-55c3-4f61-b633-35e447a4d82d-u2",
+ "status": "COMPLETED",
+ "output": {"gpu": "NVIDIA RTX A5000", "matrix_size": 500, "result_mean": 124.87},
+ "delayTime": 15,
+ "executionTime": 189
+}
+```
+
+Use `/run` and `/status` for long jobs and cold starts. `/runsync` returns the result in one request but waits at most 90 seconds by default, then returns `IN_QUEUE` or `IN_PROGRESS` and you poll `/status` anyway.
+
+
+Always send at least one key in `input`. Runpod drops a job whose `input` is empty (`{"input": {}}`), and after about 40 seconds it fails with `job timed out after 1 retries`. The curl example that `flash deploy` prints in `runpod-flash` 1.21.0 and earlier uses an empty `input`, so replace it with your function's arguments.
+
+
## Clean up
When you're done testing, clean up the endpoints:
@@ -358,10 +436,63 @@ You've successfully run GPU code on Runpod! Now you're ready to learn more about
## Troubleshooting
+### Endpoint not found
+
+```text
+RuntimeError: endpoint 'flash-quickstart' not found in app 'quickstart' environment 'production'. deploy it first with 'flash deploy'.
+```
+
+**Solution**: Calling an `@Endpoint` function sends the request to an endpoint you've already deployed. Run `flash deploy` from the directory that contains your script, then run the script from that same directory. Flash uses the directory name as the app name, so running from a different directory looks up a different app.
+
+### First run fails after 90 seconds
+
+**Issue**: The first run fails with one of these errors:
+
+```text
+RuntimeError: request to endpoint 'flash-quickstart' timed out after 90s. the endpoint may not be deployed or the worker is still starting. deploy with 'flash deploy' or increase timeout with FLASH_SENTINEL_TIMEOUT env var.
+```
+
+```text
+KeyError: 'matrix_size'
+```
+
+**Cause**: `runpod-flash` 1.21.0 waits up to 90 seconds for a result. The first request on a new worker can take longer: the worker may wait for a free GPU, then it starts and runs your function. The endpoint is deployed, and the request keeps running after the error.
+
+**Solution**: Run the script again. The worker keeps starting after the error, so the next run is faster. Raising `FLASH_SENTINEL_TIMEOUT` above 90 doesn't help in 1.21.0: the request returns at 90 seconds with the job still running, and the script fails with the `KeyError` instead.
+
+To see whether a worker is up, check the endpoint's health with its ID from `flash deploy`. `curl` reads your API key from the `RUNPOD_API_KEY` environment variable, which `flash login` doesn't set, so export it first:
+
+```bash
+export RUNPOD_API_KEY="your_key"
+curl https://api.runpod.ai/v2/ENDPOINT_ID/health \
+ -H "Authorization: Bearer $RUNPOD_API_KEY"
+```
+
+A non-zero `workers.throttled` means the worker is waiting for a GPU. A non-zero `workers.idle`, `workers.ready`, or `workers.running` means a worker is up.
+
+### Job timed out after 1 retries
+
+```text
+RuntimeError: remote execution failed: job timed out after 1 retries
+```
+
+**Solution**: The worker stopped responding while it held the job. Two common causes:
+
+- **Empty input over HTTP**: a request with `{"input": {}}` never reaches your function. Send at least one argument, as shown in [Step 9](#step-9-call-the-endpoint-from-any-language).
+- **Redeploying during a request**: `flash deploy` replaces running workers, so a request that's in progress fails. Wait for running requests to finish before you redeploy, or run the request again.
+
### Authentication error
+`flash` commands print:
+
+```text
+No RunPod API key found. Set one with:
+```
+
+Scripts that call an endpoint fail with:
+
```text
-Error: API key is not set
+httpx.HTTPStatusError: Client error '401 Unauthorized' for url 'https://api.runpod.ai/v2/flash/runsync'
```
**Solution**: Run `flash login` to authenticate with your Runpod account:
@@ -391,26 +522,13 @@ Error: endpoint template names must be unique
### Job stuck in queue
-```text
-Initial job status: IN_QUEUE
-[Stays in queue for >60 seconds]
-```
+Requests wait for minutes, and the endpoint's [health check](#first-run-fails-after-90-seconds) shows `"throttled": 1` with no idle or running workers.
-**Solution**: No GPUs available. Use `GpuGroup.ANY` to accept any available GPU:
+**Solution**: No GPU of the types you selected is free. `GpuGroup.ANY` (used in this quickstart) accepts any available GPU. If you narrowed `gpu` to specific types, list several so Flash can fall back:
```python
-@Endpoint(
- name="flash-quickstart",
- gpu=GpuGroup.ANY,
- dependencies=["numpy", "torch"]
-)
-def gpu_matrix_multiply(size):
- ...
-```
+from runpod_flash import Endpoint, GpuType
-Or add multiple specific GPU types for fallback:
-
-```python
@Endpoint(
name="flash-quickstart",
gpu=[
@@ -418,7 +536,7 @@ Or add multiple specific GPU types for fallback:
GpuType.NVIDIA_RTX_A5000,
GpuType.NVIDIA_RTX_A6000
],
- dependencies=["numpy", "torch"]
+ dependencies=["torch"]
)
def gpu_matrix_multiply(size):
...
diff --git a/flash/troubleshooting.mdx b/flash/troubleshooting.mdx
index 263d8416d..2cb76422b 100644
--- a/flash/troubleshooting.mdx
+++ b/flash/troubleshooting.mdx
@@ -232,39 +232,19 @@ Duplicate route 'POST /process' in endpoint 'my-api'
**Error:**
```text
-Local Python 3.9 is not supported by Flash workers (supported: 3.10, 3.11, 3.12, 3.13).
-Pass --python-version, declare python_version on a resource config, or run flash from a supported interpreter.
+✗ Python 3.14 is not supported. Supported versions: 3.10, 3.11, 3.12, 3.13
```
-**Cause:** Flash supports Python 3.10, 3.11, 3.12, and 3.13. Your local Python version is outside this range.
+**Cause:** You passed a `--python-version` value that Flash workers don't support. Workers support Python 3.10, 3.11, 3.12, and 3.13. Your local Python version doesn't matter: you can run Flash from Python 3.10 to 3.14.
-**Solution:**
-
-You have three options:
-
-1. **Use the `--python-version` CLI flag** to override local detection:
- ```bash
- flash build --python-version 3.12
- flash deploy --python-version 3.12
- ```
-
-2. **Declare `python_version` on your resource configs:**
- ```python
- @Endpoint(name="my-endpoint", gpu=GpuGroup.ANY, python_version="3.12")
- ```
+**Solution:** Pass a supported version, or omit the flag to use the default (3.12):
-3. **Switch to a supported Python version** using a virtual environment:
- ```bash
- # Using pyenv
- pyenv install 3.12
- pyenv local 3.12
- # Or using uv
- uv venv --python 3.12
- source .venv/bin/activate
- ```
+```bash
+flash deploy --python-version 3.12
+```
-Python 3.12 is recommended for best performance with no cold-start overhead. Python 3.10, 3.11, and 3.13 incur additional cold-start overhead on GPU workers because an alternative Python interpreter must be installed.
+Python 3.12 is recommended for best performance with no cold-start overhead. Python 3.10, 3.11, and 3.13 incur additional cold-start overhead on GPU workers because PyTorch is reinstalled for the selected interpreter.
### Local module could not be resolved
@@ -391,18 +371,18 @@ Failed to provision resources: [error details]
**Error:**
```text
-Endpoint URL not available - endpoint may not be deployed
+RuntimeError: endpoint 'my-endpoint' not found in app 'my-app' environment 'production'. deploy it first with 'flash deploy'.
```
-**Cause:** The endpoint function was called before the endpoint finished provisioning.
+**Cause:** Calling an `@Endpoint` function sends the request to an endpoint that's already deployed. Flash doesn't create the endpoint when a script calls it. Flash looks up the endpoint by the app name (the current directory name), the environment, and the endpoint `name`.
**Solutions:**
-1. **For standalone scripts**: Ensure the endpoint has time to provision. Flash handles this automatically, but network issues can cause delays.
+1. Run `flash deploy` from the directory that contains your script, then run the script from that same directory.
-2. **For Flash apps**: Deploy the app first with `flash deploy`, then call the endpoint.
+2. If you deployed to an environment other than `production` (`flash deploy --env staging`), set the same environment when you run the script: `FLASH_ENV=staging python your_script.py`.
-3. **Check endpoint status**: View your endpoints in the [Serverless console](https://console.runpod.io/serverless).
+3. **Check endpoint status**: View your endpoints in the [Serverless console](https://console.runpod.io/serverless), or run `flash undeploy list`.
### Execution timeout
diff --git a/tutorials/flash/image-generation-with-sdxl.mdx b/tutorials/flash/image-generation-with-sdxl.mdx
index 4821a2443..f856cd800 100644
--- a/tutorials/flash/image-generation-with-sdxl.mdx
+++ b/tutorials/flash/image-generation-with-sdxl.mdx
@@ -20,7 +20,7 @@ This tutorial shows you how to build an image generation script using Flash and
- You've [created a Runpod account](/accounts-billing/manage-accounts).
- You've [created a Runpod API key](/get-started/credentials).
-- You've installed [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/).
+- You've installed [Python 3.10 to 3.14](https://www.python.org/downloads/).
- You've completed the [Flash quickstart](/flash/quickstart) or are familiar with Flash basics.
## What you'll build
@@ -46,371 +46,401 @@ Install Flash using [uv](https://docs.astral.sh/uv/):
```bash
uv venv
source .venv/bin/activate
-uv pip install runpod-flash python-dotenv
+uv pip install runpod-flash
```
-Create a `.env` file with your Runpod API key:
+Log in so Flash can deploy and call endpoints on your account:
```bash
-touch .env && echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
+flash login
```
-Replace `YOUR_API_KEY` with your actual API key from the [Runpod console](https://console.runpod.io/user/settings).
+Alternatively, create a `.env` file with your API key. Flash loads `.env` from the current directory automatically:
-## Step 2: Understand Stable Diffusion XL
+```bash
+echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
+```
-[Stable Diffusion XL (SDXL)](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) is a state-of-the-art text-to-image model from Stability AI. It offers:
+Replace `YOUR_API_KEY` with your API key from the [Runpod console](https://console.runpod.io/user/settings).
-- **High-quality images**: Generates photorealistic 1024x1024 images
-- **Better prompt understanding**: Improved text comprehension compared to SD 1.5
-- **Fine details**: Enhanced rendering of hands, faces, and text
-- **Open source**: Available for free on Hugging Face
+
+The directory name becomes your Flash app name. Run `flash deploy` and your script from this directory.
+
-SDXL requires significant GPU resources:
-- **Model size**: ~7GB of weights
-- **VRAM requirement**: Minimum 16GB (24GB recommended)
-- **Generation time**: 20-40 seconds per image on RTX 4090
+## Step 2: Understand Stable Diffusion XL
-We'll use the [diffusers](https://huggingface.co/docs/diffusers/index) library from Hugging Face, which provides a clean Python API for Stable Diffusion models.
+[Stable Diffusion XL (SDXL)](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) is a text-to-image model from Stability AI. It generates 1024x1024 images and follows prompts more closely than Stable Diffusion 1.5.
-## Step 3: Create your project file
+What to expect when you run it on Runpod:
-Create a new file called `image_generation.py`:
+- **Model size**: about 7 GB of fp16 weights, downloaded by the worker the first time it loads the model.
+- **VRAM**: at least 16 GB. This tutorial uses 24 GB GPUs.
+- **Speed**: once the model is loaded, 30 steps take about 4-5 seconds on an RTX 4090.
-```bash
-touch image_generation.py
-```
+You'll use the [diffusers](https://huggingface.co/docs/diffusers/index) library from Hugging Face to load and run the model.
-Open this file in your code editor. The following steps walk through building the image generation application.
+## Step 3: Create your project file
-## Step 4: Add imports and configuration
+Create a file called `image_generation.py` in your project directory. The next three steps build it up section by section. The [complete file](#complete-file) is at the end of Step 6.
-Add the necessary imports and Flash configuration:
+## Step 4: Add imports and configuration
```python
import asyncio
import base64
from pathlib import Path
-from dotenv import load_dotenv
+
from runpod_flash import Endpoint, GpuGroup
-# Load environment variables from .env file
-load_dotenv()
+MODEL_ID = "stabilityai/stable-diffusion-xl-base-1.0"
+
+# Each worker keeps the loaded pipeline here between requests.
+_pipeline = None
```
+`_pipeline` caches the model on the worker. Without it, every request would load 7 GB of weights from disk into GPU memory again, which adds about 20 seconds per image.
+
## Step 5: Define the image generation function
-Add the endpoint function that will run on the GPU worker:
+Add the endpoint function that runs on the GPU worker:
```python
@Endpoint(
name="image-generation",
- gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24], # 24GB GPUs
+ gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24], # 24 GB GPUs
workers=2,
- idle_timeout=900, # Keep workers active for 15 minutes
- dependencies=["diffusers", "torch", "transformers", "accelerate"]
+ idle_timeout=900, # Keep workers warm for 15 minutes
+ dependencies=["diffusers", "torch", "transformers"],
)
-def generate_image(prompt, negative_prompt="", num_steps=30, guidance_scale=7.5):
- """Generate an image using Stable Diffusion XL."""
- import torch
- from diffusers import StableDiffusionXLPipeline
- import base64
+def generate_image(prompt, negative_prompt="", num_steps=30, guidance_scale=7.5, seed=None):
+ """Generate an image with Stable Diffusion XL and return it as base64 PNG."""
+ import time
from io import BytesIO
- # Load the SDXL model
- model_id = "stabilityai/stable-diffusion-xl-base-1.0"
- pipe = StableDiffusionXLPipeline.from_pretrained(
- model_id,
- torch_dtype=torch.float16,
- use_safetensors=True,
- variant="fp16"
- )
-
- # Move model to GPU
- device = "cuda" if torch.cuda.is_available() else "cpu"
- pipe = pipe.to(device)
+ import torch
+ from diffusers import StableDiffusionXLPipeline
- # Generate image
- image = pipe(
+ global _pipeline
+ cached = _pipeline is not None
+ start = time.perf_counter()
+ if _pipeline is None:
+ _pipeline = StableDiffusionXLPipeline.from_pretrained(
+ MODEL_ID,
+ torch_dtype=torch.float16,
+ use_safetensors=True,
+ variant="fp16",
+ ).to("cuda")
+ load_seconds = time.perf_counter() - start
+
+ generator = None
+ if seed is not None:
+ generator = torch.Generator("cuda").manual_seed(seed)
+
+ start = time.perf_counter()
+ image = _pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
num_inference_steps=num_steps,
guidance_scale=guidance_scale,
- height=1024,
- width=1024
+ generator=generator,
).images[0]
+ generate_seconds = time.perf_counter() - start
- # Convert image to base64 for transmission
- buffered = BytesIO()
- image.save(buffered, format="PNG")
- img_str = base64.b64encode(buffered.getvalue()).decode()
+ buffer = BytesIO()
+ image.save(buffer, format="PNG")
return {
- "image_base64": img_str,
- "prompt": prompt,
- "negative_prompt": negative_prompt,
- "num_steps": num_steps,
- "guidance_scale": guidance_scale,
- "device": device,
- "resolution": "1024x1024"
+ "image_base64": base64.b64encode(buffer.getvalue()).decode(),
+ "gpu": torch.cuda.get_device_name(0),
+ "model_cached": cached,
+ "load_seconds": round(load_seconds, 1),
+ "generate_seconds": round(generate_seconds, 1),
}
```
**Configuration breakdown**:
- **`name="image-generation"`**: Identifies your endpoint in the Runpod console.
-- **`gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24]`**: Uses RTX 4090 or L4/A5000 GPUs (both have 24GB VRAM, sufficient for SDXL).
+- **`gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24]`**: Uses 24 GB GPUs such as the RTX 4090, L4, or A5000. SDXL needs at least 16 GB.
- **`workers=2`**: Allows up to 2 parallel workers.
-- **`idle_timeout=900`**: Keeps workers active for 15 minutes (SDXL models are large, so we want longer caching).
-
-
-SDXL requires at least 16GB VRAM. Using 24GB GPUs provides comfortable headroom and faster generation.
-
+- **`idle_timeout=900`**: Keeps a worker running for 15 minutes after its last request, so the loaded model stays in memory.
+- **`dependencies`**: Installed into your build. `torch` is already in the worker image, so Flash leaves it out of the upload.
-This function:
-- Loads the SDXL model from Hugging Face.
-- Moves the model to the GPU.
-- Generates an image from the prompt.
-- Encodes the image as base64.
-- Returns the image as a base64 string (and other metadata).
-
-Expand this section for a full breakdown:
+
+Don't add `accelerate` to `dependencies`. It requires `torch`, so on Linux `flash deploy` downloads `torch` and several GB of CUDA libraries into the local build, which can fill your disk. Without it, the build is about 50 MB.
+
-**Dependencies**: The function requires four packages:
- - `diffusers`: Hugging Face library for diffusion models
- - `torch`: PyTorch for GPU computation
- - `transformers`: Text encoder dependencies
- - `accelerate`: Efficient model loading
+**Imports inside the function**: `torch` and `diffusers` are imported inside `generate_image` because they're installed on the worker, not on your machine.
-**Model loading**:
+**Model caching**:
```python
- pipe = StableDiffusionXLPipeline.from_pretrained(
- model_id,
- torch_dtype=torch.float16,
- use_safetensors=True,
- variant="fp16"
- )
+ global _pipeline
+ if _pipeline is None:
+ _pipeline = StableDiffusionXLPipeline.from_pretrained(...).to("cuda")
```
- This downloads SDXL from Hugging Face. Key parameters:
- - `torch_dtype=torch.float16`: Use half-precision (saves VRAM, faster)
- - `use_safetensors=True`: Use safe tensor format
- - `variant="fp16"`: Download the fp16 version (~7GB instead of ~14GB)
+ The worker imports your file once and keeps it in memory between requests, so module-level variables like `_pipeline` persist. The first request on a worker loads the model. Later requests on the same worker reuse it. `model_cached` in the result tells you which happened.
-**GPU acceleration**:
- ```python
- pipe = pipe.to(device)
- ```
- Moves the entire pipeline (text encoder, UNet, VAE) to GPU.
+**Model loading parameters**:
+ - `torch_dtype=torch.float16`: Half precision. Uses less VRAM and runs faster.
+ - `use_safetensors=True`: Loads the safetensors format.
+ - `variant="fp16"`: Downloads the fp16 weights (about 7 GB instead of 14 GB).
-**Image generation**:
- ```python
- image = pipe(
- prompt=prompt,
- negative_prompt=negative_prompt,
- num_inference_steps=num_steps,
- guidance_scale=guidance_scale,
- height=1024,
- width=1024
- ).images[0]
- ```
-
- Parameters:
- - **`prompt`**: What you want to see in the image
- - **`negative_prompt`**: What you don't want (e.g., "blurry, low quality")
- - **`num_inference_steps`**: More steps = better quality but slower (20-50 typical)
- - **`guidance_scale`**: How closely to follow the prompt (7-10 recommended)
- - **`height/width`**: SDXL is trained for 1024x1024
+**Generation parameters**:
+ - **`prompt`**: What you want to see in the image.
+ - **`negative_prompt`**: What you don't want, such as "blurry, low quality".
+ - **`num_inference_steps`**: More steps give more refined images but take longer. 20-50 is typical.
+ - **`guidance_scale`**: How closely to follow the prompt. 7-10 is typical.
+ - **`generator`**: A seeded generator makes results reproducible. The same prompt and seed give the same image.
**Image encoding**:
```python
- buffered = BytesIO()
- image.save(buffered, format="PNG")
- img_str = base64.b64encode(buffered.getvalue()).decode()
+ buffer = BytesIO()
+ image.save(buffer, format="PNG")
+ base64.b64encode(buffer.getvalue()).decode()
```
- We encode the image as base64 to return it through Flash. This allows us to transmit the image data as a string.
+ Endpoint results are sent back as JSON, so the PNG is encoded as a base64 string.
## Step 6: Add the main function and image saving
-Create functions to call the generator and save images:
+Add a helper that saves images and a `main()` function that calls the endpoint. Everything in this step runs on your machine:
```python
-def save_image(base64_string, filename):
- """Save a base64-encoded image to disk."""
- import base64
- from PIL import Image
- from io import BytesIO
+def save_image(image_base64, path):
+ """Decode a base64 PNG and write it to disk."""
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_bytes(base64.b64decode(image_base64))
+ print(f"Saved {path}")
- # Decode base64 string
- img_data = base64.b64decode(base64_string)
-
- # Open and save image
- image = Image.open(BytesIO(img_data))
- image.save(filename)
- print(f"✓ Image saved to {filename}")
async def main():
- print("Generating image with Stable Diffusion XL on Runpod GPU...")
- print("This may take 1-2 minutes on first run (downloading model)...\n")
+ print("Generating an image with SDXL on a Runpod GPU...")
+ result = await generate_image(
+ prompt="A serene mountain lake at sunset, highly detailed, photorealistic",
+ negative_prompt="blurry, low quality, distorted",
+ num_steps=30,
+ seed=42,
+ )
+ save_image(result["image_base64"], Path("generated_images") / "sdxl_output.png")
+ print(f"GPU: {result['gpu']}")
+ print(f"Model already loaded: {result['model_cached']}")
+ print(f"Model load: {result['load_seconds']}s, generation: {result['generate_seconds']}s")
- # Define your prompt
- prompt = "A serene landscape with mountains, a lake, and sunset, highly detailed, photorealistic"
- negative_prompt = "blurry, low quality, distorted, ugly"
- # Generate image
- result = await generate_image(
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+
+Keep the `if __name__ == "__main__":` block at the end of the file. Without it, the script exits without doing anything.
+
+### Complete file
+
+
+
+```python
+import asyncio
+import base64
+from pathlib import Path
+
+from runpod_flash import Endpoint, GpuGroup
+
+MODEL_ID = "stabilityai/stable-diffusion-xl-base-1.0"
+
+# Each worker keeps the loaded pipeline here between requests.
+_pipeline = None
+
+
+@Endpoint(
+ name="image-generation",
+ gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24], # 24 GB GPUs
+ workers=2,
+ idle_timeout=900, # Keep workers warm for 15 minutes
+ dependencies=["diffusers", "torch", "transformers"],
+)
+def generate_image(prompt, negative_prompt="", num_steps=30, guidance_scale=7.5, seed=None):
+ """Generate an image with Stable Diffusion XL and return it as base64 PNG."""
+ import time
+ from io import BytesIO
+
+ import torch
+ from diffusers import StableDiffusionXLPipeline
+
+ global _pipeline
+ cached = _pipeline is not None
+ start = time.perf_counter()
+ if _pipeline is None:
+ _pipeline = StableDiffusionXLPipeline.from_pretrained(
+ MODEL_ID,
+ torch_dtype=torch.float16,
+ use_safetensors=True,
+ variant="fp16",
+ ).to("cuda")
+ load_seconds = time.perf_counter() - start
+
+ generator = None
+ if seed is not None:
+ generator = torch.Generator("cuda").manual_seed(seed)
+
+ start = time.perf_counter()
+ image = _pipeline(
prompt=prompt,
negative_prompt=negative_prompt,
+ num_inference_steps=num_steps,
+ guidance_scale=guidance_scale,
+ generator=generator,
+ ).images[0]
+ generate_seconds = time.perf_counter() - start
+
+ buffer = BytesIO()
+ image.save(buffer, format="PNG")
+
+ return {
+ "image_base64": base64.b64encode(buffer.getvalue()).decode(),
+ "gpu": torch.cuda.get_device_name(0),
+ "model_cached": cached,
+ "load_seconds": round(load_seconds, 1),
+ "generate_seconds": round(generate_seconds, 1),
+ }
+
+
+def save_image(image_base64, path):
+ """Decode a base64 PNG and write it to disk."""
+ path.parent.mkdir(parents=True, exist_ok=True)
+ path.write_bytes(base64.b64decode(image_base64))
+ print(f"Saved {path}")
+
+
+async def main():
+ print("Generating an image with SDXL on a Runpod GPU...")
+ result = await generate_image(
+ prompt="A serene mountain lake at sunset, highly detailed, photorealistic",
+ negative_prompt="blurry, low quality, distorted",
num_steps=30,
- guidance_scale=7.5
+ seed=42,
)
+ save_image(result["image_base64"], Path("generated_images") / "sdxl_output.png")
+ print(f"GPU: {result['gpu']}")
+ print(f"Model already loaded: {result['model_cached']}")
+ print(f"Model load: {result['load_seconds']}s, generation: {result['generate_seconds']}s")
- # Save the generated image
- output_dir = Path("generated_images")
- output_dir.mkdir(exist_ok=True)
-
- filename = output_dir / "sdxl_output.png"
- save_image(result["image_base64"], filename)
-
- # Display metadata
- print(f"\n{'='*60}")
- print("GENERATION DETAILS")
- print('='*60)
- print(f"Prompt: {result['prompt']}")
- print(f"Negative prompt: {result['negative_prompt']}")
- print(f"Steps: {result['num_steps']}")
- print(f"Guidance scale: {result['guidance_scale']}")
- print(f"Resolution: {result['resolution']}")
- print(f"Device: {result['device']}")
- print('='*60)
if __name__ == "__main__":
asyncio.run(main())
```
-This main function:
-
-- Calls the remote function with `await`.
-- Creates a `generated_images` directory if it doesn't exist.
-- Decodes and saves the base64 image to disk.
-- Displays generation metadata.
+
-## Step 7: Run your first generation
+## Step 7: Deploy and run your first generation
-Run the application:
+Deploy the endpoint from your project directory:
```bash
-python image_generation.py
+flash deploy
+
+# If using uv:
+uv run flash deploy
```
-**First run output** (takes 2-3 minutes):
+Flash installs your dependencies into a local build directory, uploads the build, and creates the `image-generation` endpoint:
```text
-Generating image with Stable Diffusion XL on Runpod GPU...
-This may take 1-2 minutes on first run (downloading model)...
-
-Creating endpoint: server_Endpoint_a1b2c3d4
-Provisioning Serverless endpoint...
-Endpoint ready
-Executing function on Runpod endpoint ID: xvf32dan8rcilp
-Initial job status: IN_QUEUE
-Downloading model weights from Hugging Face...
-Model loaded, generating image...
-Job completed, output received
-✓ Image saved to generated_images/sdxl_output.png
-
-============================================================
-GENERATION DETAILS
-============================================================
-Prompt: A serene landscape with mountains, a lake, and sunset, highly detailed, photorealistic
-Negative prompt: blurry, low quality, distorted, ugly
-Steps: 30
-Guidance scale: 7.5
-Resolution: 1024x1024
-Device: cuda
-============================================================
+✓ built flash-image-generation 1 files, 2 deps, 50.2 MB
+No app 'flash-image-generation' found. Creating app and 'production' environment...
+
+✓ uploaded 50.2 MB 3.0s
+✓ deployed to production 1.9s
+
+ image-generation https://api.runpod.ai/v2/abc123xyz/runsync
+
+curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer $RUNPOD_API_KEY" \
+ -d '{"input": {}}'
```
-**Subsequent runs** (takes 30-40 seconds):
+Then run the script from the same directory:
+
+```bash
+python image_generation.py
+```
+
+The first run takes about 80 seconds. The worker starts, downloads the SDXL weights, and loads them onto the GPU:
```text
-Generating image with Stable Diffusion XL on Runpod GPU...
+Generating an image with SDXL on a Runpod GPU...
+Saved generated_images/sdxl_output.png
+GPU: NVIDIA GeForce RTX 4090
+Model already loaded: False
+Model load: 18.0s, generation: 4.9s
+```
-Resource Endpoint_a1b2c3d4 already exists, reusing.
-Executing function on Runpod endpoint ID: xvf32dan8rcilp
-Initial job status: IN_QUEUE
-Job completed, output received
-✓ Image saved to generated_images/sdxl_output.png
+Run it again while the worker is warm. The model is already loaded, so the whole run takes about 7 seconds:
-[Results appear]
+```text
+Generating an image with SDXL on a Runpod GPU...
+Saved generated_images/sdxl_output.png
+GPU: NVIDIA GeForce RTX 4090
+Model already loaded: True
+Model load: 0.0s, generation: 3.9s
```
-Open `generated_images/sdxl_output.png` to see your generated image!
+Open `generated_images/sdxl_output.png` to see your image.
-
-The first run downloads ~7GB of model weights, which takes 1-2 minutes. Subsequent runs reuse the cached model and complete in 30-40 seconds.
-
+
+The first request on a new worker can take longer than the 90 seconds `runpod-flash` 1.21.0 waits for a result, especially if the worker has to wait for a free GPU. If the script fails with `timed out after 90s. the endpoint may not be deployed or the worker is still starting` or `KeyError: 'image_base64'`, the endpoint is deployed and the worker is still loading the model. Run the script again. See [First run fails after 90 seconds](#first-run-fails-after-90-seconds).
+
-## Step 8: Experiment with different prompts
+## Step 8: Generate several images in parallel
-Try various prompts to see SDXL's capabilities:
+Replace `main()` with a version that sends three prompts at once with `asyncio.gather`:
```python
+PROMPTS = {
+ "cyberpunk_city": "A cyberpunk city at night with neon lights and rain, cinematic",
+ "space_corgi": "A corgi puppy in a space suit floating in space, highly detailed",
+ "wizard_study": "An ancient wizard's study full of books and potions, candlelight",
+}
+
+
async def main():
- # Create output directory
- output_dir = Path("generated_images")
- output_dir.mkdir(exist_ok=True)
-
- # Try different prompts
- prompts = [
- {
- "prompt": "A cyberpunk city at night with neon lights, flying cars, rain, cinematic",
- "negative": "blurry, low quality",
- "filename": "cyberpunk_city.png"
- },
- {
- "prompt": "A cute corgi puppy wearing a space suit, floating in space, highly detailed",
- "negative": "distorted, ugly, bad anatomy",
- "filename": "space_corgi.png"
- },
- {
- "prompt": "An ancient wizard's study filled with books, potions, magical artifacts, candlelight",
- "negative": "blurry, modern, plastic",
- "filename": "wizard_study.png"
- }
- ]
-
- for i, p in enumerate(prompts, 1):
- print(f"\n{'='*60}")
- print(f"Generating image {i}/{len(prompts)}")
- print(f"Prompt: {p['prompt'][:50]}...")
- print('='*60)
-
- result = await generate_image(
- prompt=p['prompt'],
- negative_prompt=p['negative'],
- num_steps=30,
- guidance_scale=7.5
- )
-
- filename = output_dir / p['filename']
- save_image(result["image_base64"], filename)
- print(f"✓ Saved to {filename}\n")
+ print(f"Generating {len(PROMPTS)} images in parallel...")
+ results = await asyncio.gather(
+ *(generate_image(prompt=prompt, seed=42) for prompt in PROMPTS.values())
+ )
+ for name, result in zip(PROMPTS, results):
+ save_image(result["image_base64"], Path("generated_images") / f"{name}.png")
+ print(f" model already loaded: {result['model_cached']}, "
+ f"generation: {result['generate_seconds']}s")
+
if __name__ == "__main__":
asyncio.run(main())
```
-Run it:
+You only changed code outside the `@Endpoint` function, so you don't need to redeploy. Run the script:
```bash
python image_generation.py
```
-You'll see three different images generated sequentially on the same GPU worker. Each generation takes about 30-40 seconds after the first one.
+```text
+Generating 3 images in parallel...
+Saved generated_images/cyberpunk_city.png
+ model already loaded: True, generation: 3.9s
+Saved generated_images/space_corgi.png
+ model already loaded: True, generation: 3.9s
+Saved generated_images/wizard_study.png
+ model already loaded: True, generation: 3.9s
+```
+
+All three requests are queued at once. A warm worker generates them one after another, about 18 seconds in total. If the queue builds up, Flash starts the second worker (`workers=2`). A new worker loads the model on its first request, so that result shows `model already loaded: False`.
+
+
+Keep the endpoint function and the code that calls it in the same file. If a second file imports `generate_image`, `flash deploy` treats it as a second definition and fails with `endpoint 'image-generation' is defined in multiple files`.
+
## Understanding generation parameters
@@ -430,9 +460,11 @@ result = await generate_image(prompt, num_steps=50)
```
**Effects**:
-- **15-20 steps**: Faster (15-20 seconds) but less refined details.
-- **30 steps**: Good balance of quality and speed (30-40 seconds) - **recommended**.
-- **50+ steps**: Diminishing returns, minimal quality improvement.
+- **15-20 steps**: Faster, but fewer refined details.
+- **30 steps**: A good balance of quality and speed (about 4-5 seconds on an RTX 4090). **Recommended**.
+- **50+ steps**: Diminishing returns, with little visible improvement.
+
+Generation time grows roughly in proportion to the number of steps.
### Guidance scale
@@ -513,10 +545,7 @@ Use negative prompts to:
2. Check Hugging Face Hub status at [status.huggingface.co](https://status.huggingface.co).
-3. Try a smaller model first to test connectivity:
- ```python
- model_id = "runwayml/stable-diffusion-v1-5" # Smaller SD 1.5
- ```
+3. Check the worker logs for the endpoint in the [Runpod console](https://console.runpod.io/serverless). Download errors from Hugging Face appear there.
### Image quality is poor
@@ -547,25 +576,66 @@ Use negative prompts to:
### Slow generation
-**Issue**: Image generation takes >60 seconds per image.
+**Issue**: Every request takes much longer than 10 seconds.
-**Possible causes**:
-1. Worker scaled down (cold start).
-2. Model not cached.
-3. Too many inference steps.
+**Check `model_cached` in the result.** If it's `False` on most requests, workers are loading the model again:
-**Solutions**:
-1. Increase `idle_timeout` to keep workers active:
+1. Make sure `_pipeline` is a module-level variable and `generate_image` declares `global _pipeline`. A pipeline created as a local variable is loaded again on every request.
+2. Increase `idle_timeout` so workers stay up between requests:
```python
- idle_timeout=1800 # Keep active for 30 minutes
+ idle_timeout=1800 # Keep workers warm for 30 minutes
```
+3. Set `workers=(1, 2)` to keep one worker running at all times. You pay for that worker while it's idle.
-2. Reduce inference steps:
- ```python
- num_steps=20 # Faster but slightly lower quality
- ```
+If `model_cached` is `True` and generation is still slow, reduce `num_steps`.
+
+### First run fails after 90 seconds
+
+**Issue**: The first run fails with one of these errors:
+
+```text
+RuntimeError: request to endpoint 'image-generation' timed out after 90s. the endpoint may not be deployed or the worker is still starting. deploy with 'flash deploy' or increase timeout with FLASH_SENTINEL_TIMEOUT env var.
+```
+
+```text
+KeyError: 'image_base64'
+```
+
+**Cause**: `runpod-flash` 1.21.0 waits up to 90 seconds for a result. The first request on a new worker can take longer: the worker may wait for a free GPU, then it starts and loads the model. The endpoint is deployed, and the request keeps running after the error.
+
+**Solution**: Run the script again. The worker keeps starting after the error, so the next run is faster. Raising `FLASH_SENTINEL_TIMEOUT` above 90 doesn't help in 1.21.0: the request returns at 90 seconds with the job still running, and the script fails with the `KeyError` instead.
+
+To see whether a worker is up, check the endpoint's health with its ID from `flash deploy`. `curl` reads your API key from the `RUNPOD_API_KEY` environment variable, which `flash login` doesn't set, so export it first:
+
+```bash
+export RUNPOD_API_KEY="your_key"
+curl https://api.runpod.ai/v2/ENDPOINT_ID/health \
+ -H "Authorization: Bearer $RUNPOD_API_KEY"
+```
+
+A non-zero `workers.throttled` means the worker is waiting for a GPU. A non-zero `workers.idle`, `workers.ready`, or `workers.running` means a worker is up.
+
+### Build runs out of disk space
+
+**Issue**: `flash deploy` fails with `No space left on device` or `Disk quota exceeded` while installing dependencies.
+
+**Cause**: A dependency that requires `torch`, such as `accelerate`, makes the local build download `torch` and several GB of CUDA libraries on Linux.
-3. Set `workers=(1, 2)` to always have a warm worker ready.
+**Solution**: Remove that dependency, or exclude it from the build:
+
+```bash
+flash deploy --exclude accelerate
+```
+
+See [`flash deploy`](/flash/cli/deploy) for `--exclude` and `--no-deps`.
+
+### Endpoint defined in multiple files
+
+**Issue**: `flash deploy` fails with `endpoint 'image-generation' is defined in multiple files`.
+
+**Cause**: Another `.py` file in the project directory imports `generate_image`. Flash scans every file and counts the import as a second definition.
+
+**Solution**: Keep the endpoint function and the code that calls it in one file, or move the extra script out of the project directory.
### Images look distorted or have artifacts
@@ -595,17 +665,24 @@ Now that you've built an image generation script with Flash, you can:
Explore different models from Hugging Face:
+[SDXL Turbo](https://huggingface.co/stabilityai/sdxl-turbo) uses the same pipeline and generates an image in 1-4 steps at 512x512. Change `MODEL_ID` at the top of the file:
+
```python
-# SDXL Turbo - 4x faster, 1 step generation
-model_id = "stabilityai/sdxl-turbo"
+MODEL_ID = "stabilityai/sdxl-turbo"
+```
-# Stable Diffusion 1.5 - smaller, faster
-model_id = "runwayml/stable-diffusion-v1-5"
+Run `flash deploy` again so the workers pick up the new model. Then call it with few steps and no guidance in `main()`:
-# Stable Diffusion 2.1 - better at artistic styles
-model_id = "stabilityai/stable-diffusion-2-1"
+```python
+result = await generate_image(
+ prompt="A cat in a spacesuit, digital art",
+ num_steps=1,
+ guidance_scale=0.0,
+)
```
+Models outside the SDXL family, such as [Stable Diffusion 1.5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5), need a different pipeline class. Replace `StableDiffusionXLPipeline` with `AutoPipelineForText2Image`, which picks the right class for the model, and redeploy.
+
### Add image-to-image generation
Use an existing image as a starting point:
@@ -626,35 +703,36 @@ Convert your script to a production [Flash app](/flash/apps/overview):
```bash
flash init image-generation-app
-# Move your function to workers/gpu/endpoint.py
-# Add FastAPI routes for HTTP API
+cd image-generation-app
+# Move generate_image into gpu_worker.py
+# Add HTTP routes in lb_worker.py
flash deploy
-
-# If using uv:
-uv run flash init image-generation-app
-uv run flash deploy
```
+`flash init` creates `gpu_worker.py`, `cpu_worker.py`, and `lb_worker.py` at the project root. See [Build Flash apps](/flash/apps/overview) for how they fit together.
+
### Optimize with network volumes
-Use [network volumes](/flash/configuration/storage) to cache models across workers:
+Each new worker downloads the 7 GB of SDXL weights again. A [network volume](/flash/configuration/storage) keeps one copy that every worker reads, so new workers skip the download. Point the Hugging Face cache at the volume with `HF_HOME`:
```python
from runpod_flash import Endpoint, GpuGroup, NetworkVolume
-vol = NetworkVolume(name="model-cache") # Finds existing or creates new
+vol = NetworkVolume(name="model-cache") # Finds an existing volume or creates one
@Endpoint(
name="image-generation",
- gpu=GpuGroup.ADA_24,
+ gpu=[GpuGroup.ADA_24, GpuGroup.AMPERE_24],
volume=vol,
- dependencies=["diffusers", "torch", "transformers", "accelerate"]
+ env={"HF_HOME": "/runpod-volume/huggingface"},
+ dependencies=["diffusers", "torch", "transformers"],
)
-def generate_image(prompt, ...):
- # Models at /runpod-volume/ persist across workers
+def generate_image(prompt, negative_prompt="", num_steps=30, guidance_scale=7.5, seed=None):
...
```
+The volume is mounted at `/runpod-volume` on every worker. A network volume lives in one data center, so workers only run there.
+
### Explore advanced features
- **LoRA fine-tuning**: Customize SDXL for specific styles.
diff --git a/tutorials/flash/text-generation-with-transformers.mdx b/tutorials/flash/text-generation-with-transformers.mdx
index 59d5ee87f..a4fee545a 100644
--- a/tutorials/flash/text-generation-with-transformers.mdx
+++ b/tutorials/flash/text-generation-with-transformers.mdx
@@ -16,7 +16,7 @@ This tutorial shows you how to build a text generation script using Flash and Hu
- You've [created a Runpod account](/accounts-billing/manage-accounts).
- You've [created a Runpod API key](/get-started/credentials).
-- You've installed [Python 3.10, 3.11, 3.12, or 3.13](https://www.python.org/downloads/).
+- You've installed [Python 3.10 to 3.14](https://www.python.org/downloads/).
- You've completed the [Flash quickstart](/flash/quickstart) or are familiar with Flash basics.
## What you'll build
@@ -42,292 +42,322 @@ Install Flash using [uv](https://docs.astral.sh/uv/):
```bash
uv venv
source .venv/bin/activate
-uv pip install runpod-flash python-dotenv
+uv pip install runpod-flash
```
-Create a `.env` file with your Runpod API key:
+Log in so Flash can deploy and call endpoints on your account:
```bash
-touch .env && echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
+flash login
```
-Replace `YOUR_API_KEY` with your actual API key from the [Runpod console](https://console.runpod.io/user/settings).
+Alternatively, create a `.env` file with your API key. Flash loads `.env` from the current directory automatically:
-## Step 2: Understand the Hugging Face transformers library
+```bash
+echo "RUNPOD_API_KEY=YOUR_API_KEY" > .env
+```
-[Hugging Face transformers](https://huggingface.co/docs/transformers/index) is a popular Python library for working with pretrained language models. It provides:
+Replace `YOUR_API_KEY` with your API key from the [Runpod console](https://console.runpod.io/user/settings).
-- **Thousands of pretrained models**: GPT-2, GPT-3, BERT, T5, LLaMA, and more
-- **Unified API**: Same code works across different model architectures
-- **Model hub integration**: Download models directly from [Hugging Face Hub](https://huggingface.co/models)
-- **Production-ready**: Used by companies and researchers worldwide
+
+The directory name becomes your Flash app name. Run `flash deploy` and your script from this directory.
+
-For this tutorial, we'll use **GPT-2**, a 124M parameter language model from OpenAI. It's small enough to load quickly but powerful enough to generate coherent text.
+## Step 2: Understand the Hugging Face transformers library
-## Step 3: Create your project file
+[Hugging Face transformers](https://huggingface.co/docs/transformers/index) is a Python library for running pretrained language models. The same API loads thousands of models from the [Hugging Face Hub](https://huggingface.co/models).
-Create a new file called `text_generation.py`:
+This tutorial uses **GPT-2**, a 124M-parameter model from OpenAI. It's small (about 500 MB), loads in seconds, and needs less than 2 GB of VRAM. It continues text rather than following instructions, so expect creative completions, not answers.
-```bash
-touch text_generation.py
-```
+## Step 3: Create your project file
-Open this file in your code editor. The following steps walk through building the text generation application.
+Create a file called `text_generation.py` in your project directory. The next three steps build it up section by section. The [complete file](#complete-file) is at the end of Step 6.
## Step 4: Add imports and configuration
-Add the necessary imports and Flash configuration:
-
```python
import asyncio
-from dotenv import load_dotenv
+
from runpod_flash import Endpoint, GpuGroup
-# Load environment variables from .env file
-load_dotenv()
+MODEL_NAME = "gpt2"
+
+# Each worker keeps the loaded model here between requests.
+_model = None
+_tokenizer = None
```
+`_model` and `_tokenizer` cache the model on the worker, so only the first request on each worker loads it.
+
## Step 5: Define the text generation function
-Add the endpoint function that will run on the GPU worker:
+Add the endpoint function that runs on the GPU worker:
```python
@Endpoint(
name="text-generation",
- gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24], # 24GB GPUs
+ gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24], # 24 GB GPUs
workers=3,
- idle_timeout=600, # 10 minutes
- dependencies=["transformers", "torch", "accelerate"]
+ idle_timeout=600, # Keep workers warm for 10 minutes
+ dependencies=["transformers", "torch"],
)
-def generate_text(prompt, max_length=50):
- """Generate text using a pretrained language model."""
- import torch
- from transformers import AutoTokenizer, AutoModelForCausalLM
-
- # Load the GPT-2 model and tokenizer
- model_name = "gpt2"
- tokenizer = AutoTokenizer.from_pretrained(model_name)
- model = AutoModelForCausalLM.from_pretrained(model_name)
-
- # Move model to GPU if available
- device = "cuda" if torch.cuda.is_available() else "cpu"
- device_name = torch.cuda.get_device_name(0) if torch.cuda.is_available() else "CPU"
- model = model.to(device)
-
- # Tokenize the input prompt
- inputs = tokenizer(prompt, return_tensors="pt").to(device)
+def generate_text(prompt, max_new_tokens=50, temperature=0.7):
+ """Continue a prompt with GPT-2."""
+ import time
- # Generate text
+ import torch
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ global _model, _tokenizer
+ cached = _model is not None
+ start = time.perf_counter()
+ if _model is None:
+ _tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
+ _model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to("cuda")
+ load_seconds = time.perf_counter() - start
+
+ inputs = _tokenizer(prompt, return_tensors="pt").to("cuda")
+ start = time.perf_counter()
with torch.no_grad():
- outputs = model.generate(
+ outputs = _model.generate(
**inputs,
- max_length=max_length,
- num_return_sequences=1,
- temperature=0.7,
+ max_new_tokens=max_new_tokens,
do_sample=True,
- pad_token_id=tokenizer.eos_token_id
+ temperature=temperature,
+ pad_token_id=_tokenizer.eos_token_id,
)
-
- # Decode the generated tokens back to text
- generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+ generate_seconds = time.perf_counter() - start
return {
- "prompt": prompt,
- "generated_text": generated_text,
- "model_name": model_name,
- "device": device,
- "device_name": device_name,
- "max_length": max_length
+ "generated_text": _tokenizer.decode(outputs[0], skip_special_tokens=True),
+ "gpu": torch.cuda.get_device_name(0),
+ "model_cached": cached,
+ "load_seconds": round(load_seconds, 1),
+ "generate_seconds": round(generate_seconds, 2),
}
```
**Configuration breakdown**:
-- **`name="text-generation"`**: Identifies your endpoint in the Runpod console
-- **`gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24]`**: Allows workers to use L4, A5000, RTX 3090, or RTX 4090 GPUs (all have 24GB VRAM)
-- **`workers=3`**: Allows up to 3 parallel workers for concurrent requests
-- **`idle_timeout=600`**: Keeps workers active for 10 minutes after last use (reduces cold starts)
-
-
-GPT-2 only requires about 2GB of VRAM, so 24GB GPUs are more than sufficient. For larger models like LLaMA or GPT-J, you might need 48GB or 80GB GPUs.
-
+- **`name="text-generation"`**: Identifies your endpoint in the Runpod console.
+- **`gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24]`**: Uses 24 GB GPUs such as the RTX 3090, RTX 4090, L4, or A5000. GPT-2 needs far less, so any of them work.
+- **`workers=3`**: Allows up to 3 parallel workers for concurrent requests.
+- **`idle_timeout=600`**: Keeps a worker running for 10 minutes after its last request, so the loaded model stays in memory.
+- **`dependencies`**: Installed into your build. `torch` is already in the worker image, so Flash leaves it out of the upload.
-This function:
-- Loads the GPT-2 model from Hugging Face.
-- Moves the model to the GPU.
-- Tokenizes the input prompt.
-- Generates text from the prompt.
-- Decodes the generated tokens back to text.
-- Returns the generated text and other metadata.
-
-Expand this section for a full breakdown:
+
+Don't add `accelerate` to `dependencies`. GPT-2 doesn't need it, and because it requires `torch`, on Linux `flash deploy` downloads `torch` and several GB of CUDA libraries into the local build. Without it, the build is about 40 MB.
+
-**Dependencies**: The function requires three packages:
- - `transformers`: Hugging Face library for language models
- - `torch`: PyTorch for GPU computation
- - `accelerate`: Helper library for loading large models efficiently
-
-**Model loading**:
- ```python
- tokenizer = AutoTokenizer.from_pretrained(model_name)
- model = AutoModelForCausalLM.from_pretrained(model_name)
- ```
- These lines download and load the GPT-2 model from Hugging Face Hub. The first time this runs, it downloads ~500MB of model weights. Subsequent runs use the cached version.
+**Imports inside the function**: `torch` and `transformers` are imported inside `generate_text` because they're installed on the worker, not on your machine.
-**GPU acceleration**:
+**Model caching**:
```python
- device = "cuda" if torch.cuda.is_available() else "cpu"
- model = model.to(device)
+ global _model, _tokenizer
+ if _model is None:
+ _tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
+ _model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to("cuda")
```
- This moves the model to GPU for faster inference. On Runpod workers, `torch.cuda.is_available()` returns `True`.
+ The worker imports your file once and keeps it in memory between requests, so module-level variables persist. The first request on a worker downloads and loads GPT-2. Later requests on the same worker reuse it. `model_cached` in the result tells you which happened.
**Tokenization**:
```python
- inputs = tokenizer(prompt, return_tensors="pt").to(device)
+ inputs = _tokenizer(prompt, return_tensors="pt").to("cuda")
```
- Converts your text prompt into token IDs that the model understands. The `.to(device)` moves these tokens to GPU memory.
+ Converts the prompt into token IDs on the GPU.
**Generation parameters**:
- - `max_length=50`: Maximum number of tokens to generate
- - `temperature=0.7`: Controls randomness (0.0 = deterministic, 1.0+ = very random)
- - `do_sample=True`: Use sampling instead of greedy decoding for more diverse outputs
- - `num_return_sequences=1`: Generate one completion per prompt
+ - `max_new_tokens`: How many tokens to generate after the prompt.
+ - `do_sample=True`: Samples from the model's predictions instead of always picking the most likely token, so each run gives a different completion.
+ - `temperature`: Controls randomness when sampling. Lower values are more predictable, higher values more varied.
+ - `pad_token_id`: GPT-2 has no padding token, so this reuses the end-of-text token and avoids a warning.
-**No gradient tracking**:
- ```python
- with torch.no_grad():
- ```
- Disables gradient computation, reducing memory usage and speeding up inference.
+**No gradient tracking**: `torch.no_grad()` skips the bookkeeping needed for training, which saves memory during inference.
## Step 6: Add the main function
-Create the main function to test your text generator:
+Add a `main()` function that calls the endpoint. It runs on your machine:
+
+```python
+async def main():
+ print("Generating text with GPT-2 on a Runpod GPU...")
+ result = await generate_text("The future of artificial intelligence is", max_new_tokens=60)
+ print(f"\n{result['generated_text']}\n")
+ print(f"GPU: {result['gpu']}")
+ print(f"Model already loaded: {result['model_cached']}")
+ print(f"Model load: {result['load_seconds']}s, generation: {result['generate_seconds']}s")
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
+```
+
+Keep the `if __name__ == "__main__":` block at the end of the file. Without it, the script exits without doing anything.
+
+### Complete file
+
+
```python
+import asyncio
+
+from runpod_flash import Endpoint, GpuGroup
+
+MODEL_NAME = "gpt2"
+
+# Each worker keeps the loaded model here between requests.
+_model = None
+_tokenizer = None
+
+
+@Endpoint(
+ name="text-generation",
+ gpu=[GpuGroup.AMPERE_24, GpuGroup.ADA_24], # 24 GB GPUs
+ workers=3,
+ idle_timeout=600, # Keep workers warm for 10 minutes
+ dependencies=["transformers", "torch"],
+)
+def generate_text(prompt, max_new_tokens=50, temperature=0.7):
+ """Continue a prompt with GPT-2."""
+ import time
+
+ import torch
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ global _model, _tokenizer
+ cached = _model is not None
+ start = time.perf_counter()
+ if _model is None:
+ _tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
+ _model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).to("cuda")
+ load_seconds = time.perf_counter() - start
+
+ inputs = _tokenizer(prompt, return_tensors="pt").to("cuda")
+ start = time.perf_counter()
+ with torch.no_grad():
+ outputs = _model.generate(
+ **inputs,
+ max_new_tokens=max_new_tokens,
+ do_sample=True,
+ temperature=temperature,
+ pad_token_id=_tokenizer.eos_token_id,
+ )
+ generate_seconds = time.perf_counter() - start
+
+ return {
+ "generated_text": _tokenizer.decode(outputs[0], skip_special_tokens=True),
+ "gpu": torch.cuda.get_device_name(0),
+ "model_cached": cached,
+ "load_seconds": round(load_seconds, 1),
+ "generate_seconds": round(generate_seconds, 2),
+ }
+
+
async def main():
- print("Starting text generation on Runpod GPU...")
-
- # Define a prompt
- prompt = "The future of artificial intelligence is"
-
- # Generate text
- result = await generate_text(prompt, max_length=100)
-
- # Display results
- print("\n" + "="*60)
- print("TEXT GENERATION RESULTS")
- print("="*60)
- print(f"\nPrompt: {result['prompt']}")
- print(f"\nGenerated text:\n{result['generated_text']}")
- print("\n" + "-"*60)
- print(f"Model: {result['model_name']}")
- print(f"Device: {result['device']}")
- print(f"GPU: {result['device_name']}")
- print(f"Max length: {result['max_length']} tokens")
- print("="*60)
+ print("Generating text with GPT-2 on a Runpod GPU...")
+ result = await generate_text("The future of artificial intelligence is", max_new_tokens=60)
+ print(f"\n{result['generated_text']}\n")
+ print(f"GPU: {result['gpu']}")
+ print(f"Model already loaded: {result['model_cached']}")
+ print(f"Model load: {result['load_seconds']}s, generation: {result['generate_seconds']}s")
+
if __name__ == "__main__":
asyncio.run(main())
```
-This main function:
+
-- Calls the remote function with `await` (runs asynchronously).
-- Waits for the GPU worker to complete text generation.
-- Displays the results in a formatted output.
+## Step 7: Deploy and run your first generation
-## Step 7: Run your first generation
+Deploy the endpoint from your project directory:
-Run the application:
+```bash
+flash deploy
+
+# If using uv:
+uv run flash deploy
+```
+
+```text
+✓ built flash-text-generation 1 files, 1 deps, 37.8 MB
+No app 'flash-text-generation' found. Creating app and 'production' environment...
+
+✓ uploaded 37.8 MB 2.1s
+✓ deployed to production 1.9s
+
+ text-generation https://api.runpod.ai/v2/abc123xyz/runsync
+
+curl -X POST https://api.runpod.ai/v2/abc123xyz/runsync \
+ -H "Content-Type: application/json" \
+ -H "Authorization: Bearer $RUNPOD_API_KEY" \
+ -d '{"input": {}}'
+```
+
+Then run the script from the same directory:
```bash
python text_generation.py
```
-**First run output** (takes 60-90 seconds):
+The first run takes about a minute while a worker starts and loads the model:
```text
-Starting text generation on Runpod GPU...
-Creating endpoint: server_Endpoint_a1b2c3d4
-Provisioning Serverless endpoint...
-Endpoint ready
-Registering Runpod endpoint at https://api.runpod.ai/xvf32dan8rcilp
-Executing function on Runpod endpoint ID: xvf32dan8rcilp
-Initial job status: IN_QUEUE
-Installing dependencies: transformers torch accelerate
-Downloading model weights...
-Job completed, output received
-
-============================================================
-TEXT GENERATION RESULTS
-============================================================
-
-Prompt: The future of artificial intelligence is
-
-Generated text:
-The future of artificial intelligence is bright and full of possibilities. With advancements in machine learning and deep learning, we're seeing AI systems that can understand natural language, recognize images, and even create art. The potential applications are endless, from healthcare to transportation to education.
-
-------------------------------------------------------------
-Model: gpt2
-Device: cuda
+Generating text with GPT-2 on a Runpod GPU...
+
+The future of artificial intelligence is bright, and it will be exciting to see if we can add a few more layers of complexity into our lives to become a better person.
+
GPU: NVIDIA GeForce RTX 4090
-Max length: 100 tokens
-============================================================
+Model already loaded: False
+Model load: 9.1s, generation: 0.76s
```
-**Subsequent runs** (takes 2-5 seconds):
+Run it again while the worker is warm. The model is already loaded, so the whole run takes about 2 seconds:
```text
-Starting text generation on Runpod GPU...
-Resource Endpoint_a1b2c3d4 already exists, reusing.
-Registering Runpod endpoint at https://api.runpod.ai/xvf32dan8rcilp
-Executing function on Runpod endpoint ID: xvf32dan8rcilp
-Initial job status: IN_QUEUE
-Job completed, output received
-
-[Results appear immediately]
+Model already loaded: True
+Model load: 0.0s, generation: 0.51s
```
-Notice the dramatic speed improvement on subsequent runs—the endpoint is already provisioned, dependencies are installed, and the model is cached.
+Your completions will differ because `do_sample=True` picks tokens at random.
-## Step 8: Experiment with different prompts
+
+The first request on a new worker can take longer than the 90 seconds `runpod-flash` 1.21.0 waits for a result, especially if the worker has to wait for a free GPU. If the script fails with `timed out after 90s. the endpoint may not be deployed or the worker is still starting` or `KeyError: 'generated_text'`, the endpoint is deployed and the worker is still starting. Run the script again. See [First run fails after 90 seconds](#first-run-fails-after-90-seconds).
+
+
+## Step 8: Try several prompts in parallel
-Modify the main function to try different prompts:
+Replace `main()` with a version that sends three prompts at once:
```python
async def main():
- print("Starting text generation on Runpod GPU...")
-
- # Try multiple prompts
prompts = [
"Once upon a time in a distant galaxy",
"The secret to happiness is",
- "In the year 2050, technology will"
+ "In the year 2050, technology will",
]
+ results = await asyncio.gather(
+ *(generate_text(prompt, max_new_tokens=40) for prompt in prompts)
+ )
+ for result in results:
+ print(f"\n{result['generated_text']}")
- for prompt in prompts:
- print(f"\n{'='*60}")
- print(f"Generating for: {prompt}")
- print('='*60)
-
- result = await generate_text(prompt, max_length=80)
- print(f"\n{result['generated_text']}\n")
if __name__ == "__main__":
asyncio.run(main())
```
-Run it again:
+You only changed code outside the `@Endpoint` function, so you don't need to redeploy. Run it again:
```bash
python text_generation.py
```
-You'll see three different completions generated sequentially on the same GPU worker.
+All three requests are sent at once. A warm worker returned all three completions in about 3 seconds. If requests queue up, Flash starts more workers, up to `workers=3`, and each new worker loads the model on its first request.
## Troubleshooting
@@ -353,7 +383,7 @@ You'll see three different completions generated sequentially on the same GPU wo
**Solutions**:
1. Use smaller models (GPT-2 instead of GPT-2 Large).
-2. Reduce `max_length` parameter.
+2. Reduce `max_new_tokens`.
3. Use larger GPUs:
```python
gpu=GpuGroup.AMPERE_48 # 48GB GPUs
@@ -361,20 +391,44 @@ You'll see three different completions generated sequentially on the same GPU wo
### Slow generation
-**Issue**: Text generation takes >30 seconds per request.
+**Issue**: Every request takes much longer than a few seconds.
-**Possible causes**:
-1. Worker scaled down (cold start).
-2. Model not cached.
-3. Large `max_length` value.
+**Check `model_cached` in the result.** If it's `False` on most requests, workers are loading the model again:
-**Solutions**:
-1. Increase `idle_timeout` to keep workers active:
+1. Make sure `_model` and `_tokenizer` are module-level variables and `generate_text` declares `global _model, _tokenizer`.
+2. Increase `idle_timeout` so workers stay up between requests:
```python
- idle_timeout=1800 # Keep active for 30 minutes
+ idle_timeout=1800 # Keep workers warm for 30 minutes
```
-2. Set `workers=(1, 3)` to always have a warm worker ready.
-3. Reduce `max_length` to generate fewer tokens.
+3. Set `workers=(1, 3)` to keep one worker running at all times. You pay for that worker while it's idle.
+
+If `model_cached` is `True` and generation is still slow, reduce `max_new_tokens`.
+
+### First run fails after 90 seconds
+
+**Issue**: The first run fails with one of these errors:
+
+```text
+RuntimeError: request to endpoint 'text-generation' timed out after 90s. the endpoint may not be deployed or the worker is still starting. deploy with 'flash deploy' or increase timeout with FLASH_SENTINEL_TIMEOUT env var.
+```
+
+```text
+KeyError: 'generated_text'
+```
+
+**Cause**: `runpod-flash` 1.21.0 waits up to 90 seconds for a result. The first request on a new worker can take longer: the worker may wait for a free GPU, then it starts and loads the model. The endpoint is deployed, and the request keeps running after the error.
+
+**Solution**: Run the script again. The worker keeps starting after the error, so the next run is faster. Raising `FLASH_SENTINEL_TIMEOUT` above 90 doesn't help in 1.21.0: the request returns at 90 seconds with the job still running, and the script fails with the `KeyError` instead.
+
+To see whether a worker is up, check the endpoint's health with its ID from `flash deploy`. `curl` reads your API key from the `RUNPOD_API_KEY` environment variable, which `flash login` doesn't set, so export it first:
+
+```bash
+export RUNPOD_API_KEY="your_key"
+curl https://api.runpod.ai/v2/ENDPOINT_ID/health \
+ -H "Authorization: Bearer $RUNPOD_API_KEY"
+```
+
+A non-zero `workers.throttled` means the worker is waiting for a GPU. A non-zero `workers.idle`, `workers.ready`, or `workers.running` means a worker is up.
### Generation quality is poor
@@ -384,9 +438,9 @@ You'll see three different completions generated sequentially on the same GPU wo
1. Adjust `temperature` (try 0.7-0.9)
2. Add `top_p` and `top_k` sampling:
```python
- outputs = model.generate(
+ outputs = _model.generate(
**inputs,
- max_length=max_length,
+ max_new_tokens=max_new_tokens,
temperature=0.8,
top_p=0.9,
top_k=50,
@@ -403,15 +457,17 @@ Now that you've built a text generation script with Flash, you can:
Try different models from Hugging Face:
+Change `MODEL_NAME` and run `flash deploy` again:
+
```python
-# Instruction-following model
-model_name = "facebook/opt-1.3b"
+# Larger general-purpose model (1.3B parameters)
+MODEL_NAME = "facebook/opt-1.3b"
# Code generation model
-model_name = "Salesforce/codegen-350M-mono"
+MODEL_NAME = "Salesforce/codegen-350M-mono"
# Dialogue model
-model_name = "microsoft/DialoGPT-medium"
+MODEL_NAME = "microsoft/DialoGPT-medium"
```
### Build a chat interface
@@ -438,17 +494,16 @@ Convert your script to a production [Flash app](/flash/apps/overview):
```bash
flash init text-generation-app
-# Move your function to workers/gpu/endpoint.py
-# Add FastAPI routes
+cd text-generation-app
+# Move generate_text into gpu_worker.py
+# Add HTTP routes in lb_worker.py
flash deploy
-
-# If using uv:
-uv run flash init text-generation-app
-uv run flash deploy
```
+`flash init` creates `gpu_worker.py`, `cpu_worker.py`, and `lb_worker.py` at the project root.
+
-When deploying queue-based functions with `flash deploy`, each function must have its own unique endpoint configuration. If your script has multiple functions sharing the same config (like `generate_text` and `chat` in this tutorial), create separate endpoints for each function when converting to a Flash app. See [understanding endpoint architecture](/flash/apps/deploy-apps#understanding-endpoint-architecture) for details.
+When deploying queue-based functions with `flash deploy`, each function must have its own unique endpoint configuration. `generate_text` and `chat` in this tutorial already have separate configs (`name="text-generation"` and `name="chat"`), so keep them that way when you move them into a Flash app. See [understanding endpoint architecture](/flash/apps/deploy-apps#understanding-endpoint-architecture) for details.
### Optimize performance