Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion docs/cost-tracking.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,24 @@ runs use the best available `updated_at` timestamp during migration.

## API key spending limits

Starting with 0.13.3, set an initial budget when creating a token:

```sh
ow token create --name experiment --spending-limit-usd 100
```

This works with the existing `OPENWEIGHTS_API_KEY` credentials authorized to create
tokens. The dashboard's token creation dialog also accepts a lifetime USD limit.
Omit the option (or leave the field blank) for unlimited spending; `0` blocks work.
The token and budget are created atomically: failure leaves neither behind.
The REST token creation endpoint accepts `spending_limit_usd` alongside `name`
and `expires_in_days`. Apply migration `20260922180000_token_creation_limit.sql`
before upgrading the dashboard or using the new CLI option. Worker images remain
v0.13.1.

Choosing a new token's initial budget uses existing token-creation permissions.
Changing or removing an existing budget still requires a signed-in admin user.

A signed-in organization **admin user** can choose an API key in the Costs page and
save a lifetime USD limit. `0` blocks work; blank removes the limit. API-key logins
can view costs but cannot edit budgets, even though older organization APIs treat
Expand Down Expand Up @@ -86,7 +104,10 @@ can add further overhead. Budgeted jobs cannot start on unpriced workers.
before a threshold is reached, startup cost may only be allocated when a worker
first runs a job, and shutdown delays, manager outages and later overhead can cause
overspend. Limits do not reserve the estimated maximum price of pending jobs.
Previously canceled jobs require an explicit restart after increasing a limit.
For example, if 60 out of 100 jobs have completed when a key exhausts its budget,
the 60 completed jobs remain completed and the remaining queued/running jobs are
marked `canceled`. Jobs attributed to other keys are unaffected. Previously canceled
jobs require an explicit restart after increasing a limit.
Limits attach to individual keys, not to all credentials held by a person; this
feature does not turn the existing organization-admin API keys into untrusted,
restricted credentials. Use independently managed keys and trusted organization
Expand Down
12 changes: 12 additions & 0 deletions docs/release-0.13.3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# OpenWeights 0.13.3

Create a token with a lifetime USD budget using
`ow token create --name experiment --spending-limit-usd 100`, or the dashboard's
new spending-limit field. The token and budget are saved atomically. Existing
API-key credentials authorized to create tokens can choose an initial budget;
editing existing budgets still requires a signed-in organization admin.

Apply `supabase/migrations/20260922180000_token_creation_limit.sql` before using
the new option. Worker images remain v0.13.1. Budget exhaustion continues to cancel
queued/running jobs for the key and reject new submissions; completed jobs stay
completed. Limits include allocated overhead and may be exceeded during shutdown.
33 changes: 24 additions & 9 deletions openweights/cli/token.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import os
import sys
from datetime import datetime, timedelta, timezone
from decimal import Decimal

from openweights import OpenWeights

Expand All @@ -28,6 +29,11 @@ def add_token_parser(parser):
help="Number of days until token expires (optional, default: no expiration)",
)

create_parser.add_argument(
"--spending-limit-usd",
help="Lifetime USD limit including overhead (default: unlimited)",
)

# revoke command
revoke_parser = subparsers.add_parser("revoke", help="Revoke (delete) a token")
revoke_parser.add_argument(
Expand Down Expand Up @@ -131,6 +137,13 @@ def handle_token_ls(args) -> int:
def handle_token_create(args) -> int:
"""Handle the token create command."""
try:
limit = getattr(args, "spending_limit_usd", None)
if limit is not None:
limit = Decimal(str(limit))
if not limit.is_finite() or limit < 0:
raise ValueError(
"Spending limit must be a finite nonnegative USD amount"
)
ow = get_openweights_client()
org_id = ow.organization_id

Expand All @@ -141,15 +154,12 @@ def handle_token_create(args) -> int:
datetime.now(timezone.utc) + timedelta(days=args.expires_in_days)
).isoformat()

# Call the create_api_token RPC function
result = ow._supabase.rpc(
"create_api_token",
{
"org_id": org_id,
"token_name": args.name,
"expires_at": expires_at,
},
).execute()
params = {"org_id": org_id, "token_name": args.name, "expires_at": expires_at}
rpc = "create_api_token"
if limit is not None:
rpc = "create_api_token_with_limit"
params["spending_limit_usd"] = str(limit)
result = ow._supabase.rpc(rpc, params).execute()

if not result.data or len(result.data) == 0:
print("Error: Failed to create token")
Expand All @@ -161,6 +171,11 @@ def handle_token_create(args) -> int:
print("-" * 80)
print(f"Token ID: {token_data['token_id']}")
print(f"Name: {args.name}")
print(
f"Spending limit: ${limit} lifetime USD"
if limit is not None
else "Spending limit: Unlimited"
)
if expires_at:
print(f"Expires: {format_datetime(expires_at)}")
else:
Expand Down
19 changes: 10 additions & 9 deletions openweights/dashboard/backend/database.py
Original file line number Diff line number Diff line change
Expand Up @@ -368,15 +368,16 @@ async def create_token(
days=token_data.expires_in_days
)

# Create API token using the client (RLS will handle authorization)
result = self.client.rpc(
"create_api_token",
{
"org_id": organization_id,
"token_name": token_data.name,
"expires_at": expires_at.isoformat() if expires_at else None,
},
).execute()
params = {
"org_id": organization_id,
"token_name": token_data.name,
"expires_at": expires_at.isoformat() if expires_at else None,
}
rpc = "create_api_token"
if token_data.spending_limit_usd is not None:
rpc = "create_api_token_with_limit"
params["spending_limit_usd"] = str(token_data.spending_limit_usd)
result = self.client.rpc(rpc, params).execute()

if not result.data or len(result.data) == 0:
raise ValueError("Failed to create token")
Expand Down
4 changes: 4 additions & 0 deletions openweights/dashboard/backend/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -440,6 +440,10 @@ async def create_token(
):
try:
return await db.create_token(organization_id, token_data)
except APIError as exc:
raise HTTPException(
status_code=403 if exc.code == "42501" else 400, detail=exc.message
)
except ValueError as e:
raise HTTPException(status_code=403, detail=str(e))
except Exception as e:
Expand Down
6 changes: 5 additions & 1 deletion openweights/dashboard/backend/models.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
from datetime import datetime
from decimal import Decimal
from typing import Any, Dict, List, Optional

from pydantic import BaseModel
from pydantic import BaseModel, Field


class Organization(BaseModel):
Expand Down Expand Up @@ -91,6 +92,9 @@ class WorkerWithRuns(Worker):
class TokenCreate(BaseModel):
name: str
expires_in_days: Optional[int] = None # None means no expiration
spending_limit_usd: Optional[Decimal] = Field(
default=None, ge=0, allow_inf_nan=False
)


class Token(BaseModel):
Expand Down

Large diffs are not rendered by default.

3,855 changes: 3,855 additions & 0 deletions openweights/dashboard/backend/static/assets/MetricsPlots-BUaa-TxX.js

Large diffs are not rendered by default.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading