cmd/seed: synthetic dataset for local development by ZukwiZ · Pull Request #70 · csfloat/reverse-watch

ZukwiZ · 2026-05-27T17:07:49Z

Summary

Add a deterministic ~6-month synthetic dataset (~9.8k rows, gentle sinusoid with occasional spikes and quiet days) for exercising the dashboard locally without needing a real production export. The generator deliberately spans every period the chart picker offers (7d / 30d / 3m / 6m / 1y), so any range renders meaningful data.

Safety properties

Refuses to run unless `Config.Environment == "development"`.
`INSERT … ON CONFLICT (id) DO NOTHING`, so re-running is a no-op.
Steam IDs use a clearly-synthetic `76561198000000000` prefix; real Steam IDs sit higher up the 64-bit range, so generated rows are easy to spot in the DB.
Snowflake IDs encode the same `created_at` + sequence layout as the production generator (`domain/models/snowflake.go`), so synthetic rows sort chronologically alongside any real rows already present.

.gitignore

Adds `internal-docs/` (author scratch space) and `internal/devseed/fixtures/` (room for any future local-only CSV fixtures) so neither leaks into the public repo.

Test plan

`go build ./cmd/seed`
Against a local dev DB with `Environment=development`: `go run ./cmd/seed` prints `generated …` and `seed complete: N inserted, 0 already present`.
Re-run: prints `seed complete: 0 inserted, N already present` (idempotent).
With `Environment` set to anything else: exits non-zero with a refusal message.
After seeding, the dashboard at `/` shows ~6 months of activity across all period picker ranges.

Made with Cursor

Note

Low Risk
Dev-only CLI with an environment guard and idempotent inserts; not linked to the production server binary.

Overview
Adds a dev-only seed path so local Postgres can hold a deterministic ~6-month reversal history (~9.8k rows) without production exports.

go run ./cmd/seed loads config, exits unless Environment is development, then bulk-inserts generated rows into the public DB via ON CONFLICT (id) DO NOTHING (safe to re-run). internal/devseed builds daily volume with variance, marketplace mix, sources, optional expungements, synthetic Steam IDs, and snowflake IDs aligned with production ordering.

README documents the workflow and dashboard period coverage; .gitignore excludes internal-docs/ and optional internal/devseed/fixtures/.

^{Reviewed by Cursor Bugbot for commit b067b6b. Bugbot is set up for automated code reviews on this repo. Configure here.}

Add a deterministic ~6-month synthetic dataset (~9.8k rows, gentle sinusoid with occasional spikes and quiet days) for exercising the dashboard locally without needing real production exports. The generator deliberately spans every period (7d / 30d / 3m / 6m / 1y) so the chart UI has data to render at any range. Safety properties: - Refuses to run unless Config.Environment == "development". - INSERT … ON CONFLICT (id) DO NOTHING, so re-running is a no-op. - Steam IDs use a clearly-synthetic 76561198000000000 prefix. - Snowflake IDs encode the same created_at + sequence layout as the production generator, so synthetic rows sort chronologically alongside any real rows already in the DB. internal-docs/ and internal/devseed/fixtures/ are added to .gitignore to keep author scratch space and any future local CSV fixtures out of the public repo. Co-authored-by: Cursor <cursoragent@cursor.com>

zedimytch · 2026-06-22T20:50:57Z

+const (
+	syntheticRNGSeed      int64  = 42
+	syntheticDays                = 180
+	syntheticTargetTotal         = 9800


Specify the datatype here.
Also, we probably don't need 9800 entries, a couple thousand should work.

Done — syntheticTargetTotal int = 2000.

zedimytch · 2026-06-22T20:51:01Z

+// days). Snowflake IDs are unique within the slice and won't collide with
+// real CSV-seeded IDs, so callers can pipe the result straight into
+// InsertReversals.
+func GenerateSynthetic(now time.Time) []*models.Reversal {


Let's normalize now() to UTC before we derive any dates.

Done — now() is normalized to UTC before any dates are derived.

zedimytch · 2026-06-22T20:51:07Z

+The seed:
+
+- Refuses to run unless `Environment` is `development`.
+- Uses `INSERT … ON CONFLICT (id) DO NOTHING`, so it's safe to re-run.
+- Generates a deterministic 6-month dataset so the dashboard at `/` has enough data to exercise every period (7d / 30d / 3m / 6m / 1y).
+- Uses a synthetic Steam ID prefix (`76561198000000000`) so generated IDs are clearly fake.


You can just state that this must be ran in a development environment. Also, please move this to the end of the README.

Done — simplified to "must be run in a development environment" and moved the section to the end of the README.

- Give syntheticTargetTotal an explicit int type and reduce the seed to ~2,000 rows (a couple thousand) for faster local seeding. - Normalize the incoming now to UTC before deriving any dates so the generated series is timezone-independent. - Simplify the README seeding guard to "must be run in a development environment", move the seeding section to the end of the README, and stop claiming idempotency (reruns add more data since IDs derive from wall-clock time). Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit c4d8523. Configure here.}

Re-running the seed on a different day produces new snowflake IDs but the same deterministic (steam_id, marketplace_slug) pairs, which collide with the partial unique index idx_reversals_steam_id_marketplace_slug. Target that natural key (with the WHERE deleted_at IS NULL predicate) in the ON CONFLICT DO NOTHING clause so reruns skip existing rows instead of raising a unique-constraint error, and correct the README/doc comments to describe the idempotent behavior. Also clamp the synthetic createdAt to models.Epoch before the unsigned snowflake timestamp subtraction so an out-of-range clock can't underflow into a garbage ID.

ZukwiZ · 2026-07-13T09:51:05Z

hey @zedimytch, nothing new needed on this one since the last round — think it's good to go whenever you have a sec 🙏

zedimytch

Thanks for this — the overall design is solid. Builds, go vet is clean, and both internal/devseed tests pass against local Postgres. Requesting changes for three issues:

Seeding breaks the local CSFloat ingestor's cursor. csfloatIngestor.sync resumes from MAX(reporter_internal_id) over csfloat rows. ~80% of synthetic rows are csfloat with reporter IDs ~2,900,001–2,902,000, so after seeding, a locally running ingestor resumes from ~2.9M and silently skips real warnings below that. Note that just setting ReporterInternalID = nil isn't safe either: Postgres sorts NULLs first on DESC, so the cursor would become nil and the ingestor would full-resync every 30 minutes. Suggest using a non-real slug for synthetic rows, or nil reporter IDs plus NULLS LAST in the ingestor query.
The Steam IDs aren't "clearly fake". 76561198000000000 is account ID ~39.7M — real accounts created around 2009. Real IDs don't "sit higher up the range" as the description says. With #71 showing avatars/names/profile links, real people would show up as flagged. A base near the top of the 32-bit account space (e.g. 76561197960265728 + 0xF0000000) still passes SteamID.IsValid and isn't allocated.
Reruns never refresh the data. The (steam_id, marketplace_slug) pairs don't depend on now, so a rerun inserts 0 rows (the rerun test asserts exactly this). A week after seeding, the 7d window is empty and re-running doesn't help. Suggest a -reset flag that deletes rows in the synthetic Steam ID range before inserting (straightforward once #2 gives them a dedicated range), and documenting it in the README.

Minor items are inline. The PR description is also stale: it still says ~9.8k rows and ON CONFLICT (id).

zedimytch · 2026-10-09T18:43:06Z

+			sfTs := createdAtMs - models.Epoch
+			sf := models.Snowflake((sfTs << 22) | uint64(seq))
+
+			reporter := syntheticBaseReporter + uint(steamOffset)


These reporter IDs end up on csfloat rows, and the CSFloat ingestor resumes from MAX(reporter_internal_id) for that slug (ingestors/csfloat/csfloat.go sync). After seeding, a local ingestor would resume from ~2.9M and skip real warnings below it. nil isn't a safe alternative on its own, because Postgres puts NULLs first on ORDER BY ... DESC, which would reset the cursor on every sync.

zedimytch · 2026-10-09T18:43:06Z

+	syntheticRNGSeed      int64  = 42
+	syntheticDays                = 180
+	syntheticTargetTotal  int    = 2000
+	syntheticBaseSteamID  uint64 = 76561198000000000


This works out to account ID ~39.7M, which are real 2009-era accounts, so these IDs aren't actually synthetic. Consider basing off the top of the 32-bit account space, e.g. 76561197960265728 + 0xF0000000, which passes SteamID.IsValid and isn't allocated. A dedicated range would also make a -reset cleanup easy.

zedimytch · 2026-10-09T18:43:06Z

+
+// GenerateSynthetic returns a ~6-month dataset (~2,000 rows, at least one
+// per day, gentle sinusoid with occasional spikes / quiet days). Snowflake
+// IDs are unique within the slice and won't collide with real CSV-seeded


nit: "CSV-seeded IDs" doesn't exist in this repo. Looks like a leftover from an earlier iteration.

zedimytch · 2026-10-09T18:43:06Z

+		for i := 0; i < counts[d]; i++ {
+			reversedAt := dayStart.Add(time.Duration(rng.Float64() * float64(24*time.Hour)))
+			if uint64(reversedAt.UnixMilli()) > nowMs {
+				reversedAt = now.Add(-1 * time.Minute)


For today's bucket, every timestamp later than now gets clamped to now - 1m here (and createdAt to now below). If you seed early in the day, most of today's rows land on the same minute, which shows up as a spike in the 24h KPI and the recent table. Sampling uniformly in [dayStart, now) for the last day would avoid that.

zedimytch · 2026-10-09T18:43:06Z

+		if end > len(reversals) {
+			end = len(reversals)
+		}
+		res := db.Clauses(onConflict).Create(reversals[i:end])


Two small things:

ON CONFLICT (steam_id, marketplace_slug) only skips that index. A primary-key collision on id would still fail the whole batch. It's unlikely here, but worth a comment.

The factory opens the DB with gorm logger.Info, so this prints the full SQL for each 1,000-row batch. Consider db.Session(&gorm.Session{Logger: logger.Default.LogMode(logger.Warn)}) for the seed.

zedimytch · 2026-10-09T18:43:06Z

+		t.Errorf("rerun inserted %d rows, want 0 (idempotent)", n2)
+	}
+
+	if first[0].ID == second[0].ID {


nit: this assertion doesn't test anything about the seeder (different now → different IDs is a given). It can be dropped.

zedimytch · 2026-10-09T18:43:07Z

+
+- Must be run in a development environment.
+- Generates a 6-month dataset so the dashboard at `/` has enough data to exercise every period (7d / 30d / 3m / 6m / 1y).
+- Uses a synthetic Steam ID prefix (`76561198000000000`) so generated IDs are clearly fake.


76561198000000000 falls in the range of real accounts (~2009), so "clearly fake" isn't accurate. See the comment on syntheticBaseSteamID.

zedimytch · 2026-10-09T18:43:07Z

+- Must be run in a development environment.
+- Generates a 6-month dataset so the dashboard at `/` has enough data to exercise every period (7d / 30d / 3m / 6m / 1y).
+- Uses a synthetic Steam ID prefix (`76561198000000000`) so generated IDs are clearly fake.
+- Is safe to re-run: rows are de-duplicated on `(steam_id, marketplace_slug)` via `ON CONFLICT DO NOTHING`, so a rerun skips rows that already exist instead of raising a unique-constraint error.


"Safe to re-run" is true, but a rerun never adds fresh data because the dedupe key doesn't depend on the date. Developers will likely re-run expecting a refresh. Worth documenting how to reset (ideally via a -reset flag).

zedimytch · 2026-10-09T18:43:07Z

+	logging.Initialize()
+	cfg := config.Load()
+
+	if cfg.Environment != constants.EnvironmentDevelopment {


Environment defaults to development in config.go, so this guard only catches an explicit production setting. Fine for a dev tool, but a check that Database.Host is local (localhost / 127.0.0.1 / docker service name) would make it much harder to seed the wrong database by accident.

zedimytch · 2026-10-09T18:43:07Z

+	reversals := devseed.GenerateSynthetic(time.Now().UTC())
+	fmt.Printf("generated %d synthetic reversals (deterministic seed)\n", len(reversals))
+
+	inserted, err := devseed.InsertReversals(f.PublicDB(), reversals)


nit: os.Exit(1) on failure below skips the deferred f.Close(). Harmless at exit, but returning an error from a run() func and exiting in main would keep it consistent.

cursor Bot reviewed May 27, 2026

View reviewed changes

Comment thread cmd/seed/main.go

zedimytch self-requested a review June 3, 2026 16:54

zedimytch requested changes Jun 22, 2026

View reviewed changes

cursor Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread internal/devseed/synthetic.go Outdated

Comment thread internal/devseed/synthetic.go

ZukwiZ requested a review from zedimytch June 24, 2026 13:40

zedimytch requested changes Oct 9, 2026

View reviewed changes

Conversation

ZukwiZ commented May 27, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Safety properties

.gitignore

Test plan

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ZukwiZ commented Jul 13, 2026

Uh oh!

zedimytch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ZukwiZ commented May 27, 2026 •

edited by cursor Bot

Loading