Skip to content

Common random numbers across scenarios in run_sampled_sims #555

Open

Description

@RomeshA

Found in review of #539.

Problem

_run_sampled_sim() creates one generator per attempt (rng_attempt) and passes the same object to parset.sample(), progset.sample() and then to every run_sim() call across the list of progset_instructions. Because the generator is stateful, the second scenario starts where the first scenario's random stream ended. Verified: two scenarios run from the same sample have different trajectories before the program start year, when they should be identical.

The code comment ("it's important to START from the same seeds") describes the intended behaviour, not the actual one. Without common random numbers (CRN) the variance of scenario differences is much larger than it needs to be, which is the main thing an uncertainty analysis of interventions cares about.

Even with the fix below, CRN is only partial: Generator.multinomial consumes a variable number of uniforms depending on n, so streams desynchronise after the scenarios diverge.

Implementation plan

  1. In _run_sampled_sim(), draw a seed per attempt, and for each instruction set create np.random.default_rng(seed) afresh before calling run_sim(). Parameter and program sampling should use a separate child seed so that the model stream is identical across scenarios regardless of how much sampling consumed.
  2. Use np.random.SeedSequence(rand_seed).spawn(n_samples) in run_sampled_sims() and again per attempt, rather than rng.integers(1e15), which is the documented way to derive independent child streams.
  3. Add a test: run two instruction sets from one sample and assert trajectories are identical up to the first year at which the instructions differ.
  4. (Follow-on, larger) Full CRN: give each compartment its own Generator spawned from the model seed, so a draw in one compartment never affects the stream of another. This also answers the question on Stochastic transitions #474 about per-compartment generators. Draws per compartment per timestep are then synchronised across scenarios as long as n is the same, and partially otherwise. Measure the variance reduction on the SIR demo before and after.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions