Skip to content

[DOC] Gallery example: compare two methods on the same bootstrap draws #1664

Closed

Description

@scks1

The gallery shows bootstrap results for one setup at a time, and plot_bootstrap_comparison sets two setups side by side as histograms. A user who wants to know whether two methods differ needs the distribution of the difference for each draw. Fitting both methods to the same BootstrapODPSample output gives that directly. The two methods share the draws, so the correlation cancels most of the sampling noise in the difference.

On genins with 200 draws, TailCurve and TailBondy give the total IBNR differences below.

import numpy as np
import chainladder as cl

genins = cl.load_sample("genins")


def total_ibnr(tri, tail):
    tri = tail.fit_transform(cl.Development().fit_transform(tri))
    return cl.Chainladder().fit(tri).ibnr_.sum("origin").values.flatten()


sims = cl.BootstrapODPSample(n_sims=200, random_state=42).fit_transform(genins)
other = cl.BootstrapODPSample(n_sims=200, random_state=7).fit_transform(genins)

curve = total_ibnr(sims, cl.TailCurve())
bondy = total_ibnr(sims, cl.TailBondy())
bondy_other = total_ibnr(other, cl.TailBondy())

paired = bondy - curve
independent = bondy_other - curve
print("paired      mean %.0f sd %.0f" % (paired.mean(), paired.std(ddof=1)))
print("independent mean %.0f sd %.0f" % (independent.mean(), independent.std(ddof=1)))
print("corr %.4f" % np.corrcoef(curve, bondy)[0, 1])
print("share Bondy > Curve %.3f" % (bondy > curve).mean())
paired      mean -537354 sd 1488592
independent mean -707677 sd 4917204
corr 0.9059
share Bondy > Curve 0.365
Bondy minus Curve mean sd p5 p50 p95
same draws, seed 42 -537,354 1,488,592 -2,726,454 -565,983 2,146,011
independent draws, seeds 7 and 42 -707,677 4,917,204 -8,230,848 -871,496 7,110,872

The two methods correlate at 0.906 across draws, so the paired standard deviation comes out at about a third of the independent one. Bondy gives the larger IBNR in 36.5% of the paired draws. On the original triangle Curve gives 20,245,461 and Bondy gives 19,620,956, a gap of -624,504. I ran this on a59efc9.

The ODP bootstrap here follows England and Verrall, "Stochastic Claims Reserving in General Insurance", British Actuarial Journal volume 8, issue 3, pages 443 to 518, 2002, https://doi.org/10.1017/S1357321700003809, and Shapland 2016, CAS Monograph No. 4, which references.bib already lists. The example needs no new API.

The example relies on one seed giving the same draws on every call. In my run the same seed reproduced the same 200 triangles. #1605 changes how estimators record random_state, so the example should follow whatever #1605 settles.

Would you like this in the gallery, either as a new notebook or as a section of plot_bootstrap_comparison? It looks small enough for a new contributor.

(This issue was written with Árbol, an AI assistant, under my direction.)

Activity

  1. henrydingliu commented on Nov 19, 2026

    @henrydingliu
    Member

    okay. so on a total ibnr_ of around $18M, 500K or 700K diff is a reasonable indication that the these two seeds show off consistent differences in means between the two methods. so then, does the difference in standard deviation disappear with more draws?

  2. scks1 commented on Nov 19, 2026

    @scks1
    ContributorAuthor

    No, the gap stays. Each standard deviation settles at its own value as the draws grow, and the paired one stays about a third of the independent one.

    The paired difference uses the same simulated triangles for both tails, so Var(B - C) = Var B + Var C - 2 Cov(B, C). Independent draws drop the covariance term. With the two totals this correlated and their spreads close, the ratio of the two standard deviations comes out near sqrt(1 - rho). The expected column below computes it from each run's own spreads and correlation.

    n 200 paired sd 1488592 independent sd 4917204 ratio 0.303 expected 0.307 corr 0.9059
      paired mean -537354 se 105259 z -5.1
    n 1000 paired sd 1540822 independent sd 4728455 ratio 0.326 expected 0.339 corr 0.8854
      paired mean -696715 se 48725 z -14.3
    n 2000 paired sd 1580905 independent sd 4773806 ratio 0.331 expected 0.335 corr 0.8881
      paired mean -688972 se 35350 z -19.5
    
    Snippet
    import numpy as np
    import chainladder as cl
    
    genins = cl.load_sample("genins")
    
    
    def total_ibnr(tri, tail):
        tri = tail.fit_transform(cl.Development().fit_transform(tri))
        return cl.Chainladder().fit(tri).ibnr_.sum("origin").values.flatten()
    
    
    for n in (200, 1000, 2000):
        sims = cl.BootstrapODPSample(n_sims=n, random_state=42).fit_transform(genins)
        other = cl.BootstrapODPSample(n_sims=n, random_state=7).fit_transform(genins)
        curve = total_ibnr(sims, cl.TailCurve())
        bondy = total_ibnr(sims, cl.TailBondy())
        paired = bondy - curve
        independent = total_ibnr(other, cl.TailBondy()) - curve
        rho = np.corrcoef(curve, bondy)[0, 1]
        sb, sc = bondy.std(ddof=1), curve.std(ddof=1)
        expected = np.sqrt((sb**2 + sc**2 - 2 * rho * sb * sc) / (sb**2 + sc**2))
        sd_p, sd_i = paired.std(ddof=1), independent.std(ddof=1)
        se = sd_p / np.sqrt(n)
        print("n %d paired sd %.0f independent sd %.0f ratio %.3f expected %.3f corr %.4f"
              % (n, sd_p, sd_i, sd_p / sd_i, expected, rho))
        print("  paired mean %.0f se %.0f z %.1f" % (paired.mean(), se, paired.mean() / se))

    What shrinks with more draws is the standard error of the paired mean, as 1/sqrt(n). It falls from 105259 at 200 draws to 35350 at 2000, and the mean difference holds near -690000 from 1000 draws on. So more draws sharpen the estimate of the difference in means, and pairing keeps the spread of each draw's difference small.

    (This comment was written with Árbol, an AI assistant, under my direction.)

  3. henrydingliu commented on Nov 19, 2026

    @henrydingliu
    Member

    i see. yeah this is a great gallery example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Scope

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions