Skip to content

[FEAT] Robust chain ladder that detects and adjusts outlying cells (Verdonck, Van Wouwe and Dhaene 2009) #1679

Open

Description

@scks1

Verdonck, Van Wouwe and Dhaene, A Robustification of the Chain-Ladder Method, North American Actuarial Journal 13(2), 2009, pages 280 to 298, multiply one cell of the Taylor and Ashe triangle by 10 and refit. The chain-ladder reserve moves from 18.7M to anything between 12.6M and 60.3M, depending on the cell, and Table 9 lists each one.

Table 7 on page 287 gives their fix in five steps.

  1. Fit the triangle with median link ratios and compute Pearson residuals as in England and Verrall (1999).
  2. Flag residuals outside Q1 - 3 IQR and Q3 + 3 IQR.
  3. Replace a flagged first-column cell by the next cell divided by the median ratio of the second column to the first, or by the first-column median when the next cell is also flagged.
  4. Refit the later columns as median multiples of the first column and pull each flagged residual back to the median residual.
  5. Run the classical chain ladder on the adjusted triangle.

On clean data the method flags nothing and returns the classical reserve. Section 5 recommends running both and looking at the data when they differ.

chainladder has no step that adjusts cells before a fit. BerquistSherman already takes a Triangle and returns an adjusted Triangle, so a transformer in chainladder.adjustments could follow that shape, list the flagged cells and feed Development, Chainladder or MackChainladder unchanged.

Step 1 needs no new average value. Development(average="simple") with per-age drop_high and drop_low counts reproduces np.nanmedian of the link ratios on genins.

genins is the paper's Table 8. The classical reserves in the snippet match Table 9 to the unit. A NumPy reading of the five steps matches the robust column of Table 9 to the unit in 40 of 55 rows, and X12 and X52 below are two of them. Four of the rows that miss by more than 1% put the outlier in a corner cell or in the last two columns, where Section 5.2 adds a curve fit to the late link ratios. The snippet leaves that step out.

The paper does not use Mack's sigma. Section 7.1 bootstraps the robust estimator and swaps the sample standard deviation for the median absolute deviation, after Stromberg (1997). Table 10 reports a bootstrap standard error of 2,190,625 for the robust method on clean data, against 1,599,681 for the classical one. MackChainladder on an adjusted triangle would treat each replaced cell as observed and would leave out the uncertainty the replacement adds.

Should the issue cover only the adjusted triangle, with a docstring note on std_err_, or should it also add the Section 7.1 bootstrap?

Snippet
import numpy as np
import chainladder as cl

genins = cl.load_sample("genins")
inc = genins.cum_to_incr().values[0, 0]
n = inc.shape[0]
obs = ~np.isnan(inc)


def reserve(x):
    tri = genins.copy()
    tri.values = np.where(obs, np.nancumsum(x, axis=1), np.nan)[None, None]
    return round(float(cl.Chainladder().fit(tri).ibnr_.sum()))


def outlying(r, pool):
    q1, q3 = np.percentile(pool, [25, 75])
    return (r < q1 - 3 * (q3 - q1)) | (r > q3 + 3 * (q3 - q1))


def robust_cl(x):
    # Table 7 of Verdonck, Van Wouwe and Dhaene (2009), steps 1 to 5.
    x = x.copy()
    c = np.where(obs, np.nancumsum(x, axis=1), np.nan)
    f = [np.nanmedian(c[:, j] / c[:, j - 1]) for j in range(1, n)]
    fit = np.full_like(c, np.nan)
    for i in range(n):
        fit[i, n - i - 1] = c[i, n - i - 1]
        for j in range(n - i - 1, 0, -1):
            fit[i, j - 1] = fit[i, j] / f[j - 1]
    m = np.diff(np.concatenate([np.zeros((n, 1)), fit], axis=1), axis=1)
    r = (x - m) / np.sqrt(np.abs(m))
    out = outlying(r, r[obs]) & obs
    flagged = 0
    for k in range(n):
        if out[k, 0]:
            flagged += 1
            if k < n - 1 and not out[k, 1]:
                x[k, 0] = x[k, 1] / np.nanmedian(x[: n - 1, 1] / x[: n - 1, 0])
            else:
                x[k, 0] = np.nanmedian(c[:, 0])
    f1 = np.array([np.nanmedian(x[: n - j, j] / x[: n - j, 0]) for j in range(n)])
    fit1 = x[:, [0]] * f1[None, :]
    r1 = (x - fit1) / np.sqrt(np.abs(fit1))
    upper = obs.copy()
    upper[:, 0] = False
    out1 = outlying(r1, r1[upper]) & upper
    x = np.where(out1, fit1 + np.median(r1[upper]) * np.sqrt(np.abs(fit1)), x)
    return reserve(x), flagged + int(out1.sum())


print("clean", reserve(inc), robust_cl(inc))
for i, j in [(0, 1), (4, 1)]:
    x = inc.copy()
    x[i, j] *= 10
    print(f"X{i + 1}{j + 1} x 10", reserve(x), robust_cl(x))

lr = genins.link_ratio.values[0, 0]
k = (~np.isnan(lr)).sum(axis=0)
cut = [int((v - 1) // 2) if v % 2 else int((v - 2) // 2) for v in k]
dev = cl.Development(average="simple", drop_high=cut, drop_low=cut).fit(genins)
print("median ldf", np.allclose(dev.ldf_.values[0, 0, 0], np.nanmedian(lr, axis=0)))
clean 18680856 (18680856, 0)
X12 x 10 12851784 (18411731, 1)
X52 x 10 17682939 (18483164, 1)
median ldf True

(This issue was written with Árbol, an AI assistant, under my direction.)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    Priority

    None yet

    Scope

    None yet

    Effort

    None yet

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions