Reproducing the Paper#
“Interpretable and Validatable Uplift Modeling” [Wil26] makes two kinds of quantitative claim. Some are exact: the levels at which each procedure tests a hypothesis follow from the definitions, and this guide recomputes them with trickle-down as doctests, executed each time the documentation is built. The rest come from the simulation study of §7, which we reproduce from the paper’s description and report beside the paper’s own numbers.
The second kind needs a caveat up front. The paper’s tables were generated by a script that is not public, and its text leaves several details unstated: the distribution of each feature, the learner, and the effect assigned to one subgroup, among others. We filled every gap with an assumption, listed in Assumptions. Two of them were calibrated against the paper’s results, and we say which. A faithful reconstruction should therefore agree with the paper to within Monte Carlo error, not digit for digit, and the tables below show how close it comes.
Generated on 2026-11-02 at revision f84ea1c with 4,000 replications per cell, root seed 261037966, Python 3.11.4, trickle-down 0.1.0, numpy 2.4.6, scipy 1.17.1, scikit-learn 1.9.1.
Levels by Depth (§5, Table 1)#
The procedures of §5 descend from sequential tests that “pay only for failures”: the fixed sequence procedure [Mau95] and the fallback procedure [Wie03]. The Fixed Hierarchy procedure is the hierarchical test of [Mei08] with arbitrary sibling weights, and trickle down is an instance of the graphical approach of [BMBP09], developed in parallel by [BSG09].
Table 1 of the paper compares the level at which each procedure tests a hypothesis at a given depth of a binary tree with equal weights and, for trickle down, a stake of \(1/2\) at every node. Hierarchical Bonferroni tests depth \(k\) at \(2^{-k} \alpha / n_k\) with \(n_k = 2^{k-1}\), which does not depend on the data:
>>> hierarchical_bonferroni = [2.0**-k / 2 ** (k - 1) for k in range(1, 5)]
>>> [f"1/{1 / level:g}" for level in hierarchical_bonferroni]
['1/2', '1/8', '1/32', '1/128']
Trickle down’s levels depend on what was rejected above. We build a
binary tree of depth four from FixedPValue
hypotheses, all with the same p-value, run
Hierarchy, and read off the level at each depth
as a fraction of \(\alpha\):
>>> from trickle_down import FixedPValue, Hierarchy
>>> def levels_by_depth(p, stake, depth=4, alpha=0.05):
... root = FixedPValue(p=p, stake=stake)
... frontier = [root]
... for _ in range(depth - 1):
... children = []
... for node in frontier:
... for _ in range(2):
... child = FixedPValue(p=p, stake=stake)
... node.add_child(child)
... children.append(child)
... frontier = children
... Hierarchy(fwer=alpha, root=root).test_hypotheses()
... levels, frontier = [], [root]
... while frontier:
... node = frontier[0]
... levels.append(node.stake * node.available_level / alpha)
... frontier = [c for n in frontier for c in n.children]
... return [f"1/{1 / level:g}" for level in levels]
With every hypothesis retained (\(p = 1\)) trickle down matches Hierarchical Bonferroni exactly; with every hypothesis rejected (\(p = 0\)) it tests each depth at a level only half as small as the one above:
>>> levels_by_depth(p=1.0, stake=0.5)
['1/2', '1/8', '1/32', '1/128']
>>> levels_by_depth(p=0.0, stake=0.5)
['1/2', '1/4', '1/8', '1/16']
The Fixed Hierarchy procedure is trickle down with every stake equal to one (§5.2). Following a chain of rejections it tests depth \(k\) at \(2^{1-k} \alpha\):
>>> levels_by_depth(p=0.0, stake=1.0)
['1/1', '1/2', '1/4', '1/8']
These are the four columns of Table 1.
A Worked Example (§7.1)#
The paper splits the data in two, fitting the uplift model and its summary tree on one half and testing on the other, which is the “honest” estimation of [AI16]. The split also licenses choosing each test’s direction on the training half [Cox75].
The paper generates 10,000 individuals with ten features: a continuous outcome predictor \(x_0\), a three-level categorical outcome predictor \(x_1\), binary effect moderators \(x_2\) and \(x_3\), and six standard-normal noise variables. The baseline outcome is \(Y(0) = x_0 + 0.3 x_1 + \varepsilon_0\) with \(\varepsilon_0 \sim N(0, 9)\), and \(Y(1) = Y(0) + \tau(x) + \varepsilon_1\), where the effect depends only on the moderators:
\(x_2\) |
\(x_3\) |
\(\tau\) |
|---|---|---|
0 |
0 |
−1.00 |
0 |
1 |
1.20 |
1 |
0 |
2.00 |
1 |
1 |
1.50 |
The average effect, \(+0.925\), conceals a quarter of the
population the treatment harms; among \(x_2 = 0\) the average is
\(+0.1\), a strong benefit canceling a strong harm.
paper.simulation.worked_example() follows the paper’s recipe. It
fits a T-learner and a depth-three regression tree on the training
half, builds the interweaved hierarchy with \(\alpha = 0.05\),
stake \(1/2\) and \(\delta = 0\), chooses every direction on
the training half, and tests on the evaluation half.
The paper’s fitted tree (its Figure 2) found \(x_2\) at the root and \(x_3\) beneath \(x_2 = 0\), plus spurious splits on noise features:
Node |
τ̂ train |
τ̂ eval |
Effect certified |
|---|---|---|---|
everyone |
+1.09 |
+1.17 |
yes |
x2 = 0 |
+0.31 |
+0.38 |
yes |
x2 = 1 |
+1.83 |
+1.91 |
yes |
x2 = 0, x3 = 0 |
-0.94 |
-0.60 |
yes |
x2 = 0, x3 = 1 |
+1.55 |
+1.37 |
yes |
x2 = 1, x9 <= -0.425 |
+1.12 |
+1.92 |
yes |
x2 = 1, x9 > -0.425 |
+2.18 |
+1.91 |
yes |
x2 = 0, x3 = 0, x7 <= 1.17 |
-0.76 |
-0.47 |
no |
x2 = 0, x3 = 0, x7 > 1.17 |
-2.13 |
-1.40 |
no |
x2 = 0, x3 = 1, x6 <= 1.38 |
+1.33 |
+1.33 |
yes |
x2 = 0, x3 = 1, x6 > 1.38 |
+3.78 |
+1.76 |
no |
x2 = 1, x9 <= -0.425, x5 <= 0.758 |
+0.68 |
+1.84 |
yes |
x2 = 1, x9 <= -0.425, x5 > 0.758 |
+2.53 |
+2.16 |
yes |
x2 = 1, x9 > -0.425, x9 <= -0.151 |
+3.22 |
+1.91 |
yes |
x2 = 1, x9 > -0.425, x9 > -0.151 |
+1.98 |
+1.91 |
yes |
The reconstruction draws a different population, so its spurious splits differ; the structure that matters should not:
Node |
τ̂ train |
τ̂ eval |
True τ |
Effect certified |
|---|---|---|---|---|
everyone |
+1.06 |
+0.90 |
+0.93 |
yes |
x2 = 0 |
+0.22 |
-0.01 |
+0.10 |
no |
x2 = 1 |
+1.81 |
+1.87 |
+1.75 |
yes |
x2 = 0, x3 = 0 |
-0.70 |
-1.11 |
-1.00 |
yes |
x2 = 0, x3 = 1 |
+1.17 |
+1.25 |
+1.20 |
yes |
x2 = 1, x0 <= 1.85 |
+1.73 |
+1.89 |
+1.75 |
yes |
x2 = 1, x0 > 1.85 |
+4.13 |
+1.91 |
+1.75 |
yes |
x2 = 0, x3 = 0, x5 <= 1.42 |
-0.87 |
-1.16 |
-1.00 |
yes |
x2 = 0, x3 = 0, x5 > 1.42 |
+1.52 |
-0.45 |
-1.00 |
no |
x2 = 0, x3 = 1, x5 <= -0.675 |
+0.05 |
+1.99 |
+1.20 |
yes |
x2 = 0, x3 = 1, x5 > -0.675 |
+1.59 |
+0.99 |
+1.20 |
yes |
x2 = 1, x0 <= 1.85, x4 <= -2.02 |
-0.39 |
+0.96 |
+1.75 |
no |
x2 = 1, x0 <= 1.85, x4 > -2.02 |
+1.77 |
+1.91 |
+1.75 |
yes |
Like the paper’s, the reconstructed tree puts \(x_2\) at the root and finds \(x_3\) beneath \(x_2 = 0\) but not beneath \(x_2 = 1\), where the true contrast is only \(-0.5\). It splits the \(x_2 = 1\) branch on \(x_0\), an outcome predictor rather than a moderator, where the paper’s tree split on noise. The training estimates on those spurious branches swing widely (+4.13 for \(x_2 = 1,\ x_0 > 1.85\), against a true +1.75) and the evaluation estimates do not follow them, which is the reason for testing on a held-out half.
The procedure separates the real structure from the spurious. It certifies that the effect is positive overall, that it is larger for \(x_2 = 1\) than \(x_2 = 0\), that within \(x_2 = 0\) it is larger for \(x_3 = 1\), and that the treatment harms \(x_2 = 0,\ x_3 = 0\), while every heterogeneity hypothesis at a spurious split is retained. As in the paper, there are no Type I errors. Unlike the paper, the reconstruction does not certify the sign of the effect in \(x_2 = 0\), where it is a modest +0.1 and this evaluation half estimates \(-0.01\). A retention still says something: the effect there is above \(-0.358\).
The tested hierarchy, with the level of each hypothesis, its p-value, and either what the rejection certifies or the bound of §6 that a retention still supports:
Node |
Hypothesis |
Level |
p |
Decision |
What we may say |
|---|---|---|---|---|---|
everyone |
effect (greater) |
0.0250 |
2.7e-16 |
rejected |
the effect is positive for everyone |
everyone |
heterogeneity (greater) |
0.0250 |
5.2e-18 |
rejected |
the effect is larger for x2 = 1 than for x2 = 0 |
x2 = 0 |
effect (greater) |
0.0125 |
0.53 |
retained |
the effect for x2 = 0 is above -0.358 |
x2 = 0 |
heterogeneity (greater) |
0.0063 |
5.8e-15 |
rejected |
the effect is larger for x2 = 0, x3 = 1 than for x2 = 0, x3 = 0 |
x2 = 0, x3 = 0 |
effect (less) |
0.0031 |
1.5e-07 |
rejected |
the effect is negative for x2 = 0, x3 = 0 |
x2 = 0, x3 = 0 |
heterogeneity (greater) |
0.0031 |
0.23 |
retained |
the effect for x2 = 0, x3 = 0, x5 > 1.42 relative to x2 = 0, x3 = 0, x5 <= 1.42 is above -1.91 |
x2 = 0, x3 = 0, x5 <= 1.42 |
effect (less) |
0.0016 |
1e-07 |
rejected |
the effect is negative for x2 = 0, x3 = 0, x5 <= 1.42 |
x2 = 0, x3 = 0, x5 > 1.42 |
effect (greater) |
0.0016 |
0.69 |
retained |
the effect for x2 = 0, x3 = 0, x5 > 1.42 is above -3.22 |
x2 = 0, x3 = 1 |
effect (greater) |
0.0031 |
3.5e-09 |
rejected |
the effect is positive for x2 = 0, x3 = 1 |
x2 = 0, x3 = 1 |
heterogeneity (greater) |
0.0031 |
0.98 |
retained |
the effect for x2 = 0, x3 = 1, x5 > -0.675 relative to x2 = 0, x3 = 1, x5 <= -0.675 is above -2.37 |
x2 = 0, x3 = 1, x5 <= -0.675 |
effect (greater) |
0.0016 |
3.2e-06 |
rejected |
the effect is positive for x2 = 0, x3 = 1, x5 <= -0.675 |
x2 = 0, x3 = 1, x5 > -0.675 |
effect (greater) |
0.0016 |
3.1e-05 |
rejected |
the effect is positive for x2 = 0, x3 = 1, x5 > -0.675 |
x2 = 1 |
effect (greater) |
0.0125 |
2.7e-33 |
rejected |
the effect is positive for x2 = 1 |
x2 = 1 |
heterogeneity (greater) |
0.0125 |
0.49 |
retained |
the effect for x2 = 1, x0 > 1.85 relative to x2 = 1, x0 <= 1.85 is above -1.56 |
x2 = 1, x0 <= 1.85 |
effect (greater) |
0.0031 |
6.3e-33 |
rejected |
the effect is positive for x2 = 1, x0 <= 1.85 |
x2 = 1, x0 <= 1.85 |
heterogeneity (greater) |
0.0031 |
0.18 |
retained |
the effect for x2 = 1, x0 <= 1.85, x4 > -2.02 relative to x2 = 1, x0 <= 1.85, x4 <= -2.02 is above -2.03 |
x2 = 1, x0 <= 1.85, x4 <= -2.02 |
effect (less) |
0.0016 |
0.82 |
retained |
the effect for x2 = 1, x0 <= 1.85, x4 <= -2.02 is below 4.18 |
x2 = 1, x0 <= 1.85, x4 > -2.02 |
effect (greater) |
0.0016 |
8.2e-33 |
rejected |
the effect is positive for x2 = 1, x0 <= 1.85, x4 > -2.02 |
x2 = 1, x0 > 1.85 |
effect (greater) |
0.0063 |
0.003 |
rejected |
the effect is positive for x2 = 1, x0 > 1.85 |
Familywise Error Rate (§7.2)#
The paper compares five procedures on the same interweaved hierarchy:
Bonferroni: every hypothesis at \(\alpha / m\);
Hierarchical Bonferroni: halving the level at each depth (§5.1);
Fixed hierarchy: trickle down with \(v \equiv 1\) (§5.2);
Trickle down: \(v \equiv 1/2\) (§5.3);
Non-adaptive: trickle down at \(v \equiv 1/2\), but with every hypothesis two-sided rather than directed by the training split.
All but the last test the directions chosen on the training half.
Under the global null, \(\tau \equiv 0\), every hypothesis is true,
so the FWER is the chance of rejecting anything. Each of 4,000
populations of \(n = 4{,}000\) is tested twice: over a partition
fixed in advance (\(x_2\), then \(x_3\)), and over a
depth-three tree distilled from a T-learner on the training half. The
columns therefore share populations, and their difference is paired.
The z columns give the difference between paper and reconstruction
in units of its standard error:
Procedure |
Fixed: paper |
Fixed: ours |
z |
Fitted: paper |
Fitted: ours |
z |
|---|---|---|---|---|---|---|
Bonferroni |
0.0447 (0.0033) |
0.0403 (0.0031) |
-1.0 |
0.0345 (0.0029) |
0.0297 (0.0027) |
-1.2 |
Hierarchical Bonferroni |
0.0447 (0.0033) |
0.0425 (0.0032) |
-0.5 |
0.0457 (0.0033) |
0.0435 (0.0032) |
-0.5 |
Fixed hierarchy |
0.0508 (0.0035) |
0.0480 (0.0034) |
-0.6 |
0.0508 (0.0035) |
0.0480 (0.0034) |
-0.6 |
Trickle down |
0.0447 (0.0033) |
0.0430 (0.0032) |
-0.4 |
0.0457 (0.0033) |
0.0432 (0.0032) |
-0.5 |
Non-adaptive |
0.0460 (0.0033) |
0.0440 (0.0032) |
-0.4 |
0.0435 (0.0032) |
0.0420 (0.0032) |
-0.3 |
Plain Bonferroni’s paired difference, fitted tree minus fixed partition: paper -0.0103 (s.e. 0.0042); reconstruction -0.0105 (s.e. 0.0040).
Trickle down and Hierarchical Bonferroni disagree on whether a familywise error occurred in 2 of 4,000 replications on the fixed partition and 3 on the fitted tree.
Every procedure controls the FWER, on both partitions, and every cell lies within 1.2 standard errors of the paper’s. The reconstruction’s rates sit slightly below the paper’s in every row; the rows share their populations, so one draw moves them together. Fixed hierarchy’s row repeats across the columns, in the paper and here, for a structural reason: under the global null it can reject nothing before the root, so its familywise error is the root’s rejection, and the root is the same in both partitions. Its exact rate is \(\alpha = 0.05\).
Plain Bonferroni is the most conservative, and more so through the fitted tree, whose larger family makes \(\alpha / m\) smaller. Its paired drop matches the paper’s to within a tenth of a standard error.
The paper says trickle down and Hierarchical Bonferroni “agree
exactly” under the global null because at \(v \equiv 1/2\) their
allocations coincide. That holds at every internal node, but
interweaved_hierarchy() gives leaves a stake of
one, which tests each leaf at twice the Hierarchical Bonferroni level.
The two procedures therefore part in a handful of replications, as
counted above. The paper’s own Figure 3 tests leaves at stake one, and
the power tables below match only with leaves at stake one, so the
paper’s identical FWER counts are a near-certainty rather than an
identity.
Statistical Power (§7.3)#
The power study runs on the fixed partition, whose ten hypotheses are all false in both effect configurations: the running example, and the same configuration with the harmed quarter helped instead. Each configuration’s effects are multiplied by a scale from 0.10 to 1.00, and each cell averages the true claims certified over 4,000 populations. A rejection counts as a true claim only if its null is false in the direction tested; a one-sided rejection in the wrong direction is a Type I error. Each cell shows the paper’s mean, then the reconstruction’s:
Procedure |
0.10 |
0.15 |
0.20 |
0.30 |
0.45 |
1.00 |
|---|---|---|---|---|---|---|
Bonferroni |
0.13 / 0.12 |
0.24 / 0.24 |
0.43 / 0.43 |
1.11 / 1.14 |
2.86 / 2.88 |
7.45 / 7.42 |
Hierarchical Bonferroni |
0.12 / 0.11 |
0.23 / 0.22 |
0.39 / 0.39 |
0.96 / 1.00 |
2.36 / 2.39 |
6.68 / 6.67 |
Fixed hierarchy |
0.11 / 0.10 |
0.20 / 0.22 |
0.35 / 0.36 |
0.84 / 0.86 |
1.90 / 1.89 |
3.60 / 3.60 |
Trickle down |
0.15 / 0.13 |
0.28 / 0.28 |
0.49 / 0.48 |
1.25 / 1.31 |
3.15 / 3.16 |
7.58 / 7.56 |
Non-adaptive |
0.12 / 0.13 |
0.22 / 0.21 |
0.37 / 0.35 |
0.93 / 0.97 |
2.53 / 2.55 |
7.36 / 7.32 |
Procedure |
0.10 |
0.15 |
0.20 |
0.30 |
0.45 |
1.00 |
|---|---|---|---|---|---|---|
Bonferroni |
0.13 / 0.12 |
0.24 / 0.23 |
0.44 / 0.43 |
1.13 / 1.11 |
2.69 / 2.72 |
6.42 / 6.44 |
Hierarchical Bonferroni |
0.13 / 0.14 |
0.26 / 0.27 |
0.46 / 0.46 |
1.06 / 1.05 |
2.19 / 2.22 |
5.91 / 5.92 |
Fixed hierarchy |
0.15 / 0.17 |
0.29 / 0.30 |
0.48 / 0.49 |
0.96 / 0.97 |
1.65 / 1.64 |
4.65 / 4.64 |
Trickle down |
0.16 / 0.16 |
0.32 / 0.32 |
0.57 / 0.56 |
1.34 / 1.35 |
2.86 / 2.91 |
6.84 / 6.84 |
Non-adaptive |
0.13 / 0.12 |
0.25 / 0.25 |
0.42 / 0.43 |
1.05 / 1.04 |
2.42 / 2.47 |
6.45 / 6.46 |
The reconstruction reproduces the paper’s ranking at every scale. Trickle down certifies the most true claims at small and moderate effects. Fixed hierarchy falls furthest behind as effects grow, because one retention near the root ends its testing. Directing each test with the training split beats testing both sides, even at the smallest effects, where the training split often picks the wrong side.
59 of 60 cells of Table 3 lie within two standard errors of the paper’s value. The paper rounds its standard errors to 0.01, so these z-scores are approximate.
Configuration |
Procedure |
Scale |
Paper |
Ours |
Gap (s.e.) |
|---|---|---|---|---|---|
The running example: one quarter harmed |
Non-adaptive |
1.00 |
7.36 |
7.32 |
2.4 |
The running example: one quarter harmed |
Bonferroni |
1.00 |
7.45 |
7.42 |
2.0 |
The running example: one quarter harmed |
Trickle down |
0.30 |
1.25 |
1.31 |
1.8 |
Every subgroup helped |
Non-adaptive |
0.45 |
2.42 |
2.47 |
1.7 |
The running example: one quarter harmed |
Trickle down |
0.10 |
0.15 |
0.13 |
1.6 |
The running example: one quarter harmed |
Hierarchical Bonferroni |
0.30 |
0.96 |
1.00 |
1.5 |
Every subgroup helped |
Trickle down |
0.45 |
2.86 |
2.91 |
1.5 |
Every subgroup helped |
Fixed hierarchy |
0.10 |
0.15 |
0.17 |
1.4 |
Table 4 of the paper splits the 0.45 column by the depth of the claim. It shows where each procedure spends its power: fixed hierarchy is strongest at the root and collapses below it, while trickle down degrades gracefully.
Procedure |
Root (of 2) |
Depth 1 (of 4) |
Depth 2 (of 4) |
|---|---|---|---|
Bonferroni |
0.77 / 0.78 |
1.06 / 1.06 |
1.02 / 1.05 |
Hierarchical Bonferroni |
1.13 / 1.14 |
0.88 / 0.87 |
0.36 / 0.39 |
Fixed hierarchy |
1.30 / 1.31 |
0.56 / 0.56 |
0.03 / 0.03 |
Trickle down |
1.20 / 1.21 |
1.04 / 1.03 |
0.90 / 0.92 |
Non-adaptive |
0.99 / 0.99 |
0.86 / 0.86 |
0.68 / 0.69 |
Procedure |
Root (of 2) |
Depth 1 (of 4) |
Depth 2 (of 4) |
|---|---|---|---|
Bonferroni |
0.86 / 0.85 |
0.90 / 0.92 |
0.93 / 0.95 |
Hierarchical Bonferroni |
1.07 / 1.07 |
0.78 / 0.81 |
0.34 / 0.35 |
Fixed hierarchy |
1.28 / 1.27 |
0.36 / 0.36 |
0.02 / 0.02 |
Trickle down |
1.14 / 1.14 |
0.94 / 0.95 |
0.79 / 0.81 |
Non-adaptive |
1.02 / 1.02 |
0.79 / 0.82 |
0.61 / 0.63 |
Every entry agrees with the paper’s to within 0.03. Fixed hierarchy leads at the root and certifies almost nothing at depth two; trickle down gives up a little at the root and keeps most of its power at the leaves. These two tables pin down the reconstruction most tightly: the depth-two column moves sharply with the leaves’ stake, and the fixed-hierarchy rows with the effect in the helped quarter.
Assumptions#
Each entry below fills a gap in the paper’s description of §7. The
module paper.simulation records the same list in
ASSUMPTIONS, so code and documentation cannot disagree.
- Outcome predictors
x0 ~ N(0, 1); x1 uniform on {0, 1, 2}. The paper says only ‘continuous’ and ‘three-level categorical’; these follow the author’s earlier simulation script.
- Effect moderators
x2 and x3 independent fair coins, which gives the paper’s ‘split the population evenly across the four moderator levels’.
- Effect noise
epsilon_1 ~ N(0, 9), independent of epsilon_0: the paper states epsilon_0 ~ N(0, 9) and then only that epsilon_1 is independent of it, which we read as the same distribution. This choice was made after a pilot: N(0, 1) gave every procedure more power than the paper reports, uniformly, by about the factor the extra variance of N(0, 9) removes. It is the one assumption calibrated against the paper’s results.
- Treatment assignment
Independent Bernoulli(1/2) per individual (‘with equal probability’).
- Training and evaluation halves
A uniformly random split into two halves of equal size.
- T-learner
scikit-learn HistGradientBoostingRegressor(max_iter=200, max_depth=4, learning_rate=0.1) on each arm. The paper names no learner; the author’s earlier script used XGBoost with the same settings, which needs OpenMP and is slower to run 4,000 times.
- Summary tree
scikit-learn DecisionTreeRegressor(max_depth=3, min_samples_leaf=0.01): ‘a hundredth of the sample’ read as 1% of the training half the tree is fit on.
- Node estimates
tau_train is the difference of arm means in the node on the training half, which is what chooses each direction.
- Fixed partition
The true moderator structure: split on x2, then on x3 within each half, giving 7 nodes and 10 hypotheses. This matches the paper’s ‘ten true claims available’ and Table 4’s 2 / 4 / 4.
- Global null
tau = 0 for everyone, so every hypothesis in every hierarchy is true and the FWER is the chance of rejecting anything.
- Effect scale
Every subgroup’s tau multiplied by the scale; 1.00 is the unscaled configuration.
- Every subgroup helped
The harmed quarter’s tau of -1.00 replaced by +0.30, the value the author’s earlier simulation script used for that cell. Chosen after a pilot of 1,000 replications per cell, scored against all 30 cells of the paper’s lower Table 3: +0.30 fit best (mean squared z of 1.7) among 0.00, 0.30, 0.50 and 1.00 (2.7, 3.6 and 31.2). +1.00, the first value we tried, left the x3 split within x2 = 0 with a contrast of only 0.2, and fixed hierarchy far below the paper.
- True claims
A rejection counts as a true claim only if its null is false under the direction actually tested. A one-sided rejection in the wrong direction is a Type I error, not a claim. A two-sided rejection of a false null counts, as the paper says Table 3 ‘credits both procedures for reaching a node’.
- Splits too small to test
If either arm of a node holds fewer than two evaluation observations, the split above it is dropped and its parent becomes a leaf. With leaves of 1% this almost never happens.
Running the Reconstruction#
The reconstruction lives in paper/ in the repository and uses only
the public API of trickle-down. Its results are committed, so building
these docs never reruns it. To regenerate them:
uv sync --extra dev
uv run python -m paper.reproduce
The full run takes about five minutes on ten cores. Every replicate
draws from its own child of one seed, so the results do not depend on
the number of workers; --replications runs a smaller study, and
--render-only rebuilds these tables from paper/results.json.