API Reference

Testing Statistical Hypotheses in a Tree-Like Hierarchy.

Hypotheses

Abstract base class for representing a hypothesis.

trickle_down.hypothesis.select_alternative(tau_train: float, delta: float = 0.0) → Literal['two-sided', 'greater', 'less']

Choose a one-sided alternative from an independent training split.

Parameters:

• tau_train (float) – Estimated effect in this node, computed on the training split.

• delta (float) – Threshold defining the hypothesis. Zero gives an ordinary test of benefit or harm; a positive value a superiority test against a break-even point; a negative value a non-inferiority test.

Returns:

“greater” when the training estimate exceeds delta, so that rejecting certifies a benefit beyond delta; “less” otherwise, so that rejecting certifies harm.

Return type:

Alternative

Notes

The choice is legitimate because the training and evaluation splits are independent. Conditional on the training split the selected hypotheses are fixed, so a procedure controlling the FWER over a fixed family controls it here too. Choosing the wrong direction costs power; it does not inflate the error rate.

References

Cox, D. R. (1975). A note on data-splitting for the evaluation of significance levels. Biometrika, 62(2), 441-444.

trickle_down.hypothesis.contrast_moments(means: Sequence[float], variances: Sequence[float], sizes: Sequence[int], coefficients: Sequence[float]) → tuple[float, float, float]

Estimate, standard error and degrees of freedom for a contrast.

For independent groups with means mu_i, variances sigma_i^2 and sizes n_i, and contrast coefficients c_i, this estimates

theta = sum_i c_i mu_i

with standard error sqrt{ sum_i c_i^2 sigma_i^2 / n_i } and the Welch-Satterthwaite degrees of freedom

nu = ( sum_i u_i )^2 / sum_i u_i^2 / (n_i - 1),

where u_i = c_i^2 sigma_i^2 / n_i.

Parameters:

• means (Sequence) – Per-group sample moments. Variances are the usual unbiased estimates (ddof=1).

• variances (Sequence) – Per-group sample moments. Variances are the usual unbiased estimates (ddof=1).

• sizes (Sequence) – Per-group sample moments. Variances are the usual unbiased estimates (ddof=1).

• coefficients (Sequence[float]) – Contrast coefficients. Use (1, -1) for a difference of means and (1, -1, -1, 1) for a difference in differences.

Returns:

Estimate, standard error, and degrees of freedom.

Return type:

tuple[float, float, float]

Notes

Groups with a zero coefficient drop out of every term, including the degrees of freedom, so they may be passed harmlessly.

trickle_down.hypothesis.t_pvalue(t: float, df: float, alternative: Literal['two-sided', 'greater', 'less']) → float

Convert a t statistic to a p-value under the given alternative.

trickle_down.hypothesis.invert_test(estimate: float, standard_error: float, df: float, level: float, alternative: Literal['two-sided', 'greater', 'less']) → tuple[float, float]

Invert a t-test to get the confidence set it implies.

A test of H_0: tau <= delta at level level rejects exactly those thresholds delta lying below

L = tau_hat - t_{1 - level, df} * SE,

so the set of thresholds the data can rule out is an interval, and reporting its endpoint says strictly more than reporting the verdict at one threshold. When the test rejects at delta, the endpoint is a sharper claim than delta. When it retains, the endpoint still bounds how far the other way the truth can be — which is the difference between “we cannot show the treatment helps” and “we cannot show the treatment helps, and it costs at most this much”.

Returns:

Lower and upper endpoints, with an infinite endpoint on the side a one-sided test says nothing about.

Return type:

tuple[float, float]

class trickle_down.hypothesis.Hypothesis(weight: float = 1.0, stake: float = 1.0, name: str | None = None)

Abstract base class for representing a hypothesis.

add_child(child: Hypothesis) → None

Add child hypothesis.

set_available_level(available_level: float) → None

Set available level.

test_hypothesis() → None

Reject when the p-value clears this node’s share of the level.

Subclasses supply the p-value, not the decision rule: the comparison against stake * available_level is what makes the node part of the procedure, and is the same for every test.

abstractmethod pvalue() → float

Calculate p-value.

confidence_interval(level: float) → tuple[float, float]

Confidence set obtained by inverting this test at level.

Not every hypothesis admits one. A p-value computed elsewhere carries no estimate or standard error, so there is nothing to invert.

class trickle_down.hypothesis.FixedPValue(p: float, weight: float = 1.0, stake: float = 1.0, name: str | None = None)

A hypothesis whose p-value was computed elsewhere.

Useful when the test is one this library does not implement — a randomization test, a bootstrap, a model-based Wald test — and for exercising a hierarchy against prescribed p-values in simulation.

pvalue() → float

Return the supplied p-value.

class trickle_down.hypothesis.TTest(mean1: float, mean2: float, var1: float, var2: float, n1: int, n2: int, weight: float = 1.0, stake: float = 1.0, name: str | None = None, delta: float = 0.0, alternative: Literal['two-sided', 'greater', 'less'] = 'two-sided')

Test a hypothesis about a difference of means using Welch’s t-test.

With delta and alternative left at their defaults this is the familiar two-sided test of equal means. Setting them tests

H_0: mu_1 - mu_2 <= delta (alternative=”greater”)

or the reverse inequality (alternative=”less”), which is what the effect hypotheses attached to the nodes of an uplift tree need: rejecting proves the treatment effect clears, or falls short of, a threshold chosen for its business meaning.

Parameters:

• mean1 (float) – Sample means. The contrast estimated is mean1 - mean2, so for a treatment effect pass the treated mean first.

• mean2 (float) – Sample means. The contrast estimated is mean1 - mean2, so for a treatment effect pass the treated mean first.

• var1 (float) – Unbiased sample variances (ddof=1).

• var2 (float) – Unbiased sample variances (ddof=1).

• n1 (int) – Sample sizes.

• n2 (int) – Sample sizes.

• delta (float) – Threshold the difference is compared against. Zero recovers a test of no effect.

• alternative (Alternative) – Which departures count as evidence. See select_alternative for choosing this from an independent training split.

pvalue() → float

Calculate p-value.

Uses Welch’s t-test. The two-sided p-value is computed by taking both one-sided p-values, doubling the smaller, and capping at 1.

confidence_interval(level: float) → tuple[float, float]

The confidence set this test’s inversion yields at level.

class trickle_down.hypothesis.ContrastTest(means: Sequence[float], variances: Sequence[float], sizes: Sequence[int], coefficients: Sequence[float], weight: float = 1.0, stake: float = 1.0, name: str | None = None, delta: float = 0.0, alternative: Literal['two-sided', 'greater', 'less'] = 'two-sided')

Test a linear contrast across independent groups.

Generalizes TTest to any number of groups and any contrast. The case this library was written for is the difference in treatment effects across the two children of a tree node,

(mu_{R1} - mu_{R0}) - (mu_{L1} - mu_{L0}),

obtained with coefficients (1, -1, -1, 1) over the four arm-by-subgroup cells ordered R1, R0, L1, L0. Rejecting the two-sided null proves the treatment effect is heterogeneous across that split.

Parameters:

• means (Sequence) – Per-cell sample moments. Variances are unbiased (ddof=1).

• variances (Sequence) – Per-cell sample moments. Variances are unbiased (ddof=1).

• sizes (Sequence) – Per-cell sample moments. Variances are unbiased (ddof=1).

• coefficients (Sequence[float]) – Contrast coefficients, aligned with the moments.

• delta (float) – Threshold the contrast is compared against.

• alternative (Alternative) – Which departures count as evidence.

classmethod difference_in_differences(treated_right: tuple[float, float, int], control_right: tuple[float, float, int], treated_left: tuple[float, float, int], control_left: tuple[float, float, int], weight: float = 1.0, stake: float = 1.0, name: str | None = None, delta: float = 0.0, alternative: Literal['two-sided', 'greater', 'less'] = 'two-sided') → ContrastTest

Build the heterogeneity test for one split of an uplift tree.

Each argument is a (mean, variance, size) triple for one arm-by-subgroup cell. The contrast estimated is tau_right - tau_left.

pvalue() → float

Calculate p-value.

confidence_interval(level: float) → tuple[float, float]

The confidence set this test’s inversion yields at level.

class trickle_down.hypothesis.FTest(*samples: NDArray[float64], weight: float = 1.0, stake: float = 1.0, name: str | None = None, **kwargs: int)

One-way ANOVA for equality of means across two or more groups.

Useful as the parent of the pairwise comparisons in a hierarchy: there is often power to show that the groups differ without power to say which is best, and testing the omnibus hypothesis first spends nothing on the pairwise tests until that much is established.

pvalue() → float

Calculate p-value.

Hierarchy

Classes for representing hypothesis hierarchies.

class trickle_down.hierarchy.Hierarchy(fwer: float, root: Hypothesis)

Statistical hypothesis hierarchy.

test_hypotheses() → None

Test every hypothesis at the level the procedure assigns it.

Safe to call again, for instance after changing fwer: every node is re-tested, so verdicts always agree with the levels recorded alongside them. P-values are cached on each hypothesis, so a re-run costs only the comparisons.

hypotheses() → list[Hypothesis]

Return every hypothesis in the tree, parents before children.

visualize_hierarchy() → Figure

Draw the tree, coloring rejected hypotheses blue.

Uplift Trees

The hypothesis architecture for validating an uplift model.

An uplift model is distilled into a shallow decision tree, and each node of that tree carries two kinds of claim we might want to certify:

• an effect claim about the average treatment effect in the node’s subpopulation, tau_S, against a threshold delta_S chosen for its business meaning; and

• a heterogeneity claim that tau differs between the node’s two children, which is what it means for the split to identify real heterogeneity.

These interweave into a single tree of hypotheses. Writing E_S for the effect hypothesis at node S and H_S for the heterogeneity hypothesis at its split, as the paper does (Wilson 2026, section 3),

E_root -> H_root -> { E_L , E_R } -> …

Each effect hypothesis has exactly one child, the heterogeneity hypothesis at the same node; each heterogeneity hypothesis has the effect hypotheses of the two child nodes as its children. Leaves of the decision tree carry an effect hypothesis and nothing below it.

The payoff is a set of claims a decision-maker can act on. If the first split is gender, the procedure can certify simultaneously that the effect is larger for men than for women, that it is positive for men, and that it is negative for women — three sentences, all at the stated familywise error rate.

class trickle_down.uplift.ArmSummary(mean: float, variance: float, size: int)

Sample moments for one treatment arm within one subpopulation.

as_cell() → tuple[float, float, int]

Return the (mean, variance, size) triple the tests expect.

classmethod from_samples(y: ArrayLike) → ArmSummary

Summarize one arm’s observations.

The variance is the unbiased sample variance, ddof=1. np.var defaults to ddof=0, which understates it by a factor of (n-1)/n; a Welch test fed that understated variance is anticonservative, and the error is invisible at a glance. That is the whole reason this exists rather than leaving every caller to write the three lines itself.

Non-finite values are refused for the same reason. A single NaN turns the variance into NaN, which propagates through the Welch statistic into a NaN p-value that compares false against every level, so the node silently retains. Dropping the offending observations instead would be worse: the caller would never learn that the slice it handed over was not the slice it was summarizing.

class trickle_down.uplift.UpliftNode(name: str, treated: ArmSummary, control: ArmSummary, tau_train: float, children: list[UpliftNode] = <factory>)

A node of the fitted uplift tree, with what is needed to test it.

Parameters:

• name (str) – Human-readable label, e.g. “men” or “x0 < 0.5”.

• treated (ArmSummary) – Moments computed on the evaluation split. These are what the hypotheses are tested against.

• control (ArmSummary) – Moments computed on the evaluation split. These are what the hypotheses are tested against.

• tau_train (float) – The effect estimated on the training split. Used only to choose the direction of the one-sided effect test, never to compute a p-value.

• children (list[UpliftNode]) – Either empty, or exactly two.

property tau_hat: float

Evaluation-split estimate of the average treatment effect.

property size: int

Total evaluation-split sample size in this subpopulation.

walk() → list[UpliftNode]

Every node at or below this one, parents before children.

class trickle_down.uplift.Claim(node: str, kind: str, hypothesis: Hypothesis, delta: float, alternative: Literal['two-sided', 'greater', 'less'], depth: int, left: str | None = None, right: str | None = None)

One hypothesis in the architecture, and what rejecting it proves.

property rejected: bool

Whether the procedure rejected this hypothesis.

property pvalue: float

The p-value, computed on the evaluation split.

property level: float

The level this hypothesis was actually tested at.

property interval: tuple[float, float]

Confidence set for this claim’s estimand, at its own level.

The level a hypothesis is tested at is the level its inversion is valid at, so these bounds are simultaneous with every other claim the procedure makes, and with each other. They are not the marginal intervals one would report for a single node, and for a deep node they are correspondingly wide.

A claim tested at level zero has the whole line for its confidence set. That is not a degenerate case to guard against but the honest answer: a test that can never reject excludes nothing, so its inversion excludes nothing either.

property reportable_bound: tuple[float, float]

The part of interval the procedure is entitled to claim.

Inverting the test gives an endpoint that may lie on either side of the node’s threshold, but only the weaker side is free. A node that rejected has passed its level down to its children, so a threshold past the one it cleared is a fresh claim it has nothing left to pay for; truncating at delta is what keeps the whole collection of endpoints simultaneously valid. A node that retained kept its level, and its endpoint already lies on the weaker side.

Nothing is free for a two-sided test: its enlarged family is not nested, so the node holds a true member whatever the effect is, and rejecting never releases the level. Such a claim reports no bound, and so does a claim tested at level zero, whose interval is already the whole line.

bound_sentence() → str

The quantitative claim, which survives a retention.

sentence says what rejecting proves and is silent when the hypothesis is retained. This says what the data support either way, within what the procedure can pay for: at a retention it bounds how far the other way the truth can lie, and at a rejection it restates the threshold already cleared.

A node handed no level has nothing to say. It is reported as such rather than omitted, so that the absence is visible in the output instead of being mistaken for a node the procedure never reached.

sentence() → str

What rejecting this hypothesis certifies, in words.

class trickle_down.uplift.UpliftClaims(hierarchy: Hierarchy, claims: list[Claim])

The hypotheses attached to an uplift tree, and their verdicts.

Holds the hierarchy so it can be tested, and knows how to report what was certified once it has been.

test() → UpliftClaims

Run the procedure. Returns self, so calls can be chained.

certified() → list[Claim]

The claims the procedure was able to certify.

trickle_down.uplift.interweaved_hierarchy(root: UpliftNode, alpha: float, delta: float = 0.0, effect_stake: float = 0.5, heterogeneity_stake: float = 0.5, adaptive: bool = True, size_weighted: bool = False) → UpliftClaims

Build the interweaved hierarchy over a fitted uplift tree.

Parameters:

• root (UpliftNode) – Root of the fitted tree.

• alpha (float) – Familywise error rate.

• delta (float) – Threshold for the effect hypotheses. Zero tests for benefit or harm; a positive value a break-even point an expensive treatment must clear; a negative value a non-inferiority margin.

• effect_stake (float) – Stakes for the two kinds of hypothesis. A stake below one keeps part of the level in reserve, so testing continues past a retention at reduced power.

• heterogeneity_stake (float) – Stakes for the two kinds of hypothesis. A stake below one keeps part of the level in reserve, so testing continues past a retention at reduced power.

• adaptive (bool) – Choose the direction of every test from the training split. For an effect hypothesis that is the sign of the node’s effect; for a heterogeneity hypothesis it is the sign of tau_right - tau_left, so rejecting says which child responds more rather than merely that they differ. When False, every hypothesis of both kinds is two-sided — the natural comparator. Two things separate them: the factor of two in level that committing to a side in advance buys, and the strength of the claim earned. A two-sided rejection certifies that the effect differs from the threshold, not which way it differs; going on to name the direction is a Type III error argument this library does not make.

• size_weighted (bool) – Split a node’s level between its children in proportion to their subpopulation sizes rather than equally. The procedure controls the familywise error rate under any weights, so this is a preference and not a correctness matter: equal weights treat a small subpopulation as worth as much attention as a large one, which is what finding a carve-out requires, while size weighting buys power where the most people are. A node’s level is recycled only to its own descendants, never across to a sibling, so the split between siblings is zero-sum and the default is the neutral one.

Returns:

The hierarchy and the claims attached to it, untested.

Return type:

UpliftClaims

Notes

Hypotheses with no children are given a stake of one. There is nothing below them to hold level in reserve for, and spending it keeps the procedure closer to exhausting alpha — a procedure that leaves level unspent is uniformly improvable.

Choosing the direction on the training split is legitimate because the splits are independent: conditional on the training split the family of hypotheses is fixed. A wrong guess costs power, not validity.

A one-sided heterogeneity test is not merely more powerful than the two-sided one; it answers a different and more useful question. Rejecting a two-sided test licenses “the effect differs across this split”, which leaves the reader to work out which way from an uncertified point estimate. Rejecting the one-sided test licenses “the effect is larger in this child than in that one”, which is a sentence someone can act on.

trickle_down.uplift.bonferroni_rejections(claims: list[Claim], alpha: float) → list[bool]

Test every hypothesis at alpha / m, ignoring the tree.

The comparator the paper argues against: valid, but it pays the full multiplicity price at every node however deep, and cannot borrow level from a rejection higher up.

trickle_down.uplift.hierarchical_bonferroni_rejections(claims: list[Claim], alpha: float) → list[bool]

The Hierarchical Bonferroni Procedure of the paper’s section 5.1.

Hypotheses at depth k of the hypothesis tree are tested at level 2^{-k} alpha / n_k, where n_k counts the hypotheses at that depth. The levels sum to at most alpha, so the FWER is controlled, but power falls off exponentially with depth. The depth is depth in the hypothesis tree, counted from one, which interleaves the two kinds of claim: a node at decision-tree depth d contributes its effect hypothesis at 2d + 1 and its heterogeneity hypothesis at 2d + 2. That is not Claim.depth, which is the decision-tree depth. Substituting one for the other leaves a procedure that still looks valid and spends quite different levels.

Graphical Approach

Reference implementation of the graphical approach to multiple testing.

The trickle down procedure is a special case of the graphical approach of Bretz, Maurer, Brannath and Posch (2009). This module implements that approach directly, and provides the encoding that turns a trickle down hierarchy into an equivalent graph, so the claim can be checked empirically rather than only argued on paper.

References

Bretz, F., Maurer, W., Brannath, W., and Posch, M. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine, 28(4), 586-604.

class trickle_down.graphical.GraphicalProcedure(allocation: ArrayLike, transitions: ArrayLike)

A sequentially rejective procedure defined by weights and a graph.

A procedure is specified by an initial allocation of the overall level, a = (a_1, …, a_m) with sum_i a_i <= alpha, and a transition matrix G = (g_ij) satisfying g_ii = 0 and sum_j g_ij <= 1. Testing H_i at level a_i and, on rejecting H_j, passing a fraction g_jl of a_j to each surviving H_l controls the FWER strongly at alpha.

Parameters:

• allocation (npt.ArrayLike) – Initial levels a_i, in absolute terms rather than as fractions of alpha. Must sum to at most the intended alpha. Array-like rather than a sequence because trickle_down_graph, the main producer of this argument, hands back an ndarray.

• transitions (npt.ArrayLike) – The m-by-m matrix G.

Notes

Algorithm 1 of Bretz et al. (2009): repeatedly find a hypothesis whose p-value is at or below its current level, reject it, add a_j * g_jl to every surviving level, and update the graph by

g_lk <- (g_lk + g_lj g_jk) / (1 - g_lj g_jl).

Their remark (iii) notes the final rejection set does not depend on which rejectable hypothesis is taken first. selection exposes that choice so the invariance can be tested.

property alpha: float

Total level allocated, which may be less than the nominal alpha.

reject(pvalues: Sequence[float], selection: str = 'argmin', rng: Generator | None = None) → NDArray[bool]

Return the rejection indicator for each hypothesis.

Parameters:

• pvalues (Sequence[float]) – One p-value per hypothesis, in the same order as the allocation.

• selection (str) – Which rejectable hypothesis to take at each step: “argmin” for the smallest p_i / a_i as in Bretz et al., “first” for the lowest index, or “random”. All three give the same answer; the choice exists to test that.

• rng (np.random.Generator | None) – Source of randomness when selection is “random”.

trickle_down.graphical.trickle_down_graph(root: Hypothesis, alpha: float) → tuple[list[Hypothesis], NDArray[float64], NDArray[float64]]

Encode a trickle down hierarchy as an equivalent graph.

Let w_i be node i’s share of its parent’s level, v_i its stake, W_i the product of weights along the path from the root to i, and anc(i) the strict ancestors of i. The initial allocation is

a_i = alpha * v_i * W_i * prod_{k in anc(i)} (1 - v_k),

and there is an edge from every node k to every strict descendant i with weight

g_{ki} = v_i * W_{k to i} * prod_u (1 - v_u),

the last product running over the nodes strictly between k and i.

Parameters:

• root (Hypothesis) – Root of the hierarchy. Child weights are normalized in place, as Hierarchy.test_hypotheses also does.

• alpha (float) – Familywise error rate.

Returns:

The hypotheses in breadth-first order, the allocation, and the transition matrix, aligned with one another.

Return type:

tuple

Notes

Edges run to every strict descendant, not merely to the children. A node’s level under the trickle down procedure is a product over all of its ancestors’ outcomes, and with parent-only edges just one of those factors would ever fire.

The allocation sums to alpha only when every root-to-leaf path ends at a node with stake 1. Otherwise the procedure leaves some of the level unspent, and can be uniformly improved by adding edges out of the leaves.