Reproducing the Paper ===================== "Interpretable and Validatable Uplift Modeling" [Wil26]_ makes two kinds of quantitative claim. Some are exact: the levels at which each procedure tests a hypothesis follow from the definitions, and this guide recomputes them with trickle-down as doctests, executed each time the documentation is built. The rest come from the simulation study of §7, which we reproduce from the paper's description and report beside the paper's own numbers. The second kind needs a caveat up front. The paper's tables were generated by a script that is not public, and its text leaves several details unstated: the distribution of each feature, the learner, and the effect assigned to one subgroup, among others. We filled every gap with an assumption, listed in :ref:`assumptions`. Two of them were calibrated against the paper's results, and we say which. A faithful reconstruction should therefore agree with the paper to within Monte Carlo error, not digit for digit, and the tables below show how close it comes. .. include:: _generated/meta.rst Levels by Depth (§5, Table 1) ----------------------------- The procedures of §5 descend from sequential tests that "pay only for failures": the fixed sequence procedure [Mau95]_ and the fallback procedure [Wie03]_. The Fixed Hierarchy procedure is the hierarchical test of [Mei08]_ with arbitrary sibling weights, and trickle down is an instance of the graphical approach of [BMBP09]_, developed in parallel by [BSG09]_. Table 1 of the paper compares the level at which each procedure tests a hypothesis at a given depth of a binary tree with equal weights and, for trickle down, a stake of :math:`1/2` at every node. Hierarchical Bonferroni tests depth :math:`k` at :math:`2^{-k} \alpha / n_k` with :math:`n_k = 2^{k-1}`, which does not depend on the data: .. doctest:: >>> hierarchical_bonferroni = [2.0**-k / 2 ** (k - 1) for k in range(1, 5)] >>> [f"1/{1 / level:g}" for level in hierarchical_bonferroni] ['1/2', '1/8', '1/32', '1/128'] Trickle down's levels depend on what was rejected above. We build a binary tree of depth four from :class:`~trickle_down.FixedPValue` hypotheses, all with the same p-value, run :class:`~trickle_down.Hierarchy`, and read off the level at each depth as a fraction of :math:`\alpha`: .. doctest:: >>> from trickle_down import FixedPValue, Hierarchy >>> def levels_by_depth(p, stake, depth=4, alpha=0.05): ... root = FixedPValue(p=p, stake=stake) ... frontier = [root] ... for _ in range(depth - 1): ... children = [] ... for node in frontier: ... for _ in range(2): ... child = FixedPValue(p=p, stake=stake) ... node.add_child(child) ... children.append(child) ... frontier = children ... Hierarchy(fwer=alpha, root=root).test_hypotheses() ... levels, frontier = [], [root] ... while frontier: ... node = frontier[0] ... levels.append(node.stake * node.available_level / alpha) ... frontier = [c for n in frontier for c in n.children] ... return [f"1/{1 / level:g}" for level in levels] With every hypothesis retained (:math:`p = 1`) trickle down matches Hierarchical Bonferroni exactly; with every hypothesis rejected (:math:`p = 0`) it tests each depth at a level only half as small as the one above: .. doctest:: >>> levels_by_depth(p=1.0, stake=0.5) ['1/2', '1/8', '1/32', '1/128'] >>> levels_by_depth(p=0.0, stake=0.5) ['1/2', '1/4', '1/8', '1/16'] The Fixed Hierarchy procedure is trickle down with every stake equal to one (§5.2). Following a chain of rejections it tests depth :math:`k` at :math:`2^{1-k} \alpha`: .. doctest:: >>> levels_by_depth(p=0.0, stake=1.0) ['1/1', '1/2', '1/4', '1/8'] These are the four columns of Table 1. A Worked Example (§7.1) ----------------------- The paper splits the data in two, fitting the uplift model and its summary tree on one half and testing on the other, which is the "honest" estimation of [AI16]_. The split also licenses choosing each test's direction on the training half [Cox75]_. The paper generates 10,000 individuals with ten features: a continuous outcome predictor :math:`x_0`, a three-level categorical outcome predictor :math:`x_1`, binary effect moderators :math:`x_2` and :math:`x_3`, and six standard-normal noise variables. The baseline outcome is :math:`Y(0) = x_0 + 0.3 x_1 + \varepsilon_0` with :math:`\varepsilon_0 \sim N(0, 9)`, and :math:`Y(1) = Y(0) + \tau(x) + \varepsilon_1`, where the effect depends only on the moderators: .. list-table:: :header-rows: 1 :widths: 10 10 10 * - :math:`x_2` - :math:`x_3` - :math:`\tau` * - 0 - 0 - −1.00 * - 0 - 1 - 1.20 * - 1 - 0 - 2.00 * - 1 - 1 - 1.50 The average effect, :math:`+0.925`, conceals a quarter of the population the treatment harms; among :math:`x_2 = 0` the average is :math:`+0.1`, a strong benefit canceling a strong harm. :func:`paper.simulation.worked_example` follows the paper's recipe. It fits a T-learner and a depth-three regression tree on the training half, builds the interweaved hierarchy with :math:`\alpha = 0.05`, stake :math:`1/2` and :math:`\delta = 0`, chooses every direction on the training half, and tests on the evaluation half. The paper's fitted tree (its Figure 2) found :math:`x_2` at the root and :math:`x_3` beneath :math:`x_2 = 0`, plus spurious splits on noise features: .. include:: _generated/paper_tree.rst The reconstruction draws a different population, so its spurious splits differ; the structure that matters should not: .. include:: _generated/example_tree.rst Like the paper's, the reconstructed tree puts :math:`x_2` at the root and finds :math:`x_3` beneath :math:`x_2 = 0` but not beneath :math:`x_2 = 1`, where the true contrast is only :math:`-0.5`. It splits the :math:`x_2 = 1` branch on :math:`x_0`, an outcome predictor rather than a moderator, where the paper's tree split on noise. The training estimates on those spurious branches swing widely (+4.13 for :math:`x_2 = 1,\ x_0 > 1.85`, against a true +1.75) and the evaluation estimates do not follow them, which is the reason for testing on a held-out half. The procedure separates the real structure from the spurious. It certifies that the effect is positive overall, that it is larger for :math:`x_2 = 1` than :math:`x_2 = 0`, that within :math:`x_2 = 0` it is larger for :math:`x_3 = 1`, and that the treatment *harms* :math:`x_2 = 0,\ x_3 = 0`, while every heterogeneity hypothesis at a spurious split is retained. As in the paper, there are no Type I errors. Unlike the paper, the reconstruction does not certify the sign of the effect in :math:`x_2 = 0`, where it is a modest +0.1 and this evaluation half estimates :math:`-0.01`. A retention still says something: the effect there is above :math:`-0.358`. The tested hierarchy, with the level of each hypothesis, its p-value, and either what the rejection certifies or the bound of §6 that a retention still supports: .. include:: _generated/example_hierarchy.rst Familywise Error Rate (§7.2) ---------------------------- The paper compares five procedures on the same interweaved hierarchy: - **Bonferroni**: every hypothesis at :math:`\alpha / m`; - **Hierarchical Bonferroni**: halving the level at each depth (§5.1); - **Fixed hierarchy**: trickle down with :math:`v \equiv 1` (§5.2); - **Trickle down**: :math:`v \equiv 1/2` (§5.3); - **Non-adaptive**: trickle down at :math:`v \equiv 1/2`, but with every hypothesis two-sided rather than directed by the training split. All but the last test the directions chosen on the training half. Under the global null, :math:`\tau \equiv 0`, every hypothesis is true, so the FWER is the chance of rejecting anything. Each of 4,000 populations of :math:`n = 4{,}000` is tested twice: over a partition fixed in advance (:math:`x_2`, then :math:`x_3`), and over a depth-three tree distilled from a T-learner on the training half. The columns therefore share populations, and their difference is paired. The ``z`` columns give the difference between paper and reconstruction in units of its standard error: .. include:: _generated/fwer.rst Every procedure controls the FWER, on both partitions, and every cell lies within 1.2 standard errors of the paper's. The reconstruction's rates sit slightly below the paper's in every row; the rows share their populations, so one draw moves them together. Fixed hierarchy's row repeats across the columns, in the paper and here, for a structural reason: under the global null it can reject nothing before the root, so its familywise error is the root's rejection, and the root is the same in both partitions. Its exact rate is :math:`\alpha = 0.05`. Plain Bonferroni is the most conservative, and more so through the fitted tree, whose larger family makes :math:`\alpha / m` smaller. Its paired drop matches the paper's to within a tenth of a standard error. The paper says trickle down and Hierarchical Bonferroni "agree exactly" under the global null because at :math:`v \equiv 1/2` their allocations coincide. That holds at every internal node, but :func:`~trickle_down.interweaved_hierarchy` gives leaves a stake of one, which tests each leaf at twice the Hierarchical Bonferroni level. The two procedures therefore part in a handful of replications, as counted above. The paper's own Figure 3 tests leaves at stake one, and the power tables below match only with leaves at stake one, so the paper's identical FWER counts are a near-certainty rather than an identity. Statistical Power (§7.3) ------------------------ The power study runs on the fixed partition, whose ten hypotheses are all false in both effect configurations: the running example, and the same configuration with the harmed quarter helped instead. Each configuration's effects are multiplied by a scale from 0.10 to 1.00, and each cell averages the true claims certified over 4,000 populations. A rejection counts as a true claim only if its null is false in the direction tested; a one-sided rejection in the wrong direction is a Type I error. Each cell shows the paper's mean, then the reconstruction's: .. include:: _generated/power.rst The reconstruction reproduces the paper's ranking at every scale. Trickle down certifies the most true claims at small and moderate effects. Fixed hierarchy falls furthest behind as effects grow, because one retention near the root ends its testing. Directing each test with the training split beats testing both sides, even at the smallest effects, where the training split often picks the wrong side. .. include:: _generated/power_gaps.rst Table 4 of the paper splits the 0.45 column by the depth of the claim. It shows where each procedure spends its power: fixed hierarchy is strongest at the root and collapses below it, while trickle down degrades gracefully. .. include:: _generated/depth.rst Every entry agrees with the paper's to within 0.03. Fixed hierarchy leads at the root and certifies almost nothing at depth two; trickle down gives up a little at the root and keeps most of its power at the leaves. These two tables pin down the reconstruction most tightly: the depth-two column moves sharply with the leaves' stake, and the fixed-hierarchy rows with the effect in the helped quarter. .. _assumptions: Assumptions ----------- Each entry below fills a gap in the paper's description of §7. The module :mod:`paper.simulation` records the same list in ``ASSUMPTIONS``, so code and documentation cannot disagree. .. include:: _generated/assumptions.rst Running the Reconstruction -------------------------- The reconstruction lives in ``paper/`` in the repository and uses only the public API of trickle-down. Its results are committed, so building these docs never reruns it. To regenerate them: .. code-block:: bash uv sync --extra dev uv run python -m paper.reproduce The full run takes about five minutes on ten cores. Every replicate draws from its own child of one seed, so the results do not depend on the number of workers; ``--replications`` runs a smaller study, and ``--render-only`` rebuilds these tables from ``paper/results.json``.