Visualizations#

Balance#

Balance plots and the rank test behind them.

plot_balance and plot_balance_2_sample draw standardized differences between weighted sample means and their targets; wilcoxon_signed_rank is the one-sample rank alternative offered wherever plot_balance takes test.

pyrake.calibration.visualizations.plot_balance(X, mu, covariates, weights, test=None, sigma=None, signed=True, title=None, quantify_imbalance=True, imbalance_pval_threshold=0.2, legend_placement='lower right', ax=None, means=None)[source]#

Plot one-sample z-statistics for covariates.

The one-sample z-statistic tests a null hypothesis regarding the mean or center of symmetry of the sample. For covariates without outliers, this can be as simple as the difference in sample mean and population mean, but for covariates with fat tails, it’s better to use a non-parametric test for location.

Parameters:
  • X (npt.NDArray[np.float64]) – X_ij = feature j for respondent i.

  • mu (npt.NDArray[np.float64]) – Mean (or median; see Notes) of covariates in target population.

  • covariates (list[str]) – Feature names.

  • weights (dict[str, npt.NDArray[np.float64]]) – One or more sets of weights.

  • test (["z", "wilcoxon"] or list of such options, optional) – Test to apply when comparing sample to population, for each covariate. The default behavior is to use simple z tests, which just compare the weighted sample mean to the population mean. For continuous covariates with potentially large values, means can be unstable, so we also support using the one-sample Wilcoxon signed rank test. See Notes. This input can be specified for each covariate separately.

  • sigma (npt.NDArray[np.float64], optional) – Sqrt of population variance. Only used with test='z'. If not specified, the (sqrt of the) sample variance of X will be used.

  • signed (bool, optional) – If True (default), plot the signed z-statistics. Otherwise, plot the absolute value of the z-statistics, which results in more condensed plots.

  • title (str, optional) – A title for the plot.

  • quantify_imbalance (bool, optional) – If True (default), annotate legend with the imbalance metric for each set of weights. Larger values indicate greater imbalance. See Notes.

  • imbalance_pval_threshold (float, optional) – P-value threshold for calculating imbalance metric. See Notes. Should be in (0, 1]. Defaults to 0.2.

  • legend_placement (str, optional) – Where to place the legend. Defaults to “lower right”.

  • ax (matplotlib Axes, optional) – If specified, plot on ax.

  • means (dict of covariate vectors, optional) – Additional methods’ model-estimated means, in covariate order and original units. A Series may name a subset of covariates; omitted entries are not plotted or counted in that method’s imbalance. Method names must not overlap weights. Only the ‘z’ diagnostic supports means; there are no synthetic respondent weights. Uses the same nominal M and sigma as weighted means. These coordinates need not be inferential z-statistics for a model-based estimate.

Returns:

  • df_balance (pandas DataFrame) – The z-statistics for each covariate and each set of weights.

  • ax (Axes) – The figure.

Raises:

ValueError – On dimension mismatches between X, mu, covariates, sigma and test; on an unknown test name; when model means accompany a non-‘z’ diagnostic; or when imbalance_pval_threshold falls outside (0, 1].

Notes

The Wilcoxon signed rank test computes the difference between the sample values and mu, which should be the population median for this test. The test statistic is the weighted sum of the ranks corresponding to sample values larger than the population median.

The imbalance is quantified by aggregating the p-values for each covariate. Any p-values less than imbalance_pval_threshold represent covariates with large imbalance; other covariates are ignored. The product of these p-values is a measure of imbalance, but to avoid numeric underflow when working with very small p-values (and very out-of-balance covariates), we report the logarithm of the product of p-values, which is equivalent to the sum of logs of the p-values. We negate the result to get a non-negative metric, with larger values indicating greater imbalance. When quantify_imbalance is True, the resulting metrics are included in the plot legend.

pyrake.calibration.visualizations.plot_balance_2_sample(X1, X2, covariates, weights1, weights2, sigma1=None, sigma2=None, signed=True, title=None, quantify_imbalance=True, imbalance_pval_threshold=0.2, legend_placement='lower right', ax=None, means1=None, means2=None)[source]#

Plot z-statistics for covariates.

If we were testing the null hypothesis that the two populations have equal covariate averages, then the z-statistic for each covariate would be:

(1/M2) X2ᵀw2 - (1/M1) X1ᵀw1
---------------------------.
  √(σ1² / M1 + σ2² / M2)
Parameters:
  • X1 (npt.NDArray[np.float64]) – X_ij = feature j for respondent i in population 1.

  • X2 (npt.NDArray[np.float64]) – X_ij = feature j for respondent i in population 2.

  • covariates (list[str]) – Feature names.

  • weights1 (dict[str, npt.NDArray[np.float64]]) – One or more sets of weights for population 1.

  • weights2 (dict[str, npt.NDArray[np.float64]]) – One or more sets of weights for population 2.

  • sigma1 (npt.NDArray[np.float64], optional) – Square root of variances for population 1, for putting all covariates on the same scale. See below for which values to pass.

  • sigma2 (npt.NDArray[np.float64], optional) – Square root of variances for population 2. The exact values to pass here depend on context, so use judgment about which z-statistic you want to calculate. The key here is to use the same variances regardless of which weights are applied, so you could use variances of each sample, or variances of the respective populations.

  • signed (bool, optional) – If True (default), plot the signed z-statistics. Otherwise, plot the two-sided z-statistics, which are always >= 0. This results in more condensed plots.

  • title (str, optional) – A title for the plot.

  • quantify_imbalance (bool, optional) – If True (default), annotate legend with the imbalance metric for each set of weights. Larger values indicate greater imbalance. See Notes.

  • imbalance_pval_threshold (float, optional) – P-value threshold for calculating imbalance metric. See Notes. Should be in (0, 1]. Defaults to 0.2.

  • legend_placement (str, optional) – Where to place the legend. Defaults to “lower right”.

  • ax (matplotlib Axes, optional) – If specified, plot on ax.

  • means1 (dict of covariate vectors, optional) – Model means for additional matching method names for population 1, distinct from weights.

  • means2 (dict of covariate vectors, optional) – Model means for population 2, with keys matching means1. Uses (mean2 - mean1) on the same nominal-count/population-SD scale as weighted means; no weights or posterior tests are implied.

Returns:

  • df_balance (pandas DataFrame) – The z-statistics for each covariate and each set of weights.

  • ax (Axes) – The figure.

Raises:

ValueError – When the weight or mean method keys disagree across groups, X1 and X2 differ in columns, or imbalance_pval_threshold falls outside (0, 1].

Notes

The imbalance is quantified by aggregating the p-values for each covariate. Presently we are just doing a z-test on the difference in means, but a future state might use something non-parametric.

Any p-values less than imbalance_pval_threshold represent covariates with large imbalance; other covariates are ignored. The product of these p-values is a measure of imbalance, but to avoid numeric underflow when working with very small p-values (and very out-of-balance covariates), we report the logarithm of the product of p-values, which is equivalent to the sum of logs of the p-values. We negate the result to get a non-negative metric, with larger values indicating greater imbalance.

When quantify_imbalance is True, the resulting metrics are included in the plot legend.

Meta-Analysis#

Estimation visualizations.

pyrake.estimation.visualizations.meta_analysis(estimators, null_min, null_max, num_points=1000, alpha=0.1, title=None, xlabel=None, xtick_format='.0%', axis_label_size=None, tick_label_size=None, legend_label_size=None, legend_placement=None, ax=None)[source]#

Combine estimates from multiple platforms.

Parameters:
  • estimators (dict_like) – Estimators for each platform. Keys should be the name of the platform, and value should be a WeightingEstimator corresponding to a particular metric.

  • null_min (float) – Range of null hypotheses to test.

  • null_max (float) – Range of null hypotheses to test.

  • num_points (int, optional) – Number of hypotheses to check. Defaults to 1_000.

  • alpha (float, optional) – Threshold for statistical significance. Defaults to 0.10.

  • title (str, optional) – Optional title for figure.

  • xlabel (str, optional) – Optional xlabel for figure. Defaults to “Null Hypothesis”.

  • xtick_format (str, optional) – Format for tick marks on x-axis. Defaults to “.0%”, leading to xticks like “50%”.

  • axis_label_size (int, optional) – Font sizes for various figure elements.

  • tick_label_size (int, optional) – Font sizes for various figure elements.

  • legend_label_size (int, optional) – Font sizes for various figure elements.

  • legend_placement (str, optional) – Where to put the legend.

  • ax (Axes, optional) – Where to plot the figure.

Returns:

  • point_estimate (float) – Combined point estimate.

  • confidence_interval (tuple[float, float]) – Combined confidence interval.

  • ax (Axes) – The plot axes.