Basics

An end-to-end tutorial on how to use the dabest library.

Load libraries

import numpy as np
import pandas as pd
import dabest

print("We're using DABEST v{}".format(dabest.__version__))
Pre-compiling numba functions for DABEST...
Compiling numba functions: 100%|██████████| 11/11 [00:00<00:00, 44.16it/s]
Numba compilation complete!
We're using DABEST v2025.03.27

Create dataset for demo

Here, we create a dataset to illustrate how dabest works. In this dataset, each column corresponds to a group of observations.

from scipy.stats import norm # Used in generation of populations.

np.random.seed(9999) # Fix the seed to ensure reproducibility of results.

Ns = 20 # The number of samples taken from each population

# Create samples
c1 = norm.rvs(loc=3, scale=0.4, size=Ns)
c2 = norm.rvs(loc=3.5, scale=0.75, size=Ns)
c3 = norm.rvs(loc=3.25, scale=0.4, size=Ns)

t1 = norm.rvs(loc=3.5, scale=0.5, size=Ns)
t2 = norm.rvs(loc=2.5, scale=0.6, size=Ns)
t3 = norm.rvs(loc=3, scale=0.75, size=Ns)
t4 = norm.rvs(loc=3.5, scale=0.75, size=Ns)
t5 = norm.rvs(loc=3.25, scale=0.4, size=Ns)
t6 = norm.rvs(loc=3.25, scale=0.4, size=Ns)

# Add a `gender` column for coloring the data.
females = np.repeat('Female', Ns/2).tolist()
males = np.repeat('Male', Ns/2).tolist()
gender = females + males

# Add an `id` column for paired data plotting.
id_col = pd.Series(range(1, Ns+1))

# Combine samples and gender into a DataFrame.
df = pd.DataFrame({'Control 1' : c1,     'Test 1' : t1,
                 'Control 2' : c2,     'Test 2' : t2,
                 'Control 3' : c3,     'Test 3' : t3,
                 'Test 4'    : t4,     'Test 5' : t5, 'Test 6' : t6,
                 'Gender'    : gender, 'ID'  : id_col
                })
df.head()
Control 1 Test 1 Control 2 Test 2 Control 3 Test 3 Test 4 Test 5 Test 6 Gender ID
0 2.793984 3.420875 3.324661 1.707467 3.816940 1.796581 4.440050 2.937284 3.486127 Female 1
1 3.236759 3.467972 3.685186 1.121846 3.750358 3.944566 3.723494 2.837062 2.338094 Female 2
2 3.019149 4.377179 5.616891 3.301381 2.945397 2.832188 3.214014 3.111950 3.270897 Female 3
3 2.804638 4.564780 2.773152 2.534018 3.575179 3.048267 4.968278 3.743378 3.151188 Female 4
4 2.858019 3.220058 2.550361 2.796365 3.692138 3.276575 2.662104 2.977341 2.328601 Female 5

Note that we have 9 groups (3 Control samples and 6 Test samples). Our dataset has also a non-numerical column indicating gender, and another column indicating the identity of each observation.

This is known as a wide dataset. See this writeup for more details.

Loading data

Before creating estimation plots and obtaining confidence intervals for our effect sizes, we need to load the data and specify the relevant groups.

We can achieve this by supplying the dataframe to dabest.load(). Additionally, we must provide the two groups to be compared in the idx argument as a tuple or list.

two_groups_unpaired = dabest.load(df, idx=("Control 1", "Test 1"), resamples=5000)

Calling this Dabest object gives you a gentle greeting, as well as the comparisons that can be computed.

two_groups_unpaired
DABEST v2025.03.27
==================
                  
Good afternoon!
The current time is Tue Mar 25 16:02:11 2025.

Effect size(s) with 95% confidence intervals will be computed for:
1. Test 1 minus Control 1

5000 resamples will be used to generate the effect size bootstraps.

Changing statistical parameters

You can change the width of the confidence interval by manipulating the ci argument.

two_groups_unpaired_ci90 = dabest.load(df, idx=("Control 1", "Test 1"), ci=90)
two_groups_unpaired_ci90
DABEST v2025.03.27
==================
                  
Good afternoon!
The current time is Tue Mar 25 16:02:11 2025.

Effect size(s) with 90% confidence intervals will be computed for:
1. Test 1 minus Control 1

5000 resamples will be used to generate the effect size bootstraps.

Effect sizes

The dabest library now features a range of effect sizes:

Each of these are attributes of the Dabest object.

two_groups_unpaired.mean_diff
DABEST v2025.03.27
==================
                  
Good afternoon!
The current time is Tue Mar 25 16:02:11 2025.

The unpaired mean difference between Control 1 and Test 1 is 0.48 [95%CI 0.205, 0.774].
The p-value of the two-sided permutation t-test is 0.001, calculated for legacy purposes only. 

5000 bootstrap samples were taken; the confidence interval is bias-corrected and accelerated.
Any p-value reported is the probability of observing theeffect size (or greater),
assuming the null hypothesis of zero difference is true.
For each p-value, 5000 reshuffles of the control and test labels were performed.

To get the results of all valid statistical tests, use `.mean_diff.statistical_tests`

For each comparison, the type of effect size is reported (here, it’s the “unpaired mean difference”). The confidence interval is reported as: [confidenceIntervalWidth LowerBound, UpperBound]

This confidence interval is generated through bootstrap resampling. See bootstraps for more details.

Since v0.3.0, DABEST will report the p-value of the non-parametric two-sided approximate permutation t-test. This is also known as the Monte Carlo permutation test.

For unpaired comparisons, the p-values and test statistics of Welch’s t test, Student’s t test, and Mann-Whitney U test can be found. For paired comparisons, the p-values and test statistics of the paired Student’s t and Wilcoxon tests are presented.

pd.options.display.max_columns = 50
two_groups_unpaired.mean_diff.results
control test control_N test_N effect_size is_paired difference ci bca_low bca_high bca_interval_idx pct_low pct_high pct_interval_idx bootstraps resamples random_seed permutations pvalue_permutation permutation_count permutations_var pvalue_welch statistic_welch pvalue_students_t statistic_students_t pvalue_mann_whitney statistic_mann_whitney bec_difference bec_bootstraps bec_bca_interval_idx bec_bca_low bec_bca_high bec_pct_interval_idx bec_pct_low bec_pct_high
0 Control 1 Test 1 20 20 mean difference None 0.48029 95 0.205161 0.773647 (145, 4893) 0.197427 0.758752 (125, 4875) [0.6148498102262239, 0.6752095203445543, 0.300... 5000 12345 [-0.17259843762502491, 0.03802293852634886, -0... 0.001 5000 [0.26356588154404337, 0.2710249543904699, 0.26... 0.002094 -3.308806 0.002057 -3.308806 0.001625 83.0 0.0 [-0.09732932551566487, 0.08087009665445155, -0... (127, 4877) -0.256862 0.259558 (125, 4875) -0.25826 0.25759 
two_groups_unpaired.mean_diff.statistical_tests
control test control_N test_N effect_size is_paired difference ci bca_low bca_high pvalue_permutation pvalue_welch statistic_welch pvalue_students_t statistic_students_t pvalue_mann_whitney statistic_mann_whitney
0 Control 1 Test 1 20 20 mean difference None 0.48029 95 0.205161 0.773647 0.001 0.002094 -3.308806 0.002057 -3.308806 0.001625 83.0

Note: A research paper Phipson & Smyth (2010) suggested that permutation p-values should never be zero, and provided a slightly adjusted formula to compute permutation p-values.

Since v2025.03.27, DABEST provides a ps_adjust parameter in the .load() function. This parameter allows you to adjust the permutation p-values using the formula suggested by Phipson & Smyth (2010). By default, DABEST uses the unadjusted p-values.

two_groups_unpaired_adjusted = dabest.load(df, idx=("Control 1", "Test 1"), resamples=5000, ps_adjust=True)
two_groups_unpaired_adjusted.mean_diff.statistical_tests
control test control_N test_N effect_size is_paired difference ci bca_low bca_high pvalue_permutation pvalue_welch statistic_welch pvalue_students_t statistic_students_t pvalue_mann_whitney statistic_mann_whitney
0 Control 1 Test 1 20 20 mean difference None 0.48029 95 0.205161 0.773647 0.0012 0.002094 -3.308806 0.002057 -3.308806 0.001625 83.0

Let’s compute the Hedges’g for our comparison.

two_groups_unpaired.hedges_g
DABEST v2025.03.27
==================
                  
Good afternoon!
The current time is Tue Mar 25 16:02:11 2025.

The unpaired Hedges' g between Control 1 and Test 1 is 1.03 [95%CI 0.317, 1.62].
The p-value of the two-sided permutation t-test is 0.001, calculated for legacy purposes only. 

5000 bootstrap samples were taken; the confidence interval is bias-corrected and accelerated.
Any p-value reported is the probability of observing theeffect size (or greater),
assuming the null hypothesis of zero difference is true.
For each p-value, 5000 reshuffles of the control and test labels were performed.

To get the results of all valid statistical tests, use `.hedges_g.statistical_tests`
two_groups_unpaired.hedges_g.results
control test control_N test_N effect_size is_paired difference ci bca_low bca_high bca_interval_idx pct_low pct_high pct_interval_idx bootstraps resamples random_seed permutations pvalue_permutation permutation_count permutations_var pvalue_welch statistic_welch pvalue_students_t statistic_students_t pvalue_mann_whitney statistic_mann_whitney bec_difference bec_bootstraps bec_bca_interval_idx bec_bca_low bec_bca_high bec_pct_interval_idx bec_pct_low bec_pct_high
0 Control 1 Test 1 20 20 Hedges' g None 1.025525 95 0.316506 1.616235 (42, 4725) 0.44486 1.745146 (125, 4875) [1.469217954462509, 1.5972518056777079, 0.6051... 5000 12345 [-0.329508986559053, 0.07158401210924781, -0.2... 0.001 5000 [0.26356588154404337, 0.2710249543904699, 0.26... 0.002094 -3.308806 0.002057 -3.308806 0.001625 83.0 0.0 [-0.2669450878059954, 0.21187593591106418, -0.... (127, 4877) -0.642387 0.629464 (125, 4875) -0.643604 0.627968

Producing estimation plots

To generate a Gardner-Altman estimation plot, simply use the .plot() method. You can learn more about its genesis and design inspiration at robust-beautiful.

Each instance of an effect size has access to the .plot() method. This allows you to quickly create plots for different effect sizes with ease.

two_groups_unpaired.mean_diff.plot();

two_groups_unpaired.hedges_g.plot();

Instead of a Gardner-Altman plot, you can generate a Cumming estimation plot by setting float_contrast=False in the .plot() method. This will plot the bootstrap effect sizes below the raw data, and also displays the the mean (gap) and ± standard deviation of each group (vertical ends) as gapped lines. This design was inspired by Edward Tufte’s dictum to maximise the data-ink ratio.

two_groups_unpaired.hedges_g.plot(float_contrast=False);

The confidence interval shown on the contrast axis is a BCa confidence interval by default. This can be modified using the ci_type parameter in the .plot() method, whereby you can select between bca and pct (percentile).

two_groups_unpaired.mean_diff.plot(ci_type='pct');

Using long (aka ‘melted’) data frames

dabest can also handle ‘melted’ or ‘long’ data. This term is used because each row now corresponds to a single data point, with one column carrying the value and other columns containing ‘metadata’ describing that data point.

For more details on wide vs long or ‘melted’ data, refer to this Wikipedia article. The pandas documentation provides recipes for melting dataframes.

x='group'
y='metric'

value_cols = df.columns[:-2] # select all but the "Gender" and "ID" columns.

df_melted = pd.melt(df.reset_index(),
                    id_vars=["Gender", "ID"],
                    value_vars=value_cols,
                    value_name=y,
                    var_name=x)

df_melted.head() # Gives the first five rows of `df_melted`.
Gender ID group metric
0 Female 1 Control 1 2.793984
1 Female 2 Control 1 3.236759
2 Female 3 Control 1 3.019149
3 Female 4 Control 1 2.804638
4 Female 5 Control 1 2.858019

When your data is in this format, you need to specify the x and y columns in dabest.load().

analysis_of_long_df = dabest.load(df_melted, idx=("Control 1", "Test 1"),
                                     x="group", y="metric")

analysis_of_long_df
DABEST v2025.03.27
==================
                  
Good afternoon!
The current time is Tue Mar 25 16:02:12 2025.

Effect size(s) with 95% confidence intervals will be computed for:
1. Test 1 minus Control 1

5000 resamples will be used to generate the effect size bootstraps.

Dabest estimation plot designs

The dabest package implements a range of estimation plot designs aimed at depicting common experimental designs:

  1. Two-Group

  2. Shared Control (Unpaired) and Repeated Measures (Paired)

  3. Proportion Plots

  4. Mini-Meta

  5. Delta-Delta

  6. Forest Plots

In addition, as of Dabest v2025.03.27, we introduce a new plotting orientation: Horizontal Plots.

Lastly, we have a whole tutorial page for making aesthetic changes to dabest plots.