Effectsize objects

The auxiliary classes involved in the computations of bootstrapped effect sizes.

TwoGroupsEffectSize

 TwoGroupsEffectSize (control, test, effect_size, proportional=False,
                      is_paired=None, ci=95, resamples=5000,
                      permutation_count=5000, random_seed=12345,
                      ps_adjust=False)

*A class to compute and store the results of bootstrapped mean differences between two groups.

Compute the effect size between two groups.*

	Type	Default	Details
control	array-like
test	array-like		These should be numerical iterables.
effect_size	string.		Any one of the following are accepted inputs: ‘mean_diff’, ‘median_diff’, ‘cohens_d’, ‘hedges_g’, or ‘cliffs_delta’
proportional	bool	False
is_paired	NoneType	None
ci	int	95	The confidence interval width. The default of 95 produces 95% confidence intervals.
resamples	int	5000	The number of bootstrap resamples to be taken for the calculation of the confidence interval limits.
permutation_count	int	5000	The number of permutations (reshuffles) to perform for the computation of the permutation p-value
random_seed	int	12345	`random_seed` is used to seed the random number generator during bootstrap resampling. This ensures that the confidence intervals reported are replicable.
ps_adjust	bool	False	If True, adjust calculated p-value according to Phipson & Smyth (2010) # https://doi.org/10.2202/1544-6115.1585
Returns	py:class:`TwoGroupEffectSize` object:		`difference` : float The effect size of the difference between the control and the test. `effect_size` : string The type of effect size reported. `is_paired` : string The type of repeated-measures experiment. `ci` : float Returns the width of the confidence interval, in percent. `alpha` : float Returns the significance level of the statistical test as a float between 0 and 1. `resamples` : int The number of resamples performed during the bootstrap procedure. `bootstraps` : numpy ndarray The generated bootstraps of the effect size. `random_seed` : int The number used to initialise the numpy random seed generator, ie.`seed_value` from `numpy.random.seed(seed_value)` is returned. `bca_low, bca_high` : float The bias-corrected and accelerated confidence interval lower limit and upper limits, respectively. `pct_low, pct_high` : float The percentile confidence interval lower limit and upper limits, respectively.

Example

random.seed(12345)
control = norm.rvs(loc=0, size=30)
test = norm.rvs(loc=0.5, size=30)
effsize = dabest.TwoGroupsEffectSize(control, test, "mean_diff")
effsize

The unpaired mean difference is -0.253 [95%CI -0.782, 0.241].
The p-value of the two-sided permutation t-test is 0.348, calculated for legacy purposes only. 

5000 bootstrap samples were taken; the confidence interval is bias-corrected and accelerated.
Any p-value reported is the probability of observing theeffect size (or greater),
assuming the null hypothesis of zero difference is true.
For each p-value, 5000 reshuffles of the control and test labels were performed.

effsize.to_dict()

{'alpha': 0.05,
 'bca_high': 0.2413346581369784,
 'bca_interval_idx': (109, 4858),
 'bca_low': -0.7818088458343655,
 'bec_bca_high': 0.5352403905584314,
 'bec_bca_interval_idx': (130, 4880),
 'bec_bca_low': -0.4982839949134528,
 'bec_bootstraps': array([-0.48953946, -0.18565285, -0.23896785, ..., -0.55130928,
         0.16037238, -0.07364879]),
 'bec_difference': 0.0,
 'bec_pct_high': 0.5280564736117328,
 'bec_pct_interval_idx': (125, 4875),
 'bec_pct_low': -0.5041777340626885,
 'bootstraps': array([-0.23923425, -0.66013733, -0.42672232, ..., -0.33191074,
        -0.16543251, -0.34179536]),
 'ci': 95,
 'difference': -0.25315417702752846,
 'effect_size': 'mean difference',
 'is_paired': None,
 'is_proportional': False,
 'pct_high': 0.25135646125431527,
 'pct_interval_idx': (125, 4875),
 'pct_low': -0.763588353717278,
 'permutation_count': 5000,
 'permutations': array([ 0.17221029,  0.03112419, -0.13911387, ..., -0.38007941,
         0.30261507, -0.09073054]),
 'permutations_var': array([0.07201642, 0.07251104, 0.07219407, ..., 0.07003705, 0.07094885,
        0.07238581]),
 'proportional_difference': nan,
 'pvalue_brunner_munzel': nan,
 'pvalue_kruskal': nan,
 'pvalue_mann_whitney': 0.5201446121616038,
 'pvalue_mcnemar': nan,
 'pvalue_paired_students_t': nan,
 'pvalue_permutation': 0.3484,
 'pvalue_students_t': 0.34743913903372836,
 'pvalue_welch': 0.3474493875548964,
 'pvalue_wilcoxon': nan,
 'random_seed': 12345,
 'resamples': 5000,
 'statistic_brunner_munzel': nan,
 'statistic_kruskal': nan,
 'statistic_mann_whitney': 494.0,
 'statistic_mcnemar': nan,
 'statistic_paired_students_t': nan,
 'statistic_students_t': 0.9472545159069105,
 'statistic_welch': 0.9472545159069105,
 'statistic_wilcoxon': nan}

EffectSizeDataFrame

 EffectSizeDataFrame (dabest, effect_size, is_paired, ci=95,
                      proportional=False, resamples=5000,
                      permutation_count=5000, random_seed=12345,
                      x1_level=None, x2=None, delta2=False,
                      experiment_label=None, mini_meta=False,
                      ps_adjust=False)

A class that generates and stores the results of bootstrapped effect sizes for several comparisons.

Example: plot

Create a Gardner-Altman estimation plot for the mean difference.

random.seed(9999) # Fix the seed so the results are replicable.
# pop_size = 10000 # Size of each population.
Ns = 20 # The number of samples taken from each population

# Create samples
c1 = norm.rvs(loc=3, scale=0.4, size=Ns)
c2 = norm.rvs(loc=3.5, scale=0.75, size=Ns)
c3 = norm.rvs(loc=3.25, scale=0.4, size=Ns)

t1 = norm.rvs(loc=3.5, scale=0.5, size=Ns)
t2 = norm.rvs(loc=2.5, scale=0.6, size=Ns)
t3 = norm.rvs(loc=3, scale=0.75, size=Ns)
t4 = norm.rvs(loc=3.5, scale=0.75, size=Ns)
t5 = norm.rvs(loc=3.25, scale=0.4, size=Ns)
t6 = norm.rvs(loc=3.25, scale=0.4, size=Ns)


# Add a `gender` column for coloring the data.
females = repeat('Female', Ns/2).tolist()
males = repeat('Male', Ns/2).tolist()
gender = females + males

# Add an `id` column for paired data plotting.
id_col = pd.Series(range(1, Ns+1))

# Combine samples and gender into a DataFrame.
df = pd.DataFrame({'Control 1' : c1,     'Test 1' : t1,
                 'Control 2' : c2,     'Test 2' : t2,
                 'Control 3' : c3,     'Test 3' : t3,
                 'Test 4'    : t4,     'Test 5' : t5, 'Test 6' : t6,
                 'Gender'    : gender, 'ID'  : id_col
                })
my_data = dabest.load(df, idx=("Control 1", "Test 1"))

fig1 = my_data.mean_diff.plot();

Create a Gardner-Altman plot for the Hedges’ g effect size.

fig2 = my_data.hedges_g.plot();

Create a Cumming estimation plot for the mean difference.

fig3 = my_data.mean_diff.plot(float_contrast=True);

Create a paired Gardner-Altman plot.

my_data_paired = dabest.load(df, idx=("Control 1", "Test 1"),
                       id_col = "ID", paired='baseline')
fig4 = my_data_paired.mean_diff.plot();

Create a multi-group Cumming plot.

my_multi_groups = dabest.load(df, id_col = "ID", 
                             idx=(("Control 1", "Test 1"),
                                 ("Control 2", "Test 2")))
fig5 = my_multi_groups.mean_diff.plot();

/Users/jonathananns/GitHub/DABEST-python/dabest/plot_tools.py:2778: UserWarning: 5.0% of the points cannot be placed. You might want to decrease the size of the markers.
  warnings.warn(err)
/Users/jonathananns/GitHub/DABEST-python/dabest/plot_tools.py:2778: UserWarning: 10.0% of the points cannot be placed. You might want to decrease the size of the markers.
  warnings.warn(err)

Create a shared control Cumming plot.

my_shared_control = dabest.load(df, id_col = "ID",
                                 idx=("Control 1", "Test 1",
                                          "Test 2", "Test 3"))
fig6 = my_shared_control.mean_diff.plot();

/Users/jonathananns/GitHub/DABEST-python/dabest/plot_tools.py:2778: UserWarning: 10.0% of the points cannot be placed. You might want to decrease the size of the markers.
  warnings.warn(err)

Create a repeated meausures (against baseline) Slopeplot.

my_rm_baseline = dabest.load(df, id_col = "ID", paired = "baseline",
                                 idx=("Control 1", "Test 1",
                                          "Test 2", "Test 3"))
fig7 = my_rm_baseline.mean_diff.plot();

Create a repeated meausures (sequential) Slopeplot.

my_rm_sequential = dabest.load(df, id_col = "ID", paired = "sequential",
                                 idx=("Control 1", "Test 1",
                                          "Test 2", "Test 3"))
fig8 = my_rm_sequential.mean_diff.plot();

PermutationTest

 PermutationTest (control:<built-infunctionarray>, test:<built-
                  infunctionarray>, effect_size:str, is_paired:str=None,
                  permutation_count:int=5000, random_seed:int=12345,
                  ps_adjust:bool=False, **kwargs)

A class to compute and report permutation tests.

	Type	Default	Details
control	array
test	array		These should be numerical iterables.
effect_size	str		Any one of the following are accepted inputs: ‘mean_diff’, ‘median_diff’, ‘cohens_d’, ‘hedges_g’, or ‘cliffs_delta’
is_paired	str	None
permutation_count	int	5000	The number of permutations (reshuffles) to perform.
random_seed	int	12345	`random_seed` is used to seed the random number generator during bootstrap resampling. This ensures that the generated permutations are replicable.
ps_adjust	bool	False	If True, the p-value is adjusted according to Phipson & Smyth (2010). # https://doi.org/10.2202/1544-6115.1585
kwargs	VAR_KEYWORD
Returns	py:class:`PermutationTest` object:		`difference`:float The effect size of the difference between the control and the test. `effect_size`:string The type of effect size reported.

Notes:

The basic concept of permutation tests is the same as that behind bootstrapping. In an “exact” permutation test, all possible resuffles of the control and test labels are performed, and the proportion of effect sizes that equal or exceed the observed effect size is computed. This is the probability, under the null hypothesis of zero difference between test and control groups, of observing the effect size: the p-value of the Student’s t-test.

Exact permutation tests are impractical: computing the effect sizes for all reshuffles quickly exceeds trivial computational loads. A control group and a test group both with 10 observations each would have a total of $20!$ or $2.43 \times 10^{18}$ reshuffles. Therefore, in practice, “approximate” permutation tests are performed, where a sufficient number of reshuffles are performed (5,000 or 10,000), from which the p-value is computed.

More information can be found here.

Example: permutation test

control = norm.rvs(loc=0, size=30, random_state=12345)
test = norm.rvs(loc=0.5, size=30, random_state=12345)
perm_test = dabest.PermutationTest(control, test, 
                                   effect_size="mean_diff", 
                                   is_paired=None)
perm_test

5000 permutations were taken. The p-value is 0.0758.