culebra.tools.ResultsAnalyzer class

class ResultsAnalyzer(dict=None, /, **kwargs)

Perform statistical analyses over the results of several batches.

Class methods

classmethod ResultsAnalyzer.load_pickle(filename: str) → Base

Load a pickled object from a file.

Parameters:

filename (str) – The file name.

Raises:

• TypeError – If filename is not a valid file name

• ValueError – If the filename extension is not PICKLE_FILE_EXTENSION

Methods

ResultsAnalyzer.save_pickle(filename: str) → None

Pickle this object and save it to a file.

Parameters:

filename (str) – The file name.

Raises:

• TypeError – If filename is not a valid file name

• ValueError – If the filename extension is not PICKLE_FILE_EXTENSION

ResultsAnalyzer.normality_test(dataframe_key: str, column: str, alpha: float = 0.05, test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function shapiro>) → TestOutcome

Assess the normality of the same results for several batches.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

• test (Callable, optional) – Normality test to be applied, defaults to DEFAULT_NORMALITY_TEST

Returns:

The results of the normality test

Return type:

TestOutcome

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If test is not a valid normality test

• ValueError – If there aren’t any data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.homoscedasticity_test(dataframe_key: str, column: str, alpha: float = 0.05, test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function bartlett>) → TestOutcome

Assess the homoscedasticity of the same results for several batches.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

• test (Callable, optional) – Homoscedasticity test to be applied, defaults to DEFAULT_HOMOSCEDASTICITY_TEST

Returns:

The results of the homoscedasticity test

Return type:

TestOutcome

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If test is not a valid homoscedasticity test

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.parametric_test(dataframe_key: str, column: str, alpha: float = 0.05) → TestOutcome

Compare the results of several batches.

Data should be independent, follow a normal distribution and also be homoscedastic. If only two results are analyzed, the T-test (ttest_ind()) is applied. For more results, the one-way ANOVA test (f_oneway()) is used instead.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

Returns:

The results of the comparison

Return type:

TestOutcome

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.non_parametric_test(dataframe_key: str, column: str, alpha: float = 0.05) → TestOutcome

Compare the results of several batches.

Data should be independent. If only two results are analyzed, the Mann-Whitney U-test (mannwhitneyu()) is applied. For more results, the Kruskal-Wallis H-test (kruskal()) is used instead.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

Returns:

The results of the comparison

Return type:

TestOutcome

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.parametric_pairwise_test(dataframe_key: str, column: str, alpha: float = 0.05) → TestOutcome

Pairwise comparison the results of several batches.

Data should be independent, follow a normal distribution and also be homoscedastic.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

Returns:

The results of the comparison

Return type:

TestOutcome

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.non_parametric_pairwise_test(dataframe_key: str, column: str, alpha: float = 0.05, p_adjust: str = 'fdr_tsbky') → TestOutcome

Pairwise comparison the results of several batches.

Data should be independent.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

• p_adjust (str or None, optional) – Method for adjusting the p-values, defaults to DEFAULT_P_ADJUST

Returns:

The results of the comparison

Return type:

TestOutcome

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If p_adjust is not None or any valid p-value adjustment method.

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.effect_size(dataframe_key: str, column: str) → TestOutcome

Effect size calculation for several batches.

The effect size is calculated with the Cohen’s d index.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

Returns:

The results of the effect sizes estimation

Return type:

EffectSize

Raises:

ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.compare(dataframe_key: str, column: str, alpha: float = 0.05, normality_test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function shapiro>, homoscedasticity_test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function bartlett>, p_adjust: str = 'fdr_tsbky') → ResultsComparison

Pairwise comparison the results of several batches.

Data should be independent.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

• normality_test (Callable, optional) – Normality test to be applied, defaults to DEFAULT_NORMALITY_TEST

• homoscedasticity_test (Callable, optional) – Homoscedasticity test to be applied, defaults to DEFAULT_HOMOSCEDASTICITY_TEST

• p_adjust (str or None, optional) – Method for adjusting the p-values, defaults to DEFAULT_P_ADJUST

Returns:

The results of the comparison

Return type:

ResultsComparison

Raises:

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If normality_test is not a valid normality test

• ValueError – If homoscedasticity_test is not a valid homoscedasticity test

• ValueError – If p_adjust is not None or any valid p-value adjustment method.

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.rank(dataframe_key: str, column: str, weight: int, alpha: float = 0.05, normality_test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function shapiro>, homoscedasticity_test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function bartlett>, p_adjust: str = 'fdr_tsbky') → Series

Rank the batches according to a concrete result.

Batches are ranked according to the procedure proposed in [Gonzalez2021]. The rank of each batch is calculated as the number of batches whose result is better (with a statistical significative difference) than that of it. Batches with a statistically similar result share the same rank.

Parameters:

• dataframe_key (str) – Key to select a dataframe from the results of all the batches

• column (str) – Column label to be analyzed in the selected dataframes from the results of all the batches

• weight (int) – Used for the comparison of the batches results selected by the dataframe_key and column provided. A negative value implies minimization (lower values are better), while a positive weight implies maximization.

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

• normality_test (Callable, optional) – Normality test to be applied, defaults to DEFAULT_NORMALITY_TEST

• homoscedasticity_test (Callable, optional) – Homoscedasticity test to be applied, defaults to DEFAULT_HOMOSCEDASTICITY_TEST

• p_adjust (str or None, optional) – Method for adjusting the p-values, defaults to DEFAULT_P_ADJUST

Returns:

The ranked batches

Return type:

Series

Raises:

• TypeError – If weight is not an integer number

• ValueError – If weight is 0

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If normality_test is not a valid normality test

• ValueError – If homoscedasticity_test is not a valid homoscedasticity test

• ValueError – If p_adjust is not None or any valid p-value adjustment method.

• ValueError – If there aren’t sufficient data in the analyzed results with such dataframe key and column label

ResultsAnalyzer.multiple_rank(dataframe_keys: ~collections.abc.Sequence[str], columns: ~collections.abc.Sequence[str], weights: ~collections.abc.Sequence[int], alpha: float = 0.05, normality_test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function shapiro>, homoscedasticity_test: ~typing.Callable[[~collections.abc.Sequence], ~collections.namedtuple] = <function bartlett>, p_adjust: str = 'fdr_tsbky') → DataFrame

Rank the batches according to multiple results.

Batches are ranked according to the procedure proposed in [Gonzalez2021]. The rank of each batch is calculated as the number of batches whose result is better (with a statistical significative difference) than that of it. Batches with a statistically similar result share the same rank.

The dataframe_keys, columns and weights must have the same length, and will be used to obtain different ranks for the results.

Parameters:

• dataframe_keys (Sequence of str) – Sequence of dataframe keys to select the different results of the batches

• columns (Sequence of str) – Sequence of column labels to select the different results of the batches

• weights (Sequence of int) – Sequence of weights to be applied to the different results of the batches. Negative values imply minimization (lower values are better), while a positive weights imply maximization.

• alpha (float, optional) – Significance level, defaults to DEFAULT_ALPHA

• normality_test (Callable, optional) – Normality test to be applied, defaults to DEFAULT_NORMALITY_TEST

• homoscedasticity_test (Callable, optional) – Homoscedasticity test to be applied, defaults to DEFAULT_HOMOSCEDASTICITY_TEST

• p_adjust (str or None, optional) – Method for adjusting the p-values, defaults to DEFAULT_P_ADJUST

Returns:

The ranked batches

Return type:

DataFrame

Raises:

• TypeError – If any weight is not an integer number

• ValueError – If any weight is 0

• TypeError – If alpha is not a real number

• ValueError – If alpha is not in [0, 1]

• ValueError – If normality_test is not a valid normality test

• ValueError – If homoscedasticity_test is not a valid homoscedasticity test

• ValueError – If p_adjust is not None or any valid p-value adjustment method.

• ValueError – If there aren’t sufficient data in the analyzed results with any given dataframe key and column label

Dunder methods

ResultsAnalyzer.__setitem__(batch_key: str, batch_results: Results) → Results

Overridden to verify the batch_key and batch_results.

Assure that batch_key is a str and batch_results is a Results.

Parameters:

• batch_key (str) – Key to identify the batch_results within the analyzer

• batch_results (Results) – The results of a batch of experiments

Returns:

The inserted results

Return type:

Results