culebra.tools.Batch class

class Batch(trainer: Trainer, untie_best_fitness_function: FitnessFunction | None = None, test_fitness_function: FitnessFunction | None = None, results_base_filename: str | None = None, hyperparameters: dict | None = None, num_experiments: int | None = None)

Generate a batch of experiments.

Parameters:

  • trainer (Trainer) – The trainer method

  • untie_best_fitness_function (FitnessFunction, optional) – The fitness function used to select the best solution from those found by the trainer in case of a tie. If set to None, the training fitness function will be used. Defaults to None.

  • test_fitness_function (FitnessFunction, optional) – The fitness used to test. If None, the training fitness function will be used. Defaults to None.

  • results_base_filename (str, optional) – The base filename to save the results. If set to None, DEFAULT_RESULTS_BASENAME is used. Defaults to None

  • hyperparameters (dict, optional) – Hyperparameter values used in this evaluation

  • num_experiments (int, optional) – Number of experiments in the batch, defaults to DEFAULT_NUM_EXPERIMENTS

Raises:

  • TypeError – If trainer is not a valid trainer

  • TypeError – If test_fitness_function is not a valid fitness function

  • TypeError – If results_base_filename is not a valid file name

  • TypeError – If hyperparameters is not a dictionary

  • ValueError – If the keys in hyperparameters are not strings

  • ValueError – If any key in hyperparameters is reserved

  • TypeError – If num_experiments is not an integer

  • ValueError – If num_experiments is not greater than zero

Class attributes

Batch.feature_metric_functions = {'Rank': <function Metrics.rank>, 'Relevance': <function Metrics.relevance>}

Metrics calculated for the features in the set of solutions.

Batch.stats_functions = {'Avg': <function Series.mean>, 'Max': <function Series.max>, 'Min': <function Series.min>, 'Std': <function Series.std>}

Statistics calculated for the results gathered from all the experiments.

Class methods

classmethod Batch.load_pickle(filename: str) → Base

Load a pickled object from a file.

Parameters:

filename (str) – The file name.

Raises:
classmethod Batch.from_config(config_script_filename: str) → Batch

Generate a new evaluation from a configuration file.

Parameters:

config_script_filename (str, optional) – Path to the configuration file. If set to None, DEFAULT_CONFIG_SCRIPT_FILENAME is used. Defaults to None

Raises:

RuntimeError – If config_script_filename is an invalid file path or an invalid configuration file

classmethod Batch.generate_run_script(config_filename: str | None = None, run_script_filename: str | None = None) → None

Generate a script to run an evaluation.

The parameters for the evaluation are taken from a configuration file.

Parameters:
Raises:

  • TypeError – If config_filename or run_script_filename are not a valid filename

  • ValueError – If the extensions of config_filename or run_script_filename are not valid.

Properties

property Batch.trainer: Trainer

Get and set the trainer method.

Getter:

Return the trainer method

Setter:

Set a new trainer method

Type:

Trainer

Raises:

TypeError – If set to a value which is not a valid trainer

property Batch.untie_best_fitness_function: FitnessFunction | None

Get and set the fitness function to untie the best solutions.

Getter:

Return the untie fitness function

Setter:

Set a new untie fitness function. If set to None and several tied solutions are found by the trainer, the first of them will be returned.

Type:

FitnessFunction

Raises:

TypeError – If set to a value which is not a valid fitness funtion

property Batch.test_fitness_function: FitnessFunction | None

Get and set the test fitness function.

Getter:

Return the test fitness function

Setter:

Set a new test fitness function. If set to None, the training fitness function will also be used for testing.

Type:

FitnessFunction

Raises:

TypeError – If set to a value which is not a valid fitness funtion

property Batch.results_base_filename: str | None

Get and set the results base filename.

Getter:

Return the results base filename

Setter:

Set a new results base filename. If set to None, DEFAULT_RESULTS_BASENAME is used.

Raises:

TypeError – If set to an invalid file name

property Batch.results_pickle_filename: str

Get the filename used to save the pickled results.

Type:

str

property Batch.results_excel_filename: str

Get the filename used to save the results in Excel format.

Type:

str

property Batch.hyperparameters: dict | None

Get and set the hyperparameter values used for the evaluation.

Getter:

Return the hyperparameter values

Setter:

Set a new set of hyperparameter values.

Raises:

  • TypeError – If the hyperparameters are not in a dictionary

  • ValueError – If the keys of the dictionary are not strings

  • ValueError – If any key in the dictionary is reserved

property Batch.results: Dict[str, DataFrame] | None

Get all the results provided.

Type:

Results

property Batch.num_experiments: int

Get and set the number of experiments in the batch.

Getter:

Return the number of experiments

Setter:

Set a new number of experiments. If set to None, DEFAULT_NUM_EXPERIMENTS is used

Type:

int

Raises:

  • TypeError – If set to a value which is not an integer

  • ValueError – If set to a value which is not greater than zero

property Batch.experiment_basename: str

Return the experiments basename.

Type:

str

property Batch.experiment_labels: Tuple[str]

Get the label to identify each one of the experiments in the batch.

Type:

tuple of str

Methods

Batch.save_pickle(filename: str) → None

Pickle this object and save it to a file.

Parameters:

filename (str) – The file name.

Raises:
Batch.reset() → None

Reset the results.

Batch.setup() → None

Set up the batch.

Create all the experiments for the batch.

Batch.run() → None

Execute the batch and save the results.

Private methods

Batch._append_data(result_key: str, exp_label: str, exp_data: Series | DataFrame) → None

Append data from an experiment to a results dataframe.

Parameters:

  • result_key (str) – Key of the result

  • exp_label (str) – Label of the experiment

  • exp_data (Series or DataFrame) – Data of the result

Batch._add_execution_metrics_stats() → None

Perform some stats on the execution metrics.

Batch._add_feature_metrics_stats() → None

Perform stats on the feature metrics of all the experiments.

Batch._add_fitness_stats(input_data_key: str, result_key: str) → None

Perform some stats on the best solutions fitness.

Parameters:

  • input_data_key (str) – Input data key.

  • result_key (str) – Result key.

Batch._execute() → None

Execute a batch of experiments.

Batch._is_reserved(name: str) → bool

Return True if the given hyperparameter name is reserved.

Parameters:

name – Hyperparameter name