culebra.tools.Experiment class

class Experiment(trainer: Trainer, untie_best_fitness_function: FitnessFunction | None = None, test_fitness_function: FitnessFunction | None = None, results_base_filename: str | None = None, hyperparameters: dict | None = None)

Set a trainer evaluation.

Parameters:

  • trainer (Trainer) – The trainer method

  • untie_best_fitness_function (FitnessFunction, optional) – The fitness function used to select the best solution from those found by the trainer in case of a tie. If set to None, the training fitness function will be used. Defaults to None.

  • test_fitness_function (FitnessFunction, optional) – The fitness function used to test. If set to None, the training fitness function will be used. Defaults to None.

  • results_base_filename (str, optional) – The base filename to save the results. If set to None, DEFAULT_RESULTS_BASENAME is used. Defaults to None

  • hyperparameters (dict, optional) – Hyperparameter values used in this evaluation

Raises:

  • TypeError – If trainer is not a valid trainer

  • TypeError – If test_fitness_function is not a valid fitness function

  • TypeError – If results_base_filename is not a valid file name

  • TypeError – If hyperparameters is not a dictionary

  • ValueError – If the keys in hyperparameters are not strings

  • ValueError – If any key in hyperparameters is reserved

Class attributes

Experiment.feature_metric_functions = {'Rank': <function Metrics.rank>, 'Relevance': <function Metrics.relevance>}

Metrics calculated for the features in the set of solutions.

Experiment.stats_functions = {'Avg': <function mean>, 'Max': <function max>, 'Min': <function min>, 'Std': <function std>}

Statistics calculated for the solutions.

Class methods

classmethod Experiment.load_pickle(filename: str) → Base

Load a pickled object from a file.

Parameters:

filename (str) – The file name.

Raises:
classmethod Experiment.from_config(config_script_filename: str | None = None) → Evaluation

Generate a new evaluation from a configuration file.

Parameters:

config_script_filename (str, optional) – Path to the configuration file. If set to None, DEFAULT_CONFIG_SCRIPT_FILENAME is used. Defaults to None

Raises:

RuntimeError – If config_script_filename is an invalid file path or an invalid configuration file

classmethod Experiment.generate_run_script(config_filename: str | None = None, run_script_filename: str | None = None) → None

Generate a script to run an evaluation.

The parameters for the evaluation are taken from a configuration file.

Parameters:
Raises:

  • TypeError – If config_filename or run_script_filename are not a valid filename

  • ValueError – If the extensions of config_filename or run_script_filename are not valid.

Properties

property Experiment.trainer: Trainer

Get and set the trainer method.

Getter:

Return the trainer method

Setter:

Set a new trainer method

Type:

Trainer

Raises:

TypeError – If set to a value which is not a valid trainer

property Experiment.untie_best_fitness_function: FitnessFunction | None

Get and set the fitness function to untie the best solutions.

Getter:

Return the untie fitness function

Setter:

Set a new untie fitness function. If set to None and several tied solutions are found by the trainer, the first of them will be returned.

Type:

FitnessFunction

Raises:

TypeError – If set to a value which is not a valid fitness funtion

property Experiment.test_fitness_function: FitnessFunction | None

Get and set the test fitness function.

Getter:

Return the test fitness function

Setter:

Set a new test fitness function. If set to None, the training fitness function will also be used for testing.

Type:

FitnessFunction

Raises:

TypeError – If set to a value which is not a valid fitness funtion

property Experiment.results_base_filename: str | None

Get and set the results base filename.

Getter:

Return the results base filename

Setter:

Set a new results base filename. If set to None, DEFAULT_RESULTS_BASENAME is used.

Raises:

TypeError – If set to an invalid file name

property Experiment.results_pickle_filename: str

Get the filename used to save the pickled results.

Type:

str

property Experiment.results_excel_filename: str

Get the filename used to save the results in Excel format.

Type:

str

property Experiment.hyperparameters: dict | None

Get and set the hyperparameter values used for the evaluation.

Getter:

Return the hyperparameter values

Setter:

Set a new set of hyperparameter values.

Raises:

  • TypeError – If the hyperparameters are not in a dictionary

  • ValueError – If the keys of the dictionary are not strings

  • ValueError – If any key in the dictionary is reserved

property Experiment.results: Dict[str, DataFrame] | None

Get all the results provided.

Type:

Results

property Experiment.best_solutions: Sequence[HallOfFame] | None

Return the best solutions found by the trainer.

property Experiment.best_representatives: List[List[Solution]] | None

Return the best representatives found by the trainer.

Methods

Experiment.save_pickle(filename: str) → None

Pickle this object and save it to a file.

Parameters:

filename (str) – The file name.

Raises:
Experiment.reset() → None

Reset the results.

Overridden to reset the best solutions and best representatives.

Experiment.run() → None

Execute the evaluation and save the results.

Private methods

Experiment._do_training() → None

Perform the training step.

Train the trainer and get the best solutions and the training stats.

Experiment._add_training_stats() → None

Add the training stats to the experiment results.

Experiment._add_fitness(result_key: str) → None

Add the fitness values to the solutions found.

Parameters:

result_key (str) – Result key.

Experiment._add_fitness_stats(result_key: str) → None

Perform some stats on the best solutions fitness.

Parameters:

result_key (str) – Result key.

Experiment._add_execution_metric(metric: str, value: Any) → None

Add an execution metric to the experiment results.

Parameters:

  • metric (str) – Name of the metric

  • value (object) – Value of the metric

Experiment._add_feature_metrics() → None

Perform stats about features frequency.

Experiment._do_test() → None

Perform the test step.

Test the solutions found by the trainer append their fitness to the best solutions dataframe.

Experiment._execute() → None

Execute the trainer method.

Experiment._is_reserved(name: str) → bool

Return True if the given hyperparameter name is reserved.

Parameters:

name – Hyperparameter name