culebra.tools.evaluation.Experiment class

class Experiment(trainer: Trainer, decision_manager: DecisionManager, test_fitness_func: FitnessFunction | None = None, results_base_filename: str | None = None, hyperparameters: dict | None = None)

Bases: Evaluation

Set a trainer evaluation.

Parameters:

  • trainer (Trainer) – The trainer

  • decision_manager (DecisionManager) – A decision manager to select the best solution from the set of best solutions found by the trainer

  • test_fitness_func (FitnessFunction) – The fitness function used to test. If omitted, the training fitness function will be used. Defaults to None

  • results_base_filename (str) – The base filename to save the results. If omitted, _default_results_base_filename is used. Defaults to None

  • hyperparameters (dict) – Hyperparameter values used in this evaluation, optional

Raises:

  • TypeError – If any of the parameters has an incorrect type

  • ValueError – If any of the parameters has an incorrect value

Class attributes

Experiment.feature_metric_funcs = {'Rank': <function Metrics.rank>, 'Relevance': <function Metrics.relevance>}

Metrics calculated for the features in the set of solutions.

Experiment.stats_funcs = {'Avg': <function mean>, 'Max': <function max>, 'Min': <function min>, 'Std': <function std>}

Statistics calculated for the solutions.

Experiment.ResultsKeys = <enum 'ResultsKeys'>
Experiment.ResultsLabels = <enum 'ResultsLabels'>

Class methods

classmethod Experiment.from_config(config_script_filename: str | None = None) → Evaluation

Generate a new evaluation from a configuration file.

Parameters:

config_script_filename (str) – Path to the configuration file. If omitted, DEFAULT_CONFIG_SCRIPT_FILENAME is used. Defaults to None

Raises:

RuntimeError – If config_script_filename is an invalid file path or an invalid configuration file

classmethod Experiment.generate_run_script(config_filename: str | None = None, run_script_filename: str | None = None) → None

Generate a script to run an evaluation.

The parameters for the evaluation are taken from a configuration file.

Parameters:
Raises:

  • TypeError – If config_filename or run_script_filename are not a valid filename

  • ValueError – If the extensions of config_filename or run_script_filename are not valid

classmethod Experiment.load(filename: str) → Base

Load a serialized object from a file.

Parameters:

filename (str) – The file name.

Returns:

The loaded object

Raises:

Properties

property Experiment.best_cooperators: list[list[Solution]] | None

Best cooperators found by the trainer.

Return type:

list[list[Solution]]

property Experiment.best_solutions: tuple[HallOfFame] | None

Best solutions found by the trainer.

Returns:

One Hall of Fame for each species

Return type:

tuple[HallOfFame]

property Experiment.decision_manager: DecisionManager

Return the decicion manager.

Return type:

DecisionManager

Setter:

Set a new decision manager

Parameters:

dm (DecisionManager) – The new decision manager

Raises:

TypeError – If dm is not a valid decision manager

property Experiment.excel_results_filename: str

Filename used to save the results in Excel format.

Return type:

str

property Experiment.hyperparameters: dict | None

Hyperparameter values used for the evaluation.

Return type:

dict

Setter:

Set the hyperparameter values used for the evaluation

Parameters:

values (dict) – Hyperparameter values used in this evaluation

Raises:

  • TypeError – If values is not a dictionary

  • ValueError – If the keys in values are not strings

  • ValueError – If any key in values is reserved

property Experiment.results: Results | None

Results obtained.

Return type:

Results

property Experiment.results_base_filename: str | None

Results base filename.

Return type:

str

Setter:

Set a new results base filename.

Parameters:

filename (str) – New results base filename. If set to None, _default_results_base_filename is used

Raises:

TypeError – If filename is not a valid file name

property Experiment.serialized_results_filename: str

Filename used to save the serialized results.

Return type:

str

property Experiment.test_fitness_func: FitnessFunction

Test fitness function.

Return type:

FitnessFunction

Setter:

Set a new test fitness function.

Parameters:

func (FitnessFunction) – New test fitness function. If set to None, the training fitness function will also be used for testing

Raises:

TypeError – If func is not a valid fitness function

property Experiment.trainer: Trainer

Return the trainer.

Return type:

Trainer

Setter:

Set a new trainer

Parameters:

value (Trainer) – The new trainer

Raises:

TypeError – If trainer is not a valid trainer

Private properties

property Experiment._default_results_base_filename: str

Default base name for results files.

Returns:

DEFAULT_RESULTS_BASE_FILENAME

Return type:

str

property Experiment._default_test_fitness_func: FitnessFunction

Default test fitness function.

Returns:

The trainer’s training function

Return type:

FitnessFunction

Methods

Experiment.dump(filename: str) → None

Serialize this object and save it to a file.

Parameters:

filename (str) – The file name.

Raises:
Experiment.reset() → None

Reset the experiment.

Experiment.run() → None

Execute the evaluation and save the results.

Private methods

Experiment._add_best(best: Sequence[Solution], fitness_func: FitnessFunction, results_key: str) → None

Add the best solution to the experiment results.

For cooperative approaches, the solution is evaluated only with the species that compose the best solution, without any other cooperator

Parameters:

  • best (Sequence[Solution]) – The best solution (one per species)

  • fitness_func (FitnessFunction) – Fitness fuction to evaluate the best solution

  • results_key (str) – Results key

Experiment._add_execution_metric(metric: str, value: Any) → None

Add an execution metric to the experiment results.

Parameters:

  • metric (str) – Name of the metric

  • value (object) – Value of the metric

Experiment._add_feature_metrics() → None

Perform stats about features frequency.

Experiment._add_fitness(results_key: str) → None

Add the fitness values to the solutions found.

Parameters:

results_key (str) – Results key.

Experiment._add_fitness_stats(results_key: str) → None

Perform some stats on the best solutions fitness.

Parameters:

results_key (str) – Results key.

Experiment._add_training_stats() → None

Add the training stats to the experiment results.

Experiment._do_test() → None

Perform the test step.

Test the solutions found by the trainer append their fitness to the best solutions dataframe.

Experiment._do_training() → None

Perform the training step.

Train the trainer and get the best solutions and the training stats.

Experiment._execute() → None

Execute the trainer.

Experiment._get_repr_properties() → dict[str, object]

Return the subset of properties used for __repr__.

Filters and evaluates all class-level @property attributes, returning only those intended for representation purposes. Private properties (names starting with _) are excluded.

Returns:

Mapping of property names to their corresponding values.

Return type:

dict[str, object]