aiaccel.hpo.optuna.samplers package#

Submodules#

aiaccel.hpo.optuna.samplers.nelder_mead_sampler module#

exception aiaccel.hpo.optuna.samplers.nelder_mead_sampler.NelderMeadEmptyError[source]#

Bases: Exception

class aiaccel.hpo.optuna.samplers.nelder_mead_sampler.NelderMeadSampler(search_space: dict[str, tuple[int | float, int | float]], seed: int | None = None, rng: RandomState | None = None, coeff: NelderMeadCoefficient | None = None, block: bool = False, sub_sampler: BaseSampler | None = None)[source]#

Bases: BaseSampler

Sampler using the NelderMead algorithm

Only the initial point and shrink (number of parameters - 1) can be calculated in parallel. Others are basically series calculations. (Even if set by e.g. optuna.optimize(n_jobs=2), the calculation is performed in series except in initial and shrink.) If parallelisation is enabled, set block = True.

When using optuna.enqueue_trial(), the enqueued parameters are calculated separately from the parameters determined by NelderMeadSampler and are taken into NelderMead if a good result is obtained. (Simplex is reconstituted). The enqueued parameters are calculated in parallel with the parameters determined by NelderMead if parallelisation is enabled.

Example

An example of a single-objective optimization is as follows:

import optuna
from aiaccel.hpo.optuna.samplers.nelder_mead_sampler import NelderMeadSampler


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return x**2


search_space = {"x": {"low": -10, "high": 10}}
study = optuna.create_study(sampler=NelderMeadSampler(search_space=search_space, seed=42))
study.optimize(objective, n_trials=10)
Parameters:

  • search_space – dict[str, tuple[float, float]] Parameter names and corresponding lower and upper limits. Must be set separately from suggest_uniform (as the parameters must be determined at the time of before_trial).

  • seed – int | None = None Random seed used for initial point calculation.

  • rng – np.random.RandomState | None = None RandomState used for initial point calculation. If specified with seed, rng takes precedence.

  • coeff – NelderMeadCoefficient | None = None Parameters used in NelderMead Algorism.

  • block – bool = False Indicates whether the queue used internally is blocked or not. If parallelisation by optuna.optimize is enabled, it must be set with block = True

  • sub_sampler – optuna.samplers.BaseSampler | None = None Sampler to output parameters when NelderMead cannot output parameters. Mainly intended for use on free computation nodes in parallel. If the sub_sampler function is enabled, it must be set with block = False.

nm#

NelderMeadAlgorism Instance of a class that manages the NelderMead algorithm.

after_trial(study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None) → None[source]#

Trial post-processing.

Pass the parameter-result pairs to the NelderMead algorithm. If trial.user_attr[“sub_trial”] = True, execute sub_sampler.after_trial().

Parameters:

  • study – Study Target study object.

  • trial – FrozenTrial Target trial object. Take a copy before modifying this object.

  • state – TrialState Resulting trial state.

  • values – Sequence[float] | None Resulting trial values. Guaranteed to not be None if trial succeeded.

Returns:

None

before_trial(study: Study, trial: FrozenTrial) → None[source]#

Trial pre-processing.

Preprocessing of the trial

This determines the parameters for NelderMead. The determined parameters are stored in trial.user_attr[“params”]. If the NelderMead parameters cannot be output and sub_sampler is None, a NelderMeadEmptyError is raised. If sub_sampler is specified, sub_sampler.before_trial() is executed, and trial.user_attr[“sub_trial”] is set to True.

Parameters:

  • study – Study Target study object.

  • trial – FrozenTrial Target trial object.

Returns:

None

infer_relative_search_space(study: Study, trial: FrozenTrial) → dict[str, BaseDistribution][source]#

Infer the search space that will be used by relative sampling in the target trial.

This method is called right before sample_relative() method, and the search space returned by this method is passed to it. The parameters not contained in the search space will be sampled by using sample_independent() method.

Parameters:

  • study – Target study object.

  • trial – Target trial object. Take a copy before modifying this object.

Returns:

A dictionary containing the parameter names and parameter’s distributions.

See also

Please refer to intersection_search_space() as an implementation of infer_relative_search_space().

sample_independent(study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution) → Any[source]#

Sample a parameter

For NelderMeadSampler, since the parameters are already determined in before_trial, return the corresponding parameters in trial.user_attr[“params”]. If trial.user_attr[“sub_trial”] = True, execute sub_sampler.sample_independent() and return its parameters.

Parameters:

  • study – Study Target study object.

  • trial – FrozenTrial Target trial object. Take a copy before modifying this object.

  • param_name – str Name of the sampled parameter.

  • param_distribution – BaseDistribution Distribution object that specifies a prior and/or scale of the sampling algorithm.

Returns:

Any

A parameter value.

sample_relative(study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution]) → dict[str, Any][source]#

Sample parameters in a given search space.

This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use relationship between parameters such as Gaussian Process and CMA-ES.

Note

The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers’ perspective.

Parameters:

  • study – Target study object.

  • trial – Target trial object. Take a copy before modifying this object.

  • search_space – The search space returned by infer_relative_search_space().

Returns:

A dictionary containing the parameter names and the values.

Module contents#

class aiaccel.hpo.optuna.samplers.NelderMeadSampler(search_space: dict[str, tuple[int | float, int | float]], seed: int | None = None, rng: RandomState | None = None, coeff: NelderMeadCoefficient | None = None, block: bool = False, sub_sampler: BaseSampler | None = None)[source]#

Bases: BaseSampler

Sampler using the NelderMead algorithm

Only the initial point and shrink (number of parameters - 1) can be calculated in parallel. Others are basically series calculations. (Even if set by e.g. optuna.optimize(n_jobs=2), the calculation is performed in series except in initial and shrink.) If parallelisation is enabled, set block = True.

When using optuna.enqueue_trial(), the enqueued parameters are calculated separately from the parameters determined by NelderMeadSampler and are taken into NelderMead if a good result is obtained. (Simplex is reconstituted). The enqueued parameters are calculated in parallel with the parameters determined by NelderMead if parallelisation is enabled.

Example

An example of a single-objective optimization is as follows:

import optuna
from aiaccel.hpo.optuna.samplers.nelder_mead_sampler import NelderMeadSampler


def objective(trial):
    x = trial.suggest_float("x", -10, 10)
    return x**2


search_space = {"x": {"low": -10, "high": 10}}
study = optuna.create_study(sampler=NelderMeadSampler(search_space=search_space, seed=42))
study.optimize(objective, n_trials=10)
Parameters:

  • search_space – dict[str, tuple[float, float]] Parameter names and corresponding lower and upper limits. Must be set separately from suggest_uniform (as the parameters must be determined at the time of before_trial).

  • seed – int | None = None Random seed used for initial point calculation.

  • rng – np.random.RandomState | None = None RandomState used for initial point calculation. If specified with seed, rng takes precedence.

  • coeff – NelderMeadCoefficient | None = None Parameters used in NelderMead Algorism.

  • block – bool = False Indicates whether the queue used internally is blocked or not. If parallelisation by optuna.optimize is enabled, it must be set with block = True

  • sub_sampler – optuna.samplers.BaseSampler | None = None Sampler to output parameters when NelderMead cannot output parameters. Mainly intended for use on free computation nodes in parallel. If the sub_sampler function is enabled, it must be set with block = False.

nm#

NelderMeadAlgorism Instance of a class that manages the NelderMead algorithm.

after_trial(study: Study, trial: FrozenTrial, state: TrialState, values: Sequence[float] | None) → None[source]#

Trial post-processing.

Pass the parameter-result pairs to the NelderMead algorithm. If trial.user_attr[“sub_trial”] = True, execute sub_sampler.after_trial().

Parameters:

  • study – Study Target study object.

  • trial – FrozenTrial Target trial object. Take a copy before modifying this object.

  • state – TrialState Resulting trial state.

  • values – Sequence[float] | None Resulting trial values. Guaranteed to not be None if trial succeeded.

Returns:

None

before_trial(study: Study, trial: FrozenTrial) → None[source]#

Trial pre-processing.

Preprocessing of the trial

This determines the parameters for NelderMead. The determined parameters are stored in trial.user_attr[“params”]. If the NelderMead parameters cannot be output and sub_sampler is None, a NelderMeadEmptyError is raised. If sub_sampler is specified, sub_sampler.before_trial() is executed, and trial.user_attr[“sub_trial”] is set to True.

Parameters:

  • study – Study Target study object.

  • trial – FrozenTrial Target trial object.

Returns:

None

infer_relative_search_space(study: Study, trial: FrozenTrial) → dict[str, BaseDistribution][source]#

Infer the search space that will be used by relative sampling in the target trial.

This method is called right before sample_relative() method, and the search space returned by this method is passed to it. The parameters not contained in the search space will be sampled by using sample_independent() method.

Parameters:

  • study – Target study object.

  • trial – Target trial object. Take a copy before modifying this object.

Returns:

A dictionary containing the parameter names and parameter’s distributions.

See also

Please refer to intersection_search_space() as an implementation of infer_relative_search_space().

sample_independent(study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution) → Any[source]#

Sample a parameter

For NelderMeadSampler, since the parameters are already determined in before_trial, return the corresponding parameters in trial.user_attr[“params”]. If trial.user_attr[“sub_trial”] = True, execute sub_sampler.sample_independent() and return its parameters.

Parameters:

  • study – Study Target study object.

  • trial – FrozenTrial Target trial object. Take a copy before modifying this object.

  • param_name – str Name of the sampled parameter.

  • param_distribution – BaseDistribution Distribution object that specifies a prior and/or scale of the sampling algorithm.

Returns:

Any

A parameter value.

sample_relative(study: Study, trial: FrozenTrial, search_space: dict[str, BaseDistribution]) → dict[str, Any][source]#

Sample parameters in a given search space.

This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use relationship between parameters such as Gaussian Process and CMA-ES.

Note

The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers’ perspective.

Parameters:

  • study – Target study object.

  • trial – Target trial object. Take a copy before modifying this object.

  • search_space – The search space returned by infer_relative_search_space().

Returns:

A dictionary containing the parameter names and the values.