Optimizing Hyperparameters¶
Hyperparameter optimization (HPO) is an indispensable step to make it work in real world.
Getting Started¶
Create a file that defines the objective function to be optimized:
import argparse
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("out_filename", type=str)
parser.add_argument("--x1", type=float)
parser.add_argument("--x2", type=float)
args = parser.parse_args()
y = (args.x1**2) - (4.0 * args.x1) + (args.x2**2) - args.x2 - (args.x1 * args.x2)
with open(args.out_filename, "w") as f:
f.write(f"{y}")
if __name__ == "__main__":
main()
Run the following command:
python -m aiaccel.hpo.apps.optimize params.x1="[0,2]" params.x2="[0,2]" n_trials=30 -- python ./objective.py --x1={x1} --x2={x2} {out_filename}
The parameters are set as params.x1=”[0,2]” and params.x2=”[0,2]” and the number of trials is set to n_trials=30. Specify the command to execute the objective function after ‘–’. In the arguments, include the parameters and ‘{out_filename}’. In objective.py, output the result of the objective function to ‘{out_filename}’ in JSON format.
Basic Usage¶
Configuration¶
Basic configuration example:
study:
_target_: optuna.create_study
direction: minimize
storage: # Set this item if results need to be stored in DB
_target_: optuna.storages.RDBStorage
url: sqlite:///aiaccel_storage.db
engine_kwargs:
connect_args:
timeout: 30
study_name: my_study # Set this item if results need to be stored in DB
sampler:
_target_: optuna.samplers.TPESampler
seed: 0
command: ["python", "./objective.py", "--x1={x1}", "--x2={x2}", "{out_filename}"]
params:
x1: [0, 1]
x2: [0, 1]
n_trials: 30
n_max_jobs: 1
Study Configuration¶
The study configuration controls the overall behavior of the optimization process:
study:
_target_: optuna.create_study # default
direction: minimize # 'minimize' or 'maximize' depending on your objective
study_name: my_study # Name of the study (optional)
storage: # This item is not required. This item is not required if there is no need to record it in the file.
_target_: optuna.storages.RDBStorage
url: sqlite:///example.db
engine_kwargs:
connect_args:
timeout: 30
load_if_exists: true # Load existing study if it exists
sampler:
_target_: optuna.samplers.TPESampler
seed: 42
Sampler Configuration¶
The sampler determines the algorithm used to search the hyperparameter space:
study:
_target_: optuna.create_study
direction: minimize
sampler:
_target_: optuna.samplers.TPESampler # Tree-structured Parzen Estimator (default)
# TPE-specific parameter
seed: 42 # For reproducibility
n_startup_trials: 10 # Number of random trials before using TPE
Available samplers include:
TPESampler: Efficient Bayesian optimization approach (recommended for most cases)
RandomSampler: Simple random search (useful as baseline)
CmaEsSampler: Covariance Matrix Adaptation Evolution Strategy (good for continuous parameters)
GridSampler: Exhaustive grid search (for small parameter spaces)
NSGAIISampler: For multi-objective optimization
NelderMeadSampler: Nelder-Mead optimization
Parameters Configuration¶
The parameters section defines the hyperparameter search space using Optuna’s suggestion methods wrapped by aiaccel:
params:
_convert_: partial
_target_: aiaccel.hpo.optuna.hparams_manager.HparamsManager # default
# Float parameter example
x1:
_target_: aiaccel.hpo.optuna.hparams.Float
low: 0.0
high: 1.0
log: false
# Another float parameter
x2:
_target_: aiaccel.hpo.optuna.hparams.Float
low: 0.0
high: 1.0
log: false
# Shorthand for float parameters
x3: [0, 1]
Parameter Types¶
aiaccel supports multiple parameter types through different suggestion wrappers:
Float: For continuous parameters
learning_rate:
_target_: aiaccel.hpo.optuna.hparams.Float
name: learning_rate
low: 0.0001
high: 0.1
log: true
Int: For integer parameters
num_layers:
_target_: aiaccel.hpo.optuna.hparams.Int
name: num_layers
low: 1
high: 10
Categorical: For categorical parameters
optimizer:
_target_: aiaccel.hpo.optuna.hparams.Categorical
name: optimizer
choices: ['adam', 'sgd', 'rmsprop']
DiscreteUniform: For discrete uniform parameters
batch_size:
_target_: aiaccel.hpo.optuna.hparams.Float
name: batch_size
low: 32
high: 256
step: 32
LogUniform: For log-uniform parameters
learning_rate:
_target_: aiaccel.hpo.optuna.hparams.Float
name: learning_rate
low: 0.0001
high: 0.1
log: true
LogInt: For log-int parameters
num_layers:
_target_: aiaccel.hpo.optuna.hparams.Int
name: num_layers
low: 1
high: 10
log: true
Command¶
Command to run the objective function. The objective function is the main function to be optimized:
command: ["python", "./objective.py", "--x1={x1}", "--x2={x2}", "{out_filename}"]
Other Configuration Options¶
n_trials: Number of trials to run
n_max_jobs: Maximum number of parallel jobs
n_trials: 100
n_max_jobs: 1 # default : 1
Usage Examples¶
Here are some common usage patterns:
Start a new study with configuration file:
aiaccel-hpo optimize --config=config.yaml
Start a new study with cli:
python -m aiaccel.hpo.apps.optimize working_directory=./cli/ params.x1="[0,2]" params.x2="[0,2]" study.sampler._target_=optuna.samplers.TPESampler study.sampler.seed=0 n_trials=30 n_max_jobs=1 -- python ./objective.py --x1={x1} --x2={x2} {out_filename}
Start a new study with configuration file and cli:
python -m aiaccel.hpo.apps.optimize --config config.yaml params.x1="[0,2]" params.x2="[0,2]" --
HPO Using NelderMeadSampler¶
Basic Usage¶
Basic optimization example using NelderMeadSampler:
Search Space¶
NelderMeadSampler requires a search space as an argument.
search_space = {
"x": (-10.0, 10.0),
"y": (-10.0, 10.0),
}
Objective Function¶
Set the Objective Function in the same way as in regular Optuna. The optimization target is the benchmark function Sphere.
def sphere(trial: optuna.trial.Trial) -> float:
params = []
for name, distribution in search_space.items():
params.append(trial.suggest_float(name, *distribution))
return float(np.sum(np.asarray(params) ** 2))
Execute Optimization¶
Specify NelderMeadSampler as the sampler and execute the optimization.
study = optuna.create_study(
sampler=NelderMeadSampler(search_space=search_space, seed=42)
)
study.optimize(func=sphere, n_trials=100)
Full code is examples/hpo/samplers/example.py
Pallarel Optimization¶
Example pallarel optimization:
study = optuna.create_study(
sampler=NelderMeadSampler(search_space=search_space, seed=42, block=True)
)
study.optimize(func=sphere, n_trials=100, n_jobs=3)
Parallel execution is enabled by setting the NelderMeadSampler argument block=True and the study.optimize argument n_jobs>2. By enabling parallel execution, the initial point calculation and the computation during shrinking can be parallelized, leading to faster execution compared to serial execution.
Full code is examples/hpo/samplers/example_parallel.py
optuna.study.enqueue_trial¶
Example using optuna.study.enqueue_trial:
study = optuna.create_study(
sampler=NelderMeadSampler(search_space=search_space, seed=42)
)
study.enqueue_trial({"x": 1.0, "y": 1.0})
study.enqueue_trial({"x": 1.0, "y": 2.0})
study.enqueue_trial({"x": 2.0, "y": 1.0})
study.optimize(func=sphere, n_trials=100)
Utilizing the ask-tell interface, random parameters are explored using enqueue_trial when NelderMeadSampler fails to output parameters.
Full code is examples/hpo/samplers/example_enqueue.py
Sub Sampler¶
Example using sub_sampler as optuna.samplers.TPESampler:
study = optuna.create_study(
sampler=NelderMeadSampler(
search_space=search_space,
seed=42,
sub_sampler=optuna.samplers.TPESampler(seed=42),
)
)
study.optimize(func=sphere, n_trials=100, n_jobs=3)
When sub_sampler=optuna.samplers.TPESampler is set as an argument for NelderMeadSampler, TPESampler is used for exploration when NelderMeadSampler fails to output parameters. When using the sub_sampler function, the argument block=False must be set even if it is parallel. (Parallel execution is possible even with block=False.)
Full code is examples/hpo/samplers/example_sub_sampler.py