Non-Iterative Optimizers

AKA Level 2 optimizers, are unified 3rd party solutions for random expressions. Look at this space:

from tune import Space, Grid, Rand

space = Space(a=Grid(1,2), b=Rand(0,1))
list(space)

[{'a': 1, 'b': Rand(low=0, high=1, q=None, log=False, include_high=True)},
 {'a': 2, 'b': Rand(low=0, high=1, q=None, log=False, include_high=True)}]

Grid is for level 1 optimization, all level 1 parameters will be converted to static values before execution. And level 2 parameters will be optimized during runtime using level 2 optimizers. So for the above example, if we have a Spark cluster and Hyperopt, then we can use Hyperot to search for the best b on each of the 2 configurations. And the 2 jobs are parallelized by Spark.

from tune import noniterative_objective, Trial

@noniterative_objective
def objective(a ,b) -> float:
    return a**2 + b**2

trial = Trial("dummy", params=list(space)[0])

Use Directly

Notice normally you don’t use them directly, instead you should use them through top level APIs. This is just to demo how they work.

Hyperopt

from tune_hyperopt import HyperoptLocalOptimizer

hyperopt_optimizer = HyperoptLocalOptimizer(max_iter=200, seed=0)
report = hyperopt_optimizer.run(objective, trial)

print(report.sort_metric, report)

1.0000000001665414 {'trial': {'trial_id': 'dummy', 'params': {'a': 1, 'b': 1.2905089873156781e-05}, 'metadata': {}, 'keys': []}, 'metric': 1.0000000001665414, 'params': {'a': 1, 'b': 1.2905089873156781e-05}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 1.0000000001665414, 'log_time': datetime.datetime(2021, 10, 6, 23, 30, 51, 970344)}

Optuna

from tune_optuna import OptunaLocalOptimizer
import optuna

optuna.logging.disable_default_handler()

optuna_optimizer = OptunaLocalOptimizer(max_iter=200)
report = optuna_optimizer.run(objective, trial)

print(report.sort_metric, report)

1.0000000003655019 {'trial': {'trial_id': 'dummy', 'params': {'a': 1, 'b': 1.9118105424729645e-05}, 'metadata': {}, 'keys': []}, 'metric': 1.0000000003655019, 'params': {'a': 1, 'b': 1.9118105424729645e-05}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 1.0000000003655019, 'log_time': datetime.datetime(2021, 10, 6, 23, 31, 26, 6566)}

As you see, we have unified the interfaces for using these frameworks. In addition, we also unified the semantic of the random expressions, so the random sampling behavior will be highly consistent on different 3rd party solutions.

Use Top Level API

In the following example, we directly use the entire space where you can mix grid search, random search and Bayesian Optimization.

from tune import suggest_for_noniterative_objective

report = suggest_for_noniterative_objective(
    objective, space, top_n=1,
    local_optimizer=hyperopt_optimizer
)[0]

print(report.sort_metric, report)

NativeExecutionEngine doesn't respect num_partitions ROWCOUNT

1.0000000001665414 {'trial': {'trial_id': '971ef4a5-71a9-5bf2-b2a4-f0f1acd02b78', 'params': {'a': 1, 'b': 1.2905089873156781e-05}, 'metadata': {}, 'keys': []}, 'metric': 1.0000000001665414, 'params': {'a': 1, 'b': 1.2905089873156781e-05}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 1.0000000001665414, 'log_time': datetime.datetime(2021, 10, 6, 23, 31, 43, 784128)}

You can also provide only random expressions in space, and use in the same way so it looks like a common case similar to the examples

report = suggest_for_noniterative_objective(
    objective, Space(a=Rand(-1,1), b=Rand(-100,100)), top_n=1,
    local_optimizer=optuna_optimizer
)[0]

print(report.sort_metric, report)

NativeExecutionEngine doesn't respect num_partitions ROWCOUNT

0.04085386621249434 {'trial': {'trial_id': '45179c01-7358-5546-8f41-d7c6f120523f', 'params': {'a': 0.01604913454189394, 'b': 0.20148521408021614}, 'metadata': {}, 'keys': []}, 'metric': 0.04085386621249434, 'params': {'a': 0.01604913454189394, 'b': 0.20148521408021614}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 0.04085386621249434, 'log_time': datetime.datetime(2021, 10, 6, 23, 34, 47, 379901)}

Factory Method

In the above example, if we don’t set local_optimizer, then the default level 2 optimizer will be used which can’t handle a configuration with random expressions.

So we have a nice way to make certain optimizer the default one.

from tune import NonIterativeObjectiveLocalOptimizer, TUNE_OBJECT_FACTORY

def to_optimizer(obj):
    if isinstance(obj, NonIterativeObjectiveLocalOptimizer):
        return obj
    if obj is None or "hyperopt"==obj:
        return HyperoptLocalOptimizer(max_iter=200, seed=0)
    if "optuna" == obj:
        return OptunaLocalOptimizer(max_iter=200)
    raise NotImplementedError

TUNE_OBJECT_FACTORY.set_noniterative_local_optimizer_converter(to_optimizer)

Now Hyperopt becomes the default level 2 optimizer, and you can switch to Optuna by specifying a string parameter

report = suggest_for_noniterative_objective(
    objective, Space(a=Rand(-1,1), b=Rand(-100,100)), top_n=1
)[0]  # using hyperopt

print(report.sort_metric, report)

report = suggest_for_noniterative_objective(
    objective, Space(a=Rand(-1,1), b=Rand(-100,100)), top_n=1,
    local_optimizer="optuna"
)[0]  # using hyperopt

print(report.sort_metric, report)

NativeExecutionEngine doesn't respect num_partitions ROWCOUNT
NativeExecutionEngine doesn't respect num_partitions ROWCOUNT

0.02788888054657708 {'trial': {'trial_id': '45179c01-7358-5546-8f41-d7c6f120523f', 'params': {'a': -0.13745463941867586, 'b': -0.09484251498594332}, 'metadata': {}, 'keys': []}, 'metric': 0.02788888054657708, 'params': {'a': -0.13745463941867586, 'b': -0.09484251498594332}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 0.02788888054657708, 'log_time': datetime.datetime(2021, 10, 6, 23, 35, 19, 961138)}
0.010490219126635992 {'trial': {'trial_id': '45179c01-7358-5546-8f41-d7c6f120523f', 'params': {'a': 0.06699961867542388, 'b': -0.07746786575079878}, 'metadata': {}, 'keys': []}, 'metric': 0.010490219126635992, 'params': {'a': 0.06699961867542388, 'b': -0.07746786575079878}, 'metadata': {}, 'cost': 1.0, 'rung': 0, 'sort_metric': 0.010490219126635992, 'log_time': datetime.datetime(2021, 10, 6, 23, 35, 21, 593974)}