How to do Bayesian hyper-parameter tuning on a blackbox modelOptimization of arbitrary functions on Cloud ML EngineLak LakshmananBlockedUnblockFollowFollowingJan 25Google Cloud ML Engine offers a hyper-parameter tuning service that uses Bayesian methods.
It is not restricted to TensorFlow or scikit-learn.
In fact, it is not even limited to machine learning.
You can use the Bayesian approach to tune pretty much any blackbox model.
To demonstrate, I’ll tune a traffic model to find the configuration of traffic that results in the minimum delay for a given flow.
I’ll demonstrate the hyper-parameter tuning on a traffic model.
Photo by Denys Nevozhai on Unsplash1.
Invoke the blackbox function to be optimizedFor simplicity, I’ll implement the “blackbox function” in Python itself (it’s adapted from an example used in several numerical software packages).
In real-life, you can call out to any executable, so you are not limited to Python.
Just make sure to get back a floating point value corresponding to the thing (delay, in this instance) that you want to minimize or maximize.
def compute_delay(flow, x12, x32): # traffic on other roads, assuming all traffic that arrives # at an intersection has to leave it x13 = flow – x12 x24 = x12 + x32 x34 = x13 – x32 # travel time on each road segment t12 = 5 + .
1 * x12 / (1.
– x12 / 10.
); t13 = x13 / (1.
– x13 / 30.
); t32 = 1.
+ x32 / (1.
– x32 / 10.
); t24 = x24 / (1.
– x24 / 30.
); t34 = 5 + .
1 * x34 / (1.
– x34 / 10.
); # total delay f = t12*x12 + t13*x13 + t32*x32 + t24*x24 + t34*x34; return(f);The model above computes the total traffic delay in a network of roads given the traffic moving between nodes and our desired total flow in vehicles/hour.
We need to find optimal values of x12 and x32 for some value of flow.
2.
Create command-line parametersMake sure to make each of the optimizable parameters a command-line parameter: parser = argparse.
ArgumentParser() parser.
add_argument('–x12', type=float, required=True) parser.
add_argument('–x32', type=float, required=True)3.
Call the hypertune packageThen, write out the metric that we want to optimize using the hypertune package: hpt = hypertune.
HyperTune() hpt.
report_hyperparameter_tuning_metric( hyperparameter_metric_tag='delay', metric_value=delay, global_step=1)4.
Write a hyperparam.
yaml fileWrite a configuration file.
The file should contain the type of machine you want to run the code on, the name of the metric that you want to optimize, and constraints for each of the parameters you want to optimize:trainingInput: scaleTier: CUSTOM masterType: standard # machine-type hyperparameters: goal: MINIMIZE maxTrials: 10 maxParallelTrials: 2 hyperparameterMetricTag: delay params: – parameterName: x12 type: DOUBLE minValue: 0 maxValue: 10 scaleType: UNIT_LINEAR_SCALE – parameterName: x32 type: DOUBLE minValue: 0 maxValue: 10 scaleType: UNIT_LINEAR_SCALE5.
Submit the hyperparameter tuning job to ML EngineYou can use the REST API or Python, but the simplest way is from the command-line using gcloud:#!/bin/bashBUCKET=<yourbuckethere>JOBNAME=hparam_$(date -u +%y%m%d_%H%M%S)REGION=us-central1gcloud ml-engine jobs submit training $JOBNAME –region=$REGION –module-name=trainer.
flow –package-path=$(pwd)/trainer –job-dir=gs://$BUCKET/hparam/ –config=hyperparam.
yaml — –flow=56.
View results on GCP ConsoleWait for the job to finish and view the results on the GCP console.
I got: { "trialId": "8", "hyperparameters": { "x12": "0.
70135653339854276", "x32": "0.
018214831148084532" }, "finalMetric": { "trainingStep": "1", "objectiveValue": 50.
2831830645 } },The actual optimum for this road configuration is a delay of 40.
3; we get relatively close within 10 trials even though the search space was large and the problem is non-linear.
Next steps:Try it out.
The full code is on GitHub.
Read the ML Engine docs on hyperparameter tuning.
Read the ML Engine docs on machine types available.
.