Tune Hyperparameters

SigOpt is an ensemble of Bayesian Optimization techniques, which finds a natural use case in hyperparameter tuning. Here, we assume you are already familiar with our optimization loop, and we provide some tips for implementing SigOpt to tune the hyperparameters of your models, regardless of model type.

Picking Your Objective Metric

Before using SigOpt to tune the hyperparameters of a machine learning model, you must first pick the objective metric you wish to maximize or minimize. This is a crucially important step, as an improperly formed objective can lead to improperly formed models! Here are some suggestions for a good objective to use for several common model types:

Classification models (logistic regression, gradient boosting, neural nets): Mean accuracy of K-fold cross validation. Learn more.
Regression models: Root MeanSquared Error of K-fold cross validation.

To learn more, visit Scikit-learn’s cross-validation tutorial.

Evaluating Your Model

Every time you receive a new Suggestion in the optimization loop, train a new model using the suggested set of hyperparameters. Evaluate the objective metric for this new model, and pass that value back to SigOpt’s optimization loop:

def evaluate_metric(assignments, dataset):
  # Make a model using the new hyperparameters
  model = make_model(assignments)

  # Score the new model on the dataset
  return score_model(model, dataset)

# optimization loop
for _ in range(N):
  suggestion = conn.experiments(experiment.id).suggestions().create()
  value = evaluate_metric(suggestion.assignments)
  conn.experiments(experiment.id).observations().create(
    suggestion=suggestion.id,
    value=value,
  )

Avoid Overfitting

Overfitting occurs when a model is over-optimized for your training data, and does not perform well when presented with new data. Avoid this common issue by employing k-fold cross validation every time you evaluate a new set of hyperparameters for your model. For a walkthrough example, see our notebook on tuning a text classifier.

Training the Final Model

When you're done, you can use the API to get the best hyperparameters you've seen. Create a new model using these hyperparameters, and train it with all of the available data:

best_assignments = (
  conn.experiments(experiment_id).best_assignments().fetch().data[0]
)
model = train_model(best_assignments, dataset)

Scikit-learn Integration