Conditional Experiments

Image: The parameters you need in an algorithm may vary based on conditions like the number of layers in a deep neural net, or the algorithm you’re using in a model ensemble.

Definition

Broadly speaking, a “conditional experiment” can turn “on” and “off” parameters subject to their satisfaction of a set of “conditions”. Use cases include a deep learning model where a parameter representing nodes in layer 3 is ignored for models with 2 hidden layers.

Experiments Define Conditionals

Experiments may now include the conditionals field, which will define how to split up the parameter space. For example, conditionals may represent the number of hidden layers in a deep neural net, or the choice between two different neural network architectures.

Currently, we only support categorical (string) values for conditional values.

Examples

Here is a snippet defining conditionals in a Python Experiment create request

[
  dict(
    name="num_hidden_layers",
    values=["1", "2", "3"]
  )
]

Parameters Define Conditions

Conditions is a mapping of conditional name to valid conditional values that “satisfy” this parameter. Values may be provided as a string (when there is only one value) or an array of strings (for any number of values).

Examples

This parameter is satisfied when num_conv_layers takes on the value “2”, or “3”. We only want to turn “on” the filter size in layer 2 when our neural network has 2 or more hidden layers.

dict(
  name="layer_2_num_filters",
  conditions=dict(num_hidden_layers=["2", "3"]),
  type="int",
  bounds=dict(min=100, max=300),
)

Parameters without conditions will behave as normal -- they will always be “on”. The same applies for parameters with an empty mapping for conditions.

dict(
  name="batch_size",
  conditions=dict(),
  type="int",
  bounds=dict(min=10, max=100),
)

Assignments Include Conditionals, Exclude Unsatisfied Parameters

The assignments mapping of a Suggestion will always contain values for all conditionals. However, an assignment for a parameter will only be provided if that parameter’s conditions are satisfied. This means that if conditions are defined for a parameter it may or may not be present in the assignments.

We currently do not support manually providing assignments for Suggestion objects, Observation objects, or other, related, objects associated with experiments using conditionals.

See below for some examples defining conditionals, conditions, and sample assignment mappings.

Deep Learning Example: Convolutional Neural Network

Sample Experiment Create

conn.experiments().create(
  name="Deep Learning with Conditionals",
  project="sigopt-examples",
  conditionals=[
    dict(
      name="num_conv_layers",
      values=["1", "2", "3"],
    ),
  ],
  parameters=[
    dict(
      name="layer_1_num_filters",
      conditions=dict(num_conv_layers=["1", "2", "3"]),
      type="int",
      bounds=dict(min=100, max=300),
    ),
    dict(
      name="layer_1_filter_size",
      conditions=dict(num_conv_layers=["1", "2", "3"]),
      type="int",
      bounds=dict(min=2, max=10),
    ),
    dict(
      name="layer_2_num_filters",
      conditions=dict(num_conv_layers=["2", "3"]),
      type="int",
      bounds=dict(min=100, max=300),
    ),
    dict(
      name="layer_2_filter_size",
      conditions=dict(num_conv_layers=["2", "3"]),
      type="int",
      bounds=dict(min=4, max=8),
    ),
    dict(
      name="layer_3_num_filters",
      conditions=dict(num_conv_layers="3"),
      type="int",
      bounds=dict(min=100, max=300),
    ),
    dict(
      name="layer_3_filter_size",
      conditions=dict(num_conv_layers="3"),
      type="int",
      bounds=dict(min=3, max=5),
    ),
  ],
  metrics=[dict(name='Accuracy', objective='maximize')],
  parallel_bandwidth=1,
  observation_budget=200,
)

Sample Assignments

With “3” convolutional (conv) layers, all parameters are included

{
  "layer_1_filter_size": 5,
  "layer_1_num_filters": 150,
  "layer_2_filter_size": 5,
  "layer_2_num_filters": 237,
  "layer_3_filter_size": 4,
  "layer_3_num_filters": 262,
  "num_conv_layers": "3"
}

With “1” convolutional layer, some parameters are missing from the assignments

{
  "layer_1_filter_size": 9,
  "layer_1_num_filters": 117,
  "num_conv_layers": "1"
}

Limitations

• The experiment type must be offline (which is the default).
• Experiments can have at most 20 parameters.
• Multisolution experiments are not supported.
• Parameter Constraints can only be defined on unconditioned parameters.