By default, SigOpt will aim to optimize your metric or search for the efficient frontier when more than one metric is being optimized. However, it can also be helpful to track metrics even when they are not being optimized. For example, some metrics may be very strongly correlated so you want to focus optimization on one. Another case is when investigating a new field while still needing to report an industry–standard one. The goal of Metric Strategy is to enable you to more easily track these metrics. Now you can store up to 50 metrics on an experiment while focusing optimization on one or two.
Metric Constraints can be used when you have some guardrail metrics in your system. Metric Constraints allow SigOpt to factor in the values of additional metrics during the optimization process.
Stored metrics allow you to better track all the relevant information you uncover in your experiment and enable more informative, more flexible graphs on the Analysis Page.
Metric Strategy works with the Metric Objective field to help make analysis more intuitive. Whether the strategy is set to
constraint, the objective can still be set to
minimize. The objective defines the direction of improvement for the metric, whether or not it gets optimized.
Example: Storing Infrastructure Measurements
Neural networks can perform well in very complicated situations but may need to balance model complexity with inference time, ensuring viable solutions can be reached within business constraints for prediction time. While accuracy and inference time need to be optimized, it may be beneficial to track the memory usage, to help monitor the expected load on the system and how that impacts the current infrastructure.
Now, we can analyze whether memory usage is correlated, positively or negatively, with our optimized metrics, giving us a more natural way to explore this interaction.
Defining the Metric Strategy
To assign a strategy to one of your metrics, specify the
strategy field in the desired Metric object when creating your experiment. You can specify
optimize, which is the default. Set a metric strategy to
store when you only want to track the metric values. Set a metric strategy to
constraint when you want the metric to be a guardrail metric. See the Metric Constraints documentation for more information on how to use the feature effectively.
Creating the Experiment
Below, we create a new experiment, using the above example of storing memory usage while optimizing accuracy and inference time.
from sigopt import Connection conn = Connection(client_token="SIGOPT_API_TOKEN") conn.set_api_url("https://api.staging.sigopt.com") experiment = conn.experiments().create( name="Neural network", parameters=[ dict( name="learning_rate", bounds=dict( min=0.0001, max=1 ), transformation="log", type="double" ), dict( name="nodes_per_layer", bounds=dict( min=5, max=20 ), type="int" ) ], metrics=[ dict( name="inference_time_milliseconds", objective="minimize", strategy="optimize" ), dict( name="validation_accuracy", objective="maximize", strategy="optimize" ), dict( name="memory_usage", objective="minimize", strategy="store" ) ], observation_budget=65, parallel_bandwidth=2, project="sigopt-examples", type="offline" ) print("Created experiment: https://app.staging.sigopt.com/experiment/" + experiment.id)
- Metrics whose strategy is set to
storewill not be optimized in any way.
strategycannot be modified after Experiment Create.
- There can be up to 50 metrics of any kind for an experiment. See the Metric Constraints documentation for the limit on the number of constraint metrics.
- All stored metrics must be reported for successful Observations.
- No stored metrics can be reported for Failed Observations.
thresholdcan only be applied to