Adaptive Experimentation with Meta’s Ax: A Practical Coding Guide

In this tutorial, we explore adaptive experimentation using Meta’s Ax with the modern Client API. We walk through a complete workflow where we tune a RandomForest model on a synthetic classification dataset, balancing predictive accuracy against model footprint. We start by defining a mixed search space with integer, float, log-scaled, and categorical parameters, then use Ax’s ask-tell optimization loop to run constrained Bayesian optimization, multi-objective optimization, and parameter-constrained experimentation. Along the way, we visualize convergence, inspect the Pareto frontier, leverage Ax’s built-in analysis tools, and persist the experiment for future reuse.

Setting Up the Environment

We begin by preparing the Colab environment and installing the required packages for Ax and scikit-learn. We import core libraries for optimization, machine learning, plotting, logging, and reproducibility. We also configure warnings and Ax logging to keep the notebook output clean and focused on the experimental results.

import importlib, subprocess, sys
def _ensure(module, pip_name=None):
    try:
        importlib.import_module(module)
    except ImportError:
        print(f"Installing {pip_name or module} ...")
        subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", pip_name or module])
_ensure("ax", "ax-platform")
_ensure("sklearn", "scikit-learn")
import logging, warnings, time
import numpy as np
import matplotlib.pyplot as plt
warnings.filterwarnings("ignore")
logging.getLogger("ax").setLevel(logging.WARNING)
from ax.api.client import Client
from ax.api.configs import RangeParameterConfig, ChoiceParameterConfig
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score
np.random.seed(0)

Defining the Search Space and Experiment

With the environment ready, we define a mixed search space that includes integer, float, log-scaled, and categorical parameters. This diversity allows Ax to efficiently explore different types of hyperparameters, such as the number of estimators (integer), maximum depth (integer), minimum samples split (float), and criterion (categorical). We then set up the Client API to manage the experiment.

# Define parameter configurations
parameters = [
    RangeParameterConfig(name="n_estimators", lower=50, upper=500, parameter_type="int"),
    RangeParameterConfig(name="max_depth", lower=5, upper=50, parameter_type="int"),
    RangeParameterConfig(name="min_samples_split", lower=2, upper=10, parameter_type="float"),
    RangeParameterConfig(name="max_features", lower=0.1, upper=1.0, parameter_type="float", log_scale=True),
    ChoiceParameterConfig(name="criterion", values=["gini", "entropy"]),
]

# Initialize the Ax client
client = Client()
client.configure_experiment(
    name="random_forest_tuning",
    parameters=parameters,
    objective_name="accuracy",
    minimize=False,
    parameter_constraints=["max_depth >= min_samples_split"],  # Parameter constraint example
)

Running the Optimization Loop

We use Ax’s ask-tell loop to run Bayesian optimization. At each iteration, we ask Ax for the next set of parameters, train a RandomForest model using cross-validation, and tell Ax the resulting accuracy. This iterative approach adapts based on previous results, focusing on promising regions of the search space.

# Define the evaluation function
def evaluate(parameters):
    model = RandomForestClassifier(**parameters, random_state=0)
    cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
    scores = cross_val_score(model, X, y, cv=cv, scoring="accuracy")
    return {"accuracy": scores.mean()}

# Load synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=15, random_state=0)

# Run optimization
for i in range(20):  # 20 iterations for demonstration
    parameters, trial_index = client.ask()
    result = evaluate(parameters)
    client.tell(trial_index, result)
    print(f"Iteration {i+1}: {parameters} -> Accuracy: {result['accuracy']:.4f}")

Multi-Objective Optimization

Ax also supports multi-objective optimization, allowing us to balance accuracy against model footprint (e.g., the number of estimators). We can configure a second objective and inspect the Pareto frontier to identify trade-offs.

# Add a second objective (model size as a proxy)
client.configure_experiment(
    name="random_forest_multiobjective",
    parameters=parameters,
    objectives=["accuracy", "n_estimators"],  # Second objective to minimize
    minimize=[False, True],  # Accuracy maximize, n_estimators minimize
)

# Re-run optimization loop with both objectives
for i in range(20):
    parameters, trial_index = client.ask()
    metrics = evaluate(parameters)
    metrics["n_estimators"] = parameters["n_estimators"]  # Simulate model size
    client.tell(trial_index, metrics)

Visualizing and Analyzing Results

Ax provides built-in tools to visualize convergence and the Pareto frontier. We can plot the best accuracy over iterations and inspect the trade-off between accuracy and model size.

# Plot convergence
from ax.api.utils import get_best_parameters
best_parameters, best_values = get_best_parameters(client.experiment)
print("Best parameters:", best_parameters)
print("Best values:", best_values)

# Generate plots
from ax.api.utils import get_pareto_frontier_plot
pareto_plot = get_pareto_frontier_plot(client.experiment, objective_names=["accuracy", "n_estimators"])
pareto_plot.render()

Persisting the Experiment

To save the experiment for future reuse, we can store it in a JSON file or a database. Ax’s persistence capabilities ensure that we can resume optimization or analyze results later without losing progress.

from ax.storage.json_store.save import save_experiment
save_experiment(client.experiment, "random_forest_exp.json")
print("Experiment saved.")

Conclusion

This guide demonstrates how to use Meta’s Ax Client API for adaptive experimentation, covering constrained Bayesian optimization, multi-objective optimization, and parameter constraints. By leveraging Ax’s ask-tell loop, we efficiently explored a complex search space and balanced competing objectives. The built-in visualization and persistence tools make it a robust framework for real-world machine learning workflows. As of 2026, Ax continues to evolve, offering even more sophisticated algorithms and integrations, making it a go-to tool for practitioners seeking efficient hyperparameter tuning.

via MarkTechPost