Tuning Module

Hyperparameter tuning utilities for MCGrad models.

This module provides utilities for automatic hyperparameter optimization of MCGrad calibrators using the Ax experimentation platform.

mcgrad.tuning.tune_mcgrad_params(model, df_train, prediction_column_name, label_column_name, df_val=None, weight_column_name=None, categorical_feature_column_names=None, numerical_feature_column_names=None, n_trials=20, n_warmup_random_trials=None, parameter_configurations=None, pass_df_val_into_tuning=False, pass_df_val_into_final_fit=False, use_model_predictions=False)[source]

Tune the hyperparameters of an MCGrad model using Ax.

Parameters:

  • model (_BaseMCGrad) – The MCGrad model to be tuned. It could be a fitted model or an unfitted model.

  • df_train (DataFrame) – The training data: 80% of the data is used for training the model, and the remaining 20% is used for validation.

  • prediction_column_name (str) – The name of the prediction column in the data.

  • label_column_name (str) – The name of the label column in the data.

  • df_val (DataFrame | None) – The validation data. If None, 20% of the training data is used for validation.

  • weight_column_name (str | None) – The name of the weight column in the data. If None, all samples are treated equally.

  • categorical_feature_column_names (list[str] | None) – The names of the categorical feature columns in the data.

  • numerical_feature_column_names (list[str] | None) – The names of the numerical feature columns in the data.

  • n_trials (int) – The number of trials to run. Defaults to 20.

  • n_warmup_random_trials (int | None) – The number of random trials to run before starting the Ax optimization. Defaults to None, which uses calculate_num_initialization_trials to determine the number of warmup trials, which uses the following rules: (i) At least 16 (Twice the number of tunable parameters), (ii) At most 1/5th of num_trials.

  • parameter_configurations (list[RangeParameterConfig] | None) – The list of parameter configurations to tune. If None, the default parameter configurations are used.

  • pass_df_val_into_tuning (bool) – Whether to pass the validation data into the tuning process. If True, the validation data is passed into the tuning process.

  • pass_df_val_into_final_fit (bool) – Whether to pass the validation data into the final fit. If True, the validation data is passed into the final fit.

  • use_model_predictions (bool) – Whether to return the surrogate model’s predicted best (True) or the actual best observed trial (False). Defaults to False, which is safer when running few trials.

Return type:

tuple[_BaseMCGrad | None, DataFrame]

Returns:

A tuple containing: - The fitted MCGrad model with the best hyperparameters found during tuning. - A DataFrame containing the results of all trials, sorted by the evaluation metric.