torchcp.regression

score function

ABS

Absolute value of the difference between prediction and true value.

CQR

Conformalized Quantile Regression (CQR) This score function allows for calculating scores and generating prediction intervals using quantile regression model.

CQRFM

Conformal Quantile Regression Fraction Median

CQRM

CQR-M

CQRR

CQR-R

R2CCP

Regression-to-Classification Conformal Prediction.

Sign

Value of the difference between prediction and true value.
class torchcp.regression.score.ABS

Absolute value of the difference between prediction and true value.

This score function allows for calculating scores and generating prediction intervals using a single-point regression model.

Reference:

Book: Algorithmic Learning in a Random World (Vovk et al., 2005) Link: https://link.springer.com/book/10.1007/b106715

generate_intervals(predicts_batch, q_hat)

Generate prediction intervals by adjusting predictions with the calibrated q_hat threshold.

Parameters:

  • predicts_batch (torch.Tensor) – A batch of predictions with shape (batch_size, …).

  • q_hat (torch.Tensor) – A tensor containing the calibrated thresholds with shape (num_thresholds,).

Returns:

A tensor containing the prediction intervals with shape (batch_size, num_thresholds, 2).

The last dimension represents the lower and upper bounds of the intervals.

Return type:

torch.Tensor

train(train_dataloader, **kwargs)

Trains the model using the provided training data.

Parameters:

  • train_dataloader (DataLoader) – DataLoader for the training data.

  • **kwargs – Additional keyword arguments for training configuration. - model (nn.Module, optional): The model to be trained. Defaults to the model passed to the predictor. - epochs (int, optional): Number of training epochs. Defaults to \(100\). - criterion (nn.Module, optional): Loss function. Defaults to torch.nn.MSELoss(). - lr (float, optional): Learning rate for the optimizer. Defaults to \(0.01\). - optimizer (torch.optim.Optimizer, optional): Optimizer for training. Defaults to torch.optim.Adam(). - verbose (bool, optional): If True, prints training progress. Defaults to True.

class torchcp.regression.score.CQR

Conformalized Quantile Regression (CQR) This score function allows for calculating scores and generating prediction intervals using quantile regression model.

Reference:

Paper: Conformalized Quantile Regression (Romano et al., 2019) Link: https://proceedings.neurips.cc/paper_files/paper/2019/file/5103c3584b063c431bd1268e9b5e76fb-Paper.pdf Github: https://github.com/yromano/cqr

generate_intervals(predicts_batch, q_hat)

Generates prediction intervals for a batch of predictions.

Prediction intervals are constructed by adding and subtracting the calibrated thresholds (q_hat) from the predicted values.

Parameters:

  • predicts_batch (torch.Tensor) – A batch of predictions, shape (batch_size, …).

  • q_hat (torch.Tensor) – Calibrated thresholds, shape (num_thresholds,).

Returns:

Prediction intervals, shape (batch_size, num_thresholds, 2).

The last dimension contains the lower and upper bounds of the intervals.

Return type:

torch.Tensor

train(train_dataloader, **kwargs)

Trains the model on provided training data with \([alpha/2, 1-alpha/2]\) quantile regression loss.

Parameters:

  • train_dataloader (torch.utils.data.DataLoader) – DataLoader providing training data.

  • **kwargs – Additional training parameters. - model (torch.nn.Module, optional): Model to be trained; defaults to the model passed to the predictor. - criterion (callable, optional): Loss function for training. If not provided, uses QuantileLoss(). - alpha (float, optional): Significance level (e.g., 0.1) for quantiles, required if criterion is None. - epochs (int, optional): Number of training epochs. Default is \(100\). - lr (float, optional): Learning rate for optimizer. Default is \(0.01\). - optimizer (torch.optim.Optimizer, optional): Optimizer for training; defaults to torch.optim.Adam(). - verbose (bool, optional): If True, displays training progress. Default is True.

Raises:

ValueError – If criterion is not provided and alpha is not specified.

class torchcp.regression.score.CQRFM

Conformal Quantile Regression Fraction Median

Reference:

Paper: Adaptive, Distribution-Free Prediction Intervals for Deep Networks (Kivaranovic et al., 2019) Link: https://proceedings.mlr.press/v108/kivaranovic20a.html

generate_intervals(predicts_batch, q_hat)

Generates prediction intervals for a batch of predictions.

Prediction intervals are constructed by adding and subtracting the calibrated thresholds (q_hat) from the predicted values.

Parameters:

  • predicts_batch (torch.Tensor) – A batch of predictions, shape (batch_size, …).

  • q_hat (torch.Tensor) – Calibrated thresholds, shape (num_thresholds,).

Returns:

Prediction intervals, shape (batch_size, num_thresholds, 2).

The last dimension contains the lower and upper bounds of the intervals.

Return type:

torch.Tensor

class torchcp.regression.score.CQRM

CQR-M

Reference:

Paper: A comparison of some conformal quantile regression methods (Matteo Sesia and Emmanuel J. Candes, 2019) Link: https://onlinelibrary.wiley.com/doi/epdf/10.1002/sta4.261 Github: https://github.com/msesia/cqr-comparison

generate_intervals(predicts_batch, q_hat)

Generates prediction intervals for a batch of predictions.

Prediction intervals are constructed by adding and subtracting the calibrated thresholds (q_hat) from the predicted values.

Parameters:

  • predicts_batch (torch.Tensor) – A batch of predictions, shape (batch_size, …).

  • q_hat (torch.Tensor) – Calibrated thresholds, shape (num_thresholds,).

Returns:

Prediction intervals, shape (batch_size, num_thresholds, 2).

The last dimension contains the lower and upper bounds of the intervals.

Return type:

torch.Tensor

train(train_dataloader, **kwargs)

Trains the model on provided training data with \([alpha/2, 1/2, 1-alpha/2]\) quantile regression loss.

Parameters:

  • train_dataloader (torch.utils.data.DataLoader) – DataLoader providing training data.

  • **kwargs – Additional training parameters. - model (torch.nn.Module, optional): Model to be trained; defaults to the model passed to the predictor. - criterion (callable, optional): Loss function for training. If not provided, uses QuantileLoss(). - alpha (float, optional): Significance level (e.g., 0.1) for quantiles, required if criterion is None. - epochs (int, optional): Number of training epochs. Default is \(100\). - lr (float, optional): Learning rate for optimizer. Default is \(0.01\). - optimizer (torch.optim.Optimizer, optional): Optimizer for training; defaults to torch.optim.Adam(). - verbose (bool, optional): If True, displays training progress. Default is True.

Raises:

ValueError – If criterion is not provided and alpha is not specified.

Note

This function is optional but recommended, because the training process for each preditor’s model is different. We provide a default training method, and users can change the hyperparameters kwargs to modify the training process. If the train function is not used, users should pass the trained model to the predictor at the beginning.

class torchcp.regression.score.CQRR

CQR-R

Reference:

Paper: A comparison of some conformal quantile regression methods (Matteo Sesia and Emmanuel J. Candes, 2019) Link: https://onlinelibrary.wiley.com/doi/epdf/10.1002/sta4.261 Github: https://github.com/msesia/cqr-comparison

generate_intervals(predicts_batch, q_hat)

Generates prediction intervals for a batch of predictions.

Prediction intervals are constructed by adding and subtracting the calibrated thresholds (q_hat) from the predicted values.

Parameters:

  • predicts_batch (torch.Tensor) – A batch of predictions, shape (batch_size, …).

  • q_hat (torch.Tensor) – Calibrated thresholds, shape (num_thresholds,).

Returns:

Prediction intervals, shape (batch_size, num_thresholds, 2).

The last dimension contains the lower and upper bounds of the intervals.

Return type:

torch.Tensor

class torchcp.regression.score.R2CCP(midpoints, device=None)

Regression-to-Classification Conformal Prediction.

This method converting regression to a classification problem and then use CP for classification to obtain CP sets for regression.

Parameters:

midpoints (torch.Tensor) – the midpoints of the equidistant bins.

Reference:

Paper: Conformal Prediction via Regression-as-Classification (Etash Guha et al., 2021) Link: https://neurips.cc/virtual/2023/80610 Github: https://github.com/EtashGuha/R2CCP

generate_intervals(predicts_batch, q_hat)

Generate the prediction interval for the given batch of predictions.

Parameters:

  • predicts_batch – the batch of predictions.

  • q_hat – the quantile level.

train(train_dataloader, **kwargs)

Trains regression-to-classification model with the R2ccpLoss.

Parameters:

  • train_dataloader (torch.utils.data.DataLoader) – DataLoader for training data.

  • kwargs – Additional parameters for training. - model (torch.nn.Module, optional): Model to be trained; defaults to the model passed to the predictor. - epochs (int, optional): Number of training epochs. Default is \(100\). - p (float, optional): Probability parameter for the loss function. Default is \(0.5\). - tau (float, optional): Threshold parameter for the loss function. Default is \(0.2\). - criterion (callable, optional): Loss function; defaults to R2ccpLoss. - lr (float, optional): Learning rate. Default is \(1e-4\). - weight_decay (float, optional): Weight decay for the optimizer. Default is \(1e-4\). - optimizer (torch.optim.Optimizer, optional): Optimizer; defaults to torch.optim.AdamW(). - verbose (bool, optional): If True, displays training progress. Default is True.

class torchcp.regression.score.Sign

Value of the difference between prediction and true value.

This score function allows for calculating scores.

predictor

SplitPredictor

Split Conformal Prediction for Regression.

EnsemblePredictor

Ensemble Conformal Prediction Interval

ACIPredictor

Adaptive Conformal Inference.

ConformalPredictiveDistribution

Obtain conformal predictive distributions from conformal predictive system.
class torchcp.regression.predictor.SplitPredictor(score_function, model=None, alpha=0.1, device=None)

Split Conformal Prediction for Regression.

This predictor allows for the construction of a prediction band for the response variable using any estimator of the regression function.

Parameters:

  • score_function (torchcp.regression.scores) – A class that implements the score function.

  • model (torch.nn.Module) – A pytorch regression model that can output predicted point.

  • alpha (float, optional) – The significance level. Default is 0.1.

  • device (torch.device, optional) – The device on which the model is located. Default is None.

Reference:

Paper: Distribution-Free Predictive Inference For Regression (Lei et al., 2017) Link: https://arxiv.org/abs/1604.04173 Github: https://github.com/ryantibs/conformal

calculate_score(predicts, y_truth, x_batch=None)

Calculate the nonconformity scores based on predictions and true values.

The exact score calculation is determined by the configured score_function.

Parameters:

  • predicts (torch.Tensor) – Model predictions.

  • y_truth (torch.Tensor) – Ground truth values.

  • x_batch (torch.Tensor, optional) – The corresponding batch of input features. This is only required by normalized score functions (e.g., ‘NorABS’) to compute adaptive nonconformity scores. Defaults to None.

Returns:

The computed nonconformity score for each sample.

Return type:

torch.Tensor

calibrate(cal_dataloader, alpha=None)

Calibrate the predictor using a calibration dataset and a given significance level alpha.

Parameters:

  • cal_dataloader (torch.utils.data.DataLoader) – The dataloader containing the calibration dataset.

  • alpha (float) – The significance level for calibration. Should be in the range (0, 1). Default is None.

evaluate(data_loader)

Evaluate the model on a test dataloader, returning coverage rate and interval size.

Parameters:

data_loader (torch.utils.data.DataLoader) – The dataloader containing the test dataset.

Returns:

A dictionary containing the coverage rate and average interval size with keys: - coverage_rate (float): The coverage rate of the prediction intervals. - average_size (float): The average size of the prediction intervals.

Return type:

dict

generate_intervals(predicts_batch, q_hat, x_batch=None)

Generate prediction intervals based on the model’s predictions and the conformal value.

Parameters:

  • predicts_batch (torch.Tensor) – Batch of predictions from the model.

  • q_hat (float) – Conformal value computed during calibration.

Returns:

Prediction intervals.

Return type:

torch.Tensor

predict(x_batch)

Generates prediction intervals for a batch of input data.

Parameters:

x_batch (torch.Tensor) – Input batch of data points, shape (batch_size, input_dim).

Returns:

Prediction intervals, shape (batch_size, 2).

Return type:

torch.Tensor

predict_p(x_batch, y_batch, smooth=False)

Compute p-values for conformal prediction.

Parameters:

  • x_batch (torch.Tensor) – A batch of instances.

  • y_batch (torch.Tensor) – A batch of labels for instances.

  • smooth (bool) – Whether to apply randomized smoothing when calibration scores equal test scores.

Returns:

p-values for each test sample and class, shape (n_test, k)

Return type:

Tensor

train(train_dataloader, **kwargs)

Trains the model using the provided train_dataloader and score_function.

Parameters:

  • train_dataloader (DataLoader) – DataLoader for the training data.

  • **kwargs – Additional keyword arguments for training configuration. - model (nn.Module, optional): The model to be trained. - epochs (int, optional): Number of training epochs. - criterion (nn.Module, optional): Loss function. - lr (float, optional): Learning rate for the optimizer. - optimizer (torch.optim.Optimizer, optional): Optimizer for training. - verbose (bool, optional): If True, prints training progress.

Note

This function is optional but recommended, because the training process for each score_function is different. We provide a default training method, and users can change the hyperparameters kwargs to modify the training process. If the train function is not used, users should pass the trained model to the predictor at the beginning.

class torchcp.regression.predictor.EnsemblePredictor(score_function, model, aggregation_function='mean', alpha=0.1, device=None)

Ensemble Conformal Prediction Interval

There are two main implementations for score_function:

  1. EnbPI:
  2. EnCQR:
Parameters:

  • model (torch.nn.Module) – The base model to be used in the ensemble.

  • score_function (torchcp.regression.scores) – The method for calculating scores and prediction intervals. Can be torchcp.regression.scores.split for EnbPI, torchcp.regression.scores.CQR for EnCQR, or other variants.

  • aggregation_function (str or callable) – A function to aggregate predictions from multiple models in the ensemble. Options include: - torch.mean: Computes the mean of the predictions. - torch.median: Computes the median of the predictions. - Custom function: Should accept a tensor and dimension as input, returning the result.

  • alpha (float, optional) – The significance level. Default is 0.1.

  • device (torch.device, optional) – The device on which the model is located. Default is None.

evaluate(data_loader, alpha=None, verbose=True)

Evaluates the performance of the ensemble model on a test dataset by calculating coverage rates and average sizes of the prediction intervals.

Parameters:

  • data_loader (DataLoader) – The DataLoader providing the test data batches.

  • alpha (float) – The significance level for conformal prediction, which controls the width of the prediction intervals (e.g., 0.1 for 90% prediction intervals). Default is None.

  • verbose (bool) – If True, prints the coverage rate and average size for each batch. Default is True.

Returns:

A dictionary containing:

  • ”Total batches”: The number of batches evaluated.

  • ”Average coverage rate”: The average coverage rate across all batches.

  • ”Average prediction interval size”: The average size of the prediction intervals.

Return type:

dict

Example:

>>> eval_results = model.evaluate(test_data_loader, alpha=0.1)
>>> print(eval_results)

Note

Procedure: 1. Iterates through each batch in the test data. 2. For each batch:

  • Generates prediction intervals using the predict method.

  • Calculates the coverage rate (percentage of true values within the interval).

  • Calculates the average interval size.

  • (Optional) If verbose is True, prints batch-wise coverage rate and average size.

  1. Aggregates and returns the overall average coverage rate and interval size across batches.

predict(x_batch, alpha=None, y_batch_last=None, aggr_pred_last=None)

Generates conformal prediction intervals for new data.

Parameters:

  • x_batch (Tensor) – Batch of input features.

  • alpha (float) – Significance level for conformal intervals. Default is None.

  • y_batch_last (Tensor, optional) – Labels from the previous batch for score updates.

  • aggr_pred_last (Tensor, optional) – Aggregated predictions from the previous batch.

Returns:

A tuple containing:

  • Prediction intervals for the input batch.

  • Aggregated predictions for the input batch.

Return type:

Tuple

train(train_dataloader, ensemble_num, subset_num, **kwargs)

Trains an ensemble of models on randomly sampled subsets of the training data.

Parameters:

  • train_dataloader (DataLoader) – The DataLoader for the training data, providing batches of data for training.

  • ensemble_num (int) – The number of models to ensemble.

  • subset_num (int) – The size of the subset of data for training each individual model in the ensemble.

  • **kwargs – Additional parameters for the score_predictor.train() method. - criterion (callable, optional): Loss function for training. If not provided, uses QuantileLoss(). - alpha (float, optional): Significance level (e.g., 0.1) for quantiles, required if criterion is None. - epochs (int, optional): Number of training epochs. Default is \(100\). - lr (float, optional): Learning rate for optimizer. Default is \(0.01\). - optimizer (torch.optim.Optimizer, optional): Optimizer for training; defaults to torch.optim.Adam(). - verbose (bool, optional): If True, displays training progress. Default is True.

Raises:

ValueError – If criterion is not provided and alpha is not specified.

Example:

>>> predictor = Ensemble(model, score_predictor, aggregation_function)
>>> predictor.train(train_dataloader=train_data_loader, ensemble_num=5, subset_num=500)

Note

Procedure:

  1. Creates ensemble_num models by sampling subsets from the training data.

2. For each model in the ensemble: - Samples a random subset of indices of size subset_num from the dataset. - Trains a copy of the base model on this subset. - Stores each trained model along with the subset indices used for training.

Post-training:

Computes and stores the conformal scores on the training dataset for later use in prediction intervals.

class torchcp.regression.predictor.ACIPredictor(score_function, model, gamma, alpha=0.1, device=None)

Adaptive Conformal Inference.

methods for forming prediction sets in an online setting where the data generating distribution is allowed to vary over time in an unknown fashion.

Parameters:

  • score_function (torchcp.regression.scores) – A class that implements the score function.

  • model (torch.nn.Module) – A PyTorch model capable of outputting quantile values. The model should be an initialization model that has not been trained.

  • gamma (float) – Step size parameter for adaptive adjustment of alpha. Must be greater than 0.

  • alpha (float, optional) – The significance level. Default is 0.1.

  • device (torch.device, optional) – The device on which the model is located. Default is None.

Reference:

Paper: Adaptive conformal inference Under Distribution Shift (Gibbs et al., 2021) Link: https://arxiv.org/abs/2106.00170 Github: https://github.com/isgibbs/AdaptiveConformal

calculate_err_rate(x_batch, y_batch_last, pred_interval_last, weight=True)

Calculate the error rate for the previous prediction intervals.

Parameters:

  • x_batch (torch.Tensor) – Input features for the current batch.

  • y_batch_last (torch.Tensor) – True labels from the previous step.

  • pred_interval_last (torch.Tensor) – Prediction intervals from the previous step.

Returns:

Weighted error rate based on historical predictions.

Return type:

float

evaluate(data_loader, lookback=200, retrain_gap=1, update_alpha_gap=1)

Evaluate the model using a test dataset and compute performance metrics such as coverage rate and average prediction interval size. The evaluation process optionally includes periodic retraining of the model and updating of the conformal score threshold (alpha).

Parameters:

  • data_loader (torch.utils.data.DataLoader) – The data loader for the evaluation dataset. It should provide batches of input features and ground-truth labels for testing.

  • lookback (int, optional) – The number of historical data points (from the training dataset) to use for initializing lookback buffers (x_lookback, y_lookback, and pred_interval_lookback). This value must be less than or equal to the size of the training dataset. Default is 200.

  • retrain_gap (int, optional) – The interval (in terms of number of test samples processed) at which the model is retrained using the lookback data. If set to 0, retraining is disabled. Default is 1.

  • update_alpha_gap (int, optional) – The interval (in terms of number of test samples processed) at which the conformal score threshold (alpha) is updated using the lookback data. If set to 0, updating alpha is disabled. Default is 1.

Returns:

A dictionary containing the following metrics:

  • ”Coverage_rate” (float): The proportion of true labels that fall within the

predicted intervals. - “Average_size” (float): The average size of the prediction intervals.

Return type:

dict

Raises:

ValueError – If lookback is greater than the size of the training dataset.

Notes

  • The lookback buffers (x_lookback, y_lookback, and pred_interval_lookback) are

initialized using the last lookback samples from the training dataset. - During the evaluation process:

  • If retrain_gap > 0, the model is retrained periodically (every retrain_gap

samples) using the lookback buffers. - If update_alpha_gap > 0, the conformal score threshold (alpha) is updated periodically (every update_alpha_gap samples) based on the lookback buffers.

  • The lookback buffers are updated dynamically after processing each test sample

with the latest predictions, inputs, and ground-truth labels from the evaluation dataset.

generate_aci_intervals(x_batch, x_lookback, y_lookback, pred_interval_lookback, predicts_batch)

Generates Adaptive Conformal Inference (ACI) prediction intervals.

Parameters:

  • x_batch (Tensor) – A batch of input features for which predictions and prediction intervals are to be generated.

  • x_lookback (Tensor) – Historical input features used for updating model calibration.

  • y_lookback (Tensor) – Historical target values corresponding to x_lookback.

  • pred_interval_lookback (Tensor) – Previously generated prediction intervals.

  • predicts_batch (Tensor) – Model predictions for the current input batch x_batch.

Returns:

The ACI prediction intervals for the input batch x_batch.

Return type:

prediction intervals (Tensor)

predict(x_batch, x_lookback=None, y_lookback=None, pred_interval_lookback=None, train=False, update_alpha=True)

Generates conformal prediction intervals for a given batch of input data. This function can also optionally retrain the model or update the conformal score threshold (alpha) based on historical data.

Parameters:

  • x_batch (Tensor) – A batch of input features for which predictions and prediction intervals are to be generated. Shape depends on the model’s input requirements (e.g., [batch_size, num_features]).

  • x_lookback (Tensor, optional) – Historical input features used for retraining or updating model calibration. If provided, y_lookback must also be provided. Default is None.

  • y_lookback (Tensor, optional) – Historical target values corresponding to x_lookback. Used for retraining or updating model calibration. If provided, x_lookback must also be provided. Default is None.

  • pred_interval_lookback (Tensor, optional) – Previously generated prediction intervals that can be used for calibration. If not provided, prediction intervals will be computed using the model’s predictions and the current quantile value q_hat. Default is None.

  • train (bool, optional) – Whether to retrain the model using the x_lookback and y_lookback data. If True, both x_lookback and y_lookback must be provided. If False, the model will not be retrained. Default is False.

  • update_alpha (bool, optional) – Whether to update the conformal score threshold (alpha) based on the error rate observed in the prediction intervals. If True, both x_lookback and y_lookback must be provided. Default is False.

Returns:

The conformal prediction intervals for the input batch x_batch

Return type:

Prediction intervals (Tensor)

Raises:

ValueError – If x_lookback is provided but y_lookback is not, or vice versa. Both must be provided together or both must be None.

Notes

  • If train is set to True but x_lookback and y_lookback are not provided,

the function will issue a warning and skip retraining. - If update_alpha is set to True but x_lookback and y_lookback are not provided, the function will use the current value of alpha instead of recalibrating it. - The conformal score threshold (alpha) is updated using a time-decayed approach, where the rate of adjustment depends on the parameter gamma.

train(train_dataloader, **kwargs)

Train and calibrate the predictor using the training data.

Parameters:

  • train_dataloader (torch.utils.data.DataLoader) – DataLoader for training data.

  • alpha (float) – Desired initial coverage rate.

  • **kwargs – Additional keyword arguments for training configuration. - model (nn.Module, optional): The model to be trained. - epochs (int, optional): Number of training epochs. - criterion (nn.Module, optional): Loss function. - lr (float, optional): Learning rate for the optimizer. - optimizer (torch.optim.Optimizer, optional): Optimizer for training. - verbose (bool, optional): If True, prints training progress.

Note

This function is necessary for ACI predictor. We provide a default training method, and users can change the hyperparameters kwargs to modify the training process. If the user wants to fully control the training process, it can be achieved by rewriting the train() of the score function.

class torchcp.regression.predictor.ConformalPredictiveDistribution(score_function=<torchcp.regression.score.sign.Sign object>, model=None, alpha=0.1, device=None)

Obtain conformal predictive distributions from conformal predictive system.

Parameters:

  • score_function (torchcp.regression.scores) – the score function must be Sign.

  • model (torch.nn.Module) – A pytorch regression model that can output predicted point.

  • alpha (float, optional) – The significance level. Default is 0.1.

  • device (torch.device, optional) – The device on which the model is located. Default is None.

Reference:

Paper: Algorithmic Learning in a Random World (vovk et al., 2022) Link: https://link.springer.com/book/10.1007/978-3-031-06649-8

calculate_score(predicts, y_truth)

Calculate the nonconformity scores based on the model’s predictions and true values.

Parameters:

  • predicts (torch.Tensor) – Model predictions.

  • y_truth (torch.Tensor) – Ground truth values.

Returns:

Computed scores for each prediction.

Return type:

torch.Tensor

calibrate(cal_dataloader)

Calibrate the predictor using a calibration dataset and a given significance level alpha.

Parameters:

  • cal_dataloader (torch.utils.data.DataLoader) – The dataloader containing the calibration dataset.

  • alpha (float) – The significance level for calibration. Should be in the range (0, 1). Default is None.

predict(x_batch)

Obtain conformal predictive distributions from conformal predictive

Parameters:

x_batch (torch.Tensor) – A batch of instances.

Returns:

conformal predictive distributions., shape (n_test, n_calib)

Return type:

Tensor

loss

QuantileLoss

Pinball Loss (Quantile Loss) for Quantile Regression.

R2ccpLoss

Conformal Prediction via Regression-as-Classification Loss.
class torchcp.regression.loss.QuantileLoss(quantiles)

Pinball Loss (Quantile Loss) for Quantile Regression. This loss, also known as the pinball loss, is commonly used in quantile regression to estimate the conditional quantiles of a target variable. It applies different penalties based on whether the predictions fall above or below the actual target, making it useful for tasks requiring interval or quantile estimation.

Parameters:

quantiles (list of float) – List of quantiles to compute, typically in the range [0, 1], e.g., [0.025, 0.975] for the 2.5th and 97.5th percentiles.

Shape:

  • Input: preds of shape (batch_size, num_quantiles) where num_quantiles is the number of specified quantiles.

  • Target: target of shape (batch_size, 1).

  • Output: A scalar representing the mean quantile loss.

Reference:

Paper: Conformalized Quantile Regression (Romano, Y., et al., 2019) Link: https://proceedings.neurips.cc/paper_files/paper/2019/file/5103c3584b063c431bd1268e9b5e76fb-Paper.pdf Github: https://github.com/yromano/cqr

The quantile loss for each quantile level \(q \in (0, 1)\) is defined as:

\[\begin{split}L_q(y, \hat{y}) = \begin{cases} q \cdot (y - \hat{y}) & \text{if } y > \hat{y} \\ (q - 1) \cdot (y - \hat{y}) & \text{if } y \leq \hat{y} \end{cases}\end{split}\]

where: - \(y\) is the target value, - \(\hat{y}\) is the predicted quantile value.

The total loss across all quantiles is averaged over the batch.

Examples:

>>> loss_fn = QuantileLoss(quantiles=[0.1, 0.5, 0.9])
>>> preds = torch.rand(4, 3, requires_grad=True)
>>> target = torch.rand(4, 1)
>>> loss = loss_fn(preds, target)
>>> loss.backward()
forward(preds, target)

Computes the mean pinball loss between predictions and targets.

Parameters:

  • preds (torch.Tensor) – Predicted values for each quantile, shaped as (batch_size, num_quantiles).

  • target (torch.Tensor) – Ground truth target values, shaped as (batch_size, 1).

Returns:

The scalar mean pinball loss across the batch.

Return type:

torch.Tensor

class torchcp.regression.loss.R2ccpLoss(p, tau, midpoints)

Conformal Prediction via Regression-as-Classification Loss. This loss combines cross-entropy and entropy regularization to provide uncertainty estimates for predictions, supporting conformal prediction in regression tasks by treating it as a classification problem.

Parameters:

  • p (float) – The norm degree for the distance measure in the cross-entropy term.

  • tau (float) – Weighting factor for the entropy regularization term.

  • midpoints (torch.Tensor) – A tensor containing midpoint values for each bin.

Shape:

  • Input: preds of shape (batch_size, K) where K is the number of bins.

  • Target: target of shape (batch_size, 1).

  • Output: A scalar representing the computed loss.

Reference:

Paper: Conformal Prediction via Regression-as-Classification (Etash Guha et al., 2021) Link: https://neurips.cc/virtual/2023/80610 Github: https://github.com/EtashGuha/R2CCP

The loss is composed of two main parts:

  1. Cross-entropy component: This term computes the weighted distance between the prediction probabilities and target midpoint values based on a specified distance norm p.

  2. Entropy regularization term: This term applies a Shannon entropy penalty, controlled by the hyperparameter tau, which helps balance between prediction certainty and regularization for the model’s output distribution.

The total loss is given by:

\[\mathcal{L} = \sum_{i=1}^N \left( \sum_{k=1}^K |y_i - m_k|^p \cdot \hat{p}_{ik} - \tau \sum_{k=1}^K \hat{p}_{ik} \log(\hat{p}_{ik}) \right)\]

where: - \(y_i\) is the target for instance i, - \(m_k\) is the midpoint for bin k, - \(\hat{p}_{ik}\) is the predicted probability for bin k and instance i, - \(\tau\) is the weight for the entropy regularization term, - \(p\) is the norm for the distance measure.

Examples:

>>> loss_fn = R2ccpLoss(p=2, tau=0.5, midpoints=torch.tensor([0.1, 0.5, 0.9]))
>>> preds = torch.rand(3, 3, requires_grad=True).softmax(dim=1)
>>> target = torch.tensor([[0.2], [0.6], [0.8]])
>>> loss = loss_fn(preds, target)
>>> loss.backward()
forward(preds, target)

Computes the R2ccp loss between model predictions and targets.

Parameters:

  • preds (torch.Tensor) – Model predictions after applying softmax, shape (batch_size, K).

  • target (torch.Tensor) – Ground truth values, shape (batch_size, 1).

Returns:

Scalar loss value.

Return type:

torch.Tensor

metrics

coverage_rate

Calculates the coverage rate of prediction intervals.

average_size

Computes the average size of prediction intervals.
torchcp.regression.utils.metrics.coverage_rate(prediction_intervals, y_truth)

Calculates the coverage rate of prediction intervals.

Parameters:

  • prediction_intervals (torch.Tensor) – A tensor of shape (batch_size, num_intervals * 2), where each interval has a lower and upper bound.

  • y_truth (torch.Tensor) – A tensor of ground truth values of shape (batch_size,).

Returns:

The coverage rate, representing the proportion of ground truth values

within the specified prediction intervals.

Return type:

torch.Tensor

torchcp.regression.utils.metrics.average_size(prediction_intervals)

Computes the average size of prediction intervals.

Parameters:

prediction_intervals (torch.Tensor) – A tensor of shape (batch_size, num_intervals * 2), where each interval has a lower and upper bound.

Returns:

The average size of the prediction intervals across all samples.

Return type:

torch.Tensor

utils

calculate_midpoints

Calculates K evenly spaced midpoints between the minimum and maximum values in a dataset loaded from data_loader.
torchcp.regression.utils.utils.calculate_midpoints(data_loader, K)

Calculates K evenly spaced midpoints between the minimum and maximum values in a dataset loaded from data_loader.

Used for R2CCP predictor.

Parameters:

  • data_loader (DataLoader) – A PyTorch DataLoader containing tuples where the second element (data[1]) is the tensor of interest.

  • K (int) – The number of midpoints to calculate.

Returns:

A tensor containing K midpoints evenly spaced between the minimum

and maximum values of the concatenated dataset.

Return type:

torch.Tensor