`torchcp.llm`¶

predictor¶

ConformalLM

Method: Conformal Language Modeling Paper: Conformal Language Modeling (Victor Quach et al., ICLR'24) Link: https://openreview.net/forum?id=pzUhfQ74c5 Github: https://github.com/Varal7/conformal-language-modeling

class torchcp.llm.predictor.ConformalLM(tokenizer=None, model=None, epsilons=None, scaling_type='none', scale_kwargs=None, set_score_function_name='none', rejection=False, seed=2024, alpha=0.1, device=None)¶

Method: Conformal Language Modeling Paper: Conformal Language Modeling (Victor Quach et al., ICLR’24) Link: https://openreview.net/forum?id=pzUhfQ74c5 Github: https://github.com/Varal7/conformal-language-modeling

Parameters:

tokenizer (Any, optional) – A tokenizer for the language model. Default is None.
model (Any, optional) – A language model based on PyTorch. Default is None.
epsilons (torch.Tensor, optional) – The risk levels that need to be controlled. Default is None.
scaling_type (str, optional) – The scaling type for scores. Default is “none”. Score scaling method, one of {platt, bin, platt_bin, rnn, none}.
scale_kwargs (dict, optional) – The parameters for the scaling function. Default is None.
set_score_function_name (str, optional) – The name of the score function to use. Default is “none”. Score function name, one of {geo, marginal, first_k, first_k_no_mask, max, sum, none}.
rejection (bool, optional) – Indicates whether to use rejection sampling. Default is False.
seed (int, optional) – The random seed. Default is 2024.
alpha (float, optional) – The significance level. Default is 0.1.
device (torch.device, optional) – The device on which the model is located. Default is None.

get_pareto_frontier(item_scores, similarity_scores, item_labels)¶: Compute a pareto frontier.

predict_with_config(config, item_scores, similarity_scores)¶

Construct the prediction set for a given config.

Parameters:

set_scores – [num_examples, max_generations] set_scores[i, j] = score of set after sample j for example i.
set_sizes – [num_examples, max_generations] set_sizes[i, j] = effective set size after sample j for example i.
set_losses – [num_examples, max_generations] set_loss[i, j] = loss of set after sample j for example i.
lambdas – [num_thresholds] Array of thresholds to test.

Returns:

Dictionary of metrics (per lambda).

Implementation of different scoring variants.

scoring¶

`geometric`	Score of a set is based on a geometric distribution approximation:
`marginal`	Similar to geometric, but with p(y_k is the only y with A(y) = 1).
`first_k`	Scores are equal to the number of draws.
`first_k_no_mask`	Scores are equal to the number of draws.
`max`
`sum`

torchcp.llm.utils.scoring.geometric(p, mask=None)¶

Score of a set is based on a geometric distribution approximation:

p(exists y in C : L(y) = 0) = 1 - prod 1 - p(A(y_i) = 1)

This is consistent with -sum log (1 - p(A(y_i) = 1)).

Parameters:: p – Matrix of size [num_examples, max_size]. Each entry of p approximates p(A(y_ij) = 1).
Returns:: Log geometric scores.

torchcp.llm.utils.scoring.marginal(p, mask=None)¶: Similar to geometric, but with p(y_k is the only y with A(y) = 1).

torchcp.llm.utils.scoring.first_k(X, mask=None)¶: Scores are equal to the number of draws.

torchcp.llm.utils.scoring.first_k_no_mask(X, mask=None)¶: Scores are equal to the number of draws.

torchcp.llm.utils.scoring.max(X, mask=None)¶

torchcp.llm.utils.scoring.sum(X, mask=None)¶

Utilities for score scaling.

scaling¶

`LogisticRegression`
`PlattScaler`	Parametric calibration using logistic regression on uncalibrated scores.
`BinningScaler`	Non-parametric equal-mass histogram regression on uncalibrated scores.
`PlattBinningScaler`	Combined parametric + non-parametric calibration (Kumar et.
`RecurrentScaler`	RNN calibration of C given sequential scores.

torchcp.llm.utils.scaling.LogisticRegression(dtype=torch.float32)¶

Base class for all neural network modules.

Your models should also subclass this class.

Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes:

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

Submodules assigned in this way will be registered, and will have their parameters converted too when you call to(), etc.

Note

As per the example above, an __init__() call to the parent class must be made before assignment on the child.

Variables:: training (bool) – Boolean represents whether this module is in training or evaluation mode.

torchcp.llm.utils.scaling.PlattScaler(**kwargs)¶: Parametric calibration using logistic regression on uncalibrated scores.

torchcp.llm.utils.scaling.BinningScaler(n_bins=20)¶: Non-parametric equal-mass histogram regression on uncalibrated scores.

torchcp.llm.utils.scaling.PlattBinningScaler(*args, n_bins=20, **kwargs)¶: Combined parametric + non-parametric calibration (Kumar et. al., 2019)

torchcp.llm.utils.scaling.RecurrentScaler(hidden_size=64, num_layers=2, num_iters=100, batch_size=128, dropout=0.0, target='set', verbose=True)¶: RNN calibration of C given sequential scores.

metrics¶

`average_size`	the average size of the prediction sets
`average_sample_size`	the average number of sample size
`average_set_loss`	Calculate the average set loss.
`SSCL`	Size-stratified conditional loss.

torchcp.llm.utils.metrics.average_size(prediction_sets)¶: the average size of the prediction sets

torchcp.llm.utils.metrics.average_sample_size(prediction_sets)¶: the average number of sample size

torchcp.llm.utils.metrics.average_set_loss(prediction_sets, prediction_set_loss)¶

Calculate the average set loss.

Parameters:

prediction_sets (torch.Tensor) – The prediction sets generated by CP algorithms.
prediction_set_loss (torch.Tensor) – The loss associated with each prediction set.

Returns:

The average set loss.

Return type:

torch.Tensor

torchcp.llm.utils.metrics.SSCL(prediction_sets, prediction_set_loss, num_bins=20)¶

Size-stratified conditional loss.

Paper: Conformal Language Modeling (Victor Quach et al., ICLR’24)

Parameters:

prediction_sets (torch.Tensor) – The prediction sets generated by CP algorithms.
prediction_set_loss (torch.Tensor) – The loss associated with each prediction set.
num_bins (int, optional) – The number of bins for stratification. Default is 20.

Returns:

The size-stratified conditional loss.

Return type:

torch.Tensor

`torchcp.llm`¶

predictor¶

scoring¶

scaling¶

metrics¶

TorchCP

Navigation

Related Topics

torchcp.llm¶

predictor¶

scoring¶

scaling¶

metrics¶

`torchcp.llm`¶