Evaluation

It provides the tools to evaluate the performance of predictive models.

BettorGridSearchCV(estimator, param_grid, *, scoring=None, n_jobs=None, refit=True, cv=TSCV, verbose=0, pre_dispatch='2*n_jobs', error_score=np.nan, return_train_score=False)

Bases: GridSearchCV, _BaseBettor

Exhaustive search over specified parameter values for a bettor.

BettorGridSearchCV implements a fit, apredict, a predict_proba', abetand ascore` method.

The parameters of the bettor used to apply these methods are optimized by cross-validated grid-search over a parameter grid.

Read more in the user guide.

Parameters:

Name	Type	Description	Default
estimator	_BaseBettor	This is assumed to implement the bettor interface.	required
param_grid	dict | list	Dictionary with parameters names (str) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.	required
scoring	str | Callable | list | tuple | dict[str, Callable] | None	Strategy to evaluate the performance of the cross-validated model on the test set. If scoring represents a single score, one can use: a single string a callable (see :ref:scoring) that returns a single value If scoring represents multiple scores, one can use: a list or tuple of unique strings a callable returning a dictionary where the keys are the metric names and the values are the metric scores a dictionary with metric names as keys and callables a values	None
n_jobs	int | None	Number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors.	None
refit	bool | str | Callable	Refit an estimator using the best found parameters on the whole dataset. For multiple metric evaluation, this needs to be a str denoting the scorer that would be used to find the best parameters for refitting the estimator at the end. Where there are considerations other than maximum score in choosing a best estimator, refit can be set to a function which returns the selected best_index_ given cv_results_. In that case, the best_estimator_ and best_params_ will be set according to the returned best_index_ while the best_score_ attribute will not be available. The refitted estimator is made available at the best_estimator_ attribute and permits using predict directly on this BettorGridSearchCV instance. Also for multiple metric evaluation, the attributes best_index_, best_score_ and best_params_ will only be available if refit is set and all of them will be determined w.r.t this specific scorer. See scoring parameter to know more about multiple metric evaluation.	True
cv	TimeSeriesSplit	Provides train/test indices to split time series data samples that are observed at fixed time intervals, in train/test sets.	TSCV
verbose	int	Controls the verbosity: the higher, the more messages.	0
pre_dispatch	int | str	Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be: - `None`, in which case all the jobs are immediately created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs - An int, giving the exact number of total jobs that are spawned - A str, giving an expression as a function of n_jobs, as in '2*n_jobs'	'2*n_jobs'
error_score	str | float | int	Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.	nan
return_train_score	bool	If False, the cv_results_ attribute will not include training scores. Computing training scores is used to get insights on how different parameter settings impact the overfitting/underfitting trade-off. However computing the scores on the training set can be computationally expensive and is not strictly required to select the parameters that yield the best generalization performance.	False

Attributes:

Name	Type	Description
cv_results_		A dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame. The key 'params' is used to store a list of parameter settings dicts for all the parameter candidates. The mean_fit_time, std_fit_time, mean_score_time and std_score_time are all in seconds. For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name.
best_estimator_		Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.
best_score_		Mean cross-validated score of the best_estimator For multi-metric evaluation, this is present only if refit is specified. This attribute is not available if refit is a function.
best_params_		Parameter setting that gave the best results on the hold out data. For multi-metric evaluation, this is present only if refit is specified.
best_index_		The index (of the cv_results_ arrays) which corresponds to the best candidate parameter setting. For multi-metric evaluation, this is present only if refit is specified.
scorer_		Scorer function used on the held out data to choose the best parameters for the model. For multi-metric evaluation, this attribute holds the validated scoring dict which maps the scorer key to the scorer callable.
n_splits_		The number of cross-validation splits (folds/iterations).
refit_time_		Seconds used for refitting the best model on the whole dataset. This is present only if refit is not False.
multimetric_		Whether or not the scorers compute several metrics.
classes_	list	The classes labels. This is present only if refit is specified and the underlying estimator is a classifier.
n_features_in_	list	Number of features seen during fit. Only defined if best_estimator_ is defined and that best_estimator_ exposes n_features_in_ when fit.
feature_names_in_	list	Names of features seen during fit. Only defined if best_estimator_ is defined and that best_estimator_ exposes feature_names_in_ when fit.

Examples:

>>> from sportsbet.evaluation import BettorGridSearchCV, OddsComparisonBettor
>>> from sportsbet.datasets import SoccerDataLoader
>>> from sklearn.model_selection import TimeSeriesSplit
>>> # Select only backtesting data for the Italian and Spanish leagues and years 2019 - 2022
>>> param_grid = {'league': ['Italy', 'Spain'], 'year': [2019, 2020, 2021, 2022]}
>>> dataloader = SoccerDataLoader(param_grid)
>>> # Select the market maximum odds
>>> X, Y, O = dataloader.extract_train_data(
... odds_type='market_maximum',
... )
>>> # Backtest the bettor
>>> bettor = BettorGridSearchCV(
    estimator=OddsComparisonBettor(),
    param_grid={'alpha': [0.02, 0.05, 0.1, 0.2, 0.3]},
    cv=TimeSeriesSplit(2),
... )
>>> backtest(bettor, X, Y, O, cv=TimeSeriesSplit(2))
Training Start ... Yield percentage per bet (away_win__full_time_goals)
...

Source code in src/sportsbet/evaluation/_model_selection.py

def __init__(
    self: Self,
    estimator: _BaseBettor,
    param_grid: dict | list,
    *,
    scoring: str | Callable | list | tuple | dict[str, Callable] | None = None,
    n_jobs: int | None = None,
    refit: bool | str | Callable = True,
    cv: TimeSeriesSplit = TSCV,
    verbose: int = 0,
    pre_dispatch: int | str = '2*n_jobs',
    error_score: str | float | int = np.nan,
    return_train_score: bool = False,
) -> None:
    GridSearchCV.__init__(
        self,
        estimator=estimator,
        param_grid=param_grid,
        scoring=scoring,
        n_jobs=n_jobs,
        refit=refit,
        cv=cv,
        verbose=verbose,
        pre_dispatch=pre_dispatch,
        error_score=error_score,
        return_train_score=return_train_score,
    )

bet(X, O)

Predict the value bets for the provided input data and odds.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required
O	DataFrame	The odds data.	required

Returns:

Name	Type	Description
B	BoolData	The value bets.

Source code in src/sportsbet/evaluation/_model_selection.py

def bet(self: Self, X: pd.DataFrame, O: pd.DataFrame) -> BoolData:
    """Predict the value bets for the provided input data and odds.

    Args:
        X:
            The input data.

        O:
            The odds data.

    Returns:
        B:
            The value bets.
    """
    self._check_attr('bet', False, True)
    return self.best_estimator_.bet(X, O)

fit(X, Y, O=None)

Fit the bettor to the input data and multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required
Y	DataFrame	The multi-output targets.	required
O	DataFrame | None	The odds data.	None

Returns:

Name	Type	Description
self	Self	The fitted bettor object.

Source code in src/sportsbet/evaluation/_model_selection.py

def fit(self: Self, X: pd.DataFrame, Y: pd.DataFrame, O: pd.DataFrame | None = None) -> Self:
    """Fit the bettor to the input data and multi-output targets.

    Args:
        X:
            The input data.

        Y:
            The multi-output targets.

        O:
            The odds data.

    Returns:
        self:
            The fitted bettor object.
    """
    self._fit(X, Y, O)
    if hasattr(self, 'best_estimator_'):
        self.init_cash_ = self.best_estimator_.init_cash_
        self.stake_ = self.best_estimator_.stake_
    if O is not None and hasattr(self, 'best_estimator_'):
        self.feature_names_odds_ = self.best_estimator_._get_feature_names_odds(O)
    return self

predict(X)

Predict class labels for multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required

Returns:

Name	Type	Description
Y	BoolData	The positive class labels.

Source code in src/sportsbet/evaluation/_model_selection.py

def predict(self: Self, X: pd.DataFrame) -> BoolData:
    """Predict class labels for multi-output targets.

    Args:
        X:
            The input data.

    Returns:
        Y:
            The positive class labels.
    """
    self._check_attr('predict', False, True)
    return self.best_estimator_.predict(X)

predict_proba(X)

Predict class probabilities for multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required

Returns:

Name	Type	Description
Y	Data	The positive class probabilities.

Source code in src/sportsbet/evaluation/_model_selection.py

def predict_proba(self: Self, X: pd.DataFrame) -> Data:
    """Predict class probabilities for multi-output targets.

    Args:
        X:
            The input data.

    Returns:
        Y:
            The positive class probabilities.
    """
    self._check_attr('predict_proba', False, True)
    return self.best_estimator_.predict_proba(X)

ClassifierBettor(classifier, betting_markets=None, init_cash=None, stake=None)

Bases: MetaEstimatorMixin, _BaseBettor

Bettor based on a Scikit-Learn classifier.

Read more in the user guide.

Parameters:

Name	Type	Description	Default
classifier	BaseEstimator	A scikit-learn classifier object implementing fit, score and predict_proba.	required
betting_markets	list[str] | None	Select the betting markets from the ones included in the data.	None
init_cash	float | None	The initial cash to use when betting.	None
stake	float | None	The stake of each bet.	None

Attributes:

Name	Type	Description
tscv_	TimeSeriesSplit	The checked value of time series cross-validator object. If tscv is None, it uses the default TimeSeriesSplit object.
init_cash_	TimeSeriesSplit	The checked value of initial cash. If init_cash is None, it uses the value of 1e3.
backtesting_results_	DataFrame	The backtesting results.

Examples:

>>> from sklearn.tree import DecisionTreeClassifier
>>> from sklearn.preprocessing import OneHotEncoder
>>> from sklearn.impute import SimpleImputer
>>> from sklearn.pipeline import make_pipeline
>>> from sklearn.compose import make_column_transformer
>>> from sportsbet.evaluation import ClassifierBettor
>>> from sportsbet.datasets import SoccerDataLoader
>>> # Select only backtesting data for the Italian league and years 2020, 2021
>>> param_grid = {'league': ['Italy'], 'year': [2020, 2021]}
>>> dataloader = SoccerDataLoader(param_grid)
>>> # Select the odds of Pinnacle bookmaker
>>> X, Y, O = dataloader.extract_train_data(
... odds_type='pinnacle',
... drop_na_thres=1.0
... )
>>> # Create a pipeline to handle categorical features and missing values
>>> clf_pipeline = make_pipeline(
... make_column_transformer(
... (OneHotEncoder(handle_unknown='ignore'), ['league', 'home_team', 'away_team']),
... remainder='passthrough'
... ),
... SimpleImputer(),
... DecisionTreeClassifier(random_state=0)
... )
>>> # Backtest the bettor
>>> bettor = ClassifierBettor(clf_pipeline)
>>> backtest(bettor, X, Y, O)
Training Start ... Yield percentage per bet (away_win__full_time_goals)
...

Source code in src/sportsbet/evaluation/_classifier.py

def __init__(
    self: Self,
    classifier: BaseEstimator,
    betting_markets: list[str] | None = None,
    init_cash: float | None = None,
    stake: float | None = None,
) -> None:
    super().__init__(betting_markets, init_cash, stake)
    self.classifier = classifier

bet(X, O)

Predict the value bets for the provided input data and odds.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required
O	DataFrame	The odds data.	required

Returns:

Name	Type	Description
B	BoolData	The value bets.

Source code in src/sportsbet/evaluation/_classifier.py

def bet(self: Self, X: pd.DataFrame, O: pd.DataFrame) -> BoolData:
    """Predict the value bets for the provided input data and odds.

    Args:
        X:
            The input data.

        O:
            The odds data.

    Returns:
        B:
            The value bets.
    """
    return super().bet(X, O)

fit(X, Y, O=None)

Fit the bettor to the input data and multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required
Y	DataFrame	The multi-output targets.	required
O	DataFrame | None	The odds data.	None

Returns:

Name	Type	Description
self	Self	The fitted bettor object.

Source code in src/sportsbet/evaluation/_classifier.py

def fit(self: Self, X: pd.DataFrame, Y: pd.DataFrame, O: pd.DataFrame | None = None) -> Self:
    """Fit the bettor to the input data and multi-output targets.

    Args:
        X:
            The input data.

        Y:
            The multi-output targets.

        O:
            The odds data.

    Returns:
        self:
            The fitted bettor object.
    """
    return super().fit(X, Y, O)

predict(X)

Predict class labels for multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required

Returns:

Name	Type	Description
Y	BoolData	The positive class labels.

Source code in src/sportsbet/evaluation/_classifier.py

def predict(self: Self, X: pd.DataFrame) -> BoolData:
    """Predict class labels for multi-output targets.

    Args:
        X:
            The input data.

    Returns:
        Y:
            The positive class labels.
    """
    return super().predict(X)

predict_proba(X)

Predict class probabilities for multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required

Returns:

Name	Type	Description
Y	Data	The positive class probabilities.

Source code in src/sportsbet/evaluation/_classifier.py

def predict_proba(self: Self, X: pd.DataFrame) -> Data:
    """Predict class probabilities for multi-output targets.

    Args:
        X:
            The input data.

    Returns:
        Y:
            The positive class probabilities.
    """
    return super().predict_proba(X)

OddsComparisonBettor(odds_types=None, alpha=0.05, betting_markets=None, init_cash=None, stake=None)

Bases: _BaseBettor

Bettor based on comparison of odds.

It implements the betting strategy as described in the paper Beating the bookies with their own numbers. Predicted probabilities of events are based on the average of selected odds types for the corresponding events, adjusted by a constant value called alpha. You can read more in the user guide.

Parameters:

Name	Type	Description	Default
odds_types	list[str] | None	The odds types to use for the calculation of concensus probabilities. The default value corresponds to 'market_average' if this odds type exists or the average of all the other odds columns if 'market_average' is missing.	None
alpha	float	An adjustment term that corresponds to the difference between the consensus and real probabilities.	0.05
betting_markets	list[str] | None	Select the betting markets from the ones included in the data.	None
init_cash	float | None	The initial cash to use when betting.	None
stake	float | None	The stake of each bet.	None

Attributes:

Name	Type	Description
odds_types_	Index	The checked value of the odds types.
alpha_	float	The checked value of the alpha parameter.
output_keys_	list[str]	The keys of the output columns. They are used to identify the consensus columns.
backtesting_results_	DataFrame	The backtesting resuts.

Examples:

>>> from sportsbet.evaluation import OddsComparisonBettor
>>> from sportsbet.datasets import SoccerDataLoader
>>> # Select only backtesting data for the Italian and Spanish leagues and years 2019 - 2022
>>> param_grid = {'league': ['Italy', 'Spain'], 'year': [2019, 2020, 2021, 2022]}
>>> dataloader = SoccerDataLoader(param_grid)
>>> # Select the market maximum odds
>>> X, Y, O = dataloader.extract_train_data(
... odds_type='market_maximum',
... )
>>> # Backtest the bettor
>>> bettor = OddsComparisonBettor(alpha=0.03)
>>> backtest(bettor, X, Y, O)
Training Start ... Yield percentage per bet (away_win__full_time_goals)
...

Source code in src/sportsbet/evaluation/_rules.py

def __init__(
    self: Self,
    odds_types: list[str] | None = None,
    alpha: float = 0.05,
    betting_markets: list[str] | None = None,
    init_cash: float | None = None,
    stake: float | None = None,
) -> None:
    super().__init__(betting_markets, init_cash, stake)
    self.odds_types = odds_types
    self.alpha = alpha

bet(X, O)

Predict the value bets for the provided input data and odds.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required
O	DataFrame	The odds data.	required

Returns:

Name	Type	Description
B	BoolData	The value bets.

Source code in src/sportsbet/evaluation/_rules.py

def bet(self: Self, X: pd.DataFrame, O: pd.DataFrame) -> BoolData:
    """Predict the value bets for the provided input data and odds.

    Args:
        X:
            The input data.

        O:
            The odds data.

    Returns:
        B:
            The value bets.
    """
    return super().bet(X, O)

fit(X, Y, O=None)

Fit the bettor to the input data and multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required
Y	DataFrame	The multi-output targets.	required
O	DataFrame | None	The odds data.	None

Returns:

Name	Type	Description
self	Self	The fitted bettor object.

Source code in src/sportsbet/evaluation/_rules.py

def fit(self: Self, X: pd.DataFrame, Y: pd.DataFrame, O: pd.DataFrame | None = None) -> Self:
    """Fit the bettor to the input data and multi-output targets.

    Args:
        X:
            The input data.

        Y:
            The multi-output targets.

        O:
            The odds data.

    Returns:
        self:
            The fitted bettor object.
    """
    return super().fit(X, Y, O)

predict(X)

Predict class labels for multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required

Returns:

Name	Type	Description
Y	BoolData	The positive class labels.

Source code in src/sportsbet/evaluation/_rules.py

def predict(self: Self, X: pd.DataFrame) -> BoolData:
    """Predict class labels for multi-output targets.

    Args:
        X:
            The input data.

    Returns:
        Y:
            The positive class labels.
    """
    return super().predict(X)

predict_proba(X)

Predict class probabilities for multi-output targets.

Parameters:

Name	Type	Description	Default
X	DataFrame	The input data.	required

Returns:

Name	Type	Description
Y	Data	The positive class probabilities.

Source code in src/sportsbet/evaluation/_rules.py

def predict_proba(self: Self, X: pd.DataFrame) -> Data:
    """Predict class probabilities for multi-output targets.

    Args:
        X:
            The input data.

    Returns:
        Y:
            The positive class probabilities.
    """
    return super().predict_proba(X)

backtest(bettor, X, Y, O, cv=None, n_jobs=-1, verbose=0)

Backtest the bettor.

Parameters:

Name	Type	Description	Default
bettor	_BaseBettor	The bettor object.	required
X	DataFrame	The input data. Each row of X represents information that is available before the start of a specific match. The index should be of type datetime, named as 'date'.	required
Y	DataFrame	The multi-output targets. Each row of Y represents information that is available after the end of a specific event. The column names follow the convention for the output data Y of the method extract_train_data of dataloaders.	required
O	DataFrame	The odds data. The column names follow the convention for the odds data O of the method extract_train_data of dataloaders.	required
cv	TimeSeriesSplit | None	Provides train/test indices to split time series data samples that are observed at fixed time intervals, in train/test sets. The default value of the parameter is None, corresponding to the default TimeSeriesSplit object.	None
n_jobs	int	Number of CPU cores to use when parallelizing the backtesting runs. The default value of -1 means using all processors.	-1
verbose	int	The verbosity level.	0

Returns:

Name	Type	Description
results	DataFrame	The backtesting results.

Source code in src/sportsbet/evaluation/_model_selection.py

def backtest(
    bettor: _BaseBettor,
    X: pd.DataFrame,
    Y: pd.DataFrame,
    O: pd.DataFrame,
    cv: TimeSeriesSplit | None = None,
    n_jobs: int = -1,
    verbose: int = 0,
) -> pd.DataFrame:
    """Backtest the bettor.

    Args:
        bettor:
            The bettor object.

        X:
            The input data. Each row of `X` represents information that is available
            before the start of a specific match. The index should be of type
            `datetime`, named as `'date'`.

        Y:
            The multi-output targets. Each row of `Y` represents information
            that is available after the end of a specific event. The column
            names follow the convention for the output data `Y` of the method
            `extract_train_data` of dataloaders.

        O:
            The odds data. The column names follow the convention for the odds
            data `O` of the method `extract_train_data` of dataloaders.

        cv:
            Provides train/test indices to split time series data samples
            that are observed at fixed time intervals, in train/test sets. The
            default value of the parameter is `None`, corresponding to the default
            `TimeSeriesSplit` object.

        n_jobs:
            Number of CPU cores to use when parallelizing the backtesting runs.
            The default value of `-1` means using all processors.

        verbose:
            The verbosity level.

    Returns:
        results:
            The backtesting results.
    """

    # Check data
    check_consistent_length(X, Y, O)
    if not isinstance(X, pd.DataFrame) or not isinstance(X.index, pd.DatetimeIndex):
        error_msg = 'Input data `X` should be pandas dataframe with a date index.'
        raise TypeError(error_msg)
    if not isinstance(Y, pd.DataFrame):
        error_msg = 'Output data `Y` should be pandas dataframe.'
        raise TypeError(error_msg)
    if not isinstance(O, pd.DataFrame):
        error_msg = 'Odds data `O` should be pandas dataframe.'
        raise TypeError(error_msg)

    # Sort data
    indices = np.argsort(X.index)
    X, Y, O = X.iloc[indices], Y.iloc[indices], O.iloc[indices]

    # Check cross validator
    if cv is None:
        cv = TimeSeriesSplit()
    if not isinstance(cv, TimeSeriesSplit):
        error_msg = 'Parameter `cv` should be a TimeSeriesSplit cross-validator object.'
        raise TypeError(error_msg)

    # Calculate results
    results = Parallel(n_jobs=n_jobs, verbose=verbose)(
        delayed(_fit_bet)(train_ind, test_ind, bettor, X, Y, O) for train_ind, test_ind in cv.split(X)
    )
    results = pd.DataFrame(results).set_index(['Training start', 'Training end', 'Testing start', 'Testing end'])

    return results

load_bettor(path)

Load the bettor object.

Parameters:

Name	Type	Description	Default
path	str	The path of the bettor pickled file.	required

Returns:

Name	Type	Description
bettor	_BaseBettor	The bettor object.

Source code in src/sportsbet/evaluation/_base.py

def load_bettor(path: str) -> _BaseBettor:
    """Load the bettor object.

    Args:
        path:
            The path of the bettor pickled file.

    Returns:
        bettor:
            The bettor object.
    """
    with Path(path).open('rb') as file:
        bettor = cloudpickle.load(file)
    return bettor

save_bettor(bettor, path)

Save the bettor object.

Parameters:

Name	Type	Description	Default
bettor	_BaseBettor	The bettor object.	required
path	str	The path to save the object.	required

Returns:

Name	Type	Description
self	None	The bettor object.

Source code in src/sportsbet/evaluation/_base.py

def save_bettor(bettor: _BaseBettor, path: str) -> None:
    """Save the bettor object.

    Args:
        bettor:
            The bettor object.

        path:
            The path to save the object.

    Returns:
        self:
            The bettor object.
    """
    with Path(path).open('wb') as file:
        cloudpickle.dump(bettor, file)