U }h<$@sdZddlZddlZddlmZddlmZddlmZddl m Z ddl Z ddl mZdd lmZdd lmZmZmZdd lmZdd lmZmZmZmZmZmZdd lmZddl m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7ddl8m9Z9m:Z:m;Z;mZ>m?Z?m@Z@mAZAddZBGdddZCGdddeZDGdddeDZEGdddeDZFGdddeDZGedeHeIdgiddd d!ZJGd"d#d#ZKd$d%ZLeeIgd&gd&gd&gd'dddd(d(d)d*d+ZMeMe&ZNeMe4ZOeMe+d(d,ZPeMe0d(d,ZQeMe1d(d,ZReMe,d(d,ZSeMe-d(d,ZTeMe2d(d,ZUeMe0d(d(d-ZVeMe/d(d,ZWeMe.d(d,ZXeMe!ZYeMe#ZZeMe*Z[d.d/Z\d0d1Z]eMe\Z^eMe]d(d,Z_eMe7ddd2Z`eMe6ddd2ZaeMe"dd3ZbeMe6dd4d5ZceMe6dd4d6d7ZdeMe6dd8d5ZeeMe6dd8d6d7ZfeMe)d(dd9ZgeMe$d(dd9ZheMe$d(dd9ZieMe:ZjeMe@ZkeMe=ZleMe;ZmeMeAZneMe>ZoeMe9ZpeMe?ZqeMe<ZreseNeOePe[eUeSeTeQeReVeWeXeYe`eaeeecefedeZebegehe^e_ejekelemeneoepeqerd:"Ztd;d<Zud=e3fd>e5fd?e'fd@e(ffD]D\ZvZweMewdAdBetev<dCD]$ZxdDyevexZzeMewdexdEetez<qqeedFgee{eueIdgd&gdGdddKd(dHdIdJZ|dS)LaH The :mod:`sklearn.metrics.scorer` submodule implements a flexible interface for model selection and evaluation using arbitrary score functions. A scorer object is a callable that can be passed to :class:`~sklearn.model_selection.GridSearchCV` or :func:`sklearn.model_selection.cross_val_score` as the ``scoring`` parameter, to specify how a model should be evaluated. The signature of the call is ``(estimator, X, y)`` where ``estimator`` is the model to be evaluated, ``X`` is the test data and ``y`` is the ground truth labeling (or ``None`` in the case of unsupervised models). N)Counter)partial) signature) format_exc) is_regressorBunch) HasMethods StrOptionsvalidate_params_get_response_values)MetadataRequestMetadataRouter_MetadataRequester_routing_enabledget_routing_for_objectprocess_routing)type_of_target)accuracy_scoreaverage_precision_scorebalanced_accuracy_scorebrier_score_lossclass_likelihood_ratiosexplained_variance_scoref1_score jaccard_scorelog_lossmatthews_corrcoef max_errormean_absolute_errormean_absolute_percentage_errormean_gamma_deviancemean_poisson_deviancemean_squared_errormean_squared_log_errormedian_absolute_errorprecision_scorer2_score recall_score roc_auc_scoretop_k_accuracy_score) adjusted_mutual_info_scoreadjusted_rand_scorecompleteness_scorefowlkes_mallows_scorehomogeneity_scoremutual_info_scorenormalized_mutual_info_score rand_scorev_measure_scorecOsH|dk r||kr||St|f|d|i|\}}|dk rD|||<|S)z/Call estimator with method and args and kwargs.Nresponse_methodr )cache estimatorr7argskwargsresult_r>;/tmp/pip-unpacked-wheel-ig1s1lm8/sklearn/metrics/_scorer.py _cached_callQs r@c@s6eZdZdZddddZddZdd Zd d Zd S) _MultimetricScoreraCallable for multimetric scoring used to avoid repeated calls to `predict_proba`, `predict`, and `decision_function`. `_MultimetricScorer` will return a dictionary of scores corresponding to the scorers in the dictionary. Note that `_MultimetricScorer` can be created with a dictionary with one key (i.e. only one actual scorer). Parameters ---------- scorers : dict Dictionary mapping names to callable scorers. raise_exc : bool, default=True Whether to raise the exception in `__call__` or not. If set to `False` a formatted string of the exception details is passed as result of the failing scorer. T) raise_exccCs||_||_dSN)_scorers _raise_exc)selfscorersrBr>r>r?__init__ssz_MultimetricScorer.__init__c si}||rind}tt|}tr4t|d}ntffdd|jD}|jD]\}} zLt| t r| j ||f|| |j } n| |f|| |j } | ||<WqXt k r} z|jr| n t||<W5d} ~ XYqXXqX|S)z!Evaluate predicted target values.Nscorecsi|]}|tdqS)rIr).0namer;r>r? sz/_MultimetricScorer.__call__..) _use_cacherr@rrr rDitems isinstance _BaseScorer_scoregetrI ExceptionrEr) rFr9r:r;Zscoresr8Z cached_callZ routed_paramsrLscorerrIer>rMr?__call__ws4    z_MultimetricScorer.__call__cst|jdkrdStdd|jDtfddtttfDrJdStr|t|rftrfdStr|t |ds|dSdS) aReturn True if using a cache is beneficial. Caching may be beneficial when one of these conditions holds: - `_ProbaScorer` will be called twice. - `_PredictScorer` will be called twice. - `_ThresholdScorer` will be called twice. - `_ThresholdScorer` and `_PredictScorer` are called and estimator is a regressor. - `_ThresholdScorer` and `_ProbaScorer` are called and estimator does not have a `decision_function` attribute. rFcSsg|] }t|qSr>)type)rKvr>r>r? sz1_MultimetricScorer._use_cache..c3s|]}|dkVqdS)rNr>)rKZ known_typecounterr>r? sz0_MultimetricScorer._use_cache..Tdecision_function) lenrDrvaluesany_PredictScorer _ProbaScorer_ThresholdScorerrhasattrrFr9r>r\r?rOs  z_MultimetricScorer._use_cachecCs t|jjdjf|jddiS)abGet metadata routing of this object. Please check :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.3 Returns ------- routing : MetadataRouter A :class:`~utils.metadata_routing.MetadataRouter` encapsulating routing information. ownerZmethod_mappingrI)r __class____name__addrDrFr>r>r?get_metadata_routings z'_MultimetricScorer.get_metadata_routingN)rk __module__ __qualname____doc__rHrXrOrnr>r>r>r?rA`s rAc@sFeZdZddZddZddZddd Zd d Zd d ZddZ dS)rRcCs||_||_||_dSrC)_kwargs _score_func_sign)rF score_funcsignr;r>r>r?rHsz_BaseScorer.__init__cCs6d|jkr|jdSt|jj}d|kr2|djSdS)N pos_label)rrrrs parametersdefault)rFZscore_func_paramsr>r>r?_get_pos_labels     z_BaseScorer._get_pos_labelcCs@ddd|jD}d|jj|jdkr0dnd||fS)NcSs$g|]\}}dt|t|fqS)z, %s=%s)str)rKkrZr>r>r?r[sz(_BaseScorer.__repr__..zmake_scorer(%s%s%s%s)rz, greater_is_better=False)joinrrrPrsrkrt _factory_args)rFZ kwargs_stringr>r>r?__repr__sz_BaseScorer.__repr__NcKsF|rtstdt|}|dk r,||d<|jttd|||f|S)aEvaluate predicted target values for X relative to y_true. Parameters ---------- estimator : object Trained estimator to use for scoring. Must have a predict_proba method; the output of that is used to compute the score. X : {array-like, sparse matrix} Test data that will be fed to estimator.predict. y_true : array-like Gold standard target values for X. sample_weight : array-like of shape (n_samples,), default=None Sample weights. **kwargs : dict Other parameters passed to the scorer. Refer to :func:`set_score_request` for more details. Only available if `enable_metadata_routing=True`. See the :ref:`User Guide `. .. versionadded:: 1.3 Returns ------- score : float Score function applied to prediction of estimator on X. zbkwargs is only supported if enable_metadata_routing=True. See the User Guide for more information.N sample_weight)r ValueErrorcopydeepcopyrSrr@)rFr9Xy_truerr;rrr>r>r?rXs  z_BaseScorer.__call__cCsdS)z2Return non-default make_scorer arguments for repr.r{r>rmr>r>r?r sz_BaseScorer._factory_argscCsJ|jdkrtn t|j}||}|rFt|d|tdS)zWarn if there is any overlap between ``self._kwargs`` and ``kwargs``. This method is intended to be used to check for overlap between ``self._kwargs`` and ``kwargs`` passed as metadata. Nz Overlapping parameters are: )rrsetkeys intersectionwarningswarn UserWarning)rFmessager;rroverlapr>r>r? _warn_overlaps z_BaseScorer._warn_overlapcKsF|jd|dt|jjd|_|D]\}}|jjj||dq&|S)awSet requested parameters by the scorer. Please see :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.3 Parameters ---------- kwargs : dict Arguments should be of the form ``param_name=alias``, and `alias` can be one of ``{True, False, None, str}``. zYou are setting metadata request for parameters which are already set as kwargs for this metric. These set values will be overridden by passed metadata if provided. Please pass them either as metadata or kwargs to `make_scorer`.rr;rh)paramalias)rrrjrkZ_metadata_requestrPrIZ add_request)rFr;rrr>r>r?set_score_requests z_BaseScorer.set_score_request)N) rkrorprHrzrrXrrrr>r>r>r?rRs , rRc@seZdZddZdS)rccKs:|jd|d||d|}|j|}|j|j||f|S)aEvaluate predicted target values for X relative to y_true. Parameters ---------- method_caller : callable Returns predictions given an estimator, method name, and other arguments, potentially caching results. estimator : object Trained estimator to use for scoring. Must have a `predict` method; the output of that is used to compute the score. X : {array-like, sparse matrix} Test data that will be fed to estimator.predict. y_true : array-like Gold standard target values for X. **kwargs : dict Other parameters passed to the scorer. Refer to :func:`set_score_request` for more details. .. versionadded:: 1.3 Returns ------- score : float Score function applied to prediction of estimator on X. There is an overlap between set kwargs of this scorer instance and passed metadata. Please pass them either as kwargs to `make_scorer` or metadata, but not both.rpredict)rrrrtrs)rF method_callerr9rrr;y_predscoring_kwargsr>r>r?rS;s  z_PredictScorer._scoreN)rkrorprSr>r>r>r?rc:srcc@seZdZddZddZdS)rdcKsB|jd|d||d||d}|j|}|j|j||f|S)aEvaluate predicted probabilities for X relative to y_true. Parameters ---------- method_caller : callable Returns predictions given an estimator, method name, and other arguments, potentially caching results. clf : object Trained classifier to use for scoring. Must have a `predict_proba` method; the output of that is used to compute the score. X : {array-like, sparse matrix} Test data that will be fed to clf.predict_proba. y : array-like Gold standard target values for X. These must be class labels, not probabilities. **kwargs : dict Other parameters passed to the scorer. Refer to :func:`set_score_request` for more details. .. versionadded:: 1.3 Returns ------- score : float Score function applied to prediction of estimator on X. rr predict_probarw)rrzrrrtrs)rFrclfryr;rrr>r>r?rSgs  z_ProbaScorer._scorecCsdS)Nz, needs_proba=Truer>rmr>r>r?rsz_ProbaScorer._factory_argsNrkrorprSrr>r>r>r?rdfs,rdc@seZdZddZddZdS)rec Ks|jd|dt|}|dkr,td|t|rB||d|}n|}z4||d||d}t|tr|t dd |Dj }WnHt t fk r||d ||d}t|trt d d |Dj }YnX|j |} |j|j||f| S) a'Evaluate decision function output for X relative to y_true. Parameters ---------- method_caller : callable Returns predictions given an estimator, method name, and other arguments, potentially caching results. clf : object Trained classifier to use for scoring. Must have either a decision_function method or a predict_proba method; the output of that is used to compute the score. X : {array-like, sparse matrix} Test data that will be fed to clf.decision_function or clf.predict_proba. y : array-like Gold standard target values for X. These must be class labels, not decision function values. **kwargs : dict Other parameters passed to the scorer. Refer to :func:`set_score_request` for more details. .. versionadded:: 1.3 Returns ------- score : float Score function applied to prediction of estimator on X. rr)binaryzmultilabel-indicatorz{0} format is not supportedrr_rcSsg|]}|qSr>r>rKpr>r>r?r[sz+_ThresholdScorer._score..rcSsg|]}|dddfqS)Nr>rr>r>r?r[s)rrrformatrrzrQlistnpZvstackTNotImplementedErrorAttributeErrorrrrtrs) rFrrrrr;Zy_typerrwrr>r>r?rSs(!    z_ThresholdScorer._scorecCsdS)Nz, needs_threshold=Truer>rmr>r>r?rsz_ThresholdScorer._factory_argsNrr>r>r>r?resArescoringT)Zprefer_skip_nested_validationcCsHt|tr@ztt|}WqDtk r<td|YqDXn|}|S)azGet a scorer from string. Read more in the :ref:`User Guide `. :func:`~sklearn.metrics.get_scorer_names` can be used to retrieve the names of all available scorers. Parameters ---------- scoring : str, callable or None Scoring method as string. If callable it is returned as is. If None, returns None. Returns ------- scorer : callable The scorer. Notes ----- When passed a string, this function always returns a copy of the scorer object. Calling `get_scorer` twice for the same scorer results in two separate scorer objects. z]%r is not a valid scoring value. Use sklearn.metrics.get_scorer_names() to get valid options.)rQr|rr_SCORERSKeyErrorr)rrVr>r>r? get_scorers  rc@s$eZdZddZddZddZdS)_PassthroughScorercCs ||_dSrC) _estimatorrgr>r>r?rH sz_PassthroughScorer.__init__cOs |j||S)z!Method that wraps estimator.scorerJ)rFr9r:r;r>r>r?rX sz_PassthroughScorer.__call__cCs"t|jjjd}t|jj|_|S)a\Get requested data properties. Please check :ref:`User Guide ` on how the routing mechanism works. .. versionadded:: 1.3 Returns ------- routing : MetadataRouter A :class:`~utils.metadata_routing.MetadataRouter` encapsulating routing information. rh)rrrjrkrrI)rFresr>r>r?rnsz'_PassthroughScorer.get_metadata_routingN)rkrorprHrXrnr>r>r>r?r src srd|d}t|tttfrd}z t|}Wn,tk rX}zt||W5d}~XYnXt|t|kr~t|d|n|t|dkrtdd|Dstd d|Drt|d |nt|d |fd d |D}nt|d|nrt|t rft|}tdd|Ds2td|t|dkrNtd|fdd | D}nt||S)aCheck the scoring parameter in cases when multiple metrics are allowed. Parameters ---------- estimator : sklearn estimator instance The estimator for which the scoring will be applied. scoring : list, tuple or dict Strategy to evaluate the performance of the cross-validated model on the test set. The possibilities are: - a list or tuple of unique strings; - a callable returning a dictionary where they keys are the metric names and the values are the metric scores; - a dictionary with metric names as keys and callables a values. See :ref:`multimetric_grid_search` for an example. Returns ------- scorers_dict : dict A dict mapping each scorer name to its validated scorer. zscoring is invalid (got zh). Refer to the scoring glossary for details: https://scikit-learn.org/stable/glossary.html#term-scoringzFThe list/tuple elements must be unique strings of predefined scorers. Nz2 Duplicate elements were found in the given list. rcss|]}t|tVqdSrCrQr|rKr}r>r>r?r^Vsz-_check_multimetric_scoring..css|]}t|VqdSrC)callablerr>r>r?r^Wszi One or more of the elements were callables. Use a dict of score name mapped to the scorer callable. Got z4 Non-string types were found in the given list. Got csi|]}|t|dqS)r check_scoring)rKrVr9r>r?rNcsz._check_multimetric_scoring..z Empty list was given. css|]}t|tVqdSrCrrr>r>r?r^kszCNon-string types were found in the keys of the given dict. scoring=zAn empty dict was passed. csi|]\}}|t|dqSrr)rKkeyrVrr>r?rNrs) rQrtupler TypeErrorrr`allrbdictrP)r9rZerr_msg_genericerr_msgrrWrGr>rr?_check_multimetric_scoring'sN         rboolean)rugreater_is_better needs_probaneeds_thresholdF)rrrcKs@|rdnd}|r|rtd|r&t}n|r0t}nt}||||S)a Make a scorer from a performance metric or loss function. This factory function wraps scoring functions for use in :class:`~sklearn.model_selection.GridSearchCV` and :func:`~sklearn.model_selection.cross_val_score`. It takes a score function, such as :func:`~sklearn.metrics.accuracy_score`, :func:`~sklearn.metrics.mean_squared_error`, :func:`~sklearn.metrics.adjusted_rand_score` or :func:`~sklearn.metrics.average_precision_score` and returns a callable that scores an estimator's output. The signature of the call is `(estimator, X, y)` where `estimator` is the model to be evaluated, `X` is the data and `y` is the ground truth labeling (or `None` in the case of unsupervised models). Read more in the :ref:`User Guide `. Parameters ---------- score_func : callable Score function (or loss function) with signature ``score_func(y, y_pred, **kwargs)``. greater_is_better : bool, default=True Whether `score_func` is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the `score_func`. needs_proba : bool, default=False Whether `score_func` requires `predict_proba` to get probability estimates out of a classifier. If True, for binary `y_true`, the score function is supposed to accept a 1D `y_pred` (i.e., probability of the positive class, shape `(n_samples,)`). needs_threshold : bool, default=False Whether `score_func` takes a continuous decision certainty. This only works for binary classification using estimators that have either a `decision_function` or `predict_proba` method. If True, for binary `y_true`, the score function is supposed to accept a 1D `y_pred` (i.e., probability of the positive class or the decision function, shape `(n_samples,)`). For example `average_precision` or the area under the roc curve can not be computed using discrete predictions alone. **kwargs : additional arguments Additional parameters to be passed to `score_func`. Returns ------- scorer : callable Callable object that returns a scalar score; greater is better. Notes ----- If `needs_proba=False` and `needs_threshold=False`, the score function is supposed to accept the output of :term:`predict`. If `needs_proba=True`, the score function is supposed to accept the output of :term:`predict_proba` (For binary `y_true`, the score function is supposed to accept probability of the positive class). If `needs_threshold=True`, the score function is supposed to accept the output of :term:`decision_function` or :term:`predict_proba` when :term:`decision_function` is not present. Examples -------- >>> from sklearn.metrics import fbeta_score, make_scorer >>> ftwo_scorer = make_scorer(fbeta_score, beta=2) >>> ftwo_scorer make_scorer(fbeta_score, beta=2) >>> from sklearn.model_selection import GridSearchCV >>> from sklearn.svm import LinearSVC >>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]}, ... scoring=ftwo_scorer) rrz@Set either needs_proba or needs_threshold to True, but not both.)rrdrerc)rurrrr;rvclsr>r>r? make_scorer{s^ r)r)rZsquaredcCst||dS)Nrrrrr>r>r?positive_likelihood_ratio srcCst||dS)Nrrrr>r>r?negative_likelihood_ratio sr)rr)rZovo)r multi_classweighted)rraverageZovr)rr)"Zexplained_varianceZr2r!r Zneg_median_absolute_errorZneg_mean_absolute_errorZ"neg_mean_absolute_percentage_errorZneg_mean_squared_errorZneg_mean_squared_log_errorZneg_root_mean_squared_errorZneg_mean_poisson_devianceZneg_mean_gamma_devianceZaccuracyZtop_k_accuracyZroc_aucZ roc_auc_ovrZ roc_auc_ovoZroc_auc_ovr_weightedZroc_auc_ovo_weightedZbalanced_accuracyZaverage_precisionZ neg_log_lossZneg_brier_scorerZneg_negative_likelihood_ratior/r5r2r0r6r3r.r4r1cCs ttS)zGet the names of all available scorers. These names can be passed to :func:`~sklearn.metrics.get_scorer` to retrieve the scorer object. Returns ------- list of str Names of all available scorers. )sortedrrr>r>r>r?get_scorer_namesds rZ precisionZrecallf1Zjaccardr)r)macromicroZsamplesrz{0}_{1})rwrZfit)r9r allow_none)rcCst|trt|St|rbt|dd}t|drZ|drZ|dsZ|dsZtd|t|S|dkrt|dr|t|S|rdSt d |dS) a3Determine scorer from user options. A TypeError will be thrown if the estimator cannot be scored. Parameters ---------- estimator : estimator object implementing 'fit' The object to use to fit the data. scoring : str or callable, default=None A string (see model evaluation documentation) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. If None, the provided estimator object's `score` method is used. allow_none : bool, default=False If no scoring is specified and the estimator has no score function, we can either return None or raise an exception. Returns ------- scoring : callable A scorer callable object / function with signature ``scorer(estimator, X, y)``. roN startswithzsklearn.metrics.zsklearn.metrics._scorerzsklearn.metrics.tests.zscoring value %r looks like it is a metric function rather than a scorer. A scorer should require an estimator as its first parameter. Please use `make_scorer` to convert a metric to a scorer.rIziIf no scoring is specified, the estimator passed should have a 'score' method. The estimator %r does not.) rQr|rrgetattrrfrrrr)r9rrmoduler>r>r?r~s8"   r)N)}rqrr collectionsr functoolsrinspectr tracebackrZnumpyrbaserutilsr Zutils._param_validationr r r Zutils._responserZutils.metadata_routingrrrrrrZutils.multiclassrr{rrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-Zclusterr.r/r0r1r2r3r4r5r6r@rArRrcrdrer|rrrrrZexplained_variance_scorerZ r2_scorerZmax_error_scorerZneg_mean_squared_error_scorerZ!neg_mean_squared_log_error_scorerZneg_mean_absolute_error_scorerZ)neg_mean_absolute_percentage_error_scorerZ neg_median_absolute_error_scorerZ"neg_root_mean_squared_error_scorerZ neg_mean_poisson_deviance_scorerZneg_mean_gamma_deviance_scorerZaccuracy_scorerZbalanced_accuracy_scorerZmatthews_corrcoef_scorerrrZ positive_likelihood_ratio_scorerZ$neg_negative_likelihood_ratio_scorerZtop_k_accuracy_scorerZroc_auc_scorerZaverage_precision_scorerZroc_auc_ovo_scorerZroc_auc_ovo_weighted_scorerZroc_auc_ovr_scorerZroc_auc_ovr_weighted_scorerZneg_log_loss_scorerZneg_brier_score_scorerZbrier_score_loss_scorerZadjusted_rand_scorerZ rand_scorerZhomogeneity_scorerZcompleteness_scorerZv_measure_scorerZmutual_info_scorerZadjusted_mutual_info_scorerZnormalized_mutual_info_scorerZfowlkes_mallows_scorerrrrrLZmetricrrZqualified_namerrr>r>r>r?sp         d, gs,1F &T d   '