o Eb@s\ddlmZddlmZddlZddlZddlmZm Z m Z ddl Z ddl mZddlmZddlmZdd lmZmZdd lmZmZmZmZmZdd lmZddlmmZdd l m!Z!dd l"m#Z#m$Z$m%Z%gdZ&eddZ'd_ddZ(GdddZ)ddZ*ddZ+d`ddZ,daddZ-dd Z.d!d"Z/dbd$d%Z0ed&d'Z1dcd(d)Z2d*d+Z3d,d-Z4ed.d/e5fd0e5fgZ6ddd3d4Z7ed5d/e5fd0e5fgZ8ded6d7Z9d8d9Z:d:d;Z;dfd=d>Zd`dAdBZ?d`dCdDZ@d`dEdFZAdGdHZBdIdJZC d`dKdLZD d`dMdNZE d`dOdPZFdQdRZGdSdTdUdd#dddVdWdXZHGdYdZdZZId[d\ZJd]d^ZKdS)g) namedtuple)make_dataclassN) combinations permutationsproduct)shgo) distributions)ConfidenceInterval)chi2norm)gammakvgammalncomb factorial)_wilcoxon_data)check_random_state)_Q_P_a_ij_Aij_Dij2)epps_singleton_2sampcramervonmisessomersd barnard_exactboschloo_exactcramervonmises_2samppermutation_test tukey_hsdEpps_Singleton_2sampResult statisticpvalueg?g?cCsZt|t|t|}}}|jdkrtd|j|jdkr+td|jt|t|}}|dks<|dkrDtd||t|sOtdt|sZtd||}|jdkrktd|jt|d  rwtd d d l m }|t ||fd }t |d |}tt||t||fj} tt||t||fj} tj| jdd} tj| jdd} ||| ||| } tj| }tj|}|d t|krtdtj| d dtj| d d}|t|jt||}t||dkr"dd|dd|d|d}||}t||}t||S)a Compute the Epps-Singleton (ES) test statistic. Test the null hypothesis that two samples have the same underlying probability distribution. Parameters ---------- x, y : array-like The two samples of observations to be tested. Input must not have more than one dimension. Samples can have different lengths. t : array-like, optional The points (t1, ..., tn) where the empirical characteristic function is to be evaluated. It should be positive distinct numbers. The default value (0.4, 0.8) is proposed in [1]_. Input must not have more than one dimension. Returns ------- statistic : float The test statistic. pvalue : float The associated p-value based on the asymptotic chi2-distribution. See Also -------- ks_2samp, anderson_ksamp Notes ----- Testing whether two samples are generated by the same underlying distribution is a classical question in statistics. A widely used test is the Kolmogorov-Smirnov (KS) test which relies on the empirical distribution function. Epps and Singleton introduce a test based on the empirical characteristic function in [1]_. One advantage of the ES test compared to the KS test is that is does not assume a continuous distribution. In [1]_, the authors conclude that the test also has a higher power than the KS test in many examples. They recommend the use of the ES test for discrete samples as well as continuous samples with at least 25 observations each, whereas `anderson_ksamp` is recommended for smaller sample sizes in the continuous case. The p-value is computed from the asymptotic distribution of the test statistic which follows a `chi2` distribution. If the sample size of both `x` and `y` is below 25, the small sample correction proposed in [1]_ is applied to the test statistic. The default values of `t` are determined in [1]_ by considering various distributions and finding good values that lead to a high power of the test in general. Table III in [1]_ gives the optimal values for the distributions tested in that study. The values of `t` are scaled by the semi-interquartile range in the implementation, see [1]_. References ---------- .. [1] T. W. Epps and K. J. Singleton, "An omnibus test for the two-sample problem using the empirical characteristic function", Journal of Statistical Computation and Simulation 26, p. 177--203, 1986. .. [2] S. J. Goerg and J. Kaiser, "Nonparametric testing of distributions - the Epps-Singleton two-sample test using the empirical characteristic function", The Stata Journal 9(3), p. 454--465, 2009. rz#x must be 1d, but x.ndim equals {}.z#y must be 1d, but y.ndim equals {}.zIx and y should have at least 5 elements, but len(x) = {} and len(y) = {}.z$x must not contain nonfinite values.z$y must not contain nonfinite values.z#t must be 1d, but t.ndim equals {}.rz&t must contain positive elements only.)iqr)rT)ZbiaszEstimated covariance matrix does not have full rank. This indicates a bad choice of the input t and the test might not be consistent.axis?gܿg333333$@g333333)npasarrayndim ValueErrorformatlenZisfiniteallZ less_equalany scipy.statsr%ZhstackreshapeZvstackZcosZsinTZcovZlinalgZpinvZ matrix_rankwarningswarnmeandotmaxr sfr)xytnxnynr%ZsigmatsZgxZgyZcov_xZcov_yZest_covZ est_cov_invrZg_diffwZcorrprG8/usr/lib/python3/dist-packages/scipy/stats/_hypotests.pyrsL"B    $$   $  rc@seZdZddZddZdS)CramerVonMisesResultcCs||_||_dSNr )selfr!r"rGrGrH__init__s zCramerVonMisesResult.__init__cCs|jjd|jd|jdS)Nz (statistic=z , pvalue=)) __class____name__r!r")rKrGrGrH__repr__szCramerVonMisesResult.__repr__N)rO __module__ __qualname__rLrPrGrGrGrHrIs rIcsddddfdd}t|}tj|dd}tj|d d}d }t|rT|||| tjt|d }|||||<t|d k||<|d 7}t|s)|S) a psi1 is defined in equation 1.10 in Csorgo, S. and Faraway, J. (1996). This implements a modified version by excluding the term V(x) / 12 (here: _cdf_cvm_inf(x) / 12) to avoid evaluating _cdf_cvm_inf(x) twice in _cdf_cvm. Implementation based on MAPLE code of Julian Faraway and R code of the function pCvM in the package goftest (v1.1.1), permission granted by Adrian Baddeley. Main difference in the implementation: the code here keeps adding terms of the series until the terms are small enough. cSsH|dd}td|td|}t| |dd|ttjS)Nr&???)rr,expsqrtpi)r>zbrGrGrH_ed2s (z_psi1_mod.._ed2cSsZ|dd}t| ttj}||dddtd|dtd|td|S)Nr&rSg@rTrU?)r,rWrXrYr)r>rZcrGrGrH_ed3s 6z_psi1_mod.._ed3c s&d|d}dt|}|d}|d}|t|dd|d|d|}t|dd|d|d |}d|dt|d d|d |d |}d |t|dd|d|d|} d |t|dd|d |d|} |||| | S)Nr&rrUr^?rSr] HrVr$ )r,rXr ) kr=mZsxZy1Zy2Ze1Ze2Ze3Ze4Ze5r\r`rGrH_Aks ,(400z_psi1_mod.._AkfloatZdtypeboolrrHz>)r,r- zeros_like ones_liker3rYr abs)r=rjtotcondrgrZrGrirH _psi1_mods   " rtcCst|}dd}tj|dd}tj|dd}d}t|r?||||}|||||<t|dk||<|d7}t|s|S) u0 Calculate the cdf of the Cramér-von Mises statistic (infinite sample size). See equation 1.2 in Csorgo, S. and Faraway, J. (1996). Implementation based on MAPLE code of Julian Faraway and R code of the function pCvM in the package goftest (v1.1.1), permission granted by Adrian Baddeley. Main difference in the implementation: the code here keeps adding terms of the series until the terms are small enough. The function is not expected to be accurate for large values of x, say x > 4, when the cdf is very close to 1. cSsvtt|dt|dtjdt|}d|d}|dd|}td|}|t|t| |S)NrarrVrSr&rT)r,rWrrYrXr)r=rgur>qr[rGrGrHterms 2  z_cdf_cvm_inf..termrkrlrmrrnr)r,r-rorpr3rq)r=rxrrrsrgrZrGrGrH _cdf_cvm_infs   rycCst|}|durt|}n5tj|dd}dd||k||dk@}t||ddd|t|||||<d|||dk<|jd krL|d S|S) u Calculate the cdf of the Cramér-von Mises statistic for a finite sample size n. If N is None, use the asymptotic cdf (n=inf). See equation 1.8 in Csorgo, S. and Faraway, J. (1996) for finite samples, 1.2 for the asymptotic cdf. The function is not expected to be accurate for large values of x, say x > 2, when the cdf is very close to 1 and it might return values > 1 in that case, e.g. _cdf_cvm(2.0, 12) = 1.0000027556716846. Nrkrlr+rdg@rr]rrG)r,r-ryrortr.)r=rBr>suprGrGrH_cdf_cvms  0 r{rGc Cst|tr tt|j}tt|}|jdkrt d|j dkr%t dt |}||g|R}dt d|ddd|}dd|t ||d}tddt||}t||dS) uIPerform the one-sample Cramér-von Mises test for goodness of fit. This performs a test of the goodness of fit of a cumulative distribution function (cdf) :math:`F` compared to the empirical distribution function :math:`F_n` of observed random variates :math:`X_1, ..., X_n` that are assumed to be independent and identically distributed ([1]_). The null hypothesis is that the :math:`X_i` have cumulative distribution :math:`F`. Parameters ---------- rvs : array_like A 1-D array of observed values of the random variables :math:`X_i`. cdf : str or callable The cumulative distribution function :math:`F` to test the observations against. If a string, it should be the name of a distribution in `scipy.stats`. If a callable, that callable is used to calculate the cdf: ``cdf(x, *args) -> float``. args : tuple, optional Distribution parameters. These are assumed to be known; see Notes. Returns ------- res : object with attributes statistic : float Cramér-von Mises statistic. pvalue : float The p-value. See Also -------- kstest, cramervonmises_2samp Notes ----- .. versionadded:: 1.6.0 The p-value relies on the approximation given by equation 1.8 in [2]_. It is important to keep in mind that the p-value is only accurate if one tests a simple hypothesis, i.e. the parameters of the reference distribution are known. If the parameters are estimated from the data (composite hypothesis), the computed p-value is not reliable. References ---------- .. [1] Cramér-von Mises criterion, Wikipedia, https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93von_Mises_criterion .. [2] Csorgo, S. and Faraway, J. (1996). The Exact and Asymptotic Distribution of Cramér-von Mises Statistics. Journal of the Royal Statistical Society, pp. 221-234. Examples -------- Suppose we wish to test whether data generated by ``scipy.stats.norm.rvs`` were, in fact, drawn from the standard normal distribution. We choose a significance level of alpha=0.05. >>> from scipy import stats >>> rng = np.random.default_rng() >>> x = stats.norm.rvs(size=500, random_state=rng) >>> res = stats.cramervonmises(x, 'norm') >>> res.statistic, res.pvalue (0.49121480855028343, 0.04189256516661377) The p-value 0.79 exceeds our chosen significance level, so we do not reject the null hypothesis that the observed sample is drawn from the standard normal distribution. Now suppose we wish to check whether the same samples shifted by 2.1 is consistent with being drawn from a normal distribution with a mean of 2. >>> y = x + 2.1 >>> res = stats.cramervonmises(y, 'norm', args=(2,)) >>> res.statistic, res.pvalue (0.07400330012187435, 0.7274595666160468) Here we have used the `args` keyword to specify the mean (``loc``) of the normal distribution to test the data against. This is equivalent to the following, in which we create a frozen normal distribution with mean 2.1, then pass its ``cdf`` method as an argument. >>> frozen_dist = stats.norm(loc=2) >>> res = stats.cramervonmises(y, frozen_dist.cdf) >>> res.statistic, res.pvalue (0.07400330012187435, 0.7274595666160468) In either case, we would reject the null hypothesis that the observed sample is drawn from a normal distribution with a mean of 2 (and default variance of 1) because the p-value 0.04 is less than our chosen significance level. rz2The sample must contain at least two observations.z#The sample must be one-dimensional.r&rdrr+r ) isinstancestrgetattrr cdfr,sortr-sizer/r.r1arangesumr;r{rI) ZrvsrargsZvalsrBZcdfvalsrvrErFrGrGrHrs ^     rcCs0tj|}|durtd|tj|tdS)z Distribution of counts of the Wilcoxon ranksum statistic r_plus (sum of ranks of positive differences). Returns an array with the counts/frequencies of all the possible ranks r = 0, ..., n*(n+1)/2 NzQThe exact distribution of the Wilcoxon test statistic is not implemented for n={}rl)rZCOUNTSgetr/r0r,arrayint)rBcntrGrGrH_get_wilcoxon_distrs rc Cs|jddks|jddkrtjtjfS|}t|}t|}|jddd}|jddd}|d||d|}|||d}dt|||d|}||} | dkrb|dfS|| d} dtt | } || fS)z=Calculate Kendall's tau-b and p-value from contingency table.rrr(r&rarS) shaper,nanrrrrr r<rq) ANAPAQASri2ZScj2 denominatorZtau numeratorZ s02_tau_bZrFrGrGrH_tau_bs   r two-sidedc Cs|jddks|jddkrtjtjfS|}|d}t|}t|}|jddd}||||}t|||d|}tjdd||d|d} Wd n1s]wYtj j | |\} } || fS) z7Calculate Somers' D and p-value from contingency table.rrr&r(ignore)dividerSraN) rr,rrrrrerrstatescipystatsZ _stats_pyZ_normtest_finish) r alternativerZNA2rrrdSr_rFrGrGrH _somers_ds r SomersDResult)r!r"tablecCst|t|}}|jdkr%|j|jkrtdtjj||d}n2|jdkrSt |dkr5tdt || t krCtd| djdkrPtd|}ntdt ||\}}t|||S) aCalculates Somers' D, an asymmetric measure of ordinal association. Like Kendall's :math:`\tau`, Somers' :math:`D` is a measure of the correspondence between two rankings. Both statistics consider the difference between the number of concordant and discordant pairs in two rankings :math:`X` and :math:`Y`, and both are normalized such that values close to 1 indicate strong agreement and values close to -1 indicate strong disagreement. They differ in how they are normalized. To show the relationship, Somers' :math:`D` can be defined in terms of Kendall's :math:`\tau_a`: .. math:: D(Y|X) = \frac{\tau_a(X, Y)}{\tau_a(X, X)} Suppose the first ranking :math:`X` has :math:`r` distinct ranks and the second ranking :math:`Y` has :math:`s` distinct ranks. These two lists of :math:`n` rankings can also be viewed as an :math:`r \times s` contingency table in which element :math:`i, j` is the number of rank pairs with rank :math:`i` in ranking :math:`X` and rank :math:`j` in ranking :math:`Y`. Accordingly, `somersd` also allows the input data to be supplied as a single, 2D contingency table instead of as two separate, 1D rankings. Note that the definition of Somers' :math:`D` is asymmetric: in general, :math:`D(Y|X) \neq D(X|Y)`. ``somersd(x, y)`` calculates Somers' :math:`D(Y|X)`: the "row" variable :math:`X` is treated as an independent variable, and the "column" variable :math:`Y` is dependent. For Somers' :math:`D(X|Y)`, swap the input lists or transpose the input table. Parameters ---------- x: array_like 1D array of rankings, treated as the (row) independent variable. Alternatively, a 2D contingency table. y: array_like, optional If `x` is a 1D array of rankings, `y` is a 1D array of rankings of the same length, treated as the (column) dependent variable. If `x` is 2D, `y` is ignored. alternative : {'two-sided', 'less', 'greater'}, optional Defines the alternative hypothesis. Default is 'two-sided'. The following options are available: * 'two-sided': the rank correlation is nonzero * 'less': the rank correlation is negative (less than zero) * 'greater': the rank correlation is positive (greater than zero) Returns ------- res : SomersDResult A `SomersDResult` object with the following fields: correlation : float The Somers' :math:`D` statistic. pvalue : float The p-value for a hypothesis test whose null hypothesis is an absence of association, :math:`D=0`. See notes for more information. table : 2D array The contingency table formed from rankings `x` and `y` (or the provided contingency table, if `x` is a 2D array) See Also -------- kendalltau : Calculates Kendall's tau, another correlation measure. weightedtau : Computes a weighted version of Kendall's tau. spearmanr : Calculates a Spearman rank-order correlation coefficient. pearsonr : Calculates a Pearson correlation coefficient. Notes ----- This function follows the contingency table approach of [2]_ and [3]_. *p*-values are computed based on an asymptotic approximation of the test statistic distribution under the null hypothesis :math:`D=0`. Theoretically, hypothesis tests based on Kendall's :math:`tau` and Somers' :math:`D` should be identical. However, the *p*-values returned by `kendalltau` are based on the null hypothesis of *independence* between :math:`X` and :math:`Y` (i.e. the population from which pairs in :math:`X` and :math:`Y` are sampled contains equal numbers of all possible pairs), which is more specific than the null hypothesis :math:`D=0` used here. If the null hypothesis of independence is desired, it is acceptable to use the *p*-value returned by `kendalltau` with the statistic returned by `somersd` and vice versa. For more information, see [2]_. Contingency tables are formatted according to the convention used by SAS and R: the first ranking supplied (``x``) is the "row" variable, and the second ranking supplied (``y``) is the "column" variable. This is opposite the convention of Somers' original paper [1]_. References ---------- .. [1] Robert H. Somers, "A New Asymmetric Measure of Association for Ordinal Variables", *American Sociological Review*, Vol. 27, No. 6, pp. 799--811, 1962. .. [2] Morton B. Brown and Jacqueline K. Benedetti, "Sampling Behavior of Tests for Correlation in Two-Way Contingency Tables", *Journal of the American Statistical Association* Vol. 72, No. 358, pp. 309--315, 1977. .. [3] SAS Institute, Inc., "The FREQ Procedure (Book Excerpt)", *SAS/STAT 9.2 User's Guide, Second Edition*, SAS Publishing, 2009. .. [4] Laerd Statistics, "Somers' d using SPSS Statistics", *SPSS Statistics Tutorials and Statistical Guides*, https://statistics.laerd.com/spss-tutorials/somers-d-using-spss-statistics.php, Accessed July 31, 2020. Examples -------- We calculate Somers' D for the example given in [4]_, in which a hotel chain owner seeks to determine the association between hotel room cleanliness and customer satisfaction. The independent variable, hotel room cleanliness, is ranked on an ordinal scale: "below average (1)", "average (2)", or "above average (3)". The dependent variable, customer satisfaction, is ranked on a second scale: "very dissatisfied (1)", "moderately dissatisfied (2)", "neither dissatisfied nor satisfied (3)", "moderately satisfied (4)", or "very satisfied (5)". 189 customers respond to the survey, and the results are cast into a contingency table with the hotel room cleanliness as the "row" variable and customer satisfaction as the "column" variable. +-----+-----+-----+-----+-----+-----+ | | (1) | (2) | (3) | (4) | (5) | +=====+=====+=====+=====+=====+=====+ | (1) | 27 | 25 | 14 | 7 | 0 | +-----+-----+-----+-----+-----+-----+ | (2) | 7 | 14 | 18 | 35 | 12 | +-----+-----+-----+-----+-----+-----+ | (3) | 1 | 3 | 2 | 7 | 17 | +-----+-----+-----+-----+-----+-----+ For example, 27 customers assigned their room a cleanliness ranking of "below average (1)" and a corresponding satisfaction of "very dissatisfied (1)". We perform the analysis as follows. >>> from scipy.stats import somersd >>> table = [[27, 25, 14, 7, 0], [7, 14, 18, 35, 12], [1, 3, 2, 7, 17]] >>> res = somersd(table) >>> res.statistic 0.6032766111513396 >>> res.pvalue 1.0007091191074533e-27 The value of the Somers' D statistic is approximately 0.6, indicating a positive correlation between room cleanliness and customer satisfaction in the sample. The *p*-value is very small, indicating a very small probability of observing such an extreme value of the statistic under the null hypothesis that the statistic of the entire population (from which our sample of 189 customers is drawn) is zero. This supports the alternative hypothesis that the true value of Somers' D for the population is nonzero. rz!Rankings must be of equal length.r&rz;All elements of the contingency table must be non-negative.z6All elements of the contingency table must be integer.z?At least two elements of the contingency table must be nonzero.z!x must be either a 1D or 2D array)r,rr.rr/rrZ contingencyZcrosstabr3astyperZnonzerorr)r=r>rrrrFrGrGrHrs"    rccsXt||}t||D]}t|}t||t}d||<||}||fVq dS)z Partition a set of indices into two fixed-length sets in all possible ways Partition a set of indices 0 ... nx + ny - 1 into two sets of length nx and ny in all possible ways (ignoring order of elements). FN)r,rrronesrm)r@rArZr_r=maskr>rGrGrH_all_partitions}s  rcCs4tt|dd}t|d||dddS)z'Compute all log combination of C(n, k).rNr')rr,r)rBZ gammaln_arrrGrGrH_compute_log_combinationssrBarnardExactResultr!r"T cCs0|dkr td|tj|tjd}|jdkstdt|dkr'tdd|jddvr5ttjdS|jdd\}}tj |d tjd d d }tj |d tjd d d }||||}} |r{||||} | d | d |d |} n|d ||| d | |} tj d d d t || t | } Wd n1swYd| || k<| |d|df} |dkrt| t| k}n|dkr| | k}n|dkr| | k}n d|}t|||}t|}t|}||||}tt|||fd|dd}tjt|j dd d}t| |S)aPerform a Barnard exact test on a 2x2 contingency table. Parameters ---------- table : array_like of ints A 2x2 contingency table. Elements should be non-negative integers. alternative : {'two-sided', 'less', 'greater'}, optional Defines the null and alternative hypotheses. Default is 'two-sided'. Please see explanations in the Notes section below. pooled : bool, optional Whether to compute score statistic with pooled variance (as in Student's t-test, for example) or unpooled variance (as in Welch's t-test). Default is ``True``. n : int, optional Number of sampling points used in the construction of the sampling method. Note that this argument will automatically be converted to the next higher power of 2 since `scipy.stats.qmc.Sobol` is used to select sample points. Default is 32. Must be positive. In most cases, 32 points is enough to reach good precision. More points comes at performance cost. Returns ------- ber : BarnardExactResult A result object with the following attributes. statistic : float The Wald statistic with pooled or unpooled variance, depending on the user choice of `pooled`. pvalue : float P-value, the probability of obtaining a distribution at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. See Also -------- chi2_contingency : Chi-square test of independence of variables in a contingency table. fisher_exact : Fisher exact test on a 2x2 contingency table. boschloo_exact : Boschloo's exact test on a 2x2 contingency table, which is an uniformly more powerful alternative to Fisher's exact test. Notes ----- Barnard's test is an exact test used in the analysis of contingency tables. It examines the association of two categorical variables, and is a more powerful alternative than Fisher's exact test for 2x2 contingency tables. Let's define :math:`X_0` a 2x2 matrix representing the observed sample, where each column stores the binomial experiment, as in the example below. Let's also define :math:`p_1, p_2` the theoretical binomial probabilities for :math:`x_{11}` and :math:`x_{12}`. When using Barnard exact test, we can assert three different null hypotheses : - :math:`H_0 : p_1 \geq p_2` versus :math:`H_1 : p_1 < p_2`, with `alternative` = "less" - :math:`H_0 : p_1 \leq p_2` versus :math:`H_1 : p_1 > p_2`, with `alternative` = "greater" - :math:`H_0 : p_1 = p_2` versus :math:`H_1 : p_1 \neq p_2`, with `alternative` = "two-sided" (default one) In order to compute Barnard's exact test, we are using the Wald statistic [3]_ with pooled or unpooled variance. Under the default assumption that both variances are equal (``pooled = True``), the statistic is computed as: .. math:: T(X) = \frac{ \hat{p}_1 - \hat{p}_2 }{ \sqrt{ \hat{p}(1 - \hat{p}) (\frac{1}{c_1} + \frac{1}{c_2}) } } with :math:`\hat{p}_1, \hat{p}_2` and :math:`\hat{p}` the estimator of :math:`p_1, p_2` and :math:`p`, the latter being the combined probability, given the assumption that :math:`p_1 = p_2`. If this assumption is invalid (``pooled = False``), the statistic is: .. math:: T(X) = \frac{ \hat{p}_1 - \hat{p}_2 }{ \sqrt{ \frac{\hat{p}_1 (1 - \hat{p}_1)}{c_1} + \frac{\hat{p}_2 (1 - \hat{p}_2)}{c_2} } } The p-value is then computed as: .. math:: \sum \binom{c_1}{x_{11}} \binom{c_2}{x_{12}} \pi^{x_{11} + x_{12}} (1 - \pi)^{t - x_{11} - x_{12}} where the sum is over all 2x2 contingency tables :math:`X` such that: * :math:`T(X) \leq T(X_0)` when `alternative` = "less", * :math:`T(X) \geq T(X_0)` when `alternative` = "greater", or * :math:`T(X) \geq |T(X_0)|` when `alternative` = "two-sided". Above, :math:`c_1, c_2` are the sum of the columns 1 and 2, and :math:`t` the total (sum of the 4 sample's element). The returned p-value is the maximum p-value taken over the nuisance parameter :math:`\pi`, where :math:`0 \leq \pi \leq 1`. This function's complexity is :math:`O(n c_1 c_2)`, where `n` is the number of sample points. References ---------- .. [1] Barnard, G. A. "Significance Tests for 2x2 Tables". *Biometrika*. 34.1/2 (1947): 123-138. :doi:`dpgkg3` .. [2] Mehta, Cyrus R., and Pralay Senchaudhuri. "Conditional versus unconditional exact tests for comparing two binomials." *Cytel Software Corporation* 675 (2003): 1-5. .. [3] "Wald Test". *Wikipedia*. https://en.wikipedia.org/wiki/Wald_test Examples -------- An example use of Barnard's test is presented in [2]_. Consider the following example of a vaccine efficacy study (Chan, 1998). In a randomized clinical trial of 30 subjects, 15 were inoculated with a recombinant DNA influenza vaccine and the 15 were inoculated with a placebo. Twelve of the 15 subjects in the placebo group (80%) eventually became infected with influenza whereas for the vaccine group, only 7 of the 15 subjects (47%) became infected. The data are tabulated as a 2 x 2 table:: Vaccine Placebo Yes 7 12 No 8 3 When working with statistical hypothesis testing, we usually use a threshold probability or significance level upon which we decide to reject the null hypothesis :math:`H_0`. Suppose we choose the common significance level of 5%. Our alternative hypothesis is that the vaccine will lower the chance of becoming infected with the virus; that is, the probability :math:`p_1` of catching the virus with the vaccine will be *less than* the probability :math:`p_2` of catching the virus without the vaccine. Therefore, we call `barnard_exact` with the ``alternative="less"`` option: >>> import scipy.stats as stats >>> res = stats.barnard_exact([[7, 12], [8, 3]], alternative="less") >>> res.statistic -1.894... >>> res.pvalue 0.03407... Under the null hypothesis that the vaccine will not lower the chance of becoming infected, the probability of obtaining test results at least as extreme as the observed data is approximately 3.4%. Since this p-value is less than our chosen significance level, we have evidence to reject :math:`H_0` in favor of the alternative. Suppose we had used Fisher's exact test instead: >>> _, pvalue = stats.fisher_exact([[7, 12], [8, 3]], alternative="less") >>> pvalue 0.0640... With the same threshold significance of 5%, we would not have been able to reject the null hypothesis in favor of the alternative. As stated in [2]_, Barnard's test is uniformly more powerful than Fisher's exact test because Barnard's test does not condition on any margin. Fisher's test should only be used when both sets of marginals are fixed. r6Number of points `n` must be strictly positive, found rlr&r&*The input `table` must be of shape (2, 2).*All values in `table` must be nonnegative.r(r+rr'rrZinvalidNrrrrrlessgreaterzG`alternative` should be one of {'two-sided', 'less', 'greater'}, found rsobolrZboundsrBZsampling_methodZa_minZa_max)r/r,r-int64rr3rrrrr5rrrXrqrr-_get_binomial_log_p_value_with_nuisance_paramcliprWfun)rrZpooledrB total_col_1 total_col_2x1x2Zp1Zp2rFZ variancesZwald_statisticZ wald_stat_obs index_arrmsg x1_sum_x2 x1_log_comb x2_log_combx1_sum_x2_log_combresultp_valuerGrGrHrsf?       rBoschlooExactResultcCstj}|dkrtd|tj|tjd}|jdkstdt|dkr*tdd|jddvr9t tj tj S|jdd\}}||}tj |dtjd dd }tj |dtjd d d}||} |d krs| ||| |j} nA|d kr| ||| |j} n3|d krt|d |d } t|d |d } | j| jkr| n| } d| j}t | j|Sddd|}t|| |d|df}| |dk}|j|j| j}}} t|}t|}||||}tt| ||fd|dd}tjt|j ddd}t ||S)aqPerform Boschloo's exact test on a 2x2 contingency table. Parameters ---------- table : array_like of ints A 2x2 contingency table. Elements should be non-negative integers. alternative : {'two-sided', 'less', 'greater'}, optional Defines the null and alternative hypotheses. Default is 'two-sided'. Please see explanations in the Notes section below. n : int, optional Number of sampling points used in the construction of the sampling method. Note that this argument will automatically be converted to the next higher power of 2 since `scipy.stats.qmc.Sobol` is used to select sample points. Default is 32. Must be positive. In most cases, 32 points is enough to reach good precision. More points comes at performance cost. Returns ------- ber : BoschlooExactResult A result object with the following attributes. statistic : float The statistic used in Boschloo's test; that is, the p-value from Fisher's exact test. pvalue : float P-value, the probability of obtaining a distribution at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. See Also -------- chi2_contingency : Chi-square test of independence of variables in a contingency table. fisher_exact : Fisher exact test on a 2x2 contingency table. barnard_exact : Barnard's exact test, which is a more powerful alternative than Fisher's exact test for 2x2 contingency tables. Notes ----- Boschloo's test is an exact test used in the analysis of contingency tables. It examines the association of two categorical variables, and is a uniformly more powerful alternative to Fisher's exact test for 2x2 contingency tables. Let's define :math:`X_0` a 2x2 matrix representing the observed sample, where each column stores the binomial experiment, as in the example below. Let's also define :math:`p_1, p_2` the theoretical binomial probabilities for :math:`x_{11}` and :math:`x_{12}`. When using Boschloo exact test, we can assert three different null hypotheses : - :math:`H_0 : p_1=p_2` versus :math:`H_1 : p_1 < p_2`, with `alternative` = "less" - :math:`H_0 : p_1=p_2` versus :math:`H_1 : p_1 > p_2`, with `alternative` = "greater" - :math:`H_0 : p_1=p_2` versus :math:`H_1 : p_1 \neq p_2`, with `alternative` = "two-sided" (default one) Boschloo's exact test uses the p-value of Fisher's exact test as a statistic, and Boschloo's p-value is the probability under the null hypothesis of observing such an extreme value of this statistic. Boschloo's and Barnard's are both more powerful than Fisher's exact test. .. versionadded:: 1.7.0 References ---------- .. [1] R.D. Boschloo. "Raised conditional level of significance for the 2 x 2-table when testing the equality of two probabilities", Statistica Neerlandica, 24(1), 1970 .. [2] "Boschloo's test", Wikipedia, https://en.wikipedia.org/wiki/Boschloo%27s_test .. [3] Lise M. Saari et al. "Employee attitudes and job satisfaction", Human Resource Management, 43(4), 395-407, 2004, :doi:`10.1002/hrm.20032`. Examples -------- In the following example, we consider the article "Employee attitudes and job satisfaction" [3]_ which reports the results of a survey from 63 scientists and 117 college professors. Of the 63 scientists, 31 said they were very satisfied with their jobs, whereas 74 of the college professors were very satisfied with their work. Is this significant evidence that college professors are happier with their work than scientists? The following table summarizes the data mentioned above:: college professors scientists Very Satisfied 74 31 Dissatisfied 43 32 When working with statistical hypothesis testing, we usually use a threshold probability or significance level upon which we decide to reject the null hypothesis :math:`H_0`. Suppose we choose the common significance level of 5%. Our alternative hypothesis is that college professors are truly more satisfied with their work than scientists. Therefore, we expect :math:`p_1` the proportion of very satisfied college professors to be greater than :math:`p_2`, the proportion of very satisfied scientists. We thus call `boschloo_exact` with the ``alternative="greater"`` option: >>> import scipy.stats as stats >>> res = stats.boschloo_exact([[74, 31], [43, 32]], alternative="greater") >>> res.statistic 0.0483... >>> res.pvalue 0.0355... Under the null hypothesis that scientists are happier in their work than college professors, the probability of obtaining test results at least as extreme as the observed data is approximately 3.55%. Since this p-value is less than our chosen significance level, we have evidence to reject :math:`H_0` in favor of the alternative hypothesis. rrrlrrrr(rr'rrr)rrBr&z`alternative` should be one of )rrrz, found rrg?rrrr)r hypergeomr/r,r-rrr3rrrrr5rr6rr"r!rrrrrWr)rrrBrrrtotalrrrpvaluesZ boschloo_lessZboschloo_greaterresr"rZ fisher_statrrrrrrrGrGrHrsf~     rcCsV|j\}}||d}tjdddRtj|t||dkd}tjd|t|d|dkd}||} d| |dkddddf<|||} d| ||kddddf<|| | } Wdn1sgwY| |} | } tjddd%t| | }| tj|t|tj |dkd}Wd| S1swY| S)a* Compute the log pvalue in respect of a nuisance parameter considering a 2x2 sample space. Parameters ---------- nuisance_param : float nuisance parameter used in the computation of the maximisation of the p-value. Must be between 0 and 1 x1_sum_x2 : ndarray Sum of x1 and x2 inside barnard_exact x1_sum_x2_log_comb : ndarray sum of the log combination of x1 and x2 index_arr : ndarray of boolean Returns ------- p_value : float Return the maximum p-value considering every nuisance paramater between 0 and 1 Notes ----- Both Barnard's test and Boschloo's test iterate over a nuisance parameter :math:`\pi \in [0, 1]` to find the maximum p-value. To search this maxima, this function return the negative log pvalue with respect to the nuisance parameter passed in params. This negative log p-value is then used in `shgo` to find the minimum negative pvalue which is our maximum pvalue. Also, to compute the different combination used in the p-values' computation formula, this function uses `gammaln` which is more tolerant for large value than `scipy.special.comb`. `gammaln` gives a log combination. For the little precision loss, performances are improved a lot. r&rrr)outwhererN) rr,rlogror;rWrZ full_likeinf)Znuisance_paramrrrt1t2rBZ log_nuisanceZlog_1_minus_nuisanceZnuisance_power_x1_x2Znuisance_power_n_minus_x1_x2Ztmp_log_values_arrZtmp_values_from_indexZ max_valueZ log_probsZ log_pvaluerGrGrHrrsJ +      rc Cst|}t|}g}t||D]!\}}|t||d}||t||d7}||qtj|dd\}} t| ||kt| S)z Compute the exact p-value of the Cramer-von Mises two-sample test for a given value s (float) of the test statistic by enumerating all possible combinations. nx and ny are the sizes of the samples. r&T)Z return_counts)r,rrrappendunique) sr@rAZrangexZrangeyusr=r>rvrrGrGrH_pval_cvm_2samp_exacts   rautoc Cstt|}tt|}|jdks|jdkrtd|jdks(|jdkr,td|dvr4tdt|}t|}|dkrLt||dkrJd}nd }t||g}t j j |d d }|d |} ||d } |t | t d|dd } | |t | t d|dd 7} ||||} } | | | d| dd| }|d krt| ||}nIdd| d}| dd| | d|d |d d | }|d| d d| }d||td|}|dkrd}n tddt|}t||dS)uMPerform the two-sample Cramér-von Mises test for goodness of fit. This is the two-sample version of the Cramér-von Mises test ([1]_): for two independent samples :math:`X_1, ..., X_n` and :math:`Y_1, ..., Y_m`, the null hypothesis is that the samples come from the same (unspecified) continuous distribution. Parameters ---------- x : array_like A 1-D array of observed values of the random variables :math:`X_i`. y : array_like A 1-D array of observed values of the random variables :math:`Y_i`. method : {'auto', 'asymptotic', 'exact'}, optional The method used to compute the p-value, see Notes for details. The default is 'auto'. Returns ------- res : object with attributes statistic : float Cramér-von Mises statistic. pvalue : float The p-value. See Also -------- cramervonmises, anderson_ksamp, epps_singleton_2samp, ks_2samp Notes ----- .. versionadded:: 1.7.0 The statistic is computed according to equation 9 in [2]_. The calculation of the p-value depends on the keyword `method`: - ``asymptotic``: The p-value is approximated by using the limiting distribution of the test statistic. - ``exact``: The exact p-value is computed by enumerating all possible combinations of the test statistic, see [2]_. The exact calculation will be very slow even for moderate sample sizes as the number of combinations increases rapidly with the size of the samples. If ``method=='auto'``, the exact approach is used if both samples contain less than 10 observations, otherwise the asymptotic distribution is used. If the underlying distribution is not continuous, the p-value is likely to be conservative (Section 6.2 in [3]_). When ranking the data to compute the test statistic, midranks are used if there are ties. References ---------- .. [1] https://en.wikipedia.org/wiki/Cramer-von_Mises_criterion .. [2] Anderson, T.W. (1962). On the distribution of the two-sample Cramer-von-Mises criterion. The Annals of Mathematical Statistics, pp. 1148-1159. .. [3] Conover, W.J., Practical Nonparametric Statistics, 1971. Examples -------- Suppose we wish to test whether two samples generated by ``scipy.stats.norm.rvs`` have the same distribution. We choose a significance level of alpha=0.05. >>> from scipy import stats >>> rng = np.random.default_rng() >>> x = stats.norm.rvs(size=100, random_state=rng) >>> y = stats.norm.rvs(size=70, random_state=rng) >>> res = stats.cramervonmises_2samp(x, y) >>> res.statistic, res.pvalue (0.29376470588235293, 0.1412873014573014) The p-value exceeds our chosen significance level, so we do not reject the null hypothesis that the observed samples are drawn from the same distribution. For small sample sizes, one can compute the exact p-values: >>> x = stats.norm.rvs(size=7, random_state=rng) >>> y = stats.t.rvs(df=2, size=6, random_state=rng) >>> res = stats.cramervonmises_2samp(x, y, method='exact') >>> res.statistic, res.pvalue (0.197802197802198, 0.31643356643356646) The p-value based on the asymptotic distribution is a good approximation even though the sample size is small. >>> res = stats.cramervonmises_2samp(x, y, method='asymptotic') >>> res.statistic, res.pvalue (0.197802197802198, 0.2966041181527128) Independent of the method, one would not reject the null hypothesis at the chosen significance level in this example. rz/x and y must contain at least two observations.z$The samples must be one-dimensional.)rexact asymptoticz0method must be either auto, exact or asymptotic.r rrZaverage)methodNr&rSr]-gUUUUUU?g~jth?r+rr )r,rr-rr/r.r1r; concatenaterrZrankdatarrrrXryrI)r=r>rZxaZyar@rArZrDZrxZryrvrgNr?rFetZvttnrGrGrHrs@b  "& 0 r)r!r"null_distributionPermutationTestResultcCs6t||d}|dur|gt|}ddt||DS)P Broadcast shapes of arrays, ignoring incompatibility of specified axes r(NcSsg|] \}}t||qSrG)r,Z broadcast_to).0r new_shaperGrGrH sz%_broadcast_arrays..)_broadcast_array_shapesr1zip)arraysr) new_shapesrGrGrH_broadcast_arrayss rcCsdd|D}t||S)rcSsg|]}t|jqSrG)r,r-r)rZarrrGrGrHrsz+_broadcast_array_shapes..)_broadcast_shapes)rr)shapesrGrGrHrs rcstdd|D}tjt||ftd}t||D]\}}||t|t|d<q|durOt|}|||dk||dk<|dd|f}tj||dd}tj|dd|jdd9t |dk|kBrnt d|dur|t t|fd d|D}|St S) zF Broadcast shapes, ignoring incompatibility of specified axes cSg|]}t|qSrGr1)rrrGrGrHrsz%_broadcast_shapes..rlNrrr(z/Array shapes are incompatible for broadcasting.csg|] }tt|qSrG)tupler,insert)rZ removed_shapeZnew_axisrrGrHrs) r;r,rr1rr atleast_1ddeleter2r3r/rr)rr)Zn_dimsrrowrZremoved_shapesrGrrHrs(  rc#sfddfddttt|}||ddD]}tdd|Dt}|VqdS)zu Generate all partitions of indices of groups of given sizes, concatenated `ns` is an iterable of ints. css0t||D]}t|}||}||gVqdSrJ)rset)rZrBr_Zx0rrGrGrHall_partitionss  z4_all_partitions_concatenated..all_partitionsc3s`t|dkr |gVdS||dD]}|d|ddD] }|dd|Vq!qdS)Nrrr)rZnsr_rrall_partitions_nrGrHrs z6_all_partitions_concatenated..all_partitions_nNcSrrG)list)r partitionrGrGrHrs z0_all_partitions_concatenated..)rranger,rrrr)rrZZ partitioningr=rGrrH_all_partitions_concatenateds rccsht|}|dkr td g}zt|D] }|t|q|VWnty2|r/|VYdSwq)z.r'cs"g|] }t||dqS)r)rrr)n_obs_icrGrHrsrrTFc3s|]}VqdSrJZ permutationr )n_obs random_staterGrH sz'_calculate_null_both..r(r.Nr)) r1r,ZcumsumZprodrrrrrrmoveaxissplitr) datar!n_permutationsrr  n_samplesZn_obs_in_max exact_testperm_generatorrindices data_batchrG)r r r rH_calculate_null_boths4       rc st||djdt}||kr'd}|}tfddtD}nd}fddt|D}|p;t|}g}t||dD];} t| } t | dd } d g} tD]} || d | | f| | <t | | d d| | <q[| || d diqDtj |dd}|||fS)z< Calculate null distribution for association tests. rr'Tc3s|] }ttVqdSrJ)rrr ) n_obs_samplerGrHr;sz+_calculate_null_pairings..Fc3s(|]}fddtDVqdS)csg|]}qSrGr r )rr rGrHrBsz6_calculate_null_pairings...N)r)rjrrr rGrHrBs  rrN.rr)r() r1rrrrrrr,rswapaxesrrr) rr!rrr rrrrrrrrGrrH_calculate_null_pairings*s2        r csNt|dkr|d|d g}t|dd}fdd}t|||||S)z> Calculate null distribution for paired-sample tests. rrr'cs0t|dd}dkr|dd}|d|iS)Nrr'rr))r,r)r)rrr!rGrHstatistic_wrappedws z2_calculate_null_samples..statistic_wrapped)r1r,rr )rr!rrr r"rGr!rH_calculate_null_samplesbsr#c  Cst|} || kr tdhd} |}|| vr td| d|dvr(td|s/t|}d} zt|dkr@|d kr@t| Wn tyLt| wt||}g} |D]} t | } | j |d krhtd t | | d } | | qVt |s~t|ntj}||ks|d krtd|dur|}nt|}||ks|d krtdhd}|}||vrtd|t|}| ||||||| |f S)z(Input validation for `permutation_test`.z`axis` must be an integer.>samplespairings independentz`permutation_type` must be in .>FTz'`vectorized` must be `True` or `False`.z6`data` must be a tuple containing at least two samplesr&r&rzIeach sample in `data` must contain two or more observations along `axis`.r'rz)`n_resamples` must be a positive integer.Nz+`batch` must be a positive integer or None.>rrrz`alternative` must be in )rr/lower _bootstrapZ_vectorize_statisticr1 TypeErrorrr,rrrrisinfrr)rr!permutation_type vectorized n_resamplesrrr)r Zaxis_intZpermutation_typesmessageZdata_ivrZn_resamples_intZbatch_ivZ alternativesrGrGrH_permutation_test_ivsZ      r0r&Fi')r,r-r.rrr)r c st|||||||| } | \ }}}}}}}}||ddi} tttd} ||||f} | |} | | \}}|r:dndfddfdd fd d }|d }|||| }t|dd}t| ||S) aHD Performs a permutation test of a given statistic on provided data. For independent sample statistics, the null hypothesis is that the data are randomly sampled from the same distribution. For paired sample statistics, two null hypothesis can be tested: that the data are paired at random or that the data are assigned to samples at random. Parameters ---------- data : iterable of array-like Contains the samples, each of which is an array of observations. Dimensions of sample arrays must be compatible for broadcasting except along `axis`. statistic : callable Statistic for which the p-value of the hypothesis test is to be calculated. `statistic` must be a callable that accepts samples as separate arguments (e.g. ``statistic(*data)``) and returns the resulting statistic. If `vectorized` is set ``True``, `statistic` must also accept a keyword argument `axis` and be vectorized to compute the statistic along the provided `axis` of the sample arrays. permutation_type : {'independent', 'samples', 'pairings'}, optional The type of permutations to be performed, in accordance with the null hypothesis. The first two permutation types are for paired sample statistics, in which all samples contain the same number of observations and observations with corresponding indices along `axis` are considered to be paired; the third is for independent sample statistics. - ``'samples'`` : observations are assigned to different samples but remain paired with the same observations from other samples. This permutation type is appropriate for paired sample hypothesis tests such as the Wilcoxon signed-rank test and the paired t-test. - ``'pairings'`` : observations are paired with different observations, but they remain within the same sample. This permutation type is appropriate for association/correlation tests with statistics such as Spearman's :math:`\rho`, Kendall's :math:`\tau`, and Pearson's :math:`r`. - ``'independent'`` (default) : observations are assigned to different samples. Samples may contain different numbers of observations. This permutation type is appropriate for independent sample hypothesis tests such as the Mann-Whitney :math:`U` test and the independent sample t-test. Please see the Notes section below for more detailed descriptions of the permutation types. vectorized : bool, default: ``False`` By default, `statistic` is assumed to calculate the statistic only for 1D arrays contained in `data`. If `vectorized` is set ``True``, `statistic` must also accept a keyword argument `axis` and be vectorized to compute the statistic along the provided `axis` of the ND arrays in `data`. Use of a vectorized statistic can reduce computation time. n_resamples : int or np.inf, default: 9999 Number of random permutations (resamples) used to approximate the null distribution. If greater than or equal to the number of distinct permutations, the exact null distribution will be computed. Note that the number of distinct permutations grows very rapidly with the sizes of samples, so exact tests are feasible only for very small data sets. batch : int, optional The number of permutations to process in each call to `statistic`. Memory usage is O(`batch`*``n``), where ``n`` is the total size of all samples, regardless of the value of `vectorized`. Default is ``None``, in which case ``batch`` is the number of permutations. alternative : {'two-sided', 'less', 'greater'}, optional The alternative hypothesis for which the p-value is calculated. For each alternative, the p-value is defined for exact tests as follows. - ``'greater'`` : the percentage of the null distribution that is greater than or equal to the observed value of the test statistic. - ``'less'`` : the percentage of the null distribution that is less than or equal to the observed value of the test statistic. - ``'two-sided'`` (default) : twice the smaller of the p-values above. Note that p-values for randomized tests are calculated according to the conservative (over-estimated) approximation suggested in [2]_ and [3]_ rather than the unbiased estimator suggested in [4]_. That is, when calculating the proportion of the randomized null distribution that is as extreme as the observed value of the test statistic, the values in the numerator and denominator are both increased by one. An interpretation of this adjustment is that the observed value of the test statistic is always included as an element of the randomized null distribution. The convention used for two-sided p-values is not universal; the observed test statistic and null distribution are returned in case a different definition is preferred. axis : int, default: 0 The axis of the (broadcasted) samples over which to calculate the statistic. If samples have a different number of dimensions, singleton dimensions are prepended to samples with fewer dimensions before `axis` is considered. random_state : {None, int, `numpy.random.Generator`, `numpy.random.RandomState`}, optional Pseudorandom number generator state used to generate permutations. If `random_state` is ``None`` (default), the `numpy.random.RandomState` singleton is used. If `random_state` is an int, a new ``RandomState`` instance is used, seeded with `random_state`. If `random_state` is already a ``Generator`` or ``RandomState`` instance then that instance is used. Returns ------- statistic : float or ndarray The observed test statistic of the data. pvalue : float or ndarray The p-value for the given alternative. null_distribution : ndarray The values of the test statistic generated under the null hypothesis. Notes ----- The three types of permutation tests supported by this function are described below. **Unpaired statistics** (``permutation_type='independent'``): The null hypothesis associated with this permutation type is that all observations are sampled from the same underlying distribution and that they have been assigned to one of the samples at random. Suppose ``data`` contains two samples; e.g. ``a, b = data``. When ``1 < n_resamples < binom(n, k)``, where * ``k`` is the number of observations in ``a``, * ``n`` is the total number of observations in ``a`` and ``b``, and * ``binom(n, k)`` is the binomial coefficient (``n`` choose ``k``), the data are pooled (concatenated), randomly assigned to either the first or second sample, and the statistic is calculated. This process is performed repeatedly, `permutation` times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value. When ``n_resamples >= binom(n, k)``, an exact test is performed: the data are *partitioned* between the samples in each distinct way exactly once, and the exact null distribution is formed. Note that for a given partitioning of the data between the samples, only one ordering/permutation of the data *within* each sample is considered. For statistics that do not depend on the order of the data within samples, this dramatically reduces computational cost without affecting the shape of the null distribution (because the frequency/count of each value is affected by the same factor). For ``a = [a1, a2, a3, a4]`` and ``b = [b1, b2, b3]``, an example of this permutation type is ``x = [b3, a1, a2, b2]`` and ``y = [a4, b1, a3]``. Because only one ordering/permutation of the data *within* each sample is considered in an exact test, a resampling like ``x = [b3, a1, b2, a2]`` and ``y = [a4, a3, b1]`` would *not* be considered distinct from the example above. ``permutation_type='independent'`` does not support one-sample statistics, but it can be applied to statistics with more than two samples. In this case, if ``n`` is an array of the number of observations within each sample, the number of distinct partitions is:: np.product([binom(sum(n[i:]), sum(n[i+1:])) for i in range(len(n)-1)]) **Paired statistics, permute pairings** (``permutation_type='pairings'``): The null hypothesis associated with this permutation type is that observations within each sample are drawn from the same underlying distribution and that pairings with elements of other samples are assigned at random. Suppose ``data`` contains only one sample; e.g. ``a, = data``, and we wish to consider all possible pairings of elements of ``a`` with elements of a second sample, ``b``. Let ``n`` be the number of observations in ``a``, which must also equal the number of observations in ``b``. When ``1 < n_resamples < factorial(n)``, the elements of ``a`` are randomly permuted. The user-supplied statistic accepts one data argument, say ``a_perm``, and calculates the statistic considering ``a_perm`` and ``b``. This process is performed repeatedly, `permutation` times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value. When ``n_resamples >= factorial(n)``, an exact test is performed: ``a`` is permuted in each distinct way exactly once. Therefore, the `statistic` is computed for each unique pairing of samples between ``a`` and ``b`` exactly once. For ``a = [a1, a2, a3]`` and ``b = [b1, b2, b3]``, an example of this permutation type is ``a_perm = [a3, a1, a2]`` while ``b`` is left in its original order. ``permutation_type='pairings'`` supports ``data`` containing any number of samples, each of which must contain the same number of observations. All samples provided in ``data`` are permuted *independently*. Therefore, if ``m`` is the number of samples and ``n`` is the number of observations within each sample, then the number of permutations in an exact test is:: factorial(n)**m Note that if a two-sample statistic, for example, does not inherently depend on the order in which observations are provided - only on the *pairings* of observations - then only one of the two samples should be provided in ``data``. This dramatically reduces computational cost without affecting the shape of the null distribution (because the frequency/count of each value is affected by the same factor). **Paired statistics, permute samples** (``permutation_type='samples'``): The null hypothesis associated with this permutation type is that observations within each pair are drawn from the same underlying distribution and that the sample to which they are assigned is random. Suppose ``data`` contains two samples; e.g. ``a, b = data``. Let ``n`` be the number of observations in ``a``, which must also equal the number of observations in ``b``. When ``1 < n_resamples < 2**n``, the elements of ``a`` are ``b`` are randomly swapped between samples (maintaining their pairings) and the statistic is calculated. This process is performed repeatedly, `permutation` times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value. When ``n_resamples >= 2**n``, an exact test is performed: the observations are assigned to the two samples in each distinct way (while maintaining pairings) exactly once. For ``a = [a1, a2, a3]`` and ``b = [b1, b2, b3]``, an example of this permutation type is ``x = [b1, a2, b3]`` and ``y = [a1, b2, a3]``. ``permutation_type='samples'`` supports ``data`` containing any number of samples, each of which must contain the same number of observations. If ``data`` contains more than one sample, paired observations within ``data`` are exchanged between samples *independently*. Therefore, if ``m`` is the number of samples and ``n`` is the number of observations within each sample, then the number of permutations in an exact test is:: factorial(m)**n Several paired-sample statistical tests, such as the Wilcoxon signed rank test and paired-sample t-test, can be performed considering only the *difference* between two paired elements. Accordingly, if ``data`` contains only one sample, then the null distribution is formed by independently changing the *sign* of each observation. References ---------- .. [1] R. A. Fisher. The Design of Experiments, 6th Ed (1951). .. [2] B. Phipson and G. K. Smyth. "Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn." Statistical Applications in Genetics and Molecular Biology 9.1 (2010). .. [3] M. D. Ernst. "Permutation Methods: A Basis for Exact Inference". Statistical Science (2004). .. [4] B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap (1993). Examples -------- Suppose we wish to test whether two samples are drawn from the same distribution. Assume that the underlying distributions are unknown to us, and that before observing the data, we hypothesized that the mean of the first sample would be less than that of the second sample. We decide that we will use the difference between the sample means as a test statistic, and we will consider a p-value of 0.05 to be statistically significant. For efficiency, we write the function defining the test statistic in a vectorized fashion: the samples ``x`` and ``y`` can be ND arrays, and the statistic will be calculated for each axis-slice along `axis`. >>> def statistic(x, y, axis): ... return np.mean(x, axis=axis) - np.mean(y, axis=axis) After collecting our data, we calculate the observed value of the test statistic. >>> from scipy.stats import norm >>> rng = np.random.default_rng() >>> x = norm.rvs(size=5, random_state=rng) >>> y = norm.rvs(size=6, loc = 3, random_state=rng) >>> statistic(x, y, 0) -3.5411688580987266 Indeed, the test statistic is negative, suggesting that the true mean of the distribution underlying ``x`` is less than that of the distribution underlying ``y``. To determine the probability of this occuring by chance if the two samples were drawn from the same distribution, we perform a permutation test. >>> from scipy.stats import permutation_test >>> # because our statistic is vectorized, we pass `vectorized=True` >>> # `n_resamples=np.inf` indicates that an exact test is to be performed >>> res = permutation_test((x, y), statistic, vectorized=True, ... n_resamples=np.inf, alternative='less') >>> print(res.statistic) -3.5411688580987266 >>> print(res.pvalue) 0.004329004329004329 The probability of obtaining a test statistic less than or equal to the observed value under the null hypothesis is 0.4329%. This is less than our chosen threshold of 5%, so we consider this to to be significant evidence against the null hypothesis in favor of the alternative. Because the size of the samples above was small, `permutation_test` could perform an exact test. For larger samples, we resort to a randomized permutation test. >>> x = norm.rvs(size=100, random_state=rng) >>> y = norm.rvs(size=120, loc=0.3, random_state=rng) >>> res = permutation_test((x, y), statistic, n_resamples=100000, ... vectorized=True, alternative='less', ... random_state=rng) >>> print(res.statistic) -0.5230459671240913 >>> print(res.pvalue) 0.00016999830001699983 The approximate probability of obtaining a test statistic less than or equal to the observed value under the null hypothesis is 0.0225%. This is again less than our chosen threshold of 5%, so again we have significant evidence to reject the null hypothesis in favor of the alternative. For large samples and number of permutations, the result is comparable to that of the corresponding asymptotic test, the independent sample t-test. >>> from scipy.stats import ttest_ind >>> res_asymptotic = ttest_ind(x, y, alternative='less') >>> print(res_asymptotic.pvalue) 0.00012688101537979522 The permutation distribution of the test statistic is provided for further investigation. >>> import matplotlib.pyplot as plt >>> plt.hist(res.null_distribution, bins=50) >>> plt.title("Permutation distribution of test statistic") >>> plt.xlabel("Value of Statistic") >>> plt.ylabel("Frequency") r)r')r%r$r&rrcs$||k}|jdd}|SNrr(rrobservedZcmpsr adjustmentr.rGrHr.zpermutation_test..lesscs$||k}|jdd}|Sr1r2r3r5rGrHr3r7z!permutation_test..greatercs(||}||}t||d}|S)Nr&)r,Zminimum)rr4Z pvalues_lessZpvalues_greaterr)rrrGrH two_sided8s  z#permutation_test..two_sided)rrr)r0r r#rr,rr)rr!r,r-r.rrr)r rr4Znull_calculatorsZnull_calculator_argsZcalculate_nullrrr8ZcomparerrG)r6rrr.rHrs> `  rc@s*eZdZdZddZddZd ddZd S) TukeyHSDResultaResult of `scipy.stats.tukey_hsd`. Attributes ---------- statistic : float ndarray The computed statistic of the test for each comparison. The element at index ``(i, j)`` is the statistic for the comparison between groups ``i`` and ``j``. pvalue : float ndarray The associated p-value from the studentized range distribution. The element at index ``(i, j)`` is the p-value for the comparison between groups ``i`` and ``j``. Notes ----- The string representation of this object displays the most recently calculated confidence interval, and if none have been previously calculated, it will evaluate ``confidence_interval()``. References ---------- .. [1] NIST/SEMATECH e-Handbook of Statistical Methods, "7.4.7.1. Tukey's Method." https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm, 28 November 2020. cCs.||_||_||_||_||_d|_d|_dSrJ)r!r" _ntreatments_nobs _stand_err_ci_ci_cl)rKr!r"r;r:r<rGrGrHrLds zTukeyHSDResult.__init__cCs|jdur |jddd|jddd}|d7}t|jjdD]?}t|jjdD]4}||kr_|d |d |d |j||fd |j||fd |jj||fd |jj||fd d 7}q+q!|S)Nffffff?)confidence_levelz(Tukey's HSD Pairwise Group Comparisons (dz.1fz% Confidence Interval) z3Comparison Statistic p-value Lower CI Upper CI rz (z - z) z>10.3f ) r=confidence_intervalr>rr"rr!lowhigh)rKrrrrGrGrH__str__ms&     zTukeyHSDResult.__str__r?cCs|jdur|jdur||jkr|jSd|krdks$tdtd||j|j|jf}tjj|}||j}|j |}|j |}t ||d|_||_|jS)a3Compute the confidence interval for the specified confidence level. Parameters ---------- confidence_level : float, optional Confidence level for the computed confidence interval of the estimated proportion. Default is .95. Returns ------- ci : ``ConfidenceInterval`` object The object has attributes ``low`` and ``high`` that hold the lower and upper bounds of the confidence intervals for each comparison. The high and low values are accessible for each comparison at index ``(i, j)`` between groups ``i`` and ``j``. References ---------- .. [1] NIST/SEMATECH e-Handbook of Statistical Methods, "7.4.7.1. Tukey's Method." https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm, 28 November 2020. Examples -------- >>> from scipy.stats import tukey_hsd >>> group0 = [24.5, 23.5, 26.4, 27.1, 29.9] >>> group1 = [28.4, 34.2, 29.5, 32.2, 30.1] >>> group2 = [26.1, 28.3, 24.3, 26.2, 27.8] >>> result = tukey_hsd(group0, group1, group2) >>> ci = result.confidence_interval() >>> ci.low array([[-3.649159, -8.249159, -3.909159], [ 0.950841, -3.649159, 0.690841], [-3.389159, -7.989159, -3.649159]]) >>> ci.high array([[ 3.649159, -0.950841, 3.389159], [ 8.249159, 3.649159, 7.989159], [ 3.909159, -0.690841, 3.649159]]) Nrrz)Confidence level must be between 0 and 1.)rDrE) r=r>r/r;r:r studentized_rangeZppfr<r!r )rKr@paramsZsrdZtukey_criterionZ upper_confZ lower_confrGrGrHrCs +    z"TukeyHSDResult.confidence_intervalN)r?)rOrQrR__doc__rLrFrCrGrGrGrHr9Hs  r9cCsjt|dkr tddd|D}|D]}|jdkrtd|jdkr'tdt|r2tdq|S) Nr&z$There must be more than 1 treatment.cSg|]}t|qSrG)r,r-rargrGrGrHrrz!_tukey_hsd_iv..rz&Input samples must be one-dimensional.z+Input sample size must be greater than one.zInput samples must be finite.)r1r/r.rr,r+r3)rrLrGrGrH _tukey_hsd_ivs   rMc Gst|}t|}tdd|D}tdd|D}t|}tdd|D|d||}t|jdkrBd|d}n d|d|dj}t||d}|dj|}t ||} | |||f} t j j | } t || |||S) aPerform Tukey's HSD test for equality of means over multiple treatments. Tukey's honestly significant difference (HSD) test performs pairwise comparison of means for a set of samples. Whereas ANOVA (e.g. `f_oneway`) assesses whether the true means underlying each sample are identical, Tukey's HSD is a post hoc test used to compare the mean of each sample to the mean of each other sample. The null hypothesis is that the distributions underlying the samples all have the same mean. The test statistic, which is computed for every possible pairing of samples, is simply the difference between the sample means. For each pair, the p-value is the probability under the null hypothesis (and other assumptions; see notes) of observing such an extreme value of the statistic, considering that many pairwise comparisons are being performed. Confidence intervals for the difference between each pair of means are also available. Parameters ---------- sample1, sample2, ... : array_like The sample measurements for each group. There must be at least two arguments. Returns ------- result : `~scipy.stats._result_classes.TukeyHSDResult` instance The return value is an object with the following attributes: statistic : float ndarray The computed statistic of the test for each comparison. The element at index ``(i, j)`` is the statistic for the comparison between groups ``i`` and ``j``. pvalue : float ndarray The computed p-value of the test for each comparison. The element at index ``(i, j)`` is the p-value for the comparison between groups ``i`` and ``j``. The object has the following methods: confidence_interval(confidence_level=0.95): Compute the confidence interval for the specified confidence level. Notes ----- The use of this test relies on several assumptions. 1. The observations are independent within and among groups. 2. The observations within each group are normally distributed. 3. The distributions from which the samples are drawn have the same finite variance. The original formulation of the test was for samples of equal size [6]_. In case of unequal sample sizes, the test uses the Tukey-Kramer method [4]_. References ---------- .. [1] NIST/SEMATECH e-Handbook of Statistical Methods, "7.4.7.1. Tukey's Method." https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm, 28 November 2020. .. [2] Abdi, Herve & Williams, Lynne. (2021). "Tukey's Honestly Significant Difference (HSD) Test." https://personal.utdallas.edu/~herve/abdi-HSD2010-pretty.pdf .. [3] "One-Way ANOVA Using SAS PROC ANOVA & PROC GLM." SAS Tutorials, 2007, www.stattutorials.com/SAS/TUTORIAL-PROC-GLM.htm. .. [4] Kramer, Clyde Young. "Extension of Multiple Range Tests to Group Means with Unequal Numbers of Replications." Biometrics, vol. 12, no. 3, 1956, pp. 307-310. JSTOR, www.jstor.org/stable/3001469. Accessed 25 May 2021. .. [5] NIST/SEMATECH e-Handbook of Statistical Methods, "7.4.3.3. The ANOVA table and tests of hypotheses about means" https://www.itl.nist.gov/div898/handbook/prc/section4/prc433.htm, 2 June 2021. .. [6] Tukey, John W. "Comparing Individual Means in the Analysis of Variance." Biometrics, vol. 5, no. 2, 1949, pp. 99-114. JSTOR, www.jstor.org/stable/3001913. Accessed 14 June 2021. Examples -------- Here are some data comparing the time to relief of three brands of headache medicine, reported in minutes. Data adapted from [3]_. >>> from scipy.stats import tukey_hsd >>> group0 = [24.5, 23.5, 26.4, 27.1, 29.9] >>> group1 = [28.4, 34.2, 29.5, 32.2, 30.1] >>> group2 = [26.1, 28.3, 24.3, 26.2, 27.8] We would like to see if the means between any of the groups are significantly different. First, visually examine a box and whisker plot. >>> import matplotlib.pyplot as plt >>> fig, ax = plt.subplots(1, 1) >>> ax.boxplot([group0, group1, group2]) >>> ax.set_xticklabels(["group0", "group1", "group2"]) # doctest: +SKIP >>> ax.set_ylabel("mean") # doctest: +SKIP >>> plt.show() From the box and whisker plot, we can see overlap in the interquartile ranges group 1 to group 2 and group 3, but we can apply the ``tukey_hsd`` test to determine if the difference between means is significant. We set a significance level of .05 to reject the null hypothesis. >>> res = tukey_hsd(group0, group1, group2) >>> print(res) Tukey's HSD Pairwise Group Comparisons (95.0% Confidence Interval) Comparison Statistic p-value Lower CI Upper CI (0 - 1) -4.600 0.014 -8.249 -0.951 (0 - 2) -0.260 0.980 -3.909 3.389 (1 - 0) 4.600 0.014 0.951 8.249 (1 - 2) 4.340 0.020 0.691 7.989 (2 - 0) 0.260 0.980 -3.389 3.909 (2 - 1) -4.340 0.020 -7.989 -0.691 The null hypothesis is that each group has the same mean. The p-value for comparisons between ``group0`` and ``group1`` as well as ``group1`` and ``group2`` do not exceed .05, so we reject the null hypothesis that they have the same means. The p-value of the comparison between ``group0`` and ``group2`` exceeds .05, so we accept the null hypothesis that there is not a significant difference between their means. We can also compute the confidence interval associated with our chosen confidence level. >>> group0 = [24.5, 23.5, 26.4, 27.1, 29.9] >>> group1 = [28.4, 34.2, 29.5, 32.2, 30.1] >>> group2 = [26.1, 28.3, 24.3, 26.2, 27.8] >>> result = tukey_hsd(group0, group1, group2) >>> conf = res.confidence_interval(confidence_level=.99) >>> for ((i, j), l) in np.ndenumerate(conf.low): ... # filter out self comparisons ... if i != j: ... h = conf.high[i,j] ... print(f"({i} - {j}) {l:>6.3f} {h:>6.3f}") (0 - 1) -9.480 0.280 (0 - 2) -5.140 4.620 (1 - 0) -0.280 9.480 (1 - 2) -0.540 9.220 (2 - 0) -4.620 5.140 (2 - 1) -9.220 0.540 cSrJrG)r,r9rKrGrGrHrd rztukey_hsd..cSsg|]}|jqSrG)r)rarGrGrHre scSsg|] }tj|ddqS)r)Zddof)r,varrKrGrGrHrj srr&rN)rMr1r,r-rrrr6rXrqr rGr<r9) rZ ntreatmentsZmeansZnsamples_treatmentsZnobsZmse normalizeZ stand_errZmean_differencesZt_statrHrrGrGrHrs,  r)r#rJ)rG)r)Nr)rTr)rr)r)L collectionsrZ dataclassesrZnumpyr,r7 itertoolsrrrr4rZscipy.optimizerr Z_commonr Z_continuous_distnsr r Z scipy.specialr rrrrrZscipy.stats._bootstraprr)Zscipy._lib._utilrZ_hypotests_pythranrrr__all__rrrIrtryr{rrrrrrrrrkrrrrrrrZ attributesrrrrrrrr r#r0rr9rMrrGrGrGrHs       y 2 $ t  2   M[    ( = 9 ; }