o èEbˆPã@sJdZddlmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZmZe e¡jZdd „Z  d@d d „Zdd„Zdd„ZdAdd„ZdBdd„ZdCdd„Zdd„Zdd„ZdDdd„ZdDdd „Zd!d"„Zd#d$„Z d%d&„Z!d'd(„Z"d)d*„Z#d+d,„Z$d-d.„Z%dCd/d0„Z&d1d2„Z'd3d4„Z(d5d6„Z)dEd8d9„Z*dEd:d;„Z+dd?„Z-dS)Fz+Functions used by least-squares algorithms.é)ÚcopysignN)Únorm)Ú cho_factorÚ cho_solveÚ LinAlgError)Úissparse)ÚLinearOperatorÚaslinearoperatorc Csšt ||¡}|dkrtdƒ‚t ||¡}t ||¡|d}|dkr&tdƒ‚t ||||¡}|t||ƒ }||}||} || krI|| fS| |fS)aqFind the intersection of a line with the boundary of a trust region. This function solves the quadratic equation with respect to t ||(x + s*t)||**2 = Delta**2. Returns ------- t_neg, t_pos : tuple of float Negative and positive roots. Raises ------ ValueError If `s` is zero or `x` is not within the trust region. rz `s` is zero.éz#`x` is not within the trust region.)ÚnpÚdotÚ ValueErrorZsqrtr) ÚxÚsÚDeltaÚaÚbÚcÚdÚqÚt1Út2©rú.phi_and_derivativeréÿÿÿÿFgNgü©ñÒMbP?çà?r é)ÚEPSr rÚmaxÚranger Úabs)ÚnÚmZufrÚVrZ initial_alphaÚrtolZmax_iterr#r Z thresholdZ full_rankÚpZ alpha_upperr!r"Z alpha_lowerrÚitÚratiorrrÚsolve_lsq_trust_region9sB1      ÿr2cCsvzt|ƒ\}}t||f|ƒ }t ||¡|dkr|dfWSWn ty(Ynw|d|d}|d|d}|d|d}|d|} |d|} t | | d||| d|d| || | | g¡} t | ¡} t | t | ¡¡} |t  d| d| dd| dd| df¡}d tj || |¡dd t ||¡} t  | ¡}|d d …|f}|d fS) azSolve a general trust-region problem in 2 dimensions. The problem is reformulated as a 4th order algebraic equation, the solution of which is found by numpy.roots. Parameters ---------- B : ndarray, shape (2, 2) Symmetric matrix, defines a quadratic term of the function. g : ndarray, shape (2,) Defines a linear term of the function. Delta : float Radius of a trust region. Returns ------- p : ndarray, shape (2,) Found solution. newton_step : bool Whether the returned solution is the Newton step which lies within the trust region. r T)rr)rr&)r&r&rr&ér%©ZaxisNF) rrr r rZarrayÚrootsÚrealZisrealZvstackrÚargmin)ÚBÚgrÚRÚlowerr/rrrrÚfZcoeffsÚtÚvalueÚirrrÚsolve_trust_region_2d«s0  ÿ ÿ  6ÿ 6( r@cCsh|dkr ||}n||krdkrnnd}nd}|dkr&d|}||fS|dkr0|r0|d9}||fS)zÍUpdate the radius of a trust region based on the cost reduction. Returns ------- Delta : float New radius. ratio : float Ratio between actual and predicted reductions. rr&çÐ?gè?g@r)rZactual_reductionZpredicted_reductionÚ step_normZ bound_hitr1rrrÚupdate_tr_radiusÞs  ýrCc CsÊ| |¡}t ||¡}|dur|t |||¡7}|d9}t ||¡}|dura| |¡}|t ||¡7}dt ||¡t ||¡} |dur\|t |||¡7}| dt |||¡7} ||| fS||fS)a²Parameterize a multivariate quadratic function along a line. The resulting univariate quadratic function is given as follows: :: f(t) = 0.5 * (s0 + s*t).T * (J.T*J + diag) * (s0 + s*t) + g.T * (s0 + s*t) Parameters ---------- J : ndarray, sparse matrix or LinearOperator shape (m, n) Jacobian matrix, affects the quadratic term. g : ndarray, shape (n,) Gradient, defines the linear term. s : ndarray, shape (n,) Direction vector of a line. diag : None or ndarray with shape (n,), optional Addition diagonal part, affects the quadratic term. If None, assumed to be 0. s0 : None or ndarray with shape (n,), optional Initial point. If None, assumed to be 0. Returns ------- a : float Coefficient for t**2. b : float Coefficient for t. c : float Free term. Returned only if `s0` is provided. Nr%)r r ) ÚJr9rÚdiagZs0ÚvrrÚurrrrÚbuild_quadratic_1dûs     rHc Csv||g}|dkrd||}||kr|krnn| |¡t |¡}|||||}t |¡}||||fS)zÛMinimize a 1-D quadratic function subject to bounds. The free term `c` is 0 by default. Bounds must be finite. Returns ------- t : float Minimum point. y : float Minimum value. rgà¿)Úappendr Úasarrayr7) rrÚlbÚubrr=ZextremumÚyZ min_indexrrrÚminimize_quadratic_1d.s     rNcCs–|jdkr| |¡}t ||¡}|dur|t |||¡7}n | |j¡}tj|ddd}|dur?|tj||ddd7}t ||¡}d||S)aìCompute values of a quadratic function arising in least squares. The function is 0.5 * s.T * (J.T * J + diag) * s + g.T * s. Parameters ---------- J : ndarray, sparse matrix or LinearOperator, shape (m, n) Jacobian matrix, affects the quadratic term. g : ndarray, shape (n,) Gradient, defines the linear term. s : ndarray, shape (k, n) or (n,) Array containing steps as rows. diag : ndarray, shape (n,), optional Addition diagonal part, affects the quadratic term. If None, assumed to be 0. Returns ------- values : ndarray with shape (k,) or float Values of the function. If `s` was 2-D, then ndarray is returned, otherwise, float is returned. r&Nr rr4r%)Úndimr r ÚTr)rDr9rrEZJsrÚlrrrÚevaluate_quadraticEs   €   rRcCst ||k||k@¡S)z$Check if a point lies within bounds.)r Úall)rrKrLrrrÚ in_boundsosrTcCs¦t |¡}||}t |¡}| tj¡tjddt ||||||||¡||<Wdƒn1s9wYt |¡}|t ||¡t  |¡  t ¡fS)aðCompute a min_step size required to reach a bound. The function computes a positive scalar t, such that x + s * t is on the bound. Returns ------- step : float Computed step. Non-negative value. hits : ndarray of int with shape of x Each element indicates whether a corresponding variable reaches the bound: * 0 - the bound was not hit. * -1 - the lower bound was hit. * 1 - the upper bound was hit. Úignore)ZoverN) r ZnonzeroZ empty_likeÚfillÚinfZerrstateÚmaximumÚminÚequalÚsignZastypeÚint)rrrKrLZnon_zeroZ s_non_zeroZstepsZmin_steprrrÚstep_size_to_boundts    ÿÿ  r]绽×Ùß|Û=c Cs¶tj|td}|dkrd|||k<d|||k<|S||}||}|t dt |¡¡}|t dt |¡¡}t |¡|t ||¡k@} d|| <t |¡|t ||¡k@} d|| <|S)a·Determine which constraints are active in a given point. The threshold is computed using `rtol` and the absolute value of the closest bound. Returns ------- active : ndarray of int with shape of x Each component shows whether the corresponding constraint is active: * 0 - a constraint is not active. * -1 - a lower bound is active. * 1 - a upper bound is active. ©Zdtyperr$r&)r Ú zeros_liker\rXr*ÚisfiniteÚminimum) rrKrLr.ÚactiveZ lower_distZ upper_distZlower_thresholdZupper_thresholdZ lower_activeZ upper_activerrrÚfind_active_constraints‘s$  ÿÿrdc Csà| ¡}t||||ƒ}t |d¡}t |d¡}|dkr4t ||||¡||<t ||||¡||<n&|||t dt ||¡¡||<|||t dt ||¡¡||<||k||kB}d||||||<|S)zÔShift a point to the interior of a feasible region. Each element of the returned vector is at least at a relative distance `rstep` from the closest bound. If ``rstep=0`` then `np.nextafter` is used. r$r&rr%)Úcopyrdr rZZ nextafterrXr*) rrKrLZrstepZx_newrcZ lower_maskZ upper_maskZ tight_boundsrrrÚmake_strictly_feasible¸s   ÿÿrfcCsxt |¡}t |¡}|dkt |¡@}||||||<d||<|dkt |¡@}||||||<d||<||fS)a9Compute Coleman-Li scaling vector and its derivatives. Components of a vector v are defined as follows: :: | ub[i] - x[i], if g[i] < 0 and ub[i] < np.inf v[i] = | x[i] - lb[i], if g[i] > 0 and lb[i] > -np.inf | 1, otherwise According to this definition v[i] >= 0 for all i. It differs from the definition in paper [1]_ (eq. (2.2)), where the absolute value of v is used. Both definitions are equivalent down the line. Derivatives of v with respect to x take value 1, -1 or 0 depending on a case. Returns ------- v : ndarray with shape of x Scaling vector. dv : ndarray with shape of x Derivatives of v[i] with respect to x[i], diagonal elements of v's Jacobian. References ---------- .. [1] M.A. Branch, T.F. Coleman, and Y. Li, "A Subspace, Interior, and Conjugate Gradient Method for Large-Scale Bound-Constrained Minimization Problems," SIAM Journal on Scientific Computing, Vol. 21, Number 1, pp 1-23, 1999. rr$r&)r Ú ones_liker`ra)rr9rKrLrFZdvÚmaskrrrÚCL_scaling_vectorÓs  ric CsFt|||ƒr |t |¡fSt |¡}t |¡}| ¡}tj|td}||@}t ||d||||¡||<||||k||<||@}t ||d||||¡||<||||k||<||@}||}t  ||||d||¡} ||t | d||| ¡||<| ||k||<t |¡} d| |<|| fS)z3Compute reflective transformation and its gradient.r_r r$) rTr rgrarer`ÚboolrXrbZ remainder) rMrKrLZ lb_finiteZ ub_finiterZ g_negativerhrr=r9rrrÚreflective_transformationÿs(    $ $ $ rkc Cstd dddddd¡ƒdS)Nz*{0:^15}{1:^15}{2:^15}{3:^15}{4:^15}{5:^15}Ú Iterationz Total nfevÚCostúCost reductionú Step normÚ Optimality©ÚprintÚformatrrrrÚprint_header_nonlinear!s  ÿÿrtc CsL|durd}nd |¡}|durd}nd |¡}td ||||||¡ƒdS)Nú ú {0:^15.2e}z({0:^15}{1:^15}{2:^15.4e}{3}{4}{5:^15.2e}©rsrr)Ú iterationZnfevÚcostÚcost_reductionrBÚ optimalityrrrÚprint_iteration_nonlinear's   ÿÿr|cCstd ddddd¡ƒdS)Nz#{0:^15}{1:^15}{2:^15}{3:^15}{4:^15}rlrmrnrorprqrrrrÚprint_header_linear8s  ÿÿr}cCsJ|durd}nd |¡}|durd}nd |¡}td |||||¡ƒdS)Nrurvz!{0:^15}{1:^15.4e}{2}{3}{4:^15.2e}rw)rxryrzrBr{rrrÚprint_iteration_linear>s    ÿr~cCs t|tƒr | |¡S|j |¡S)z4Compute gradient of the least-squares cost function.)Ú isinstancerÚrmatvecrPr )rDr<rrrÚ compute_gradQs   rcCsnt|ƒrt | d¡jdd¡ ¡d}n tj|dddd}|dur+d||dk<nt ||¡}d||fS)z5Compute variables scale based on the Jacobian matrix.r rr4r%Nr&)rr rJZpowerrÚravelrX)rDZ scale_inv_oldZ scale_invrrrÚcompute_jac_scaleYs"  rƒcóDtˆƒ‰‡‡fdd„}‡‡fdd„}‡‡fdd„}tˆj|||dS)z#Return diag(d) J as LinearOperator.c󈈠|¡S©N)Úmatvec©r©rDrrrr‡lóz(left_multiplied_operator..matveccsˆdd…tjfˆ |¡Sr†)r ÚnewaxisÚmatmat©ÚXr‰rrrŒoóz(left_multiplied_operator..matmatcsˆ | ¡ˆ¡Sr†)r€r‚rˆr‰rrr€rsz)left_multiplied_operator..rmatvec©r‡rŒr€©r rÚshape©rDrr‡rŒr€rr‰rÚleft_multiplied_operatorhó ÿr”cr„)z#Return J diag(d) as LinearOperator.csˆ t |¡ˆ¡Sr†)r‡r r‚rˆr‰rrr‡}sz)right_multiplied_operator..matveccsˆ |ˆdd…tjf¡Sr†)rŒr r‹rr‰rrrŒ€rz)right_multiplied_operator..matmatcr…r†©r€rˆr‰rrr€ƒrŠz*right_multiplied_operator..rmatvecrr‘r“rr‰rÚright_multiplied_operatoryr•r—csFtˆƒ‰ˆj\‰}‡‡fdd„}‡‡‡fdd„}tˆ||f||dS)zµReturn a matrix arising in regularized least squares as LinearOperator. The matrix is [ J ] [ D ] where D is diagonal matrix with elements from `diag`. cst ˆ |¡ˆ|f¡Sr†)r Zhstackr‡rˆ)rDrErrr‡•sz(regularized_lsq_operator..matveccs*|dˆ…}|ˆd…}ˆ |¡ˆ|Sr†r–)rZx1Zx2©rDrEr,rrr€˜s  z)regularized_lsq_operator..rmatvec)r‡r€)r r’r)rDrEr+r‡r€rr˜rÚregularized_lsq_operatorŠs  r™TcCs`|r t|tƒs | ¡}t|ƒr|j|j|jdd9_|St|tƒr*t||ƒ}|S||9}|S)zhCompute J diag(d). If `copy` is False, `J` is modified in place (unless being LinearOperator). Zclip)Úmode)rrrerÚdataZtakeÚindicesr—©rDrrerrrÚright_multiply s û þržcCsr|r t|tƒs | ¡}t|ƒr |jt |t |j¡¡9_|St|tƒr,t ||ƒ}|S||dd…tj f9}|S)zhCompute diag(d) J. If `copy` is False, `J` is modified in place (unless being LinearOperator). N) rrrerr›r ÚrepeatZdiffZindptrr”r‹rrrrÚ left_multiply²s û þr c CsD|||ko |dk}||||k}|r|rdS|rdS|r dSdS)z8Check termination condition for nonlinear least squares.rAér rNr) ZdFÚFZdx_normZx_normr1ZftolZxtolZftol_satisfiedZxtol_satisfiedrrrÚcheck_terminationÄsr£cCsR|dd|d|d}t||tk<|dC}||d|9}t||dd|fS)z`Scale Jacobian and residuals for a robust loss function. Arrays are modified in place. r&r r%F)re)r'r )rDr<ZrhoZJ_scalerrrÚscale_for_robust_loss_functionÓs  r¤)Nrr)NN)rr†)r^)T).Ú__doc__ZmathrZnumpyr Z numpy.linalgrZ scipy.linalgrrrZ scipy.sparserZscipy.sparse.linalgrr ZfinfoÚfloatZepsr'rr2r@rCrHrNrRrTr]rdrfrirkrtr|r}r~rrƒr”r—r™ržr r£r¤rrrrÚsH    ' ÿr3  3 *  ',"