U }hw"@s>dZddlZGdddZGdddeZGdddeZdS) z(Stochastic optimization methods for MLP Nc@s2eZdZdZd ddZddZddZd d Zd S) BaseOptimizera9Base (Stochastic) gradient descent optimizer Parameters ---------- learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights Attributes ---------- learning_rate : float the current learning rate 皙?cCs||_t||_dSN)learning_rate_initfloat learning_rate)selfrr Q/tmp/pip-unpacked-wheel-ig1s1lm8/sklearn/neural_network/_stochastic_optimizers.py__init__szBaseOptimizer.__init__cCs4||}tdd|D|D]\}}||7}qdS)aUpdate parameters with given gradients Parameters ---------- params : list of length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params grads : list of length = len(params) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params css|] }|VqdSrr ).0pr r r +sz.BaseOptimizer.update_params..N) _get_updateszip)rparamsgradsupdatesparamupdater r r update_paramss zBaseOptimizer.update_paramscCsdS)zhPerform update to learning rate and potentially other states at the end of an iteration Nr rZ time_stepr r r iteration_ends.szBaseOptimizer.iteration_endscCs|rt|ddS)aHDecides whether it is time to stop training Parameters ---------- msg : str Message passed in for verbose output verbose : bool Print message to stdin if True Returns ------- is_stopping : bool True if training needs to stop Stopping.T)printrmsgverboser r r trigger_stopping4s zBaseOptimizer.trigger_stoppingN)r)__name__ __module__ __qualname____doc__r rrrr r r r r s  rcs:eZdZdZdfdd Zd d Zd d Zd dZZS) SGDOptimizeraStochastic gradient descent optimizer with momentum Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.1 The initial learning rate used. It controls the step-size in updating the weights lr_schedule : {'constant', 'adaptive', 'invscaling'}, default='constant' Learning rate schedule for weight updates. -'constant', is a constant learning rate given by 'learning_rate_init'. -'invscaling' gradually decreases the learning rate 'learning_rate_' at each time step 't' using an inverse scaling exponent of 'power_t'. learning_rate_ = learning_rate_init / pow(t, power_t) -'adaptive', keeps the learning rate constant to 'learning_rate_init' as long as the training keeps decreasing. Each time 2 consecutive epochs fail to decrease the training loss by tol, or fail to increase validation score by tol if 'early_stopping' is on, the current learning rate is divided by 5. momentum : float, default=0.9 Value of momentum used, must be larger than or equal to 0 nesterov : bool, default=True Whether to use nesterov's momentum or not. Use nesterov's if True power_t : float, default=0.5 Power of time step 't' in inverse scaling. See `lr_schedule` for more details. Attributes ---------- learning_rate : float the current learning rate velocities : list, length = len(params) velocities that are used to update params rconstant?T?cs8t|||_||_||_||_dd|D|_dS)NcSsg|]}t|qSr npZ zeros_liker rr r r sz)SGDOptimizer.__init__..)superr lr_schedulemomentumnesterovpower_t velocities)rrrr,r-r.r/ __class__r r r ys zSGDOptimizer.__init__cCs(|jdkr$t|j|d|j|_dS)aPerform updates to learning rate and potential other states at the end of an iteration Parameters ---------- time_step : int number of training samples trained on so far, used to update learning rate for 'invscaling' Z invscalingN)r,rrr/rrr r r rs zSGDOptimizer.iteration_endscCsd|jdkr|rt|ddS|jdkr<|r8t|ddS|jd_|r`t|d|jdS) NZadaptiverTgư>z# Learning rate too small. Stopping.g@z Setting learning rate to %fF)r,rrrr r r rs    zSGDOptimizer.trigger_stoppingcsDfddtj|D}|_jr@fddtj|D}|S)Get the values used to update params with given gradients Parameters ---------- grads : list, length = len(coefs_) + len(intercepts_) Containing gradients with respect to coefs_ and intercepts_ in MLP model. So length should be aligned with params Returns ------- updates : list, length = len(grads) The values to add to params cs$g|]\}}j|j|qSr r-rr Zvelocitygradrr r r*sz-SGDOptimizer._get_updates..cs$g|]\}}j|j|qSr r5r6r8r r r*s)rr0r.rrrr r8r rs    zSGDOptimizer._get_updates)rr$r%Tr&) rr r!r"r rrr __classcell__r r r1r r#Is2r#cs*eZdZdZd fdd Zdd ZZS) AdamOptimizeraStochastic gradient descent optimizer with Adam Note: All default values are from the original Adam paper Parameters ---------- params : list, length = len(coefs_) + len(intercepts_) The concatenated list containing coefs_ and intercepts_ in MLP model. Used for initializing velocities and updating params learning_rate_init : float, default=0.001 The initial learning rate used. It controls the step-size in updating the weights beta_1 : float, default=0.9 Exponential decay rate for estimates of first moment vector, should be in [0, 1) beta_2 : float, default=0.999 Exponential decay rate for estimates of second moment vector, should be in [0, 1) epsilon : float, default=1e-8 Value for numerical stability Attributes ---------- learning_rate : float The current learning rate t : int Timestep ms : list, length = len(params) First moment vectors vs : list, length = len(params) Second moment vectors References ---------- :arxiv:`Kingma, Diederik, and Jimmy Ba (2014) "Adam: A method for stochastic optimization." <1412.6980> MbP?r%+?:0yE>csHt|||_||_||_d|_dd|D|_dd|D|_dS)NrcSsg|]}t|qSr r'r)r r r r*sz*AdamOptimizer.__init__..cSsg|]}t|qSr r'r)r r r r*s)r+r beta_1beta_2epsilontmsvs)rrrr?r@rAr1r r r s zAdamOptimizer.__init__csjd7_fddtj|D_fddtj|D_jtdjjdjj_ fddtjjD}|S)r4r3cs(g|] \}}j|dj|qS)r3)r?)r mr7r8r r r*sz.AdamOptimizer._get_updates..cs,g|]$\}}j|dj|dqS)r3)r@)r vr7r8r r r*scs,g|]$\}}j |t|jqSr )rr(sqrtrA)r rErGr8r r r*s) rBrrCrDrr(rHr@r?rr9r r8r rs"      zAdamOptimizer._get_updates)r<r%r=r>)rr r!r"r rr:r r r1r r;s. r;)r"Znumpyr(rr#r;r r r r s?}