h--SSKJr SSKrSSKrSSKrSSKrSSKrSSKJ r SSK r SSK J r SSKJrJr SSKJrJrJrJrJrJr SSKJr SSKJr SSKJr SS KJr SS K J!r! SSK"r#\RH"S 5r%SS jr&SS jr'Sr(SSSjjr)SSjr*/4SSjjr+\,S:Xa7Sr-\R\R[\-5 \ R^"\-5 \+"5 gg)) annotationsN) setup_logger)get_rankget_size)add_io_bindings_as_ortvaluesconvert_inputs_for_ort%get_merged_sample_with_past_kv_inputsget_sample_inputsget_sample_with_past_kv_inputsverify_ort_inputs)setup_torch_model)make_dynamic_cache) AutoConfig) __version__) DynamicCachecPUR(aSOSup#URnX#U4$)N))rr) use_past_kvmax_position_embeddings)argsconfigpast_sequence_lengthcurr_sequence_lengthmax_sequence_lengths l/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/transformers/models/llama/llama_parity.pyget_sequence_lengthsr%s.;?;K;KQW. 88 7J JJc V[5nSn[X5upEnUR(a2[UURUUUUUR UR SUS9 nU$UR(a%[UURUUUR SUS9nU$[XRX5SS9nU$)NT)seq_len past_seq_len max_seq_lenuse_fp16use_buffer_share return_dict world_size)r%r'r()r') rrmergedr devicer%r&rr r )rrr( batch_sizersequence_lengthrinputss r get_inputsr.+sJJAUVZAc>+> {{6  KK #-+]]!22!  2 M   /  KK  ]]!  M#6;; aef Mrc l[U[[[45(aU$[U[5(a[ SU55$[U[ 5(aUVs/sHn[ U5PM sn$[U[5(aUVs1sHn[ U5iM sn$[U[5(a/UR5VVs0sHup!U[ U5_M snn$[U[R5(aUR5$[US5(aUR5$[U[5(a:[![ [ [#UR$UR&SS9555$[)S[+U535es snfs snfs snnf)Nc38# UHn[U5v M g7f)N)torch_deepcopy).0vs r !torch_deepcopy..Rs61^A&&scloneF)strictz(torch_deepcopy not implemented for type ) isinstanceintfloatstrtuplelistr1setdictitemsnpndarraycopyhasattrr6rrzip key_cache value_cacheNotImplementedErrortype)valuer3ks rr1r1NsK%#uc*++ %6666%+015aq!511%+015aq!511%16?>!$$??%$$zz|ug{{}%&&!.c%//5K\K\ej6k1l"mnn  He V WW21?s#F&F+F0c UnUcH[UUUUR(a[RO[RUR S9upV[ X5nSU;aC[R"[5[R"S5:a[US5US'[U5nURS:wa[RR5 [R"5n U"S0UD6R R#5R%5R'5n URS:wa[RR5 [R"5n [(R+SX- S35 UR,(a"UbA[RR/5 [1X5upn[3UUR4U US9nURR75S3nUS :XaUS UR804n[:R<"UR>[:R@"5U/S 9n[CUU5nURS:wa[EUUUR[GUR85UR4US 9unnURI5 [R"5n URKU5 URM5 [R"5n URO5S nAOA[R"5n URQSU5n[R"5n US n[(R+SX- S35 SUR>;dSUR>;aSOSn[RRT"U UUUS9n[(RWSU35 U(d/[(RWS[RRX"U U- 535 U$)N torch_dtyper*past_key_valuesz4.45cpuz PyTorch took z s)r&r#r$ExecutionProviderCUDAExecutionProvider device_id) sess_options providers) ort_inputsr*rSr&kv_cache_ortvaluesrzONNX Runtime took int4int8g4@g?)rtolatolz,Are PyTorch and ONNX Runtime results close? z Max diff: )-r r%torchfloat16float32r*r.pvVersiontransformers_versionrr1execution_providercuda synchronizetimelogitsdetachrPnumpyloggerinfo small_gpu empty_cacherrr&upperrankortInferenceSessiononnx_model_pathSessionOptionsr rr9synchronize_inputsrun_with_iobindingsynchronize_outputscopy_outputs_to_cpurunrAallclosewarningmax)rlocationuse_auth_tokenrW pytorch_modelrpy_modelr-inputs_after_deepcopy start_time pt_outputsend_timer_rep ort_model io_binding ort_outputstolparitys r verify_parityrdsKH,   *.--U]];;   %FF"rzz2F'G2::V\K]']$6v>O7P$Q !+62 %'  J21299@@BFFHNNPJ %'  yy{H KK- 56b9: ~~(.   4H3U00 #..)' F  # # ) ) + ,,= >B $$; * +$$ '')$I y& 1F %')E **$))n!221 * & & %%'YY[ $$Z0&&(99; 446q9 YY[ mmD&1 99;!!n  KK$X%:$;2>?4///6T=Q=Q3Q#W[C [[[s EF NNA&JK BFF: +C$D#EFG rc [R"5nURSSSSS9 URSSS[RR S5S S 9 URS S S [RR S5SS 9 URSSSS/SQSS9 URSSSSS9 UR SS9 URSSSSS9 UR SS9 URSS SS!S9 UR SS"9 URS#SS$S9 UR SS%9 URS&S'S /S(QS)S*9 URS+S[S,S-S.9 URS/SS0S9 U/:XaUR5OURU5nURS1;d URS2:XaURS:Xa S3Ul U$S4Ul U$)5Nz-mz --model_nameFzModel name in Hugging Face)requiredhelpz-tz--torch_model_directory.zMPath to folder containing PyTorch model and associated files if saved on disk)rdefaultrz-oz--onnx_model_pathTzSPath to ONNX model (with external data files saved in the same folder as the model)z-epz--execution_providerrP)rPrdrocmz(Execution provider to verify parity with)rrchoicesrz-vz --verbose store_truezPrint verbose logs)actionr)verbosez-pz --use_past_kvzfUse past key and past value as inputs to the model. Necessary for decoder_with_past_model.onnx models.)rz-gz--use_buffer_sharezWUse if model has GroupQueryAttention and you want to enable past-present buffer sharing)r&z--mergedz2Use merged model (i.e. decoder_merged_model.onnx).)r)z-fpz --precision)rXrYfp16fp32zPrecision of model)rrrz --cache_dirz ./model_cachezQmodel cache dir to override default HF cache dir to avoid overflood the /home dir)rrIrrz --small_gpuzhLoad the llama in GPU every time for parity_check if it's running in a machine which GPU memory < 36GB. >rrYrXrr) argparseArgumentParser add_argumentospathjoin set_defaultsr; parse_args precisionrc)argvparserrs rget_argsrsT  $ $ &F   )   ! S! \    S! b   ' 7    !   &   u   E*   f   /  A  u%  0 !    `   w #'"*6   &2C2CD2ID >>- -$..F2JtOfOfjoOo  N K N Krc 8[U5n[UR5 [R SU35 [ 5n[ USURS:H5 X!l[ USURS:XaSOSU35 [ US[R"UR55 UR[RR!S5:HnU(a UR"O URn0nUR$(d ['XX55 gS=pgUR((dH[+UUUUR,(a[R.O[R0URS 9upgS Ul['XX5XvS 9nS Ul['XX5XvS 9 g) Nz Arguments: r%r device_namerPzcuda:r*rrMF)r~rT)rrrrjrkrsetattrrrorcr]r*rtorch_model_directoryrrr model_namer)rrlr r%r^r_r)rrror}r|rWrllamas rmainr,sE D>D KK+dV$% :D D*dnn67I D-$*A*AU*JRWX\W]P^_ D(ELL)9)9:;//277<<3DDN"0td6P6PH ;;dnI~~-.2mmU]]{{ MF!* Ne   dnX]mr__main__r!)rargparse.Namespacerr)NN) rrr|r;r}boolrWr?r~zNone | torch.nn.ModulerzNone | AutoConfig)rz list[str])0 __future__rrloggingrrfrirApackaging.versionversionr`r]benchmark_helperr dist_settingsrr llama_inputsrrr r r r llama_torchr (models.torch_export_patches.cache_helperr transformersrrrbtransformers.cache_utilsr onnxruntimerp getLoggerrjrr.r1rrr__name__seedrandom manual_seedr\rrrs #  ),*G#<1   2 K FX6-1 $ a aaa a * a  aHaH$nN z DIINN4 dF r