hGESSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK J r J r SSK J r SSKJrJr SSKJr \R&"S5rS'Sjr"SS 5rS rS rS rS rSrS\4SjrSrSS/4S\4SjjrSr Sr!Sr"Sr#S\4Sjr$\%S:XGa\"5r&\ "\&RN5 \&RPS:d-\&RRS:d\&RP\&RR-S:a\RUS5 \RVRY\&R5(ap\&RZ(d_\&R\(d\/"S \&RS!35e\RaS"\&R5 \Rb"\&R5 \"\&RP\&RR\&R5r2\&RZ(d \$"\2\&5 \"\2Rf5r4\RaS$5 \2RfRqS%S&5r9\"\4\95 gg!\5a- \RmS#\2Rf35 \2Rnr4Njf=f)(N)get_ort_environment_variables setup_logger)main)PRETRAINED_GPT2_MODELS Gpt2Helper) OnnxModelc [R"5nURSSS[SSR [ 5-S9 URSS[S S S 9 URS S[ S SS 9 URSS[ SSS 9 URSSSSS9 URSS9 URSSSSS9 URSS9 URSSSSS9 URSS9 URSSSS9 URSS9 URS SSS!S9 URSS"9 URS#SSS$S9 URSS%9 URU5nU$)&Nz-mz--model_name_or_pathTz2Model path, or pretrained model name in the list: z, )requiredtypehelpz--csvFzgpt2_parity_results.csvz#path of csv file to save the result)r r defaultr z --test_casesiznumber of test cases per runz--runs(znumber of repeated runs --use_gpu store_truezuse GPU for inference)r actionr )use_gpuz--allz'run all combinations of mixed precision)allz-e--use_external_data_format)r r)use_external_data_formatz --verbose)verbosez --skip_testzEdo not run test, and only rank experiments based on existing csv file) skip_testz --overwritezOverwrite existing csv file) overwrite) argparseArgumentParser add_argumentstrjoinrint set_defaults parse_args)argvparserargss j/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/transformers/models/gpt2/gpt2_parity.pyparse_argumentsr&s  $ $ &F    ADIINdDe e   ) 2    +  5sBMfg  eLOfg &  6   E" :US_` 7  eLI &  T   %(  *   %(   T "D Kc \rSrSrSrSrSrg) ParityTaskacDX lXlX0l/UlSUlg)Nr) total_runs test_casescsv_pathresultsrun_id)selfr-r,r.s r%__init__ParityTask.__init__bs$$   r'c[RR5RS5nUSUR3nU=RS- sl[ /UQSPUR PSPUR PUUURS9nU(aURRU5 U$![a [RSU35 SnU$f=f)Nz %Y%m%d%H%M%S_z-tz-r)experiment_namer0 csv_filenamezFailed to run experiment ) datetimenowstrftimer0rr-r,r.r/append Exceptionlogger exception)r1r"r7 start_timer0results r%runParityTask.runis&&**,55nE US[TUS54$)Nr6r)ra)itemrow_maps r%print_wins..sd1guWT!W-='>?r'T)keyreversez Wins:z Based on z* wins and a scoring function, the ranking:rza{:02d}: WINs={:02d}, run_id={}, latency={:5.2f}, top1_match={:.4f}, size={}_MB, experiment={}, {}r\r] experiment) printdictsorteditemsr>debuginfo enumerateformatrZrWr) winsrP test_namerS sorted_winsrankprevious_valuecountrhvalueres @r% print_winsr{s" G (OG!$H  JJL? K LLI;f[M23 KK)I;&PQR DN():):)<=|  "DC8}# w~~#C(c"345-.L)57   >r'c:0n0nUHnUSnSX%'SX5'M [USSS9n/SQn[R"XgS9nUR5 /SQn [ U5n [ U S - 5GHn X n [ U S [5(a[R"U S 5n OU S n [ U S -U S 5GHynXnS nU HnU UUU:wdMS n O U(dM,[ US [5(a[R"US 5nOUS n[RRU US S S9unn[RRU USS S9unnUbCUS:a=[U S5[US5:aX,S==S - ss'OX/S==S - ss'US:a=[U S5[US5:aXrqr{)rPoutput_csv_path utest_wins ttest_winsrSr0rQ column_nameswriterrequired_match_columns num_resultsiresult1ajresult2 all_matchedcolumnbutest_statistic utest_pvaluettest_statistic ttest_pvalues r%run_significance_testrsJJX   osB /7  A!E$i {Q'AgG'";rqwarning)rmodel onnx_modelrnodes r%get_last_matmul_node_namer"s IIn %E5!J$88: ;;  a % %)< << < {{11!499 :D ||x 9$))EFyy NNB4<<.PYZ^ZcZcYdef r'cURnSUS3R5nUR(aURS5 USSSU/- nU(aUR S/UQ5 U$)N-m  -o --use_gpu -p fp16r--io_block_listlogitsz--node_block_list--op_block_list)model_name_or_pathsplitrr<extend)r$last_matmul_node_name op_block_listr parameterss r%get_mixed_precision_parametersr1sz  # #Eug2399;J $$67 J,=}=> r'FastGeluLayerNormalizationtaskc[XU5nSR[U55nU(aSUS3nOSUS3n[5nU(a USUS3-nUR XF5 g)N,Mixed precision baseline +  in FP32z=Mixed precision baseline (logits output and last MatMul node z in FP32)z ())rrrnrrB)rr$rrrop_block_list_strrYenv_varss r% run_candidaterCsr 0][J !67,->,?xHNOdNeeno,.H8*A&&HHZr'c<URnSUS3R5nUR(aURS5 UR(aURS5 SUS3R5nUR(aURS5 X#4$)Nrz -o -p fp32rrr)rrrr<r)r$r fp32_baseline fp16_baselines r% get_baselinesrXs  # #E% ,224M ||[) $$9:% 56<<>M $$9:  ''r'cSS/nURX-S5 S/nURX-S5 URX-S/-[U5-S/-S5 /nUnUHVnS/UV s/sH oU:wdM U PM sn -n URX-U -S US 35n U (dMEURU 5 MX [US S 9n [ S U 5 gs sn f)z:Step 0 is to check which operator in FP16 causes most lossrrzFP16 except logitsz--keep_io_typeszGraph I/O FP32, Other FP16rz--force_fp16_initializerszFP32 except weights in FP16z FP32 except z in FP16c US$)Nr\rI)ys r%rf"run_tuning_step0..~s !? !GHH] $&BC HH#4"55W EIdHee% G*+'.M'Q"Wq'.MM -1MA\RTQUU]C^_ 6 ! ( ( 0 *0NOJ H*U/Ns ' C4CcNUHnSU/nURX-SUS35 M! g)zKStep 1 is to figure out which optimized operator in FP32 could benefit mostrrrN)rB)rmixed_precision_baselinerrrs r%run_tuning_step1rs4*B/   $ 4)"X 6 r'c /SQnUVs/sH oDU;dM UPM nnUHGnXe;dM /UQUPnUR/UQSPUQSRSRU5U55 MI gs snf)zAssumed that you have run step 0 and 1 to figure out that Logits FP32 and some operators shall be in FP32, This step will try add one more operator. )rrSkipLayerNormalizationrz(Mixed precision baseline + {},{} in FP32rN)rBrsr)rrrcandidate_fp32_opsxfp32_opsrrs r%run_tuning_step2rsV-D-am1C-HD +hOOM HHN*N,=N N:AA#((8BTVXY Es A,A,c @^ [R"SURUR/S9n[ U5up4UR US5n/nU(a%SU;aUS(aUSR S5nO [S5e/m U(a%SU;aUS(aUSR S5m O [S5eUR(d[RS 5 gUR US 5 [US 5n[XU/S 9 U 4S jnUR(a0[XT U5 [X/S 9n [!X U5 [#X U5 O [UUUU"/SQ5S 9 [XUS/S 9 [UUUU"/SQ5S 9 [UUUU"/SQ5S 9 g)N onnx_models) new_folderremove_existingz FP32 baselineoptimized_operatorsrz!Failed to get optimized operators operatorszFailed to get operatorsz5skip mixed precision since --use_gpu is not specifiedz FP16 baselineraw)rcB>UVs/sH oT;dM UPM sn$s snfNrI)rrrs r% get_fp32_ops run_parity..get_fp32_opss0Qr-Q000s )rrAddr)rrrr)rEmbedLayerNormalizationrrr)rget_onnx_pathsrrrrBrrXrr>rqrrrrrrr) rr$onnx_model_pathsrrrArrrrrs @r% run_parityrs!00 00 $1#6 M XXm_ 5FM (F2?T8U45;;C@ >??G ;&(f[.A%++C0455 << KLHH]O,56Fu6MN$32F1 xxg}E#A$mo#p GG   !&'^_  d"7 |T  "#fg    " j r'__main__r_i'zNot enough test cases or runs to get stable results or test significance. Recommend test_cases >= 100, runs >= 20, test_cases * runs >= 10000.z Output file zK existed. Please remove the file, or use either --skip_test or --overwrite.z6Remove existing file %s since --overwrite is specifiedzFailed to load csv z#Start running significance tests...z.csvz .stats.csvr):rrMr9rloggingosr scipy.statsrbenchmark_helperrrconvert_to_onnxr gpt2_helperrrrr getLoggerr>r&r)rTrZrar{rrrrrrrrrrrDr$rr-rrpathexistsrrrXrqremoverr.rPr=r?r/replace summary_csvrIr'r%rs%   H :   2 @F:DO'T[+| c ,34  * (V6  CZCL z  D  B$//DII2MPU2U S  ww~~dhh~~txxj(st  KKPRVRZRZ [ IIdhh  dootyy$(( ;D >>4$T]]3  KK56--'' =K$ ,A4 .t}}o>?||s!H++0II