hSSKJr SSKrSSKJr SSKJr SSKJ r J r SSK J r J r JrJr SSKJr \"\5r"SS 5r"S S \5rg) ) getLoggerN)Fusion)AttentionMaskFormat) FusionUtils NumpyHelper) NodeProto TensorProtohelper numpy_helper) OnnxModelcX\rSrSrSrS\4SjrS\4SjrSr Sr S \ S \ S -4S jr S r g ) AttentionMask2 Fuse Attention subgraph into one Attention node. modelcXl0Ul0Ul[U5Ul[ R UlUR5Ul gN) r mask_indice mask_castedrutilsr MaskIndexEnd mask_formatget_opset_version opset_version)selfrs c/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/transformers/fusion_attention.py__init__AttentionMask.__init__sB  ' .;;"446rcXlgr)r)rrs rset_mask_formatAttentionMask.set_mask_format s&rcfXR;aX RU:XdeX RU'gr)r)rmask mask_indexs rset_mask_indiceAttentionMask.set_mask_indice#s3 ## #!1!1$!77 77!+rct[UR5S:de[[UR55$Nr)lenrnextiter)rs rget_first_maskAttentionMask.get_first_mask(s14##$q(((D))*++rmask_2dreturnNc UR[R:XagXR;aURU$URR U5(aUR RU5up#OUR RU5up4SnU(aX0RU'UR[R:XaX0RU'U$URRS5nURS:a|[R"SU/U/URRSS5S9nURR![R""SS/5[R""S S 5/5 OS nURR%U5cAURR'[R("U[*R,S/S/S S 95 [R"SX7/U/URRSS5S9nURR![R""S S 5/5 URR/U5 XPRU'U$)NTr% ReduceSum MaskReduceSuminputsoutputsnameaxeskeepdimsrort_const_1_reduce_sum_axesFr8 data_typedimsvalsraw)rrNoMaskrrfind_graph_inputrcast_graph_input_to_int32cast_input_to_int32rrcreate_node_namerr make_node attributeextendmake_attributeget_initializeradd_initializer make_tensorr INT64add_node)rr/casted input_name _cast_node output_namemask_index_node axes_names r process_maskAttentionMask.process_mask,s$   299 9 && &##G, , :: & &w / /!%!E!Eg!N FJ%)ZZ%C%CG%L "JF (2  W %   2@@ @(2  W % jj11,?    "$.."|$ ZZ00oN O  % % , ,f.C.CFQC.PRXRgRghrtuRv-w x6Izz)))4< **&&&"-"3"3SS! %..".$ ZZ00oN O  % % , ,f.C.CJPQ.R-S T O,$/!r)rrrrrr)__name__ __module__ __qualname____firstlineno____doc__r rrr!r&r-strrV__static_attributes__rrrrsA7i7'+>', ,8C8C$J8rrc(f^\rSrSrSrSSSSS/4S\S\S \S \S-S \S \S \ \ 4U4Sjjjr S\ S\ \\44SjrS\ S\ \\44SjrS\ 4SjrS\ 4SjrS\ S\ S\ 4SjrS\ S\ S\ 4SjrS\ S\ S-S \ S-S!\ S\ S-4 S"jrS#\ S$\ S%\ S\ S\ S-S \ S-S\ \ \ \ 44S&jrS6S#\ S$\ \ -S-S%\ \ -S-S\ S\ S-S \ S-S \S\S'\ S(\ S\ S)\S\ S\ S*\ S+\ S,\S\ S-4$S-jjrS7S.\ S-S#\ S$\ S%\ S\ S\ S \ S \S\S/\ S'\ S0\ S1\S\ S\ S*\ S+\ S2\S-S\ S-4&S3jjrS4rS5rU=r$)8FusionAttentiongrNFSkipLayerNormalizationLayerNormalizationr hidden_size num_headsattention_maskuse_multi_head_attention!disable_multi_head_attention_biassearch_op_typesc>U(aSOSn[T U]XU5 X lX0lU(aUO [ U5UlXPlX`lSUlSUl SUl SUl SUl g)NMultiHeadAttention AttentionT) superrrerfrrgrhrimask_filter_valuenum_heads_warninghidden_size_warning shape_infershape_infer_done) rrrerfrgrhrirjattention_op_name __class__s rrFusionAttention.__init__lst5M0R] ?C&"0>nMRWDX(@%1R.!%"&#'  $rconcatr0c[UR5S:XaURRURS5nURRURS5n[ U[ R 5(aOURS:Xa?[ U[ R 5(a URS:XaUSUSUS-4$URUR4$)a Detect num_heads and hidden_size from Concat node in the following subgraph: SkipLayerNormalization or EmbedLayerNormalization / | MatMul Shape | | Add Gather(indices=0) | | | Unsqueeze | | | Concat (*, -1, 12, 64) | / Reshape | Transpose r:r) r*inputrget_constant_value isinstancenpndarraysizerfre)rrwrf head_sizes r)get_num_heads_and_hidden_size_from_concat9FusionAttention.get_num_heads_and_hidden_size_from_concats$ v||  ! 55fll1oFI 55fll1oFI9bjj11NNa'y"**55NNa' |Yq\IaL%@@@~~t////r reshape_qcURRURS5nUc{URRUS5nUb!URS:XaUR U5$[ RSURS5 URUR4$[U[R5(a![U5S:wdUSS::d USS::a.[ RSU5 URUR4$USnUSnXE-nURS:aHX@R:wa9UR(a([ RS URU5 S UlURS:aHX`R:wa9UR (a([ RS URU5 S UlXF4$) zDetect num_heads and hidden_size from a reshape node. Args: reshape_q (NodeProto): reshape node for Q Returns: Tuple[int, int]: num_heads and hidden_size r:Concatz%s is not initializer.ryrzrr{zGq_shape_value=%s. Expected value are like [0, 0, num_heads, head_size].z>--num_heads is %d. Detected value is %d. Using detected value.Fz@--hidden_size is %d. Detected value is %d. Using detected value.)rr}r| get_parentop_typerloggerdebugrfrer~rrr*rpwarningrq)rr q_shape_valuerwrfrres rget_num_heads_and_hidden_size-FusionAttention.get_num_heads_and_hidden_sizes 55iooa6HI  ZZ**9a8F!fnn&@EEfMM LL19??13E F>>4#3#33 3M2::66=!Q&a A%q)9Q)> LLbdq r>>4#3#33 3!!$ !!$ + >>A )~~"=%%TVZVdVdfo*/&   a K3C3C$C''VX\XhXhju,1(%%radd_qkcUR(d%URRSS9UlSUlURcgURR UR S5nURR UR S5nUbUc[ RSU5 gX#:wa[ RSU5 gUR S$)NT)updaterr:zone of the inputs of %s is Nonez)the shape of two inputs of %s is not same)rsrinfer_runtime_shaperrget_edge_shaper|rr)rr input_0_shape input_1_shapes rget_add_qk_strFusionAttention.get_add_qk_strs$$#zz==T=JD $(D !    #((77 QH ((77 QH  M$9 LL:F C  ) LLDf M||Arc^US-m[[U4SjUR55n[U5S:XaT$[U5S:XdeURR S5n[ R"S[UR5Vs/sHoAPM snT/USS9nURRU5 URURU'T$s snf)N_maskc(>URST:H$r))output)nodemask_output_names r0FusionAttention.reshape_add_qk..st{{1~AQ/Qrr:rrr6r7r8axis) listfilter nodes_to_addr*rrFr rGrangerfappendthis_graph_namenode_name_to_graph_name)rr concat_nodeconcat_node_name_concat_add_qk_fp32rs @rreshape_add_qkFusionAttention.reshape_add_qks"G+6"QSWSdSdef { q # #;1$$$::66x@#-- $)$..$9:$9qF$9:%&!     !349=9M9M$$%56;s C past_kpast_vc8URRS5nURRS5nUS-RSS5nUS-RSS5n[R"SU/U/US/S9n[R"SU/U/US/S9nUR R U5 UR R U5 URURU'URURU'URRS5n URSS 5RSS5RS S 5n [R"SXV/U /U SS 9n UR R U 5 URURU 'U $) zConcatenate past_k and past_v inputs to create past_kv input. Args: past_k (str): name of past K value past_v (str): name of past V value Returns: kv_output_name (str): name of past KV value Unsqueeze_5d.rr)r6r7r8r9rz.valuez.kv_value_kvr) rrFreplacer rGrrrr) rrrunsqueeze_k_nameunsqueeze_v_name k_5d_name v_5d_namek_5dv_5drkv_output_name concat_kvs rrFusionAttention.concat_kvs ::66{C::66{Ce^,,S#6 e^,,S#6  8K!   8K!     &   &9=9M9M$$%569=9M9M$$%56 ::66x@%8@@cJRRS[]bc$$ )#$!     +9=9M9M$$%56rpresent_k_namepresent_v_namekv_nodec\SupEURRU5nURRU5nUcM[R"[R "SSS9US9nURR X`R5 UcM[R"[R "SSS9US9nURR XpR5 URRS5nURRS5n [R"SX4/U/USS 9n [R"SX5/U/U SS 9n URRU 5 URRU 5 URURU'URURU 'g) aSplit kv_node containing present KV values into separate present K and present V values. Args: present_k_name (str): name of output to store present K value in present_v_name (str): name of output to store present V value in kv_node (str): name of present KV values )index_0index_1Nrint64)dtype)r8r:Gatherr)rrKr from_arrayrarrayrLrrFr rGrrr) rrrrk_indexv_indexk_dimv_dim gather_k_name gather_v_name present_k present_vs rsplit_kvFusionAttention.split_kv8st0 **73 **73 = ++BHHQg,FWUE JJ & &u.B.B C = ++BHHQg,FWUE JJ & &u.B.B C 33H=  33H= $$ %#$  $$ %#$     +   +6:6J6J$$]36:6J6J$$]3rq_addk_addv_add name_prefixcURRURS5=(d( URRURS5n[R"U5n[ R "U5n[ R "U5nUbmURRURS5=(d( URRURS5n [R"U 5nUbmURRURS5=(d( URRURS5n [R"U 5n[ R"XgU4SS9n S[ R"UR5-n US-n URU URU /U S9 U $)Nr:rrr{ _qkv_biasr8r>r?r@) rrKr|rto_arrayr zeros_likestackprodshaperLr>)rrrrrq_biasqbkbvbk_biasv_biasqkv_bias qkv_bias_dim bias_names rcreate_combined_qkv_bias(FusionAttention.create_combined_qkv_biasesw++EKKN;itzz?Y?YZ_ZeZefgZh?i  ! !& ) ]]2  ]]2   ZZ// A?m4::C]C]^c^i^ijk^lCmF%%f-B  ZZ// A?m4::C]C]^c^i^ijk^lCmF%%f-B88RRLq1277288,, +-  &&  rq_matmulk_matmulv_matmulcb URRS5nURSURS:Xa URSURS:XdeURRURS5nURRURS5n URRURS5n [R "U5n [R "U 5n [R "U 5n U R U R :XaU R U R :XdeU R Sn[R"XU 4SS9RUSU-45nUS-nURUURUR SUR S/US9 US-n[R"SURSU/U/US 9nURURU'U/nUS -nURU[ R"S/S/S S 9 US -nURU[ R"S/U/S S 9 US-nURU[ R"S/SU-/S S 9 US-nURU[ R"S/SU-/S S 9 US-nURU[ R"S/S/S S 9 US-n[R"SUUUU/U/URRS5S 9nURURUR$'US-n[R"SUUUU/U/URRS5S 9nURURUR$'US-n[R"SUUUU/U/URRS5S 9nURURUR$'UnUn Un!UR'UUU/5 UR((GasUbURRURS5(aSOSn"[R*"[R "URRURU"555(aHUURSU"- 'UnUR-U5 URURUR$'UbURRURS5(aSOSn"[R*"[R "URRURU"555(aHUURSU"- 'Un UR-U5 URURUR$'UbURRURS5(aSOSn"[R*"[R "URRURU"555(aHUURSU"- 'Un!UR-U5 URURUR$'UR.R'U5 UU U!4$)a9Create packed QKV MatMul node before MultiHeadAttention node. This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path Returns: q_output (NodeProto): Slice node for Q k_output (NodeProto): Slice node for K v_output (NodeProto): Slice node for V MatMulrr:rr{ _qkv_weightr_qkv_outr5_q_start_indexFr=_k_start_index_v_start_indexrz_end_of_qkv_index_qkv_last_axis_q_outSlice_k_out_v_out)rrFr|rKrrrrrreshaperLr>r rGrrr rNr8rIrianyrr)#rrrrrrrmatmul_node_nameq_weightk_weightv_weightqwkwvwd qkv_weightqkv_weight_nameqkv_matmul_output qkv_matmul qkv_nodes q_slice_name k_slice_name v_slice_nameend_of_qkv_nameqkv_last_axis_nameq_slice_outputq_slicek_slice_outputk_slicev_slice_outputv_sliceq_outputk_outputv_outputinitializer_inputs# rcreate_packed_qkv_matmul_node-FusionAttention.create_packed_qkv_matmul_nodes74 ::66x@~~a HNN1$55(..:Kx~~^_O`:```::--hnnQ.?@::--hnnQ.?@::--hnnQ.?@  ! !( +  ! !( +  ! !( +xx288#BHH(<<< HHQKXXrrl3;;QAJG *]:  ((""1%z'7'7':;  "-z9%% NN1%7&'!  :>9M9M$$%56L (*::  ,+:K:KSTRU]^\_ejk'*::  ,+:K:KSTRU]^\_ejk'*::  ,+:K:KSTRU]^ab]b\cino*-@@ /[=N=NVWUX`ade`e_flqr-0@@ "4 @Q@QYZX[cebflqr)H4"" %|\CUV#$,,W5   6:5I5I$$W\\2)H4"" %|\CUV#$,,W5   6:5I5I$$W\\2)H4"" %|_FXY#$,,W5   6:5I5I$$W\\2'7G45  1 1 1 )-)C)CEKKPQN)S)SAYZ!66+..tzz/I/I%++VgJh/ijkk9GEKK$5 56$H$$U+?C?S?SD00< )-)C)CEKKPQN)S)SAYZ!66+..tzz/I/I%++VgJh/ijkk9GEKK$5 56$H$$U+?C?S?SD00< )-)C)CEKKPQN)S)SAYZ!66+..tzz/I/I%++VgJh/ijkk9GEKK$5 56$H$$U+?C?S?SD00<   +8++rrkey_padding_maskunidirectionalrr packed_qkvcUS:deUS:aX-S:wa[RSX5 gURR5RVs1sHnUR iM nnURR S5n/nU(aWURUUUUUU5unnnURURSURSURS/5 GO=[U[5(a[U[5(aUR(a<URURSURSURS/5 OURURSURSURS/5 O[U[5(at[U[5(a_UU;aYUU;aSUR(a!URURSX#/5 O"URURSX#/5 OgUR(d%URXEUU5nURU5 OURS5 U (aU(aURXX/5 O U (dU (aURX/5 U /nU(aU(aURUU/5 [ R""SUUUS9nSUlUR&R[ R("S U55 U (a9UR&R[ R("S [+U 555 UR-S5 U$s snf) aCreate a MultiHeadAttention node. Args: q_matmul (NodeProto): name of MatMul from Q path - (batch_size, sequence_length, hidden_size) k_matmul (NodeProto): name of MatMul from K path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) v_matmul (NodeProto): name of MatMul from V path - (batch_size, sequence_length, hidden_size) or (batch_size, num_heads, past_sequence_length, head_size) q_add (NodeProto): name of Add from Q path k_add (NodeProto): name of Add from K path v_add (NodeProto): name of Add from V path num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. output (str): output name of MHA key_padding_mask (str): name of key padding mask add_qk (str): name of add after Q x K' unidirectional (bool): whether to apply causal attention mask automatically or not past_k (str): name of past K value - (batch_size, num_heads, past_sequence_length, head_size) past_v (str): name of past V value - (batch_size, num_heads, past_sequence_length, head_size) present_k (str): name of present K value - (batch_size, num_heads, sequence_length, head_size) present_v (str): name of present V value - (batch_size, num_heads, sequence_length, head_size) packed_qkv (bool): whether to combine MatMuls from Q, K, V paths Note: This is for the scenario where an Attention node should be created but cannot be created because past_key and past_value are separate inputs and not one concatenated input. Returns: Union[NodeProto, None]: the node created or None if failed. r9input hidden size %d is not a multiple of num of heads %dNrmrlr5 com.microsoftrfr!)rrrgraphr|r8rFrrIrr~rrir]rrr rGdomainrHrJintincrease_counter)rrrrrrrrfrerr rr!rrrrr"rgraph_input_names mha_node_name mha_inputsrrrr mha_outputsmha_nodes rcreate_multihead_attention_node/FusionAttention.create_multihead_attention_nodes^1}} ? 7A= LLTVa m37::3C3C3E3K3KL3K4TYY3KL 33K@  (,(J(J ) %GWg   w~~a0'..2CW^^TUEVW X ) , ,Hi1P1P55!!5<<?HOOA4F UV"XY!!8??1#5xq7I8??[\K]"^_ x % %8S))----55!!5<<?H"GH!!8??1#5x"JK 5555eE=YI   i (   b ! f   /H I    /8 9h     95 6##    *!!&"7"7 Y"OP     % %f&;&;US:deU S:aX-S:wa[RSX5 gSnUcUcUcSnURRURS5nURRURS5nURRURS5nSunnnU(Ga"URRURS5=(d( URRURS5nURRURS5=(d( URRURS5nURRURS5=(d( URRURS5nU(aU(aU(aU(dgUc[ URSS35 g[ R"U5n[ R"U5n[ R"U5nURUR:XdeURSnURSnURSnUUs=:XaU:Xde eU S:aU U:wa[RS U U5 Sn URUR:waSn [R"URSS5n![R"URSS5n"[R"URSS5n#Sn$U (a![R"UUU4SS 9n%U!U"-U#-n$O[R"UUU4SS 9n%S U!-n$Sn&Sn'U(Ga[ R"U5n([ R"U5n)[ R"U5n*[R"U(R5n+[R"U)R5n,[R"U*R5n-U+U,s=:XaU!:Xde eU-U#:XdeU (a![R"U(U)U*4SS 9n'U+U,-U--n&O[R"U(U)U*4SS 9n'S U+-n&URRS 5n.UR(d*UR!U.S -UR"U[%U$5/U%S9 U(a)UR!U.S-UR"[%U&5/U'S9 UR(aU (a[RS5 gUR&SUR&SUR&SU.S-/n/UbU/R)U5 [*R,"SU/U /U.S9n0UR/S5 GO-U U.S -U(aU.S-OS/n/UbU/R)U5 OU/R)S5 U=(a Un1U1(a"UR1X5n2U/R)U25 U (a)U1(dU/R)S5 U/R)U 5 U /n3U(a]U(aVUR3SS5R3SS5R3SS5n4U3R)U45 UR5UUU45 [*R,"S U/U3U.S9n0UR/S 5 SU0lU0R8R;[*R<"SU5/5 U (a1U0R8R;[*R<"SS5/5 Ub1U0R8R;[*R<"SU5/5 U (a4U0R8R;[*R<"SU!U"U#/5/5 UR>bDU0R8R;[*R<"S[AUR>55/5 U0$)aCreate an Attention node. Args: mask_index (str | None): mask input q_matmul (NodeProto): MatMul node in fully connection for Q k_matmul (NodeProto): MatMul node in fully connection for K v_matmul (NodeProto): MatMul node in fully connection for V q_add (NodeProto): Add bias node in fully connection for Q k_add (NodeProto): Add bias node in fully connection for K v_add (NodeProto): Add bias node in fully connection for V num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. hidden_size (int): hidden dimension. If a model is pruned, it is the hidden dimension after pruning. first_input (str): first input name output (str): output name add_qk_str (str): name of Add node after Q x K' causal: whether it is uni-directional mask. past_k (str): name of input for past K value past_v (str): name of input for past V value present_k (str): name of output to store present K value present_v (str): name of output to store present V value scale: scale before softmax Returns: Union[NodeProto, None]: the node created or None if failed. rr$NTFr:)NNNzl is not an initializer. Please set do_constant_folding=True in torch.onnx.export to unblock attention fusionzInput hidden size (%d) is not same as weight matrix dimension of q,k,v (%d). Please provide a correct input hidden size or pass in 0rr{rmrrrzVMultiHeadAttention does not support relative_position_bias: cannot fuse the attention.rlr5r%z.key_keyrrr&rfr!r5qkv_hidden_sizesro)!rrrrKr|printrrrrrr concatenaterrFrhrLr>r)rrr rGr*rrrr(rHrIrJrofloat)5rr%rrrrrrrfrer2rr3r4rrrrr5has_biasrrrrrrrrr qw_in_size kw_in_size vw_in_sizeis_qkv_diff_dims qw_out_size kw_out_size vw_out_sizeqkv_weight_dimr rrrrr q_bias_shape k_bias_shape v_bias_shapeattention_node_nameattention_inputsattention_node past_existspast_kvattention_outputs present_kvs5 rcreate_attention_node%FusionAttention.create_attention_nodes\1}} ? 7A= LLTVa m =U]u}H::--hnnQ.?@::--hnnQ.?@::--hnnQ.?@!1 ZZ// A?m4::C]C]^c^i^ijk^lCmFZZ// A?m4::C]C]^c^i^ijk^lCmFZZ// A?m4::C]C]^c^i^ijk^lCmFf   >>!$%&gg   ! !( +  ! !( +  ! !( +xx288###XXa[ XXa[ XXa[ Z5:55555 ?{j8 NNJ  ! 88rxx #  ggbhhqrl+ ggbhhqrl+ ggbhhqrl+  R 1=J(;6DN2r2,Q7J_N &* %%f-B%%f-B%%f-B77288,L77288,L77288,L<>;> >> >>;. ..>>2r2,Q?+l:\I 88RRLq9 </ "jj99+F,,  (=8",, #n"56 !    (;6 **,'( !   ( ( uv"""#k1   % '' 3#--$'( N  ! !"6 7#m35=#k12   % '' 3 ''+ +VK..8 ''0"$++B/ '' 3!' Y&..vr:BB62NVVWZ\_` !((4 iJ?#--')( N  ! !+ . /  '')>)>{I)V(WX   $ $ + +V-B-BCSUV-W,X Y    $ $ + +V-B-B7E-R,S T   $ $ + +&&'9KVa;bcd   ! ! -  $ $ + +V-B-BCVX]^b^t^tXu-v,w xrcUnUnURS:Xa$URRUSS5nUbUnOgURRU/SQ/SQ5nSnUbUu ppO+URRU/SQ/SQ5nUbUuppOg/n [ UR 5H4upX;aM XSR S:XaM#U RU5 M6 [U 5S:wagU SnURRUS S5nUbmUUR SnUb5[U5S :Xa&USnURS:XaUR SnOcgUb[U5S :XaUR SnO@gURS:Xa/UUnUH$nURS:XdMUR SnM& UUnURS :Xa([UR 5S :XaUR SnUUnUVs/sHnURPM nnURS5S:wagURRU /SQ/SQ5nUc[RS5 gUu n nnSnSnSnSn/SQ/SQ4/SQ/SQ4/SQ/SQ4/SQ/SQ4/SQ/SQ4/SQ/SQ4S.nSn UR5HQun!n"URRU U"SU"S5n U cM.U!S :XaS!nOU!S":XaS!nOU!S#:XaS!nOU!S$:XaS!n O U c[RS%5 gSn#Sn$Sn%Sn&U(aU un n%n$n O4U(aU un n#n%n$O%U(aU u n n$OU(a U un n#n$n&n OU un n#n n$U&=(d U$n&URRU&/SQ/S&Q5n'U'c:URRU&/S'Q/S(Q5n'U'c[RS)5 gU'S*n(U'S+n)U'S,n*U$n+U(a?URRU$S S-/SS/5n,U,c[RS.5 gU,un+n URRU+/SQU(aSOSSSS/5n-U-c:URRU$/S/Q/S0Q5n-U-c[RS15 gU-S+n.U-S,n/Sn0S2n1U(a7URRU%/S3Q/SQ4/S4Q/SQ4/S5Q/S6Q4/U5un n0n OU(a^URRU%/S7Q/S6Q4/S4Q/SQ4/U5un n0n U#b+URU#5n1U1c[RS8U#5 gOEU(aO=URRU#/S9Q/S:Q4/S;Q/SQ4/S?Q/S@Q4/U5un n0n U(dU0c[RSA5 gU(d[U05S:aURRU0S5un n2U2b>[!U2["R$5(aU2R&S:Xa[)U25S:ag[)U25SB:wa[)U25UlUR SU:XGaU*R SU:XGawU/R SU:XGabU(d+UR,R/U0S,R S5OSn3UcW OU n4UR1U(5un5n6U5S::dU6S::a[R3SC5 gUR5U3U*U/UU)U.UU5U6UU4R SU1SD9 n7U7cgUR6RU75 UR8UR:U7R<'UbUR Sn8SEU8-n9UR?SFU8-[@RBS /SSU5[EU6U5- 5/SSG9n:URRG[HRJ"SHU4R SU:R</U9/SIU8-5UR85 U9UR S'URLROU4X/5 URLROU 5 URLROURP(dU'OU'SS,5 URLROURP(dU-OU-SS,5 URLROURP(dUOUSS,5 S!Ul)ggggs snf)JNrdAddr)rRrReshape Transposer)NNrrr)rREinsumrTr)r:Nrrr:Mulrzrcryrr{)rTrSrRr)r:rrNz&fuse_attention: failed to match v pathF)SoftmaxrRDivr)rrNr)rXrRrVr)rXWhererrY)rrrzr)rXrRrZr)rrrrz)rXrYr)rrr)rXrRrrVSqrt)rrNrr:)path1path2path3path4path5sdpar^Tr_r`raz'fuse_attention: failed to match qk path)rrrN)rYrTrSrRr)rrrrNz&fuse_attention: failed to match q pathrr[z/fuse_attention: failed to match mul sqrt q path)rTrTrSrRr)r:rrrNz&fuse_attention: failed to match k pathr%)ExpandrSEqual)rerr)CastrdrSre)rrrr)rfrerrz6fuse_attention: failed to verify shape inference of %s)rVSubrfrr)Nrr:rr)rVrgrr)Nrr:r)rZrfrgrdrr)Nrrr:rr)rZrfrgrfrdrr)Nrrr:rrrz)fuse_attention: failed to match mask pathizmFailed to detect num_heads and hidden_size for Attention fusion. Please specify those parameters in argument.) r%rrrrrrrfrer2rr3edge_modified_shape_modified_tensorr=rSreshape_modified_)*rr match_parentmatch_parent_path enumerater|rrr*countrritemsmatch_parent_pathsrget_constant_inputr~rrrr;rorgrVrrrOrrrr8rLr rNr)rOr rGnodes_to_removerIrh prune_graph);rrinput_name_to_nodesoutput_name_to_nodenormalize_node start_nodeadd_before_layernormr einsum_noder reshape_qkv transpose_qkv matmul_qkv other_inputs_i node_input root_inputmul_before_layernorm mul_childrenlayernorm_nodechildrenchild parent_nodechildren_typesv_nodesadd_vmatmul_v is_distillis_distill_addis_no_mask_attentionis_sdpaqk_pathsqk_nodeskvr matmul_qkwhere_qkafter_qq_nodesradd_qmatmul_qafter_k mul_k_nodesk_nodesadd_kmatmul_k mask_nodesr3mul_valr%attention_last_node q_num_heads q_hidden_sizenew_node unique_indexnew_edge shape_tensors; rfuseFusionAttention.fusevsy #  ! !%9 9#'::#:#:>5RS#T #/1 JJ00  ? !   =F :Q z 44DoI$>G;  ' (8(89NB4q\0033    + : |  ! !!_  $zz66z5!L  +./C/J/J1/MNL'C ,=,B!-a!))-AA!/!6!6q!9J)c,.?1.D188;   # #'; ;*:6H!==$88!&aJ"**5   ": :s;CUCU?VZ[?[$++A.J&z25=>XE%--X>    )Q . **..z;dfuv ? LLA B ")Auh $9?K9?K;\J;\J2I>@BTU  NN$DAqzz33J!adKHG|! g!%g'+$f %   LLB C   *2 'Q)Q /7 ,Q) ! ( Q9 19 .Q 7A(0 %Q9&Y**..w8acrs ?jj22@"G  EFBK  2; **66y5&/TUW[S\]K" NO&LWa**.. >gSTVWYZ\`@a  ?jj22F"G  EF 2;  #zz<<3Y?8)D;\J $  Az1#zz<<@,O8)D$   Az1!!008 %LL!Y[ab ! #zz<<EGYZ=OQShiY[st $  Az1$ (: LLD E #J!(;66z!}EJAw"7BJJ77GLLA>!  *x~~a/@J/NS[SaSabcSdhrSrZn,,99*R.:N:Nq:QRtxJ1<1D+- )-)K)KI)V &Ka=A#5C11%!!!%)&*11!4%2 H    $ $X .:>:N:ND ( ( 7&*003 +l:#330<?)//Q S1L-MN 4   ##$$!,33A6 8I8IJ! +l:  (((0 !!!$  ' ')@T%U%%% % &, % #' %,0%c%%40 0eTWY\T\o0>,&y,&U3PS8_,&\Y* S 255S5S5n+Ks+KC+K#+KZ44    T  rsD .1== 8 SSl~$f~$r