h=bzSSKJr SSKrSSKJr SSKJr SSKJ r J r J r J r SSK Jr \"\5r"SS\5rg) ) getLoggerN)Fusion) FusionUtils) NodeProto TensorProtohelper numpy_helper) OnnxModelc H^\rSrSrSrS\4U4SjjrS"S\S\4Sjjr S\S \ S\4S jr S \ S \ S\ 4S jr S\S\ S-4SjrS\S\ S-4SjrS\S\ 4SjrS\S\\ \4S\ 4SjrS\S\ S-4SjrS\S\ S-4SjrS\ S\ S-4SjrS\ S\ S\ S\ S\S\4 SjrS rS!rU=r$)#FusionMultiHeadAttentionMMDitzG Fuse MultiHeadAttention for Multimodal Diffusion Transformer (MMDiT). modelc4>[TU]USS/S9 0Ulg)NMultiHeadAttentionSoftmax) fused_op_typesearch_op_types)super__init__unsqueeze_update_map)selfr __class__s c/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/transformers/fusion_mha_mmdit.pyr&FusionMultiHeadAttentionMMDit.__init__s$ .BU^T_`$&! start_nodereturnc2URRU/SQUSS/US9nUcgUSn[UR5S:wagURR URS5nUcg[UR 5S:wag[ US5$)a8 Detect num_heads from Reshape & Transpose of q/k/v for both Stable Diffusion 3.x and Flux 1.x: MatMul .. [-1] [24] .. | | | / / Add Concat(axis=0) | / Reshape | Transpose(perm=0,1,3,2) | (start_node) ) TransposeReshapeConcatroutput_name_to_node)rmatch_parent_pathleninputget_constant_valueshapeint)rrr$ input_indexnodes concat_shapevalues r get_num_heads+FusionMultiHeadAttentionMMDit.get_num_headss ,, :[!QURRUSS/SS/US9nU(aURUSU5$gURRUS/S/US9nU(aURUSU5$g)a Detect num_heads from subgraph like the following (num_heads=24 in this example): MatMu .. [-1] [24] .. | | | / / Add Concat | / Reshape | Transpose(perm=0,2,1,3) | SimplifiedLayerNormalization | Transpose(perm=0,1,3,2) Another variant is to an extra Concat node to join two symmetrical subgraphs: | | MatMul MatMul .. [-1] [24] .. | | | | / / Add Concat Add Concat | / | / Reshape Reshape | | Transpose Transpose(perm=0,2,1,3) | | SimplifiedLayerNormalization SimplifiedLayerNormalization | / Concat | Transpose(perm=0,1,3,2) Both patterns are used in stable diffusion 3.5 model. r!SimplifiedLayerNormalizationrr"r#)rr(r2)rr4r$r5r/s rget_num_heads_from_k2FusionMultiHeadAttentionMMDit.get_num_heads_from_k:sD #JJ00h(FG!Qex1E))%(4GHH JJ00<=sXk1E))%(4GHHr input_name output_namecSnURRU5nUcO[R"[R "/SQSS9US9nURR X@R5 [R"SX/U/URRS5S9nURRU5 URURUR'URS$) zAdd a Reshape node to convert 4D BxSxNxH to 3D BxSxD. Args: input_name (str): input name for the 4D tensor of shape BxSxNxH. output_name (str): output name for the 3D tensor of shape BxSxD, where D = N * H. Returns: str: the output name bsnh_to_bsd_reshape_dims)rrr%int64)dtype)namer inputsoutputsr@r)rget_initializerr from_arraynparrayadd_initializerthis_graph_namer make_nodecreate_node_name nodes_to_addappendnode_name_to_graph_namer@output)rr:r; new_dims_namenew_dims reshape_qs r reshape_to_3d+FusionMultiHeadAttentionMMDit.reshape_to_3dks3 ::--m<  #..rxx '/RYfgH JJ & &x1E1E F$$ . M,,Y7    +7;7K7K$$Y^^4""rmul_qNcRURRUSS/SS/5nUcgUupE[R"US/SQ5(dgURSURS'UR SnUS-UR S'UR UR SUS-5$) a  MultiHeadAttenion requires query in BSD format. This function adjusts query from BNSH to BSD format. Before: MatMul | Add Concat | / Reshape | Transpose(perm=0,2,1,3) | SimplifiedLayerNorm | Mul After: MatMul | Add Concat | / Reshape | SimplifiedLayerNorm | Reshape (shape=[0, 0, -1]) r7rrNpermrr'r"_BSNH_BSD)rr(rcheck_node_attributer*rOrS)rrUr$pathsln_a transpose_a sln_outputs r'adjust_query_from_bnsh_to_bsd_no_concatEFusionMultiHeadAttentionMMDit.adjust_query_from_bnsh_to_bsd_no_concats:zz++  +[ 9 F  <!// V\RR%**1- A\\!_ $w. Q!!%,,q/:3FGGrcURRU/SQ/SQ5nUcgUupEn[UR5S:wagURRUSS/SS/5nUcgUupx[R "US /S Q5(dg[R "US /S Q5(dg[R "US S5(dgURSURS'URSURS'[ R"S URSURS/URSS -/URRS 5SS9n URRU 5 URURU R'URU RSURSS-5$)a MultiHeadAttenion requires query in BSD format. This function adjusts query from BNSH to BSD format. Before: MatMul MatMul | | Add Concat Add Concat | / | / Reshape Reshape | | Transpose(perm=0,2,1,3) Transpose(perm=0,2,1,3) | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=2) | Mul After: MatMul MatMul | | Add Concat Add Concat | / | / Reshape Reshape | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=1) | Reshape (shape=[0, 0, -1]) )r!r7r)rrrNr'r7rr"rrWrXaxisr!rZrBrCr@rdr[)rr(r)r*rr\rrJrOrKrLrMrIrNr@rS) rrUr$r]concatr^r_sln_b transpose_bnew_concat_nodes radjust_query_from_bnsh_to_bsd;FusionMultiHeadAttentionMMDit.adjust_query_from_bnsh_to_bsdsBzz++  C   <%)"{ v||  !zz++  +[ 9 F  <!// V\RR// V\RR//BB%**1- A$**1- A ** LLOU\\!_5]]1%/0,,X6     1=A=Q=Q$$_%9%9:!!/"8"8";V]]1=MPV=VWWr unsqueezecdURRUR5nUGc[UR5S:XaN[ R "SURURSS-/URRS5S/S9nOSnURRU5cL[ R"U[RS/S/S9nURRXPR5 [ R "SURSU/URSS-/URRS5S 9nUR R#U5 URUR$UR'URSnX RUR'U$) Nr" UnsqueezerrZr')rBrCr@axesunsqueeze_axes_2)r@ data_typedimsvalsrA)rgetr@r)r*rrJrOrrKrD make_tensorrINT64rHrIrLrMrN)rrlupdated_unsqueeze_outputnew_nodeinitializer_namerps rupdate_unsqueeze_axes_1_to_2:FusionMultiHeadAttentionMMDit.update_unsqueeze_axes_1_to_2s#'#<#<#@#@#P # +9??#q(!++$??&--a07:;44[A $6 ::--.>?G'-'9'9-"-"3"3SS ($ JJ../?AUAUV!++%OOA.0@A&--a07:;44[A     $ $X .:>:N:ND ( ( 7'/q'9 $8P % %inn 5''raddr$c[UR5S:wagURRU/SQ/SQU5nUcg[ UR5nUR US5nUbUS/:wagUR US5nUbUS/:wagURRU/SQ/SQU5nUcgUR US5nUbUS/:wagUR US5nUbUS/:wagUR US5USRS'UR US5USRS'g) a. Update axes of Unsqueeze from [1] to [2] in the following pattern: Unsqueeze Unsqueeze (axes=[0]) (axes=[0]) | | Unsqueeze Unsqueeze ... (axes=[1]) ... (axes=[1]) | / | / Mul Mul | / Add Args: add (NodeProto): the Add node output_name_to_node (Dict[str, NodeProto]): mapping from output name to node Returns: bool: True if the pattern is matched and updated successfully, False otherwise. r'F)Mulrnrn)r"r"rr"r)rr"rT)r)r*rr(rget_squeeze_or_unsqueeze_axesrz)rr|r$nodes_b fusion_utilsaxes_1axes_0nodes_as rupdate_unsqueeze_axes3FusionMultiHeadAttentionMMDit.update_unsqueeze_axes(sL& syy>Q **..s4UW`buv ?"4::. ;;GAJG >Vs];;GAJG >Vs]**..s4UW`buv ?;;GAJG >Vs];;GAJG >Vs]"?? K "?? K rc:URRU/SQ/SQ5nUcgUupEpgn[UR5S:wagURRUSS/SS/5nUcgUup[R "US /S Q5(dg[R "U S /S Q5(dg[R "US S5(dgUR XB5(dgURSURS'U RSU RS'[R"S URSU RS/URSS -/URRS 5SS9n URRU 5 URURU R'URR!URSU RS5 UR#URSURSS-5$)a[ Adjust graph to change query format from BNSH to BSD for Flux model. Note that the graph pattern is complex, and we only do a shallow match here. Before: | | Transpose(perm=0,2,1,3) Transpose(perm=0,2,1,3) | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=2) | Mul Mul | / Add | Mul After (Transpose nods are removed, and a Reshape is added): | | SimplifiedLayerNorm SimplifiedLayerNorm | / Concat(axis=1) | Mul Mul | / Add | Reshape (shape=[0, 0, -1]) )Addr~r!r7r)rrrrrNr'r7rr"rrWrXrdr!rZrer[)rr(r)r*rr\rrrJrOrKrLrMrIrNr@replace_input_of_all_nodesrS) rrUr$r]r|_mul_arfr^r_rgrhris r"adjust_flux_query_from_bnsh_to_bsd@FusionMultiHeadAttentionMMDit.adjust_flux_query_from_bnsh_to_bsd]sBzz++  Q   <26/VK v||  !zz++  +[ 9 F  <!// V\RR// V\RR//BB))#CC%**1- A$**1- A ** LLOU\\!_5]]1%/0,,X6     1=A=Q=Q$$_%9%9: --fmmA.>@V@VWX@YZ!!#**Q-A1GHHrcURRU/SQ/SQ5nUcgUupEpg[R"US/SQ5(dgUR XB5(dgUR SUR S'UR SS-UR S'URUR SUR SS-5$) ax Adjust graph to change query format from BNSH to BSD for Flux model. Note that the graph pattern is complex, and we only do a shallow match here. Before: | Transpose(perm=0,2,1,3) | SimplifiedLayerNorm | Mul Mul | / Add | Mul After (Transpose is removed, and a Reshape is added): | SimplifiedLayerNorm | Mul Mul | / Add | Reshape (shape=[0, 0, -1]) )rr~r7r)rrrrNrWrXrrZr[)rr(rr\rr*rOrS)rrUr$r]r|rr^r_s r)adjust_flux_single_query_from_bnsh_to_bsdGFusionMultiHeadAttentionMMDit.adjust_flux_single_query_from_bnsh_to_bsds:zz++  G   <*.'U// V\RR))#CC%**1- A 1 / 1 !!#**Q-A1GHHrqc [R"SU/US-/URRSSS9/SQS9nURR U5 UR URUR'URUS-US-5$)NrrZTranspose_BNSH_to_BSNH) name_prefixrX)r@rWr[) rrJrrKrLrMrIrNr@rS)rrr$ transpose_qs rtranspose_reshape_bnsh_to_bsd;FusionMultiHeadAttentionMMDit.transpose_reshape_bnsh_to_bsds&&  C [M,,[F^,_     -9=9M9M$$[%5%56!!!g+q6z::rkvrO num_headscUS:deXU/nU/n[R"SUUURRS5S9nSUlUR R [R"SU5/5 U$)a. Create a MultiHeadAttention node. Args: q (str): name of q k (str): name of k v (str): name of v output (str): output name of MHA num_heads (int): number of attention heads. If a model is pruned, it is the number of heads after pruning. Returns: NodeProto: the node created. rrrAz com.microsoftr)rrJrrKdomain attributeextendmake_attribute) rrrrrOr mha_inputs mha_outputsmha_nodes rcreate_multihead_attention_node=FusionMultiHeadAttentionMMDit.create_multihead_attention_nodes,1}}AY h ## ,,-AB  *!!6#8#8i#P"QRrcBURS:XdeUnURRURS5(agURR U/SQ/SQU5nUcgUupgn[ R "US/SQ5(dgURRU/SQ/SQ5n U cgU uppn nnU RSnUURS:wagURRU S S /S S/5nUcgUunnURSn[ R "US/S Q5(dgURRUS S/S S/5nUcgUSRSU RS:wagURS nURRUSS US9nUbURRUS SUS9nUcg[ R "US/SQ5(dgURRUS S US9nUcg[ R "US/SQ5(dgO@URRUS S US9nUcg[ R "US/SQ5(dgU(aURUU5OURXcS S9nUS:XaURUUUSL5nUS::agUbURX5nOURX5nUc:URX5nUc&URX5nUcUR!UU5nUR#UUUURSUS9nUR$R'U5 UR(UR*UR,'UR.R1XgU/5 SUlg)Nrr)MatMulrr )rrrrrWrX)rr~SqrtDivrCastSliceShape)rrr"rr"rrrr~rr")rr"rYr'rrr!)r.r$)r.)rrrrOrT)op_typerfind_graph_outputrOmatch_child_pathrr\r(r* match_parentr2r8rjrarrrrrLrMrIrNr@nodes_to_remover prune_graph)rnodeinput_name_to_nodesr$softmaxr/ matmul_s_v transpose_out reshape_outq_nodes matmul_qkrUsqrt_q_2div_qsqrt_q_shape_qq_bnshk_nodesmul_kr4r k_scale_nodesrconcat_v transpose_1 transpose_2rqueryrxs rfuse"FusionMultiHeadAttentionMMDit.fuses||y((( :: ' 'q(9 : :  ++ 79QSf  = 16. ;// v|TT **..  N $  ? CJ@ (61aQ W]]1% % **..y5+:NQRTUPVW ? ${   a // V\RR  44UVUOaQRVT     ! !! $q(9 9    Q ::**:xQdw*x   **11+1J]2K"33KVV**11+1J]2K"33KVVW **11KQL_2K"33KVV    x)< =##JQR#S  >11+?RT\dhThiIA~  66uRE@@\E =;;EWE}FFub=!>>vGZ[E77%%a( 8    *6:6J6J$$X]]3 ##Z $LM r)rr)r)__name__ __module__ __qualname____firstlineno____doc__r rrr-r2boolr8strrSrarjrzdictrrrrrr__static_attributes__ __classcell__)rs@rr r s|'i' Z]B/ /im/ru/b######4.HY.H`cfj`j.H`MX9MXVY\`V`MX^"(i"(C"(H33cS\nI]3bf3jRI RI[^ae[eRIh1Iy1Ibehlbl1If ;s ;CRVJ ;) ) )  )  )  ) )V  rr )loggingrnumpyrF fusion_baserrronnxrrrr onnx_modelr rloggerr rrrs4 $== 8 K  FK  r