hSSKJr SSKJr SSKJr SSKJrJrJ r SSK J r \"\ 5r "SS\5r"SS \5rg ) ) getLogger)Fusion) FusionUtils) NodeProto TensorProtohelper) OnnxModelc*^\rSrSrSrS"S\S\4U4SjjjrS\SS\ \\4-4S jr S \S \ \\ \4S \ S\ 4S jrSrSrSrSrSrS\S\ \S\-44SjrS#S\S \S\S\SS\-S\S-4 SjjrSrSrS$SjrSrSrS rS!rU=r$)%FusionEmbedLayerNoMaskz Fuse embedding layer into one node (EmbedLayerNormalization). It supports the following model types: BERT, DistilBert, ALBert. model descriptionc>[TU]USSS/U5 [U5UlSUlSUlSUlSUlg)NEmbedLayerNormalizationLayerNormalizationSkipLayerNormalizationF)super__init__rutils shape_infershape_infer_done attention embed_node)selfr r __class__s d/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/transformers/fusion_embedlayer.pyrFusionEmbedLayerNoMask.__init__sP   % !#; <   !'  %addreturnNcURRUS/S/5nUcgURRUS/S/5nUcgUSUS4$)NGatherr)r match_parent_path)rr gather_0_path gather_1_paths rmatch_two_gather'FusionEmbedLayerNoMask.match_two_gather%sa 44S8*qcJ   44S8*qcJ  Qq!111r layernorminput_name_to_nodesis_distil_bertcnURRUSUSS9UlURbgURSU;agX!RSn[ UVs/sHoUR PM sn5nU/SQ:XapUHjnUR S:XdMURR U/SQ/S Q5nUcM;US RSURS:XdM`US Ul g [U5S :XaUSR S :XaUSRSU;aX$SRSn [U 5S :XaU SR S:XapU SRSU;aZX)SRSn U HnUR S:XdMXpl g [ U Vs/sHoUR PM sn5nU(a/U/SQ:wa&U/SQ:waU/SQ:wa[RS5 ggU/SQ:waU/SQ:wa[RS5 ggs snfs snf)aoCheck that LayerNormalization has a child of Attention node or subgraph like Attention. Args: layernorm (NodeProto): LayerNormalization node input_name_to_nodes (Dict[str, List[NodeProto]]): map from input name to nodes is_distil_bert (bool): whether it is DistilBert or not Returns: bool: whether there is Attention node or subgraph like Attention AttentionF) recursiveTr)MatMulr/r/rr)Addr/MultiHeadAttentionr/)NNrrr#r/r0)r/r/r/Shaper)r0r/r/r/r4r4)r0r/r/r/r4z> %   A &9 9&'7'7':;H EH5H EF U U <<#;; JJ88I*E (U2Y__Q-?9CSCSTUCV-V/4Qx,#! x=A (1+"5"5"AhqkFXFXYZF[_rFr/ 0B0B10EFMM"a'!!$,,5!!$++A.2EE+!,<,C,CA,FG!D||{2)-#""(E(JE5E(J!K "cc"&]]"&TT [\ " !%  [\q!F:)Ks H-4H2cJURRUSS/SS/5nUc%URRU/SQ/SQ5nUcgUSUSpeURSU:wagURRU/S Q/S Q4/S Q/S Q4/U5upxnUcgUSn URR U SS5(a"URR U S S5(dgUSn URR U SS5(dgUSn U RSU:wagg)aMatch position embedding path from input_ids to Gather for DistilBert. Pattern is like the following: (input_ids) | Shape | | Gather (indices=1) | | | Cast (optional) | | | Range (start=0, end=*, delta=1) | | | Unsqueeze | / Expand | Gather Expandr4r#)rHWhereReshaper4)r#r#r3rFrr2) UnsqueezeRangeCastr"r4)rrr#rr)rKrLr"r4)rrr#rr3T)r r$r9match_parent_pathsrcheck_node_input_value) rposition_embedding_gather input_idsoutput_name_to_noderBexpandshape_path2 range_node gather_node shape_nodes r#match_position_embedding_distilbert:FusionEmbedLayerNoMask.match_position_embedding_distilberts>* ,,-FSZH[^_ab]cd =JJ00)7E }a%) ;;q>Y &jj33 BOT:LI   ! =1X JJ - -j!Q ? ?DJJDeDefprsuvDwDwBi  11+q!DD2Y   A ) +rcg)aIMatch position embedding path from input_ids to Gather for Roberta. Roberta Embedding Layer Pattern (* is optional since it might be removed by ORT, ? is the padding word id): (input_ids) --> Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Mul -- Cast(to=7) -- Add(B=1) -- Cast(to=7)* --> Gather | ^ V | +------------------------------+ Roberta new pattern from transformers v4.9: (input_ids) --> Equal(B=?) -- Not -- Cast(to=6) -- CumSum(axis=1) -- Add(B=0) -- Mul -- Cast(to=7) -- Add(B=1) --> Gather | ^ V | +-------------------------------------------+ start_node = position_embedding_gather start_index = 1 # match optional Cast node. parent = self.model.get_parent(start_node, start_index, output_name_to_node) if parent is None: return if parent.op_type == "Cast": if OnnxModel.get_node_attribute(parent, "to") != 7: return start_node = parent start_index = 0 i, path, return_indices = self.model.match_parent_paths( start_node, [ (['Add', 'Cast', 'Mul', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0]), (['Add', 'Cast', 'Mul', 'Add', 'CumSum', 'Cast', 'Not', 'Equal'], [start_index, 0, 0, 0, 0, 0, 0, 0])], output_name_to_node) if path is not None: # constant input of Add shall be 1. i, value = self.model.get_constant_input(path[0]) if value != 1: return False _, self.padding_word_id = self.model.get_constant_input(path[-1]) return input_ids == path[-1].input[0] FrrQrRrSs r match_position_embedding_roberta7FusionEmbedLayerNoMask.match_position_embedding_robertasZrcURRUSS/SS/U5nUcgUupVURRURS5nUb[ UR 5S:XaUR SS:XaUR RUSS/5(a_UR RUSS/5(a<[ UR5S:Xd$UR RUSS/5(dgURR5nUS :a [R"US S/5(dgO$UR RUSS/5(dgURRUSU5n U cgU RS :XaAUR RU SS5(dgURRU SU5n OU n U bU RS :wagUR RU SS5(dgURRU SU5n U bU RS :wagX+RS:H$)awMatch position embedding path from input_ids to Gather for BERT. BERT Embedding Layer Pattern: (input_ids) / / Shape / | / Gather (indices=1) / | / Add (optional, B=0) / | Gather (segment_ids) Unsqueeze (axes=0) \ | | \ Gather Slice (data[1,512], starts=0, ends=*, axes=1, steps=1) \ / | Add Gather \ / Add | LayerNormalization SlicerKr#r3Fr axesr0r"r4) r r$get_constant_valuer9r;rUrrPget_opset_versionrcheck_node_attribute get_parentr8) rrQrRrSpathslice unsqueeze slice_weight opset_versionrAgatherrUs rmatch_position_embedding_bert4FusionEmbedLayerNoMask.match_position_embedding_berts,zz++ % k " F    <zz44U[[^D  $L&&'1,""1%* 11%QC@@ 11%QC@@U[[!Q&$***K*KESTWXVY*Z*Z 446 2 33IvsKKL::44YA3GGzz$$Y3FG < <<5 ::44T1a@@ZZ**44GHFF >V^^x7 11&!Q?? %%fa1DE =EMMW4KKN**rcdURXU5(agURXU5(agg)NTF)rrr[r_s rmatch_position_embedding/FusionEmbedLayerNoMask.match_position_embedding9s7  - -.GTg h h  3 34MZm n nrc <URSnU(aURSOSnURSnUR(d%URRSS9UlSUlURbURR U5nURR U5nU(aU(de[ U5S:Xa[ U5S:Xa USUS:Xd[RSUSU35 gU(aUURRXE5(d5[RS US URR U535 gURRURS 5n U b[ U R5S:wa[RS 5 gURRURS 5n U b9[ U R5S:wd U RSU RS:wa[RS 5 gU(azURRURS 5n U b9[ U R5S:wd U RSU RS:wa[RS5 gU RS U RS ::aU[RSURS SU RS SURS SU RS 35 U(aU RS W RS ::aU[RSURS SU RS SURS SU RS 35 U RS U RS ::aU[RSURS SU RS SURS SU RS 35 g)zXSanity check of embedding weights, and match hidden_size of weights and shape of inputs.r#NT)updater3z^Cannot fuse EmbedLayerNormalization: input_ids and position_ids not matched in 2nd dimension: z vs FzYCannot fuse EmbedLayerNormalization: input_ids and segment_ids does not have same shape: z != rzICannot fuse EmbedLayerNormalization: word embedding table is not expectedzMCannot fuse EmbedLayerNormalization: position embedding table is not expectedzLCannot fuse EmbedLayerNormalization: segment embedding table is not expectedzword_embedding_table (z) size z <= position_embedding_table (z <= segment_embedding_table (zposition_embedding_table () r9rr infer_runtime_shaperget_edge_shaper;r<info compare_shaperhrUwarning) rword_embedding_gathersegment_embedding_gatherrQrR segment_ids position_idsinput_ids_shapeposition_ids_shapeword_embedding_tableposition_embedding_tablesegment_embedding_tables rcheck_embedding&FusionEmbedLayerNoMask.check_embeddingGs)//2 ;S.44Q7Y] 066q9 $$#zz==T=JD $(D !    '"..==iHO!%!1!1!@!@!N "'9 99O$)*+q0#A&*)D)DQ)G(HPdPjPjklPmOnoMNgNmNmnoNpMqqxyQyWyWXYyZx[\  #))!,0G0M0Ma0PP,-B-H-H-K,LGThTnTnopTqSrsPQiQoQopqQrPssz{R{X{XYZ{[z\](--a04K4Q4QRS4TT01J1P1PQR1S0TT[\t\z\z{|\}[~\]u]{]{|}]~\FG^GdGdefGgFhir input_namec2SnURRU5nUbXURRR[ R :wa URRU5upBXB4$UnXB4$URRU5upBXB4$)apCast a graph input or node input to int32. Args: input_name (str): name of graph input or node input Returns: A tuple of casted input name and the cast node. int32_output (str): If input is int32, it is the input name, Otherwise it is output name of Cast node. input_cast_node (Union[None, NodeProto]): Cast node. It could be None if input is int32. N) r find_graph_inputtype tensor_type elem_typerINT32rcast_input_to_int32)rrinput_cast_node graph_input int32_outputs r cast_to_int32$FusionEmbedLayerNoMask.cast_to_int32sjj11*=  "++559J9JJ04 0N0Nz0Z- ,, * ,,-1JJ,J,J:,V )L,,rrRr~rQrrc /n URU5upURRS5n URS:XaURSn URSn OURSn URSn SnUbQURURS5upUUURSURSURSU U /nO#USURSURSSU U /nUb5UR S5 URU5upjUR U5 U S -U S -/nU(aUbUOU S -nUR U5 [ R"SUUU S 9nS UlURH1nURS:XdMURRU/5 M3 [UR5S:Xa1URR[ R"SS5/5 U R U5 U H&nURURUR'M( UR RU 5 UUlU$)aCreate an EmbedLayerNormalization node. Note that segment embedding is optional. Args: input_ids (str): input_ids for word embeddings layernorm (NodeProto): LayerNormalization or SkipLayerNormalization node. word_embedding_gather (NodeProto): the Gather node for word embedding position_embedding_gather (NodeProto): the Gather node for position embedding segment_embedding_gather (Union[None, NodeProto]): the Gather node for segment embedding, or None. Returns: NodeProto: the EmbedLayerNormalization node created. rrr#r3rdNr_output_dummy_mask_index_embedding_sum)outputsnamez com.microsoftepsilong-q=)rr create_node_namer8r9appendr make_nodedomain attributerextendr;make_attributethis_graph_namenode_name_to_graph_name nodes_to_addr)rrRr)r~rQrrembedding_sum_outputembedding_sum_namerrV node_namegammabetaembed_node_inputsrembed_node_outputsrrattrAs rcreate_fused_node(FusionEmbedLayerNoMask.create_fused_nodesn. )))4 JJ//0IJ    4 4OOA&E??1%DOOA&E??1%D  # /!//0H0N0Nq0QRNK%++A.)//2(..q1! %++A.)//2!   #  $ $R ("00>OL  $ $\ 2')3YAT5TU );)G%YYiMiD  % %d +%% % &  , &&Cxx9$$$++SE2' z## $ )  ' ')>)>y')R(S T J' D6:6J6JD ( ( 3!   .$rc~URRURSURS5 SUlg)NrT)r replace_input_of_all_nodesr6 prune_graph)rr)rs r finish_fusion$FusionEmbedLayerNoMask.finish_fusion s5 --i.>.>q.A:CTCTUVCWXrcURS:H=(a: [UR5S:=(a [URS5S:$)Nrrdr)r8r;r6)rrAs r"is_skip_layer_norm_with_sum_output9FusionEmbedLayerNoMask.is_skip_layer_norm_with_sum_outputsD 88nc$++>NQR>RnWZ[_[f[fgh[iWjmnWnnrc URU5nUcgUupxURSn URSn URXSS9(dgURUSU5(dgURS:XaVUR U5n Sn Un U (aUR SOSnUSL=(a URRU5SLnOUn URS:XaSOSn [UR 5U :aUR U OSnUSL=(a URRU5SLnU=(a X;=(a [X>5S:nUSL=(a! URS:g=(d U=(d Un URU UUUUU U U(aUOSS9nU (a?S U R U 'U(d)URRUUR S 5 URUU5 g ) NFr#r+rrdr0r)rr_no_use__to_be_removed_r3T) r'r9rErr8rr6r find_graph_outputr;rrr)rr)add_before_layernormr*rSoptional_segment_gather two_gatherr~rQrRrneed_embedding_sum_outputsum_output_indexnode_with_sum_output sum_outputis_sum_graph_outputis_sum_used_by_multiple_nodesrs r fuse_gpt2 FusionEmbedLayerNoMask.fuse_gpt2s&(**+?@  ;E8)//2 066q9 ,,Y\a,b##$94AZ[[    8 8(,(O(OPY(Z % #, 0I))!,tJ#-T#9"u @\@\]g@hpt@t #7 $8$@$@E$Iqq +2236FF%++,<=  $.T#9"u @\@\]g@hpt@t o AosK^KjGknoGo **44)?)$,,5m9LmPm & ++   ! % # !:-@zd,   %">?SV^U_bcad"e " *$;A$>!,,-FSfgg001IVijj+D'@ $(, %##$9Unoo++   ! % $   91rc"URRUS/S/5nURS:Xa UcgUSnSnOURRUS/S/5nURRUS/S/5nUcUbUcgUSnUSnO8Ub1Uc.URRUS/S/5nUcgUSnUSnOUnSnURXX#U5(agUR XX#5(agUR XX#5(agg)Nr0rrr"r#)r r$r8rrr) rrAr*rSfirst_add_pathrrr%r&s rfuseFusionEmbedLayerNoMask.fusesJ55dUGaSI <</ /%#1!#4 &* # JJ88zA3OM JJ88zA3OM$)B!)'5a'8$*7*:'*}/D!%!=!=dUGaS!Q!)'5a'8$*7*:''+$*.' >> (;Ri      >$6I _ _  `r)rr:rrrrr)zno mask)NFN)N)__name__ __module__ __qualname____firstlineno____doc__r strrrtupler'dictlistboolrEr[r`rrrurrrrrrrrr__static_attributes__ __classcell__rs@rr r sL i c   2I 2$y)?S9T2T 2RR"#tI"67R R  Rh<|-^F+P HT--c4);K6K0L-<$("``` ) ` $- ` #'"2 `Dj`D orvOb'R0d""rr cF^\rSrSrSS\4U4SjjjrSrU4SjrSrU=r $)FusionEmbedLayerNormalizationir c2>[TU]US5 X lg)Nz with mask)rruse_mask_index)rr rrs rr&FusionEmbedLayerNormalization.__init__s  ,,rcURn[UR5S:Xa<URRU5 [R SUR 5 O}[UR5S:aCURS(d/XRS'[R SUR 5 O![R SUR 5 gUHn[R SUR 5 URS:XaURSURS'MQURS :XdMcURSURS 'M g) Nzappend mask to %szreplace mask in %szskip mask in %szupdate mask_index in %sr-r#rdr1re) rr;r9rr<r=rr8r6)r mask_int32attention_nodesrattention_nodes r replace_mask*FusionEmbedLayerNormalization.replace_masks__ z A %    # #J / LL,joo > !! "Q &z/?/?/B",  Q  LL-z ? LL*JOO < -N LL2N4G4G H%%4*4*;*;A*>$$Q'''+??*4*;*;A*>$$Q' .rc>SUlSUlSUl[TU]XU5 URcgUR (d'[ RS5 URS5 gURc4URc'[ RS5 URS5 gUR(aURRSnOURRSnX$nURRU5(aFUVs/sHoRS;dMUPM nnURXF5 URS5 gXC;a([ RSU5 URS5 gX4nURS ;aUVs/sHoRS;dMUPM nnWRS :XaBURS n[U5[U5:XaURR!U5 URXF5 URS5 ggs snfs snf) NzG--use_mask_index is not set: EmbedLayerNormalization will not have maskz EmbedLayerNormalization(no mask)zLEmbedLayerNormalization will not have mask since attention node is not foundrdre)r-r1z"EmbedLayerNormalization(with mask)zHEmbedLayerNormalization will not have mask since %s is not a node output) ReduceSumrMrr)rr:rrrrr<r=increase_counterr9r rr8rr;nodes_to_remover)rrAr*rSrchildren_nodesrrs rr"FusionEmbedLayerNormalization.fuses#  T0CD ?? " "" LLb c  ! !"D E  >> !d&:&:&B LLg h  ! !"D E  >>--a0J--33A6J,8 :: & &z 2 20>v,,RuButOv   j :  ! !"F G   0 LLceo p  ! !"D E ". <<0 00>v,,RuButOv||{*!ZZ] ~&#o*>>((//5   j :  ! !"F G 1wwsH61H6!H;8H;)rr:rr)F) rrrrr rrrrrrs@rrrs%-i--?*-H-HrrN)loggingr fusion_baser fusion_utilsronnxrrr onnx_modelr rr<r rr^rrrsC $// 8 P VP fGH$:GHr