U h.@sddlmZddlmZddlZddlmZmZddlmZm Z m Z m Z ddlm Z ddlmZeeZGdd d ZGd d d ZdS) ) getLogger)TupleN) array_equalndarray) NodeProto TensorProtohelper numpy_helper)onnx_pb) OnnxModelc@seZdZedddZeeeefdddZd*edd d Z edd d Z edd dZ e ddZ e d+edddZe d,edddZe ejdddZe d-eedddZedd d!Zd"d#Zd$d%Zd&d'Zd(d)ZdS). FusionUtilsmodelcCs ||_dSNr )selfrrI/tmp/pip-unpacked-wheel-socb9apf/onnxruntime/transformers/fusion_utils.py__init__szFusionUtils.__init__) input_namereturncCsn|j|}|dk rL|jjjtjkrL||\}}t d|dd|fSt d|d|dk d|fS)NzCasted graph input z to int32TzDid not cast graph input z to int32: found F) rZfind_graph_inputtypeZ tensor_typeZ elem_typerINT32cast_input_to_int32loggerdebug)rrZ graph_input cast_output cast_noderrrcast_graph_input_to_int32s z%FusionUtils.cast_graph_input_to_int32int32)rc Cs|d|}|g}|j}||krF||}|rF|jdkrF|jdg}tjd||gd}|dkrlttj}n0|dkrttj }n|dkrttj }nt d|j td |g|j|||fS) N_Castr)inputsoutputsrZfloat32Zfloat16z"Invalid target_type: {target_type}to)routput_name_to_nodeop_typeinputrZ make_nodeintrrFLOATZFLOAT16 ValueError attributeextendZmake_attributeadd_node) rrZ target_typerr!r$ parent_noderZto_typerrr cast_inputs$       zFusionUtils.cast_inputcCs ||dS)Nr)r.)rrrrrr:szFusionUtils.cast_input_to_int32cCs|j}||}|D]h}|jdkrd}|jD]&}|jdkr.|jttjkr.d}qVq.|r|j d}|j ||j ||qdS)Nr Fr#Tr) rinput_name_to_nodesr%r*nameir'rroutput remove_nodereplace_input_of_all_nodes)rrr/nodesnodeZis_int32ZattZ output_namerrrremove_cast_int32=s     zFusionUtils.remove_cast_int32cCszd}|j||krJ|||j|krJ||j||t||j|}||j|<||krl|||n |g||<|S)Nr)r&removelenappend)r6r1new_input_namer/old_input_referencerrrupdate_node_inputLs   zFusionUtils.update_node_inputrc Cs<|j|}|j|}t||||}|dko6|| } | S)a Before: (input)-->parent-->node-->(output) After: (input)-->parent--> | +----->node-->(output) This function returns a flag whether the parent node can be removed. r)r&r r=Zfind_graph_output) rr6r-r/Znode_input_indexZparent_input_indexZold_input_namer;r<Zparent_can_be_removedrrr skip_parent\s  zFusionUtils.skip_parentN)attribute_namecCsV|}|jD]}|j|kr t|}q t|trJt|ttfoHt||ddS||kSdS)aVerify that a node has expected value for an attribute. Args: node (NodeProto): a node to check attribute_name (str): name of attribute expected_value (Any): expected value of the attribute default_value (Any, optional): default value if the attribute does not exist. Defaults to None. Returns: bool: whether the check is passed or not FZ equal_nanN)r*r0rZget_attribute_value isinstancelistrr)r6r?expected_value default_valuevalueattrrrrcheck_node_attributers     z FusionUtils.check_node_attribute)tensorcCst|tjstdt|t|jdks8|jtjjkr@td|j r|t t j |j dd|j}t |ddg}||_ ntd|S) zTranspose a 2-D INT8 TensorProto Args: tensor (TensorProto): tensor to be transposed Returns: tensor (TensorProto): transposed tensor z5Expected input type is an ONNX TensorProto but got %sz'Only INT8 2-D tensors can be transposedZint8)dtyperzonly raw buffer supported)rA onnx_protorr)rr9dims data_typeZINT8raw_datanumpyZreshapeZ frombufferZ transposetobytes)rHZ int32_dataZint32_transposed_datarrrtranspose_2d_int8_tensors  z$FusionUtils.transpose_2d_int8_tensorT)r6rcCs|jdkrtd|j||jd}|dkr8dS|jdkpX|jdkoX|jddk}|rf|sfdSt|jdkrxdS||jd}|j|jkrdS|dkrdSt |dkS) aVerify if a provided QuantizeLinear (Q) / DequantizeLinear (DQ) node is a good candidate for fusion. It is a good candidate for fusion if: (1) The Q/DQ node is for per-tensor quantization if allow_per_tensor_quantization_only is `True` (2) The Q/DQ node should have constant scale (3) The Q/DQ node should have a zero point of 0 Args: node (NodeProto): a Q/DQ node to check Returns: bool: whether the check is passed or not >DequantizeLinearQuantizeLinearz+Provided node is not a Q/DQ node. Op Type: rKNFrrIT) r%rrget_constant_valuer&ndimshaper9rPall)r6rZ"allow_per_tensor_quantization_onlyZscaleZscale_has_single_elementZ zero_pointrrrcheck_qdq_node_for_fusions " z%FusionUtils.check_qdq_node_for_fusion) input_indexcCsVt|j|kst|j|j|}t|trJt|ttfoHt||ddS||kSdS)a7Verify that a node has expected input value Args: node (NodeProto): a node to check input_index (int): index of its input to be verified expected_value (Any): expected value of the input Returns: bool: whether the check is passed or not Fr@N) r9r&AssertionErrorrrUrArBrr)rr6rZrCrErrrcheck_node_input_values  z"FusionUtils.check_node_input_valuecCsg}|jD]F}|jdkr|jd|jkr|j|jd|jd||q|r||j|t dt |ddS)z>Remove Identity nodes, except those right before graph output.ZIdentityrzRemoved z Identity nodesN) rr5r%r2get_graphs_output_namesr4r&r:Z remove_nodesrinfor9)rnodes_to_remover6rrrremove_identity_nodess   z!FusionUtils.remove_identity_nodescCs|jdSr)rremove_cascaded_cast_nodesrrrrrasz&FusionUtils.remove_cascaded_cast_nodescCs|jdSr)rremove_useless_cast_nodesrbrrrrcsz%FusionUtils.remove_useless_cast_nodescCsB|jjdd}|dkrdSg}|jD]`}|jdkr(||jd}||jd}|r(|r(||kr(td|j d|| |q(|r>t |j }t |j }|D]}tt |j|@rtt |j|@st|j|jddkr|j|jd|jdnqn|j|jd|jd|j|qdS) ziRemove reshape node that is not needed based on symbolic shape inference: input and output has same shapeT)updateNZReshaperzRemove reshape node z* since its input shape is same as output: rK)rZinfer_runtime_shaper5r%Zget_edge_shaper&r2rr^r0r:setZget_graphs_input_namesr]boolr9r/Zreplace_output_of_all_nodesr4r3)rZ shape_inferr_r6Z input_shapeZ output_shapeZgraph_input_namesZgraph_output_namesrrrremove_useless_reshape_nodess4  z(FusionUtils.remove_useless_reshape_nodes)r)rr)N)T)__name__ __module__ __qualname__r rstrrrfrr.rr7 staticmethodr=r>rGrLrrRrrYr'r\r`rarcrgrrrrr s(  * r c@s$eZdZedeeedddZdS) NumpyHelperF)rH fill_zerosrcCs0|r&ddlm}t|j|j|jdSt|S)Nr)mapping)rWrJ)onnxrorrMZTENSOR_TYPE_TO_NP_TYPErNr to_array)rHrnrorrrrqs  zNumpyHelper.to_arrayN)F)rhrirjrlrrfrrqrrrrrmsrm)loggingrtypingrrPrrrprrrr r rLZ onnx_modelr rhrr rmrrrrs