U hY@sddlZddlZddlZddlmZddlmZddlZddlZddlm Z m Z m Z ddlm Z ddlmZmZmZmZddlmZddlmZmZmZzdd lmZWnek rdZYnXd Zd Zd Zd ZdZ dZ!dZ"dZ#dZ$iZ%dde&e DZ'GdddeZ(GdddeZ)GdddeZ*GdddeZ+e j j,e-de j j.e-de j j/eiZ0djdd Z1dkd"d#Z2d$d%Z3dld&d'Z4dmd(d)Z5dnd*d+Z6Gd,d-d-Z7Gd.d/d/Z8Gd0d1d1Z9d2d3Z:d4d5Z;d6d7Zed:d;d<Z?d=d>Z@d?d@ZAdodBdCZBedDdEdFZCeedGdHdIZDe dJdKdLZEe eFdMdNdOZGe dJdPdQZHe eFdMdRdSZIee dTdUdVZJe e dMdWdXZKe ejLdYdZd[ZMe>e>d\d]d^ZNe>e>d\d_d`ZOe>dadbdcZPe>dadddeZQe>dadfdgZRe>dadhdiZSdS)pN)Enum)Path) ModelProto TensorProtoexternal_data_helper)onnx_pb) make_graph make_model make_nodemake_tensor_value_info)ReferenceEvaluator)GraphOptimizationLevelInferenceSessionSessionOptions) float8e4m3fnz onnx.quantizez0.1.0zai.onnxz com.microsoftQuantizeLinearZ_QuantizeLinear_InputZDequantizeLinearZ_DequantizeLinear_OutputZ _quantizedcCs(i|] }ttt|trtt||qS) isinstancegetattrrint).0krrH/tmp/pip-unpacked-wheel-socb9apf/onnxruntime/quantization/quant_utils.py "src@s(eZdZdZdZddZeddZdS)QuantizationModercCs|jSNnameselfrrr__str__-szQuantizationMode.__str__cCs*z t|WStk r$tYnXdSr)rKeyError ValueError)moderrr from_string0s zQuantizationMode.from_stringN)__name__ __module__ __qualname__Z IntegerOpsZ QLinearOpsr! staticmethodr%rrrrr)s rc@s(eZdZdZdZddZeddZdS)QuantizedValueTyperrcCs|jSrrrrrrr!<szQuantizedValueType.__str__cCs*z t|WStk r$tYnXdSr)r*r"r#)vrrrr%?s zQuantizedValueType.from_stringN)r&r'r(ZInputZ Initializerr!r)r%rrrrr*8s r*c@s8eZdZdZdZdZddZeddZe dd Z d S) QuantTyperrcCs|jSrrrrrrr!LszQuantType.__str__cCs*z t|WStk r$tYnXdSr)r,r"r#)trrrr%Os zQuantType.from_stringcCsD|tjkrtjS|tjkr tjS|tjkr0tjStd|ddS)NzUnexpected value qtype=.) r,QInt8rINT8QUInt8UINT8 QFLOAT8E4M3FN FLOAT8E4M3FNr#rrrr tensor_typeVs   zQuantType.tensor_typeN) r&r'r(r0r2r4r!r)r%propertyr6rrrrr,Gs r,c@s(eZdZdZdZddZeddZdS) QuantFormatrrcCs|jSrrrrrrr!eszQuantFormat.__str__cCs*z t|WStk r$tYnXdSr)r8r"r#)formatrrrr%hs zQuantFormat.from_stringN)r&r'r(Z QOperatorZQDQr!r)r%rrrrr8as r8Zint8uint8c Csx|tkstd|d|tjjtjjtjjtjjfkr|dkrPtd|dt t t dgdgt j d|gdgdt d d d dgd ggd td tjdtd tjdgtd |dg}t|}|d|tj|tjddSt|}t|tjkrdnd|dkr dn|} t|tjkr$dnd|dkr4dn|} t|tj||} tj| | | | d| |SdS)NUnexpected data type . requested. Only INT8 and UINT8 are supported.rz2zero_point is expected to be null for float 8 not r/Constant zero_point)valuerXscaleYZqu)r@rA)out)ONNX_TYPE_TO_NP_TYPEAssertionError onnx_protorr5ZFLOAT8E4M3FNUZZ FLOAT8E5M2ZFLOAT8E5M2FNUZNotImplementedErrorr rr onnxhelperZ make_tensorr FLOATr runastypenumpyfloat32maxr:minasarrayroundZclip) qTypeZarrrAr>lowhighZ onnx_modelrefdtypeZcliplowZcliphighZarr_fp32rrrquantize_nparraywsD   &&(r[FcCs|dks|dkr$td|d|t|d}t|d}|rZtt|t|}| }| }||t||}|ttjjkrd}d}nt |||}||gS)aCalculate the scale s and zero point z for the quantization relation r = s(q-z), where r are the original values and q are the corresponding quantized values. r and z are calculated such that every value within [rmin,rmax] has an approximate representation within [qmin,qmax]. In addition, qmin <= z <= qmax is enforced. If the symmetric flag is set to True, the interval [rmin,rmax] is symmetrized to [-absmax, +absmax], where absmax = max(abs(rmin), abs(rmax)). :parameter rmin: minimum value of r :parameter rmax: maximum value of r :parameter qmin: minimum value representable by the target quantization data type :parameter qmax: maximum value representable by the target quantization data type :return: zero and scale [z, s] rzBqmin and qmax must meet requirement: qmin <= 0 <= qmax while qmin:z, qmmax:?) r#rSrRabsfloatrPZfinforQZtinyrU)rminrmaxqminqmax symmetricZabsmaxrAr>rrrcompute_scale_zps  rdcs|tkrj|tjkrRddlmfddtddD}tjdd|Dtjd}nt d|d |t|<t t|}d}||}||gS) arCalculate the scale s for a float8 type (E4M3FN). The function assumes the coefficient distribution and the float 8 distribution are similar to two gaussian laws. :return: zero and scale [z, s] More details in notebook `quantization_fp8.ipynb `_. rfloat8e4m3_to_float32csg|] }|qSrr)rirerr sz+compute_scale_zp_float8..cSs$g|]}t|st|s|qSr)rPisnanisinf)rfrrrrhs )rZzQuantization to element_type=z not implemented.) FLOAT8_DISTRIBUTIONSrr5Zonnx.numpy_helperrfrangerParrayrQr#std)Z element_typerpZ all_valuesvaluesZstd_f8ZzerorArrercompute_scale_zp_float8s    rrc CsHd}d}d}d}t|r(t|}t|}|tjkr|r>tdt|}t||\}}t |t |||} t | tj d@dkrt |} td| d| d| d| d ||||| fS|tjtjfkr4t|rt|||d \} } t||| | |\}}t |t |||} ||||| fStd |d d S) a :param data: data to quantize :param qType: data type to quantize to. Supported types UINT8 and INT8 :param symmetric: whether symmetric quantization is used or not. This is applied to INT8. :return: minimum, maximum, zero point, scale, and quantized weights To pack weights, we compute a linear transformation - when data `type == uint8` mode, from `[rmin, rmax]` -> :math:`[0, 2^{b-1}]` and - when data `type == int8`, from `[-m , m]` -> :math:`[-(2^{b-1}-1), 2^{b-1}-1]` where `m = max(abs(rmin), abs(rmax))` and add necessary intermediate nodes to trasnform quantized weight to full weight using the equation :math:`r = S(q-z)`, where - *r*: real original value - *q*: quantized value - *S*: scale - *z*: zero point rr\z1Unsupported option reduce_range=True for float 8.rEz+One of the quantized value is NaN data in [z, z], quantized_data in [z].rczUnexpected value for qType=r/N)lenrSrRrr5 RuntimeErrorrPrprrr[rTanyrOr:Zravelr1r3get_qmin_qmax_for_qTyperdr#) datarVrc reduce_ranger_r`r>rArpquantized_dataZnp_datararbrrr quantize_datas4   , r{cCs|tjjkr|rdnd\}}nZ|tjjkrR|r@|r6dnd\}}qx|rHdnd\}}n&|tjjkrhtdntd|d||fS) z Return qmin and qmax, the minimum and maximum value representable by the given qType :parameter qType: onnx.onnx_pb.TensorProto.UINT8 or onnx.onnx_pb.TensorProto.UINT8 :return: qmin, qmax )rrE)rrD)i@)rCrE)irEz;This function is not implemented for float 8 as not needed.r;r<)rIrr3r1r5rJr#rVryrcrarbrrrrws    rwcCst|||d\}}||S)z Helper function to get the quantization range for a type. parameter qType: quantization type. return: quantization range. rs)rwr}rrrget_qrange_for_qType,sr~c@s eZdZdZggdfddZdS)QuantizedInitializerzJ Represents a linearly quantized weight input from ONNX operators Nc Cs:||_||_||_||_||_||_||_||_| |_dSr) r initializerrminsrmaxs zero_pointsscalesrxrzaxis) r rrrrrrrxrzrrrr__init__;s zQuantizedInitializer.__init__r&r'r(__doc__rrrrrr6s  rc@seZdZdZdddZdS)QuantizedValuezI Represents a linearly quantized value (input\output\intializer) Nc Cs4||_||_||_||_||_||_||_||_dSr) original_nameZq_name scale_nameZzp_nameZ value_typer node_type node_qtype) r rZnew_quantized_namerZzero_point_nameZquantized_value_typerrrrrrrZs zQuantizedValue.__init__)NNNrrrrrrUs  rc@seZdZdZddZdS)BiasToQuantizez+ Represents a bias to be quantized cCs||_||_||_dSr) bias_name input_name weight_name)r rrrrrrrtszBiasToQuantize.__init__NrrrrrrosrcCs|jdkrtd|jd|jdkr.|j}n|jdkr@|j}n|jdkrR|j}n|jdkrd|j}n|jdkrv|j}nt|jd kr|j}nb|jd kr|j }nP|jd kr|j }n>|jd kr|j }n,|jd kr|j }ntd|jd|jd|j|iS)z Convert attribute to kwarg format for use with onnx.helper.make_node. :parameter attribute: attribute in AttributeProto format. :return: attribute in {key: value} format. rz attribute z does not have type specified.rr- z has unsupported type r/) typer#rrlrgsr.gZfloatsZintsstringsZtensorsZgraphs) attributer?rrrattribute_to_kwargzs0           rcs*fdd|D}t|dkr&|dSdS)z Helper function to find item by name in a list. parameter item_name: name of the item. parameter item_list: list of items. return: item if found. None otherwise. csg|]}|jkr|qSrr)ritem item_namerrrhs z find_by_name..rN)rt)rZ item_listitemsrrr find_by_namesrcCs,d}tdt|D]}|||kr|}q|S)zC Helper function to return index of an item in a node list r)rnrt)Z elem_nameZ elem_listZelem_idxrgrrrget_elem_indexs  rcCstjd||g|S)z Helper function to create a Mul node. parameter inputs: list of input names. parameter output: output name. parameter name: name of the node. return: Mul node in NodeProto format. ZMul)rKrLr )inputsoutputrrrr get_mul_nodesr)filename identifierreturncCs|j|j||jS)zp Helper function to generate a identifiable filepath by concatenating the given identifier as a suffix. )parentjoinpathstemsuffix)rrrrrgenerate_identified_filenamesrcCsddl}ddlm}ddl}|j|jdtdt|tdt||j||dd|d| d| d | dS) Nr) thresholdz Histogram:zHistogram Edges:T)fillz Tensor valueZCountszTensor value V.S. Counts) sysZmatplotlib.pyplotZpyplotrPZset_printoptionsmaxsizeprintZstairsZxlabelZylabeltitleshow)histZ hist_edgesrZpltrPrrr apply_plots    rc CsVddl}ddl}ddlmmm}ddlmmm}t d|t dd}| | |W5QRX| d}g}t|D]t}||} ttt| dt| d} ||} || } ||||| ||| ||} || q||t||D]} || q|}|||||||}| ||!}t dd }| |W5QRXt"j#$d dd kr|j%|d}|&}t'|D],}|(|} t | )t | *qt d d\}t|D]H}||} |d ttt| dt| d}| || dqW5QRXdS)z> Helper function to write calibration table to files. rNzcalibration cache: zcalibration.jsonwirzcalibration.flatbufferswbZQUANTIZATION_DEBUG)r1zcalibration.cache  )+json flatbuffersZ5onnxruntime.quantization.CalTableFlatBuffers.KeyValueZ quantizationZCalTableFlatBuffersKeyValueZ5onnxruntime.quantization.CalTableFlatBuffers.TrtTableTrtTablelogginginfoopenwritedumpsZBuildersortedkeysstrrRr]Z CreateStringZ KeyValueStartZKeyValueAddKeyZKeyValueAddValueZ KeyValueEndappendZTrtTableStartDictVectorrtZPrependUOffsetTRelativeZ EndVectorZ TrtTableStartZTrtTableAddDictZ TrtTableEndZFinishZOutputosenvirongetZGetRootAsTrtTableZ DictLengthrnDictZKeyValue)Zcalibration_cacherrrrfileZbuilderZkey_value_listkeyrqr?Zflat_keyZ flat_value key_valueZ main_dictZ cal_tablebufZdict_lenrgrrrrwrite_calibration_tablesV                 & r-C6?cCs|dktj}|dktj}|}|j|}|s:dS|t|t|}|dkshtd|||f|tj}|||| |7}|dkdkst|S)a~Given a discrete distribution (may have not been normalized to 1), smooth it by replacing zeros with eps multiplied by a scaling factor and taking the corresponding amount off the non-zero values. Ref: http://web.engr.illinois.edu/~hanj/cs412/bk3/KL-divergence.pdf https://github.com//apache/incubator-mxnet/blob/master/python/mxnet/contrib/quantization.py rrr\z"n_zeros=%d, n_nonzeros=%d, eps1=%f)rOrPrQsumsizer^rH)pZepsZis_zerosZ is_nonzerosZn_zerosZ n_nonzerosZeps1rrrrsmooth_distributions    r) model_pathcCs4tj|dd}|jjD]}t|rdSqdS)NF)Zload_external_dataT)rKloadas_posixgraphrrZuses_external_data)rmodelZ intializerrrrmodel_has_external_data9s   r)ropt_model_pathcCsFt}||_tj|_i}dg|d<t||fddgi|}dS)z Generate model that applies graph optimization (constant folding, etc.) parameter model_path: path to the original onnx model parameter opt_model_path: path to the optimized onnx model :return: optimized onnx model ZConstantSharingZdisabled_optimizersZ providersZCPUExecutionProviderN)rrZoptimized_model_filepathr ZORT_ENABLE_BASICZgraph_optimization_levelr)rrZ sess_optionkwargs_rrroptimize_modelAs   r)rcCs>ddi}|jr,|jD]}||j|jiqtj||dS)z>Tag the model that it went through quantization pre-processingonnx.quant.pre_processonnxruntime.quantNmetadata_propsupdaterr?rKrLZset_model_props)rrproprrradd_pre_process_metadataQs  r)rrcCs0|jr,|jD]}|jdkr |jdkr dSq dS)zCCheck the model whether it went through quantization pre-processingrrTFrrr?)rrrrrmodel_has_pre_process_metadataZs  rcCs>ddi}|jr,|jD]}||j|jiqtj||dS)N onnx.inferrr)rrrrrradd_infer_metadatacs  rcCs0|jr,|jD]}|jdkr |jdkr dSq dS)NrrTFr)rrrrrmodel_has_infer_metadataks  r)rrcCsBt|d}tjt|t|t|}t|||S)Nz -inferred) rrKZshape_inferenceZinfer_shapes_pathrrrrunlink)rZinferred_model_pathrrrrload_model_with_shape_inferss  rc CsNtjdd8}t|d}tj||ddt|W5QRSQRXdS)Nz ort.quant.)prefixz model.onnxT)Zsave_as_external_data)tempfileTemporaryDirectoryrrrKZ save_modelrr)rZ quant_tmp_dirrrrr&save_and_reload_model_with_shape_infer|sr)rrcCs:|jtjjkrtj|Std|jdt |jdS)Nz&Only float type is supported. Weights z is ) Z data_typerIrrMrKZ numpy_helperZto_arrayr#r type_to_name)rrrrtensor_proto_to_arrays  r) tensor_namercCs|dS)NZ_QuantizeLinearrrrrradd_quant_suffixsrcCs|tSr)QUANT_INPUT_SUFFIXrrrradd_quant_input_suffixsr)rcCs|dS)NZ_QuantizeLinear_Outputrrrrradd_quant_output_suffixsrcCs|dS)NZ_DequantizeLinearrrrrradd_dequant_suffixsrcCs|dS)NZ_DequantizeLinear_Inputrrrrradd_dequant_input_suffixsrcCs|tSr)DEQUANT_OUTPUT_SUFFIXrrrradd_dequant_output_suffixsr)NN)F)F)FF)FF)r)TrrrenumrpathlibrrPrKrrrrrIZ onnx.helperrr r r Zonnx.referencer Z onnxruntimer rrZ#onnx.reference.custom_element_typesr ImportErrorZ __producer__ __version__Z onnx_domainZ ms_domainZ QUANT_OP_NAMErZDEQUANT_OP_NAMErZTENSOR_NAME_QUANT_SUFFIXrmdirrrr*r,r8r1rZr3r5rGr[rdrrr{rwr~rrrrrrrrrrrrrrrboolrrrrrZndarrayrrrrrrrrrrrs      % * 6   %   C