U h%{ @s$ddlZddlZddlmZddlmZmZmZmZddl m Z ddl m Z ddl mZmZmZmZmZddlmZmZmZGd d d ZGd d d eZGd ddeZeeedddZejdddejejdddejdf edddZdddejddddfeedddZeeedddZ dS)N)Path)CalibrationDataReaderCalibrationMethod TensorsDatacreate_calibrator) ONNXQuantizer) QDQQuantizer) QuantFormatQuantizationMode QuantTypeload_model_with_shape_infermodel_has_pre_process_metadata)IntegerOpsRegistry QDQRegistryQLinearOpsRegistryc@s*eZdZejejddddddfddZdS) QuantConfigNFc CsL|pg}|pg}|pg}||_||_||_||_||_||_||_||_dS)a This is the Base class for both Static and Dynamic Quantize Configuration Args: activation_type: quantization data type of activation. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection weight_type: quantization data type of weight. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection op_types_to_quantize: specify the types of operators to quantize, like ['Conv'] to quantize Conv only. It quantizes all supported operators by default. nodes_to_quantize: List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. example: [ 'Conv__224', 'Conv__252' ] nodes_to_exclude: List of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None. per_channel: quantize weights per channel reduce_range: quantize weights with 7-bits. It may improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode use_external_data_format: option used for large size (>2GB) model. Set to False by default. N)op_types_to_quantize per_channel reduce_range weight_typeactivation_typenodes_to_quantizenodes_to_excludeuse_external_data_format) selfrrrrrrrrrE/tmp/pip-unpacked-wheel-socb9apf/onnxruntime/quantization/quantize.py__init__s)zQuantConfig.__init__)__name__ __module__ __qualname__r QUInt8QInt8rrrrrrsrc sBeZdZejejejejdddddddf e dfdd Z Z S)StaticQuantConfigNFcalibration_data_readerc s<tj|||||| | | d||_||_||_| p4i|_dS)a This is the derived class for static Quantize Configuration Args: calibration_data_reader: a calibration data reader. It enumerates calibration data and generates inputs for the original model. calibrate_method: Current calibration methods supported are MinMax, Entropy and Percentile. quant_format: QuantFormat{QOperator, QDQ}. QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dyanmic mode currently is supported. Will support more in future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is False for static mode. If enabled, only MatMul with const B will be quantized. AddQDQPairToWeight = True/False : Default is False which quantizes floating-point weight and feeds it to solely inserted DeQuantizeLinear node. If True, it remains floating-point weight and inserts both QuantizeLinear/DeQuantizeLinear nodes to weight. OpTypesToExcludeOutputQuantization = list of op type : Default is []. If any op type is specified, it won't quantize the output of ops with this specific op types. DedicatedQDQPair = True/False : Default is False. When inserting QDQ pair, multiple nodes can share a single QDQ pair as their inputs. If True, it will create identical and dedicated QDQ pair for each node. QDQOpTypePerChannelSupportToAxis = dictionary : Default is {}. Set channel axis for specific op type, for example: {'MatMul': 1}, and it's effective only when per channel quantization is supported and per_channel is True. If specific op type supports per channel quantization but not explicitly specified with channel axis, default channel axis will be used. CalibTensorRangeSymmetric = True/False : Default is False. If enabled, the final range of tensor during calibration will be explicitly set to symmetric to central point "0". CalibMovingAverage = True/False : Default is False. If enabled, the moving average of the minimum and maximum values will be computed when the calibration method selected is MinMax. CalibMovingAverageConstant = float : Default is 0.01. Constant smoothing factor to use when computing the moving average of the minimum and maximum values. Effective only when the calibration method selected is MinMax and when CalibMovingAverage is set to True. QuantizeBias = True/False : Default is True which quantizes floating-point biases and it solely inserts a DeQuantizeLinear node. If False, it remains floating-point bias and does not insert any quantization nodes associated with biases. This extra option is only effective when quant_format is QuantFormat.QDQ. SmoothQuant = True/False : Default is False. If enabled, SmoothQuant algorithm will be applied before quantization to do fake input channel quantization. SmoothQuantAlpha = float : Default is 0.5. It only works if SmoothQuant is True. It controls the difficulty of weight and activation quantization. A larger alpha value could be used on models with more significant activation outliers to migrate more quantization difficulty to weights. SmoothQuantFolding = True/False : Default is True. It only works if SmoothQuant is True. If enabled, inserted Mul ops during SmoothQuant will be folded into the previous op if the previous op is foldable. execution_provider : A enum indicates the Execution Provider such as: CPU, TRT, NNAPI, SNE, etc. Raises: ValueError: Raise ValueError if execution provider is unknown )rrrrrrrrN)superrr&calibrate_method quant_format extra_options) rr&r(r)rrrrrrrrr* __class__rrrOsT zStaticQuantConfig.__init__) rr r!rMinMaxr QDQr r#rr __classcell__rrr+rr$Nsr$cs0eZdZejdddddddffdd ZZS)DynamicQuantConfigNFc s(tj|||||||d|p i|_dS)a This is a class for dynamic Quant Configuration Args: extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dynamic mode currently is supported. Will support more in the future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is True for dynamic mode. If enabled, only MatMul with const B will be quantized. execution_provider : A enum indicates the Execution Provider such as: CPU, TRT, NNAPI, SNE, etc. Raises: ValueError: Raise ValueError if execution provider is unknown )rrrrrrrN)r'rr*) rrrrrrrrr*r+rrrs! zDynamicQuantConfig.__init__)rr r!r r#rr/rrr+rr0sr0r)rrcCs|tjkr|tjkrtd|tjkr@|tjkr@td|d|tjkrd|tjkrdtd|d|tjkr|tjkr|tjkrtddS)NzrONNXRuntime quantization doesn't support data format:activation_type=QuantType.QInt8, weight_type=QuantType.QUInt8zFONNXRuntime quantization doesn't support data format: activation_type=z@ !=QuantType.QFLOAT8E4M3FN, weight_type=QuantType.QFLOAT8E4M3FN.zkONNXRuntime quantization doesn't support data format: activation_type=QuantType.QFLOAT8E4M3FN, weight_type=z!=QuantType.QFLOAT8E4M3FNzvPlease use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.) r r#r" ValueError QFLOAT8E4M3FNr r.loggingwarningr1rrrcheck_static_quant_argumentss   r6Fr%c! s|tjks|tjkr&| tjkr&tdp,i| p4g} | ptk r$}zt|d td|W5d }~XYnXdd ldd l}ddlm}fdd}dd|jjD|}||||}~|ddddj}| fdd|jjDt!j"dd}t|j#$d%}|j&||ddt t|}t!j"ddj}t't||t|$d%| | |d}|(|)}t*|t+s\t,dt-|d t-|d ~W5QRXt.||||t/j0krt1||||d|||| | | } nt2||||d|||| | | } | 3| j4|| |std!d d r|5d S)"a Given an onnx model and calibration data reader, create a quantized onnx model and save it into a file It is recommended to use QuantFormat.QDQ format from 1.11 with activation_type = QuantType.QInt8 and weight_type = QuantType.QInt8. If model is targeted to GPU/TRT, symmetric activation and weight are required. If model is targeted to CPU, asymmetric activation and symmetric weight are recommended for balance of performance and accuracy. Args: model_input: file path of model to quantize model_output: file path of quantized model calibration_data_reader: a calibration data reader. It enumerates calibration data and generates inputs for the original model. quant_format: QuantFormat{QOperator, QDQ}. QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. activation_type: quantization data type of activation. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection calibrate_method: Current calibration methods supported are MinMax and Entropy. Please use CalibrationMethod.MinMax or CalibrationMethod.Entropy as options. op_types_to_quantize: specify the types of operators to quantize, like ['Conv'] to quantize Conv only. It quantizes all supported operators by default. per_channel: quantize weights per channel reduce_range: quantize weights with 7-bits. It may improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode weight_type: quantization data type of weight. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection nodes_to_quantize: List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. example: [ 'Conv__224', 'Conv__252' ] nodes_to_exclude: List of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None. use_external_data_format: option used for large size (>2GB) model. Set to False by default. extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dyanmic mode currently is supported. Will support more in the future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also, the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is False for static mode. If enabled, only MatMul with const B will be quantized. AddQDQPairToWeight = True/False : Default is False which quantizes floating-point weight and feeds it to solely inserted DeQuantizeLinear node. If True, it remains floating-point weight and inserts both QuantizeLinear/DeQuantizeLinear nodes to weight. OpTypesToExcludeOutputQuantization = list of op type : Default is []. If any op type is specified, it won't quantize the output of ops with this specific op types. DedicatedQDQPair = True/False : Default is False. When inserting QDQ pair, multiple nodes can share a single QDQ pair as their inputs. If True, it will create identical and dedicated QDQ pair for each node. QDQOpTypePerChannelSupportToAxis = dictionary : Default is {}. Set channel axis for specific op type, for example: {'MatMul': 1}, and it's effective only when per channel quantization is supported and per_channel is True. If specific op type supports per channel quantization but not explicitly specified with channel axis, default channel axis will be used. CalibTensorRangeSymmetric = True/False : Default is False. If enabled, the final range of tensor during calibration will be explicitly set to symmetric to central point "0". CalibMovingAverage = True/False : Default is False. If enabled, the moving average of the minimum and maximum values will be computed when the calibration method selected is MinMax. CalibMovingAverageConstant = float : Default is 0.01. Constant smoothing factor to use when computing the moving average of the minimum and maximum values. Effective only when the calibration method selected is MinMax and when CalibMovingAverage is set to True. SmoothQuant = True/False : Default is False. If enabled, SmoothQuant algorithm will be applied before quantization to do fake input channel quantization. SmoothQuantAlpha = float : Default is 0.5. It only works if SmoothQuant is True. It controls the difficulty of weight and activation quantization. A larger alpha value could be used on models with more significant activation outliers to migrate more quantization difficulty to weights. SmoothQuantFolding = True/False : Default is True. It only works if SmoothQuant is True. If enabled, inserted Mul ops during SmoothQuant will be folded into the previous op if the previous op is foldable. zIOnly Distribution calibration method is supported for float quantization.rPlease consider to run pre-processing before quantization. Refer to example: https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md )ZCalibTensorRangeSymmetricZ symmetric)ZCalibMovingAverageZmoving_average)ZCalibMovingAverageConstantZaveraging_constantcs$i|]\}}|kr||qSr)get).0namekey)r*rr sz#quantize_static..Z SmoothQuantFNz/neural_compressor.adaptor.ox_utils.smooth_quant.zLneural-compressor is not correctly installed. Please check your environment.)ORTSmoothQuantc3s"}|D]}|dfVqdS)N)deepcopy)Z data_readerdata)r&copyrrinc_dataloaders z'quantize_static..inc_dataloadercSsg|] }|jqSrr:r9irrr sz#quantize_static..ZSmoothQuantAlphag?ZSmoothQuantFoldingTcsg|]}|jkr|jqSrrCrD) orig_nodesrrrFs z ort.quant.)prefixz sq_model.onnx)Zsave_as_external_datazaugmented_model.onnx)Zaugmented_model_pathr(rr*zUnexpected type z" for tensors_range and calibrator=zPlease consider pre-processing before quantization. See https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md )6r r3r Distributionr2r Z QLinearOpslenlistrkeysrsetr rrr4r5r8 importlib import_module Exceptionerror RuntimeErrorrAonnxZ/neural_compressor.adaptor.ox_utils.smooth_quantr>graphnodeZ transformmodelextendtempfileTemporaryDirectoryr:joinpathas_posixZ save_modelrZ collect_dataZ compute_data isinstancer TypeErrortyper6r Z QOperatorrr quantize_modelsave_model_to_filecleanup)! model_input model_outputr&r)rrrrrrrrr(r*modeZ q_linear_opsZqdq_opsrV pre_processedZcalib_extra_options_keysZcalib_extra_optionsrNerSr>rBZ dataloadersqZsq_pathZ quant_tmp_dirZ calibratorZ tensors_range quantizerr)r&rAr*rGrquantize_staticsn               ri)rbrcc  Cs| pi} |pg}|pg}|pg}tj} |r6t|dkrBtt}tt|} t| } | sdt dd| krtd| d<t | ||| d|t j d|||| } | | j||dS)a Given an onnx model, create a quantized onnx model and save it into a file Args: model_input: file path of model to quantize model_output: file path of quantized model op_types_to_quantize: specify the types of operators to quantize, like ['Conv'] to quantize Conv only. It quantizes all supported operators by default. per_channel: quantize weights per channel reduce_range: quantize weights with 7-bits. It may improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode weight_type: quantization data type of weight. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection nodes_to_quantize: List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. example: [ 'Conv__224', 'Conv__252' ] nodes_to_exclude: List of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None. use_external_data_format: option used for large size (>2GB) model. Set to False by default. extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dynamic mode currently is supported. Will support more in the future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is True for dynamic mode. If enabled, only MatMul with const B will be quantized. rr7ZMatMulConstBOnlyTFN)r Z IntegerOpsrJrKrrLr rrr4r5rr r"r_rVr`)rbrcrrrrrrrr*rdrVrerhrrrquantize_dynamics>6  rjrbrcZ quant_configcCst|trHt|||j|j|j|j|j|j|j |j |j |j |j |jdn@t|trt|||j|j|j |j |j |j |j |jd ntddS)zQuantize a model with QuantConfig. Args: model_input (Path): Path to the model to quantize. model_output (Path): Path to save the quantized model. quant_config (QuantConfig): Quantization Configuration. ) r(r)rrrrrrrrr*)rrrrrrrr*z\Invalid quantization config type, it must be either StaticQuantConfig or DynamicQuantConfig.N)r\r$rir&r(r)rrrrrrrrr*r0rjr]rkrrrquantizeIs>   rl)!r4rXpathlibrZ calibraterrrrZonnx_quantizerrZ qdq_quantizerr Z quant_utilsr r r r rregistryrrrrr$r0r6r.r#r-rirjrlrrrrsT   7e. s a