h@#SSKrSSKrSSKrSSKrSSKJr SSKrSSK J r J r J r J r SSKJr SSKJr SSKJr \R("\5r"SS5rS r\S :Xa\"5r\R4(a\R7\R85 \R:r\R>r \RBRE\ 5(a!\RGS \ S 35 \$"S \ S 35e\RJ"\5r&\"\&\RN\RP\RRS 9r*\*RW5 \*RLRY\ S5 gg)N) GraphProto ModelProto NodeProto TensorProto)quantize_matmul_bnb4) ONNXModel)attribute_to_kwargc\rSrSrSrSrSrSS\S\S\4S jjr \ S \ \ S \ \\ 44S j5rS \R"S \R&4SjrS\S\ \ S \4SjrS\ \ 4SjrSrSrg)MatMulBnb4QuantizerzMPerform 4b quantization of constant MatMul weights using FP4 or NF4 data typerrNmodel quant_type block_sizecU=(d /nU[R[R4;de[U5UlX lX0l[U5UlgN) r FP4NF4r rrrsetnodes_to_exclude)selfrrrrs h/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/quantization/matmul_bnb4_quantizer.py__init__MatMulBnb4Quantizer.__init__%sR+1r1557J7N7NOOOOu% $$ #$4 5 graph_pathreturnc[[U5S- SS5H1nXnURHnURU:XdMXC4s s $ M3 g)Nr)NN)rangelen initializername)r#rgidgraphtensors r__get_initializer%MatMulBnb4Quantizer.__get_initializer-sNZ1,b"5COE++;;$&!=(,6 rfpweightc [UR5S:wa [S5eUR5R 5nURup4X4-nUR nXV-S- U-nUS-S-n[ R"USS9n [ R"XqRS9n [XXURXC5 X4$)z4b quantize fp32/fp16 weightz9Current bnb4 block quantization only supports 2D tensors!ruint8)dtype) r!shape ValueError transposecopyrnpzerosr-rr) rr) fpweight_trowscolsnumelr num_blocksquantized_numelpackedabsmaxs rbnb4_block_quant$MatMulBnb4Quantizer.bnb4_block_quant6s x~~ ! #XY Y'')..0 ^^  __ (1,;  19*/9*NN;VT__VZarnode graph_stackcXURS:waU$[RSURS35 URUR;a%[RSURS35 U$UR Sn[ RX25upEUc[RS5 U$[RRU5n[UR5S:wa[RS 5 U$URU5upx[RRU5n URS -U lUR H0n U RU:XdMUR RU 5 O [RRU5n URS -U lUR R#X/5 0n URupXS 'XS 'UR$U S'UR&U S'[R(R*"SUR SU RU R/UR,S/UR(aURS -OSSS.U D6n[RSURS35 U$)zdIf the node is MatMul with fp32 const weight, quantize the weight with int4, and return the new nodeMatMulzstart to quantize z ...zexclude to quantize z$ as specified by nodes_to_exclude...rz2MatMul doesn't have const weight. Skip to quantizer+z)MatMul weight is not 2D. Skip to quantize_Bnb4_absmaxKNrrr com.microsoft)inputsoutputsr#domainzcomplete quantization of ) MatMulBnb4)op_typeloggerdebugr#rinputr %_MatMulBnb4Quantizer__get_initializeronnx numpy_helperto_arrayr!r.r< from_arrayremover"extendrrhelper make_nodeoutput)rr>r?inputBBBs_graphB_arrayr:r;B_quantrO absmax_tensorkwargsr5r6matmul_bnb4_nodes r_bnb4_matmul_node_weight,MatMulBnb4Quantizer._bnb4_matmul_node_weightLs; <<8 #K )$))D9: 99-- - LL/ {:^_ `KA);;FP  9 LLM NK##,,Q/ w}}  " LLD EK..w7##..v6vv' ^^EzzV#%%e,$ ))44V< VVi/ ##W$<=]] s s #|#|;;00  JJqM7<<1C1CD[[^$(, W$r"      0 4@Arc/nUSnURGHnURVs/sHWnUR[RR :Xd*UR[RR :XdMUUPMY nnU(Ga^0nURHnUR[RR :Xa9URUR5 URURU50nOUR[RR :XaV/n URH5n URU 5 U RURU5/5 M7 URU 0nO [U5nURU5 GM [RR "UR"UR$UR&4SUR0UD6nURUR)XA55 GM UR+S5 URRU5 UR-5 U$s snf)Nrr#r>)r> attributetyperQAttributeProtoGRAPHGRAPHSappendgr#_process_subgraphgraphsrVr updaterWrXrLrOrYrb ClearFieldpop) rr? new_nodesr%r>attr graph_attrsr`kvvaluesubgraphs rrl%MatMulBnb4Quantizer._process_subgraphs BJJD!NN*D99 3 3 9 99TYY$J]J]JdJd=d*   NNDyyD$7$7$=$==#**4662"ii)?)? )LMd&9&9&@&@@ "(, H'..x8!LL$*@*@*M)NO)4#ii//5MM"%+{{,,LL$**dkk@D MS   T::4M N7:   )$ ?s AI>IcpURR5/nURR5nSnUHnURS:XdMSnM U(d0UR [ R RSS5/5 URU5 URR5 g)NFrGTr) rr% opset_importrJrVrQrW make_opsetidrlclean_initializers)rr?ry has_ms_domainopsets rprocessMatMulBnb4Quantizer.processszz'')* zz..0  !E||. $ "   !9!9/1!M N O {+ %%'r)rrrrr)__name__ __module__ __qualname____firstlineno____doc__rrrintr staticmethodlistrtuplerrPnpt ArrayLiker2ndarrayr<rrbrlr~__static_attributes__rrr r sW C C6j6c6s6D,<{T^G^A_  2:: ,5 Y5 T*EU5 Zc5 n$T*-=$L (rr c [R"SS9nURSSSS9 URSSSS9 URS S S [R[R /S S 9 URSS SSS9 URSSS SS9 UR S S9 URSS[S /SS9 UR5$)NaBlockwise FP4/NF4 quantization for MatMul 2D weight matrices. A weight matrix is partitioned into blocks, where each block is a contiguous subset inside the flattened transposed weight matrix. Each block is quantized into a set of 4b integers with an absolute value scaling factor. ) descriptionz --input_modelTzPath to the input model file)requiredhelpz--output_modelzPath to the output model filez --quant_typeFrz&Quantization data type. 0: FP4, 1: NF4)rdefaultchoicesrz --block_size@zVBlock size for blockwise quantization. Note: bnb.nn.Linear4bit only uses block_size=64)rrrz-vz --verbose store_true)raction)verbosez--nodes_to_exclude+zBSpecify the nodes to be excluded from quantization with node names)nargsrfrrr) argparseArgumentParser add_argumentr rr set_defaultsstr parse_args)parsers rrrs  $ $F $=[\ (4>]^ $((*=*A*AB 5   e   kE,O &   Q     r__main__zfile z already exists)rT)-rloggingosnumpyr2 numpy.typingtypingrrQ onnx.onnx_pbrrrronnxruntime.capi._pybind_stater onnx_modelr quant_utilsr getLoggerrrMr rargsrsetLevelDEBUG input_modelinput_model_path output_modeloutput_model_pathpathexistserror Exceptionloadrrrrquantr~save_model_to_filerrrrs  GG?!+   8 $^(^(B$N z