hǍ 4SrSSKrSSKrSSKrSSKrSSKrSSKrSSKJr SSK J r SSK r SSK Jr SSKJr \R""S5rSS jrSS jrSSjrSSjrSSjrSr0S S S 0SS/S4SjrSrSrS Sjr0S S S SSSSSSSS/4 Sjrg)!zWeightOnly for onnxrt adaptor.N) numpy_helper)np_dtype_to_tensor_dtype) ONNXModel)simple_progress_barneural_compressorc X2-S-n [R"URSU 4SS9n URSSU<SU<3-n URSU /n /n 0nSnUS :Xa1US S 2S S S 24US S 2SS S 24S --nUS S 2S U 24U S S 2S S 24'O"US:XaUn O[R S US 35 [R "U SXI45n [R "USU45nUR[R:Xd UR[R:Xde[RRURSS-[UR5URUR5SS9nU RUR 5 U RU5 UGb}US:XaUR#S5nOUS :Xa[R$"URSS-S -SSS9n[R&"URSU-U-5R S5nUS S S 2nUSS S 2nUUS -S-UUR)5-UUS -'UUS -S-UUR)5S --UUS -'O[+S US 35e[R "UUSS45n[RRURSS-S URUR5SS9nU RUR 5 U RU5 USUS'USUS'X.S'X>S'US:aXS'[RRU S U RU R5SS9nU RU5 [RR,"U4U UR.UR (aUR S-[1U5-O S[1U5-SS.UD6nUU 4$)a Build MatMulNBits node. Args: node: original matmul node weight_shape: original weight shape num_bits (int): num_bits group_size (int): how many elements share one scale/zp k_blocks (int): block number q_weight (array): quantized weight scale (array): scale zero_point (array): zero point accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). Returns: matmul_weight_only_node: MatMulNBits node new_inits: initializers of the new node ruint8dtyper_QG MatMulNBitsNz8MatMulNBits does not have kernel support for num_bits = ._scaleTname data_typedimsvalsraw_zpKNbits block_sizeaccuracy_levelz com.microsoft)inputsoutputsrdomain)npzerosshapeinputloggererrorreshaper float32float16onnxhelper make_tensorrtobytesappendrastypefullarangeravel ValueError make_nodeoutputstr)node weight_shapenum_bits group_sizek_blocksq_weightscale zero_pointr$ blob_sizepacked q_weight_name input_names new_initskwargsop_typeq_weight_pairs scale_tensor packed_zpidxeven_idxodd_idx zp_tensorq_weight_tensormatmul_weight_only_nodes p/var/www/fran/franai/venv/lib/python3.13/site-packages/onnxruntime/quantization/neural_compressor/weight_only.pymake_matmul_weight_only_noderW,s8%*I XXx~~a()4G DFJJqMb Aj^$DDM::a=-0KI FG1}!!SqS&)HQ1W,=,BB%a)m4q!t Q OPXzYZ[\ ZZX 9 :F JJur8n -E ;;"** $ rzz(AA A;;** ZZ]X %*5;;7 [[ ]]_  +L|(() \" q="))'2I ]!1!1!!4q!8Q >7SI))J,,Q/8;hFGOOPRSC3Q3xH!$Q$iG(1(a-(@4(G:V^K_KeKeKg'gIh!m $'0A'>'E*U\J]JcJcJeijJj&kIgl #WX`Waabcd dJJy<?B*?@ KK++A&!)//PYPaPaPcim,  9>>*#q/F3Kq/F3K6N%<#1 kk--  \\ ^^   .O_%"kk33 15TYY H -s8}@T    #I --r asymc ([R"USU45nUS:XdUS:Xa SU-S- nSnO*US:Xa$US:wa SUS- -S- OSnUS:wa SUS- -*OSn[R"USSS 9U-n[R"USSS 9U-n US:Xa[R"[R "U5[R "U 55n [R "U R5n U S:n XS -R[R5WW- - X'US :Xa [R"U R5O'[R "U RS S 9SUS- --n GO%[R "U R5n [R"X- X:gR5R5Vs/sHn[U5WW- - PM sn5XU :g'US :Xa4[R"U R5U- U - R5Ol[R"S[R "W[R"U R5U- U - R555RS 5n [R""X R$S 9n[R&"X US9 [R("XUS9 [R"XS9 [R*"UWWUS9 XU 4$s snf)aQuantize tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quantized weight scale: scale zero_point: zero point rrZuintrrrsymTaxiskeepdimsg@intr r out)r(r.minmaxmaximumabsonesr*r6float64r)arrayflattentolistfloatroundminimum empty_liker divideaddclip)datar@rAschemer ratiomaxqminqrminrmax max_rangerDmaskrEirCs rV quant_tensorr~s ::dR, -D 5F?({Q 5*2a-qX\"Q&Q)1Qx!|$%B 66$Q . 6D 66$Q . 6D JJrvvd|RVVD\:  #1} ,44RZZ@D4KP %*e^BHHU[[ !SZ9[_`empqeq_r9s  # hh04 T\/J/R/R/T/[/[/] ^/]!U1X %/] ^ dl ~hhu{{#d*e 3 : : <Arzz$"((5;;2G$2NRW1W0^0^0`abiijqr  }}T5HIIdx(FF8X.HHX$GGHdDh/ J && _s Lc[R"USU45R[R5nSU-S- nSn[R"US-SSS9n[R "XR- 5n[R "U[R"U55n[R"USSS9n[R"USSS9n [R"USSS9n [R"Xp-SSS9n [R"U RURS9n X:gn X4- XX- - X'SU - n[R"[R"XU- -5XC5nX-U-U- n[R"UUS--SSS9nSnS nSn[U5GHn[R"U RURS9n[R "UUU--U-U- /5RUR5SnX:gn UXX- - UU '[R"[R"UX- -5XC5nUU-n[R"USSS9n[R"UU-SSS9n"UU-SSS9n[R""U U-US-5nU U-U U-- U- nUU -UU-- U- nUU-U-U- n[R"UUS--SSS9n [R "U 5n![R "U5n"[R$"U!U":5Sn#UU#S S 24UU#S S 24'U U#UU#'UU#UU#'UU#UU#'GM [R"U*U- R5SU5RS 5n$UR[R&5n[R("XRS9n%[R*"XU%S 9 [R "U%U$U%S 9 [R"U%U%S 9 [R"U%XCU%S 9 U%UU$4$) aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 32. Returns: output: quantized weight scale: scale zero_point: zero point rrrrTr^r 皙?Nr rb)r(r.r6r/sumsqrtrrrgrdrerhr*r rsrnrangerjsubtractwhererirprq)&rtr@rArwrxsum_x2av_xweightsryrzsum_wsum_xiscaler|rD quant_datadiffbest_madnsteprdeltarrminis_ iscale_newfactorquant_data_newmul_weights_quant_data_newsum_lsum_l2sum_xlD this_scalethis_minmadmad_1 best_mad_1idx_to_replacerErCs& rVquant_tensor_k_quant_cpurs ::dR, - 4 4RZZ @D h;?D D VVD!G!d 3F 776& 'DffT266$<(G 66$Q .D 66$Q .D FF7T 2E FF7>D 9E WWTZZtzz 2F ?FFtzzRSTU|!TZ$*%<= 4* *D!EtR%,~%="1DI2^C!VZ[2T9DQ KKq 1fnuu}49 UNUV^3q8N*X5<ffWtQw&Q> XXh' %*"45a8(6~q7H(I >1$%#&~#6 *> :n'7^14D5E/002At<CCGLJ LL $E}}T5HIIdx(FF8ZX.HHX8$GGHdh/ UJ &&rXcSSKnSSKnURR5(GaUR U5nUR SU45R UR5nSU-S- nSnURUS-SSS9nURXr- 5nURXRU55n URUSSS9n URUSSS9n URU SSS9n URX-SSS9n URU RUR S9nX:gnXV- XX- - X'SU- nUR#UR%XU - -5Xe5nUU-U -U- nURU US--SSS9nS nS nSn['U5GHnURU RUR S9nUR)UUU--U-U- /5R UR 5SnX:gnUXX- - UU'UR#UR%UX - -5Xe5nU U-nURUSSS9nURUU-SSS9nURUU-SSS9nUR+U U-US-5nU U-U U-- U- n UU -UU-- U- n!U U-U!-U- nURU US--SSS9n"UR)U"5n#UR)U5n$UR-U#U$:5Sn%UU%SS24UU%SS24'U"U%UU%'U U%UU%'U!U%U U%'GM UR#U *U- R%5SU5R S 5n&UR UR.5nUR1UUR S9n'UR3UUU'S 9 URU'U&U'S 9 UR%U'U'S 9 UR#U'XeU'S 9 U'R55UR55U&R554$[6R9S 5 [;XU5$![<a$ [6R?S5 [;XU5s$f=f)aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. Returns: output: quantized weight scale: scale zero_point: zero point rNrrrTr^r rrr rbzqTry to use k-quant quantization on CUDA. However, CUDA is not available.Fall back to k-quant quantization on CPU.zNow we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.) cupytorchcuda is_availableasarrayr.r6r/rrrrrgrdrerhr*r rsrnrrjrrrirprqgetr,warningr ImportErrorinfo)(rtr@rAcprrwrxrrrryrzrrrr|rDrrrrrrrrrrrrrrrrrrrrrrErCs( rVquant_tensor_k_quant_cudarsID :: " " $ $::d#D<<Z 0188DDh;?DDVVD!G!dV;F776./DffT66$<0G66$Q66D66$Q66DFF7TF:EFF7>DFAEWWTZZtzzW:F#E"FGNNtzzZ[\]|#)TZ$*-D#E 4 !#* 2L)Mt!Z-4~-E*9DQ :^ KRS^bc :T ATXYKKq9#fnuu}<A "UNUV^;q@!N2X=DffWtQw.QfF XXh/ !#%**~q?P0Q >1,-+.~+>((2>(Bn%'/'?^$1$4D5E/!8!8!:AtDKKGTJLL,E}}T}=H IIdExI 0 FF8ZXF 6 HHX8H , GGHdhG 7<<>599; 0@@ @ NN< ,DJG G D  D ( CC DsPP% P%%+QQcrURn[XX#XE5upxn [R"XU - -U5$)aQuant dequant tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quant-dequant weight )r*r~r(r.) rtr@rArur rv org_shapeweightrDzps rV qdq_tensorrys8 I$TZVF2 ::e{+Y 77rXcUS:XaU$URnX!-nXCS- nUS:a[R"USU4S4S5nU$)zPad tensor rowi so that it can be is divisible by group_size. Args: weight (array): weight group_size (int): how many elements share one scale/zp k_blocks (int): the number of block Returns: weight: paded weight rr)rrconstant)r*r(pad)rrArB org_w_shape padded_rowspad_lens rV pad_tensorrsUR ,,K'KN*G{!Wv 6 C MrXCPUExecutionProviderk_quantc [U5nURb)[RR UR5OSn /n /n [ UR 5V s/sHoRS;dMU PM sn 5n SnUR 5GHnURS;aUS- n[X5 URS;dM6URURS5cMWURUR05S:wdMyURURS5n[R"UU S9R5n[ UR 5S:waMUR"nURU;a3XRS nXRS nXRS nUR nUS :waUOUSnUSS- U-S-nUR%URS5n['UUU5nUS :H=(d US:HnU(aUS:Xa[)UR*X#5unnnO:[-UR*X#USURURSS55unnn[/UUUUUUR1S5UR1U5US:XdUS:XaUOSUS9 unnUR3U5 U R5U5 U R5U5 GO [7UR*X#USURURSS55n[8R:"UUSS 45n[8R<"U5nUSUS2SS24R1U5n[>R@RCURSSU<SU<3-[EU5UR URG5SS9nURIU5 URURS'US:XdGMpURKU5 GM URMU 5 UROU 5 URQ5 U$s sn f)a|Quant the model with round to nearst method. Args: model (ModelProto or ONNXModel): onnx model weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'RTN' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". ratios (dict, optional): percentile of clip. Defaults to {}. accuracy_level (int): accuracy level. Support 0 (unset),1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NMatMulrrfp32)base_dirrr"rArurrr rr\r rZ r>r?r@rArBrCrDrEr$rarrTr))r model_pathospathdirnamelennodesrLrget_initializerr+rrrto_arraycopyr*r get_initializer_share_numrrTr~rWr6add_initializersr5rr(r. transposer1r2r3rr4add_initializerremove_initializer add_nodes remove_nodestopological_sort)model weight_configr@rAruratiosr$ providers algorithmr new_nodesrr} total_numcurr_idr> weight_tensorrr rrBinit_share_numsatisfy_MatMulNBits_conditionrCrDr q_matmul_noderJrTs rV rtn_quantizersH e E494D4D4Prwwu//0VXHIL I 1j1HQ IJIG  <<: % qLG  3 LLJ &%%djjm4@!!$))R0F:!11$**Q-@M!**=8LQQSF6<< A%LLEyyM)(3F;*995lC &yy1(; ,,K'1R'7[^J#A*z9A=H"<%>tAw<3|#44 1#l2C1DIcRVWXRYl^\ 4 d1gt $ $ MM$QUVWQXQ^Q^Q`aQ`~tx'9 :Q`ab c Q , , MM$SUVPWyaf=gh=g  =ghi j MM$s<^_Y`inGopGo)$x} 5Gopq r) :%: bhpsK >K$K"K{Gz?FTc ^^^ ^#^$^%^&SU-S- m$Sm#Sm%Sm&U#U$U%UU&U U4Sjn URn U "U5up[R"U5S:HnSXU4'SXSS24'U(aG[R"[R"U55SSS 2nXSS24nXSS24SS2U4n[R"U5n[R"U5nU[R "[R"U55-n[R "U S5nUUU4==U- ss'[RR[RRU55RnUn[SU SU5GHn[UU-U S5nUU- n[R"UUU2SS245n[R"U5n[R"U5n[R"U5nUUU2UU24n[U5GH!nUUSS24nUUU4nUS :wa(UU-U-S:XaU "UUU-UU-U-2SS245upU [R"[R "USS2[R"4U - 5U -ST$5U - -R%5n U UUSS24'UU - S-US-- UUSS24'UU - U- n!UUS2SS24==[R&"[R("UUS2U4SS 9[R("U!SS 95-ss'U!UUSS24'GM$ UUUU2SS24'US- UUU2SS24'UUS2SS24==[R&"UUS2UU24U5-ss'GM U(a[R"W5n"UU"SS24n[R*"UUR5nAU$) aQuant the weight with GPTQ method. Args: W (array): weight. H (array): Hessian matrix. num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". blocksize (int, optional): blocksize to quantize weight. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. Returns: Q: fake quantized weight rrdg?g333333@cF>URnT(d#[R"UR5SS9n[R"URS5n[R "[R "USS9U5n[R"[R"USS9U5nTS:XaR[R"[R"U5U5nUS:n[R"U5(aXB*X2'US:HUS:H-nSX2'SXB'XC- T- nTS:Xa*[R"UR5TS--S- nO[R"U*U- 5nT(Ga1[R"URS/5[S5-n[[TT-55HnSUT- - n X-n X-n X- T- n TS:wa[R"U *U - 5OUn [R "[R"X - 5U -ST5nX-n[R""[R"U5T5n[R$"US5nX:n[R"U5(dMXXr'XXR'XXb'M T(d1USn[R&"XR5n[R&"Xb5nS/S/[)U5S- --n[R*"UU5n[R*"UU5nXV4$)Nrrrr]rrinf)r*r( expand_dimsrkr)rordrfrerganyrhrnrmrrarspowerrrepeatrr.)rrtmpxminxmaxrDzerobestr}pxmin1xmax1scale1zero1qerrr*gridrw maxshrinkmsenorm perchannelrus rV find_paramsgptq..find_params~sLL ^^FNN$41=Fhhv||A'zz"&&a0#6zz"&&a0#6 U?::bffTlD1D(Cvvc{{!YJ qyTQY'  $ U?775;;'4!84q8D88TEEM*D 77FLLO,-e >%A,Q#GX\cdIef fIDAJ "R%( "Q;r"uax "#q& RYYtBCBJ/66 ;,>**T" gqjM 1aggA HrXcR[U5nURb)[RR UR5OSn[ XX5unnA[ R"URRR5nURUVs/sHnURPM sn5 /nUR5HnURS;dMURUR05S:wdM7URUR05RSS5S:XdMiUR!UR"S5 M [%['U55nUR)U5 UR*(a/[,R."URURS-S S S S 9 UR*(d.[0R2"URR55UU S 9O"[0R2"URS-UU S 9n[7U5GHunn[9[;U5US -5 /n/nUR<UGH<nURS;dMURUR05S:wdM8URUR05RSS5S:XdMjUR?UR"S 5cM[@RB"UR?UREUR5R"S 5U5R 5n[;URF5S:waGMUR!U5 UR!UREUR55 GM? [;U5S:XaGMUVs/sH5n[HRJ"URFSURFS45PM7 nnSnUHnURMU/U5SnURFSn[HRN"USURFS45nUVs/sHnUUUU-- -PM nnUU- n[HRP"SU- 5U-nUVs/sH'nU[HRR"URTU5-PM) nnM [WUUUS S9GH)unnn URU;a6UURSnUURSnUURSnUS:waUOURFSnURXn![[UU UUUUUU U U S9 n"UR?UR"S 5n#UR]UR"S 5n$US:Hn%U%(aURFn&U&SU-S - U-n'[_U"UU'5n"[aU"RTX4US5un"n(n)[cUU&UUU'U"ReS5U(ReU!5US:XaU)OSU S9 un*n+URgU+5 URiU5 URkU*5 O[,RlRoUR"S SU<SU<3-[qU!5U"RFU"ReU!5Rs5S S9n,URuU,5 U,RUR"S 'U$S :XdGMURwU#5 GM, GM URU5 URRRRyU5 UR{5 UR*(aCSSK>J?n- U-"UR[RRUR5S5 U$s snfs snfs snfs snf)a+Quant the model with GPTQ method. Args: model (ModelProto or ONNXModel): onnx model dataloader (object): dataloader for calibration. weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'GPTQ' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". n_samples (int, optional): calibration sample number. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. blocksize (int, optional): blocksize to quantize weight. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel NrrrrGPTQrrTFrrrrrrr"rAru)r@rArur9r:r;r)r+rr\r rZrrrr)load_external_data_for_model)Arrrrrrrr4rgraphr<remove_tensors_from_outputsrrrLrr5r+listsetadd_tensors_to_outputsrr1rrrrrrrinput_name_to_nodesrrrget_noder*r(r)runr.rr6rrr rNrrr~rWr6r remove_nodeadd_noder2r3rr4rr MergeFromronnx.external_data_helperrQsplit).rr rr@rArur r:r9r;r)r+r$rrr%r  org_outputr} output_namesr>r rP input_name node_listrrHsnsamplesrtrrr8r rCrrrrrBrDrrrJrTrQs. rV gptq_quantizeres^ e E494D4D4Prwwu//0VXH*HJFBu{{00778J %%z&Bz!qvvz&BCL  LLJ &!!$))R0F:!!$))R044[&IVS    1 . L)*L   .   KK    ."&$(#  ## U[[::!%%dii488fMQWW))$**Q-8D%..))%..*C*I*I!*LMx$&v||$)v&   !:;: w<1  :A B'Qbhh AGGAJ/0' BD++zlD1!4C))A,C**S2syy}"56C=?@R!x8c>23RB@ OH''!h,'#-C578R!biis++RB8BGR 6    yyM)(3F;*4995lC &tyy1(;'1R'7V\\!_JLLE!%#!!% H"11$**Q-@M"<rns0%   0!%   . /p.f3'lF'RXDv8&4   %&tn6x    I ^     %&|rX