8h2$SSKrSSKrSSKrSSKJrJr SSKrSSKJr SSK J r J r SSK J r JrJrJrJrJrJrJrJrJrJrJrJr SSKJr SSKJrJrJrJ r J!r!J"r" SSK#J$r$J%r%J&r&J'r'J(r(J)r) SS K*J+r+ SS K,J-r- S S /r."S S \ 5r/"SS \ 5r0g)N)AnyOptional)narrow_tensor_by_index) FsspecReader FsspecWriter) _gen_file_name _get_dtype_get_safetensors_file_metadata_HFStorageInfo _metadata_fnCUSTOM_METADATA_KEYDATA_KEYDATA_OFFSETS_KEYDEFAULT_EXTRA_METADATA_KEY DTYPE_KEYSAVED_OFFSETS_KEY SHAPE_KEYSUFFIX)SerializationFormat)ChunkStorageMetadataMetadata MetadataIndex StorageMetaTensorPropertiesTensorStorageMetadata)LoadPlan LoadPlannerReadItemSavePlan SavePlanner WriteItem) WriteResult)FutureHuggingFaceStorageWriterHuggingFaceStorageReaderc ^\rSrSrSrSS\S\\\\4S\\S\ SS4 U4S jjjr S \ \ S\ \ 4S jr S \ S \S\\ \4U4SjjrS\S\ \ \SS4SjrS\\\\4S\ \S\\\ \44Sjr\S\4Sj5rSrU=r$)r$1z A writer that writes to a huggingface repository in the huggingface format. Uses Fsspec back-end to communicate with back-end storage. Fsspec registration of the storage solution is required. Npathfqn_to_index_mappingtoken save_shardedreturnc>Ub[TU]UU[RS9 O[TU]U[RS9 X lX@lg)aw Initialize the huggingface writer pointing to path. Args: path: hf directory where the checkpoint will be read from. Needs to have .safetensors files, but can be from any fsspec supported storage, including localFS and hf://. fqn_to_index_mapping: A mapping from tensor FQN to the index of the file that the tensor should be written to. Indices are from 1 to N, where N is the number of files. If not provided, the tensors will be written to a single file. If none, then all the tensors on the same rank will be written to the same file. token: The token to use to authenticate with huggingface hub. save_sharded: If True, save the checkpoint as a sharded checkpoint where every rank saves its own shard. Default is False which assumes full tensors are being saved. N)r(r*serialization_format)r(r.)super__init__r SAFETENSORS_fqn_to_index_mapping _save_sharded)selfr(r)r*r+ __class__s a/var/www/fran/franai/venv/lib/python3.13/site-packages/torch/distributed/checkpoint/hf_storage.pyr0!HuggingFaceStorageWriter.__init__8s\0   G %8%D%D   G %8%D%D  @T")plansc/n[USS9H[up40nURbURUS'UR(aX5S'UR[R "XES95 M] U$)N)startr) shard_index) storage_data) enumerater2r3append dataclassesreplace)r4r9 new_plansiplanr>s r6prepare_global_plan,HuggingFaceStorageWriter.prepare_global_plan^sn  a0GA+-L))57;7Q7Q 34!!./]+   [00Q R1r8rEplannerc.>[UR5S:Xa[5nUR/5 U$URnSnSnSU;aUSnSU;aUSnUR XQR5nUb[ UR55OSn[R"5n UR5HHup[XU5n U RURRURU 5X45 MJ [T U]AX)5$)Nrr)r=r;)lenitemsr# set_resultr>_split_by_storage_planmaxvaluesqueueQueuerputfs concat_pathr(r/ _write_data)r4rErHfutr> storage_planr=buckets highest_index file_queue file_index write_items file_namer5s r6 write_data#HuggingFaceStorageWriter.write_dataks tzz?a  (C NN2 J(,'8'8 15 %) !\ 1'(>?L L (&}5K--lJJG6B6NL//12TU "'++- '.}} #J&z+NI NN$$TYY :IS (7 w"777r8metadataresultscTUR(ag0n0nSnUHnURUVs0sH-owRRURR _M/ sn5 U[ UVs/sHowRRPM sn5- nM SU0US'XCS'URRUR[5nURRUS5n [R"X9SS9 SSS5 gs snfs snf!,(df  g=f)Nr total_sizer` weight_mapw)indent)r3updateindexfqnr> relative_pathsumlengthrSrTr(r create_streamjsondump) r4r`rametadata_to_write storage_mdrcwr_listwr metadata_path metadata_files r6finishHuggingFaceStorageWriter.finishs      G   GNOwr<< U$)Nr;)rirjr@)r4rWrKrXitemkeyidxs r6rM/HuggingFaceStorageWriter._split_by_storage_plans^  u: D**..C#C! $v  ##D)r8c[$N)r )r4s r6ru&HuggingFaceStorageWriter.metadata_pathsr8)r2r3)NNF)__name__ __module__ __qualname____firstlineno____doc__strrdictintboolr0listrrFr r#r"r^rrwr!rMpropertyru__static_attributes__ __classcell__r5s@r6r$r$1s$:>#" $*$*'tCH~6$*} $*  $*  $*$*L h DN 888 [! " 8>BxB$tK7H2IBdB&$T#s(^4=A)_ c4 ?" #&sr8cp^\rSrSrSrS S\S\\SS4U4SjjjrS\S \ S\ S4S jr S\ 4S jr S rU=r$)r%z A reader that reads from a huggingface repository in the huggingface format. Uses in Fsspec back-end to communicate with storage. Fsspec registration of the storage solution is required. Nr(r*r,cB>Ub[TU]XS9 g[TU]US9 g)a1 Initialize the huggingface reader pointing to path. Args: path: hf directory where the checkpoint will be read from. Needs to have .safetensors file, but can be from any fsspec supported storage, including localFS and hf://. token: The token to use to authenticate with huggingface hub. N)r(r*)r()r/r0)r4r(r*r5s r6r0!HuggingFaceStorageReader.__init__s+   G $  4 G $  'r8rErHc xSSKJn 0nURHInURURnUR nUR U/5RU5 MK UR5GHupxURRUS5n U"U R55n U V s0sH oSU S_M n n UGH0n URU RnXRR[n[R"UUR S9nUR#UR$5n['XR(U R*5nUR-U 5R/5nUR15UR15:Xd6SU RSUR15SUR1535eUR3U5 UR5U U5 GM3 SSS5 GM [75nUR9S5 U$s sn f!,(df  GM=f) Nr) deserializerbr;dtypezreq z mismatch sizes z vs ) safetensorsrrKr> storage_indexrk setdefaultr@rSrnread dest_indexrjrtorch frombufferrreshapeshaperstorage_offsetslengthsresolve_tensordetachsizecopy_ commit_tensorr#rL)r4rErHrper_file read_itemitem_mdr]reqsstream deserialized tensor_infodeserialized_dictreq tensor_bytestensor target_tensorrVs r6 read_data"HuggingFaceStorageReader.read_datas+.0I&*&7&7 8O8O&PG--I    2 . 5 5i @$ (~~/OI&&y$76 +6;;=9 FR@FR{NKN2l"@ C"//0A0ABG#4^^5G5G#H#RL"--$%mmF$^^GMM:F3 3 3S[[F%,$:$:3$?$F$F$HM(--/6;;=@s0011A-BTBTBVAWW[\b\g\g\i[jk@"''/))#}=) 87 0>h t 9@ 87sH)/H$D9H)$H)) H9 c60n0n/nURRUR5H0nUR[5(dMUR U5 M2 UGHnURR US5n[U5upxUR[5n Sn U (aCU R[5(a)[R"U R[55n UR5GH+upU [:XaMU b X[n OS/[U [ 5-n X;a[#[%['U [(5S9[*R,"[/U [ U 5VV s/sH upX-PM sn n5[1[*R,"W 5[*R,"U [ 5S9/S9X'OXR2R [1[*R,"U 5[*R,"U [ 5S95 [5XR65n[9[U55H%n[;UUU [ UU U-5UU'M' [*R,"U5XlU b[=XU [S9nO[=U S/[U [ 5-S9n[?UU [@SU [@SU [@S- [*R,"U [ 5['U [(5S 9UU'GM. SSS5 GM [CUUS 9n[EUS S5c[G5Ul$URJURHl%U$s sn nf!,(df  GM6=f) Nrrr)offsetssizes) propertiesrchunks)r)rjoffsetr;)rkrrmrr)state_dict_metadatar> storage_meta)&rSlsr(endswithrr@rnr getrr roloadsrKrrJrrrr rrSizeziprrrrrangerNrr rrgetattrrrload_id)r4rr>safetensors_filesfilesafetensor_filefsafetensors_metadata_custom_metadatadcp_sharding_infor{valrsavedrrDmetadata_indexr`s r6 read_metadata&HuggingFaceStorageReader.read_metadatas;@B<> GGJJtyy)D}}V$$!((.* 1O&&=*H*K'$"6":":;U"V$(!"':':;N'O'O(, '++,?@)%!5 : : ':!:53H'7&0Y&@("':=S^V9T!"9T %*N9T!""!5,1JJv,>*/**S^*D!"$4+0$,077>>0 % 6 2%**S^:T $$7$<$A$AB!&s4y!1A&)$q'3y>!3Dvay3P&QDG"28= 48H+05)4)6 #c,BCT,U**7 #QC#c)n2E,E*4B&5"#34Q7"#34Q7#>N:OPQ:RR#jjY8(Y8 4L0a!=>= 1F 3%  8^T 2 :$/MH !(, %c!"5>=s C>NNF!NN N r)rrrrrrrr0rrr#rrrrrrs@r6r%r%s^ (S(#($(( +h+++ZUxUUr8)1rArorPtypingrrrtorch.distributed._shard._utilsr/torch.distributed.checkpoint._fsspec_filesystemrr&torch.distributed.checkpoint._hf_utilsrr r r r r rrrrrrr'torch.distributed.checkpoint.filesystemr%torch.distributed.checkpoint.metadatarrrrrr$torch.distributed.checkpoint.plannerrrrrr r!$torch.distributed.checkpoint.storager" torch.futuresr#__all__r$r%rr8r6rsz BVH=  &'A BA|AHY|Yr8