U qhE @sdZddlmZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z mZmZmZddlmZddlmZddlZd ejkre ejd d an4d ejkre ejd d d ane ejd dd d aGdddZdddddZddddZdddddZdddddZd;dddd!d"d#Zdddd$d%d&Z dddd.ddd2d3d4d5Z#d?ddd.ddd(d3d6d7Z$d@ddddd.ddd2d8d9d:Z%dS)AzJONNX Model Hub This implements the python client for the ONNX model hub. ) annotationsN)BytesIO)join)IOAnyDictListcast) HTTPError)urlopenZ ONNX_HOMEZhubXDG_CACHE_HOMEonnx~z.cachec@s<eZdZdZdddddZddd d Zddd d Zd S) ModelInfoaMA class to represent a model's property and metadata in the ONNX Hub. It extracts model name, path, sha, tags, etc. from the passed in raw_model_info dict. Attributes: model: The name of the model. model_path: The path to the model, relative to the model zoo (https://github.com/onnx/models/) repo root. metadata: Additional metadata of the model, such as the size of the model, IO ports, etc. model_sha: The SHA256 digest of the model file. tags: A set of tags associated with the model. opset: The opset version of the model. zdict[str, Any]None)raw_model_inforeturncCstt|d|_tt|d|_ttttf|d|_d|_d|jkrZtt|jd|_t|_ d|jkrttt t|jd|_ tt |d|_ ||_ dS)zgInitializer. Args: raw_model_info: A JSON dict containing the model info. model model_pathmetadataN model_shatagsZ opset_version)r strrrrrrrsetrrintopsetr)selfrr,/tmp/pip-unpacked-wheel-xnis5xre/onnx/hub.py__init__+s  zModelInfo.__init__rrc Cs&d|jd|jd|jd|jd S)NzModelInfo(model=z, opset=z, path=z , metadata=))rrrrrrrr__str__@szModelInfo.__str__cCs|SN)r#r"rrr__repr__CszModelInfo.__repr__N)__name__ __module__ __qualname____doc__rr#r%rrrrrs rrr)new_dirrcCs|adS)zkSets the current ONNX hub cache location. Args: new_dir: Location of new model hub cache. N _ONNX_HUB_DIR)r*rrrset_dirGsr-r cCstS)znGets the current ONNX hub cache location. Returns: The location of the ONNX hub model cache. r+rrrrget_dirQsr.ztuple[str, str, str])reporcCsV|dddd}|dddd}d|krH|dd}nd}|||fS)zCGets the repo owner, name and ref from a repo specification string.:r/main)splitr/ repo_owner repo_namerepo_refrrr_parse_repo_infoZs r9boolcCs&t|\}}}|dko$|dko$|dkS)zzVerifies whether the given model repo can be trusted. A model repo can be trusted if it matches onnx/models:main. r modelsr3r9r5rrr_verify_repo_refesr=F)r/lfsrcCsBt|\}}}|r*d|d|d|dSd|d|d|dS)aTGets the base github url from a repo specification string. Args: repo: The location of the model repo in format "user/repo[:branch]". If no branch is found will default to "main". lfs: Whether the url is for downloading lfs models. Returns: The base github url for downloading. z*https://media.githubusercontent.com/media/r1z"https://raw.githubusercontent.com/r<)r/r>r6r7r8rrr _get_base_urlms r?)url file_namerc CsNd}t|8}t|d"}||}|s*q6||qW5QRXW5QRXdS)zDownloads the file with specified file_name from the url. Args: url: A url of download link. file_name: A specified file name for the downloaded file. i@wbN)r openreadwrite)r@rA chunk_sizeresponsefchunkrrr_download_files  rJonnx/models:mainz str | Nonezlist[str] | Nonezlist[ModelInfo])r/rrrc st|}|d}z6t|$}ddtttt|D}W5QRXWn2tk rx}ztd||W5d}~XYnXdkr|nfdd|D}|dkr|Sdd|D} g} |D]0} d d| j D} t | | d kr| | q| S) aGets the list of model info consistent with a given name and tags Args: repo: The location of the model repo in format "user/repo[:branch]". If no branch is found will default to "main" model: The name of the model to search for. If `None`, will return all models with matching tags. tags: A list of tags to filter models by. If `None`, will return all models with matching name. Returns: ``ModelInfo``s. zONNX_HUB_MANIFEST.jsoncSsg|] }t|qSr)r).0inforrr szlist_models..zCould not find manifest at Ncs"g|]}|jkr|qSr)rlowerrLmrrrrNscSsh|] }|qSrrOrLtrrr szlist_models..cSsh|] }|qSrrSrTrrrrVsr) r?r jsonloadr rrr AssertionErrorrlen intersectionappend) r/rrbase_urlZ manifest_urlrGmanifestematching_modelsZcanonical_tagsZmatching_info_listrQZ model_tagsrrRr list_modelss, " raz int | None)rr/rrcszt||}|std|dkr6t|ddd}n<fdd|D}|srdd|D}t|d d ||d S) aGets the model info matching the given name and opset. Args: model: The name of the onnx model in the manifest. This field is case-sensitive repo: The location of the model repo in format "user/repo[:branch]". If no branch is found will default to "main" opset: The opset of the model to get. The default of `None` will return the model with largest opset. Returns: ``ModelInfo``. zNo models found with name NcSs|j Sr$r)rQrrrz get_model_info..)keycsg|]}|jkr|qSrrbrPrbrrrNs z"get_model_info..cSsg|] }|jqSrrbrPrrrrNsz has no version with opset z. Valid opsets: r)rarYsorted)rr/rr`Zselected_modelsZ valid_opsetsrrbrget_model_infos rgzonnx.ModelProto | None)rr/r force_reloadsilentrc Csrt|||}|jd}|jdk r:|jd|d|d<tttj|}|s\tj |st |s|sd|d}t |t j dt dtd krdStjtj|d d t|d } t d |d |t| |j|nt d|d|t|d} | } W5QRX|jdk rZt| } | |jksZtd|jd| d|jdttttt | S)aDownloads a model by name from the onnx model hub. Args: model: The name of the onnx model in the manifest. This field is case-sensitive repo: The location of the model repo in format "user/repo[:branch]". If no branch is found will default to "main" opset: The opset of the model to download. The default of `None` automatically chooses the largest opset force_reload: Whether to force the model to re-download even if its already found in the cache silent: Whether to suppress the warning message if the repo is not trusted. Returns: ModelProto or None r1N_The model repo specification _ is not trusted and may contain security vulnerabilities. Only continue if you trust this repo.fileContinue?[y/n]yTexist_ok Downloading  to local path Using cached model from rbThe cached model has SHA256  while checksum should be h. The model in the hub may have been updated. Use force_reload to download the model from the model hub.)!rgrr4rrr,osseppathexistsr=printsysstderrinputrOmakedirsdirnamer?rJrCrDhashlibsha256 hexdigestrYrr rXr rbytesr) rr/rrhriselected_modelZlocal_model_path_arrZlocal_model_pathmsglfs_urlrHZ model_bytesdownloaded_sharrrrXs6         rXc Cst|||}|jdd}|jd}|dk rD|d|d|d<tttj|}|sftj|st |s|sd|d} t | t j d t d t d krdStjtj|d d t|d } t d|d|t| |jd|nt d|d|t|d} | } W5QRX|dk rbt| } | |ksbtd|jd| d|d|dt|d}tj|||dt|d}|S)awDownloads a model along with test data by name from the onnx model hub and returns the directory to which the files have been extracted. Users are responsible for making sure the model comes from a trusted source, and the data is safe to be extracted. Args: model: The name of the onnx model in the manifest. This field is case-sensitive repo: The location of the model repo in format "user/repo[:branch]". If no branch is found will default to "main" opset: The opset of the model to download. The default of `None` automatically chooses the largest opset force_reload: Whether to force the model to re-download even if its already found in the cache silent: Whether to suppress the warning message if the repo is not trusted. Returns: str or None model_with_data_pathr1model_with_data_shaNrjrkrlrmrnrprqTrrrtrurvrwrxryrzr{r|r)rgrr4rr,r}r~rrr=rrrrrOrrr?rJrCrDrrrrYrrZr utilsZ_extract_model_safelistdir)rr/rrhrirZlocal_model_with_data_path_arrrZlocal_model_with_data_pathrrrHZmodel_with_data_bytesrZlocal_model_with_data_dir_pathrrrrdownload_model_with_test_data$sh             r) network_modelpreprocessing_model network_repopreprocessing_reporrhrirc Cst|||||}|dkr&td|t|||||}|dkrLtd|t} i} i} |jD],} | jdkrrdn| j} | | | j| | <q`|jD],} | jdkrdn| j} | | | j| | <qd}d}| D]n} | dkr| | }| | }q| | kr| | kr| | | | krtd|d|d | d | | d | | d q||krltj ||}|j |_ tj |n,||krtj ||}|j |_ tj |d dt|jj|jjD}tjj|||d}|S)aBuilds a composite model including data preprocessing by downloading a network and a preprocessing model and combine it into a single model Args: network_model: The name of the onnx model in the manifest. preprocessing_model: The name of the preprocessing model. network_repo: The location of the model repo in format "user/repo[:branch]". If no branch is found will default to "main" preprocessing_repo: The location of the proprocessing model repo in format "user/repo[:branch]". If no branch is found will default to "main" opset: The opset of the model to download. The default of `None` automatically chooses the largest opset force_reload: Whether to force the model to re-download even if its already found in the cache silent: Whether to suppress the warning message if the repo is not trusted. Returns: ModelProto or None Nz(Could not load the preprocessing model: z"Could not load the network model: zai.onnxrkzCan not merge z and z: because they contain different opset versions for domain z (z) and z\). Only the default domain can be automatically converted to the highest version of the two.cSsg|]\}}|j|jfqSr)name)rLZ out_entryZin_entryrrrrNsz(load_composite_model..)io_map)rX RuntimeErrorrZ opset_importdomainaddversion ValueErrorr Zversion_converterZconvert_versionZ ir_versioncheckerZ check_modelzipgraphoutputrZcomposeZ merge_models)rrrrrrhriZ preprocessingnetworkZ all_domainsZdomains_to_version_networkZ domains_to_version_preprocessingZopset_import_entryrZpreprocessing_opset_versionZnetwork_opset_versionrZmodel_with_preprocessingrrrload_composite_models       *   r)F)rKNN)rKN)rKNFF)rKNFF)rKrKNFF)&r) __future__rrrWr}rioros.pathrtypingrrrrr urllib.errorr urllib.requestr r environr,r expanduserrr-r.r9r=r?rJrargrXrrrrrrs\       )   2#C^