U qh. @sddlmZddlZddlZddlZddlZddlmZddlm Z m Z ddl m m ZddlmZmZmZmZmZGdddZdd d d d d Zdd d dddZdDdd ddddd dddZdEdddddd dddZdd dd d!Zdd d d"d#d$Zdd%d&d'd(Zd)d*d%d+d,d-Zd.d%d/d0d1Zdd%d&d2d3Zd.d%d/d4d5Z dd%d&d6d7Z!d dd8d9d:Z"ddd;dd?d@Z$dd ddAdBdCZ%dS)F) annotationsN)chain)CallableIterable)AttributeProto FunctionProto GraphProto ModelProto TensorProtoc@seZdZdddddZdS)ExternalDataInfor NonetensorreturncCsbd|_d|_d|_d|_d|_|jD]}t||j|jq$|jrLt |j|_|jr^t |j|_dS)N) locationoffsetlengthchecksumbasepath external_datasetattrkeyvalueint)selfrentryr=/tmp/pip-unpacked-wheel-xnis5xre/onnx/external_data_helper.py__init__s  zExternalDataInfo.__init__N)__name__ __module__ __qualname__rrrrrr sr r strr )rbase_dirrc Csft|}t||j|j}t|d8}|jr8||j|jrN| |j|_ n | |_ W5QRXdS)zLoads data from an external file for tensor. Ideally TensorProto should not hold any raw data but if it does it will be ignored. Arguments: tensor: a TensorProto object. base_dir: directory that contains the external data. rbN) r c_checkerZ_resolve_external_data_locationrnameopenrseekrreadraw_data)rr$infoexternal_data_file_path data_filerrrload_external_data_for_tensor)s  r/r )modelr$rcCs8t|D]*}t|rt||tj|_|jdd=qdS)zLoads external tensors into model Arguments: model: ModelProto to load external data to base_dir: directory that contains external data N)_get_all_tensorsuses_external_datar/r DEFAULT data_locationr)r0r$rrrrload_external_data_for_model?s   r5z int | Nonez str | None)rrrrrrrc Cs|dstd|jd|jdd=tj|_||dk rBt|nd|dk rTt|nd||dD]*\}}|dk rd|j }||_ t ||_ qddS)Nr+zTensor zGdoes not have raw_data field. Cannot set external data for this tensor.)rrrrr) HasField ValueErrorr'rr EXTERNALr4ritemsaddrr#r) rrrrrrkvrrrrset_external_dataOs(   r=TFboolr)r0all_tensors_to_one_filersize_thresholdconvert_attributerc Cst|}|rt|}|rptt}|r@tj|r= the size_threshold it will be converted to external data. To convert every tensor with raw data to external data set size_threshold=0. convert_attribute (bool): If true, convert all tensors to external data If false, convert only non-attribute tensors to external data zDlocation must be a relative path that is relative to the model path.r+N)_get_initializer_tensorsr1r#uuidZuuid1ospathisabsr7r6sys getsizeofr+r=r'_is_valid_filename) r0r@rrArBtensors file_namerZtensor_locationrrrconvert_model_to_external_datams4   rM)r0rcCs@t|D]2}t|r|ds&td|jdd=tj|_qdS)zCall to set all tensors which use external data as embedded data. save_model saves all the tensors data as embedded data after calling this function. Arguments: model (ModelProto): Model to be converted. r+raw_data field doesn't exist.N)r1r2r6r7rr r3r4)r0rrrr convert_model_from_external_datas    rO)r base_pathrc Cst|}tj||j}|ds*tdtj|sLt|dW5QRXt|dz}| dd|j dk r| }|j |kr| d|j || |j | }| |j t||j|| |W5QRXdS) zWrites tensor data to an external file according to information in the `external_data` field. Arguments: tensor (TensorProto): Tensor object to be serialized base_path: System path of a folder where tensor data is to be stored r+rNabzr+brN)r rErFjoinrr6r7isfiler(r)rtellwriter+r=)rrPr,r-r. file_sizerrrrsave_external_datas"          rYzIterable[TensorProto])onnx_model_protorcCstt|t|S)z=Scan an ONNX model for all tensors and return as an iterator.)rrC_get_attribute_tensors)rZrrrr1sr1rz-Callable[[GraphProto], Iterable[TensorProto]]) attributefuncrccsF|jtjkr||jEdH|jtjkrB|jD]}||EdHq.dS)zICreate an iterator through processing ONNX model attributes with functor.N)typerZGRAPHgZGRAPHSZgraphs)r\r]graphrrr_recursive_attribute_processors    razGraphProto | FunctionProto)graph_or_functionrccsBt|tr|jEdH|jD] }|jD]}t|tEdHq&qdS)zICreate an iterator of initializer tensors from ONNX model graph/function.N) isinstancerZ initializernoder\ra#_get_initializer_tensors_from_graphrbrdr\rrrres    reccs.t|jEdH|jD]}t|EdHqdS)z:Create an iterator of initializer tensors from ONNX model.N)rer` functions!_get_attribute_tensors_from_graphrZfunctionrrrrCs rCccsJ|jD]>}|jD]2}|dr&|jV|jEdHt|tEdHqqdS)zSCreate an iterator of tensors from node attributes of an ONNX model graph/function.tN)rdr\r6rkrKrarhrfrrrrhs    rhccs.t|jEdH|jD]}t|EdHqdS)zDCreate an iterator of tensors from node attributes of an ONNX model.N)rhr`rgrirrrr[s r[)filenamercCstd}||}t|S)z8Utility to check whether the provided filename is valid.z^[^<>:;,?"*|/]+$)recompilematchr?)rlexprorrrrJs  rJr cCs|do|jtjkS)z?Returns true if the tensor stores data in an external location.r4)r6r4r r8)rrrrr2s  r2)r field_keyrcCs*t|jD]\}}|j|kr |j|=q dS)aRemoves a field from a Tensor's external_data key-value store. Modifies tensor object in place. Arguments: tensor (TensorProto): Tensor object from which value will be removed field_key (string): The key of the field to be removed N) enumeraterr)rrqifieldrrrremove_external_data_fields  ru)r0filepathrcCs8t|D]*}t|r|drt|||dq|S)aSerializes data for all the tensors which have data location set to TensorProto.External. Note: This function also strips basepath information from all tensors' external_data fields. Arguments: model (ModelProto): Model object which is the source of tensors to serialize. filepath: System path to the directory which should be treated as base path for external data. Returns: ModelProto: The modified model object. r+)r1r2r6rYZ ClearField)r0rvrrrrwrite_external_data_tensors,s   rw)NNNN)TNr>F)& __future__rrErmrHrD itertoolsrtypingrrZonnx.onnx_cpp2py_export.checkerZonnx_cpp2py_exportcheckerr&Z onnx.onnx_pbrrrr r r r/r5r=rMrOrYr1rarerCrhr[rJr2rurwrrrrsB    4"