U h=@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z e eZddZddZdd Zd d Zd d ZedkredS)N) setup_logger) WhisperConfigWhisperProcessorcCst}|jddtddd|jddddd |jd d dd d gdd|jddtdd|jddtdd|jddddd|jddddd|jdtdd |jd!td"d |jd#tdd$d|jd%tdd&d'd(gd)d*|jd+tdd,d-d.gd/d*|jd0td1d2d3|jd4ddd5d|jd6tdd7d3|jd8ddd9|}t|d:|jd;d< d=d>d?|j d>|j }t|d@|t j |jddA|jdB9_|S)CNz-a --audio-pathTz0Path to folder of audio files for E2E evaluation)typerequiredhelpz-lz --languagezLanguage of audio file)defaultrz-tz--taskZ transcribe translatezTask to complete)r choicesrz-w --warmup-runs)rr z-n --num-runs z --hf-pt-eagerF store_truez,Benchmark in PyTorch without `torch.compile`)r actionrz--hf-pt-compilez)Benchmark in PyTorch with `torch.compile`--hf-ort-dir-pathzDPath to folder containing ONNX models for Optimum + ORT benchmarking)rr--ort-model-pathz'Path to ONNX model for ORT benchmarking --model-namez9Model name in Hugging Face (e.g. openai/whisper-large-v2) --precisionZint8Zfp16Zfp32zPrecision to run model)rrr r--devicecpuZcudaZrocmzDevice to benchmark models --device-idrz GPU device ID)rr rz --verbosezPrint detailed logsz --timeoutz8Number of mins to attempt the benchmark before moving on--tune)r r model_size/.-z./ log_folder)exist_ok<)argparseArgumentParser add_argumentstrint parse_argssetattr model_namesplitreplacer precisionosmakedirsrtimeout)parserargsZlog_folder_namer2Y/tmp/pip-unpacked-wheel-socb9apf/onnxruntime/transformers/models/whisper/benchmark_all.pyget_argss  r4c Csg}d}d}d}d}d}d\} } d\} } d\} }}}d\}}d}d}d }d }t|(}|D]}|d d }||kr|d }n"||krd}n||ks||krd}||krt|t||d}q\||krt|t||d}|d kr||} } d}|dkrx||} } d}q\||krNt|t||d} || }|d}q\||kr\d|krt||dd|dd}n@|||t|ddd}t||d}t|d}| r| nd| r| nd|r|nd}|d}|r||nd}t d|dt d|dt d||| | | | rP| nd| r\| nd|||||g }| |q\W5QRX|S)Nz Load audio: zFeature extraction: zEvaluating PyTorch...zEvaluating ONNX Runtime...)NN)NNNNz Latency: z Throughput: zGenerated token length: zpeak= z load-audiozfeature-extractionprocess iZCPU=z MB'"Z max_used_MBrrzTotal latency: z szAudio duration: zReal-time factor: ) openr+floatlenrfindr&findjsonloadsloggerinfoappend) device_idlog_file base_resultsentriesstepZload_audio_patternZfeat_ext_patternZpytorch_patternZonnxruntime_patternZload_audio_latency_sZload_audio_throughput_sZfeat_ext_latency_sZfeat_ext_throughput_sZ token_lengthZ latency_sZper_token_latency_sZper_token_latency_msZ throughputZmemoryZlatency_patternZthroughput_patternZtoken_length_patternZmemory_patternfZ input_linelineZpeakusageZ total_latencyZaudio_durationZrtfentryr2r2r3process_log_files            &"       rPcCsddl}|j|dddddddd d d d d dddgd}|dd|d<|dd|d<|dd|d<|d d|d <|d d|d <|d d|d <|d d|d <|d d|d <|dd|d<|dd|d<|dd|d<|j|ddtd|ddS)NrZEngineZ PrecisionZDevicez Audio Filez Duration (s)z Token LengthzLoad Audio Latency (s)zLoad Audio Throughput (qps)zFeature Extractor Latency (s)z"Feature Extractor Throughput (qps)z Latency (s)zPer Token Latency (ms/token)zThroughput (qps)z Memory (GB)zReal Time Factor (RTF))columnsr>r&F)indexzResults saved in !)ZpandasZ DataFrameZastypeZto_csvrDrE)resultsfilenamepdZdfr2r2r3 save_resultssBrWc Cs|dtjdd}tj|j|}t|dF}tj|||d}z| |j Wntj k rr| YnXW5QRXt d||j|j||g} t|j|| } | S)N_%Y-%m-%d_%H:%M:%Sz.logw)stdoutstderrz Gathering data from log files...)datetimenowr-pathjoinrr= subprocessPopenwaitr/TimeoutExpiredkillrDrEr,devicerPrG) r1 benchmark_cmdZengine audio_filedurationZ log_filenameZlog_pathrHr7rIrTr2r2r3 benchmarks  rjcCsvt}t|jt|jdtjj_ t |j }t |j }|j|j|jd}|jgttdd|}|jr|jrdt|gng}|jr|jrdt|gng}|jrdgng}g}t|jD]t} tj|j| } ztj| d} WnNtk r6} z.d} tjd | dd td | d W5d} ~ XYnXtd | d|jrdddd| ddd|j d|j d|j!dt|j"dt|j#dt|j$d|j%g|} tdt || d| | }|&||j'r:dddd| ddd|j d|j d|j!dt|j"dt|j#dt|j$d|j%g|} tdt || d | | }|&||j(rdddd| dd!d"|j(d|j d|j d|j!dt|j"dt|j#dt|j$d|j%g|} td#t || d$| | }|&||j)rdddd| dd%d&|j)d|j d|j d|j!dt|j"dt|j#dt|j$d|j%g||} td't || d(| | }|&|q|j*d)|j d*t+j+,d+d,}t-|tj|j%|dS)-NT)languagetaskcSs|dS)Nr:r2)Ztoken_idr2r2r3/zmain..z--decoder-input-idsr)r_rz@An error occurred while trying to calculate the audio duration: )exc_infozJIf you get an error that says: soundfile.LibsndfileError: Error opening 'zt': File contains data in an unknown format. you may not have installed `ffmpeg` in addition to installing `librosa`.zTesting z...pythonz-mzmodels.whisper.benchmarkrz--benchmark-typez hf-pt-eagerrrrrr rz --log-folderz'Benchmark PyTorch without torch.compilez pytorch-eagerz hf-pt-compilez$Benchmark PyTorch with torch.compilezpytorch-compilezhf-ortrz Benchmark Optimum + ONNX Runtimez optimum-ortZortrzBenchmark ONNX RuntimeZ onnxruntimerrXrYz.csv).r4rverboserDrE__dict__torchbackendsZcudnnrjrZfrom_pretrainedr)rZget_decoder_prompt_idsrkrlZdecoder_start_token_idlistmapr%Ztuner-listdir audio_pathr_r`librosaZ get_duration ExceptionwarningZ hf_pt_eagerr,rfrGZ warmup_runsZnum_runsrextendZ hf_pt_compileZhf_ort_dir_pathZort_model_pathrr]r^rW)r1config processorZhf_forced_decoder_idsZort_forced_decoder_idsZhf_decoder_input_ids_cmdZort_decoder_input_ids_cmdZ ort_tune_cmdZ all_resultsrhrxriergrTZcsv_filer2r2r3main#s(               "r__main__)r"r]rBloggingr-raryrsZbenchmark_helperrZ transformersrr getLogger__name__rDr4rPrWrjrr2r2r2r3s$  }[)$