U hn@sDddlZddlmZdZddZddZddZdd Zd d Z dS) NcCsdtjkrHttjdttjd}ttjd}tjdd||dnfdtjkrdd lm}|j}ttjddttjd d}ttjd d }tjdd ||dndS)NZ LOCAL_RANKZRANKZ WORLD_SIZEZncclztcp://127.0.0.1:7645)Z init_method world_sizerankZOMPI_COMM_WORLD_LOCAL_RANKr)MPIZOMPI_COMM_WORLD_RANKZOMPI_COMM_WORLD_SIZEztcp://127.0.0.1:7647) osenvironintdistZinit_process_groupZmpi4pyrZ COMM_WORLDget)rrrcommr W/tmp/pip-unpacked-wheel-socb9apf/onnxruntime/transformers/models/llama/dist_settings.py init_dists   rcCstdk rtSdSNr)r ZGet_rankr r r r get_ranksrcCstdk rtSdS)Nr)r ZGet_sizer r r r get_size"srcCstdk rtdS)N)r Barrierr r r r barrier&srcGstdkrt|dSr)rprint)argsr r r print_out+s r) rZtorch.distributedZ distributedr r rrrrrr r r r s