7h^ SSKrSSKrSSKJr SSKrSSKrSSKJr SSKJ r J r SSK J r "SS\5r "S S \5r\RS \4S j5rS \R"S \ 4SjrS \R"S \4SjrS \R"S \4Sjr"SS\5r"SS\5r"SS\5rS/S//rS/S//S/S//S/S///r/SQ/SQ/SQ/rS \R"S \4Sjrg) N)IntEnum)ir)get_dtype_size sympy_product)Vc \rSrSrSrSrSrSrg) NCCL_COLLrrN)__name__ __module__ __qualname____firstlineno__ ALL_REDUCE ALL_GATHERREDUCE_SCATTER__static_attributes__r W/var/www/fran/franai/venv/lib/python3.13/site-packages/torch/_inductor/comm_analysis.pyr r sJJNrr c \rSrSrSrSrSrSrg)NVIDIA_GPU_TYPErrr r N)rrrrVOLTAAMPEREHOPPERrr rrrrs E F FrrreturncN[RRR[RRR5=(d SnSU;a[ R $SU;a[ R$SU;a[ R$[ R$)NV100A100H100) torchutils collect_env get_gpu_inforunrrrr)gpu_infos r get_gpu_typer*s|{{&&33EKK4K4K4O4OPVTVH $$$ 8 %%% 8 %%%%%%rnodec[U[R5(d[SU35eURnUceSU;a[ R $SU;a[ R$SU;a[ R$[SU35e)Nz!node is not a collective kernel: all_reduce all_gatherreduce_scatterzUnsupported collective kernel: ) isinstancer_CollectiveKernel ValueErrorpython_kernel_namer rrr)r+ kernel_names rget_collective_typer5(s dB00 1 1>7v>??rc \rSrSrSrSrSrSrg)NCCL_HWSrrr r N)rrrrNVLINKPCINETrr rrrOrOSs F C CrrOc\rSrSrSrSrSrg) NCCL_ALGOYrrr N)rrrrTREERINGrr rrrUrUYs D DrrUc\rSrSrSrSrg) NCCL_PROTO^rr N)rrrrLLrr rrrZrZ^s  BrrZg333333@gffffff@g333333??g@g@)C@r^gffffff4@)gU@g6@g3@c[U5nUS- S- S- nSn[U5n[R"XC- 5nUnUS::ag[R n[ Rn[U5n [RRRn [RRRn [5n US::aUS- OSn US:XaU OSn[UU nUS:XaU OU nSnUU-n[!UUUS:dU ["R$:XaSOS-5nU ["R$:Xa SUS- -nO)U ["R&["R(4;aUS- nSU-W- nUU-nUS - n[*R,nU ["R$:XaUS:aSU-nO,SnO)U ["R&["R(4;aUS- n[.UUn[0UUUn[0[*R2UUnS nUS:aSn[5UU5nUUW- U-UU--- nUS -nUU- nUU-$) a Returns estimated NCCL collective runtime in nanoseconds (ns). The following heuristics are copied from https://github.com/NVIDIA/nccl/blob/master/src/graph/tuning.cc. We aim to estimate the runtime as accurately as possible. Assumptions: - only ring algorithm (NCCL_ALGO_RING) is used - only Low-Latency protocol (NCCL_PROTO_LL) is used, i.e. Simple or LL128 is not used - 8 gpus per node # TODO: Need to find a way to get accurate "gpus per node" and "# nodes" info. - collective is one of: allreduce, reducescatter, allgather irrr g?gUUUUUU?r]geAgg@@)rErMmathceilrUrXrZr\r5r$ _inductorconfig intra_node_bw inter_node_bwr*llMaxBwsminr rrrrOrQbaseLathwLatrSmax)r+tensor_storage_size_bytestensor_storage_size_GBnum_gpus_per_node group_sizenNodesnRanks nccl_algo nccl_protocollbwIntrabwInter compCapIndexindex2index1llMaxBwbw nChannelsbusBwnstepsratio bandwidthbandwidth_GB_per_nsintraHw nInterStepslatencyintraLatinterLat netOverhead latency_ns transport_nss r estimate_nccl_collective_runtimerso!@ E6=DtK*40J YYz5 6F F {IJ t $D oo$$22Goo$$22G>L!Q;VaZAF#q[\aFvv&GaKWBI NE  !ty/C/C'C9) U E  y###fqj! )**I,@,@A A!6\V #E I#c/nnG y### A:f*KK )**I,@,@A Aqj i ,GW~i(4HW[[!),Z8HK z 8[)H $0;3IIIG3J*,??L * $$r) functoolsraenumrr;r$r rr%rr virtualizedrr r lru_cacher*IRNoder5r=rErMrOrUrZrirjrgfloatrr rrrsJ  0 g   &o & & Jbii JI J "))  @BII@#@ g               ,   ,b%299b%b%r