7h^SSKJr SSKrSSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r SSK J r Jr SSKrSSKJr SSKJr SSKJr S S KJrJr S S KJr S S KJrJrJr S S KJ r J!r!J"r"J#r#J$r$J%r%J&r& S SK'J(r( \RR"\*5r+\RXR[\*S5r.\(aSSK/J0r0 S"Sjr1S"Sjr2S"Sjr3S"Sjr4\ "SS55r5S#Sjr6S$Sjr7S%Sjr8S&Sjr9Sr:Sr;S"SjrS r?S)S!jr@g)*) annotationsN) defaultdict) dataclass)Any TYPE_CHECKING)trace_structured)StorageWeakRef) OrderedSet)configir)WeakDep)estimate_peak_memoryFreeableInputBufferget_freeable_input_buf)contains_collective contains_waitfind_recursive_deps_of_nodefind_recursive_users_of_node is_collectiveis_fallback_opis_wait)Voverlap)BaseSchedulerNodec[USSSS9$)z/ Greedily schedules waits as late as possible. FT raise_comms sink_waitsreorder_for_overlap_schedule_for_commsnodess O/var/www/fran/franai/venv/lib/python3.13/site-packages/torch/_inductor/comms.pyrr*s Ed c[USSSS9$)z0 Greedily schedules comms as early as possible. TFrr!r#s r%rr3s DU r&c[USSSS9$)a This achieves the following overall scheduling procedure: Step 1: Given that we've currently scheduled comm N, we now schedule all compute nodes that are required for comm N + 1 but do not depend on comm N, to run at the same time with comm N. Step 2: If all those compute nodes are sufficient to overlap comm N, we're done. Otherwise, we now need to look elsewhere to find compute that overlaps with comm N. We prioritize compute nodes that are needed sooner. Step 3: We schedule the compute nodes dependent on comm N and required for comm N + 1. Step 4: We schedule comm N + 1. Repeat this for subsequent comm nodes. Trr!r#s r%reorder_compute_for_overlapr)<s DTt r&c ^ [U5upUVs0sHo3X#R_M nn[UVs/sHo4UPM sn5n[UVs/sHo2URPM sn5nSUSUS3m /SQnUR 5VVs/sHHup8[ U5UR URURURUR/PMJ n nn[RRS5(aSSK J n T U "U US9- m O8T S - m T [U5S -- m T S R[[U 55- m [ R#T 5 [%S S U 4S jS9 U$s snfs snfs snfs snnf)a+ Reorders communication ops relative to computation ops to improve communication-compute overlapping and hide comm latency. Stops moving a particular op if it reaches a point that would have increased the peak memory footprint. Currently, follows these heuristics (subject to change or tune): - never reorders collectives relative to one another, for SPMD safety - has an option for per-collective prefetch limit, but does not enable it by default - limits the total number of reorder steps to some factor of the graph size to prevent worst-case quadratic performance Prerequisite: sink_comms_and_waits - ensure comm and wait nodes are scheduled as late as possible, respecting data dependencies. That allows reorder_communication_preserving_peak_memory to take a best case peak-memory snapshot, and then monotonically improve latency by moving collectives backward in time. Peak memory impact is computed in an iterative fashion. First, memory use at each timestep is computed, and global peak memory is computed as a max over timesteps. Then, when swapping any two adjacent nodes, only the curr-memory for the earlier of the nodes after the swap is affected. This enables checking step by step whether a swap is peak-memory-safe, and bailing out if not. Example: 0 n0 C0 1 n1 C0 + Allocs(n1) - Frees(n1) 2 n2 C0 + Allocs(n1) - Frees(n1) + Allocs(n2) - Frees(n2) 0 n0 C0 1 n2 C0 + Allocs(n2) - Frees(n2) <-- After moving n2 to Time 1, only time1 memory changes 2 n1 C0 + Allocs(n2) - Frees(n2) + Allocs(n1) - Frees(n1) zAreorder_communication_preserving_peak_memory improved overlap by z ns after z reorders. )zCollective nodezinitial exposedz final exposed improvementzlimiting factormovestabulater)r-)headersz>Please `pip install tabulate` to nicely render overlap stats.  artifactc SSS.$)N,reorder_communication_preserving_peak_memorystring)nameencodingr6r&r%>reorder_communication_preserving_peak_memory..s B  r&c>T$Nr6)reorder_log_strsr%r7r8s?r&) metadata_fn payload_fn)6_reorder_communication_preserving_peak_memory_internalr+sumr,items node_summaryinitial_exposed final_exposedlimiting_factor importlibutil find_specr-strjoinmap overlap_loginfor) r$reordered_snodes node_statssnoder+total_improvement total_movesr.node_reorder_inforowsr-r;s @r%r2r2Os@ ?vF!FPPZE*+777ZKP[I[E/[IJJGJ5%(..JGHK LL]K^_l ,G")3(8(8(: ); $E    - -  + +  ) )  - -  # #  ); ~~ ++%8     M  3w<$..499Sd^44_% +  eQIG sE9E>FAFcb\rSrSr%SrSrS\S'SrS\S'SrS\S 'S r S \S '\ S 5r Sr g) ReorderInfoz= Debug info describing how an individual snode was reordered floatrBrCNonerHrDrintr,c4URUR- $r:)rBrC)selfs r%r+ReorderInfo.improvements##d&8&888r&r6N) __name__ __module__ __qualname____firstlineno____doc__rB__annotations__rCrDr,propertyr+__static_attributes__r6r&r%rUrUsB OUM5!OS!E3N 99r&rUcP^^[U5S-nSn[U5n[Rb[Rn[[R R R55n[[R R55n[X5n[XU5upxUV s0sHo[U 5_M sn m0n U4Sjn [U5GHup[U 5(dM[5=oU 'U "XU S-S5=U lU lX!:a SU lMN[%U S- SS5GHTnXnU['SX- 5:a SU l M[U5(a S U l M[U R(Vs/sHnUR*PM sn5m[-U4S jUR/555(a5(d S U l GMXxU- XS- X- :a S U l GM/U R TU :a S U l GMMU =R2S- slUS- nXnXS-X'UXS-'XS-X- nXXS- - nXU- U-X'U "XUS-S5U lGMW GM X 4$s sn fs snf)z| Internal testing helper that also returns debug info. Returns: - reordered snodes list - dict {snode: ReorderInfo} drNc>[U5nSnUH/n[U5(aM[U5(a O UTU- nM1 [SX#- 5$)Nr)estimate_op_runtimerrmax)collective_snoderemaining_snodes comm_time compute_timerOruntimess r%exposed_communication_timeZ_reorder_communication_preserving_peak_memory_internal..exposed_communication_times['(89  %E"5))U## HUO +L&1i.//r&r z move limitrWzprefetch limitzcollective orderingc3H># UHoR5T;v M g7fr:)get_name).0o dep_namess r% I_reorder_communication_preserving_peak_memory_internal..s7O!JJLI-7Os"zdata dependencyz peak memoryzsufficient overlapping)lenr reorder_prefetch_limitr rgraph graph_inputskeysget_output_namesrrrj enumeraterrUrBrCrDrangerkunmet_dependenciesr4any get_outputsrr,)r$ MOVE_LIMITrQPER_COLLECTIVE_PREFETCH_LIMITr} graph_outputsname_to_freeable_input_buf peak_memory curr_memoryrOstatsrqi reorder_infoj prev_snodestmpj_plus_one_allocj_allocrwrps @@r%r>r>sVs"JK$'K! $$0(.(E(E%$.qww/C/C/H/H/J$KL%/0H0H0J%KMAWB 4M K@FFve*511vFH35E 0f% u % %*5- 7L<*5Q/B L (<+E(/; ,1q5"b)#Y s1a?@@3CL0&z223HL0&8P8P'Q8P18P'QR 7A7M7M7O' 333DL0Q/+!e2D{~2UU3@L0--?3KL0""a'"q i"q5M #1u #.1u#5 #F %.;1u+==!,'!9\rSrSrSUUUU4SjjrSrSrg)$_schedule_for_comm..RunnableiPc>Xl[[UR555nTUR 5nTUTUTU4Ulgr:)rOnextiterget_operation_namesrtscore)r\rOr4 fused_namename_to_fused_nodescores_0scores_1scores_2s r%__init__-_schedule_for_comm..Runnable.__init__QsTJU6689:D+D1::.Runnable.__lt__[s:: + +r&)rrON)returnrY)r^r_r`rarrre)rrrrsr%RunnablerPs   ,r&rc38# UHoRv M g7fr:)r4)rudeps r%rx%_schedule_for_comm.._sG.Fs((.Fsc>TRU5 UR5HTnTUHHnTURU5 [TU5S:XdM+[R "TT"U55 MJ MV g)zE Schedules `snode` and put all unblocked nodes onto the ready queue. rN)appendget_buffer_namesremoverzheapqheappush)rObuf_namer buffer_usersready scheduled unmet_depss r%schedule$_schedule_for_comm..scheduleosl ..0H%h/5!((2z%()Q.NN5(5/:01r&c>TVs/sH=n[UR5(aM[UR5(aM;UPM? nn[U5S:Xag[ USS9$s snf)zP Return the next node in the ready queue that's neither a collective or a wait. rNcUR$r:rxs r%r7G_schedule_for_comm..get_overlapping_candidate..sQWWr&key)rrOrrzmin)r candidatesrs r%get_overlapping_candidate5_schedule_for_comm..get_overlapping_candidatezsf &qww/ 8Eagg8N   z?a :#455 sA'A'A'c>[U5(deT"U5 TUnUS:aQT"5=nbGTRU5 T"UR5 UTUR-nUS:a T"5=nbMG[R"T5 g)z Schedules collective node `snode`, along with one or more compute nodes to overlap with it. The strategy is described in the comment of `reorder_compute_for_overlap`. rN)rrrOrheapify)rOcollective_cost candidaterrr snode_to_costs r%schedule_collective_for_overlap;_schedule_for_comm..schedule_collective_for_overlaps #5))))'. a 799F LL # Y__ % }Y__= =O a 799F  er&z;Detected unscheduled nodes. Nodes with unmet dependencies: )rrrrtsysmaxsizer ancestorsrrr rrrjr@rzrraddheappoprO)r$rrr buf_name_to_snodeidxrOrop_name node_namecomm_idxancestoranc_fused_namedepsrrrrrrrrrrrrrrs @@@@@@@@@@@@r%r"r"sL#%r2 Hh' ..0H*/h '1002G*/ w '3/45>>+,NN$ !kk!(H .u55)1HU^^% &!OO!3H!=!F!F!H+.x/G+R(, MH ZM%00)*HU^^% & , , <E zGe.F.FGGG<J E=H=TLDJKF5/66FKM!'')  t9> NN5(5/ 2C   ! !% (* I ; ; 6& e** e$** #6u#=#= +E 2 UO e**"'') 4yA~ I* V ~* Q<Ls 'K Kc [RR5(dU$UVs/sHn[U5(dMUPM nn[ S[ U55H]n[ [XER555nXES- R5HnXER[XvS95 M M_ U$s snf)z Decide global ordering of comms, by just enforcing the ordering that's in the input graph (might not be the same ordering as the eager mode program). TODO: Come up with a better approach r  mutating_buf) torch distributed is_availablerrrzrrr add_fake_depr)nodes name_to_bufrn comm_nodesrrbufs r%decide_global_ordering_of_commsrs    ) ) + + "=U&9!&s CCc[RS:XaUR5nU$[[R5(de[R"U5nU$)z2 Returns estimated op runtime in nanoseconds (ns) default)r rjget_estimated_runtimecallable)rOruntimes r%rjrjsU!!Y.--/ N223333,,U3 Nr&c UR5n[U5S:XGaASn[UR[R [R 45(aSURRS3nUR5Vs/sHo3RR5PM nnSRUVs/sHAn[U[R5(aSURSURS3OSPMC sn5nURR5nURRR UUSUSUR#5SS 3$/nUHn UR%['U 55 M URR S S RU53$s snfs snf![a SnNf=f) Nr z (),z (size=z , stride=z.0fz ns): z, ) get_nodesrz isinstancenoder ExternKernelOut_CollectiveKernelpython_kernel_nameget_output_specrILayoutsizestridemaybe_get_nameAttributeError __class__r^rrrA) rOr$detailchildlayoutslayoutout_tensor_infor summaries child_snodes r%rArAs __ F 6{a ejj2#5#5r7K7K"L M M%**778:F=B__=NO=NE::--/=NO(( &  &Ffbii00&++i a@&     113I**&&//08II;VXY^YtYtYvwzX{{A AI k23oo&& 'r$))I*>)? @@)P  I s#F16r)rKdebug)stepmsgs r%step_log#visualize_overlap..step_logsT"IRu-.r&z| zEst. runtime (ms): i)rrrjrrrArKr)ordertotal_est_runtime cur_comm_noder rrOs r%visualize_overlaprs# #M/!'   "5))!%8%??! % $$!%8%??! Tl512 4"5))!%8%??! % ,u"5!68$$,u"5!68 $ L$7#89:-(. /$6=>?r&c@Un[[RRR 55n[[RR 55n[ RGHXn[U[5(aU[5;a [5Un[U5(d SUS35e[U[X5U5upV[RR!5S:Xa)["R%SUSU<S35 ['U5 [*R*"5nU"U5n[*R*"5U- n [RR!5S:Xa(["R%S US U S 35 ['U5 [U[X5U5upV[-S U<35 GM[ U$![(an["R%SUS9 SnANSnAff=f![(an["R%SUS9 SnAN|SnAff=f) Nz3Invalid reorder_compute_and_comm_for_overlap pass: z is not callablerz.==== Visualize overlap before reordering pass z, peak_memory=z ====r)exc_infoz-==== Visualize overlap after reordering pass z (ran in z sec)====zfinal peak_memory=)r rr|r}r~rr 'reorder_for_compute_comm_overlap_passesrrHglobalsrrrrrget_rankrKrr Exceptiontimeprint) r$r r}rpr_et0ts r%$reorder_compute_and_comm_for_overlaprs E$.qww/C/C/H/H/J$KL%/0H0H0J%KM  ; ; a  !wy. ! A{{ A!DT U {. *6@-      % % '1 ,   @?k^SXY  2!%(YY[% IIK"     % % '1 ,   ?s)A3iX  2!%(. *6@-   #{n%&?<@ L# 2!!"q!1 2 2!!"q!1 2s0 G  G5 G2G--G25 H?HHc ^ ^^^^^^[UR5m[[5m[[5m[T5HupURS:XdMUR [ RRRR:XdMUURSRS:XdSUSURSS35eURSnURSnUS:aTURU5 MTURU5 M UUU4Sjn[[5n[T5HupURS:XdMUR [ RRRR:XdMUUnURSmTRS:XdS TS US35eU"T5(dMUTRU5 M S nS mTHnURS:XdM[UR [ R R"5(dMJUR R$R&(dMqU"U5(aMT"X&R)55(dMS US35e UR+5GH3umn [U 5GHupTU nURSTLdeURun mU S-n U [-U 5S- :aXS-O [-T5S- nTXn[/UU4SjU55(aSTSUSUS35eUHnURS:XdMTUR;dM'UR [ RRRR:wdMe[1UU4SjUR55nUUl M GM GM6 UR+5H0umn [U 5HupTU nUR3U5 M M2 THynURS:XdMUR [ RRRR:XdMSURSU;dMhUR3U5 M{ g)ab This FX graph pass replaces uses of FSDP2 unsharded params with their corresponding graph intermediates that were fsdp.copy_ into the unsharded params in the original graph. NOTE: Can only apply this pass to any of the FSDP2 unsharded params that have this pattern (or repetition of): `resize_(full) -> copy_ -> resize_(0)`. Because of this, for partial-graph case where `resize_(full) -> copy_` is in one graph and `resize_(0)` is in another graph, we can't remove these resize and copy ops and thus we will have worse performance there. In other words, "do we try to remove all the resize_(full) -> copy_ -> resize_(0) nodes for this unsharded param" is actually a per-unsharded-param decision, since for each unsharded param, we look at its resize sequence pattern (in `check_resize_pattern()`) to determine if its set of resize and copy nodes can be removed. call_functionr placeholderz1Resize can only operate on graph inputs, but got z# which is resizing non-graph-input r/r cn>TRU/5nTRU/5n[U5[U5:Xd2[RSUS[U5S[U5S35 g[ X5H7up4X4:dM [RSUSTUSUS TUSUS 3 5 g g ) NzH Unequal number of resize-to-full and resize-to-0 nodes for graph input z: z vs. zK. Skipping `remove_fsdp2_unsharded_param_graph_input_usage` FX graph pass. Fz For graph input z: resize-to-full node z at index z happens after resize-to-0 node zd. Skipping `remove_fsdp2_unsharded_param_graph_input_usage` FX graph pass for that unsharded param. T)getrzlogwarningzip) graph_inputresized_to_full_idxesresized_to_0_idxesresize_to_full_idxresize_to_0_idx&graph_input_to_resized_to_0_node_idxes)graph_input_to_resized_to_full_node_idxes node_lists r%check_resize_patternLremove_fsdp2_unsharded_param_graph_input_usage..check_resize_patternYs !J M M ! DGG UWX()S1C-DD KKHHS}UE#&8"9!:; 47 !4 / "4  3I>P4Q3RR\]o\pq )/ :;:oEVW4 r&z\ Assumed all FSDP2 `unsharded_param`s to be graph input, but it's not true! Offending node: z . Graph: cUR[RRRR :H=(d; UR[RR RR :H$r:)targetropsfsdpcopy_rinductorresize_storage_bytes_)rs r%is_allowed_mutationKremove_fsdp2_unsharded_param_graph_input_usage..is_allowed_mutationsO KK599>>//77 7 O{{eii00FFNNN r&c [UR[RR5(aj[ URR R5VVs/sH3up#URcMURR(dM1UPM5 snnO/n[UVs/sH6n[URURSR55PM8 sn5n[UVs/sH)n[URSR55PM+ sn5n[XW-5S:$s snnfs snfs snf)Nvalr)rr0r_ops OpOverloadr_schema arguments alias_infois_writer r argsmetauntyped_storagerz)runsharded_paramsrrmutated_arg_idxesmutated_node_arg_storagesunsharded_paramstorages_of_unsharded_paramss r%-is_node_mutating_unsharded_param_or_its_aliaseremove_fsdp2_unsharded_param_graph_input_usage..is_node_mutating_unsharded_param_or_its_aliass)$++uzz'<'<==&dkk&9&9&C&CD DDA<<01 0E0ED   %/+ *Atyy|007GGIJ* % ! (2(8 '7O33E:JJLM'7 ( $ ,KLqPP)    s D=7D=D=)=E60EzdUser mutation on FSDP2 unsharded param is not allowed when Traceable FSDP2 is used. Violating node: c38># UHnT"UT/5v M g7fr:r6)rurrHrFs r%rxAremove_fsdp2_unsharded_param_graph_input_usage..s%*D>d_DUVV*sz(Assumed no ops mutating unsharded param z in subgraph z, but it's not true! Graph: c36># UHnUTLaTOUv M g7fr:r6)ruarg replacementrFs r%rxrKs$%#,C(+o'= 3F#,sN)listrrropr0rr1r4r5rr@rr2r3rr:r;r< is_mutabler~r@rzrtuple erase_node)r|rrr%new_sizer-'unsharded_param_to_fsdp_copy_node_idxesfsdp_copy_noder6fsdp_copy_node_idxesrfsdp_copy_node_idxrsubgraph_start_idxsubgraph_end_idxsubgraph_nodesnew_argsr*r+rHr,rNrFs @@@@@@r%.remove_fsdp2_unsharded_param_graph_input_usager]7sbU[[!I1*y)  GG & uyy11GGOOO99Q> ##...'--DBBD eeidjk : 1 6 6 8 %./C%D !A&'9:N!&&q)_< <<+00NA{!3a!7 s/0144%U+^a'  ''9KN* ))8(9~FVW w 'GG.'4994 uyy'9'9'O'O'W'WW$%#'99% H!)DI')&E 9J 1 6 6 8 %./C%D !A&'9:N   ^ ,&E 9  GG & uyy11GGOOO ! GG   T " r&c@^ SSKm T RR5(deT RRR (a%T RRR (deSSK J nJ nJ nJnJn U 4SjnU"5nU"U"T RRR R U"["R$U"T RR&R(R U"S5U"S5U"S5U"S5U"S 5U"S 5U"S 5U"S 5U"S 55 U"S55U"S5U"S55USS9SU 4Sjj5nU"U5 UR+U5 g![[[4a gf=f)Nrr ) CallFunction KeywordArgMatchPatternMatcherPassregister_graph_patterncZ>[UR5nUHnUR[R:XdM#UR SRTR RRRLdMiUR SS:XdM~URU5 M g)Nrr ) rOrr0operatorgetitemr@r1r2all_gather_copy_inrrS)gr,rrs r%remove_unused_getitem8reinplace_fsdp_all_gather..remove_unused_getitem spM AH,,,FF1I$$ (I(I(Q(QQFF1IN Q r&all_gather_inputsinp_split_sizesall_gather_input_numel world_sizerankdtypedevicegroup_name_inner"allocate_memory_from_process_groupitem_idx group_size group_namec&URSS:H$)Nrtr)kwargs)matchs r%r7+reinplace_fsdp_all_gather..Ds%,,z":a"?r&) pass_dict extra_checkc>U4SjnURUUSUSUSUSUSUSUSUS US US US / 5 g) Nc>USSnUSnUSnTRRRR"U6nUSnUSnTRRR RXRX6S9nU$)NrWrr )out)r1r2rgr_c10d_functionalall_gather_into_tensor_out) r@ copy_in_argsrurvrgrf getitem_1all_gather_into_tensorrs r%replEreinplace_fsdp_all_gather..reinplace_all_gather..replGs 9LbJbJ!&!B!B!J!J" )+G*1-I **EEMMN # * )r&rkrlrmrnrorprqrrrsrurv)replace_by_example)ryr@rxrrs r%reinplace_all_gather7reinplace_fsdp_all_gather..reinplace_all_gather-s{4 *$   *+()/0|$vwx )*;<|$|$  r&)ryra)5torch.distributed.fsdp._fully_shard._fsdp_collectivesrrr1rrr ImportErrorrAssertionErrorpattern_matcherr_r`rarbrcrrerfr2rgapply) r|r_r`rarbrcri graph_passrrs @r%reinplace_fsdp_all_gatherrs D  --//// II & & = = **EE  FE   $%J II & & = = E E   IINN55==230178|,v&w'x(12CD :&  | $ | $' *?/2" 32" H% UM  8sA1FFFc[U[RRR[RRR 45(ae[ UR5SS5$)N)rr _inductor schedulerFusedSchedulerNodeGroupedSchedulerNoderZrt)rOs r% get_op_idxrnsb OO % % 8 8 OO % % : :   u~~# $$r&c  ^^^ ^!SSKJm /n[["5nSnSn0n0n0m!U U!4Sjn UGH"n [ U R [ RRRRS9(Ga[U4SjU R55(GaSnU n [5n [U U UT5 [[ RRRR[ RRRR[ RRR R/5m[#U U UTUU 4SjS 9 [%U S S 9n ['U 5nS n[)['U 55H^nU Un[+UR [ RRR R5(aUS- nUS:dM\Un O U SUn Sn[)['U 5S- 5H9n[-U US-R [.R05(dM4US-n O UceU "U SU5nU "U US5nUUU'GM;[+U R [ RRR2R5(dGMSnU n[5n[#UUUT5 [%US S 9nSn[)['U5S- 5H9n[-UUS-R [.R05(dM4US-n O UceU "USU5nU "UUS5nUUU'GM% ['T!5S :deU(a['U5S :deU(a['U5S :deUHSn U R55T!;aT!U R55n X;aM1UR7U 5 UR9U 5 MU SnUR;5HkunnUb`[=[?URA555nURC5H*nURE[GUR55US95 M, UnMm SnUR;5HkunnUb`[=[?URA555nURC5H*nURE[GUR55US95 M, UnMm U$)Nr )rFc>TRRU5nUHnUTUR5'M UTUR5'U$r:)rcreatert)snodes_to_group group_noderOrsnode_name_to_final_snodes r%_create_group_node:enforce_comm_ordering_for_fsdp.._create_group_nodesO33::?K $E:D %enn&6 7%;E!*"5"5"78r&)rPc3># UHJn[TUR[RRR R 5v ML g7fr:)rrrr1r2rgr)rurrs r%rx1enforce_comm_ordering_for_fsdp..sJ % "1%**EIINN,M,M,U,U  %sAATc>[UTR5=(d6 [UTR5=(a URRT;(+$r:)rNopKernelSchedulerNodeExternKernelSchedulerNoder op_overload)r allowed_opsrs r%r70enforce_comm_ordering_for_fsdp..sDq)"B"BC"1i&I&IJ>FF..+= 'r&) criteria_cbc[U5$r:rrs r%r7rJqMr&rrc[U5$r:rrs r%r7rrr&r)$rrr rrrrr1rrrrrr wait_tensorr2split_with_sizes_copyrsortedrzrrrr _WaitKernel chunk_catrtrrr@rrrrrr)"r$rr new_orderr ag_exists rs_exists$ag_grouped_node_to_wait_grouped_node$rs_grouped_node_to_wait_grouped_noderrOag_snodeag_related_snode_setag_related_snodesend_idx_of_current_ag_blockcopy_out_countr cur_snode wait_node_idx ag_group_nodeag_wait_group_noders_snoders_related_snode_setrs_related_snodes rs_group_noders_wait_group_node prev_ag_waitwait_group_noderrv prev_rs_waitrrrs" ` @@@r%enforce_comm_ordering_for_fsdprys )+I3!III+-(+-( "  JJ59955PPXX   __    IHLVL  ($"  %II..IIQQII..::BBIINN88@@K )$" !'$*A!  +..?*@ 'N3012-a0 !NNEIINN$H$H$P$P#a'N!A%23/3!22N3N O !M301A56/A6;;R^^LL$%EM7!, ,,./@-/PQM"44Emn4U!V BT 0 ?EJJ (@(@(H(H I IIHMWL ($"  !'$*A!  !M301A56/A6;;R^^LL$%EM7!, ,,./@-/PQM"44Emn4U!V BT 0 ?]` ( )A -- -781<<<781<<< >> 8 8-enn.>?E    e L*N*T*T*V&   #]%C%C%E FGL!--/**AJJL|D0' +WL*N*T*T*V&   #]%C%C%E FGL!--/**AJJL|D0' +W r&)r$list[BaseSchedulerNode]rr)r$rrzDtuple[list[BaseSchedulerNode], dict[BaseSchedulerNode, ReorderInfo]]) r$rrboolrrr rrr)rrrr)rOrrrX)r|torch.fx.Graph)r|rrrY)r$1list[torch._inductor.scheduler.BaseSchedulerNode]rz4dict[str, torch._inductor.scheduler.SchedulerBuffer]rzdict[str, BaseSchedulerNode]rr)A __future__rrrEloggingrerr collectionsr dataclassesrtypingrrrtorch._loggingr torch.multiprocessing.reductionsr torch.utils._ordered_setr rr r dependenciesrmemoryrrrutilsrrrrrrr virtualizedr getLoggerr^r"_logginggetArtifactLoggerrKtorch._inductor.schedulerrrrr)r2rUr>r"rrjrArrr]rrrr6r&r%rs#  #!% +;/!UU!nn..xC ; #&T #TTn  9 9  9V #VIVrW #WWW W  Wt ", A:#L& #&&RA#Hpf%n =nEn5n7 nr&