Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Improve multi-GPU BFS performance #4619

Draft
wants to merge 95 commits into
base: branch-24.10
Choose a base branch
from
Draft

[WIP] Improve multi-GPU BFS performance #4619

wants to merge 95 commits into from

Commits on Jul 15, 2024

  1. add a create_graph_from_edgelist function that takes edge list in mul… 

    …tiple chunks
    @seunghwak
    seunghwak committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a0d1f01 View commit details
    Browse the repository at this point in the history

  2. update R-mat graph generators to generate edge list in multiple chunks

    @seunghwak
    seunghwak committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    55513ae View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-24.08' of https://github.com/rapidsai/cugraph in… 

    …to enh_graph_creation
    @seunghwak
    seunghwak committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    2163fd8 View commit details
    Browse the repository at this point in the history

  4. fix build error

    @seunghwak
    seunghwak committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a9dfb92 View commit details
    Browse the repository at this point in the history

  5. delete unused functions

    @seunghwak
    seunghwak committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    e7b33ca View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. fix build errors

    @seunghwak
    seunghwak committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    27ea550 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2024

  1. add temporary performance measurement code

    @seunghwak
    seunghwak committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    e5e8257 View commit details
    Browse the repository at this point in the history

  2. add code to broadcast frontier using a bitmap

    @seunghwak
    seunghwak committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    7ec5b08 View commit details
    Browse the repository at this point in the history

  3. resolve merge conflicts

    @seunghwak
    seunghwak committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    d6123ba View commit details
    Browse the repository at this point in the history

  4. fix build error

    @seunghwak
    seunghwak committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    81f51c1 View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2024

  1. update dataframe buffer utilities

    @seunghwak
    seunghwak committed Jul 21, 2024
    Configuration menu
    Copy the full SHA
    69cb4f9 View commit details
    Browse the repository at this point in the history

  2. reduce # resizes

    @seunghwak
    seunghwak committed Jul 21, 2024
    Configuration menu
    Copy the full SHA
    6adcccb View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. remove debug statement

    @seunghwak
    seunghwak committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    bfe21fc View commit details
    Browse the repository at this point in the history

  2. rename VertexFrontierBucketType to KeyBucketType

    @seunghwak
    seunghwak committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    446435b View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. update per_v_transform_reduce_incoming|outgoing_e to support reduce_o… 

    …p::any
    @seunghwak
    seunghwak committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    df463ce View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. update kernels to take KeyIterator key_first & key_last

    @seunghwak
    seunghwak committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    222148d View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2024

  1. update per_v_transform_reduce_incoming_outgoing_e to support key list

    @seunghwak
    seunghwak committed Jul 28, 2024
    Configuration menu
    Copy the full SHA
    effc69c View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. remove pred_op.cuh

    @seunghwak
    seunghwak committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    d537290 View commit details
    Browse the repository at this point in the history

  2. update per_v_transform_reduce_incoming|outgoing_e to take a predicate

    @seunghwak
    seunghwak committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    cf92885 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. Merge branch 'branch-24.10' of https://github.com/rapidsai/cugraph in… 

    …to enh_bfs_mg
    @seunghwak
    seunghwak committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    b8d846c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.10' of https://github.com/rapidsai/cugraph in… 

    …to enh_bfs_mg
    @seunghwak
    seunghwak committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    50ffc33 View commit details
    Browse the repository at this point in the history

  3. split per_v_transform_reduce_incoming_outgoing_e implementation to tw… 

    …o files
    @seunghwak
    seunghwak committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    a6476b9 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2024

  1. implement per_v_transform_reduce_if_incoming|outgoing_e

    @seunghwak
    seunghwak committed Aug 10, 2024
    Configuration menu
    Copy the full SHA
    ec24758 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. update BFS to use per_v_transform_reduce_if_outoging_e

    @seunghwak
    seunghwak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    df751e7 View commit details
    Browse the repository at this point in the history

  2. file rename

    @seunghwak
    seunghwak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4661b9b View commit details
    Browse the repository at this point in the history

  3. remove transform_reduce_v_frontier_outgoing_e_by_src (this can be bet… 

    …ter supported by per_v_transform_reduce_if_outgoing_e)
    @seunghwak
    seunghwak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    0951741 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'branch-24.10' of https://github.com/rapidsai/cugraph in… 

    …to enh_bfs_mg
    @seunghwak
    seunghwak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    796b928 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. code cleanup, add few FIXMEs to improve performance, and add performa… 

    …nce measurement code
    @seunghwak
    seunghwak committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    7b98e3a View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. performance tuning for BFS

    @seunghwak
    seunghwak committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    3f77ee1 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2024

  1. add a utility to find iteator type in dataframe buffer

    @seunghwak
    seunghwak committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    bb75771 View commit details
    Browse the repository at this point in the history

  2. minor performance tuning

    @seunghwak
    seunghwak committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    cfce7bc View commit details
    Browse the repository at this point in the history

Commits on Aug 18, 2024

  1. delete unused code

    @seunghwak
    seunghwak committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    75d6151 View commit details
    Browse the repository at this point in the history

  2. add an option to skip edge shuffling in R-mat edge list generation

    @seunghwak
    seunghwak committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    0f88988 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'branch-24.10' of https://github.com/rapidsai/cugraph in… 

    …to enh_bfs_mg
    @seunghwak
    seunghwak committed Aug 18, 2024
    Configuration menu
    Copy the full SHA
    180ece1 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. fix build error

    @seunghwak
    seunghwak committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    2efb51e View commit details
    Browse the repository at this point in the history

  2. fix documentation error

    @seunghwak
    seunghwak committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    106a6ad View commit details
    Browse the repository at this point in the history

  3. add a query function

    @seunghwak
    seunghwak committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    98419cb View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Merge branch 'branch-24.10' of https://github.com/rapidsai/cugraph in… 

    …to enh_bfs_mg
    @seunghwak
    seunghwak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    8193a91 View commit details
    Browse the repository at this point in the history

  2. bug fix

    @seunghwak
    seunghwak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    9625e0c View commit details
    Browse the repository at this point in the history

  3. bug fix

    @seunghwak
    seunghwak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    03cfe0d View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. bug fixes

    @seunghwak
    seunghwak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    c15305f View commit details
    Browse the repository at this point in the history

  2. bug fix

    @seunghwak
    seunghwak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    4a1f150 View commit details
    Browse the repository at this point in the history

  3. replace offset vector communication with local computing

    @seunghwak
    seunghwak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    29b6834 View commit details
    Browse the repository at this point in the history

  4. add tmp perf measurement code

    @seunghwak
    seunghwak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    fcc75e0 View commit details
    Browse the repository at this point in the history

  5. map GPUs on minor_comm to consecutive GPUs

    @seunghwak
    seunghwak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    710eb88 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. additional performance tuning

    @seunghwak
    seunghwak committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    d040110 View commit details
    Browse the repository at this point in the history

  2. add a utility function

    @seunghwak
    seunghwak committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    ca816dd View commit details
    Browse the repository at this point in the history

  3. fix build error

    @seunghwak
    seunghwak committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    7712c38 View commit details
    Browse the repository at this point in the history

  4. fix build error

    @seunghwak
    seunghwak committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    31a5955 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. bug fix

    @seunghwak
    seunghwak committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    ac33784 View commit details
    Browse the repository at this point in the history

  2. perf experiment

    @seunghwak
    seunghwak committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    6d8c7ef View commit details
    Browse the repository at this point in the history

  3. perf measurement code update

    @seunghwak
    seunghwak committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    6bcdbe7 View commit details
    Browse the repository at this point in the history

  4. rename [vertex_first, vertex_last) in fill|update_edge_src|dst_proper… 

    …ty to [sorted_unique_vertex_first,sorted_unique_vertex_last) and require them to be sorted.
    @seunghwak
    seunghwak committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    3a950a5 View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. update fill|update_edge_minor_property to optionally use bitmap to br… 

    …oadcast vertex list
    @seunghwak
    seunghwak committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    d27a5e3 View commit details
    Browse the repository at this point in the history

  2. add missing includes

    @seunghwak
    seunghwak committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    97022f5 View commit details
    Browse the repository at this point in the history

  3. specialization for bool

    @seunghwak
    seunghwak committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    ecf76f8 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. add asynchronous copy_if

    @seunghwak
    seunghwak committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    350f17e View commit details
    Browse the repository at this point in the history

  2. fix implicit synchronization in multi-stream execution

    @seunghwak
    seunghwak committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    93f726f View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. fix implicit synchronizations for multi-stream execution

    @seunghwak
    seunghwak committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    d022c30 View commit details
    Browse the repository at this point in the history

  2. delete debug prints

    @seunghwak
    seunghwak committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    e53b3b8 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. fix erroneous comments (numbering)

    @seunghwak
    seunghwak committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    b6e4f28 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. reduce memory footprint in graph creation

    @seunghwak
    seunghwak committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    be504cc View commit details
    Browse the repository at this point in the history

  2. undo temporary change for benchmarking

    @seunghwak
    seunghwak committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    3b151e0 View commit details
    Browse the repository at this point in the history

  3. update comments

    @seunghwak
    seunghwak committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    ad0c879 View commit details
    Browse the repository at this point in the history

  4. cosmetic updates

    @seunghwak
    seunghwak committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    ce4ea93 View commit details
    Browse the repository at this point in the history

  5. resolve merge conflicts

    @seunghwak
    seunghwak committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    743ebf1 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. update renumbering to use binning in more places

    @seunghwak
    seunghwak committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    9445027 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.10' of https://github.com/rapidsai/cugraph in… 

    …to enh_mem_footprint
    @seunghwak
    seunghwak committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    bea1498 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'upstream_pr4642' into enh_bfs_mg

    @seunghwak
    seunghwak committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    70b1108 View commit details
    Browse the repository at this point in the history

  4. update detail::extract_transform_v_frontier_e to use multiple CUDA st… 

    …reams
    @seunghwak
    seunghwak committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    28641a6 View commit details
    Browse the repository at this point in the history

  5. exec_policy=>exec_policy_nosync

    @seunghwak
    seunghwak committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    05df778 View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2024

  1. performance-tweak detail::extract_transform_v_frontier_e

    @seunghwak
    seunghwak committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    5c4e3bd View commit details
    Browse the repository at this point in the history

  2. update comments

    @seunghwak
    seunghwak committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    dc44a7d View commit details
    Browse the repository at this point in the history

  3. improve stream concurrency

    @seunghwak
    seunghwak committed Sep 4, 2024
    Configuration menu
    Copy the full SHA
    ebcbfb7 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. update copy_if_nosync to take a pointer to store the counter

    @seunghwak
    seunghwak committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    3652c33 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. temporary paramter setting for benchmarking

    @seunghwak
    seunghwak committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    c689e35 View commit details
    Browse the repository at this point in the history

  2. bug fix

    @seunghwak
    seunghwak committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    f7b061b View commit details
    Browse the repository at this point in the history

  3. add sun_nosync for multi stream execution

    @seunghwak
    seunghwak committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    20e1c74 View commit details
    Browse the repository at this point in the history

  4. pre-filter keys

    @seunghwak
    seunghwak committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    9fa4fb4 View commit details
    Browse the repository at this point in the history

  5. multi-stream execution

    @seunghwak
    seunghwak committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    b6a1fb0 View commit details
    Browse the repository at this point in the history

  6. more performance logs

    @seunghwak
    seunghwak committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    3f71304 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. update logging

    @seunghwak
    seunghwak committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    3577699 View commit details
    Browse the repository at this point in the history

  2. use global comm to shuffle in compute_renumber_map (to avoid P2P buff… 

    …er initilaization in subcomm)
    @seunghwak
    seunghwak committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    42c4d0b View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. reduce small memory allocations

    @seunghwak
    seunghwak committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    2557668 View commit details
    Browse the repository at this point in the history

  2. bug fix

    @seunghwak
    seunghwak committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    0381f22 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. temporarily store vertex IDs in 48 bit to cut peak memory usage

    @seunghwak
    seunghwak committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    eb822da View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2024

  1. update v_list bitmap bcast

    @seunghwak
    seunghwak committed Sep 14, 2024
    Configuration menu
    Copy the full SHA
    a067f08 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2024

  1. undo a flag

    @seunghwak
    seunghwak committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    6c9118e View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. peak memory usage

    @seunghwak
    seunghwak committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    20721e6 View commit details
    Browse the repository at this point in the history

  2. use approximation in swithcing between topdown & bottomup

    @seunghwak
    seunghwak committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    9d002c5 View commit details
    Browse the repository at this point in the history

  3. update logging

    @seunghwak
    seunghwak committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    9e3574e View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. peak memory usage

    @seunghwak
    seunghwak committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    07749f4 View commit details
    Browse the repository at this point in the history

  2. improve logging

    @seunghwak
    seunghwak committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    4ddd0a1 View commit details
    Browse the repository at this point in the history

  3. NCCL bug workaround

    @seunghwak
    seunghwak committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    3bb6602 View commit details
    Browse the repository at this point in the history

  4. temporary parameter tweaks for testing

    @seunghwak
    seunghwak committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    8be2a3f View commit details
    Browse the repository at this point in the history