Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

POC: Implement from_json_to_structs using get_json_object kernel #2449

Draft
wants to merge 67 commits into
base: branch-24.10
Choose a base branch
from
Draft

POC: Implement from_json_to_structs using get_json_object kernel #2449

wants to merge 67 commits into from

Commits on Aug 14, 2024

  1. Rename function and class containing from_json 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    2459f61 View commit details
    Browse the repository at this point in the history

  2. Fix copyright year 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    3b1e067 View commit details
    Browse the repository at this point in the history

  3. Fix style 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    1959b87 View commit details
    Browse the repository at this point in the history

  4. Rename parameter 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    2f83c57 View commit details
    Browse the repository at this point in the history

  5. Change parameter type 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    8f1da4b View commit details
    Browse the repository at this point in the history

  6. Fix style 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    515bb6b View commit details
    Browse the repository at this point in the history

  7. Exclude the new files from compiling 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    c7d5ad9 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2024

  1. WIP 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    d9e52c0 View commit details
    Browse the repository at this point in the history

  2. Change signatures for from_json_to_structs 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 16, 2024
    Configuration menu
    Copy the full SHA
    e4bb460 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Merge branch 'branch-24.10' into from_json

    @ttnghia
    ttnghia committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    fe2bf9f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'branch-24.10' into from_json

    @ttnghia
    ttnghia committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    3412195 View commit details
    Browse the repository at this point in the history

  3. Add convert_schema_to_paths function and fix compile error 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    9e239cb View commit details
    Browse the repository at this point in the history

  4. Merge branch 'branch-24.10' into from_json 

    # Conflicts:
#	src/main/cpp/CMakeLists.txt
#	src/main/cpp/src/JSONUtilsJni.cpp
#	src/main/cpp/src/from_json.hpp
#	src/main/java/com/nvidia/spark/rapids/jni/JSONUtils.java
#	src/main/java/com/nvidia/spark/rapids/jni/MapUtils.java
    @ttnghia
    ttnghia committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    7d5653c View commit details
    Browse the repository at this point in the history

  5. Convert schema into paths

    @ttnghia
    ttnghia committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    187785b View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. WIP

    @ttnghia
    ttnghia committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    a02cfca View commit details
    Browse the repository at this point in the history

  2. Add keep_quotes parameter 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    49bbf7c View commit details
    Browse the repository at this point in the history

  3. Find decimal columns in schema 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    14b463e View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Allow to specify allow_leading_zero_numbers 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    ae1df4a View commit details
    Browse the repository at this point in the history

  2. Fix order when checking keep_quotes 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    1a96dd5 View commit details
    Browse the repository at this point in the history

  3. Allow to specify allow_non_numeric_numbers 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    41304a4 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2024

  1. WIP for supporting structs 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    5ba1506 View commit details
    Browse the repository at this point in the history

Commits on Sep 5, 2024

  1. Merge branch 'branch-24.10' into from_json

    @ttnghia
    ttnghia committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    79398b1 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Merge branch 'branch-24.10' into from_json

    @ttnghia
    ttnghia committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    e7a64d6 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Support struct type in schema 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    4376438 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. Apply null mask for structs 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    7c5e3ec View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2024

  1. Fix column order 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 14, 2024
    Configuration menu
    Copy the full SHA
    4127040 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Add from_json_object kernel 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    97cd60b View commit details
    Browse the repository at this point in the history

  2. Add type_ids data for paths 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    357c671 View commit details
    Browse the repository at this point in the history

  3. Fix struct null mask 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    2896875 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Add Java test 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    ba3649d View commit details
    Browse the repository at this point in the history

  2. Fix character matching 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    82ad2d5 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'fix_match_escaped' into from_json

    @ttnghia
    ttnghia committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    2c2adfc View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Fix column order in schema 

    Signed-off-by: Nghia Truong <[email protected]>
    @ttnghia
    ttnghia committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    bff6d12 View commit details
    Browse the repository at this point in the history

  2. Allow STRUCT type in JSON path

    @ttnghia
    ttnghia committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    1abd8f8 View commit details
    Browse the repository at this point in the history

  3. Add test with LIST

    @ttnghia
    ttnghia committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    a18ee48 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Output LIST as string

    @ttnghia
    ttnghia committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    adc45be View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2024

  1. Update tests and fix null mask

    @ttnghia
    ttnghia committed Sep 21, 2024
    Configuration menu
    Copy the full SHA
    77128eb View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Output list without outer brackets

    @ttnghia
    ttnghia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a78bb72 View commit details
    Browse the repository at this point in the history

  2. Output columns following the input schema

    @ttnghia
    ttnghia committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    9f13aed View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Fix struct schema, and add test

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    76a44da View commit details
    Browse the repository at this point in the history

  2. Add test

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    1f36d17 View commit details
    Browse the repository at this point in the history

  3. Fix struct child of array

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    af97f21 View commit details
    Browse the repository at this point in the history

  4. Implement element delimiter

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    b14c332 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'branch-24.10' into from_json 

    # Conflicts:
#	thirdparty/cudf
#	thirdparty/cudf-pins/versions.json
    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    233c6b5 View commit details
    Browse the repository at this point in the history

  6. Cleanup

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    54a8e96 View commit details
    Browse the repository at this point in the history

  7. Add isNullOrEmpty function

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    705dd6b View commit details
    Browse the repository at this point in the history

  8. Cleanup

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    a3bb86c View commit details
    Browse the repository at this point in the history

  9. Change memory budget

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    3edfb0d View commit details
    Browse the repository at this point in the history

  10. Add test

    @ttnghia
    ttnghia committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    92c4308 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Search for both element delimiter and null placeholder

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    395c0bc View commit details
    Browse the repository at this point in the history

  2. Fix null in array

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    a04183c View commit details
    Browse the repository at this point in the history

  3. Cleanup

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    1cad1f5 View commit details
    Browse the repository at this point in the history

  4. Add test

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    aaa7d89 View commit details
    Browse the repository at this point in the history

  5. Fix null count equal size

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3b4c1c2 View commit details
    Browse the repository at this point in the history

  6. Fix struct under list

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    f8bf8dd View commit details
    Browse the repository at this point in the history

  7. Cleanup

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    d344eec View commit details
    Browse the repository at this point in the history

  8. Remove redundant path

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3c68814 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'branch-24.10' into from_json

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    7910633 View commit details
    Browse the repository at this point in the history

  10. Fix parsing non-numeric number

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    c175583 View commit details
    Browse the repository at this point in the history

  11. Add allow_unquoted_control_chars option

    @ttnghia
    ttnghia committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    0f04032 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Fix output array/struct

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    3feefae View commit details
    Browse the repository at this point in the history

  2. Add test

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    feeeffc View commit details
    Browse the repository at this point in the history

  3. Fix empty list

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    a663606 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'branch-24.10' into from_json

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    dd60c81 View commit details
    Browse the repository at this point in the history

  5. Add test

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    de53ef0 View commit details
    Browse the repository at this point in the history

  6. Fix struct under list

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    2407dec View commit details
    Browse the repository at this point in the history

  7. Cleanup

    @ttnghia
    ttnghia committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    571d692 View commit details
    Browse the repository at this point in the history