Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade Datafusion 40 #771

Merged
merged 42 commits into from
Jul 31, 2024
Merged

Upgrade Datafusion 40 #771

merged 42 commits into from
Jul 31, 2024

Commits on Jul 25, 2024

  1. chore: update datafusion deps

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    8d3a215 View commit details
    Browse the repository at this point in the history

  2. feat: impl ExecutionPlan::static_name() for DatasetExec 

    This required trait method was added upstream [0] and recommends to simply forward to `static_name`.

[0]: apache/datafusion#10266
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    0179c6f View commit details
    Browse the repository at this point in the history

  3. feat: update first_value and last_value wrappers. 

    Upstream signatures were changed for the new new `AggregateBuilder` api [0].

This simply gets the code to work. We should better incorporate that API into `datafusion-python`.

[0] apache/datafusion#10560
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    61f5ea3 View commit details
    Browse the repository at this point in the history

  4. migrate count to UDAF 

    Builtin Count was removed upstream.

TBD whether we want to re-implement `count_star` with new API.

Ref: apache/datafusion#10893
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    86d1ad0 View commit details
    Browse the repository at this point in the history

  5. migrate approx_percentile_cont, approx_distinct, and approx_median to… 

    … UDAF

Ref: approx_distinct apache/datafusion#10851
Ref: approx_median apache/datafusion#10840
Ref: approx_percentile_cont and _with_weight apache/datafusion#10917
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    f4a0828 View commit details
    Browse the repository at this point in the history

  6. migrate avg to UDAF 

    Ref: apache/datafusion#10964
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    3a277a8 View commit details
    Browse the repository at this point in the history

  7. migrage corr to UDAF 

    Ref: apache/datafusion#10884
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    98498e9 View commit details
    Browse the repository at this point in the history

  8. migrate grouping to UDAF 

    Ref: apache/datafusion#10906
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    f1717a2 View commit details
    Browse the repository at this point in the history

  9. add alias mean for UDAF avg

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    c790454 View commit details
    Browse the repository at this point in the history

  10. migrate stddev to UDAF 

    Ref: apache/datafusion#10827
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d0018ea View commit details
    Browse the repository at this point in the history

  11. remove rust alias for stddev 

    The python wrapper now provides stddev_samp alias.
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d0ae202 View commit details
    Browse the repository at this point in the history

  12. migrage var_pop to UDAF 

    Ref: apache/datafusion#10836
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    40d9d3e View commit details
    Browse the repository at this point in the history

  13. migrate regr_* functions to UDAF 

    Ref: apache/datafusion#10898
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    86d9d9b View commit details
    Browse the repository at this point in the history

  14. migrate bitwise functions to UDAF 

    The functions now take a single expression instead of a Vec<_>.

Ref: apache/datafusion#10930
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    8881e2a View commit details
    Browse the repository at this point in the history

  15. add missing variants for ScalarValue with todo

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    c754d82 View commit details
    Browse the repository at this point in the history

  16. fix typo in approx_percentile_cont

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    e3a4a7a View commit details
    Browse the repository at this point in the history

  17. add distinct arg to count

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    9b2f63b View commit details
    Browse the repository at this point in the history

  18. comment out failing test 

    `approx_percentile_cont` is now returning a DoubleArray instead of an IntArray.

This may be a bug upstream; it requires further investigation.
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    6539c0c View commit details
    Browse the repository at this point in the history

  19. update tests to expect lowercase sum in query plans 

    This was changed upstream.

Ref: apache/datafusion#10831
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    11e601f View commit details
    Browse the repository at this point in the history

  20. update ScalarType data_type map

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    77b24e3 View commit details
    Browse the repository at this point in the history

  21. add docs dependency pickleshare

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    3b54873 View commit details
    Browse the repository at this point in the history

  22. re-implement count_star

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1d1cd84 View commit details
    Browse the repository at this point in the history

  23. lint: ruff python lint

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    e6775a3 View commit details
    Browse the repository at this point in the history

  24. lint: rust cargo fmt

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    8ca3469 View commit details
    Browse the repository at this point in the history

  25. include name of window function in error for find_window_fn

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a521310 View commit details
    Browse the repository at this point in the history

  26. refactor find_window_fn for debug clarity

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    06e2704 View commit details
    Browse the repository at this point in the history

  27. search default aggregate functions by both name and aliases 

    The alias list no longer includes the name of the function.

Ref: apache/datafusion#10658
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    df2cbad View commit details
    Browse the repository at this point in the history

  28. fix markdown in find_window_fn docs

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a8a6c9d View commit details
    Browse the repository at this point in the history

  29. parameterize test_window_functions 

    `first_value` and `last_value` are currently failing and marked as xfail.
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a38aa99 View commit details
    Browse the repository at this point in the history

  30. add test ids to test_simple_select tests marked xfail

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    b6eee28 View commit details
    Browse the repository at this point in the history

  31. update find_window_fn to search built-ins first 

    The behavior of `first_value` and `last_value` UDAFs currently does not match the built-in behavior.
This allowed me to remove `marks=pytest.xfail` from the window tests.
    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    39893fc View commit details
    Browse the repository at this point in the history

  32. improve first_call and last_call use of the builder API

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    22f372b View commit details
    Browse the repository at this point in the history

  33. remove trailing todos

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    7211ba6 View commit details
    Browse the repository at this point in the history

  34. fix examples/substrait.py

    @Michael-J-Ward
    Michael-J-Ward committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    a75861d View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. chore: remove explicit aliases from functions.rs 

    Ref: apache#779
    @Michael-J-Ward
    Michael-J-Ward committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    0c57e59 View commit details
    Browse the repository at this point in the history

  2. remove array_fn! aliases

    @Michael-J-Ward
    Michael-J-Ward committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    eb905fb View commit details
    Browse the repository at this point in the history

  3. remove alias rules for expr_fn_vec!

    @Michael-J-Ward
    Michael-J-Ward committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    00779b7 View commit details
    Browse the repository at this point in the history

  4. remove alias rules from expr_fn! macro

    @Michael-J-Ward
    Michael-J-Ward committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    dc2db41 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. remove unnecessary pyo3 var-arg signatures in functions.rs

    @Michael-J-Ward
    Michael-J-Ward committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    65ea065 View commit details
    Browse the repository at this point in the history

  2. remove pyo3 signatures that provided defaults for first_value and las… 

    …t_value
    @Michael-J-Ward
    Michael-J-Ward committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    2009741 View commit details
    Browse the repository at this point in the history

  3. parametrize test_string_functions

    @Michael-J-Ward
    Michael-J-Ward committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    e17ba64 View commit details
    Browse the repository at this point in the history

  4. test regr_ function wrappers 

    Closes apache#778
    @Michael-J-Ward
    Michael-J-Ward committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    d293398 View commit details
    Browse the repository at this point in the history