Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] casting from float32 to Decimal64Dtype is resulting in incorrect values #14169

Closed
galipremsagar opened this issue Sep 22, 2023 · 7 comments
Assignees
Labels
0 - Backlog In queue waiting for assignment bug Something isn't working libcudf Affects libcudf (C++/CUDA) code.

Comments

@galipremsagar
Copy link
Contributor

Describe the bug
When there is a type-cast from float32 to Decimal64Dtype, the decimal values yield incorrect results. This issue was surfaced when testing arrow-13. arrow-12 has the same bug and it was fixed in arrow-13, we have similar issue in our libcudf too.

Steps/Code to reproduce bug

IIn [20]: import pyarrow as pa # This is pyarrow-13, not 12

In [21]: import cudf

In [22]: data = [14.12302, 97938.2, None, 0.0, -8.302014, None, 94.31304, -112.2314, 0.3333333]

In [23]: s = cudf.Series(data, dtype='float32')

In [24]: s
Out[24]: 
0     14.12302017
1     97938.20313
2            <NA>
3             0.0
4    -8.302014351
5            <NA>
6     94.31304169
7    -112.2313995
8     0.333333313
dtype: float32

In [25]: parray = pa.array(data, type=pa.float32())

In [26]: parray
Out[26]: 
<pyarrow.lib.FloatArray object at 0x7ff00e3bf700>
[
  14.12302,
  97938.2,
  null,
  0,
  -8.302014,
  null,
  94.31304,
  -112.2314,
  0.3333333
]

In [27]: decimal_dtype = cudf.Decimal64Dtype(11, 4)

In [28]: s.astype(decimal_dtype)
Out[28]: 
0       14.1230
1    97938.2016      # Incorrect, it should be 97938.2031
2          None
3        0.0000
4       -8.3020
5          None
6       94.3130
7     -112.2314
8        0.3333
dtype: decimal64

In [29]: parray.cast(decimal_dtype.to_arrow())
Out[29]: 
<pyarrow.lib.Decimal128Array object at 0x7ff00e1bebc0>
[
  14.1230,
  97938.2031,
  null,
  0.0000,
  -8.3020,
  null,
  94.3130,
  -112.2314,
  0.3333
]

Expected behavior

In [28]: s.astype(decimal_dtype)
Out[28]: 
0       14.1230
1    97938.2031      # Incorrect, it should be 97938.2031
2          None
3        0.0000
4       -8.3020
5          None
6       94.3130
7     -112.2314
8        0.3333
dtype: decimal64

Environment overview (please complete the following information)

  • Environment location: [Bare-metal]
  • Method of cuDF install: [from source]

Environment details
Please run and paste the output of the cudf/print_env.sh script here, to gather any other relevant environment details

Click here to see environment details
 **git***
 commit 4b2a8e2ccdeff36ff58d28d80c8742a474e1d69a (HEAD -> arrow_13)
 Author: galipremsagar <[email protected]>
 Date:   Thu Sep 21 23:13:31 2023 -0700
 
 Upgrade to arrow 13
 **git submodules***
 
 ***OS Information***
 DISTRIB_ID=Ubuntu
 DISTRIB_RELEASE=18.04
 DISTRIB_CODENAME=bionic
 DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
 NAME="Ubuntu"
 VERSION="18.04.4 LTS (Bionic Beaver)"
 ID=ubuntu
 ID_LIKE=debian
 PRETTY_NAME="Ubuntu 18.04.4 LTS"
 VERSION_ID="18.04"
 HOME_URL="https://www.ubuntu.com/"
 SUPPORT_URL="https://help.ubuntu.com/"
 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
 VERSION_CODENAME=bionic
 UBUNTU_CODENAME=bionic
 Linux dt07 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
 
 ***GPU Information***
 Fri Sep 22 03:31:47 2023
 +---------------------------------------------------------------------------------------+
 | NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
 |-----------------------------------------+----------------------+----------------------+
 | GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
 | Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
 |                                         |                      |               MIG M. |
 |=========================================+======================+======================|
 |   0  Tesla T4                        On | 00000000:3B:00.0 Off |                    0 |
 | N/A   46C    P0               27W /  70W|   1348MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 |   1  Tesla T4                        On | 00000000:5E:00.0 Off |                    0 |
 | N/A   34C    P8                9W /  70W|      4MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 |   2  Tesla T4                        On | 00000000:AF:00.0 Off |                    0 |
 | N/A   29C    P8               10W /  70W|      4MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 |   3  Tesla T4                        On | 00000000:D8:00.0 Off |                    0 |
 | N/A   28C    P8               10W /  70W|      4MiB / 15360MiB |      0%      Default |
 |                                         |                      |                  N/A |
 +-----------------------------------------+----------------------+----------------------+
 
 +---------------------------------------------------------------------------------------+
 | Processes:                                                                            |
 |  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
 |        ID   ID                                                             Usage      |
 |=======================================================================================|
 +---------------------------------------------------------------------------------------+
 
 ***CPU***
 Architecture:        x86_64
 CPU op-mode(s):      32-bit, 64-bit
 Byte Order:          Little Endian
 CPU(s):              64
 On-line CPU(s) list: 0-63
 Thread(s) per core:  2
 Core(s) per socket:  16
 Socket(s):           2
 NUMA node(s):        2
 Vendor ID:           GenuineIntel
 CPU family:          6
 Model:               85
 Model name:          Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
 Stepping:            4
 CPU MHz:             1194.971
 BogoMIPS:            4200.00
 Virtualization:      VT-x
 L1d cache:           32K
 L1i cache:           32K
 L2 cache:            1024K
 L3 cache:            22528K
 NUMA node0 CPU(s):   0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
 NUMA node1 CPU(s):   1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
 Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d arch_capabilities
 
 ***CMake***
 /nvme/0/pgali/envs/cudfdev/bin/cmake
 cmake version 3.27.6
 
 CMake suite maintained and supported by Kitware (kitware.com/cmake).
 
 ***g++***
 /nvme/0/pgali/envs/cudfdev/bin/g++
 g++ (conda-forge gcc 11.4.0-2) 11.4.0
 Copyright (C) 2021 Free Software Foundation, Inc.
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
 
 ***nvcc***
 /nvme/0/pgali/envs/cudfdev/bin/nvcc
 nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2022 NVIDIA Corporation
 Built on Wed_Sep_21_10:33:58_PDT_2022
 Cuda compilation tools, release 11.8, V11.8.89
 Build cuda_11.8.r11.8/compiler.31833905_0
 
 ***Python***
 /nvme/0/pgali/envs/cudfdev/bin/python
 Python 3.10.12
 
 ***Environment Variables***
 PATH                            : /nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/anaconda3/bin:/nvme/0/pgali/envs/cudfdev/bin:/nvme/0/pgali/.cargo/bin:/home/nfs/pgali/.vscode-server/bin/abd2f3db4bdb28f9e95536dfa84d8479f1eb312d/bin/remote-cli:/nvme/0/pgali/.cargo/bin:/nvme/0/pgali/anaconda3/bin:/nvme/0/pgali/anaconda3/condabin:/nvme/0/pgali/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda/bin:/usr/local/cuda/bin
 LD_LIBRARY_PATH                 : /usr/local/cuda/lib64:/usr/local/cuda/lib64::/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
 NUMBAPRO_NVVM                   :
 NUMBAPRO_LIBDEVICE              :
 CONDA_PREFIX                    : /nvme/0/pgali/envs/cudfdev
 PYTHON_PATH                     :
 
 ***conda packages***
 /nvme/0/pgali/anaconda3/bin/conda
 # packages in environment at /nvme/0/pgali/envs/cudfdev:
 #
 # Name                    Version                   Build  Channel
 _libgcc_mutex             0.1                 conda_forge    conda-forge
 _openmp_mutex             4.5                       2_gnu    conda-forge
 _sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
 accessible-pygments       0.0.4              pyhd8ed1ab_0    conda-forge
 aiobotocore               2.5.4              pyhd8ed1ab_0    conda-forge
 aiohttp                   3.8.5           py310h2372a71_0    conda-forge
 aioitertools              0.11.0             pyhd8ed1ab_0    conda-forge
 aiosignal                 1.3.1              pyhd8ed1ab_0    conda-forge
 alabaster                 0.7.13             pyhd8ed1ab_0    conda-forge
 anyio                     4.0.0              pyhd8ed1ab_0    conda-forge
 argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
 argon2-cffi-bindings      21.2.0          py310h5764c6d_3    conda-forge
 arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
 asttokens                 2.4.0              pyhd8ed1ab_0    conda-forge
 async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
 async-timeout             4.0.3              pyhd8ed1ab_0    conda-forge
 attrs                     23.1.0             pyh71513ae_1    conda-forge
 aws-c-auth                0.7.3                he2921ad_3    conda-forge
 aws-c-cal                 0.6.2                hc309b26_1    conda-forge
 aws-c-common              0.9.0                hd590300_0    conda-forge
 aws-c-compression         0.2.17               h4d4d85c_2    conda-forge
 aws-c-event-stream        0.3.2                h2e3709c_0    conda-forge
 aws-c-http                0.7.12               hc865f51_1    conda-forge
 aws-c-io                  0.13.32              h1a03231_3    conda-forge
 aws-c-mqtt                0.9.6                h3a0376c_0    conda-forge
 aws-c-s3                  0.3.17               h1678ad6_0    conda-forge
 aws-c-sdkutils            0.1.12               h4d4d85c_1    conda-forge
 aws-checksums             0.1.17               h4d4d85c_1    conda-forge
 aws-crt-cpp               0.23.1               hf7d0843_2    conda-forge
 aws-sam-translator        1.75.0             pyhd8ed1ab_0    conda-forge
 aws-sdk-cpp               1.11.156             he6c2984_2    conda-forge
 aws-xray-sdk              2.12.0             pyhd8ed1ab_0    conda-forge
 babel                     2.12.1             pyhd8ed1ab_1    conda-forge
 backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
 backports                 1.0                pyhd8ed1ab_3    conda-forge
 backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
 backports.zoneinfo        0.2.1           py310hff52083_7    conda-forge
 bcrypt                    4.0.1           py310hcb5633a_1    conda-forge
 beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
 benchmark                 1.8.0                h59595ed_0    conda-forge
 binutils                  2.40                 hdd6e379_0    conda-forge
 binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
 binutils_linux-64         2.40                 hbdbef99_2    conda-forge
 blas                      1.0                         mkl    conda-forge
 bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
 blinker                   1.6.2              pyhd8ed1ab_0    conda-forge
 bokeh                     3.2.2              pyhd8ed1ab_0    conda-forge
 boto3                     1.28.17            pyhd8ed1ab_0    conda-forge
 botocore                  1.31.17            pyhd8ed1ab_3    conda-forge
 brotlipy                  0.7.0           py310h5764c6d_1005    conda-forge
 bzip2                     1.0.8                h7f98852_4    conda-forge
 c-ares                    1.19.1               hd590300_0    conda-forge
 c-compiler                1.5.2                h0b41bf4_0    conda-forge
 ca-certificates           2023.7.22            hbcca054_0    conda-forge
 cached-property           1.5.2                hd8ed1ab_1    conda-forge
 cached_property           1.5.2              pyha770c72_1    conda-forge
 cachetools                5.3.1              pyhd8ed1ab_0    conda-forge
 certifi                   2023.7.22          pyhd8ed1ab_0    conda-forge
 cffi                      1.15.1          py310h2fee648_5    conda-forge
 cfgv                      3.3.1              pyhd8ed1ab_0    conda-forge
 cfn-lint                  0.80.2             pyhd8ed1ab_0    conda-forge
 charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
 click                     8.1.7           unix_pyh707e725_0    conda-forge
 cloudpickle               2.2.1              pyhd8ed1ab_0    conda-forge
 cmake                     3.27.6               hcfe8598_0    conda-forge
 colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
 comm                      0.1.4              pyhd8ed1ab_0    conda-forge
 commonmark                0.9.1                      py_0    conda-forge
 contourpy                 1.1.1           py310hd41b1e2_0    conda-forge
 coverage                  7.3.1           py310h2372a71_1    conda-forge
 cryptography              41.0.4          py310h75e40e8_0    conda-forge
 cubinlinker               0.3.0           py310hfdf336d_0    rapidsai
 cuda-nvtx                 11.8.86                       0    nvidia
 cuda-python               11.8.2          py310h01a121a_0    conda-forge
 cuda-sanitizer-api        11.8.86                       0    nvidia
 cuda-version              11.8                 h70ddcb2_2    conda-forge
 cudatoolkit               11.8.0              h4ba93d1_12    conda-forge
 cudf                      23.10.0                  pypi_0    pypi
 cupy                      12.2.0          py310hbb1d8f0_1    conda-forge
 cxx-compiler              1.5.2                hf52228f_0    conda-forge
 cyrus-sasl                2.1.27               h54b06d7_7    conda-forge
 cython                    3.0.2           py310hc6cd4ac_2    conda-forge
 cytoolz                   0.12.2          py310h2372a71_0    conda-forge
 dask                      2023.9.2           pyhd8ed1ab_0    conda-forge
 dask-core                 2023.9.3a230920 py_gda256320e_1    dask/label/dev
 dask-cudf                 23.10.0                  pypi_0    pypi
 dataclasses               0.8                pyhc8e2a94_3    conda-forge
 datasets                  2.14.4             pyhd8ed1ab_0    conda-forge
 debugpy                   1.8.0           py310hc6cd4ac_0    conda-forge
 decopatch                 1.4.10             pyhd8ed1ab_0    conda-forge
 decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
 defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
 dill                      0.3.7              pyhd8ed1ab_0    conda-forge
 distlib                   0.3.7              pyhd8ed1ab_0    conda-forge
 distributed               2023.9.2           pyhd8ed1ab_0    conda-forge
 distro                    1.8.0              pyhd8ed1ab_0    conda-forge
 dlpack                    0.5                  h9c3ff4c_0    conda-forge
 docker-py                 6.1.3              pyhd8ed1ab_0    conda-forge
 docutils                  0.19            py310hff52083_1    conda-forge
 doxygen                   1.9.1                hb166930_1    conda-forge
 ecdsa                     0.18.0             pyhd8ed1ab_1    conda-forge
 entrypoints               0.4                pyhd8ed1ab_0    conda-forge
 exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
 execnet                   2.0.2              pyhd8ed1ab_0    conda-forge
 executing                 1.2.0              pyhd8ed1ab_0    conda-forge
 fastavro                  1.8.3           py310h2372a71_0    conda-forge
 fastrlock                 0.8.2           py310hc6cd4ac_0    conda-forge
 filelock                  3.12.4             pyhd8ed1ab_0    conda-forge
 flask                     2.3.3              pyhd8ed1ab_0    conda-forge
 flask_cors                3.0.10             pyhd3deb0d_0    conda-forge
 fmt                       9.1.0                h924138e_0    conda-forge
 fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
 freetype                  2.12.1               h267a509_2    conda-forge
 frozenlist                1.4.0           py310h2372a71_0    conda-forge
 fsspec                    2023.9.1           pyh1a96a4e_0    conda-forge
 future                    0.18.3             pyhd8ed1ab_0    conda-forge
 gcc                       11.4.0               h7baecda_2    conda-forge
 gcc_impl_linux-64         11.4.0               h7aa1c59_2    conda-forge
 gcc_linux-64              11.4.0               hfd045f2_2    conda-forge
 gflags                    2.2.2             he1b5a44_1004    conda-forge
 glog                      0.6.0                h6f12383_0    conda-forge
 gmock                     1.14.0               ha770c72_1    conda-forge
 gmp                       6.2.1                h58526e2_0    conda-forge
 gmpy2                     2.1.2           py310h3ec546c_1    conda-forge
 graphql-core              3.2.3              pyhd8ed1ab_0    conda-forge
 greenlet                  2.0.2           py310hc6cd4ac_1    conda-forge
 gtest                     1.14.0               h00ab1b0_1    conda-forge
 gxx                       11.4.0               h7baecda_2    conda-forge
 gxx_impl_linux-64         11.4.0               h7aa1c59_2    conda-forge
 gxx_linux-64              11.4.0               hfc1ae95_2    conda-forge
 huggingface_hub           0.17.2             pyhd8ed1ab_0    conda-forge
 hypothesis                6.86.2             pyha770c72_0    conda-forge
 identify                  2.5.29             pyhd8ed1ab_0    conda-forge
 idna                      3.4                pyhd8ed1ab_0    conda-forge
 imagesize                 1.4.1              pyhd8ed1ab_0    conda-forge
 importlib-metadata        6.8.0              pyha770c72_0    conda-forge
 importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
 importlib_resources       5.13.0             pyhd8ed1ab_0    conda-forge
 iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
 intel-openmp              2022.1.0          h9e868ea_3769
 ipykernel                 6.25.2             pyh2140261_0    conda-forge
 ipython                   8.15.0             pyh0d859eb_0    conda-forge
 isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
 itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
 jedi                      0.19.0             pyhd8ed1ab_0    conda-forge
 jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
 jmespath                  1.0.1              pyhd8ed1ab_0    conda-forge
 joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
 jschema-to-python         1.2.3              pyhd8ed1ab_0    conda-forge
 json5                     0.9.14             pyhd8ed1ab_0    conda-forge
 jsondiff                  2.0.0              pyhd8ed1ab_0    conda-forge
 jsonpatch                 1.32               pyhd8ed1ab_0    conda-forge
 jsonpickle                2.2.0              pyhd8ed1ab_0    conda-forge
 jsonpointer               2.4             py310hff52083_2    conda-forge
 jsonschema                4.19.1             pyhd8ed1ab_0    conda-forge
 jsonschema-spec           0.2.4              pyhd8ed1ab_0    conda-forge
 jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
 jsonschema-with-format-nongpl 4.19.1             pyhd8ed1ab_0    conda-forge
 junit-xml                 1.9                pyh9f0ad1d_0    conda-forge
 jupyter-cache             0.6.1              pyhd8ed1ab_0    conda-forge
 jupyter-lsp               2.2.0              pyhd8ed1ab_0    conda-forge
 jupyter_client            8.3.1              pyhd8ed1ab_0    conda-forge
 jupyter_core              5.3.1           py310hff52083_0    conda-forge
 jupyter_events            0.7.0              pyhd8ed1ab_2    conda-forge
 jupyter_server            2.7.3              pyhd8ed1ab_0    conda-forge
 jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
 jupyterlab                4.0.6              pyhd8ed1ab_0    conda-forge
 jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
 jupyterlab_server         2.25.0             pyhd8ed1ab_0    conda-forge
 kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
 keyutils                  1.6.1                h166bdaf_0    conda-forge
 krb5                      1.21.2               h659d440_0    conda-forge
 lazy-object-proxy         1.9.0           py310h1fa729e_0    conda-forge
 lcms2                     2.15                 h7f713cb_2    conda-forge
 ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
 lerc                      4.0.0                h27087fc_0    conda-forge
 libabseil                 20230802.1      cxx17_h59595ed_0    conda-forge
 libarrow                  13.0.0           h1935d02_4_cpu    conda-forge
 libblas                   3.9.0            16_linux64_mkl    conda-forge
 libbrotlicommon           1.1.0                hd590300_0    conda-forge
 libbrotlidec              1.1.0                hd590300_0    conda-forge
 libbrotlienc              1.1.0                hd590300_0    conda-forge
 libcblas                  3.9.0            16_linux64_mkl    conda-forge
 libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
 libcufile                 1.4.0.31                      0    nvidia
 libcufile-dev             1.4.0.31                      0    nvidia
 libcurand                 10.3.0.86                     0    nvidia
 libcurand-dev             10.3.0.86                     0    nvidia
 libcurl                   8.3.0                hca28451_0    conda-forge
 libdeflate                1.19                 hd590300_0    conda-forge
 libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.12               hf998b51_1    conda-forge
 libexpat                  2.5.0                hcb278e6_1    conda-forge
 libffi                    3.4.2                h7f98852_5    conda-forge
 libgcc-devel_linux-64     11.4.0               h922705a_2    conda-forge
 libgcc-ng                 13.2.0               h807b86a_2    conda-forge
 libgfortran-ng            13.2.0               h69a702a_2    conda-forge
 libgfortran5              13.2.0               ha4646dd_2    conda-forge
 libgomp                   13.2.0               h807b86a_2    conda-forge
 libgoogle-cloud           2.12.0               h8d7e28b_2    conda-forge
 libgrpc                   1.57.0               ha4d0f93_1    conda-forge
 libiconv                  1.17                 h166bdaf_0    conda-forge
 libjpeg-turbo             2.1.5.1              hd590300_1    conda-forge
 libkvikio                 23.10.00a       cuda11_230922_g8db93d0_23    rapidsai-nightly
 liblapack                 3.9.0            16_linux64_mkl    conda-forge
 libllvm14                 14.0.6               hcd5def8_4    conda-forge
 libnghttp2                1.52.0               h61bc06f_0    conda-forge
 libnsl                    2.0.0                h7f98852_0    conda-forge
 libntlm                   1.4               h7f98852_1002    conda-forge
 libnuma                   2.0.16               h0b41bf4_1    conda-forge
 libpng                    1.6.39               h753d276_0    conda-forge
 libprotobuf               4.23.4               hf27288f_6    conda-forge
 librdkafka                1.9.2                ha5a0de0_2    conda-forge
 librmm                    23.10.00a       cuda11_230922_gcd76d414_24    rapidsai-nightly
 libsanitizer              11.4.0               h4dcbe23_2    conda-forge
 libsodium                 1.0.18               h36c2ea0_1    conda-forge
 libsqlite                 3.43.0               h2797004_0    conda-forge
 libssh2                   1.11.0               h0841786_0    conda-forge
 libstdcxx-devel_linux-64  11.4.0               h922705a_2    conda-forge
 libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
 libthrift                 0.19.0               h8fd135c_0    conda-forge
 libtiff                   4.6.0                h29866fb_1    conda-forge
 libutf8proc               2.8.0                h166bdaf_0    conda-forge
 libuuid                   2.38.1               h0b41bf4_0    conda-forge
 libuv                     1.46.0               hd590300_0    conda-forge
 libwebp-base              1.3.2                hd590300_0    conda-forge
 libxcb                    1.15                 h0b41bf4_0    conda-forge
 libzlib                   1.2.13               hd590300_5    conda-forge
 livereload                2.6.3              pyh9f0ad1d_0    conda-forge
 llvmlite                  0.40.1          py310h1b8f574_0    conda-forge
 locket                    1.0.0              pyhd8ed1ab_0    conda-forge
 lz4                       4.3.2           py310h0cfdcf0_0    conda-forge
 lz4-c                     1.9.4                hcb278e6_0    conda-forge
 make                      4.3                  hd18ef5c_1    conda-forge
 makefun                   1.15.1             pyhd8ed1ab_0    conda-forge
 markdown                  3.4.4              pyhd8ed1ab_0    conda-forge
 markdown-it-py            2.2.0              pyhd8ed1ab_0    conda-forge
 markupsafe                2.1.3           py310h2372a71_1    conda-forge
 matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
 mdit-py-plugins           0.4.0              pyhd8ed1ab_0    conda-forge
 mdurl                     0.1.0              pyhd8ed1ab_0    conda-forge
 mimesis                   11.1.0             pyhd8ed1ab_0    conda-forge
 mistune                   3.0.1              pyhd8ed1ab_0    conda-forge
 mkl                       2022.1.0           hc2b9512_224
 moto                      4.2.3              pyhd8ed1ab_0    conda-forge
 mpc                       1.3.1                hfe3b2da_0    conda-forge
 mpfr                      4.2.0                hb012696_0    conda-forge
 mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
 msgpack-python            1.0.5           py310hdf3cbec_0    conda-forge
 multidict                 6.0.4           py310h1fa729e_0    conda-forge
 multiprocess              0.70.15         py310h2372a71_0    conda-forge
 myst-nb                   0.17.2             pyhd8ed1ab_0    conda-forge
 myst-parser               0.18.1             pyhd8ed1ab_0    conda-forge
 nbclient                  0.7.4              pyhd8ed1ab_0    conda-forge
 nbconvert                 7.8.0              pyhd8ed1ab_0    conda-forge
 nbconvert-core            7.8.0              pyhd8ed1ab_0    conda-forge
 nbconvert-pandoc          7.8.0              pyhd8ed1ab_0    conda-forge
 nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
 nbsphinx                  0.9.3              pyhd8ed1ab_0    conda-forge
 ncurses                   6.4                  hcb278e6_0    conda-forge
 nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
 networkx                  3.1                pyhd8ed1ab_0    conda-forge
 ninja                     1.11.1               h924138e_0    conda-forge
 nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
 notebook                  7.0.4              pyhd8ed1ab_0    conda-forge
 notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
 numba                     0.57.1          py310h0f6aa51_0    conda-forge
 numpy                     1.24.4          py310ha4c1d20_0    conda-forge
 numpydoc                  1.5.0              pyhd8ed1ab_0    conda-forge
 nvcc_linux-64             11.8                h667003e_22    conda-forge
 nvcomp                    2.6.1                h0800d71_2    conda-forge
 nvtx                      0.2.8           py310h2372a71_0    conda-forge
 openapi-schema-validator  0.6.1              pyhd8ed1ab_0    conda-forge
 openapi-spec-validator    0.6.0              pyhd8ed1ab_0    conda-forge
 openjpeg                  2.5.0                h488ebb8_3    conda-forge
 openssl                   3.1.3                hd590300_0    conda-forge
 orc                       1.9.0                h52d3b3c_2    conda-forge
 overrides                 7.4.0              pyhd8ed1ab_0    conda-forge
 packaging                 23.1               pyhd8ed1ab_0    conda-forge
 pandas                    1.5.3           py310h9b08913_1    conda-forge
 pandoc                    3.1.3                h32600fe_0    conda-forge
 pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
 paramiko                  3.3.1              pyhd8ed1ab_0    conda-forge
 parso                     0.8.3              pyhd8ed1ab_0    conda-forge
 partd                     1.4.0              pyhd8ed1ab_1    conda-forge
 pathable                  0.4.3              pyhd8ed1ab_0    conda-forge
 pbr                       5.11.1             pyhd8ed1ab_0    conda-forge
 pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
 pickleshare               0.7.5                   py_1003    conda-forge
 pillow                    10.0.1          py310h29da1c1_1    conda-forge
 pip                       23.2.1             pyhd8ed1ab_0    conda-forge
 pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
 platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
 pluggy                    1.3.0              pyhd8ed1ab_0    conda-forge
 pre-commit                3.4.0              pyha770c72_1    conda-forge
 prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
 prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
 prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
 protobuf                  4.23.4          py310h620c231_2    conda-forge
 psutil                    5.9.5           py310h2372a71_1    conda-forge
 pthread-stubs             0.4               h36c2ea0_1001    conda-forge
 ptxcompiler               0.8.1           py310h01a121a_0    conda-forge
 ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
 pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
 py-cpuinfo                9.0.0              pyhd8ed1ab_0    conda-forge
 pyarrow                   13.0.0          py310hf9e7431_4_cpu    conda-forge
 pyasn1                    0.5.0              pyhd8ed1ab_0    conda-forge
 pycparser                 2.21               pyhd8ed1ab_0    conda-forge
 pydantic                  1.10.12         py310h2372a71_1    conda-forge
 pydata-sphinx-theme       0.14.1             pyhd8ed1ab_0    conda-forge
 pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
 pynacl                    1.5.0           py310h5764c6d_2    conda-forge
 pyopenssl                 23.2.0             pyhd8ed1ab_1    conda-forge
 pyparsing                 3.1.1              pyhd8ed1ab_0    conda-forge
 pysocks                   1.7.1              pyha2e5f31_6    conda-forge
 pytest                    7.4.2              pyhd8ed1ab_0    conda-forge
 pytest-benchmark          4.0.0              pyhd8ed1ab_0    conda-forge
 pytest-cases              3.6.14             pyhd8ed1ab_0    conda-forge
 pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
 pytest-xdist              3.3.1              pyhd8ed1ab_0    conda-forge
 python                    3.10.12         hd12c33a_0_cpython    conda-forge
 python-confluent-kafka    1.9.2           py310h5764c6d_2    conda-forge
 python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
 python-fastjsonschema     2.18.0             pyhd8ed1ab_0    conda-forge
 python-jose               3.3.0              pyh6c4a22f_1    conda-forge
 python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
 python-snappy             0.6.1           py310hcee4d7c_0    conda-forge
 python-xxhash             3.3.0           py310h2372a71_0    conda-forge
 python_abi                3.10                    4_cp310    conda-forge
 pytorch                   1.11.0             py3.10_cpu_0    pytorch
 pytorch-mutex             1.0                         cpu    pytorch
 pytz                      2023.3.post1       pyhd8ed1ab_0    conda-forge
 pywin32-on-windows        0.1.0              pyh1179c8e_3    conda-forge
 pyyaml                    6.0.1           py310h2372a71_1    conda-forge
 pyzmq                     25.1.1          py310h5bbb5d0_0    conda-forge
 rdma-core                 28.9                 h59595ed_1    conda-forge
 re2                       2023.03.02           h8c504da_0    conda-forge
 readline                  8.2                  h8228510_1    conda-forge
 recommonmark              0.7.1              pyhd8ed1ab_0    conda-forge
 referencing               0.30.2             pyhd8ed1ab_0    conda-forge
 regex                     2023.8.8        py310h2372a71_0    conda-forge
 requests                  2.31.0             pyhd8ed1ab_0    conda-forge
 responses                 0.23.1             pyhd8ed1ab_0    conda-forge
 rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
 rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
 rhash                     1.4.4                hd590300_0    conda-forge
 rmm                       23.10.00a       cuda11_py310_230922_gcd76d414_24    rapidsai-nightly
 rpds-py                   0.10.3          py310hcb5633a_0    conda-forge
 rsa                       4.9                pyhd8ed1ab_0    conda-forge
 s2n                       1.3.51               h06160fa_0    conda-forge
 s3fs                      2023.9.1           pyhd8ed1ab_0    conda-forge
 s3transfer                0.6.2              pyhd8ed1ab_0    conda-forge
 sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
 sarif-om                  1.0.4              pyhd8ed1ab_0    conda-forge
 scikit-build              0.17.6             pyh4af843d_0    conda-forge
 scipy                     1.11.2          py310hb13e2d6_1    conda-forge
 sed                       4.8                  he412f7d_0    conda-forge
 send2trash                1.8.2              pyh41d4057_0    conda-forge
 setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
 six                       1.16.0             pyh6c4a22f_0    conda-forge
 snappy                    1.1.10               h9fff704_0    conda-forge
 sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
 snowballstemmer           2.2.0              pyhd8ed1ab_0    conda-forge
 sortedcontainers          2.4.0              pyhd8ed1ab_0    conda-forge
 soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
 spdlog                    1.11.0               h9b3ece8_1    conda-forge
 sphinx                    5.3.0              pyhd8ed1ab_0    conda-forge
 sphinx-autobuild          2021.3.14          pyhd8ed1ab_0    conda-forge
 sphinx-copybutton         0.5.2              pyhd8ed1ab_0    conda-forge
 sphinx-markdown-tables    0.0.17             pyh6c4a22f_0    conda-forge
 sphinxcontrib-applehelp   1.0.7              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-devhelp     1.0.5              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-htmlhelp    2.0.4              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-jsmath      1.0.1              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-qthelp      1.0.6              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-serializinghtml 1.1.9              pyhd8ed1ab_0    conda-forge
 sphinxcontrib-websupport  1.2.6              pyhd8ed1ab_0    conda-forge
 sqlalchemy                2.0.21          py310h2372a71_0    conda-forge
 sshpubkeys                3.3.1              pyhd8ed1ab_0    conda-forge
 stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
 streamz                   0.6.4              pyh6c4a22f_0    conda-forge
 sympy                     1.12            pypyh9d50eac_103    conda-forge
 sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
 tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
 tblib                     2.0.0              pyhd8ed1ab_0    conda-forge
 terminado                 0.17.1             pyh41d4057_0    conda-forge
 tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
 tk                        8.6.12               h27826a3_0    conda-forge
 tokenizers                0.13.1          py310h633acb5_2    conda-forge
 toml                      0.10.2             pyhd8ed1ab_0    conda-forge
 tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
 toolz                     0.12.0             pyhd8ed1ab_0    conda-forge
 tornado                   6.3.3           py310h2372a71_1    conda-forge
 tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
 traitlets                 5.10.0             pyhd8ed1ab_0    conda-forge
 transformers              4.24.0             pyhd8ed1ab_0    conda-forge
 types-pyyaml              6.0.12.11          pyhd8ed1ab_0    conda-forge
 typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
 typing_extensions         4.8.0              pyha770c72_0    conda-forge
 typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
 tzdata                    2023c                h71feb2d_0    conda-forge
 ucx                       1.14.1               h64cca9d_5    conda-forge
 ukkonen                   1.0.1           py310hbf28c38_3    conda-forge
 uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
 urllib3                   1.26.15            pyhd8ed1ab_0    conda-forge
 virtualenv                20.24.4            pyhd8ed1ab_0    conda-forge
 wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
 webcolors                 1.13               pyhd8ed1ab_0    conda-forge
 webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
 websocket-client          1.6.3              pyhd8ed1ab_0    conda-forge
 werkzeug                  2.3.7              pyhd8ed1ab_0    conda-forge
 wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
 wrapt                     1.15.0          py310h1fa729e_0    conda-forge
 xmltodict                 0.13.0             pyhd8ed1ab_0    conda-forge
 xorg-libxau               1.0.11               hd590300_0    conda-forge
 xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
 xxhash                    0.8.2                hd590300_0    conda-forge
 xyzservices               2023.7.0           pyhd8ed1ab_0    conda-forge
 xz                        5.2.6                h166bdaf_0    conda-forge
 yaml                      0.2.5                h7f98852_2    conda-forge
 yarl                      1.9.2           py310h2372a71_0    conda-forge
 zeromq                    4.3.4                h9c3ff4c_1    conda-forge
 zict                      3.0.0              pyhd8ed1ab_0    conda-forge
 zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
 zlib                      1.2.13               hd590300_5    conda-forge
 zstd                      1.5.5                hfc55251_0    conda-forge

Additional context
Add any other context about the problem here.

@galipremsagar galipremsagar added bug Something isn't working Needs Triage Need team to review and classify labels Sep 22, 2023
@galipremsagar
Copy link
Contributor Author

@bdice I found the PR that was responsible for resolving this issue in the arrow project: apache/arrow#35997

Would you be able to take a look at this in libcudf?

@GregoryKimball GregoryKimball added 0 - Backlog In queue waiting for assignment libcudf Affects libcudf (C++/CUDA) code. and removed Needs Triage Need team to review and classify labels Nov 9, 2023
@bdice
Copy link
Contributor

bdice commented Nov 27, 2023

I'm hoping to work on this in 24.02 but not sure if I'll have time in this release. It's on my list of tasks.

@pmattione-nvidia
Copy link
Contributor

I'm working on a set of changes to try to address this.

@pmattione-nvidia
Copy link
Contributor

Update: This work has been broken into many separate PRs, 4 of which have been merged so far:

Explicit conversion PR
Benchmarking PR
Powers-of-10 PR
Utilities PR

The primary PR will be submitted very soon. I'm putting some finishing touches on it based off of conversations with @ttnghia on handling some of the edge cases.

@pmattione-nvidia
Copy link
Contributor

There will always be discrepancies between cuDF and pyarrow, because cuDF has chosen to truncate its results, whereas pyarrow has chosen to round them.

rapids-bot bot pushed a commit that referenced this issue Jul 11, 2024
This PR contains the main algorithm for the new decimal <--> floating conversion code. This algorithm was written to address the precision issues described [here](#14169). 

### Summary
* The new algorithm is more accurate than the previous code, but it is also far more complex.  
* It can perform conversions that were not even possible in the old code due to overflow (decimal32/64/128 conversions only worked for scale factors up to 10^9/18/38, respectively). Now the entire floating-point range is convertible, including denormals. 
* This new algorithm is significantly faster in some parts of the conversion phase-space, and in some parts slightly slower.  

### Previous PR's 
These contain the supporting parts of this work:  
* [Explicit conversion PR](#15438) 
* [Benchmarking PR](#15334)
* [Powers-of-10 PR](#15353)
* [Utilities PR](#15359). These utilities are updated here to support denormals. 

### Algorithm Outline
We convert floating -> (integer) decimal by:
* Extract the floating-point mantissa (converted to integer) and power-of-2
* For float we use a uint64 to contain our data during the below shifting/scaling, for double uint128_t
* In this shifting integer, we alternately apply the extracted powers-of-2 (bit-shifts, until they're all used) and scale-factor powers-of-10 (multiply/divide) as needed to reach the desired scale factor. 

Decimal -> floating is just the reverse operation.  

### Supplemental Changes
* Testing: Add decimal128, add precise-conversion tests. Remove kludges due to inaccurate conversions. Add test for zeroes. 
* Benchmarking: Enable regions of conversion phase-space for benchmarking that were not possible in the old algorithm. 
* Unary: Cleanup by using CUDF_ENABLE_IF.  Call new conversion code for base-10 fixed-point. 

### Performance for various conversions/input-ranges
* Note: F32/F64 is float/double

New algorithm is **FASTER** by: 
* F64             --> decimal64:   60% for E8    --> E15
* F64             --> decimal128: 13% for E-8  --> E-15
* F64             --> decimal128: 22% for E8    --> E15
* F64             --> decimal128: 27% for E31  --> E38
* decimal32   --> F64:             18% for E-3   --> E4
* decimal64   --> F64:             27% for E-14 --> E-7
* decimal64   --> F64:             17% for E-3   --> E4
* decimal128 --> F64:             21% for E-14 --> E-7
* decimal128 --> F64:             11% for E-3   --> E4
* decimal128 --> F64:             13% for E31   --> E38

New algorithm is **SLOWER** by: 
* F32             --> decimal32:     3% for E-3   --> E4
* F32             --> decimal64:     2% for E-14   --> E14
* F64             --> decimal32:     3% for E-3   --> E4
* decimal32   --> F32:               5% for E-3   --> E4
* decimal128 --> F64:             36% for E-37 --> E-30

Other kernels:
* The PYMOD binary-op benchmark is 7% slower.  

### Performance discussion
* Many conversions have identical speed, indicating these algorithms are often fast and we are instead bottlenecked on overheads such as getting the input to the gpu in the first place. 
* F64 conversions are often much faster than the old algorithm as the new algorithm completely avoids the FP64 pipeline. Other than the cast to double itself, all of the operations are on integers. Thus we don't have threads competing with each other and taking turns for access to the floating-point cores. 
* The conversions are slightly slower for floats with powers-of-10 near zero.  Presumably this is due to code overhead for e.g., handling a large range of inputs, UB-checks for bit shifts, branches for denormals, etc. 
* The conversion is slower for decimal128 conversions with very small exponents, which requires several large divisions (128bit divided by 64bit). 
* The PYMOD kernel is slower due to register pressure from the introduction of the new division routines in the earlier PR. Even though this benchmark does not perform decimal <--> floating conversions, it gets hit because of inlined template code in the kernel increasing the code/register pressure.

Authors:
  - Paul Mattione (https://github.com/pmattione-nvidia)

Approvers:
  - Jason Lowe (https://github.com/jlowe)
  - Bradley Dice (https://github.com/bdice)
  - Mike Wilson (https://github.com/hyperbolic2346)

URL: #15905
@pmattione-nvidia
Copy link
Contributor

The new decimal <--> floating conversion PR has been merged.

@galipremsagar
Copy link
Contributor Author

Thanks @pmattione-nvidia ! Closing this issue, was able to verify that the issue is fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0 - Backlog In queue waiting for assignment bug Something isn't working libcudf Affects libcudf (C++/CUDA) code.
Projects
Status: Done
Development

No branches or pull requests

4 participants