Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write row group statistics #92

Merged
merged 26 commits into from
Sep 22, 2024
Merged

Write row group statistics #92

merged 26 commits into from
Sep 22, 2024

Conversation

gaborcsardi
Copy link
Member

@gaborcsardi gaborcsardi commented Sep 16, 2024 •
edited
Loading

Logical types:

  • STRING
  • ENUM
  • UUID
  • INT
  • UINT
  • DECIMAL
  • FLOAT16
  • DATE
  • TIME
  • TIMESTAMP
  • JSON
  • BSON

Primitive types:

  • BOOLEAN
  • INT32
  • INT64
  • FLOAT
  • DOUBLE
  • BYTE_ARRAY
  • FIXED_LEN_BYTE_ARRAY

Tests:

  • Uncompressed PLAIN data if probably currently, at least for some types, as we write it out directly?(tested it, it is fine)
  • UINT types might not use unsigned comparison

@gaborcsardi
Write null_count per column chunk
d1c5a01
@gaborcsardi
Report min/max stats in read_parquet_metadata()
4a1fc57
@gaborcsardi
Improve docs
d6ab47b 
[ci skip]
@gaborcsardi
Update test snapshot
a669d90
@gaborcsardi
Writer: pass row group & page to callbacks
1b5d7ec 
So they can calculate min and max values easily.
@gaborcsardi
Min/max for PLAIN INT32 and DATE
55f126f
@gaborcsardi
Work around an unrelated crash
b6ada24
@gaborcsardi
Min/max for integer dictionaries
87a3689
@gaborcsardi
Fix a bug with string dicts and row groups
5dae2d8
@gaborcsardi
Test case for writing min/max for DATEs
12016fa
@gaborcsardi
Fix tests on older R
f71bf85
@gaborcsardi
Test min/max when writing uncompressed data
d999659
@gaborcsardi
Minor refactor
d46c91c 
Move function to better place.
@gaborcsardi
Fixes for writing min/max for integers
6262342 
- fix double -> int32 conversion
- fixes for NA values
@gaborcsardi
Fix writing min/max for TIME
4b128d0
@gaborcsardi
Write min/max for DOUBLE
98b6425
@gaborcsardi
Min/max for FLOAT
6dc4840
@gaborcsardi
Write min/max for INT64
649b7fb
@gaborcsardi
Fix metadata snapshot tests
16d3c1e
@gaborcsardi
Ooops, really fix snapshots
ca834ce
@gaborcsardi
Fix compilation on Linux
105c7d1
@gaborcsardi
Better error from min/max conversion
07338d1
@gaborcsardi
Implement min/max for various string (BYTE_ARRAY) types
3d0f67b
@gaborcsardi
Min/max for TIMESTAMP
f9e5428
@gaborcsardi
Document types without min/max write support
3f96d6c
@gaborcsardi
NEWS for writing min/max
e43fc37 
[CI skip]
@gaborcsardi gaborcsardi merged commit 495c0c1 into main Sep 22, 2024
13 checks passed
@gaborcsardi gaborcsardi deleted the feature/write-stats branch September 22, 2024 10:38
@gaborcsardi gaborcsardi mentioned this pull request Sep 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@gaborcsardi