add decimal validator #763

davidhewitt · 2023-07-11T15:26:10Z

Change Summary

Adds a Rust implementation of decimal validation and serialization. Needs more tests and benchmarks to confirm it's worth going for this instead of pydantic/pydantic#6327

Related issue number

N/A

Checklist

Unit tests for the changes exist
Documentation reflects the changes where applicable
Pydantic tests pass with this pydantic-core (except for expected changes)
My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @dmontagu

codspeed-hq · 2023-07-11T15:37:15Z

CodSpeed Performance Report

Merging #763 will degrade performances by 43.74%

_{Comparing dh/decimal (997cbde) with main (c94fa15)}

Summary

🔥 1 improvements
❌ 1 regressions
✅ 133 untouched benchmarks

🆕 3 new benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`main`	`dh/decimal`	Change
🔥	`test_dont_raise_error`	36 µs	25.6 µs	+40.74%
❌	`test_tagged_union_int_keys_python`	33.3 µs	59.1 µs	-43.74%
🆕	`test_decimal_from_string_core`	N/A	67.2 µs	N/A
🆕	`test_decimal_from_string_limit`	N/A	20 µs	N/A
🆕	`test_decimal_from_string_pyd`	N/A	65.2 µs	N/A

samuelcolvin

looks like a good start.

The easiest way to check performance would be to copy the validator constructed by pydantic and add it to benchmarks here, together with the function from pydantic. Then run benchmarks. You can probably remove that once we're sure it's faster.

src/input/input_json.rs

src/input/input_python.rs

src/input/input_json.rs

src/serializers/type_serializers/decimal.rs

src/validators/decimal.rs

samuelcolvin

couple more points.

src/input/input_python.rs

src/validators/decimal.rs

davidhewitt · 2023-07-20T15:08:01Z

I think the implementation here for the validator is now complete and is as optimized as can be. Still to do:

Serializer feedback
More tests
Create known error types

I will park this until we make a decision whether to move forwards with it. The alternative is pydantic/pydantic#6781

codecov · 2023-07-24T15:45:19Z

Codecov Report

Merging #763 (997cbde) into main (c94fa15) will decrease coverage by 0.24%.
The diff coverage is 80.92%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #763      +/-   ##
==========================================
- Coverage   94.06%   93.82%   -0.24%     
==========================================
  Files         102      104       +2     
  Lines       15023    15365     +342     
  Branches       25       25              
==========================================
+ Hits        14131    14416     +285     
- Misses        886      943      +57     
  Partials        6        6

Files Changed	Coverage Δ
src/errors/mod.rs	`92.30% <ø> (ø)`
src/serializers/shared.rs	`79.56% <0.00%> (-0.98%)`	⬇️
src/validators/decimal.rs	`71.09% <71.09%> (ø)`
src/serializers/type_serializers/decimal.rs	`93.02% <93.02%> (ø)`
src/input/input_python.rs	`98.05% <96.96%> (-0.10%)`	⬇️
python/pydantic_core/core_schema.py	`96.77% <100.00%> (+0.06%)`	⬆️
src/errors/types.rs	`99.50% <100.00%> (+0.01%)`	⬆️
src/input/input_abstract.rs	`90.40% <100.00%> (+0.40%)`	⬆️
src/input/input_json.rs	`91.63% <100.00%> (+0.30%)`	⬆️
src/validators/mod.rs	`97.74% <100.00%> (+0.01%)`	⬆️

... and 3 files with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c94fa15...997cbde. Read the comment docs.

JeanArhancet · 2023-07-26T08:06:33Z

src/validators/decimal.rs

+    input: &'a impl Input<'a>,
+    decimal_type: &'a PyType,
+) -> ValResult<'a, &'a PyAny> {
+    decimal_type.call1((arg,)).map_err(|e| {


Concerning this part maybe we can do the calculation in rust and set the attributes of the python class Decimal: __slots__ = ('_exp','_int','_sign', '_is_special') WDYT @davidhewitt ?

In this PR I'm trying to keep the same logic as pydantic just move the implementation. I would definitely be interested in this as a future improvement. I believe @samuelcolvin had some previous thoughts on how the parsing might be done (by looking at the digits as a string), not sure if there was any implementation put somewhere?

python/pydantic_core/core_schema.py

dmontagu · 2023-07-27T19:09:05Z

src/input/input_json.rs

+    fn strict_decimal(&'a self, decimal_type: &'a PyType) -> ValResult<&'a PyAny> {
+        let py = decimal_type.py();
+        match self {
+            JsonInput::Float(f) => create_decimal(PyString::new(py, &f.to_string()), self, decimal_type),


Is there a way for us to make sure that when parsing numeric JSON into a decimal it is treated as a string rather than loaded into an f64 or whatever? I think @samuelcolvin said he did something like this in speedate

(Maybe it's already doing that, not sure, but it's probably worth checking that we load many-digit JSON numbers into decimal properly)

src/input/input_python.rs

dmontagu · 2023-07-27T19:17:55Z

tests/serializers/test_decimal.py

Seems to me worth adding a test that Decimal('123456789123456789123456789.123456789123456789123456789') (or some other similarly high-precision decimal) gets serialized properly (at least to JSON).

tests/validators/test_decimal.py

dmontagu · 2023-07-27T19:24:00Z

I made various comments; I think at least some merit changes (e.g., removing the print line), and I do think there are some logic changes worth making (i.e., allowing proper subclasses of Decimal to pass validation unmodified), but happy to defer to your judgement.

For hooky: please update

davidhewitt · 2023-08-07T10:18:46Z

Thanks for the thorough review; that definitely caught a couple of bugs!

I've rebased this PR and added a commit to make the desired review changes.

davidhewitt · 2023-08-08T11:21:54Z

please review

addressed

davidhewitt · 2023-08-09T22:43:15Z

Many thanks, I rebased and dropped the commit which pointed the integration tests at pydantic/pydantic#6858 👍

samuelcolvin previously requested changes Jul 13, 2023

View reviewed changes

pydantic-hooky bot added the awaiting author revision label Jul 13, 2023

pydantic-hooky bot assigned davidhewitt Jul 13, 2023

samuelcolvin reviewed Jul 13, 2023

View reviewed changes

src/input/input_python.rs Outdated Show resolved Hide resolved

src/input/input_python.rs Outdated Show resolved Hide resolved

davidhewitt mentioned this pull request Jul 18, 2023

Add uuid validator #772

Merged

4 tasks

davidhewitt force-pushed the dh/decimal branch 2 times, most recently from e8254f0 to 5bb39f3 Compare July 19, 2023 17:04

This was referenced Jul 20, 2023

fast-path checking finite decimals pydantic/pydantic#6769

Merged

use vectorcall calling convention for performance #792

Closed

samuelcolvin reviewed Jul 20, 2023

View reviewed changes

src/validators/decimal.rs Outdated Show resolved Hide resolved

davidhewitt mentioned this pull request Jul 20, 2023

add profiling make target and instructions #793

Merged

4 tasks

davidhewitt force-pushed the dh/decimal branch from 5bb39f3 to e88e436 Compare July 20, 2023 14:11

davidhewitt mentioned this pull request Jul 20, 2023

rework decimal validator for better performance pydantic/pydantic#6781

Closed

5 tasks

davidhewitt force-pushed the dh/decimal branch 3 times, most recently from 1586486 to a088741 Compare July 24, 2023 15:40

davidhewitt force-pushed the dh/decimal branch 2 times, most recently from ed0bd96 to 2224e98 Compare July 25, 2023 16:49

davidhewitt mentioned this pull request Jul 25, 2023

use pydantic-core decimal pydantic/pydantic#6858

Closed

5 tasks

davidhewitt force-pushed the dh/decimal branch 2 times, most recently from 2fa96b3 to 0f7ca0d Compare July 25, 2023 19:18

JeanArhancet reviewed Jul 26, 2023

View reviewed changes

davidhewitt force-pushed the dh/decimal branch from a31b90c to 606da42 Compare July 26, 2023 16:06

davidhewitt mentioned this pull request Jul 26, 2023

optimize Python float validation #826

Merged

4 tasks

davidhewitt force-pushed the dh/decimal branch 3 times, most recently from 9b714c9 to 0b08d2c Compare July 27, 2023 12:11