[Testing] Property based testing

[jaychia]

Summary

Start testing using hypothesis https://github.com/HypothesisWorks/hypothesis which should give us much stronger edge-case testing.

[jaychia]

Problem

Building a DataFrame library means that we have an infinite possible permutations of data and operations that we need to support. Building out tests manually that can cover every single test case is unfeasible, but important to ensure that users do not encounter unexpected behavior.

Solution

Unit test suites with simple assertions

1. Sort: all rows in the dataframe are correctly ordered
2. Filter: no rows in the dataframe violate the predicate
3. Limit: the total number of rows in the dataframe does not exceed the limit defined
4. Projection: the new column(s) produced are correctly calculated from the input columns from the previous DataFrame
5. Aggregations: the results produced are equivalent to running the aggregation locally on the single column after collection
6. Groupby Aggregations: the results produced are equivalent to running the aggregation locally on the groupby keys/agg column

Inputs to these test suites are generated using strategies

1. We define dtypes and properties for each (hashable, size [fixed/variable], ranges, truthfulness (bool("") is False) etc)
2. We define properties for columns of data (uniqueness, all nulls, all NaNs, empty, sorted-ness etc)
3. We define properties of DataFrames (number of rows, number of partitions, permutations on dtypes of columns, read source [in-memory vs CSV vs Parquet])
4. We define properties of DataFrameTransformations (partition-preserving, rowcount-preserving, etc) see: https://hypothesis.works/articles/rule-based-stateful-testing/

Now, given the unit test suite we have from before, we can pass in different DataFrameTransformation(DataFrames) into each test suite and check for correctness across various permutations of transformations/dataframes/dtypes.