Data buffer fixes #24

jgoizueta · 2016-07-29T15:56:34Z

Fixes #23

Deal with the case that the buffer for SQLGetData is too small, and also with missing trailing zeros.

To avoid loss of precision

Some types (e.g. PostGIS geometry columns which are mapped to text) may have huge sized. We now use type text for those large columns and limit the buffer size (which is no problem since GetData by parts was implemented)

* Non-supported column types (including bit-strings) are now omitted * bit(1) is interpreted as boolean * Conversion to bytea (longvarbinary) is handled properly The buf_used variable has been added to keep track of the buffer size in case of binary data (non null-terminated) so it will be available for future binary conversions.

jgoizueta · 2016-08-02T10:05:42Z

@rafatower please review. I think review may be easier looking at each commit in isolation

rafatower · 2016-08-02T11:07:46Z

odbc_fdw.c

+			 * * With options BoolsAsChar=0 this allows
+			 *   preserving boolean columns from pSQL ODBC.
+			 */
+		    appendStringInfo(sql_type, "boolean");


indentantion

please start using a linter or fomater of your choice.

I think that misalignment is due to how github handles tabs, but I sure do need a linter

rafatower · 2016-08-02T11:18:58Z

My understanding is that this fixes some issues found with pg driver, right?

What are the chances of breaking any of the working drivers with this patch? what can be done to mitigate that risk?

jgoizueta · 2016-08-02T15:20:45Z

This deals with problems found with Hive and PG (but could affect other drivers too), namely:

The driver reported size for a column may be insufficient for all its values
A number for which the read buffer is too small will be truncated and we cannot read it in parts
The reported size of column can be huge for varying-size columns
PG booleans are mapped to ODBC bit(1) or char(5) depending on a parameter
ODBC LongVarBinary type is formatted as hexadecimal when read through a C string (so this, unlike the bit strings can be read easily into a bytea).

It has been tested with PG, MySQL and Hive. (well, binary data has not been tested with Hive, but it probably didn't work as it was)

jgoizueta added 4 commits July 29, 2016 17:54

Handle partial SQLGetData results

3db51c0

Deal with the case that the buffer for SQLGetData is too small, and also with missing trailing zeros.

Use adequate minimum buffer size for numeric data

df59364

To avoid loss of precision

Limit size of varying columns and buffers

8149e32

Some types (e.g. PostGIS geometry columns which are mapped to text) may have huge sized. We now use type text for those large columns and limit the buffer size (which is no problem since GetData by parts was implemented)

rafatower reviewed Aug 2, 2016
View reviewed changes

jgoizueta added 2 commits August 2, 2016 17:52

Define macro names for magic values

f903e6a

Refactor ifs into switch/case

cb4a4c2

rafatower merged commit 3ea8a68 into master Aug 3, 2016

rafatower deleted the 23-getdata branch August 3, 2016 15:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data buffer fixes #24

Data buffer fixes #24

jgoizueta commented Jul 29, 2016

jgoizueta commented Aug 2, 2016

rafatower Aug 2, 2016

jgoizueta Aug 2, 2016

rafatower commented Aug 2, 2016

jgoizueta commented Aug 2, 2016 •

edited

Loading

Data buffer fixes #24

Data buffer fixes #24

Conversation

jgoizueta commented Jul 29, 2016

jgoizueta commented Aug 2, 2016

rafatower Aug 2, 2016

Choose a reason for hiding this comment

jgoizueta Aug 2, 2016

Choose a reason for hiding this comment

rafatower commented Aug 2, 2016

jgoizueta commented Aug 2, 2016 • edited Loading

jgoizueta commented Aug 2, 2016 •

edited

Loading