Resolving issue with obtain prior LSN number

tap-mssql 2.6.1 2024-10-09

• Resolving issue with call get the prior LSN number (passing in an unescaped table).

Adding new config settings conn_properties and enable_tds_logging

New setting for the wintersrd variant of tap-mssql.

Optional:

The "conn_properties" allows specific tweaking of database settings via SQL set statements to send to the database instance upon connection establishment. Can be a string or another kind of iterable of strings.

The default values set if this settings is not defined are:

"SET ARITHABORT ON; SET CONCAT_NULL_YIELDS_NULL ON; SET ANSI_NULLS ON; SET ANSI_NULL_DFLT_ON ON; SET ANSI_PADDING ON; SET ANSI_WARNINGS ON; SET ANSI_NULL_DFLT_ON ON; SET CURSOR_CLOSE_ON_COMMIT ON; SET QUOTED_IDENTIFIER ON; SET TEXTSIZE 2147483647;"

Example: override the built-in session properties supplied by pymssql by default, because one of the default settings (CURSOR_CLOSE_ON_COMMIT) is not available on PDW

{
  "conn_properties": "SET ARITHABORT ON; SET CONCAT_NULL_YIELDS_NULL ON; SET ANSI_NULLS ON; SET ANSI_NULL_DFLT_ON ON; SET ANSI_PADDING ON; SET ANSI_WARNINGS ON; SET ANSI_NULL_DFLT_ON ON; SET QUOTED_IDENTIFIER ON; SET TEXTSIZE 2147483647;"
}

Optional:

The "enable_tds_logging" When set it will dump out the underlying TDS driver logs. Useful for diagnosing issues if you are having connection issues to SQL Server databases. WARNING! this
does dump a lot of information and may log secure data, should be only used in Development
environments.

{
  "enable_tds_logging": true
}

Patching tap-mssql to remove CVES and bumping to latest versions

Patching tap-mssql to work with the latest versions of the pytest components. This is required to address CVES identified by dependabot.

tap-mssql 2.5.0 2024-08-26

• removing tox-poetry-install - Not well supported and not required for the pipelines.
• Updating integration tests variables to CAPS to support passing MSSQL Port
• tox-docker requires expose rather than ports syntax in tox.ini
• Bumping docker = "^6.1.3" -> "^7.1.0"
• Bumping pytest = "^7.0.0" -> "^8.3.2"
• Bumping pytest-cov = "^4.1.0" -> = "^5.0.0"
• Bumping mypy = "^1.10.1" -> = "^1.11.1"
• Bumping pytest-sugar = "^0.9.7" -> = "^1.0.0"
• Bumping pytest-datafiles = "^2.0" -> = "^3.0"
• Bumping tox-docker = "4.1.0" -> = "5.0.0"
• Bumping requests = "2.31.0" -> "2.32.3"
• Bumping tox = "^3.28.0" -> "^4.18.0"

Local testing has been as well as the integration testing to ensure the tap-mssql application still works correctly.

Patching tap-mssql (highlight singer-python with improved speed)

tap-mssql 2.4.0 2024-08-08

• Moving to patched version of Singer Framework plus using msgspec rather than orjson for JSON serialization speed.
• Adjusting tox file for test pipeline to point to correct location of the sqlcmd in the latest MSSQL docker image.
• Adjusting tox file. Added -No switch calling sqlcmd to allow user/password connection to MSSQL.
• Patching dependencies.
• Pinned Requests to "2.31.0". This is required as the tox pipeline fails with a Docker API error with requests "2.32.x".
• Explicitly setting python version 3.8 -> 3.12
• Moving to using recommended poetry-core for build. https://python-poetry.org/docs/pyproject/#poetry-and-pep-517
• NOTE: Recommend refactoring to avoid using tox-poetry-installer as restrictive dependencies are stopping tox and docker updates. This may resolve the requests/docker api issue.

Bug Fix: Change to preserve column ordering as per source

tap-mssql 2.3.1 2024-07-22

Bug Fix. Issue wintersrd#62 - change to column selection using lists instead of sets to preserve column ordering

Resolving pendulum datetime issue for incremental loads

Bug Fix. Change pendulum DateTime type to datetime.datetime as pymssql 2.3.0 is no longer compatible with query parameters as pendulum DateTime (pymssql/pymssql#889).

Singer Decimal Bug Fix for higher versions of MS SQL Server

Bug Fix. Enhancing singer-decimal output for numeric data to correctly output the correct datatype - string and precision.

Bug fix for datetimeoffset field

Bug Fix. Removing test code which slipped into the release, and adjusting offending code with correct dynamic column name.

Note: The change log points to 2.2.1 whereas the release number added was 2.2.2.

Support Datetimeoffset, Fix datetime2 and tinyint

tap-mssql 2.2.0 2023-08-23

This feature increases support for SQL Datatypes.

• Adds support for datetimeoffset - wintersrd#50
• Resolves issue with datetime2 rounding with high precision timestamps wintersrd#51
• Resolves the max and min range for tinyints. The current min and max are correct for MariaDB and MySQL only. MSSQL Server only supports
  positive integers (unsigned tinyint). wintersrd#2

Rolled all enhancement into upstream fork and re-sync'd

tap-mssql 2.1.0 2023-08-01

This is a number of new enhancements to extend capability and resolving a few bugs.

Features

• Adds use_singer_decimal setting, capability i.e. the ability to output decimal, float, and integer data as strings to avoid loss of precision. A target like this custom target-snowflake can decode this back into numeric data.
• Ordering columns by their ordinal position so the schema matches the database definition
• Support for new cursor_array_size parameter. This allows fetching records in batches
  speeding up network traffic. Recommend a setting of 10000 records.
• Fixing bug. We have encountered a scenario where a stream starts. It failed and there
  was no bookmark written.
  When the job is re-run the state file had currently_syncing for the given stream but with
  no bookmark written it didn't appear in the streams_with_state list. This small change will
  ensure that the currently_syncing_list will be populated whether there is a bookmark or not
  by filtering against the ordered_streams list which includes both streams_without_state and
  streams_with_state.
• Removed test for "INCREMENTAL" and not primary_keys.
  INCREMENTAL loads can be performed without primary keys as long as there
  is a replication key.
• Updating the CDC documentation
• Fixing bug. Where encountered a scenario where state lsn == max lsn.
  If the lsn coming from state is equal to the maximum lsn, then we should not be incrementing
  this lsn as it causes errors in the fn_cdc_get_all_changes function.
• Fixing bug. The _sdc_lsn_operation should be 2 for Inserts not Deletes.
• Fixing bug. pymssql 2.2.8 does not work. Excluding this version from pyproject.toml.
• Feature. Adding a logo to the tap.
• Resolving PR: wintersrd#36
• Resolving PR: wintersrd#16