Implement table_reset test in the new framework #161

bhuvana-talend · 2023-06-13T21:32:20Z

Description of change

https://jira.talendforge.org/browse/TDL-22960

Rewrote this test from test_salesforce_incremental_table_reset.py to use the new framework.
It also provides the opportunity to test the new testcase table_reset_test.py in the new framework

QA steps

automated tests passing
manual qa steps passing (list below)

Risks

Rollback steps

revert this branch

luandy64 · 2023-06-15T14:46:57Z

tests/test_table_reset.py

+class SFTableResetTest(TableResetTest, SFBaseTest):
+    """tap-salesforce Table reset test implementation"""
+
+    salesforce_api = 'BULK'


This should probably go in SFBaseTest since it's used by SFBaseTest.get_properties

We should probably set a default in the base case, which can be overriden by setting the property here if needed. Also, do we need to run this test for both BULK and REST? We could import TableResetTest again in another module and class and set it in that test to REST.

moved this to sfbase.py

luandy64 · 2023-06-15T14:56:40Z

tests/test_table_reset.py

+
+    salesforce_api = 'BULK'
+    reset_stream = 'User'
+    replication_method = "INCREMENTAL"


See my comment in the framework about this

This already exists in the tap-tester base_case. No need to define it again here.

INCREMENTAL = "INCREMENTAL" # replication method value in metadata FULL_TABLE = "FULL_TABLE" # replication method value in metadata

removed this

luandy64 · 2023-06-15T14:57:45Z

tests/test_table_reset.py

+    @staticmethod
+    def convert_state_to_utc(date_str):
+        """
+        Convert a saved bookmark value of form '2020-08-25T13:17:36-07:00' to
+        a string formatted utc datetime,
+        in order to compare aginast json formatted datetime values
+        """
+        date_object = dateutil.parser.parse(date_str)
+        date_object_utc = date_object.astimezone(tz=pytz.UTC)
+        return datetime.datetime.strftime(date_object_utc, "%Y-%m-%dT%H:%M:%SZ")


This doesn't seem like a salesforce-table-reset specific function. Should we move this into the framework?

This is similar to parse_date and timedelta_formatted. We should put this in base case in the same section or see if we can enhance that functionality to also convert to UTC.

moved this to framework base_case.py
This method in turn calls parse_date and then converts to utc

…alesforce_api

HarrisonMarcRose · 2023-06-26T20:16:39Z

tests/sfbase.py

+    @staticmethod
+    def get_unsupported_by_rest_api():
+        """The streams listed here are not supported by the REST API"""
+        unsupported_streams = {
+            'Announcement',
+            'CollaborationGroupRecord',
+            'ContentDocumentLink',
+            'ContentFolderMember',
+            'DataStatistics',
+            'EntityParticle',
+            'FieldDefinition',
+            'FlexQueueItem',
+            'IdeaComment',
+            'OwnerChangeOptionInfo',
+            'PicklistValueInfo',
+            'PlatformAction',
+            'RelationshipDomain',
+            'RelationshipInfo',
+            'SearchLayout',
+            'SiteDetail',
+            'UserEntityAccess',
+            'UserFieldAccess',
+            'Vote',
+            'RecordActionHistory',
+            'FlowVersionView',
+            'FlowVariableView',
+            'AppTabMember',
+            'ColorDefinition',
+            'IconDefinition',
+        }
+
+        return unsupported_streams
+
+    def get_unsupported_by_bulk_api(self):
+        unsupported_streams_rest = self.get_unsupported_by_rest_api()
+        unsupported_streams_bulk_only= {
+            'AcceptedEventRelation',
+            'AssetTokenEvent',
+            'AttachedContentNote',
+            'CaseStatus',
+            'ContentFolderItem',
+            'ContractStatus',
+            'DeclinedEventRelation',
+            'EventWhoRelation',
+            'PartnerRole',
+            'QuoteTemplateRichTextData',
+            'RecentlyViewed',
+            'SolutionStatus',
+            'TaskPriority',
+            'TaskWhoRelation',
+            'TaskStatus',
+            'UndecidedEventRelation',
+            'OrderStatus',
+            'WorkOrderStatus',
+            'WorkOrderLineItemStatus',
+            'ServiceAppointmentStatus',
+            'ServiceAppointmentStatus',
+            'FieldSecurityClassification',
+            # BUG_TODO | the following streams are undocumented
+            'WorkStepStatus',
+            'ShiftStatus',
+        }
+
+        return unsupported_streams_bulk_only | unsupported_streams_rest
+
+    def is_unsupported_by_rest_api(self, stream):
+        """returns True if stream is unsupported by REST API"""
+
+        return stream in self.get_unsupported_by_rest_api()
+
+    def is_unsupported_by_bulk_api(self, stream):
+        """
+        returns True if stream is unsupported by BULK API
+
+        BULK API does not support any streams that are unsupported by the REST API and
+        in addition does not support the streams listed below.
+        """
+        return stream in self.get_unsupported_by_bulk_api()


I thought we had discussed in slack that we were going to remove this as it isn't necessary yet and we have it in the other base and then we can add it in if needed when a test requires it as we transition to the new framework.

yeah, missed it, Removed this method now

HarrisonMarcRose

left a comment

HarrisonMarcRose · 2023-07-06T17:47:25Z

tests/sfbase.py

+    def get_unsupported_by_rest_api():
+        """The streams listed here are not supported by the REST API"""
+        unsupported_streams = {
+            'Announcement',
+            'CollaborationGroupRecord',
+            'ContentDocumentLink',
+            'ContentFolderMember',
+            'DataStatistics',
+            'EntityParticle',
+            'FieldDefinition',
+            'FlexQueueItem',
+            'IdeaComment',
+            'OwnerChangeOptionInfo',
+            'PicklistValueInfo',
+            'PlatformAction',
+            'RelationshipDomain',
+            'RelationshipInfo',
+            'SearchLayout',
+            'SiteDetail',
+            'UserEntityAccess',
+            'UserFieldAccess',
+            'Vote',
+            'RecordActionHistory',
+            'FlowVersionView',
+            'FlowVariableView',
+            'AppTabMember',
+            'ColorDefinition',
+            'IconDefinition',
+        }
+
+        return unsupported_streams
+
+    def get_unsupported_by_bulk_api(self):
+        unsupported_streams_rest = self.get_unsupported_by_rest_api()
+        unsupported_streams_bulk_only= {
+            'AcceptedEventRelation',
+            'AssetTokenEvent',
+            'AttachedContentNote',
+            'CaseStatus',
+            'ContentFolderItem',
+            'ContractStatus',
+            'DeclinedEventRelation',
+            'EventWhoRelation',
+            'PartnerRole',
+            'QuoteTemplateRichTextData',
+            'RecentlyViewed',
+            'SolutionStatus',
+            'TaskPriority',
+            'TaskWhoRelation',
+            'TaskStatus',
+            'UndecidedEventRelation',
+            'OrderStatus',
+            'WorkOrderStatus',
+            'WorkOrderLineItemStatus',
+            'ServiceAppointmentStatus',
+            'ServiceAppointmentStatus',
+            'FieldSecurityClassification',
+            # BUG_TODO | the following streams are undocumented
+            'WorkStepStatus',
+            'ShiftStatus',
+        }
+
+        return unsupported_streams_bulk_only | unsupported_streams_rest


If we aren't using these methods in our new test (which it looks like we aren't) we should remove this.

Removed these unused methods

HarrisonMarcRose · 2023-07-06T17:50:29Z

tests/test_table_reset.py

+    """tap-salesforce Table reset test implementation
+    Currently tests only the stream with Incremental replication method"""
+
+    #reset_stream = 'User'


Don't need this since we have it below. Although, if you make this a property then this would work and you could remove the other one. I think that looks better but either way works.

Made it as a property

…orted_by_bulk_api

Implement table_reset test in the new framework

94d22b9

luandy64 reviewed Jun 15, 2023

View reviewed changes

bhuvana-talend added 3 commits June 16, 2023 15:24

Removed expected_replication_keys as it is in framework base, added s…

85c7a12

…alesforce_api

Removed replication_method and convert_state_to_utc

ff692f2

Added comment

91b33a5

HarrisonMarcRose reviewed Jun 26, 2023

View reviewed changes

bhuvana-talend added 2 commits June 27, 2023 00:03

removed is_unsupported_by_rest_api and is_unsupported_by_bulk_api

fcb1e41

defined reset_stream() method

47e4fed

HarrisonMarcRose reviewed Jul 6, 2023

View reviewed changes

bhuvana-talend added 2 commits July 6, 2023 21:57

removed the unused methods get_unsupported_by_rest_api and get_unsupp…

9d537e3

…orted_by_bulk_api

made reset_stream a property

0a5142e

HarrisonMarcRose approved these changes Jul 7, 2023

View reviewed changes

bhuvana-talend and others added 2 commits July 7, 2023 07:34

Merge branch 'master' into tdl-22960

a0922e0

Update sfbase to match base for expected discovery streams

780f125

bhtowles merged commit 5d46c8e into master Jul 12, 2023
2 checks passed

bhtowles deleted the tdl-22960 branch July 12, 2023 13:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement table_reset test in the new framework #161

Implement table_reset test in the new framework #161

bhuvana-talend commented Jun 13, 2023

luandy64 Jun 15, 2023

HarrisonMarcRose Jun 15, 2023

bhuvana-talend Jun 16, 2023

luandy64 Jun 15, 2023

HarrisonMarcRose Jun 15, 2023

bhuvana-talend Jun 16, 2023

luandy64 Jun 15, 2023

HarrisonMarcRose Jun 15, 2023

bhuvana-talend Jun 16, 2023

HarrisonMarcRose Jun 26, 2023

bhuvana-talend Jun 27, 2023

HarrisonMarcRose left a comment

HarrisonMarcRose Jul 6, 2023

bhuvana-talend Jul 6, 2023

HarrisonMarcRose Jul 6, 2023

bhuvana-talend Jul 6, 2023

Implement table_reset test in the new framework #161

Implement table_reset test in the new framework #161

Conversation

bhuvana-talend commented Jun 13, 2023

Description of change

QA steps

Risks

Rollback steps

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HarrisonMarcRose left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment