Charges

[illia108]

Hey.
Rather a question than an issue. I got almost 3 times less charges in Stitch than I have in ReCharge. All other tables seems pretty close. Is there something I'm missing? Thanks

[Brobin]

I'm seeing the same thing. It seems to be missing charges. Probably due to the charge generation.

Specifically, after an April 6th billing run, new charges were generated for May 6th, but my stitch connection only picked up about half of the new charges

Edit: My thought is that charges are generated with "updated_at" in the past by the time they show up on the API. Nearly half of mine are not getting pulled in. To fix I had to do a full replication, which used up tons of my monthly row allocation.

[illia108]

I ended up building a custom integrator: ReCharge -> my app -> S3 -> BigQuery

[Brobin]

Gotcha. I already have a number of queries pointing to these tables, so I'd rather get this hooked up. Talking to Recharge support right now to test out my delayed charge generation hypothesis.

[Brobin]

Hmm,

2020-04-14 15:49:40,704Z    tap - INFO START Syncing: charges
2020-04-14 15:49:40,705Z    tap - INFO charges - Sync start since: 2020-04-14T10:48:28.000000Z, 
2020-04-14 15:49:40,705Z    tap - INFO URL for charges: https://api.rechargeapps.com/charges?page=1&updated_at_min=2020-04-14T10:48:28.000000Z&limit=250
2020-04-14 15:49:40,991Z    tap - INFO HTTP request to "charges" endpoint took 0.286s, returned status code 200
2020-04-14 15:49:41,028Z    tap - INFO replicated 6 records from "charges" endpoint
2020-04-14 15:49:41,028Z    tap - INFO Write state for stream: charges, value: 2020-04-14T10:59:04.000000Z
2020-04-14 15:49:41,028Z    tap - INFO charges - Synced Page 1 - Records 1 to 6
2020-04-14 15:49:41,028Z    tap - INFO Synced: charges, total_records: 6
2020-04-14 15:49:41,028Z    tap - INFO FINISHED Syncing: charges
2020-04-14 15:49:41,065Z target - INFO Serializing batch with 6 messages for table charges
2020-04-14 15:49:41,066Z target - INFO Sending batch of 23922 bytes to https://api.stitchdata.com/v2/import/batch
2020-04-14 15:49:41,091Z   main - INFO State update: changing bookmarks.charges from "2020-04-14T10:48:28.000000Z" to "2020-04-14T10:59:04.000000Z"
2020-04-14 15:49:41,091Z   main - INFO State update: removing currently_syncing was "charges"

But when I hit the API at that endpoint, I get 42 records

Edit: trying another to see if its still missing them. Got 6/44 :/

Edit 2: Maybe its a timezone issue? It looks like its setting the bookmarks in local time, but with GMT as the timezone, which makes up a 4 hour gap between EST.

[Brobin]

From Recharge Support

our DB is in EST time zone and when you add Z at the end you are moving time zone to UTC so you open 4 hour gap I believe

[Brobin]

Okay, so it turns out it is most likely a pagination/sorting issue on the recharge API side. Jeff is putting together a PR

[aviwebberman]

Appreciate y'all's work on this. I think I'm having the same issue with charges, just curious if there's been any update?

[Brobin]

I ended up reverting back to a custom api upload tap and just backfill today-2days every day. It doubles the rows loaded into the warehouse, but at least the data is correct.

[Brobin]

But #3 should have fixed it if the issues was pagination and bookmarking

[advinay]

hi @Brobin did you finally know what was going on ? I'm having the same problem as you had : missing some customer updates

[beam-ryan]

Hello - any updates on this? I am seeing the same issue. Stale charge dates via the recharge connector. A handful of rows have not updated since my initial bulk upload.