fix(2024-06-01): use registered domain for third parties to count sub-domains

[Kporal]

This PR is here to expose and suggest a fix to inaccurate computed data when third party runs a script from dynamic subdomain.

What's wrong here is that some third parties use a dynamic subdomain to serve its main script on websites (e.g <dynamicThirdPartySubdomain>.domain.com). Some of these subdomain scripts are saved under observed-domains JSON file as results of the sql/all-observed-domains-query.sql query but analyzing http archive database we found a lot that are ignored because of number of occurrences (less than 50 cf SQL query having clause).

In this MR we've decided to group and count on registered domain NET.REG_DOMAIN instead of hostname NET.HOST to include all subdomains and do not skip "dynamic" subdomains (as they participate in the average execution time).

We've also cleaned unused select in query (COUNT(0) as totalOccurences nested query).

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
third-party-web	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 7, 2024 1:44pm

[patrickhulce]

thanks very much for raising @Kporal!

this isn't the behavior we want though. this would erase all distinction between shared domains like cloudfront. we define *.domain.tld differently from foo.domain.tld in the entities.json for a reason.

to address this gap, we'd need to do something much more sophisticated like group on the patterns defined in entities.json (or group on both domain and tld+1 for post-processing)

do you have a sense of how large the problem is? e.g. how many entities data change by how much with this?

[patrickhulce]

we shouldn't edit historical data we've already delivered, we should just update a new one for July

[Nigui]

Hello 👋
PR #228 is here to compute fresh data with a new query fixing mentioned issue.
If you want to have a look, would be great. Thank you.