discovery 403 (forbidden) error

[haiakab]

Discussed in #2365

^{Originally posted by haiakab August 16, 2023}
Hi,
multiple subscriptions defined under our Azure account,
subscriptions have the same authentication (mode=UserAssignedManagedIdentity)
for all of the subscription, discovery is created as well - container is running, curl command returns Ok status (200).
scrapers were defined for the subscriptions with multiple metrics, but it's not working for some of the scrapers - getting 403 (forbidden) error for some of the subscriptions.
I see that this error is related to authentication- but the same auth is working for other subscriptions
what can be the issue?

error log:
Failed to scrape resource collection virtual-machine-landscape: Response status code does not indicate success: 403 (Forbidden).
System.Net.Http.HttpRequestException: Response status code does not indicate success: 403 (Forbidden).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.SendGetRequestAsync(String uriPath) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 63
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryClient.GetAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryClient.cs:line 44
at Promitor.Agents.Scraper.Discovery.ResourceDiscoveryRepository.GetResourceDiscoveryGroupAsync(String resourceDiscoveryGroupName) in /src/Promitor.Agents.Scraper/ResourceDiscoveryRepository.cs:line 25
at Promitor.Agents.Scraper.Scheduling.ResourceDiscoveryGroupScrapingJob.ExecuteAsync(CancellationToken cancellationToken) in /src/Promitor.Agents.Scraper/ResourceDiscoveryGroupScrapingJob.cs:line 80