Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Promitor scraper fails to find resource mapping for Microsoft.Network #2019

Open
Mahagon opened this issue Apr 28, 2022 · 15 comments
Open

Promitor scraper fails to find resource mapping for Microsoft.Network #2019

Mahagon opened this issue Apr 28, 2022 · 15 comments
Assignees
@tomkerkhove
Labels
bug Something isn't working

Comments

@Mahagon
Copy link

Mahagon commented Apr 28, 2022

Report

The promitor scraper seems to be unable to fetch the NetworkInterface Metrics in Azure Monitor

Expected Behavior

Scraping NetworkInterface Metrics from Azure Monitor

Actual Behavior

Fatal log msgs in promitor scraper Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces

Steps to Reproduce the Problem

  1. Configure NetworkInterface ResourceDiscoveryGroup in promitor discovery agent
  2. Scrape Metrics from NetworkInterface ind promitor agent scraper through ResourceDiscoveryGrooup

Component

Scraper

Version

2.7.3

Configuration

Configuration:

---
repositories:
  - name: promitor
    url: https://charts.promitor.io/
  - name: bedag
    url: https://bedag.github.io/helm-charts

releases:
  - name: promitor-scraper
    namespace: promitor
    chart: promitor/promitor-agent-scraper
    values:
      - azureAuthentication:
          mode: UserAssignedManagedIdentity
          identity:
            id: {{ exec "terraform" (list "-chdir=../.." "output" "-json" "promitor_scraper_client_id") }}
            binding: promitor-scraper-identity
      - azureMetadata:
          tenantId: {{ requiredEnv "ARM_TENANT_ID" }}
          subscriptionId: {{ requiredEnv "ARM_SUBSCRIPTION_ID" }}
          resourceGroupName: REDACTED
      - resourceDiscovery:
          enabled: true
          host: promitor-agent-resource-discovery
          port: 8889
      - metricSinks:
          prometheusScrapingEndpoint:
            enabled: true
            serviceMonitor:
              enabled : true
              namespace: promitor
          enableServiceDiscovery: true
      - metrics:
          - name: azure_network_interface_bytes_received_rate
            description: "Number of bytes the Network Interface received."
            resourceType: NetworkInterface
            azureMetricConfiguration:
              metricName: BytesReceivedRate
              aggregation:
                type: Total
            resourceDiscoveryGroups:
              - name: network-interfaces-landscape
          - name: azure_network_interface_bytes_sent_rate
            description: "Number of bytes the Network Interface sent."
            resourceType: NetworkInterface
            azureMetricConfiguration:
              metricName: BytesSentRate
              aggregation:
                type: Total
            resourceDiscoveryGroups:
              - name: network-interfaces-landscape
          - name: azure_network_interface_packets_received_rate
            description: "Number of packets the Network Interface received."
            resourceType: NetworkInterface
            azureMetricConfiguration:
              metricName: PacketsReceivedRate
              aggregation:
                type: Total
            resourceDiscoveryGroups:
              - name: network-interfaces-landscape
          - name: azure_network_interface_packets_sent_rate
            description: "Number of packets the Network Interface sent."
            resourceType: NetworkInterface
            azureMetricConfiguration:
              metricName: PacketsSentRate
              aggregation:
                type: Total
            resourceDiscoveryGroups:
              - name: network-interfaces-landscape
      - tolerations:
          - key: "CriticalAddonsOnly"
            operator: "Exists"
            effect: "NoSchedule"
  - name: promitor-discovery
    namespace: promitor
    chart: promitor/promitor-agent-resource-discovery
    values:
      - azureAuthentication:
          mode: UserAssignedManagedIdentity
          identity:
            id: {{ exec "terraform" (list "-chdir=../.." "output" "-json" "promitor_discovery_client_id") }}
            binding: promitor-discovery-identity
      - azureLandscape:
          cloud: Global
          tenantId: {{ requiredEnv "ARM_TENANT_ID" }}
          subscriptions:
            - {{ requiredEnv "ARM_SUBSCRIPTION_ID" }}
      - prometheus:
          serviceMonitor:
            enabled: true
            namespace: promitor
      - resourceDiscoveryGroups:
          - name: network-interfaces-landscape
            type: NetworkInterface
      - tolerations:
          - key: "CriticalAddonsOnly"
            operator: "Exists"
            effect: "NoSchedule"
  - name: promitor-scraper-identity
    namespace: promitor
    chart: ../../charts/azure-identity
    values:
      - identityName: promitor-scraper-identity
      - resourceId: {{ exec "terraform" (list "-chdir=../.." "output" "-json" "promitor_scraper_resource_id") }}
      - clientId: {{ exec "terraform" (list "-chdir=../.." "output" "-json" "promitor_scraper_client_id") }}
      - selector: promitor-scraper-identity
  - name: promitor-discovery-identity
    namespace: promitor
    chart: ../../charts/azure-identity
    values:
      - identityName: promitor-discovery-identity
      - resourceId: {{ exec "terraform" (list "-chdir=../.." "output" "-json" "promitor_discovery_resource_id") }}
      - clientId: {{ exec "terraform" (list "-chdir=../.." "output" "-json" "promitor_discovery_client_id") }}
      - selector: promitor-discovery-identity

Logs

2022-04-28T08:10:05.445290049Z stdout F [08:10:05 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/{PODNAME}.nic.ee13ddf1-772a-4855-822a-a5ba2be2a9ea, traceId: {600df2dd-b4fb-46f5-85b3-f241561f791c}
2022-04-28T08:10:04.545832034Z stdout F [08:10:04 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/{PODNAME}.nic.c88992f6-5ecd-44d1-b332-84795d619807, traceId: {be611765-5fbd-4b71-aafd-01e0a4e814e4}
2022-04-28T08:10:03.869797215Z stdout F [08:10:03 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/{PODNAME}.nic.327f47ff-6dae-4bd9-9cca-729917de5d40, traceId: {8f662120-c90c-478e-bc4b-b6b7f9bf6cf5}
2022-04-28T08:10:03.838297078Z stdout F [08:10:03 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/{PODNAME}.nic.81f30dfd-cfeb-46d9-aae9-5cb99f7355bd, traceId: {726b500d-7b77-42f2-90c3-4e49deae33d2}
2022-04-28T08:10:03.740632441Z stdout F [08:10:03 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/{PODNAME}.nic.c88992f6-5ecd-44d1-b332-84795d619807, traceId: {de2b765a-bbe8-4575-b02b-bcfe05f55a81}
2022-04-28T08:10:03.612416274Z stdout F [08:10:03 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/{PODNAME}.nic.ac66fc98-779a-421e-a4fe-98736cc3d489, traceId: {8c173ef7-156e-46c2-b617-322f5b107361}

Platform

Microsoft Azure

Contact Details

No response
@Mahagon Mahagon added the bug Something isn't working label Apr 28, 2022
@tomkerkhove
Copy link
Owner

Are these the only logs that are available? Also, are these found through discovery?

@Mahagon
Copy link
Author

Mahagon commented Apr 28, 2022 •
edited
Loading

They are found through discovery, yes :)
I copied (hopefully) a complete interval of logs:

Spoiler with logs 
2022-04-28T08:00:24.118906749Z stdout F [08:00:24 INF] Request starting HTTP/1.1 GET http://xx.xxx.x.xxx:5000/api/v1/health?includeDependencies=false - -
2022-04-28T08:00:20.229247838Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001j as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.229074637Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.229056637Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.229052337Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001g as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.229047037Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001l as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.229039137Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001m as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.228983236Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-user-33133238-vmss00001k as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:20.228881536Z stdout F [08:00:20 INF] Found value null for metric node_network_in_bytes with dimension aks-system-34776058-vmss00000i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:19.065700651Z stdout F [08:00:19 INF] Request finished HTTP/1.1 GET http://xx.xxx.x.xxx:5000/api/v1/health?includeDependencies=false - - - 200 60 application/json;+charset=utf-8 3.5161ms
2022-04-28T08:00:19.064899845Z stdout F [08:00:19 INF] Executed action Promitor.Agents.Scraper.Controllers.v1.HealthController.Get (Promitor.Agents.Scraper) in 0.2421ms
2022-04-28T08:00:19.063799036Z stdout F [08:00:19 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.Scraper.Controllers.v1.HealthController (Promitor.Agents.Scraper).
2022-04-28T08:00:19.062264225Z stdout F [08:00:19 INF] Request starting HTTP/1.1 GET http://xx.xxx.x.xxx:5000/api/v1/health?includeDependencies=false - -
2022-04-28T08:00:17.719889694Z stdout F [08:00:17 INF] Request finished HTTP/1.1 GET http://xx.xxx.x.xxx:5000/metrics - - - 200 - text/plain;+version=0.0.4 4.8503ms
2022-04-28T08:00:17.715055957Z stdout F [08:00:17 INF] Request starting HTTP/1.1 GET http://xx.xxx.x.xxx:5000/metrics - -
2022-04-28T08:00:15.761896673Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Unknown as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.761892673Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.761873172Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListContainers as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.761869272Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.761865272Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobTags as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.761860172Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.761826772Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757826542Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001j as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757822742Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001m as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757819042Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001g as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757815142Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757811142Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001l as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757807242Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757802941Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-user-33133238-vmss00001k as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757797841Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-system-34776058-vmss00000h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.757764641Z stdout F [08:00:15 INF] Found value null for metric node_cpu_usage_millicores with dimension aks-system-34776058-vmss00000i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.743818335Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerServiceMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.743566633Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721290863Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlob as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721286163Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension DeleteBlob as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721265263Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721244063Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension BlobPreflightRequest as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721239863Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721228263Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.721220463Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.720949861Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.65128823Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.650618925Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Logoff as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.64996972Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.649188014Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Flush as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.648518309Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SessionSetup as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.647778203Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.647132798Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetInfo as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.646423093Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension TreeDisconnect as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.645824388Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Read as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.645156683Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetShareAcl as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.644490578Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension DeleteShare as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.643794373Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetShareMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.643080667Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension CreateShare as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.642410262Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetShareProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.64078025Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension TreeConnect as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.640047344Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetShareProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.639415539Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Ioctl as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.638736734Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Write as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.638551533Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.637198522Z stdout F [08:00:15 INF] Found value 67320 for metric azure_storage_account_bytes_received_rate with dimension Close as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.636247515Z stdout F [08:00:15 INF] Found value 79560 for metric azure_storage_account_bytes_received_rate with dimension QueryInfo as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.634907805Z stdout F [08:00:15 INF] Found value 116280 for metric azure_storage_account_bytes_received_rate with dimension Create as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516668604Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListContainers as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516662804Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerACL as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516658304Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension CreateContainer as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516642104Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetContainerMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516637504Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetContainerACL as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516633504Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516628704Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516624504Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516620704Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerServiceMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516616104Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516611903Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension BlobPreflightRequest as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516593703Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Unknown as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516589803Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension CreateTable as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516585503Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516581403Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516576803Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516570503Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.516515203Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.44404315Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001j as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.44403955Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001m as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.44403505Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001g as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.44401595Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.442080636Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001l as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.441959235Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001k as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.441836134Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-user-33133238-vmss00001h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.435837388Z stdout F [08:00:15 INF] Found value null for metric node_memory_rss_bytes with dimension aks-system-34776058-vmss00000i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378806053Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ReleaseBlobLease as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378802153Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension AcquireBlobLease as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378794753Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension SetBlobMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378765553Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378743353Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378738053Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.378703453Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlob as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.37841515Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlob as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.367624568Z stdout F [08:00:15 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:15.114707541Z stdout F [08:00:15 INF] Found value 0 for metric azure_key_vault_api_result_count with aggregation interval 00:05:00
2022-04-28T08:00:14.961928976Z stdout F [08:00:14 INF] Found value null for metric azure_key_vault_api_latency with aggregation interval 00:05:00
2022-04-28T08:00:14.863199124Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censored-dev-key-vault.nic.c17668e1-49aa-4aa5-98f0-31b4ab44e0e6, traceId: {ab58938b-289b-465f-ad3a-0b188443b886}
2022-04-28T08:00:14.834330604Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs4.nic.1e6001e7-12d2-4d6d-98c9-440e49e7bdd3, traceId: {63c88455-74ef-42a8-ab3c-14f3db706ace}
2022-04-28T08:00:14.74060369Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001j as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739540482Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001m as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739536482Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001g as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739525882Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001l as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739521882Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739517982Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001k as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739513382Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-system-34776058-vmss00000h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739506582Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-system-34776058-vmss00000i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.739470881Z stdout F [08:00:14 INF] Found value null for metric node_memory_working_set_percentage with dimension aks-user-33133238-vmss00001h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.738547774Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddeb27.nic.ac66fc98-779a-421e-a4fe-98736cc3d489, traceId: {46bfb30e-b370-40d6-8472-a7f604f04213}
2022-04-28T08:00:14.701845595Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs3.nic.327f47ff-6dae-4bd9-9cca-729917de5d40, traceId: {0f9e8c56-65e6-4e54-8982-d5ec7994ffa1}
2022-04-28T08:00:14.687506685Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs1.nic.88b100e5-0743-43cd-a592-e0ac661c4872, traceId: {4afb39b7-a4c7-462e-aa47-16fa40f5576a}
2022-04-28T08:00:14.563997844Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs2.nic.c88992f6-5ecd-44d1-b332-84795d619807, traceId: {86084e68-979b-480e-a8ab-e4d58f1884cf}
2022-04-28T08:00:14.494192312Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/redactedazurefilesdev.nic.ee13ddf1-772a-4855-822a-a5ba2be2a9ea, traceId: {4e898d7f-aae5-4163-966b-1850b3a7e2b3}
2022-04-28T08:00:14.471290438Z stdout F [08:00:14 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebnext.nic.81f30dfd-cfeb-46d9-aae9-5cb99f7355bd, traceId: {603d83ea-4f1b-4636-bc9a-5d00c490ab44}
2022-04-28T08:00:14.335796105Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.334531895Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.332416079Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.329402556Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.328471849Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.326651335Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.321168894Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.148018174Z stdout F [08:00:14 INF] Request finished HTTP/1.1 GET http://xx.xxx.x.xxx:5000/api/v1/health?includeDependencies=false - - - 200 60 application/json;+charset=utf-8 76.2599ms
2022-04-28T08:00:14.140175614Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension Unknown as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.139324108Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.138530102Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListContainers as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.137703395Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.136909789Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.136048683Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerServiceMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.135255677Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.134470171Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension BlobPreflightRequest as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.133732665Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension DeleteBlob as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.132956759Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.132190053Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.131374547Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.130581241Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.129799235Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlob as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.129003329Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.128054022Z stdout F [08:00:14 INF] Executed action Promitor.Agents.Scraper.Controllers.v1.HealthController.Get (Promitor.Agents.Scraper) in 0.2373ms
2022-04-28T08:00:14.126331909Z stdout F [08:00:14 INF] Route matched with {action = "Get", controller = "Health"}. Executing controller action with signature System.Threading.Tasks.Task`1[Microsoft.AspNetCore.Mvc.IActionResult] Get(Boolean) on controller Promitor.Agents.Scraper.Controllers.v1.HealthController (Promitor.Agents.Scraper).
2022-04-28T08:00:14.071545891Z stdout F [08:00:14 INF] Request starting HTTP/1.1 GET http://xx.xxx.x.xxx:5000/api/v1/health?includeDependencies=false - -
2022-04-28T08:00:14.058604693Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.057389883Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.056426176Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.05567077Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.053838356Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.052687348Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.050667032Z stdout F [08:00:14 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:14.037893235Z stdout F [08:00:14 INF] Discovered 1 resources for resource collection azure-kubernetes-service.
2022-04-28T08:00:14.025267639Z stdout F [08:00:14 INF] Discovered 1 resources for resource collection key-vault-landscape.
2022-04-28T08:00:14.024381332Z stdout F [08:00:14 WRN] {"DependencyType": "Http", "DependencyName": "GET /api/v2/resources/groups/key-vault-landscape/discover", "DependencyData": null, "TargetName": "promitor-agent-resource-discovery", "ResultCode": 200, "IsSuccessful": true, "StartTime": "2022-04-28T08:00:00.5725964 +00:00", "Duration": "00:00:13.4511425", "Context": {"TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
2022-04-28T08:00:14.022957821Z stdout F [08:00:14 WRN] {"RequestMethod": "GET", "RequestHost": "http://promitor-agent-resource-discovery", "RequestUri": "/api/v2/resources/groups/key-vault-landscape/discover", "ResponseStatusCode": 200, "RequestDuration": "00:00:13.4498841", "RequestTime": "2022-04-28T08:00:14.0225069 +00:00", "OperationName": "GET /api/v2/resources/groups/key-vault-landscape/discover", "Context": {"TelemetryType": "Request"}, "$type": "RequestLogEntry"}
2022-04-28T08:00:14.02155121Z stdout F [08:00:14 INF] End processing HTTP request after 13448.8336ms - 200
2022-04-28T08:00:14.020615203Z stdout F [08:00:14 INF] Received HTTP response headers after 13447.5358ms - 200
2022-04-28T08:00:14.018539887Z stdout F [08:00:14 INF] Discovered 1 resources for resource collection azure-kubernetes-service.
2022-04-28T08:00:14.017455979Z stdout F [08:00:14 WRN] {"DependencyType": "Http", "DependencyName": "GET /api/v2/resources/groups/azure-kubernetes-service/discover", "DependencyData": null, "TargetName": "promitor-agent-resource-discovery", "ResultCode": 200, "IsSuccessful": true, "StartTime": "2022-04-28T08:00:00.6587420 +00:00", "Duration": "00:00:13.3584827", "Context": {"TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
2022-04-28T08:00:14.016345071Z stdout F [08:00:14 WRN] {"RequestMethod": "GET", "RequestHost": "http://promitor-agent-resource-discovery", "RequestUri": "/api/v2/resources/groups/azure-kubernetes-service/discover", "ResponseStatusCode": 200, "RequestDuration": "00:00:13.3573262", "RequestTime": "2022-04-28T08:00:14.0161264 +00:00", "OperationName": "GET /api/v2/resources/groups/azure-kubernetes-service/discover", "Context": {"TelemetryType": "Request"}, "$type": "RequestLogEntry"}
2022-04-28T08:00:14.015144461Z stdout F [08:00:14 INF] End processing HTTP request after 13356.2402ms - 200
2022-04-28T08:00:14.013992553Z stdout F [08:00:13 INF] Received HTTP response headers after 13336.721ms - 200
2022-04-28T08:00:14.004188178Z stdout F [08:00:14 WRN] {"DependencyType": "Http", "DependencyName": "GET /api/v2/resources/groups/azure-kubernetes-service/discover", "DependencyData": null, "TargetName": "promitor-agent-resource-discovery", "ResultCode": 200, "IsSuccessful": true, "StartTime": "2022-04-28T08:00:00.6354894 +00:00", "Duration": "00:00:13.3679270", "Context": {"TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
2022-04-28T08:00:14.002687867Z stdout F [08:00:14 WRN] {"RequestMethod": "GET", "RequestHost": "http://promitor-agent-resource-discovery", "RequestUri": "/api/v2/resources/groups/azure-kubernetes-service/discover", "ResponseStatusCode": 200, "RequestDuration": "00:00:13.3669803", "RequestTime": "2022-04-28T08:00:14.0024976 +00:00", "OperationName": "GET /api/v2/resources/groups/azure-kubernetes-service/discover", "Context": {"TelemetryType": "Request"}, "$type": "RequestLogEntry"}
2022-04-28T08:00:14.00178826Z stdout F [08:00:14 INF] End processing HTTP request after 13366.15ms - 200
2022-04-28T08:00:14.000401849Z stdout F [08:00:14 INF] Received HTTP response headers after 13364.5165ms - 200
2022-04-28T08:00:13.99926454Z stdout F [08:00:13 INF] Discovered 1 resources for resource collection key-vault-landscape.
2022-04-28T08:00:13.998391534Z stdout F [08:00:13 WRN] {"DependencyType": "Http", "DependencyName": "GET /api/v2/resources/groups/key-vault-landscape/discover", "DependencyData": null, "TargetName": "promitor-agent-resource-discovery", "ResultCode": 200, "IsSuccessful": true, "StartTime": "2022-04-28T08:00:00.6746854 +00:00", "Duration": "00:00:13.3234890", "Context": {"TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
2022-04-28T08:00:13.996922123Z stdout F [08:00:13 WRN] {"RequestMethod": "GET", "RequestHost": "http://promitor-agent-resource-discovery", "RequestUri": "/api/v2/resources/groups/key-vault-landscape/discover", "ResponseStatusCode": 200, "RequestDuration": "00:00:13.3217769", "RequestTime": "2022-04-28T08:00:13.9964902 +00:00", "OperationName": "GET /api/v2/resources/groups/key-vault-landscape/discover", "Context": {"TelemetryType": "Request"}, "$type": "RequestLogEntry"}
2022-04-28T08:00:13.995669113Z stdout F [08:00:13 INF] End processing HTTP request after 13320.8854ms - 200
2022-04-28T08:00:13.994033801Z stdout F [08:00:13 INF] Received HTTP response headers after 13319.022ms - 200
2022-04-28T08:00:13.990655975Z stdout F [08:00:13 INF] Discovered 8 resources for resource collection network-interfaces-landscape.
2022-04-28T08:00:13.989653767Z stdout F [08:00:13 WRN] {"DependencyType": "Http", "DependencyName": "GET /api/v2/resources/groups/network-interfaces-landscape/discover", "DependencyData": null, "TargetName": "promitor-agent-resource-discovery", "ResultCode": 200, "IsSuccessful": true, "StartTime": "2022-04-28T08:00:00.5976986 +00:00", "Duration": "00:00:13.3913388", "Context": {"TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
2022-04-28T08:00:13.987528851Z stdout F [08:00:13 WRN] {"RequestMethod": "GET", "RequestHost": "http://promitor-agent-resource-discovery", "RequestUri": "/api/v2/resources/groups/network-interfaces-landscape/discover", "ResponseStatusCode": 200, "RequestDuration": "00:00:13.3886317", "RequestTime": "2022-04-28T08:00:13.9871427 +00:00", "OperationName": "GET /api/v2/resources/groups/network-interfaces-landscape/discover", "Context": {"TelemetryType": "Request"}, "$type": "RequestLogEntry"}
2022-04-28T08:00:13.985885739Z stdout F [08:00:13 INF] End processing HTTP request after 13387.9956ms - 200
2022-04-28T08:00:13.964197673Z stdout F [08:00:13 INF] Received HTTP response headers after 13365.327ms - 200
2022-04-28T08:00:13.818590564Z stdout F [08:00:13 INF] Found value null for metric azure_container_registry_agent_pool_cpu_time with aggregation interval 00:05:00
2022-04-28T08:00:13.795401087Z stdout F [08:00:13 INF] Found value 0 for metric azure_container_registry_total_pull_count with aggregation interval 00:05:00
2022-04-28T08:00:13.695670327Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.694886421Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetContainerProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.693992214Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.693911913Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.684114439Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624173382Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624169182Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624160582Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624117982Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624113682Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624105282Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.624045781Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.539382636Z stdout F [08:00:13 FTL]
2022-04-28T08:00:13.532908987Z stdout F [08:00:13 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs1.nic.88b100e5-0743-43cd-a592-e0ac661c4872, traceId: {12064114-5e33-427b-8064-c5e2cc736732}
2022-04-28T08:00:13.423336452Z stdout F [08:00:13 INF] Found value null for metric azure_container_registry_run_duration with aggregation interval 00:05:00
2022-04-28T08:00:13.408858241Z stdout F [08:00:13 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs2.nic.c88992f6-5ecd-44d1-b332-84795d619807, traceId: {74dfc079-ff78-4367-9874-a2ab93681b38}
2022-04-28T08:00:13.404300707Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension SetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404296206Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension CreateContainer as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404278406Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension SetContainerACL as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404274106Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension SetContainerMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404269706Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension GetContainerACL as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404265306Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension GetContainerServiceMetadata as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404260606Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension GetQueueServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404256306Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension ListContainers as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404251906Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension BlobPreflightRequest as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404247706Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension Unknown as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404243206Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension GetFileServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404238906Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension GetContainerProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404234506Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension CreateTable as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404229506Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension ListBlobs as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404223406Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension PutBlock as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404217506Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension PutBlockList as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404201806Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.404123405Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_sent_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.267992368Z stdout F [08:00:13 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddeb27.nic.ac66fc98-779a-421e-a4fe-98736cc3d489, traceId: {06f55de0-d86d-4f0d-9b67-a93a5fd135d9}
2022-04-28T08:00:13.256272178Z stdout F [08:00:13 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs3.nic.327f47ff-6dae-4bd9-9cca-729917de5d40, traceId: {1932cad2-0ccd-46fa-b324-d1b56633f849}
2022-04-28T08:00:13.234787515Z stdout F [08:00:13 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censored-dev-key-vault.nic.c17668e1-49aa-4aa5-98f0-31b4ab44e0e6, traceId: {2751fc75-bc6d-4319-84e3-cea8b2e8f896}
2022-04-28T08:00:13.106597038Z stdout F [08:00:13 INF] Found value 53320744496 for metric azure_container_registry_storage_used with aggregation interval 00:05:00
2022-04-28T08:00:13.010466205Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension GetBlobServiceProperties as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:13.010369605Z stdout F [08:00:13 INF] Found value 0 for metric azure_storage_account_bytes_received_rate with dimension EntityGroupTransaction as part of ApiName dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.672651831Z stdout F [08:00:12 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/redactedazurefilesdev.nic.ee13ddf1-772a-4855-822a-a5ba2be2a9ea, traceId: {62dc2c44-81c8-456f-a745-281c08bdbc48}
2022-04-28T08:00:12.566974526Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001j as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.565465014Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001m as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.564656208Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001g as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.561829586Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001l as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.560345175Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.558064358Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001k as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.525929713Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-system-34776058-vmss00000h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.497788198Z stdout F [08:00:12 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebqs4.nic.1e6001e7-12d2-4d6d-98c9-440e49e7bdd3, traceId: {2ce56877-5d4e-4af9-983e-a38c517e73f4}
2022-04-28T08:00:12.461867825Z stdout F [08:00:12 FTL] NotFound: Failed to find resource mapping for provider: Microsoft.Network, resourceType: networkInterfaces, id: /subscriptions/{SUBSCRIPTIONID}/resourceGroups/{RESOURCEGROUP}/providers/Microsoft.Network/networkInterfaces/censoreddebnext.nic.81f30dfd-cfeb-46d9-aae9-5cb99f7355bd, traceId: {b645ee6c-663f-48fe-8b6f-acbf7c1a4008}
2022-04-28T08:00:12.449605731Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-system-34776058-vmss00000i as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.447759317Z stdout F [08:00:12 INF] Found value null for metric node_memory_working_set_bytes with dimension aks-user-33133238-vmss00001h as part of node dimension with aggregation interval 00:05:00
2022-04-28T08:00:12.413604757Z stdout F [08:00:12 INF] Discovered 1 resources for resource collection azure-kubernetes-service.
2022-04-28T08:00:12.412614549Z stdout F [08:00:12 WRN] {"DependencyType": "Http", "DependencyName": "GET /api/v2/resources/groups/azure-kubernetes-service/discover", "DependencyData": null, "TargetName": "promitor-agent-resource-discovery", "ResultCode": 200, "IsSuccessful": true, "StartTime": "2022-04-28T08:00:00.6441520 +00:00", "Duration": "00:00:11.7681453", "Context": {"TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
2022-04-28T08:00:12.41142154Z stdout F [08:00:12 WRN] {"RequestMethod": "GET", "RequestHost": "http://promitor-agent-resource-discovery", "RequestUri": "/api/v2/resources/groups/azure-kubernetes-service/discover", "ResponseStatusCode": 200, "RequestDuration": "00:00:11.7660972", "RequestTime": "2022-04-28T08:00:12.4110678 +00:00", "OperationName": "GET /api/v2/resources/groups/azure-kubernetes-service/discover", "Context": {"TelemetryType": "Request"}, "$type": "RequestLogEntry"}

@tomkerkhove
Copy link
Owner

Interesting. Are you using CNI on AKS? Would you mind trying to look for that ARM resource and extract the ARM template please? (if allowed)

@Mahagon
Copy link
Author

Mahagon commented Apr 28, 2022 •
edited
Loading

Yes and we are using calico as CNI plugin.

We deploy it out cluster with terraform, so I'm unsure how to extract the CNI part from the ARM template :/

our terraform template for the cluster looks like this:

Spoiler with terraform code 
resource "azurerm_kubernetes_cluster" "k8s" {
  name                = "XXXX-${var.environment}"
  location            = data.azurerm_resource_group.saas.location
  resource_group_name = data.azurerm_resource_group.saas.name
  dns_prefix          = "k8s"
  kubernetes_version  = "XXXX"
  api_server_authorized_ip_ranges = var.api_server_authorized_ip_ranges

  identity {
    type = "SystemAssigned"
  }

  default_node_pool {
    name                         = "system"
    enable_auto_scaling          = true
    min_count                    = var.min_system_agent_count
    max_count                    = var.max_system_agent_count
    vm_size                      = "Standard_D4_v4"
    max_pods                     = 100
    orchestrator_version         = "XXXXX"
    only_critical_addons_enabled = true #Sets taint to CriticalAddonsOnly=true:NoSchedule
    vnet_subnet_id               = azurerm_subnet.k8s-subnet.id
  }

  network_profile {
    load_balancer_sku  = "Standard"
    network_plugin     = "azure"
    network_policy     = "calico"
    service_cidr       = "xx.xxx.0.0/14"
    docker_bridge_cidr = "xx.xx.0.1/16"
    dns_service_ip     = "xx.xxx.0.10"
  }

  role_based_access_control {
    enabled = true
  }

  lifecycle {
    prevent_destroy = true
  }
}

@tomkerkhove
Copy link
Owner

Would you mind going to your Azure sub in the portal and look a resource group with MC_ as a name prefix and see if you see those NICs please?

@Mahagon
Copy link
Author

Mahagon commented Apr 28, 2022

I can't see them in the resource group with the MC_ prefix, just in our manually created resource group

image

image

@tomkerkhove
Copy link
Owner

image

Would it be possible to click the resource and go to "Export Template" please?

@Mahagon
Copy link
Author

Mahagon commented May 2, 2022

Oh sorry missed the notification

Is a single Interface enough?

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "networkInterfaces_redacted_nic_ac66fc98_779a_421e_a4fe_98736cc3d489_name": {
            "defaultValue": "redacted.nic.ac66fc98-779a-421e-a4fe-98736cc3d489",
            "type": "String"
        },
        "virtualNetworks_aks_network_externalid": {
            "defaultValue": "/subscriptions/SUBSCRIPTIONID/resourceGroups/RESOURCEGROUP/providers/Microsoft.Network/virtualNetworks/aks-network",
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.Network/networkInterfaces",
            "apiVersion": "2020-11-01",
            "name": "[parameters('networkInterfaces_redacted_nic_ac66fc98_779a_421e_a4fe_98736cc3d489_name')]",
            "location": "germanywestcentral",
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "privateEndpointIpConfig.64729c55-df1f-480f-9929-ac427901a20c",
                        "properties": {
                            "privateIPAddress": "xx.xxx.xx.xx",
                            "privateIPAllocationMethod": "Dynamic",
                            "subnet": {
                                "id": "[concat(parameters('virtualNetworks_aks_network_externalid'), '/subnets/storage-account')]"
                            },
                            "primary": true,
                            "privateIPAddressVersion": "IPv4"
                        }
                    }
                ],
                "dnsSettings": {
                    "dnsServers": []
                },
                "enableAcceleratedNetworking": false,
                "enableIPForwarding": false
            }
        }
    ]
}

@tomkerkhove
Copy link
Owner

Hm that looks pretty straightforward. I'll need to try to reproduce this, sorry.

@michail-nikolaev
Copy link

Got the same issue for network interface generated by private endpoint...

@krnayak2601
Copy link

Is this issue resolved? I'm having same issue errors in discovery and scraper for Microsoft.Network

@tomkerkhove
Copy link
Owner

This was not fixed as I didn't have time to look at it yet, but always open to contributions.

@tomkerkhove
Copy link
Owner

I'm not fully sure how to reproduce this, to be honest.

@amirschw
Copy link
Contributor

amirschw commented Mar 6, 2023 •
edited
Loading

I am seeing the same issue happening intermittently from Microsoft.Network when using the preview images to scrape NatGateways. Found this related issue: Azure/azure-sdk-for-python#14388.

@amirschw
Copy link
Contributor

amirschw commented Mar 6, 2023

I wonder if migrating to the new Azure SDKs might help - #2209

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomkerkhove tomkerkhove

Labels
bug Something isn't working
Projects
Promitor Roadmap
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Mahagon @michail-nikolaev @tomkerkhove @amirschw @krnayak2601