Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jira-prometheus-exporter add-on not export *jira* matrics in datacenter mode #61

Open
agentchif opened this issue Jan 30, 2020 · 13 comments
Open

jira-prometheus-exporter add-on not export *jira* matrics in datacenter mode #61

agentchif opened this issue Jan 30, 2020 · 13 comments

Comments

@agentchif
Copy link

i installed the add-on on Jira DC that contain 2 nodes and 1 Load balancer in front (haproxy).
i configure the prometheus config.yaml to scrape both nodes and put also labels on each node.
the scrape section at the prometheus config.yaml look like:

  • job_name: 'jira'
    scheme: http
    metrics_path: '/plugins/servlet/prometheus/metrics'
    static_config:
    • targets: ['145.1.45.141:8080']
      labels:
      nodeID: node01
      ip: '145.1.45.141'
    • targets: ['145.1.45.142:8080']
      labels:
      nodeID: node02
      ip: '145.1.45.142'

the metrics that the exporter export to the prometheus are only jvm & process TYPES
there is no JIRA type at all

hence i cant measure metrics that related to jira like:
jira_issue_update_count
jira_user_login_count
jira_user_logout_count
etc....

anything that start with jira_

BTW,
i have the same configuration at bitbucket and there the bitbucket exporter is exporting bitbucket metrics include jvm & process

please assist here
@AndreyVMarkelov
Copy link
Owner

AndreyVMarkelov commented Jan 30, 2020 via email

@agentchif
Copy link
Author

agentchif commented Jan 30, 2020 •
edited
Loading

Jira Software 7.13.0 (Datacenter)
Jira Core 7.13.0

from /var/atlassian/application-data/jira/log/atlassian-jira.log i can see this error:

at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.licenseMetrics(MetricCollectorImpl.java:435)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.collectInternal(MetricCollectorImpl.java:489)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.collect(MetricCollectorImpl.java:641)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.PrometheusExporter.doGet(PrometheusExporter.java:44)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.AllEndpointFilter$1.run(AllEndpointFilter.java:44)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.requestDuration(MetricCollectorImpl.java:283)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.AllEndpointFilter.doFilter(AllEndpointFilter.java:39)
2020-01-30 11:00:41,273 http-nio-8080-exec-321 ERROR anonymous 660x5246839x1 - 145.1.45.19 /plugins/servlet/prometheus/metrics [r.a.a.p.promjiraexporter.service.MetricCollectorImpl] Error collect prometheus metrics

@agentchif
Copy link
Author

here is a full part of atlassian-jira.log.
this section is repeat itself in deferent times period.

2020-01-30 13:02:17,181 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] [scheduled] Running cache replication queue stats for: 10 queues...
2020-01-30 13:02:17,181 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] Cache replication queue stats per node: node2 snapshot stats: {"timestampMillis":1580382137181,"nodeId":"node2","queueSize":0,"startQueueSize":0,"startTimestampMillis":1580382077181,"startMillisAgo":60000,"closeCounter":0,"addCounter":29,"droppedOnAddCounter":0,"criticalAddCounter":0,"criticalPeekCounter":0,"criticalRemoveCounter":0,"peekCounter":0,"peekOrBlockCounter":29,"removeCounter":29,"backupQueueCounter":0,"closeErrorsCounter":0,"addErrorsCounter":0,"peekErrorsCounter":0,"peekOrBlockErrorsCounter":0,"removeErrorsCounter":0,"backupQueueErrorsCounter":0,"lastAddTimestampMillis":1580382130261,"lastAddMillisAgo":6920,"lastPeekTimestampMillis":0,"lastPeekMillisAgo":0,"lastPeekOrBlockTimestampMillis":1580382130261,"lastPeekOrBlockMillisAgo":6920,"lastRemoveTimestampMillis":1580382130266,"lastRemoveMillisAgo":6915,"lastBackupQueueTimestampMillis":0,"lastBackupQueueMillisAgo":0,"timeToAddMillis":{"count":29,"min":0,"max":4,"sum":64,"avg":2},"timeToPeekMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"timeToPeekOrBlockMillis":{"count":29,"min":0,"max":85083,"sum":509423,"avg":17566},"timeToRemoveMillis":{"count":29,"min":0,"max":3,"sum":41,"avg":1},"timeToBackupQueueMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"staleCounter":0,"sendCounter":29,"droppedOnSendCounter":0,"timeToSendMillis":{"count":29,"min":0,"max":6,"sum":70,"avg":2},"sendRuntimeExceptionCounter":0,"sendCheckedExceptionCounter":0,"sendNotBoundExceptionCounter":0}
2020-01-30 13:02:17,182 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] Cache replication queue stats per node: node2 total stats: {"timestampMillis":1580382137181,"nodeId":"node2","queueSize":0,"startQueueSize":0,"startTimestampMillis":1578558592533,"startMillisAgo":1823544648,"closeCounter":0,"addCounter":777231,"droppedOnAddCounter":0,"criticalAddCounter":0,"criticalPeekCounter":0,"criticalRemoveCounter":0,"peekCounter":0,"peekOrBlockCounter":778148,"removeCounter":777231,"backupQueueCounter":0,"closeErrorsCounter":0,"addErrorsCounter":0,"peekErrorsCounter":0,"peekOrBlockErrorsCounter":0,"removeErrorsCounter":0,"backupQueueErrorsCounter":0,"lastAddTimestampMillis":1580382130261,"lastAddMillisAgo":6920,"lastPeekTimestampMillis":0,"lastPeekMillisAgo":0,"lastPeekOrBlockTimestampMillis":1580382130261,"lastPeekOrBlockMillisAgo":6920,"lastRemoveTimestampMillis":1580382130266,"lastRemoveMillisAgo":6915,"lastBackupQueueTimestampMillis":0,"lastBackupQueueMillisAgo":0,"timeToAddMillis":{"count":777231,"min":0,"max":4361,"sum":10206526,"avg":13},"timeToPeekMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"timeToPeekOrBlockMillis":{"count":778148,"min":0,"max":1356299,"sum":18227115548,"avg":23423},"timeToRemoveMillis":{"count":777231,"min":0,"max":3119,"sum":3081623,"avg":3},"timeToBackupQueueMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"staleCounter":0,"sendCounter":776321,"droppedOnSendCounter":0,"timeToSendMillis":{"count":776321,"min":0,"max":7052,"sum":3692014,"avg":4},"sendRuntimeExceptionCounter":1,"sendCheckedExceptionCounter":6,"sendNotBoundExceptionCounter":1820}
2020-01-30 13:02:17,182 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] [scheduled] ... done running cache replication queue stats for: 10 queues.
2020-01-30 13:02:42,704 http-nio-8080-exec-321 ERROR anonymous 782x5259684x1 - 145.1.45.19 /plugins/servlet/prometheus/metrics [r.a.a.p.promjiraexporter.service.MetricCollectorImpl] Error collect prometheus metrics
java.lang.NoSuchMethodError: com.atlassian.application.api.Application.getLicense()Lio/atlassian/fugue/Option;
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.licenseMetrics(MetricCollectorImpl.java:435)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.collectInternal(MetricCollectorImpl.java:489)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.collect(MetricCollectorImpl.java:641)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.PrometheusExporter.doGet(PrometheusExporter.java:44)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at com.atlassian.plugin.servlet.DelegatingPluginServlet.service(DelegatingPluginServlet.java:37)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at com.atlassian.plugin.servlet.ServletModuleContainerServlet.service(ServletModuleContainerServlet.java:45)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
... 53 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 20 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 3 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 41 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 16 filtered
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.AllEndpointFilter$1.run(AllEndpointFilter.java:44)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.requestDuration(MetricCollectorImpl.java:283)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.AllEndpointFilter.doFilter(AllEndpointFilter.java:39)
... 7 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 3 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 3 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 19 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at org.kantega.atlaskerb.AtlasKerberosPreFilter.doFilter(AtlasKerberosPreFilter.java:61)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 7 filtered
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at net.bull.javamelody.JiraMonitoringFilter.doFilter(JiraMonitoringFilter.java:137)
... 22 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
2020-01-30 13:03:17,181 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] [scheduled] Running cache replication queue stats for: 10 queues...
2020-01-30 13:03:17,181 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] Cache replication queue stats per node: node2 snapshot stats: {"timestampMillis":1580382197181,"nodeId":"node2","queueSize":0,"startQueueSize":0,"startTimestampMillis":1580382137181,"startMillisAgo":60000,"closeCounter":0,"addCounter":16,"droppedOnAddCounter":0,"criticalAddCounter":0,"criticalPeekCounter":0,"criticalRemoveCounter":0,"peekCounter":0,"peekOrBlockCounter":16,"removeCounter":16,"backupQueueCounter":0,"closeErrorsCounter":0,"addErrorsCounter":0,"peekErrorsCounter":0,"peekOrBlockErrorsCounter":0,"removeErrorsCounter":0,"backupQueueErrorsCounter":0,"lastAddTimestampMillis":1580382191453,"lastAddMillisAgo":5728,"lastPeekTimestampMillis":0,"lastPeekMillisAgo":0,"lastPeekOrBlockTimestampMillis":1580382191454,"lastPeekOrBlockMillisAgo":5727,"lastRemoveTimestampMillis":1580382191459,"lastRemoveMillisAgo":5722,"lastBackupQueueTimestampMillis":0,"lastBackupQueueMillisAgo":0,"timeToAddMillis":{"count":16,"min":0,"max":4,"sum":40,"avg":2},"timeToPeekMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"timeToPeekOrBlockMillis":{"count":16,"min":0,"max":92169,"sum":457998,"avg":28624},"timeToRemoveMillis":{"count":16,"min":0,"max":3,"sum":24,"avg":1},"timeToBackupQueueMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"staleCounter":0,"sendCounter":16,"droppedOnSendCounter":0,"timeToSendMillis":{"count":16,"min":0,"max":7,"sum":38,"avg":2},"sendRuntimeExceptionCounter":0,"sendCheckedExceptionCounter":0,"sendNotBoundExceptionCounter":0}
2020-01-30 13:03:17,181 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] Cache replication queue stats per node: node2 total stats: {"timestampMillis":1580382197181,"nodeId":"node2","queueSize":0,"startQueueSize":0,"startTimestampMillis":1578558592533,"startMillisAgo":1823604648,"closeCounter":0,"addCounter":777247,"droppedOnAddCounter":0,"criticalAddCounter":0,"criticalPeekCounter":0,"criticalRemoveCounter":0,"peekCounter":0,"peekOrBlockCounter":778164,"removeCounter":777247,"backupQueueCounter":0,"closeErrorsCounter":0,"addErrorsCounter":0,"peekErrorsCounter":0,"peekOrBlockErrorsCounter":0,"removeErrorsCounter":0,"backupQueueErrorsCounter":0,"lastAddTimestampMillis":1580382191453,"lastAddMillisAgo":5728,"lastPeekTimestampMillis":0,"lastPeekMillisAgo":0,"lastPeekOrBlockTimestampMillis":1580382191454,"lastPeekOrBlockMillisAgo":5727,"lastRemoveTimestampMillis":1580382191459,"lastRemoveMillisAgo":5722,"lastBackupQueueTimestampMillis":0,"lastBackupQueueMillisAgo":0,"timeToAddMillis":{"count":777247,"min":0,"max":4361,"sum":10206566,"avg":13},"timeToPeekMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"timeToPeekOrBlockMillis":{"count":778164,"min":0,"max":1356299,"sum":18227573546,"avg":23423},"timeToRemoveMillis":{"count":777247,"min":0,"max":3119,"sum":3081647,"avg":3},"timeToBackupQueueMillis":{"count":0,"min":0,"max":0,"sum":0,"avg":0},"staleCounter":0,"sendCounter":776337,"droppedOnSendCounter":0,"timeToSendMillis":{"count":776337,"min":0,"max":7052,"sum":3692052,"avg":4},"sendRuntimeExceptionCounter":1,"sendCheckedExceptionCounter":6,"sendNotBoundExceptionCounter":1820}
2020-01-30 13:03:17,181 localq-stats-0 INFO [c.a.j.c.distribution.localq.LocalQCacheManager] [scheduled] ... done running cache replication queue stats for: 10 queues.
2020-01-30 13:03:42,704 http-nio-8080-exec-191 ERROR anonymous 783x5259891x1 - 145.1.45.19 /plugins/servlet/prometheus/metrics [r.a.a.p.promjiraexporter.service.MetricCollectorImpl] Error collect prometheus metrics
java.lang.NoSuchMethodError: com.atlassian.application.api.Application.getLicense()Lio/atlassian/fugue/Option;
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.licenseMetrics(MetricCollectorImpl.java:435)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.collectInternal(MetricCollectorImpl.java:489)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.collect(MetricCollectorImpl.java:641)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:216)
at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:137)
at io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:22)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.PrometheusExporter.doGet(PrometheusExporter.java:44)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at com.atlassian.plugin.servlet.DelegatingPluginServlet.service(DelegatingPluginServlet.java:37)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at com.atlassian.plugin.servlet.ServletModuleContainerServlet.service(ServletModuleContainerServlet.java:45)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
... 53 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 20 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 3 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 41 filtered
at com.atlassian.jira.security.JiraSecurityFilter.lambda$doFilter$0(JiraSecurityFilter.java:66)
... 1 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:64)
... 16 filtered
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.AllEndpointFilter$1.run(AllEndpointFilter.java:44)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.service.MetricCollectorImpl.requestDuration(MetricCollectorImpl.java:283)
at ru.andreymarkelov.atlas.plugins.promjiraexporter.servlet.AllEndpointFilter.doFilter(AllEndpointFilter.java:39)
... 7 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 3 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 3 filtered
at org.kantega.atlaskerb.AtlasKerberosFilter.doFilter(AtlasKerberosFilter.java:90)
... 19 filtered
at com.atlassian.jira.servermetrics.CorrelationIdPopulatorFilter.doFilter(CorrelationIdPopulatorFilter.java:30)
... 5 filtered
at org.kantega.atlaskerb.AtlasKerberosPreFilter.doFilter(AtlasKerberosPreFilter.java:61)
... 8 filtered
at com.atlassian.web.servlet.plugin.request.RedirectInterceptingFilter.doFilter(RedirectInterceptingFilter.java:21)
... 4 filtered
at com.atlassian.web.servlet.plugin.LocationCleanerFilter.doFilter(LocationCleanerFilter.java:36)
... 7 filtered
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at net.bull.javamelody.JiraMonitoringFilter.doFilter(JiraMonitoringFilter.java:137)
... 22 filtered
at com.atlassian.jira.servermetrics.MetricsCollectorFilter.doFilter(MetricsCollectorFilter.java:25)
... 24 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

@AndreyVMarkelov
Copy link
Owner

AndreyVMarkelov commented Jan 30, 2020 via email

@agentchif
Copy link
Author

Hi Andrey
sorry to bother you but there is any progress on that issue?

@AndreyVMarkelov
Copy link
Owner

@agentchif please try this https://github.com/AndreyVMarkelov/jira-prometheus-exporter/releases/download/1.0.33-jira8/prom-jira-exporter-1.0.33-jira8.jar

@agentchif
Copy link
Author

Hi andrey
its still not working.
the same issue exist
i got only jvm_* metrics but not jira_* metrics

Can you please check it?

@AndreyVMarkelov
Copy link
Owner

AndreyVMarkelov commented Mar 10, 2020 via email

@agentchif
Copy link
Author

hi andrey
i think it is the same log
i attached the new log here

atlassian-jira.log

@agentchif
Copy link
Author

hi andrey
did you found some time to check that issue?

@agentchif
Copy link
Author

hi
any update please?

@agentchif
Copy link
Author

hi andrey
there is any update regarding this issue?

@agentchif
Copy link
Author

Hi
any update please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AndreyVMarkelov @agentchif