-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add loggers aimed at Elasticsearch #48
Comments
+1 for |
In my case I had to add |
Moving security discussion to a private channel. |
This seems to work: In [1]: import requests
In [2]: topic = 'test'
In [3]: r = requests.put(f'http://cmb03.cs.nsls2.local:9200/{topic}/_doc/1', json={'hello': 'DAMA tester'})
In [4]: r
Out[4]: <Response [200]>
In [5]: r.text
Out[5]: '{"_index":"test","_type":"_doc","_id":"1","_version":3,"result":"updated","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1}' |
Good. As a note to future readers, the URL Also, refer to this Dropbox Paper for suggested index names: https://paper.dropbox.com/doc/Kafka-Topics--AT8pvSkZTo5zs_yP40M6R5HFAg-Kedt0QGwc0DhH9cZzXkDy |
We should wrap the usage demonstrated by in @mrakitin's comment above in a Python logging handler. Something like: class ElasticHandler(logging.Handler):
def __init__(self, url):
self.url = url
super().__init__()
def emit(self, record):
# Extract useful info from the record and put them into a dict.
response = requests.put(...)
# Raise an exception if the server we PUT to returns a bad status code.
# The logging framework will catch the error and print a message.
response.raise_for_status() Read up on LogRecord to understand what to expect in Test like so: from bluesky import RunEngine
RE = RunEngine({})
handler = ElasticHandler(URL)
RE.log.addHandler(handler)
RE.log.setLevel("DEBUG")
RE([]) # Run an empty plan for simplicity's sake. Will still generate some log messages. |
@ke-zhang-rd If you are interested in getting involved with the Kibana stuff, this might be a good issue for you to work on. We don't need it for the deployment, but we wish we had it for the deployment and will roll it out to select beamlines as soon as it's ready. Let us know if you are interested; if not I expect @mrakitin or I will be happy to take it. |
As a separate but related issue that may be tackled in the same PR, we agreed in our meeting today that we would have both an |
Awesome, I like that the |
We need a Python logging handler that submits data to Elastic. Specifically it should submit at POST request like this:
There may already be a nice library of StackOverflow snippet for making HTTP requests from Python loggers. If not, I would just roll something using requests.
The text was updated successfully, but these errors were encountered: