-
Notifications
You must be signed in to change notification settings - Fork 0
/
Logstash_Notes.txt
276 lines (250 loc) · 7.75 KB
/
Logstash_Notes.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
--------------
Logstash Notes
--------------
LEARNING DOCS
-------------
Review
------
https://www.elastic.co/guide/en/logstash/current/execution-model.html
New/Next
--------
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
Reference
---------
https://www.elastic.co/support/matrix#show_logstash_plugins
https://www.elastic.co/guide/en/logstash/current/input-plugins.html
https://www.elastic.co/guide/en/logstash/current/filter-plugins.html
https://www.elastic.co/guide/en/logstash/current/output-plugins.html
https://www.elastic.co/guide/en/logstash/current/codec-plugins.html
https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html
TO-DO/ISSUES
------------
- add CloudWatch input filter
- find out how ELB logs are getting into S3 buckets
- set up ELB logging via Ansible
- get/look at ELB logs
- Tuning and Profiling Logstash Performance (https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html)
- consider enabling (to prevent loss): https://www.elastic.co/guide/en/logstash/current/persistent-queues.html
- are other servers named logstash or loghost?
logstash
- which AMI to use - use lookup
- for lc, image_id and image_name being define?
- security groups
- public/private network
- asg role av zone?
- create sec-grp-logstash?
- setup access logs and log bucket info for elb logging
- /etc/yum.conf
value of "releasever" wrong
- AWS keys not being set
because value of "releasever" (above) was wrong, could not install gcc
due to "distro" value in Ansible inventory (override on command line when running playbook)
- update R53 record logstash.us.vmedix.nim
- find out how to run logstash in debug mode
- create/use a patterns dir for GROK
https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
- put file and pattern in logstash role
- update role to install the pattern file
- update the inventory file to use the new/added pattern
INPUTS -> FILTERS -> OUTPUTS
INSTALL
-------
Required: Java 8 (May 2107: Java 9 not supported)
# java -version
# export JAVA_HOME=/usr/bin/java # may need
APT
---
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo apt-get update && sudo apt-get install logstash
YUM
---
# download public signing key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
# add/create yum repo
/- /etc/yum.repos.d/elastic.repo -\
[elastic-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
\- /etc/yum.repos.d/elastic.repo -/
# install with yum
sudo yum install logstash
FILES/DIRS
----------
/usr/share/logstash # install dir
/usr/local/bin/logstash -> /usr/share/logstash/bin/logstash
/etc/logstash
/etc/logstash/conf.d # pipeline config files
/etc/logstash/logstash.yml # settings config file
/etc/logstash/startup.options # linux: for system-install
/etc/logstash/jvm.options # JVM config flags
/etc/logstash/log4j2.properties # logs configurations
/var/log/logstash
/var/lib/logstash
/etc/init.d/logstash # Upstart
/etc/init/logstash # SysV
Running/Stopping
----------------
sudo systemctl start logstash.service # Systemd
sudo systemctl stop logstash # Systemd
sudo initctl start|stop logstash # Upstart
sudo /etc/init.d/logstash start|stop # SysV
sudo kill -TERM $LOGSTASH_PID
PLUG-INS
--------
/usr/share/logstash/bin/logstash-plugin
logstash-plugin install logstash-filter-fingerprint
logstash-plugin install logstash-output-amazon_es
(https://github.com/awslabs/logstash-output-amazon_es)
output { amazon_es { hosts => $HOSTS region => $REGION } }
logstash-plugin list
EXAMPLES
--------
HelloWorld.conf
--
input {
tcp {
port => 10000
}
}
filter {
grok {
match => { "message" => "Hello, %{WORD:name}" }
}
}
output {
elasticsearch {
hosts => "https://aasd2342134.us-east-1.aws.found.io:9234"
user => "elastic"
password => "SAGAS@#$%DFSGFDS"
}
}
$ echo "Hello, Fred" | nc localhost 10000
$ cat sample.log.entry | nc localhost 10000
--
# basic/simple test
[sudo] logstash -e 'input { stdin {} } output { stdout {} }'
INFO logstash.pipeline - Pipeline manin started
The stdin plugin is now waiting for input:
Hello World!
2017-05-15T18:59:06.891Z praco.dev.local Hello World!
[Ctrl-D]
--
# logstash info
$ curl -s 0:9600/_node?pretty # server info
$ curl -s 0:9600/_node/stats?pretty # stats info
COMMANDS
--------
logstash -f CONFIG -t|--config.test_and_exit # test config and exit
logstash -f CONFIG -r|--config.reload.automatic # enable automatic config reloads
# get list of logging subsystems/settings
curl -XGET 'localhost:9600/_node/logging?pretty'
# change value of logging subsystems/settings
curl -XPUT 'localhost:9600/_node/logging?pretty' -H 'Content-Type: application/json' -d' { "logger.logstash.outputs.elasticsearch" : "DEBUG" } '
# Elasticsearch test query
curl -XGET '0:9200/logstash-$DATE/_search?pretty&q=response=200'
curl -XGET '0:9200/logstash-2017.05.15/_search?pretty&q=response=200'
curl -XGET '0:9200/logstash-2017.05.15/_search?pretty&q=geoip.city_name=Mountain'
curl -XGET 'http://localhost:9200/logstash-$DATE/_search?pretty&q=client:iphone'
curl -XGET 'http://localhost:9200/logstash-$DATE/_search?pretty&q=fields.type:syslog'
sample logstash configs
-----------------------
input {
beats {
id => "vmedix_beats_input_plugin"
port => "5043"
ssl => "true"
ssl_certificate => "/path/to/cert"
ssl_key => "/path/to/key" # PKCS8 format
}
cloudwatch {
port => "5044"
}
elasticsearch {
# reads query results from an Elasticsearch cluster
port => "5045"
}
exec {
# capture output of a shell command
}
file {
# stream events from file(s)
}
generator {
# generates random log events for test purposes
}
github {
# reads events from a GitHub webhook
}
redis {
# reads events from a Redis instance
}
s3 {
# streams events from files in a S3 bucket
}
snmptrap {
# creates events based on SNMP trap messages
}
sqs {
# pulls events from AWS SQS
}
}
filter {
grok {
id => "vmedix_grok_filter_plugin"
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
date {
id => "vmedix_date_filter_plugin"
match => [ "timestamp", "ISO8601" ]
geoip {
id => "vmedix_geopip_filter_plugin"
source => "clientip"
}
}
output {
# stdout {
# codec => rubydebug
# }
elasticsearch {
hosts => [ "localhost:9200" ]
}
amazon_es {
hosts => ["{{ elasticsearch_endpoint }}"]
region => "{{ elasticsearch_region }}"
index => "{{ project | lower }}-{{ env }}-%{indexname}-logs-%{+YYYY.MM.dd}"
id => "vmedix_amazon_output_plugin"
}
}
GROK
----
syntax for a grok pattern is %{SYNTAX:SEMANTIC[:DATA_TYPE]}
SYNTAX: name of the pattern that will match the text
SEMANTIC: identifier to give to the text being matched
DATA_TYPE: (Optional - data type conversion: int or float)
Custom Patterns
---------------
# Named Capture
(?<field_name>the pattern here)
(?<queue_id>[0-9A-F]{10,11})
# Create a custom patterns file
- create a directory called "patterns"
- create a file in the patterns directory with a meaningful name
# contest of ./patterns/postfix
POSTFIX_QUEUE_ID [0-9A-F]{10,11}
- use the "patterns_dir" setting in GROK plugin
filter {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUE_ID:queue_id}: %{GREEDYDATA:syslog_message}" }
}
}
tags: logstash