Figure 1. ELK Stack |
Logstash is the workhorse that collects the log files from Application Servers, parses them, formats them and sends them to Elastic Search. Elastic Search stores and indexes the data, which is presented by Kibana. The end users access Kibana Web Interface to view the data.
One of the great things about Elasticsearch is its extensive JSON based REST API which allows you to integrate, manage and query the indexed data in countless different ways.
Elastic Search
Elastic Search is a robust search and analytics tool that stores data in a document oriented data store. It has the following features:
- Is built on top of high performance open source search engine Apache Lucene (in Java)
- Started off as scalable Lucene
- Horizontally scalable search engine
- Often a faster solution than Hadoop/Spark/Flink/etc.
- Document oriented storage
- The document oriented storage differs sharply from traditional table oriented RDBMS (Such as Oracle, MS SQL Server).
- With document oriented data storage, data is stored as structured JSON documents.
- Every field is indexed by default.
- This is why the search speed is incredible.
- Each shard is an inverted index of documents.
- Consumes data from Logstash
- Creates Indexes for log files, typically a date-based index
- Its architecture favors distribution
- You can scale your Elastic Search infrastructure massively and seamlessly.
curl 'http://localhost:9200/?pretty'
to get a response like this:[2]
{
"name" : "Tom Foster",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.1.0",
"build_hash" : "72cd1f1a3eee09505e036106146dc1949dc5dc87",
"build_timestamp" : "2015-11-18T22:40:03Z",
"build_snapshot" : false,
"lucene_version" : "5.3.1"
},
"tagline" : "You Know, for Search"
}
Video 2. Logstash Overview
Logstash
Logstash, another open source tool does the heavy lifting of consuming the logs from various systems
and sends them to Elastic Search. It has the following features:
- Is a tool for collecting & monitoring logs from remote machines
- Data pushed and stored in Elastic Search
- Is a data pipeline for Elasticsearch
- Parses, transforms, and filters data as it passes through
- Can derive structure from unstructured data
- Can anonymize personal data or exclude it entirely
- Can do geo-location lookups
- Guarantees at-least-once delivery
- Absorbs throughput from load spikes
- Can scale across many nodes
Kibana
Kibana is an open source data exploration & visualization platform that is the presentation layer in the ELK stack. It consumes
data from Elastic Search Indexes. A user accesses Kibana interface via a web browser.
- Used for
- Log and time series analytics, application monitoring & operational intelligence
- Make queries in Elastic Search
- Enables the searching & interaction with data in Elastic Search
- Allows performing advanced analytics & creation of reports
- Provide Real-time Dashboard
- Enables creation & sharing of dynamic dashboards that get updated in realtime
1 comment:
brigade eldorado is committed to creating sustainable living environments, and Brigade Eldorado is no exception.
Post a Comment