Kibana―Knowing the ELK Basics

Figure 1.  ELK Stack

The most popular usage of Elasticsearch today is Log Management. It is developed alongside a data-collection and log-parsing engine called Logstash, and an analytics and visualization platform called Kibana. The three products are designed for use as an integrated solution, referred to as the "Elastic Stack" (formerly the "ELK stack"):

Logstash is the workhorse that collects the log files from Application Servers, parses them, formats them and sends them to Elastic Search. Elastic Search stores and indexes the data, which is presented by Kibana. The end users access Kibana Web Interface to view the data. 

One of the great things about Elasticsearch is its extensive JSON based REST API which allows you to integrate, manage and query the indexed data in countless different ways.

Elastic Search 

Elastic Search is a robust search and analytics tool that stores data in a document oriented data store.  It has the following features:

• Is built on top of high performance open source search engine Apache Lucene (in Java)
• Started off as scalable Lucene
• Horizontally scalable search engine
• Often a faster solution than Hadoop/Spark/Flink/etc.
• Document oriented storage
• The document oriented storage differs sharply from traditional table oriented RDBMS (Such as Oracle, MS SQL Server).
• With document oriented data storage, data is stored as structured JSON documents.
• Every field is indexed by default. 
• This is why the search speed is incredible.
• Each shard is an inverted index of documents.
• Consumes data from Logstash 
• Creates Indexes for log files, typically a date-based index
• Its architecture favors distribution
• You can scale your Elastic Search infrastructure massively and seamlessly.

You can get the latest version of Elasticsearch from elastic.co/downloads/elasticsearch.  Once you have an instance of ElasticSearch up and running you can talk to it using it's REST API residing at localhost port 9200.  For example, you can run the following curl command:

curl 'http://localhost:9200/?pretty'

to get a response like this:^[2]

{
  "name" : "Tom Foster",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.1.0",
    "build_hash" : "72cd1f1a3eee09505e036106146dc1949dc5dc87",
    "build_timestamp" : "2015-11-18T22:40:03Z",
    "build_snapshot" : false,
    "lucene_version" : "5.3.1"
  },
  "tagline" : "You Know, for Search"
}

Video 2.  Logstash Overview

Logstash

Logstash, another open source tool does the heavy lifting of consuming the logs from various systems and sends them to Elastic Search.  It has the following features:

• Is a tool for collecting & monitoring logs from remote machines
• Data pushed and stored in Elastic Search
• Is a data pipeline for Elasticsearch
• Parses, transforms, and filters data as it passes through
• Can derive structure from unstructured data
• Can anonymize personal data or exclude it entirely
• Can do geo-location lookups
• Guarantees at-least-once delivery
• Absorbs throughput from load spikes
• Can scale across many nodes

Kibana

Kibana is an open source data  exploration & visualization platform that is the presentation layer in the ELK stack. It consumes data from Elastic Search Indexes. A user accesses Kibana interface via a web browser.

• Used for 
• Log and time series analytics, application monitoring & operational intelligence
• Make queries in Elastic Search
• Enables the searching & interaction with data in Elastic Search
• Allows performing advanced analytics & creation of reports
• Provide Real-time Dashboard
• Enables creation & sharing of dynamic dashboards that get updated in realtime

The default settings configure Kibana to run on localhost:5601. To change the host or port number, or connect to Elasticsearch running on a different machine, you'll need to update your kibana.yml file.  For more information, read:

• Configuring Kibana
• Getting Kibana Up and Running

References

1. Basic Concepts (Apache  Lucene)
2. Installing and Running Elasticsearch
3. 15 Minutes Introduction to ELK (Elastic Search,LogStash,Kibana)
4. Configuring Kibana
5. Getting Kibana Up and Running
6. Filter Plugins (Logstach)
7. Elasticsearch Architecture (youtube)