How to visualize real-time data in Kibana coming into Elasticsearch?
We all know Kibana is an open-source data visualization dashboard for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch. In today's time, it has really become necessary to visualize the data in real-time as most of the things are happening around us in real-time. Kibana lets you visualize more kinds of charts like line, pie, gauge, bar and time-series chart, here we will be discussing the time series chart.
Before you start, I presume that you have a piece of good knowledge about aggregations in Elasticsearch even though if you don’t know how to write aggregation is fine but you should know what the aggregation does. A brief explanation of aggregations is discussed here.
We will be discussing three types of aggregations in Easticsearch, first one is the metric aggregation that returns the single mathematical value as a result, for example, sum, avg, max, min, etc. are the metric aggregations that returns the sum, avg, max, min, respectively. the second aggregation is bucket aggregations that return the buckets as a result, a bucket contains key and doc_count. for example terms, range, date-histograms return the buckets. Terms aggregation is a multi-bucket value source based aggregation where buckets are dynamically built — one per unique value. Date-histogram aggregation is a multi-bucket value source based aggregation where buckets are dynamically built on the basis of interval. The third one is Pipeline aggregations which work on the outputs produced from other aggregations rather than from document sets, adding information to the output tree. There are many different types of pipeline aggregation, each computing different information from other aggregations.
As we want to visualize real data, we should have some source of data coming into Elasticsearch in real-time. here for demonstration purpose, I have used dummy script in python that inserts a record into Elasticsearch index at an interval of 30 seconds. A record consists of a timestamp, the number of bytes transferred, the number of packets transferred, and an IP. I have used some conditional statements to show the variations in the record that we are inserting at an interval of 30 seconds. I am running Elasticsearch and Kibana both on the localhost.
Step 1. Start Elasticsearch and Kibana server on your desktop, then run Demoscript.py by using command python3 Demoscript.py
Step 2. Open Kibana in your browser. In the menu, navigate to Stack Management>Index Management. Here you will see the list of indices present in Elasticsearch. You can see the myindex index is present in the list.
Step 3. Now navigate to Stack Management>Index Patterns. Now create an index pattern, type index name as myindex and select timestamp as Time filter field name, and click create index pattern.
Step 4. Now navigate to Visualize>create visualizations. Select TSVB(time Series Visual Builder).
Step 5. Now navigate to the Panel options section in the bottom right. Under the Data Panel, type myindex in Index pattern field, select timestamp as Time field. In the Interval field, you can type values as auto, 1m, 0.5m(30 seconds),≥1m, 1d, etc. I have used ≥0.5m as the data into Elasticsearch coming at an interval of 30 seconds. the time interval can be
Step 6. In TSVB visualization, the first thing that we need to understand that all the aggregation on the y-axis is applied on the basis of buckets formed on the x-axis.
The x-axis is the time axis that takes the value from time-field i.e. is a timestamp. Date-histogram applied on a timestamp field which forms multiple buckets for the time-range on the basis of interval. Time-range can be selected from top right corner as shown in Fig. 2. if time-range is selected like the last 1 month, then the interval may be extended to 1 hour dynamically even though we have set interval as ≥0.5m, cause the maximum number of buckets formed on the x-axis can be 2000, not more than this. don’t forget to use greater than equal sign(≥0.5m) before the interval, it allows the Kibana to use dynamic interval if the number of buckets goes to above 2000.
The y-axis is the axis on which different aggregations are applied which are calculated over the buckets formed on the time-axis.
Visualization 1. create a time-series graph that shows the number of bytes transferred in every 30 seconds and the maximum number of bytes transferred over the whole time-range.
To create this visualization, We will use 2 series aggregations. Now navigate to the data tab shown in Fig. 2., In the first series aggregation, In metrics tab select sum in Aggregations and select bytes in Fields. Add second series aggregation by clicking on the + sign.
In the second series aggregation, metric and pipeline aggregations are used. to create metric aggregation, In metrics tab select sum in Aggregations and select bytes in Fields. Now to create pipeline aggregations, add another metric, and in this added metric field, select overall max in Aggregations and select sum of bytes in Fields. Now navigate to Panel options, you can select Data Formatter as Bytes.
Visualization 2. create a time-series graph that shows the bytes speed in bytes/second and the maximum bytes speed over the whole time-range.
we will make some changes to the visualization1 data tab, we will add one more metric aggregation i.e. math(special aggregations) in both series aggregation.
Now navigate to the Data tab of visualization1, In the first series aggregation, add metric aggregation in last, select math in Aggregations, then type currentbytes inside variable name and select sum of bytes in Field. Now write under expression field: params.currentbytes/(params._interval/1000). In the second series aggregation, add metric aggregation in last, select math in Aggregations, then type maxbytes inside variable name and select sum of bytes in Field. Now write under expression field: params.maxbytes/(params._interval/1000).
Note 1- params._interval gives time in milliseconds, to convert into seconds, we need to divide it by 1000 in order to get current speed and max speed per second. Note 2- Navigate to Options tab, select Bytes in Data Formatter and change Template fiels as {{value}}/sec.
You can also select Refresh interval, click on the drop-down arrow in the top right corner(near to Last 30 minutes), now type interval like 30 seconds in refresh every field. It will refresh the time-series graph automatically every 30 seconds.
