Scientific works and Related works of ELK stack

Scientific works and Related works of ELK stack

Scientific works of ELK Stack

In the first paragraph helps you to grab some knowledge regarding scientific works. you can apply them to your day to day software developments and business enhancements and, the second paragraph shows some real-world example which used ELK Stack, before that it is better to read my previous article related to ELK stack.

  1. Using ELK stack, the user can view the no of clients, average response time, slow responses, slow nodes, high traffic using the volume of messages for given time period using the visualization dashboard of Kibana. Logstash collect the data and splitting and push to Elasticsearch database and it uses Kibana to visualization.
  2. ELK used to detect cyber-attacks. Logstash can ability to read iptables (is a front-end for configuring the Linux firewall) and syslogs. After those data pushing to Elasticsearch and Kibana used to display the information from iptables and syslog events provided by Elasticsearch.
  3. ELK used to manage the NGIX logs. NGIX contains valuable data like client requests, active client in the movement. Installing Logstash can ability to transport NGIX to Elasticsearch. After Kibana can be used to visualize and analyze that transported NGIX logs.
  4. ELK is working as a monitoring tool to examine IAAS (infrastructure as a service) and application running on hosted virtual instance. Accounting data of IAAS is stored in MySQL data bases. Logstash listen the MySQL database for new records, if found it sends to Elasticsearch and Kibana dash board displays relevant information in each case, according to pre-define queries. E.g. Number of queries per user, CPU monitoring.

Popular organizations who used ELK Stack

above mentioned few scientific works of ELK stack used below large companies as their best practices.

Netflix

Netflix Inc is a one of the major media service provider in America started on 1997. Their main service is based on subscriptions based on streaming services. It offers library of movies, teledrama and television programs those are constructed in-house. Beginning of the time they sold DVD and rent via mail. Then after aimed DVD rental business in 2007. The business has been expanded according to the introduction of streaming media in 2010.

After expanded internationally Netflix operated 190 countries around the world except few countries. As of July-2018 it has 130 million subscribers around the world (Anon., 2018) . In January 2014, Netflix was generating over 70 billion log events a day, Netflix’s diverse data needs include the requirement to store, search and index documents. That’s why Elasticsearch has found niche in Netflix. Usage of Elasticsearch in Netflix has been increased over the last few years. Now it is using 100+ clusters (700-800 nodes)

Netflix | Logging | Solution

Stack Overflow

The Stack overflow is a popular question and answer website introduced in 2008 that is used monolithic architecture and MySQL and C# to develop. It mainly focuses on computer programming questions and answers in different aspects. the flagship site of Stack overflow is the Stack Exchange Network. As of today registered user count is more than 9.2 million. (9191225) and 16 million questions and 25 million answers. Researchers found more than 75% of users have asked only one question, only 65% of users have answered only one question. Hence there is a mechanism in stock exchange network to delete unanswered questions for a certain amount of time period because it is the unwanted weight for the servers.
E.g. In 2016, 1.6 Million questions have been deleted.
In 2015 it handles 4 billion requests per month, 3000 requests/ peak, and 800 M SQL queries per day, 8500/s peak.
Below are few statistics for 2016 according to Nick Craver.

  • 209,420,973 (+61,336,090) HTTP requests to our load balancer
  • 66,294,789 (+30,199,477) of those were page loads
  • 1,240,266,346,053 (+406,273,363,426) bytes (1.24 TB) of HTTP traffic sent
  • 569,449,470,023 (+282,874,825,991) bytes (569 GB) total received
  • 3,084,303,599,266 (+1,958,311,041,954) bytes (3.08 TB) total sent
ELK | Stack Overflow | Stack Exchange | Solution

LinkedIn

LinkedIn started in 2003 for connect around the world and find better job opportunities. Initially it had only 2700 members but over the time today it increased till 350 million members. It serves tens of thousands of web pages to users all the day. 50% of users are coming via mobile platform. All the requests are handle from back-end system which handle millions of queries per second. LinkedIn started as many sites start today, as a single monolithic application doing it all. Leo is the name of that single application. It manage all the business logic and servlets for all the various pages. Few years later LinkedIn moved to SOA.

Over the time LinkedIn used to Elasticsearch, Logstash, Kibana and Kafka to achieve below use cases,

  • Operational troubleshooting
  • Exception searching
  • Security auditing
  • Product tracking

LinkedIn used Elasticsearch as near real-time search and analytic engine, Logstash as a collect logs and parse to JSON, Kibana used to visualization and Kafka used as a data transfer queue. Kafka at LinkedIn,

  • Overall stats
    • 875 Billion messages per day
    • 185 Terabytes In
    • 675 Terabytes Out
  • Peak Load
    • 10.5 Million messages/sec
    • 18.5 Gigabits/sec Inbound
    • 70.5 Gigabits/sec Outbound

Nowadays 100+ ELK clusters across 20+ teams and 6 data centers used LinkedIn. Below are the cluster stats.

  • 250 TB data indexed
  • 325 billion total documents
  • 40 ELK clusters per data center
ELK | LinkedIn | Solution
ELK | LinkedIn | Solution

More organizations.

ELK | Users | Facebook | LinkedIn | Stack overflow | Ebay

Reference

https://en.wikipedia.org/wiki/Netflix
https://logz.io/blog/15-tech-companies-chose-elk-stack
https://logz.io/blog/15-tech-companies-chose-elk-stack/
https://stackexchange.com/sites?view=list#questionsperday
https://www.infoq.com/news/2015/06/scaling-stack-overflow
https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/
https://engineering.linkedin.com/architecture/brief-history-scaling-linkedin