Build your management
There is always a big focus on the problem on the first day while implementing a system. Over time, new problems arise and remain to the system implementer. Questions such as;
- What are the simple steps that users can take to make changes safely and reliably?
- What are the indicators to show the user if the system is working optimally?
- In what ways can the user solve problems in this system?
Fluent and Logstash
Fluent and Logstash are the most appropriate log processing tools for day 1. Working through it, building an initial configuration can be seamless. But the big question lies in its scalability and how long it lasts.
To Fluentd or logstashManaging configurations with the appropriate configuration changes requires significant system investment.
One success feature of organizations that build viewing tubes is that they have created unique frameworks for testing individual units for intensive code and configuration review and configuration processes.
These processes were not invented for no reason. Prior to the implementation of these processes, minor configuration errors would regularly violate production. This process also includes implementing your continuous deployment pipe to safely transfer changes to production. In addition to the CI / CD, which needs to be built, monitoring comes next. Suppose the data arrives late or the system slows down for any reason. In this case, metrics should be available to determine which system is causing recurrent pressure or which particular set of configurations may be to blame.
One incorrect regular expression can ruin the performance of a data processing system. Building rich dashboards for monitoring is a significant part of the second day of building your system. Finally, when the data is not processed in the way that end users expect, system operators need to recover the data processing to verify why a given configuration produces a given output. There is no environment like manufacturing. Attempting to take a configuration and environment where the data conditions recreate the problem can be very time consuming. Tools like Fluentd and Logstash provide little introspection, data capture or troubleshooting capability.