Automating Log Reviews: A Python Script for Efficient Error Detection

Have you ever had to find a single line of text in a multiple-page document with the infamous control F function? Here at Kinetech, our Business Engineer, Brandon Bremer, has created a Python script to find exactly what we are looking for when faced with a wall of text.

Importance of Log Review

In the dynamic nature of software development, log reviews are a necessity for application excellence. Logs serve as the digital footprint of an application as it records the literal processes of the application’s functionality, errors, and user interactions. Logs are the main tool for troubleshooting and debugging errors as they provide detailed reports of what causes an error. By analyzing these logs, developers can identify underlying and current issues, find ways to improve performance and detect possible security issues. 

Log reviews allow developers to maintain the application’s functionality and health. Consistent log reviews ensure that developers know what happens within the application and proactively solve errors before they escalate into critical issues for the application and its users. Without a log reviewing program, log reviews are a manual and time-staking process to look through and understand. 

The program that Kinetech has developed allows our support team to compile the logs of our apps and generate a readable and customized output of an error log. This organized error log provides valuable insight into fixing bugs and maintaining support for Kinetech applications that are currently in production.

How we used to Vs. How we do it now

Before the implementation of this program, log reviews had to be done manually. Manually going through log reviews can take a long time as it requires a person to sift through thousands of log entries and actively search for a specific line of text or recognize trends and patterns users face. This method of scrolling manually was not only time-consuming but also led to human error as details would be overlooked by the sheer number of general data.

The new program we have now is a simple but efficient solution to the log review process. By using a Python script and the text files of the logs, we can provide a list of text to search and extract the pertinent information related to what the program finds. The program can also be provided with a list of text to ignore. After scanning the logs, an output text file is generated with the logs categorized by the time the error occurred. 

We will talk about how exactly it works in the next section.

Screenshots/GIF of how it works

We first start with an IDE (Integrated Development Environment) such as Visual Studio to run Python. Then the program file (FileReader.ipynb) is loaded in with the log files that we want to search over. The program works by opening each text file and searching each line for a keyword such as ‘Error’ and saves a customizable amount of lines around the found text. 

After scanning the logs, a compiled list of the errors is generated in an output.txt file.

Exclusions/Customizing the output

To filter out certain errors from getting picked up and ultimately excluded from the final output file, you can add specific texts to the ignoreme.txt file. This file allows us to skip over certain errors that have already been addressed by the team and lowers the amount of redundant errors for the final output.

To change the number of lines retrieved from the error log entry, there is a variable, numberOfLines, that can be adjusted. Given most errors are lengthy as a whole, this allows the program’s user to print only the most relevant information about an error. The search criteria for the logs can also be changed. Rather than searching for instances of ‘Error’, the search can include specific usernames or specific occurrences of actions. 

Output

Here is an example entry from one of our application’s logs. Each output entry starts with the error line that was detected with the program.

The next lines provide detailed information on the error such as when/where it occurred or a reason for why the action could not be completed. These lines are generated by the numberOfLines variable.

Lastly, the entry writes out which file it came from so the user can find the full error log for additional context. 

An average error entry provides us with who created the error, when/where the application the error occurred, and reasoning for why the action could not execute. This allows the support team to recreate the error and take the appropriate actions to prevent the error from happening again. Additional steps of addressing these issues range from debugging each step or reaching out to the user for more information. Once the fix has been identified and developed, our clients verify the functionality and the support team pushes the changes to production.

Conclusion

Log reviews continuously serve as an important practice in maintaining the health and efficiency of Kinetech applications. The evolution of log reviews, from manual to automated, provides an efficient way to observe the integral ongoings proactively. This is one of the ways that Kinetech supports our production applications after going live. Want to join the Kinetech team to help innovate? Click here for job postings.