Systems and processes for computer log analysis

10484,506

15/641282

July 04, 2017

Existing program code, which is executable on one or more computers forming part of a distributed computer system, is analyzed. The analysis identifies log output instructions present in the program code. Log output instructions are those statements or other code that generate log messages related to service requests processed by the program code. A log model is generated using the analysis. The log model is representative of causal relationships among service requests defined by the program code. The log model can then be applied to logs containing log messages generated by execution of the program code, during its normal operation, to group log messages for improved analysis, including visualization, of the performance and behaviour of the distributed computer system.

YScope Inc. (Toronto, CA)

YSCOPE INC. (Toronto, CA)

1. A process for performing computer log analysis, the process comprising: performing an analysis on existing program code that is executable on one or more computers forming part of a distributed computer system, the analysis identifying log output instructions present in the program code, the log output instructions being configured to generate log messages related to service requests processed by the program code; generating a log model using the analysis, the log model being representative of causal relationships among service requests defined by the program code, generating the log model including automatically generating one or more groups for grouping log messages based on the analysis on the existing program code, generating the log model further including generating a direct acyclic graph containing log points reachable from a top-level method associated with a particular service request to group a first log message with other log messages associated with the particular service request; and applying the log model to a plurality of log messages generated by execution of the program code to add log messages of the plurality of log messages into the one or more groups.

2. The process of claim 1 , wherein generating the log model comprises identifying a log output statement in the program code and parsing the log output statement to determine a format of the log output statement and any variable referenced by the log output statement to generate a signature of the log output statement.

3. The process of claim 2 , further comprising using a value of a variable referenced by the log output statement to generate the signature for the log output statement.

4. The process of claim 3 , storing in the signature of the log output statement as a regular expression.

5. The process of claim 1 , wherein generating the log model comprises analyzing dataflow of variables of the program code and storing in the log model one or more indications of one or more variables not modified by a particular service as a request identifier of the particular service request.

6. The process of claim 5 , wherein generating the log model comprises identifying a top-level method in the program code as a first method of any thread dedicated to the processing of a single type of service request and storing an identifier of the top-level method in the log model.

7. The process of claim 6 , further comprising determining that the particular service request is processed by the top-level method and storing in the log model the request identifier of the particular service request in association with the identifier of the top-level method.

8. The process of claim 1 , wherein generating the log model comprises identifying log output statements in the program code and generating indications of temporal associations or dissociations among the log output statements.

9. The process of claim 1 , wherein generating the log model comprises identifying log output statements in the program code, identifying threads that contain the log output statements, determining which pairs of the threads communicate with each other, and storing an indication of any determined communicating pairs of threads in the log model.

10. The process of claim 9 , wherein determining which pairs of the threads communicate with each other comprises detecting thread creation.

11. The process of claim 9 , wherein determining which pairs of the threads communicate with each other comprises determining whether two instructions reachable from two top-level methods operate on a shared object, wherein each of the two top-level methods is a first method of any thread dedicated to the processing of a single type of service request.

12. The process of claim 11 , wherein applying the log model comprises adding a particular log message to a particular group on a condition that the particular log message shares a common top-level method with the group.

13. The process of claim 12 , wherein the adding is further based on a condition that one or more request identifiers of the particular log message are not different from one or more request identifiers of the group.

14. The process of claim 13 , wherein the one or more request identifiers of the particular log message and the one or more request identifiers of the group are indicative of variables in the program code that are not modified during processing of a service request.

15. The process of claim 12 , wherein the adding is further based on a condition that a log point of the particular log message matches a temporal sequence in a control flow of the program code.

16. The process of claim 12 , further comprising storing the log model at a logging computer of the distributed computer system, the logging computer performing the applying of the log model.

17. A log processing system comprising: at least one computer to perform an analysis on existing program code that is executable on one or more computers forming part of a distributed computer system, the analysis identifying log output instructions present in the program code, the log output instructions being configured to generate log messages related to service requests processed by the program code; the at least one computer further to use the analysis to generate a log model of causal relationships among service requests defined by the program code, generation of the log model including automatically generating one or more groups for grouping log messages based on the analysis on the existing program code, generating of the log model further including generating a directed acyclic graph containing log points reachable from a top-level method associated with a particular service request to group a first log message with other log messages associated with the particular service request; and the at least one computer further to apply the log model to a plurality of log messages generated by execution of the program code to add log messages of the plurality of log messages into the one or more groups.

18. The system of claim 17 , further comprising a results database configured to store indications of the one or more groups.

19. The system of claim 18 , further comprising a visualization engine configured to display data contained in the results database.

5819094 October 1998 Sato
2005/0223368 October 2005 Smith
2012/0072399 March 2012 Taitel

Yuan et al., “Improving Software Diagnosability via Log Enhancement” (Year: 2012). cited by examiner
Xu et al., “Detecting Large-Scale System Problems by Mining Console Logs” (Year: 2010). cited by examiner
Zou et al., “UiLog: Improving Log-Based Fault Diagnosis by Log Analysis” (Year: 2016). cited by examiner
Cheong et al., “Automatically Generating Summary Visualizations from Game Logs” (Year: 2008). cited by examiner
Ding Yuan et al., “Improving Software Diagnosability via Log Enhancement” (Year: 2011). cited by examiner

edspgr.10484506