SYSTEMS AND METHODS FOR GENERATING DATA LINEAGE

20250110752

18/480083

October 03, 2023

Systems and methods for lineage data tracking are disclosed. Domain-specific code is obtained in response to a build request and the domain-specific code is converted to process code. The domain-specific code includes an instance of a domain-specific language configured for a specific domain. The process code includes at least one breadcrumb instrumentation element automatically inserted during the conversion process by a conversion engine and the breadcrumb instrumentation element is not defined in the domain-specific code. The process code is executed including execution of one or more activities defined by the domain-specific code and the at least one breadcrumb instrumentation element. During execution of the at least one breadcrumb instrumentation element, a lineage data structure is generated. The lineage data structure is configured to store data elements representative of a mapping between an input data element and an output data element. The lineage data structure is stored in a data store.

Wolters Kluwer Financial Services Belgium N.V. (Mechelen, BE)

1. A system, comprising: a non-transitory memory; a processor communicatively coupled to the non-transitory memory, wherein the processor is configured to read a set of instructions to: obtain domain-specific code in response to a build request, wherein the domain-specific code includes an instance of a domain-specific language configured for a specific domain; convert the domain-specific code to process code, wherein the process code includes at least one breadcrumb instrumentation element automatically inserted during conversion by a conversion engine, and wherein the breadcrumb instrumentation element is not defined in the domain-specific code; execute the process code, wherein execution of the process code includes execution of one or more activities defined by the domain-specific code and the at least one breadcrumb instrumentation element; during execution of the at least one breadcrumb instrumentation element, generate a lineage data structure configured to store data elements representative of a mapping between an input data element and an output data element; and store the lineage data structure in a data store.

2. The system of claim 1, wherein the at least one breadcrumb instrumentation element is inserted in the process code at a location corresponding to a function type defined in the domain-specific code.

3. The system of claim 1, wherein the lineage data structure comprises a tuple defining a set of inputs, a set of outputs, and a corresponding activity of the one or more activities defined by the domain-specific code.

4. The system of claim 3, wherein the set of inputs and the set of outputs each comprise at least one instance comprising a type and an identifier.

5. The system of claim 1, wherein the lineage data structure comprise a symbolic data element defining a set of inputs and a corresponding activity of the one or more activities defined by the domain-specific code.

6. The system of claim 1, wherein the at least one breadcrumb instrumentation element comprises a function call to a function defined by a processing engine configured to execute the process code.

7. The system of claim 1, wherein the breadcrumb instrumentation element comprises a scoping mechanism.

8. The system of claim 7, wherein the scoping mechanism is configured to mark a stack of potential inputs, and wherein the lineage data structure comprises inputs occurring after the mark in the stack.

9. The system of claim 1, wherein the process code comprises Java code.

10. A computer-implemented method, comprising: converting domain-specific code to process code, wherein the domain-specific code includes an instance of a domain-specific language configured for a specific domain, wherein the process code includes at least one breadcrumb instrumentation element automatically inserted during conversion by a conversion engine, and wherein the breadcrumb instrumentation element is not defined in the domain-specific code; executing the process code, wherein execution of the process code includes execution of one or more activities defined by the domain-specific code and the at least one breadcrumb instrumentation element; during execution of the at least one breadcrumb instrumentation element, generating a lineage data structure configured to store data elements representative of a mapping between an input data element and an output data element; and storing the lineage data structure in a data store.

11. The computer-implemented method of claim 10, wherein the at least one breadcrumb instrumentation element is inserted in the process code at a location corresponding to a function type defined in the domain-specific code.

12. The computer-implemented method of claim 10, wherein the lineage data structure comprises a tuple defining a set of inputs, a set of outputs, and a corresponding activity of the one or more activities defined by the domain-specific code.

13. The computer-implemented method of claim 10, wherein the lineage data structure comprise a symbolic data element defining a set of inputs and a corresponding activity of the one or more activities defined by the domain-specific code.

14. The computer-implemented method of claim 10, wherein the at least one breadcrumb instrumentation element comprises a function call to a function defined by a processing engine configured to execute the process code.

15. The computer-implemented method of claim 10, wherein the breadcrumb instrumentation element comprises a scoping mechanism.

16. The computer-implemented method of claim 15, wherein the scoping mechanism is configured to mark a stack of potential inputs, and wherein the lineage data structure comprises inputs occurring after the mark in the stack.

17. A non-transitory computer readable medium having instructions stored thereon, wherein the instructions, when executed by at least one processor, cause at least one device to perform operations comprising: obtaining domain-specific code in response to a build request, wherein the domain-specific code includes an instance of a domain-specific language configured for a specific domain; converting domain-specific code to process code, wherein the process code includes at least one breadcrumb instrumentation element automatically inserted during conversion by a conversion engine, and wherein the breadcrumb instrumentation element is not defined in the domain-specific code, wherein the at least one breadcrumb instrumentation element is inserted in the process code at a location corresponding to a function type defined in the domain-specific code; executing the process code, wherein execution of the process code includes execution of one or more activities defined by the domain-specific code and the at least one breadcrumb instrumentation element; during execution of the at least one breadcrumb instrumentation element, generating a lineage data structure configured to store data elements representative of a mapping between an input data element and an output data element; and storing the lineage data structure in a data store.

18. The non-transitory computer readable medium of claim 17, wherein the lineage data structure comprises a tuple defining a set of inputs, a set of outputs, and a corresponding activity of the one or more activities defined by the domain-specific code.

19. The non-transitory computer readable medium of claim 17, wherein the lineage data structure comprise a symbolic data element defining a set of inputs and a corresponding activity of the one or more activities defined by the domain-specific code.

20. The non-transitory computer readable medium of claim 17, wherein the at least one breadcrumb instrumentation element comprises a function call to a function defined by a processing engine configured to execute the process code.

06; 06

edspap.20250110752