Use of batch mode function execution in database engines to enable efficient calls to remote services

11797,535

17/105092

November 25, 2020

Techniques for batch mode execution for calls to remote services are described. A method of batch mode execution for calls to remote services may include generating, by a query service of a provider network, a query plan to optimize a query for batch processing of data, the query plan including at least a function reference to a function provided by at least one service of the provider network, executing the query plan to invoke the function associated with the function reference, wherein a batch function generates a request including a batch of service calls to be processed by the at least one service, sends the request including the batch of service calls to the at least one service, and obtains a plurality of machine learning responses from the at least one service, and generating a query response based on the plurality of responses.

Amazon Technologies, Inc. (Seattle, WA, US)

Amazon Technologies, Inc. (Seattle, WA, US)

1. A computer-implemented method comprising: receiving, by a query service of a provider network, a query, the query including one or more operations to be performed by a machine learning service on data stored in a storage service of the provider network; generating, by a query engine of the query service, a query plan to optimize the query for batch processing by moving one or more function references to the machine learning service in the query to a SELECT list of the query; executing, by a batch manager of the query engine, the query plan by: calling a batch operator of the query plan, causing the batch operator to call a batch function associated with the machine learning service; gathering, by the batch operator, multiple data rows; generating, by the batch operator, a batch of service calls including the multiple data rows to be processed by the machine learning service; providing, by the batch function, the batch of service calls including the multiple data rows to the machine learning service in a single request; and receiving, by the batch function, a plurality of responses from the machine learning service; and generating a query response based on the plurality of responses.

2. The computer-implemented method of claim 1 , wherein the batch of service calls is associated with a batch size determined by the query service based on observed throughput of the query engine.

3. The computer-implemented method of claim 1 , wherein the one or more function references are application programming interface calls or user-defined function calls.

4. A computer-implemented method comprising: generating, by a query service of a provider network, a query plan to optimize a query for batch processing of data, the query plan including at least a function reference to a batch function provided by at least one service of the provider network; executing, by a batch manager of the query service, the query plan by: calling a batch operator of the query plan, causing the batch operator to call the batch function; gathering, by the batch operator, multiple data rows; generating, by the batch operator, a request including a batch of service calls, the batch of service calls including the multiple data rows to be processed by the at least one service; sending, by the batch function, the request, including the batch of service calls, to the at least one service; and obtaining, by the batch function, a plurality of responses from the at least one service; and generating a query response based on the plurality of responses.

5. The computer-implemented method of claim 4 , wherein executing, by the batch manager of the query service, the query plan further comprises: launching at least one worker node that executes the query plan and calls the batch operator of the query plan, wherein the batch operator: processes a first response from the plurality of responses; and caches any remaining responses from the plurality of responses at the at least one worker node.

6. The computer-implemented method of claim 5 , further comprising: optimizing the query plan by migrating the function reference to a SELECT list of the query from any other portion of the query.

7. The computer-implemented method of claim 4 , wherein the batch manager is implemented using a batch iterator generated based at least on portions of the query plan.

8. The computer-implemented method of claim 7 , wherein executing, by the batch manager of the query service, the query plan further comprises: launching at least one worker node that executes the query plan and requests a next tuple from the batch iterator, wherein the batch iterator: sends a plurality of next requests to one or more sources of the batch iterator; receives a plurality of next responses from the one or more sources, wherein the batch of service calls is based at least on the plurality of next responses; returns a first response from the plurality of responses to the at least one worker node; and caches any remaining responses from the plurality of responses at the at least one worker node.

9. The computer-implemented method of claim 4 , wherein the at least one service is a machine learning service hosting a custom machine learning model, a machine learning-backed service having a machine learning model provided by the provider network, or a machine learning-backed service hosting the custom machine learning model.

10. The computer-implemented method of claim 9 , wherein the machine learning service or the machine learning-backed service receives the request including the batch of service calls and performs inference using a machine learning model to generate the plurality of responses.

11. The computer-implemented method of claim 9 , wherein the batch of service calls is associated with a batch size determined by the query service based on observed throughput of a query engine of the query service.

12. The computer-implemented method of claim 4 , further comprising: storing the plurality of responses in a cache; identifying, during execution of the query plan, a next service call associated with a data item associated with at least one of the plurality of responses stored in the cache; and retrieving the at least one of the plurality of responses from the cache.

13. A system comprising: a first one or more electronic devices implementing a machine learning service in a multi-tenant provider network; and a second one or more electronic devices implementing a query service in the multi-tenant provider network, the query service including instructions that upon execution by one or more processors cause the query service to: generate a query plan to optimize a query for batch processing of data, the query plan including at least a function reference to a batch function provided by the machine learning service; execute, by a batch manager of the query service, the query plan by: calling a batch operator of the query plan, causing the batch operator to call the batch function; gathering, by the batch operator, multiple data rows; generating a request including a batch of service calls, the batch of service calls including the multiple data rows to be processed by the machine learning service; sending, by the batch function, the request, including the batch of service calls, to the machine learning service; and obtaining, by the batch function, a plurality of responses from the machine learning service; and generate a query response based on the plurality of responses.

14. The system of claim 13 , wherein to execute the query plan the instructions comprise further instructions that, when executed by the one or more processors, further cause the query service to: launch at least one worker node that executes the query plan and calls the batch operator of the query plan, wherein the batch operator: processes a first response from the plurality of responses; and caches any remaining responses from the plurality of responses at the at least one worker node.

15. The system of claim 14 , wherein the instructions comprise further instructions that, when executed by the one or more processors, further cause the query service to: optimize the query plan by migrating the function reference to a SELECT list of the query from any other portion of the query.

16. The system of claim 13 , wherein the batch manager is implemented using a batch iterator generated based at least on portions of the query plan, and wherein to execute the query plan the instructions comprise further instructions that, when executed by the one or more processors, further cause the query service to: launch at least one worker node that executes the query plan and requests a next tuple from the batch iterator, wherein the batch iterator: sends a plurality of next requests to one or more sources of the batch iterator; receives a plurality of next responses from the one or more sources, wherein the batch of service calls is based at least on the plurality of next responses; returns a first response from the plurality of responses to the at least one worker node; and caches any remaining responses from the plurality of responses at the at least one worker node.

17. The system of claim 13 , wherein the machine learning service receives the request including the batch of service calls and performs inference using a machine learning model associated with the machine learning service to generate the plurality of responses.

18. The system of claim 13 , wherein the batch of service calls is associated with a batch size determined by the query service based on observed throughput of a query engine of the query service.

19. The system of claim 13 , wherein the machine learning service hosts a custom machine learning model.

20. The system of claim 13 , wherein the instructions comprise further instructions that, when executed by the one or more processors, further cause the query service to: store the plurality of responses in a cache; identify, during execution of the query plan, a next service call associated with a data item associated with at least one of the plurality of responses stored in the cache; and retrieve the at least one of the plurality of responses from the cache.

20150379424 December 2015 Dirac
20200004596 January 2020 Sengupta
20200065303 February 2020 Bhattacharjee

edspgr.11797535