Adaptive communication management system using model-based and model-free reinforcement learning

12261,893

18/921443

October 21, 2024

A system and method for managing communications through adaptive switching between model-based and model-free approaches is disclosed. The system comprises a communication management server with a master AI agent that dynamically selects between model-based and model-free reinforcement learning techniques to process incoming communications. The model-based approach utilizes a Markov Decision Process (MDP) with an environment state model, while the model-free approach employs a large language model (LLM) with retrieval-augmented generation. The system evaluates the performance and cost of both approaches in real-time, considering factors such as available historical data, observed decision quality, user goal modifications, and environmental changes. Based on this evaluation, the system adaptively switches between the two approaches or employs an ensemble method, optimizing the decision-making process for each unique communication scenario. This adaptive mechanism enables efficient handling of both familiar and novel situations, continuously improving performance while aligning with user preferences and goals.

Intelligent Communication Assistant, Inc. (Boca Raton, FL, US)

Intelligent Communication Assistant, Inc. (Boca Raton, FL, US)

1. A system for managing incoming communications, the system comprising: a multimedia gateway configured to manage real-time media; and a communication management server comprising one or more processors, a memory, and a plurality of programming instructions stored in the memory, the plurality of programming instructions when executed by the one or more processors causes the one or more processors to: receive, by a master AI agent running on the communication management server, a notification of an incoming communication associated with a user device among a plurality of user devices; determine a current performance and cost of available model-based and model-free modes for action determination; select, based on the determined performance and cost, between a model-based approach and a model-free approach for processing the incoming communication; when the model-based approach is selected: determine a current state for a user environment based on a context associated with the incoming communication, user preferences associated with the user device, and an interaction graph; invoke an action selection function (ASF) associated with an environment state model that implements a Markov Decision Process (MDP) to determine an action for processing the incoming communication; when the model-free approach is selected: generate a prompt based on the context associated with the incoming communication, the user preferences, and the interaction graph; invoke a generative completion API on a large language model (LLM) using the generated prompt to determine an action for processing the incoming communication; execute the determined action using at least one of the multimedia gateway or an AI communication agent among a plurality of AI communication agents associated with the communication management server.

2. The system of claim 1 , wherein selecting between the model-based approach and the model-free approach is based on at least one of: availability of adequate relevant historical data, observed decision quality, user goal modifications, or changes in the user's environment.

3. The system of claim 1 , wherein the plurality of programming instructions when executed by the one or more processors further causes the one or more processors to: employ an ensemble method combining outputs from both the model-based approach and the model-free approach to determine the action for processing the incoming communication.

4. The system of claim 1 , wherein the model-based approach further comprises: comparing the current state of the user environment with different states in the environment state model; responsive to non-identification of a match between the current state and the different states in the environment state model, updating the environment state model to reflect the current state; and solving the MDP to determine a new ASF.

5. The system of claim 1 , wherein the model-free approach utilizes retrieval-augmented generation to incorporate relevant information from the interaction graph and user preferences into the prompt.

6. The system of claim 1 , wherein the plurality of programming instructions when executed by the one or more processors further causes the one or more processors to: continuously evaluate the performance of the selected approach; and dynamically switch between the model-based approach and the model-free approach based on the continuous evaluation.

7. The system of claim 1 , wherein the interaction graph is processed using a Graph Neural Network (GNN) to generate embeddings used in both the model-based and model-free approaches.

8. The system of claim 1 , wherein the plurality of programming instructions when executed by the one or more processors further causes the one or more processors to store the executed action and its outcome in a vector database for future reference and learning.

9. The system of claim 1 , wherein the user preferences comprise at least one of: global objectives and goals, do not disturb (DND) hours, contact exceptions, communication type priority, integrated permissions information, and customized rules to identify senders based on keywords.

10. The system of claim 1 , wherein the plurality of programming instructions when executed by the one or more processors further causes the one or more processors to generate a reward indicative of fulfillment of the user preferences based on the executed action.

11. A method for managing incoming communications, the method comprising: receiving, by a master AI agent running on a communication management server, a notification of an incoming communication associated with a user device among a plurality of user devices; determining a current performance and cost of available model-based and model-free modes for action determination; selecting, based on the determined performance and cost, between a model-based approach and a model-free approach for processing the incoming communication; when the model-based approach is selected: determining a current state for a user environment based on a context associated with the incoming communication, user preferences associated with the user device, and an interaction graph; invoking an action selection function (ASF) associated with an environment state model that implements a Markov Decision Process (MDP) to determine an action for processing the incoming communication; when the model-free approach is selected: generating a prompt based on the context associated with the incoming communication, the user preferences, and the interaction graph; invoking a generative completion API on a large language model (LLM) using the generated prompt to determine an action for processing the incoming communication; executing the determined action using at least one of a multimedia gateway or an AI communication agent among a plurality of AI communication agents associated with the communication management server.

12. The method of claim 11 , wherein selecting between the model-based approach and the model-free approach is based on at least one of: availability of adequate relevant historical data, observed decision quality, user goal modifications, or changes in the user's environment.

13. The method of claim 11 , further comprising the step of employing an ensemble method combining outputs from both the model-based approach and the model-free approach to determine the action for processing the incoming communication.

14. The method of claim 11 , wherein the model-based approach further comprises the steps of: comparing the current state of the user environment with different states in the environment state model; responsive to non-identification of a match between the current state and the different states in the environment state model, updating the environment state model to reflect the current state; and solving the MDP to determine a new ASF.

15. The method of claim 11 , wherein the model-free approach utilizes retrieval-augmented generation to incorporate relevant information from the interaction graph and user preferences into the prompt.

16. The method of claim 11 , further comprising the steps of: continuously evaluating the performance of the selected approach; and dynamically switching between the model-based approach and the model-free approach based on the continuous evaluation.

17. The method of claim 11 , further comprising the step of processing the interaction graph using a Graph Neural Network (GNN) to generate embeddings used in both the model-based and model-free approaches.

18. The method of claim 11 , further comprising the step of storing the executed action and its outcome in a vector database for future reference and learning.

19. The method of claim 11 , wherein the user preferences comprise at least one of: global objectives and goals, do not disturb (DND) hours, contact exceptions, communication type priority, integrated permissions information, and customized rules to identify senders based on keywords.

20. The method of claim 11 , further comprising the step of generating a reward indicative of fulfillment of the user preferences based on the executed action.

10911468 February 2021 Muddu
11455576 September 2022 Dalli
11514305 November 2022 Commons
2012/0202538 August 2012 Uusitalo
2018/0121766 May 2018 McCord

edspgr.12261893