How do I add a summary of each user's history and feedback to my prompts without slowing down responses?

We recommend updating the summary in the background and letting requests read the latest finished version, even if it is slightly stale. Our Motion system (SIGMOD 2024 demo) treats a prompt's parts as views that are updated as new feedback arrives, under a weak consistency model. In our shopping demo, a fresh GPT-4 summary could take over thirty seconds, while Motion returned recommendations in 1.5 seconds.

Sources: Building Reactive Large Language Model Pipelines with Motion (SIGMOD 2024 (Demo))

More questions on this topic

All questions about using AI for data and knowledge work