This publish is co-written with Lee Rehwinkel from Planview.
Companies at present face quite a few challenges in managing intricate initiatives and packages, deriving helpful insights from huge information volumes, and making well timed selections. These hurdles often result in productiveness bottlenecks for program managers and executives, hindering their means to drive organizational success effectively.
Planview, a number one supplier of linked work administration options, launched into an bold plan in 2023 to revolutionize how 3 million world customers work together with their mission administration functions. To comprehend this imaginative and prescient, Planview developed an AI assistant known as Planview Copilot, utilizing a multi-agent system powered by Amazon Bedrock.
Growing this multi-agent system posed a number of challenges:
Reliably routing duties to acceptable AI brokers
Accessing information from varied sources and codecs
Interacting with a number of utility APIs
Enabling the self-serve creation of recent AI expertise by completely different product groups
To beat these challenges, Planview developed a multi-agent structure constructed utilizing Amazon Bedrock. Amazon Bedrock is a totally managed service that gives API entry to basis fashions (FMs) from Amazon and different main AI startups. This enables builders to decide on the FM that’s finest suited to their use case. This strategy is each architecturally and organizationally scalable, enabling Planview to quickly develop and deploy new AI expertise to fulfill the evolving wants of their clients.
This publish focuses totally on the primary problem: routing duties and managing a number of brokers in a generative AI structure. We discover Planview’s strategy to this problem throughout the growth of Planview Copilot, sharing insights into the design selections that present environment friendly and dependable job routing.
We describe custom-made home-grown brokers on this publish as a result of this mission was applied earlier than Amazon Bedrock Brokers was typically obtainable. Nonetheless, Amazon Bedrock Brokers is now the really useful answer for organizations trying to make use of AI-powered brokers of their operations. Amazon Bedrock Brokers can retain reminiscence throughout interactions, providing extra customized and seamless person experiences. You may profit from improved suggestions and recall of prior context the place required, having fun with a extra cohesive and environment friendly interplay with the agent. We share our learnings in our answer that can assist you understanding use AWS know-how to construct options to fulfill your objectives.
Answer overview
Planview’s multi-agent structure consists of a number of generative AI parts collaborating as a single system. At its core, an orchestrator is accountable for routing questions to varied brokers, accumulating the realized data, and offering customers with a synthesized response. The orchestrator is managed by a central growth staff, and the brokers are managed by every utility staff.
The orchestrator contains two principal parts known as the router and responder, that are powered by a big language mannequin (LLM). The router makes use of AI to intelligently route person questions to varied utility brokers with specialised capabilities. The brokers might be categorized into three principal sorts:
Assist agent – Makes use of Retrieval Augmented Era (RAG) to offer utility assist
Knowledge agent – Dynamically accesses and analyzes buyer information
Motion agent – Runs actions throughout the utility on the person’s behalf
After the brokers have processed the questions and supplied their responses, the responder, additionally powered by an LLM, synthesizes the realized data and formulates a coherent response to the person. This structure permits for a seamless collaboration between the centralized orchestrator and the specialised brokers, which gives customers an correct and complete solutions to their questions. The next diagram illustrates the end-to-end workflow.
Technical overview
Planview used key AWS companies to construct its multi-agent structure. The central Copilot service, powered by Amazon Elastic Kubernetes Service (Amazon EKS), is accountable for coordinating actions among the many varied companies. Its obligations embody:
Managing person session chat historical past utilizing Amazon Relational Database Service (Amazon RDS)
Coordinating site visitors between the router, utility brokers, and responder
Dealing with logging, monitoring, and accumulating user-submitted suggestions
The router and responder are AWS Lambda capabilities that work together with Amazon Bedrock. The router considers the person’s query and chat historical past from the central Copilot service, and the responder considers the person’s query, chat historical past, and responses from every agent.
Software groups handle their brokers utilizing Lambda capabilities that work together with Amazon Bedrock. For improved visibility, analysis, and monitoring, Planview has adopted a centralized immediate repository service to retailer LLM prompts.
Brokers can work together with functions utilizing varied strategies relying on the use case and information availability:
Current utility APIs – Brokers can talk with functions by way of their present API endpoints
Amazon Athena or conventional SQL information shops – Brokers can retrieve information from Amazon Athena or different SQL-based information shops to offer related data
Amazon Neptune for graph information – Brokers can entry graph information saved in Amazon Neptune to assist complicated dependency evaluation
Amazon OpenSearch Service for doc RAG – Brokers can use Amazon OpenSearch Service to carry out RAG on paperwork
The next diagram illustrates the generative AI assistant structure on AWS.
Router and responder pattern prompts
The router and responder parts work collectively to course of person queries and generate acceptable responses. The next prompts present illustrative router and responder immediate templates. Further immediate engineering can be required to enhance reliability for a manufacturing implementation.
First, the obtainable instruments are described, together with their objective and pattern questions that may be requested of every software. The instance questions assist information the pure language interactions between the orchestrator and the obtainable brokers, as represented by instruments.
Subsequent, the router immediate outlines the rules for the agent to both reply on to person queries or request data by way of particular instruments earlier than formulating a response:
The next is a pattern response from the router part that initiates the dataQuery software to retrieve and analyze job assignments for every person:
The next is a pattern response from the responder part that makes use of the dataQuery software to fetch details about the person’s assigned duties. It experiences that the person has 5 duties assigned to them.
Mannequin analysis and choice
Evaluating and monitoring generative AI mannequin efficiency is essential in any AI system. Planview’s multi-agent structure allows evaluation at varied part ranges, offering complete high quality management regardless of the system’s complexity. Planview evaluates parts at three ranges:
Prompts – Assessing LLM prompts for effectiveness and accuracy
AI brokers – Evaluating full immediate chains to take care of optimum job dealing with and response relevance
AI system – Testing user-facing interactions to confirm seamless integration of all parts
The next determine illustrates the analysis framework for prompts and scoring.
To conduct these evaluations, Planview makes use of a set of fastidiously crafted check questions that cowl typical person queries and edge instances. These evaluations are carried out throughout the growth part and proceed in manufacturing to trace the standard of responses over time. Presently, human evaluators play an important function in scoring responses. To assist within the analysis, Planview has developed an inner analysis software to retailer the library of questions and observe the responses over time.
To evaluate every part and decide essentially the most appropriate Amazon Bedrock mannequin for a given job, Planview established the next prioritized analysis standards:
High quality of response – Assuring accuracy, relevance, and helpfulness of system responses
Time of response – Minimizing latency between person queries and system responses
Scale – Ensuring the system can scale to hundreds of concurrent customers
Value of response – Optimizing operational prices, together with AWS companies and generative AI fashions, to take care of financial viability
Based mostly on these standards and the present use case, Planview chosen Anthropic’s Claude 3 Sonnet on Amazon Bedrock for the router and responder parts.
Outcomes and influence
Over the previous 12 months, Planview Copilot’s efficiency has considerably improved by way of the implementation of a multi-agent structure, growth of a sturdy analysis framework, and adoption of the newest FMs obtainable by way of Amazon Bedrock. Planview noticed the next outcomes between the primary technology of Planview Copilot developed mid-2023 and the newest model:
Accuracy – Human-evaluated accuracy has improved from 50% reply acceptance to now exceeding 95%
Response time – Common response instances have been decreased from over 1 minute to twenty seconds
Load testing – The AI assistant has efficiently handed load assessments, the place 1,000 questions had been submitted simultaneous with no noticeable influence on response time or high quality
Value-efficiency – The fee per buyer interplay has been slashed to 1 tenth of the preliminary expense
Time-to-market – New agent growth and deployment time has been decreased from months to weeks
Conclusion
On this publish, we explored how Planview was in a position to develop a generative AI assistant to handle complicated work administration course of by adopting the next methods:
Modular growth – Planview constructed a multi-agent structure with a centralized orchestrator. The answer allows environment friendly job dealing with and system scalability, whereas permitting completely different product groups to quickly develop and deploy new AI expertise by way of specialised brokers.
Analysis framework – Planview applied a sturdy analysis course of at a number of ranges, which was essential for sustaining and enhancing efficiency.
Amazon Bedrock integration – Planview used Amazon Bedrock to innovate quicker with broad mannequin selection and entry to varied FMs, permitting for versatile mannequin choice primarily based on particular job necessities.
Planview is migrating to Amazon Bedrock Brokers, which allows the combination of clever autonomous brokers inside their utility ecosystem. Amazon Bedrock Brokers automate processes by orchestrating interactions between basis fashions, information sources, functions, and person conversations.
As subsequent steps, you possibly can discover Planview’s AI assistant function constructed on Amazon Bedrock and keep up to date with new Amazon Bedrock options and releases to advance your AI journey on AWS.
About Authors
Sunil Ramachandra is a Senior Options Architect enabling hyper-growth Unbiased Software program Distributors (ISVs) to innovate and speed up on AWS. He companions with clients to construct extremely scalable and resilient cloud architectures. When not collaborating with clients, Sunil enjoys spending time with household, operating, meditating, and watching motion pictures on Prime Video.
Benedict Augustine is a thought chief in Generative AI and Machine Studying, serving as a Senior Specialist at AWS. He advises buyer CxOs on AI technique, to construct long-term visions whereas delivering speedy ROI.As VP of Machine Studying, Benedict spent the final decade constructing seven AI-first SaaS merchandise, now utilized by Fortune 100 firms, driving vital enterprise influence. His work has earned him 5 patents.
Lee Rehwinkel is a Principal Knowledge Scientist at Planview with 20 years of expertise in incorporating AI & ML into Enterprise software program. He holds superior levels from each Carnegie Mellon College and Columbia College. Lee spearheads Planview’s R&D efforts on AI capabilities inside Planview Copilot. Exterior of labor, he enjoys rowing on Austin’s Girl Chicken Lake.