Initially, when ChatGPT simply appeared, we used easy prompts to get solutions to our questions. Then, we encountered points with hallucinations and commenced utilizing RAG (Retrieval Augmented Era) to supply extra context to LLMs. After that, we began experimenting with AI brokers, the place LLMs act as a reasoning engine and might determine what to do subsequent, which instruments to make use of, and when to return the ultimate reply.
The subsequent evolutionary step is to create groups of such brokers that may collaborate with one another. This method is logical because it mirrors human interactions. We work in groups the place every member has a particular function:
The product supervisor proposes the following challenge to work on.The designer creates its appear and feel.The software program engineer develops the answer.The analyst examines the information to make sure it performs as anticipated and identifies methods to enhance the product for patrons.
Equally, we are able to create a group of AI brokers, every specializing in one area. They will collaborate and attain a ultimate conclusion collectively. Simply as specialization enhances efficiency in actual life, it might additionally profit the efficiency of AI brokers.
One other benefit of this method is elevated flexibility. Every agent can function with its personal immediate, set of instruments and even LLM. As an illustration, we are able to use completely different fashions for various components of our system. You should use GPT-4 for the agent that wants extra reasoning and GPT-3.5 for the one which does solely easy extraction. We are able to even fine-tune the mannequin for small particular duties and use it in our crew of brokers.
The potential drawbacks of this method are time and value. A number of interactions and data sharing between brokers require extra calls to LLM and eat extra tokens. This might lead to longer wait instances and elevated bills.
There are a number of frameworks obtainable for multi-agent techniques at this time. Listed below are a number of the hottest ones:
AutoGen: Developed by Microsoft, AutoGen makes use of a conversational method and was one of many earliest frameworks for multi-agent techniques,LangGraph: Whereas not strictly a multi-agent framework, LangGraph permits for outlining complicated interactions between actors utilizing a graph construction. So, it may also be tailored to create multi-agent techniques.CrewAI: Positioned as a high-level framework, CrewAI facilitates the creation of “crews” consisting of role-playing brokers able to collaborating in numerous methods.
I’ve determined to start out experimenting with multi-agent frameworks from CrewAI because it’s fairly broadly fashionable and consumer pleasant. So, it appears like possibility to start with.
On this article, I’ll stroll you thru find out how to use CrewAI. As analysts, we’re the area specialists liable for documenting numerous knowledge sources and addressing associated questions. We’ll discover find out how to automate these duties utilizing multi-agent frameworks.
Let’s begin with organising the surroundings. First, we have to set up the CrewAI essential bundle and an extension to work with instruments.
pip set up crewaipip set up ‘crewai[tools]’
CrewAI was developed to work primarily with OpenAI API, however I might additionally wish to attempt it with a neighborhood mannequin. In line with the ChatBot Area Leaderboard, one of the best mannequin you may run in your laptop computer is Llama 3 (8b parameters). It is going to be probably the most possible possibility for our use case.
We are able to entry Llama fashions utilizing Ollama. Set up is fairly simple. It’s essential obtain Ollama from the web site after which undergo the set up course of. That’s it.
Now, you may take a look at the mannequin in CLI by working the next command.
ollama run llama3
For instance, you may ask one thing like this.
Let’s create a customized Ollama mannequin to make use of later in CrewAI.
We are going to begin with a ModelFile (documentation). I solely specified the bottom mannequin (llama3), temperature and cease sequence. Nonetheless, you may add extra options. For instance, you may decide the system message utilizing SYSTEM key phrase.
FROM llama3
# set parametersPARAMETER temperature 0.5PARAMETER cease Consequence
I’ve saved it right into a Llama3ModelFile file.
Let’s create a bash script to load the bottom mannequin for Ollama and create the customized mannequin we outlined in ModelFile.
#!/bin/zsh
# outline variablesmodel_name=”llama3″custom_model_name=”crewai-llama3″
# load the bottom modelollama pull $model_name
# create the mannequin fileollama create $custom_model_name -f ./Llama3ModelFile
Let’s execute this file.
chmod +x ./llama3_setup.sh./llama3_setup.sh
You’ll find each recordsdata on GitHub: Llama3ModelFile and llama3_setup.sh
We have to initialise the next environmental variables to make use of the native Llama mannequin with CrewAI.
os.environ[“OPENAI_API_BASE”]=’http://localhost:11434/v1′
os.environ[“OPENAI_MODEL_NAME”]=’crewai-llama3′ # custom_model_name from the bash script
os.environ[“OPENAI_API_KEY”] = “NA”
We’ve completed the setup and are able to proceed our journey.
As analysts, we regularly play the function of subject material specialists for knowledge and a few data-related instruments. In my earlier group, we used to have a channel with nearly 1K members, the place we have been answering numerous questions on our knowledge and the ClickHouse database we used as storage. It took us numerous time to handle this channel. It will be fascinating to see whether or not such duties could be automated with LLMs.
For this instance, I’ll use the ClickHouse database. When you’re , You may be taught extra about ClickHouse and find out how to set it up regionally in my earlier article. Nonetheless, we received’t utilise any ClickHouse-specific options, so be happy to stay to the database you understand.
I’ve created a fairly easy knowledge mannequin to work with. There are simply two tables in our DWH (Information Warehouse): ecommerce_db.customers and ecommerce_db.classes. As you may guess, the primary desk comprises details about the customers of our service.
The ecommerce_db.classes desk shops details about consumer classes.
Concerning knowledge supply administration, analysts sometimes deal with duties like writing and updating documentation and answering questions on this knowledge. So, we’ll use LLM to put in writing documentation for the desk within the database and train it to reply questions on knowledge or ClickHouse.
However earlier than shifting on to the implementation, let’s be taught extra concerning the CrewAI framework and its core ideas.
The cornerstone of a multi-agent framework is an agent idea. In CrewAI, brokers are powered by role-playing. Position-playing is a tactic if you ask an agent to undertake a persona and behave like a top-notch backend engineer or useful buyer assist agent. So, when making a CrewAI agent, you have to specify every agent’s function, purpose, and backstory in order that LLM is aware of sufficient to play this function.
The brokers’ capabilities are restricted with out instruments (features that brokers can execute and get outcomes). With CrewAI, you should use one of many predefined instruments (for instance, to go looking the Web, parse an internet site, or do RAG on a doc), create a customized software your self or use LangChain instruments. So, it’s fairly straightforward to create a strong agent.
Let’s transfer on from brokers to the work they’re doing. Brokers are engaged on duties (particular assignments). For every activity, we have to outline an outline, anticipated output (definition of executed), set of obtainable instruments and assigned agent. I actually like that these frameworks comply with the managerial finest practices like a transparent definition of executed for the duties.
The subsequent query is find out how to outline the execution order for duties: which one to work on first, which of them can run in parallel, and many others. CrewAI carried out processes to orchestrate the duties. It gives a few choices:
Sequential —probably the most simple method when duties are known as one after one other.Hierarchical — when there’s a supervisor (specified as LLM mannequin) that creates and delegates duties to the brokers.
Additionally, CrewAI is engaged on a consensual course of. In such a course of, brokers will be capable to make selections collaboratively with a democratic method.
There are different levers you should use to tweak the method of duties’ execution:
You may mark duties as “asynchronous”, then they are going to be executed in parallel, so it is possible for you to to get a solution quicker.You should use the “human enter” flag on a activity, after which the agent will ask for human approval earlier than finalising the output of this activity. It could will let you add an oversight to the method.
We’ve outlined all the first constructing blocks and might talk about the holly grail of CrewAI — crew idea. The crew represents the group of brokers and the set of duties they are going to be engaged on. The method for collaboration (processes we mentioned above) may also be outlined on the crew stage.
Additionally, we are able to arrange the reminiscence for a crew. Reminiscence is essential for environment friendly collaboration between the brokers. CrewAI helps three ranges of reminiscence:
Quick-term reminiscence shops info associated to the present execution. It helps brokers to work collectively on the present activity.Lengthy-term reminiscence is knowledge concerning the earlier executions saved within the native database. One of these reminiscence permits brokers to be taught from earlier iterations and enhance over time.Entity reminiscence captures and constructions details about entities (like personas, cities, and many others.)
Proper now, you may solely change on all sorts of reminiscence for a crew with none additional customisation. Nonetheless, it doesn’t work with the Llama fashions.
We’ve discovered sufficient concerning the CrewAI framework, so it’s time to start out utilizing this information in apply.
Let’s begin with a easy activity: placing collectively the documentation for our DWH. As we mentioned earlier than, there are two tables in our DWH, and I want to create an in depth description for them utilizing LLMs.
First method
At first, we’d like to consider the group construction. Consider this as a typical managerial activity. Who would you rent for such a job?
I might break this activity into two components: retrieving knowledge from a database and writing documentation. So, we’d like a database specialist and a technical author. The database specialist wants entry to a database, whereas the author received’t want any particular instruments.
Now, we now have a high-level plan. Let’s create the brokers.
For every agent, I’ve specified the function, purpose and backstory. I’ve tried my finest to supply brokers with all of the wanted context.
database_specialist_agent = Agent(function = “Database specialist”,purpose = “Present knowledge to reply enterprise questions utilizing SQL”,backstory = ”’You might be an knowledgeable in SQL, so you may assist the group to assemble wanted knowledge to energy their selections. You might be very correct and take into consideration all of the nuances in knowledge.”’,allow_delegation = False,verbose = True)
tech_writer_agent = Agent(function = “Technical author”,purpose = ”’Write participating and factually correct technical documentation for knowledge sources or instruments”’,backstory = ”’ You might be an knowledgeable in each know-how and communications, so you may simply clarify even refined ideas.You base your work on the factual info supplied by your colleagues.Your texts are concise and could be simply understood by a large viewers. You utilize skilled however fairly a casual type in your communication.”’,allow_delegation = False,verbose = True)
We are going to use a easy sequential course of, so there’s no want for brokers to delegate duties to one another. That’s why I specified allow_delegation = False.
The subsequent step is setting the duties for brokers. However earlier than shifting to them, we have to create a customized software to hook up with the database.
First, I put collectively a operate to execute ClickHouse queries utilizing HTTP API.
CH_HOST = ‘http://localhost:8123’ # default tackle
def get_clickhouse_data(question, host = CH_HOST, connection_timeout = 1500):r = requests.publish(host, params = {‘question’: question}, timeout = connection_timeout)if r.status_code == 200:return r.textelse: return ‘Database returned the next error:n’ + r.textual content
When working with LLM brokers, it’s necessary to make instruments fault-tolerant. For instance, if the database returns an error (status_code != 200), my code received’t throw an exception. As a substitute, it is going to return the error description to the LLM so it might try to resolve the problem.
To create a CrewAI customized software, we have to derive our class from crewai_tools.BaseTool, implement the _run technique after which create an occasion of this class.
from crewai_tools import BaseTool
class DatabaseQuery(BaseTool):identify: str = “Database Question”description: str = “Returns the results of SQL question execution”
def _run(self, sql_query: str) -> str:# Implementation goes herereturn get_clickhouse_data(sql_query)
database_query_tool = DatabaseQuery()
Now, we are able to set the duties for the brokers. Once more, offering clear directions and all of the context to LLM is essential.
table_description_task = Job(description = ”’Present the great overview for the information in desk {desk}, in order that it is easy to grasp the construction of the information. This activity is essential to place collectively the documentation for our database”’,expected_output = ”’The great overview of {desk} within the md format. Embrace 2 sections: columns (checklist of columns with their varieties) and examples (the primary 30 rows from desk).”’,instruments = [database_query_tool],agent = database_specialist_agent)
table_documentation_task = Job(description = ”’Utilizing supplied details about the desk, put collectively the detailed documentation for this desk so that individuals can use it in apply”’,expected_output = ”’Nicely-written detailed documentation describing the information scheme for the desk {desk} in markdown format, that provides the desk overview in 1-2 sentences then then describes every columm. Construction the columns description as a markdown desk with column identify, sort and outline.”’,instruments = [],output_file=”table_documentation.md”,agent = tech_writer_agent)
You may need observed that I’ve used {desk} placeholder within the duties’ descriptions. We are going to use desk as an enter variable when executing the crew, and this worth might be inserted into all placeholders.
Additionally, I’ve specified the output file for the desk documentation activity to avoid wasting the ultimate end result regionally.
We’ve got all we’d like. Now, it’s time to create a crew and execute the method, specifying the desk we’re fascinated by. Let’s attempt it with the customers desk.
crew = Crew(brokers = [database_specialist_agent, tech_writer_agent],duties = [table_description_task, table_documentation_task],verbose = 2)
end result = crew.kickoff({‘desk’: ‘ecommerce_db.customers’})
It’s an thrilling second, and I’m actually wanting ahead to seeing the end result. Don’t fear if execution takes a while. Brokers make a number of LLM calls, so it’s completely regular for it to take a couple of minutes. It took 2.5 minutes on my laptop computer.
We requested LLM to return the documentation in markdown format. We are able to use the next code to see the formatted lead to Jupyter Pocket book.
from IPython.show import MarkdownMarkdown(end result)
At first look, it appears nice. We’ve received the legitimate markdown file describing the customers’ desk.
However wait, it’s incorrect. Let’s see what knowledge we now have in our desk.
The columns listed within the documentation are utterly completely different from what we now have within the database. It’s a case of LLM hallucinations.
We’ve set verbose = 2 to get the detailed logs from CrewAI. Let’s learn via the execution logs to determine the basis reason for the issue.
First, the database specialist couldn’t question the database resulting from problems with quotes.
The specialist didn’t handle to resolve this drawback. Lastly, this chain has been terminated by CrewAI with the next output: Agent stopped resulting from iteration restrict or time restrict.
This implies the technical author didn’t obtain any factual details about the information. Nonetheless, the agent continued and produced utterly faux outcomes. That’s how we ended up with incorrect documentation.
Fixing the problems
Though our first iteration wasn’t profitable, we’ve discovered lots. We’ve got (a minimum of) two areas for enchancment:
Our database software is simply too troublesome for the mannequin, and the agent struggles to make use of it. We are able to make the software extra tolerant by eradicating quotes from the start and finish of the queries. This resolution just isn’t superb since legitimate SQL can finish with a quote, however let’s attempt it.Our technical author isn’t basing its output on the enter from the database specialist. We have to tweak the immediate to focus on the significance of offering solely factual info.
So, let’s attempt to repair these issues. First, we’ll repair the software — we are able to leverage strip to eradicate quotes.
CH_HOST = ‘http://localhost:8123’ # default tackle
def get_clickhouse_data(question, host = CH_HOST, connection_timeout = 1500):r = requests.publish(host, params = {‘question’: question.strip(‘”‘).strip(“‘”)}, timeout = connection_timeout)if r.status_code == 200:return r.textelse: return ‘Database returned the next error:n’ + r.textual content
Then, it’s time to replace the immediate. I’ve included statements emphasizing the significance of sticking to the information in each the agent and activity definitions.
tech_writer_agent = Agent(function = “Technical author”,purpose = ”’Write participating and factually correct technical documentation for knowledge sources or instruments”’,backstory = ”’ You might be an knowledgeable in each know-how and communications, so you may simply clarify even refined ideas.Your texts are concise and could be simply understood by extensive viewers. You utilize skilled however fairly casual type in your communication.You base your work on the factual info supplied by your colleagues. You persist with the information within the documentation and use ONLY info supplied by the colleagues not including something.”’,allow_delegation = False,verbose = True)
table_documentation_task = Job(description = ”’Utilizing supplied details about the desk, put collectively the detailed documentation for this desk so that individuals can use it in apply”’,expected_output = ”’Nicely-written detailed documentation describing the information scheme for the desk {desk} in markdown format, that provides the desk overview in 1-2 sentences then then describes every columm. Construction the columns description as a markdown desk with column identify, sort and outline.The documentation relies ONLY on the data supplied by the database specialist with none additions.”’,instruments = [],output_file = “table_documentation.md”,agent = tech_writer_agent)
Let’s execute our crew as soon as once more and see the outcomes.
We’ve achieved a bit higher end result. Our database specialist was in a position to execute queries and think about the information, which is a major win for us. Moreover, we are able to see all of the related fields within the end result desk, although there are many different fields as nicely. So, it’s nonetheless not solely right.
I as soon as once more appeared via the CrewAI execution log to determine what went flawed. The problem lies in getting the checklist of columns. There’s no filter by database, so it returns some unrelated columns that seem within the end result.
SELECT column_name FROM information_schema.columns WHERE table_name = ‘customers’
Additionally, after a number of makes an attempt, I observed that the database specialist, on occasion, executes choose * from <desk> question. It would trigger some points in manufacturing as it would generate numerous knowledge and ship it to LLM.
Extra specialised instruments
We are able to present our agent with extra specialised instruments to enhance our resolution. At the moment, the agent has a software to execute any SQL question, which is versatile and highly effective however liable to errors. We are able to create extra centered instruments, similar to getting desk construction and top-N rows from the desk. Hopefully, it is going to cut back the variety of errors.
class TableStructure(BaseTool):identify: str = “Desk construction”description: str = “Returns the checklist of columns and their varieties”
def _run(self, desk: str) -> str:desk = desk.strip(‘”‘).strip(“‘”)return get_clickhouse_data(‘describe {desk} format TabSeparatedWithNames’.format(desk = desk))
class TableExamples(BaseTool):identify: str = “Desk examples”description: str = “Returns the primary N rows from the desk”
def _run(self, desk: str, n: int = 30) -> str:desk = desk.strip(‘”‘).strip(“‘”)return get_clickhouse_data(‘choose * from {desk} restrict {n} format TabSeparatedWithNames’.format(desk = desk, n = n))
table_structure_tool = TableStructure()table_examples_tool = TableExamples()
Now, we have to specify these instruments within the activity and re-run our script. After the primary try, I received the next output from the Technical Author.
Job output: This ultimate reply gives an in depth and factual description of the ecommerce_db.customers desk construction, together with column names, varieties, and descriptions. The documentation adheres to the supplied info from the database specialist with none additions or modifications.
Extra centered instruments helped the database specialist retrieve the right desk info. Nonetheless, though the author had all the mandatory info, we didn’t get the anticipated end result.
As we all know, LLMs are probabilistic, so I gave it one other attempt. And hooray, this time, the end result was fairly good.
It’s not excellent because it nonetheless contains some irrelevant feedback and lacks the general description of the desk. Nonetheless, offering extra specialised instruments has undoubtedly paid off. It additionally helped to forestall points when the agent tried to load all the information from the desk.
High quality assurance specialist
We’ve achieved fairly good outcomes, however let’s see if we are able to enhance them additional. A standard apply in multi-agent setups is high quality assurance, which provides the ultimate overview stage earlier than finalising the outcomes.
Let’s create a brand new agent — a High quality Assurance Specialist, who might be in command of overview.
qa_specialist_agent = Agent(function = “High quality Assurance specialist”,purpose = “””Guarantee the best high quality of the documentation we offer (that it is right and simple to grasp)”””,backstory = ”’You’re employed as a High quality Assurance specialist, checking the work from the technical author and making certain that it is inline with our highest requirements.It’s essential examine that the technical author gives the complete full solutions and make no assumptions. Additionally, you have to guarantee that the documentation addresses all of the questions and is simple to grasp.”’,allow_delegation = False,verbose = True)
Now, it’s time to explain the overview activity. I’ve used the context parameter to specify that this activity requires outputs from each table_description_task and table_documentation_task.
qa_review_task = Job(description = ”’Overview the draft documentation supplied by the technical author.Make sure that the documentation absolutely solutions all of the questions: the aim of the desk and its construction within the type of desk. Make it possible for the documentation is in step with the data supplied by the database specialist. Double examine that there are not any irrelevant feedback within the ultimate model of documentation.”’,expected_output = ”’The ultimate model of the documentation in markdown format that may be printed. The documentation ought to absolutely tackle all of the questions, be constant and comply with our skilled however casual tone of voice.”’,instruments = [],context = [table_description_task, table_documentation_task],output_file=”checked_table_documentation.md”,agent = qa_specialist_agent)
Let’s replace our crew and run it.
full_crew = Crew(brokers=[database_specialist_agent, tech_writer_agent, qa_specialist_agent],duties=[table_description_task, table_documentation_task, qa_review_task],verbose = 2,reminiscence = False # do not work with Llama)
full_result = full_crew.kickoff({‘desk’: ‘ecommerce_db.customers’})
We now have extra structured and detailed documentation due to the addition of the QA stage.
Delegation
With the addition of the QA specialist, it will be fascinating to check the delegation mechanism. The QA specialist agent may need questions or requests that it might delegate to different brokers.
I attempted utilizing the delegation with Llama 3, nevertheless it didn’t go nicely. Llama 3 struggled to name the co-worker software accurately. It couldn’t specify the right co-worker’s identify.
We achieved fairly good outcomes with a neighborhood mannequin that may run on any laptop computer, however now it’s time to change gears and use a far more highly effective mannequin — GPT-4o.
To do it, we simply have to replace the next surroundings variables.
os.environ[“OPENAI_MODEL_NAME”] = ‘gpt-4o’ os.environ[“OPENAI_API_KEY”] = config[‘OPENAI_API_KEY’] # your OpenAI key
To modify on the delegation, we should always specify allow_delegation = True for the QA specialist agent.
Additionally, we are able to use useful reminiscence performance for OpenAI fashions, as talked about above. The reminiscence will enable our brokers to share their data with one another throughout execution and leverage long-term reminiscence to get info from earlier executions.
full_crew = Crew(brokers = [database_specialist_agent, tech_writer_agent, qa_specialist_agent],duties = [table_description_task, table_documentation_task, qa_review_task],verbose = 2,reminiscence = True )
full_result = full_crew.kickoff({‘desk’: ‘ecommerce_db.customers’})
Let’s see the CrewAI execution logs to grasp how delegation works. Listed below are all of the logs for the QA specialist. We are able to see that it reached out to the database specialist to double-check the data.
Thought: I have to overview the supplied draft documentation to make sure it absolutely solutions all of the questions and is full, right, and simple to grasp. Particularly, I want to make sure that it explains the aim of the desk, its construction, and contains related examples with none irrelevant feedback.
First, I’ll examine the 2 drafts to determine any discrepancies and lacking info. Then, I’ll confirm the accuracy and completeness of the data supplied.
**Comparability:**1. **Introduction:**- First draft: Consists of an introduction and examples.- Second draft: Provides a extra detailed introduction concerning the function and significance of the desk.
2. **Columns:**- Each drafts have the identical column construction and descriptions.
3. **Extra Data:**- Each drafts embody comparable extra info, however the second draft is extra concise.
4. **Examples:**- The primary draft contains examples, that are lacking within the second draft.
5. **Remaining Checks:**- Guarantee there are not any irrelevant feedback.- Confirm consistency with info supplied by the database specialist.
Motion: Ask query to co-workerAction Enter: 1
<– omitted logs from the database specialist –>
Based mostly on the supplied context and examples for the `ecommerce_db.customers` desk, the column descriptions and examples seem correct. Listed below are the small print:- `user_id`: This column comprises a singular identifier for every consumer. The supplied examples (e.g., 1000001, 1000002) align with this description as they’re distinctive numeric identifiers.
– `nation`: This column specifies the nation of the consumer. The supplied examples (e.g., United Kingdom, France, Germany, Netherlands) are all legitimate nation names.
– `is_active`: This column signifies whether or not the consumer is lively (1) or not (0). The supplied examples (e.g., 0, 1) accurately characterize this binary standing.
– `age`: This column reveals the age of the consumer. The supplied examples (e.g., 70, 87, 88, 25, 48, 78, 65, 31, 66, 73) are all legitimate ages.
Subsequently, the column descriptions and examples supplied for the `ecommerce_db.customers` desk are correct and meet the anticipated standards.
Thought: Based mostly on the remark from the database specialist, the column descriptions and examples supplied for the `ecommerce_db.customers` desk are correct. Now, I’ll consolidate one of the best components from each drafts,making certain the documentation is full, clear, and freed from irrelevant feedback.
Motion: Finalize the documentation in markdown format, incorporating the detailed introduction, examples, and making certain the tone is skilled however casual.
After I tried the delegation for the primary time, I didn’t allow reminiscence, which led to incorrect outcomes. The information specialist and the technical author initially returned the right info. Nonetheless, when the QA specialist returned with the follow-up questions, they began to hallucinate. So, it appears like delegation works higher when reminiscence is enabled.
Right here’s the ultimate output from GPT-4o. The end result appears fairly good now. We undoubtedly can use LLMs to automate documentation.
So, the primary activity has been solved!
I used the identical script to generate documentation for the ecommerce_db.classes desk as nicely. It is going to be useful for our subsequent activity. So, let’s not waste any time and transfer on.
Our subsequent activity is answering questions based mostly on the documentation because it’s widespread for a lot of knowledge analysts (and different specialists).
We are going to begin easy and can create simply two brokers:
The documentation assist specialist might be answering questions based mostly on the docs,The assist QA agent will overview the reply earlier than sharing it with the client.
We might want to empower the documentation specialist with a few instruments that can enable them to see all of the recordsdata saved within the listing and browse the recordsdata. It’s fairly simple since CrewAI has carried out such instruments.
from crewai_tools import DirectoryReadTool, FileReadTool
documentation_directory_tool = DirectoryReadTool(listing = ‘~/crewai_project/ecommerce_documentation’)
base_file_read_tool = FileReadTool()
Nonetheless, since Llama 3 retains fighting quotes when calling instruments, I needed to create a customized software on prime of the FileReaderTool to beat this problem.
from crewai_tools import BaseTool
class FileReadToolUPD(BaseTool):identify: str = “Learn a file’s content material”description: str = “A software that can be utilized to learn a file’s content material.”
def _run(self, file_path: str) -> str:# Implementation goes herereturn base_file_read_tool._run(file_path = file_path.strip(‘”‘).strip(“‘”))
file_read_tool = FileReadToolUPD()
Subsequent, as we did earlier than, we have to create brokers, duties and crew.
data_support_agent = Agent(function = “Senior Information Assist Agent”,purpose = “Be probably the most useful assist for you colleagues”,backstory = ”’You’re employed as a assist for data-related questions within the firm. Though you are a giant knowledgeable in our knowledge warehouse, you double examine all of the information in documentation. Our documentation is completely up-to-date, so you may absolutely depend on it when answering questions (you needn’t examine the precise knowledge in database).Your work is essential for the group success. Nonetheless, keep in mind that examples of desk rows do not present all of the attainable values. It’s essential make sure that you present the absolute best assist: answering all of the questions, making no assumptions and sharing solely the factual knowledge.Be inventive attempt your finest to resolve the client drawback. ”’,allow_delegation = False,verbose = True)
qa_support_agent = Agent(function = “Assist High quality Assurance Agent”,purpose = “””Guarantee the best high quality of the solutions we offer to the shoppers”””,backstory = ”’You’re employed as a High quality Assurance specialist, checking the work from assist brokers and making certain that it is inline with our highest requirements.It’s essential examine that the agent gives the complete full solutions and make no assumptions. Additionally, you have to guarantee that the documentation addresses all of the questions and is simple to grasp.”’,allow_delegation = False,verbose = True)
draft_data_answer = Job(description = ”’Essential buyer {buyer} reached out to you with the next query:“`{query}“`
Your activity is to supply one of the best reply to all of the factors within the query utilizing all obtainable info and never making any assumprions. If you do not have sufficient info to reply the query, simply say that you do not know.”’,expected_output = ”’The detailed informative reply to the client’s query that addresses all the purpose talked about. Make it possible for reply is full and stict to information (with none extra info not based mostly on the factual knowledge)”’,instruments = [documentation_directory_tool, file_read_tool], agent = data_support_agent)
answer_review = Job(description = ”’Overview the draft reply supplied by the assist agent.Make sure that the it absolutely solutions all of the questions talked about within the preliminary inquiry. Make it possible for the reply is constant and would not embody any assumptions.”’,expected_output = ”’The ultimate model of the reply in markdown format that may be shared with the client. The reply ought to absolutely tackle all of the questions, be constant and comply with our skilled however casual tone of voice. We’re very chill and pleasant firm, so remember to incorporate all of the well mannered phrases.”’,instruments = [], agent = qa_support_agent)
qna_crew = Crew(brokers = [data_support_agent, qa_support_agent],duties = [draft_data_answer, answer_review],verbose = 2,reminiscence = False # do not work with Llama)
Let’s see the way it works in apply.
end result = qna_crew.kickoff({‘buyer’: “Max”, ‘query’: “””Hey group, I hope you are doing nicely. I want to search out the numbers earlier than our CEO presentation tomorrow, so I’ll actually respect your assist.I have to calculate the variety of classes from our Home windows customers in 2023. I’ve tried to search out the desk with such knowledge in our knowledge warehouse, however wasn’t in a position to. Do you’ve got any concepts whether or not we retailer the wanted knowledge someplace, in order that I can question it? “””})
We’ve received a well mannered, sensible and useful reply in return. That’s actually nice.
**Howdy Max,**
Thanks for reaching out along with your query! I am completely happy that can assist you discover the variety of classes from Home windows customers in 2023. After reviewing our documentation, I discovered that we do retailer knowledge associated to classes and customers in our ecommerce database, particularly within the `ecommerce_db.classes` desk.
To reply your query, I can give you a step-by-step information on find out how to question this desk utilizing SQL. First, you should use the `session_id` column together with the `os` column filtering for “Home windows” and the `action_date` column filtering for dates in 2023. Then, you may group the outcomes by `os` utilizing the `GROUP BY` clause to rely the variety of classes that meet these situations.
This is a pattern SQL question that ought to provide the desired output:
“`sqlSELECT COUNT(*) FROM ecommerce_db.classes WHERE os = ‘Home windows’ AND action_date BETWEEN ‘2023-01-01’ AND ‘2023-12-31’GROUP BY os;“`
This question will return the full variety of classes from Home windows customers in 2023. I hope this helps! If in case you have any additional questions or want extra help, please do not hesitate to ask.
Let’s complicate the duty a bit. Suppose we are able to get not solely questions on our knowledge but additionally about our software (ClickHouse). So, we could have one other agent within the crew — ClickHouse Guru. To present our CH agent some data, I’ll share a documentation web site with it.
from crewai_tools import ScrapeWebsiteTool, WebsiteSearchToolch_documenation_tool = ScrapeWebsiteTool(‘https://clickhouse.com/docs/en/guides/creating-tables’)
If you have to work with a prolonged doc, you may attempt utilizing RAG (Retrieval Augmented technology) — WebsiteSearchTool. It is going to calculate embeddings and retailer them regionally in ChromaDB. In our case, we’ll persist with a easy web site scraper software.
Now that we now have two subject material specialists, we have to determine who might be engaged on the questions. So, it’s time to make use of a hierarchical course of and add a supervisor to orchestrate all of the duties.
CrewAI gives the supervisor implementation, so we solely have to specify the LLM mannequin. I’ve picked the GPT-4o.
from langchain_openai import ChatOpenAIfrom crewai import Course of
complext_qna_crew = Crew(brokers = [ch_support_agent, data_support_agent, qa_support_agent],duties = [draft_ch_answer, draft_data_answer, answer_review],verbose = 2,manager_llm = ChatOpenAI(mannequin=’gpt-4o’, temperature=0), course of = Course of.hierarchical, reminiscence = False )
At this level, I needed to change from Llama 3 to OpenAI fashions once more to run a hierarchical course of because it hasn’t labored for me with Llama (just like this problem).
Now, we are able to attempt our new crew with several types of questions (both associated to our knowledge or ClickHouse database).
ch_result = complext_qna_crew.kickoff({‘buyer’: “Maria”, ‘query’: “””Good morning, group. I am utilizing ClickHouse to calculate the variety of clients. Might you please remind whether or not there’s an possibility so as to add totals in ClickHouse?”””})
doc_result = complext_qna_crew.kickoff({‘buyer’: “Max”, ‘query’: “””Hey group, I hope you are doing nicely. I want to search out the numbers earlier than our CEO presentation tomorrow, so I’ll actually respect your assist.I have to calculate the variety of classes from our Home windows customers in 2023. I’ve tried to search out the desk with such knowledge in our knowledge warehouse, however wasn’t in a position to. Do you’ve got any concepts whether or not we retailer the wanted knowledge someplace, in order that I can question it. “””})
If we take a look at the ultimate solutions and logs (I’ve omitted them right here since they’re fairly prolonged, however you will discover them and full logs on GitHub), we’ll see that the supervisor was in a position to orchestrate accurately and delegate duties to co-workers with related data to deal with the client’s query. For the primary (ClickHouse-related) query, we received an in depth reply with examples and attainable implications of utilizing WITH TOTALS performance. For the data-related query, fashions returned roughly the identical info as we’ve seen above.
So, we’ve constructed a crew that may reply numerous sorts of questions based mostly on the documentation, whether or not from a neighborhood file or an internet site. I believe it’s a wonderful end result.
You’ll find all of the code on GitHub.
On this article, we’ve explored utilizing the CrewAI multi-agent framework to create an answer for writing documentation based mostly on tables and answering associated questions.
Given the in depth performance we’ve utilised, it’s time to summarise the strengths and weaknesses of this framework.
Total, I discover CrewAI to be an extremely helpful framework for multi-agent techniques:
It’s simple, and you may construct your first prototype shortly.Its flexibility permits to resolve fairly refined enterprise issues.It encourages good practices like role-playing.It gives many useful instruments out of the field, similar to RAG and an internet site parser.The assist of several types of reminiscence enhances the brokers’ collaboration.Constructed-in guardrails assist forestall brokers from getting caught in repetitive loops.
Nonetheless, there are areas that could possibly be improved:
Whereas the framework is straightforward and simple to make use of, it’s not very customisable. As an illustration, you presently can’t create your personal LLM supervisor to orchestrate the processes.Generally, it’s fairly difficult to get the complete detailed info from the documentation. For instance, it’s clear that CrewAI carried out some guardrails to forestall repetitive operate calls, however the documentation doesn’t absolutely clarify the way it works.One other enchancment space is transparency. I like to grasp how frameworks work below the hood. For instance, in Langchain, you should use langchain.debug = True to see all of the LLM calls. Nonetheless, I haven’t found out find out how to get the identical stage of element with CrewAI.The total assist for the native fashions could be an ideal addition, as the present implementation both lacks some options or is troublesome to get working correctly.
The area and instruments for LLMs are evolving quickly, so I’m hopeful that we’ll see lots of progress within the close to future.
Thank you a large number for studying this text. I hope this text was insightful for you. If in case you have any follow-up questions or feedback, please go away them within the feedback part.
This text is impressed by the “Multi AI Agent Methods with CrewAI” quick course from DeepLearning.AI.