Building Autonomous AI Agents with CrewAI

Building Autonomous AI Agents with CrewAI

Bynamishsaxena

Last year, people focused on perfecting the art and science of prompts to get the most out of tools like ChatGPT. The results were truly awe-inspiring, and as always, the curious minds of humans wanted more.

So this year — and I believe for the foreseeable future — people are busy exploring the development of autonomous agents.

The idea itself seems to draw parallels from Dark, but the initial experimentations have been more than promising. In this journey, some are busy building frameworks to create agents (e.g., LangGraph, AutoGen, CrewAI, SuperAGI, etc.), others are creating marketplaces for agents (e.g., agent.ai), and some are building co-pilots (e.g., GitHub Copilot or Office 365 Copilot). And many, like me, are trying all of these out 🙂

So in this post, I want to share my experiences and learnings on building agents with CrewAI. You may ask, why CrewAI? Well, first, there aren’t many choices, and second, I wanted something that I could quickly prototype with. I have also experimented with LangGraph and AutoGen and will share those experiences sometime in the future.

I don’t want to bore you with too much theory, but there are a few basic building blocks I’d like to quickly share:

Agents

  • In Simple Terms: Think of an agent as a member of your team whom you work with.
  • Technical: A standalone program that can be executed to accomplish something.

Tasks

  • In Simple Terms: It’s what you want your team member or agent to do, setting expectations for the outcome.
  • Technical: It’s more than just describing what to do; it also specifies who will do it (the agent), what help is available (the tools), and at the very least, what output you are expecting.

Tools

  • In Simple Terms: They’re like the tools you use to accomplish tasks with ease, such as a screwdriver or wrench.
  • Technical: These are functions you want your agents to use to either bridge LLM knowledge gaps, provide information about your own data, or accomplish tasks that cannot or shouldn’t be done by LLMs — for example, file system operations or cloning a git repository.

Crew(s)

  • In Simple Terms: This is the team consisting of all your team members. Just like in the real world, you can have as many crews as you like.
  • Technical: In CrewAI, a crew represents a collaborative group of agents working together to achieve a set of tasks. Each crew defines the strategy for task execution, agent collaboration, and the overall workflow.

Let’s begin by exploring the steps involved

Step 1: Install CrewAI (and CrewAI tools)

# Install only the main crewAI package
pip install crewai

# Install the main crewAI package and the tools package
pip install 'crewai[tools]'

Step 2: Create CrewAI App

Now this has been a significant improvement from recent releases. There is a CLI tool that can give you pretty much all the boilerplate code

# This will create a directory or folder in your current directory with <project_name>.
crewai create crew <project_name>

This will create a directory or folder in your current directory with <project_name> and the following structure.

project_name/
├── .gitignore
├── pyproject.toml
├── README.md
└── src/
    └── my_project/
        ├── __init__.py
        ├── main.py
        ├── crew.py
        ├── tools/
        │   ├── custom_tool.py
        │   └── __init__.py
        └── config/
            ├── agents.yaml
            └── tasks.yaml

Step 3: Create a Virtual environment

I recommend using venv. While CrewAI offers excellent support for Poetry, simplifying dependency management, it’s essential to set up a virtual environment as you would with any Python project.

Step 4: Understand the code

While it’s common to jump straight into running commands, I encourage you to open the code in your favorite IDE (e.g., VSCode or PyCharm) and follow along with this post.

Let’s examine the three most important files. Open them alongside this blog to better understand the concepts discussed.

src/config/agents.yaml

researcher:
  role: >
    {topic} Senior Data Researcher
  goal: >
    Uncover cutting-edge developments in {topic}
  backstory: >
    You're a seasoned researcher with a knack for uncovering the latest
    developments in {topic}. Known for your ability to find the most relevant
    information and present it in a clear and concise manner.

reporting_analyst:
  role: >
    {topic} Reporting Analyst
  goal: >
    Create detailed reports based on {topic} data analysis and research findings
  backstory: >
    You're a meticulous analyst with a keen eye for detail. You're known for
    your ability to turn complex data into clear and concise reports, making
    it easy for others to understand and act on the information you provide.

Here we are defining the team members or agents in a declarative format.

  1. We give it a name (e.g., researcher) and then adding more definition
  2. Who they are (role)
  3. What’s their contribution in the team going to be (goal) and
  4. Their background or experience (backstory).

If you have flexed your brain muscles in prompting, all of this should feel relatable. This is like the pre-cursor to the actual prompt where we tell ChatGPT or other LLMs to give a personality. For API programmers, this is like giving a System Prompt.

There are few noticeable things here:

  • We are declaring agents in a yaml configuration file which is more human readable on one hand but also kind following the best practices of externalising prompts from code.
  • You might have noticed the use of a variable {topic}. This is a very powerful concept. This allows you to define your agents even more generic and reuse them across different use cases. We will see later, how you pass the value of this variable.

There are many other attributes or settings you can define, read about them here.

src/config/tasks.yaml

research_task:
  description: >
    Conduct a thorough research about {topic}
    Make sure you find any interesting and relevant information given
    the current year is 2024.
  expected_output: >
    A list with 10 bullet points of the most relevant information about {topic}
  agent: researcher

reporting_task:
  description: >
    Review the context you got and expand each topic into a full section for a report.
    Make sure the report is detailed and contains any and all relevant information.
  expected_output: >
    A fully fledge reports with the mains topics, each with a full section of information.
    Formatted as markdown without '```'
  agent: reporting_analyst

Here we are defining the tasks different team members or agents have to carry out and again in a declarative format.

  1. We give it a name (e.g., research_task)
  2. Describing the task in bit more detail (description)
  3. What we expect as an output of this task or activity (expected_output)
  4. Last but definitely not the least, which team-member should work on this task (agent).

Relating back to the usual prompting techniques, this is what we most tell LLMs or the main part of the prompt. For API programmers, this is the user message part.

There are few noticeable things here:

  • The use of yaml configuration format
  • You can pass variables here as well
  • Associating the task to an agent.
  • The output definition and format. CrewAI gives you the flexibility to get the output of a task as a String, Pydantic, or JSON Dict.

There are many other attributes or settings you can define, read about them here.

src/crew.py

We will see it in parts to build a better understanding. This is essentially about assembling the desired team (of agents) and weaving the tasks to accomplish your overarching goal.

Define the crewbase

@CrewBase
class TestCrew():

I had created the project with name Test and hence the name here is TestCrew. It could be different in your case.

The point of note here is the annotation @CrewBase. In this python file, like I said earlier, we are essentially defining our crew or team.

Load the configuration file

 agents_config = 'config/agents.yaml'
 tasks_config = 'config/tasks.yaml'

This doesn’t need any explanation. This is just loading the configurations of agents and tasks we saw earlier.

Defining agents and tasks

 @agent
 def researcher(self) -> Agent:
  return Agent(
   config=self.agents_config['researcher'],
   verbose=True
  )

@task
 def research_task(self) -> Task:
  return Task(
   config=self.tasks_config['research_task'],
  )

There are few noticeable things here:

  • Use of the @agent and @task decorators.
  • You can set other attributes here as well, such as verbosity or tools.
  • The declarative nature of this approach enhances readability.

Putting the Crew/ Team together

 @crew
 def crew(self) -> Crew:
  """Creates the Test crew"""
  return Crew(
   agents=self.agents, # Automatically created by the @agent decorator
   tasks=self.tasks, # Automatically created by the @task decorator
   process=Process.sequential,
   verbose=True,
   # process=Process.hierarchical, # In case you wanna use that instead https://docs.crewai.com/how-to/Hierarchical/
  )

There are few noticeable things here:

  • Use of annotations @crew
  • Adding agents and tasks. As we have added annotations, the framework takes of creating the actual objects during runtime
  • Defining additional attributes like verbosity
  • There is a new term here called Process. This is essentially telling how the execution of tasks should be. Should this be Sequential (in the order you have defined in your crew.py) or Hierarchical (like adding a team leader or manager who will decide on the execution path)
  • Like Agents and Tasks, we are just defining the Crew or Team of agents here with the tasks they must execute.

You might have a few questions at this point:

  • How are we going to execute or run this crew?
  • We haven’t defined an LLM here, so how does that part work?
  • How do we pass variables we defined in the YAML configuration files?
  • Does this actually work, or is it too good to be true?

Let’s uncover question 1 and 3 first

# main.py
def run():
    """
    Run the crew.
    """
    inputs = {
        'topic': 'AI LLMs'
    }
    TestCrew().crew().kickoff(inputs=inputs)

In main.py there is a function called run(). This will run your crew. Now if you look closely, we are defining a JSON called inputs. This is where we are giving value to our earlier defined variables (e.g., topic).

To run this, first install the dependencies

$ cd project_name
$ crewai install # your can also type poetry install

and to run this, execute the following

$ crewai run

Given the verbosity set in this example, there is going to be lot of logs on the console. Please take time to read these and it’s fairly straight forword.

There are few more interesting things to notice in main.py,

def train():
    """
    Train the crew for a given number of iterations.
    """
    inputs = {
        "topic": "AI LLMs"
    }
    try:
        TestCrew().crew().train(n_iterations=int(sys.argv[1]), filename=sys.argv[2], inputs=inputs)

    except Exception as e:
        raise Exception(f"An error occurred while training the crew: {e}")

def replay():
    """
    Replay the crew execution from a specific task.
    """
    try:
        TestCrew().crew().replay(task_id=sys.argv[1])

    except Exception as e:
        raise Exception(f"An error occurred while replaying the crew: {e}")

def test():
    """
    Test the crew execution and returns the results.
    """
    inputs = {
        "topic": "AI LLMs"
    }
    try:
        TestCrew().crew().test(n_iterations=int(sys.argv[1]), openai_model_name=sys.argv[2], inputs=inputs)

    except Exception as e:
        raise Exception(f"An error occurred while replaying the crew: {e}")

Few words on these

train():

$ crewai train -n <n_iterations> <filename> (optional)

It is essentially about training your agents (not LLM). This process enables you to give feedback at each step of execution so your agents can work in more predictable fashion. This process creates a .pkl file in your project directory. If you pass the filename parameter, it will create the file with same name. You just have to ensure that it should have .pkl as the extension.

replay():

$ crewai replay <task_id>

You have ability to trace back steps of your crew or team or any specific task.

test():

$ crewai test #or crewai test -n 5 -m gpt-4o

This will run your crew for a specified number of iterations and provide detailed performance metrics. Please see here if you are interested in getting more details.

Addressing question #2 — how does the LLM part work when we haven’t defined it?

CrewAI works with OpenAI by default; you simply need to define the OPENAI_API_KEY in your .env file.

However, you can override this and use other supported LLMs. For example, to use Azure OpenAI, you can configure it as follows:

# crew.py
@llm
def azure_llm(self) -> AzureChatOpenAI:
    return AzureChatOpenAI(
        openai_api_version=os.environ.get(
            "AZURE_OPENAI_VERSION", "2023-07-01-preview"
        ),
        azure_deployment=os.environ.get("AZURE_OPENAI_DEPLOYMENT", "gpt-4o"),
        azure_endpoint=os.environ.get(
            "AZURE_OPENAI_ENDPOINT", "https://<your-endpoint>.openai.azure.com/"
        ),
        api_key=os.environ.get("AZURE_OPENAI_KEY"),
        temperature=0.7,
    )

Like other annotations, this would create an LLM object for you as well. Make sure to pass this to your agents definition either in agents.yaml or in crew.py.

With that understanding you can now create multiple different teams or crews and automate lot of your tasks and make them run autonomously.

Does this work, or is it too good to be true?

The answer is both yes and no. CrewAI and similar frameworks are evolving rapidly. While I haven’t yet seen solutions ready for production or enterprise-level deployment, they are excellent for experimentation and learning. This doesn’t mean they won’t reach that level; given the advancements, it’s only a matter of time. CrewAI is open source, and they are developing CrewAI+, which will be their route to monetisation and also demonstrates their confidence in the product.

Onward

Well, I have just talked about the basics here, there is lot more to explore on CrewAI, e.g., Memory, Tracing, Planning, Tools, and so on. If you are enthusiastic about the autonomous AI agents, it is a fantastic framework to experiment with.

Feel free to share your thoughts in the comments, show your appreciation through claps, and don’t forget to follow me to stay updated on the future of Generative AI.

Explore More