Deep Dive Part 2: How does BabyAGI actually work?

Hi Hitchhikers,

Last week, we explored the fascinating world of autonomous AI agents and how they could be the next step toward Artificial General Intelligence (AGI). One of the standout examples of such agents is BabyAGI—an AI system that autonomously manages tasks by creating, prioritizing, and executing them based on a predefined objective and the outcomes of previous tasks. In case you missed it, here’s how I described BabyAGI last week1:

"My favorite example of this to date is BabyAGI, an agent created by Yohei Nakajima, a Venture Capitalist, that uses GPT-4 and Pinecone for memory to create and perform tasks autonomously! ... In summary, BabyAGI is an AI system that autonomously manages tasks by creating, prioritizing, and executing them based on a predefined objective and the outcomes of previous tasks. It interacts with GPT-4 for natural language processing and uses Pinecone for data storage and retrieval."​

Today, we will take a closer look at BabyAGI and go deeper into how it works. We'll explore the inner workings of its code and understand the techniques used by BabyAGI to achieve autonomy in task management. Let's dive in!

Ready to jump in?

Before we get started, please help me on my quest to reach 5,000 subscribers by sharing this newsletter with a friend!

How BabyAGI Actually Works

Before we dive into the code, let's provide a quick refresher on what BabyAGI is and how it works. BabyAGI is an autonomous agent system that uses large language models (LLMs)2 to carry out tasks based on a defined objective. The core idea is to have a series of agents that can work together to execute tasks, create new tasks based on the results of the execution, and prioritize tasks in a list. Each of these agents is powered by an LLM, making them capable of natural language understanding and generating meaningful responses to prompts.

The high-level idea of how BabyAGI works can be summarized as follows:

1. The User provides an Objective and an Initial Task. The Objective is the overall goal that the agents will work towards, and the Initial Task is the first task to be executed in order to achieve the Objective.
2. The system starts a loop where it continuously checks for tasks in the task list. If the task list has items, it will proceed with the following steps: a. Execution Agent: The first task in the list is sent to the Execution Agent. The agent is prompted with the task and generates a response as the result of the task execution. The response is saved for future context. b. Task Creation Agent: The Task Creation Agent is prompted with the last result and remaining tasks. It generates new tasks that should be added to the task list based on the result of the previous execution. c. Prioritization Agent: The Prioritization Agent is given the task list along with the knowledge of the Objective. It is responsible for cleaning the formatting and reprioritizing the tasks in the list.
3. The loop continues until all tasks are completed or the Objective is achieved.

Now that we have a high-level idea of how BabyAGI works. Let’s jump into the details…

Understanding BabyAGI's Core Components

The code for BabyAGI is written in Python and the crux of it is all in one script with 313 lines of code3. The script consists of several key components that work together to achieve autonomy:

1. Task List: task_list is a deque (double-ended queue) that holds the tasks that need to be completed. Each task is represented as a dictionary with a "task_id" and a "task_name." BabyAGI starts with an initial task and continuously adds new tasks to this list.

2. Task Creation Agent: task_creation_agent is a function that uses GPT-4 to generate new tasks based on the result of the previously completed task and a predefined objective. The function sends a prompt to OpenAI's API and receives a list of new tasks as strings.

3. Execution Agent: execution_agent is responsible for performing the tasks in the task list. It leverages GPT-4 and relevant context retrieved from Pinecone4 to complete each task based on the objective. The context is retrieved using context_agent to find relevant previous tasks.

4. Prioritization Agent: prioritization_agent reprioritizes the task list based on the objective and other factors. It sends a prompt to OpenAI's API and receives the reprioritized task list as a numbered list.

5. Pinecone: Pinecone is a vector database used to store and retrieve task results for context. BabyAGI uses Pinecone to create an index based on a specified table name and then stores the results of tasks, along with task names and metadata.

Infinite Loop and Workflow

The script operates in an infinite loop, and in each iteration, it performs the following steps:

1. Pull the First Task: BabyAGI dequeues the first task from the task list.
2. Execute the Task: BabyAGI sends the task to the execution agent, which uses OpenAI's API to complete the task based on the context retrieved from Pinecone.
3. Store the Result: The result is enriched and stored in Pinecone for future context retrieval.
4. Create New Tasks: The task creation agent function generates new tasks based on the objective and the result of the previous task. The new tasks are added to the task list.
5. Reprioritize the Task List: The prioritization agent function reprioritizes the task list based on the objective and other factors.

This process repeats continuously, allowing BabyAGI to autonomously manage tasks by creating, prioritizing, and executing them based on a predefined objective and the outcomes of previous tasks.

The Magic of Prompt Engineering

Prompt engineering5 plays a crucial role in BabyAGI's ability to generate meaningful tasks and responses. By carefully crafting the prompts sent to GPT-4, BabyAGI is able to communicate its objective, the context of previous tasks, and the specific task it needs to perform.k

For instance, in the task_creation_agent function, the prompt is designed to provide GPT-4 with the context it needs to generate new tasks:

prompt = f""" You are a task creation AI that uses the result of an execution agent to create new tasks with the following objective: {objective}, The last completed task has the result: {result}. This result was based on this task description: {task_description}. These are incomplete tasks: {', '.join(task_list)}. Based on the result, create new tasks to be completed by the AI system that do not overlap with incomplete tasks. Return the tasks as an array."""

The prompt clearly defines the AI's role and objective, provides the result of the previous task, and specifies the task description and the list of incomplete tasks. By carefully framing this information, BabyAGI enables GPT-4 to generate meaningful and relevant new tasks that align with the overall objective.

Similarly, the execution_agent function uses a prompt that includes both the objective and the task that needs to be performed:

prompt = f""" You are an AI who performs one task based on the following objective: {objective}\n. Take into account these previously completed tasks: {context}\n. Your task: {task}\nResponse:"""

Again, by providing the objective, the context of previous tasks, and the specific task, BabyAGI can guide GPT-4 toward generating a response that meets the desired criteria.

The prioritization_agent function uses a prompt that requests GPT-4 to reprioritize the task list based on the objective:

prompt = f""" You are a task prioritization AI tasked with cleaning the formatting of and reprioritizing the following tasks: {task_names}. Consider the ultimate objective of your team:{OBJECTIVE}. Do not remove any tasks. Return the result as a numbered list, like: #. First task #. Second task Start the task list with number {next_task_id}."""

This prompt ensures that GPT-4 reprioritizes tasks in a way that aligns with the objective and does not remove any tasks from the list.

The Art of Autonomy

BabyAGI's ability to manage tasks autonomously in a continuous loop makes it so intriguing. It's not just about completing individual tasks; it's about how BabyAGI dynamically generates new tasks based on the outcomes of previous ones, reprioritizes them, and iterates on this process indefinitely.

This level of autonomy is particularly exciting when we consider its potential applications in various domains, such as project management, customer support, and research. Imagine an AI system that can autonomously manage complex projects, create new action items based on progress, and prioritize tasks to meet specific goals.

As I mentioned last week, the implications of such autonomous AI agents extend beyond productivity and efficiency—they could represent a step toward AGI, where AI systems can understand and interact with the world similarly to human intelligence.

BabyAGI is growing up quickly

Since BabyAGI was released just 18 weeks ago by Yohei Nakajima, the AI community has been buzzing with excitement, and developers have been inspired to create their own projects based on the BabyAGI framework. BabyAGI’s creator Yohei Nakajima recently highlighted some of the most exciting projects that have emerged in just the last few weeks, showcasing the versatility and potential of autonomous agents:

1. Congoys: A web-based agent named "Cognosys" that serves as a version of AutoGPT/BabyAGI, accessible through a browser​.
2. GOD mode: An agent that can order coffee at Starbucks, perform market analysis, find and negotiate leases, and more. It was inspired by AutoGPT and BabyAGI and is available at godmode.space​​.
3. LLMitlessAPI: A single endpoint that triggers an agent, allowing users to define the service they want the agent to perform and the data to act upon. It can be used for chat and other applications​.
4. Baby AGI as a service: A tool that allows integration of BabyAGI with user applications using langchain-serve​.
5. Baby AGI on Streamlit: A Streamlit web app that makes BabyAGI more accessible to user.
6. Otto: A system that uses LLMs and vector databases for self-governing digital organizations, enabling equal agent participation in management​​.
7. Toddler AGI: An architecture based on BabyAGI that allows multiple instances to work on the same database and use human reinforcement to prioritize important problems​​.
8. Do Anything Machine: A to-do list that autonomously completes tasks using a GPT-4 agent. The agent has context about the user and their company and can access relevant apps.
9. GPT-Legion: An autonomous agent framework similar to BabyAGI but written in TypeScript​​.
10. BabyAGI-asi: A modified BabyAGI that runs arbitrary Python code to perform tasks such as routines for pyautogui, web scraping, and computer actions​.
11. AgentGPT: An agent that performs tasks directly in the browser. Users can give the agent a goal, and it will think, plan, and execute actions accordingly​.
12. Baby AGI as a ChatGPT plug-in: A version of BabyAGI that runs as a plug-in within ChatGPT, allowing the agent to create content like a 250-page sci-fi novel​.
13. Coding agent: An agent that follows the Test-Driven Development (TDD) methodology, allowing users to write tests while the agent iteratively develops the featur​.
14. BabyAGI + Langchain Tools: An agent that uses Langchain tools to look up real information and take actions based on the context​.
15. Baby AGI in Slack: An agent that uses Pinecone and Slack to create threads with new objectives and perform tasks autonomously​.
16. Teenage-AGI: BabyAGI but with more memory capabilities.

Unfortunately, Twitter no longer allows Tweets to be embedded in Subtack posts so I encourage readers to check out the original Twitter thread and support these projects​ here: Yohei on Twitter: "It’s been 18 days since sharing the “paper” on BabyAGI… The most exciting to me is what people have started building since then, and the discussions around it! Check them out👇" / Twitter6​

Throwing the baby out with the bathwater

As we continue to witness rapid advances in AI research and development, systems like BabyAGI remind us of the limitless possibilities of AI and its potential to shape the future in profound ways. At the same time, with the AI space moving so quickly, there’s a legitimate fear from members of the tech community that tools like BabyAGI could be used for more harm than good. The prominent VC and All In podcast host Chamath Palihapitiya is even calling for the government to step in immediately with regulations:

If you’re interested in hearing more of the arguments for/against regulation in AI, I recommend listening to the latest episode of All In. In the episode, Besties talked about tools like BabyAGI and how they could be used in both positive and dangerous ways:

As I’ve said in previous posts, I really don’t think we should act too quickly to restrict innovation in this space at this early stage. Furthermore, I think it’s entirely unrealistic for regulate language models because there will be open-source versions in the next year that will perform just as well as OpenAI’s LLMs. Regulating open-source code is exceptionally challenging because it can easily be shared, copied, and changed, and there’s no central actor that regulators can hold accountable.

I think a better path forward for regulations is to make the law clearer around the responsibility of individuals that use these tools to do damage. For example, if an individual instructs an AI Agent to cause harm to others, they should be made responsible for the harm caused. Now, the argument that you can make here is “What if the AI autonomously decides to cause harm itself”. We’re still far away from that being the case without any human instruction or objective setting. There hasn’t yet been an example of any LLMs deciding without any provocation to wish harm on humans that I’ve seen.

AI agents like BabyAGI are good at carrying out human instructions and objectives. What makes them novel over traditional computer programs is that they are able to do this using a natural language interface and they can figure out how to solve problems without step-by-step instructions. This, to me, is what makes them such an exciting technology and I can’t wait to see how they evolve in the coming weeks!

That’s all for this week. If you enjoyed my writing, then please support The Hitchhiker’s Guide to AI by subscribing - it’s completely free!

Thanks for reading The Hitchhiker's Guide to AI! Subscribe for free to receive new posts and support my work.