How to Build an AI Agent - Part 1: Vision and Planning

AI Agents are useful applications of AI and machine learning, but how are they made? This is a multi-part blog series going through the full steps of building an AI agent. 

But first, what's an AI Agent? An AI agent is a software entity that perceives its environment, processes information, plans, makes decisions, and takes actions to achieve specific goals. AI agents can range from simple rule-based software and fixed workflows to fully autonomous systems.

Key Characteristics of an AI Agent:

1. Perception – It gathers data from its environment using sensors, APIs, knowledge base, or input data streams.
2. Processing & Decision-Making – It applies logic, rules, or AI models to analyze the input and determine an appropriate response.
3. Memory – It could store and manage interaction records with users and help future decisions.
4. Action – It performs actions based on its decisions, which may include generating responses, automating tasks, or interacting with other systems.
5. Autonomy – Workflow agents automate complex/repetitive tasks for enhancing productivity. Autonomous agent operates complex tasks without constant human intervention.
6. Adaptability – Some AI agents can learn from interactions and improve over time.

Our documented steps will result in an MVP AI agent that anyone can follow. This is Part 1: Vision and Planning, where we ideate between two AI agent ideas for fun and explore what is necessary for creating a minimal viable product (MVP) before settling on which one to build.

Exploring Two AI Agent Ideas

Success comes from three key factors: a planned vision, achievable means, and efficient use of resources. — GMI's motto

We start by exploring the vision for two ideas: 

• Convention Research Assistant: an AI agent for discovering industry events and calculating the associated costs and expected returns for attending
• Outfit Seeker: an AI agent for viewing a photo or image, understanding the style and clothing, and then scraping online clothing options to generate options for buying the identified style

Both of these are projects with tangible use-cases, detailed below:

Convention Research Assistant

Keeping up with industry conventions and conferences can be overwhelming. This AI agent aims to streamline the process by:

• Finding relevant industry conventions through web scraping and data aggregation.
• Estimating costs, including travel, tickets, accommodations, and other expenses.
• Calculating the expected return on investment (ROI) based on factors such as audience, networking opportunities, and speaker lineups.
• Generating concise summaries to help users make informed decisions quickly.

Why This is a Feasible MVP

• Convention schedules and details are often available online in structured formats.
• Web scraping and simple data processing can be implemented relatively quickly.
• Cost estimation models are straightforward, relying on public travel and ticket pricing data.
• ROI estimation can start with basic heuristics and be refined over time. We actually consider this to be the hardest part of the AI agent's job as we'll need probably need to teach the agent how to score expected values of each convention.

Outfit Seeker

This AI agent would take an image of a person’s outfit—whether from a photo or a drawing—and attempt to find purchasable clothing items that match the look. The main functionalities include:

• Image recognition and classification to break down clothing items by type, color, and style.
• Searching online for similar clothing items using computer vision and web crawling.
• Handling challenges such as variations in lighting, angles, and availability of matching products.

Why This is Significantly Harder to Build

As fun as this would be to build, it's a great example of a simple idea with technical complications.

• Image recognition of fashion items requires a sophisticated deep learning model trained on extensive datasets.
• Finding exact or highly similar matches across different online stores is a complex problem involving multiple APIs and custom search engines.
• Clothing items often go out of stock or vary in price and availability, making real-time accuracy a challenge.

You want to avoid overcomplicating an AI agent, so we chose to not build this one. It's still something we'd like to probably create somewhere down the line as a fun project.

Defining the AI Agent

Once the idea is selected, it’s essential to clearly define:

• The specific problem the agent solves.
• The target users and their needs.
• Success metrics to gauge effectiveness.

This clarity ensures a focused development process with measurable goals. In our case, we can easily define these:

• Problem it solves: Automating the manual process of evaluating industry-specific events for ROI.
• Users and needs: We'll be the ones using this! In this case, our needs are for our teammates to be given a pre-researched list of conventions and industry events with the AI agent's summary of expected value weighed against associated costs. The human makes the decision at the end, and the AI agent's job is to provide conden
• Success metrics: Did it accelerate their workflows and help make them more productive? There's also the failure state where the AI agent provides wrong/incorrect/inaccurate information where it creates more work. Maybe we can compare their previous workflow against an AI-assisted workflow.

All of the above is our Vision. Now it's time to plan on how we'll execute with a Plan.

Identifying Requirements

Building a functional AI agent requires:

• Data sources (e.g., industry event listings, clothing retailer APIs, image databases).
• Core AI models and techniques (e.g., NLP for summarization, computer vision for outfit recognition).
• Infrastructure to collect, process, and present information to users.

To make this easier on us, we'll use a few open-source tools:

• Dify.ai – This is a low-code platform for building generative AI applications.
• DeepSeek-R1 – This is a lightweight and open-source LLM model. This might be overkill for the task at hand but hey, we want to play with the cool new toy. Also, GMI Cloud is hosting DeepSeek-R1 now, so we're dogfooding our own thing!

Expected Challenges & Complications

Every AI project comes with its own hurdles. Some key challenges for our project include:

• Data availability: Some information might be behind paywalls or require advanced scraping techniques.
• Technical feasibility: The convention assistant is relatively simple, but the outfit search assistant involves complex AI and search algorithms. 
• Accuracy and performance: Ensuring accurate event ROI estimates or precise outfit matches requires significant refinement over time.
• Limitations of existing AI models: Pre-trained models may need fine-tuning or additional data to be effective for our use cases.
• Solid infrastructure: We'll probably do basic refining for this project, but good infrastructure is key for continuously refining the Agent.

By anticipating these challenges, AI agent builders can make informed decisions on feasibility and development strategies.

‍

Part 2: Building an MVP – The Process

Stay tuned for part 2, where we'll document the steps we use to build an MVP of our Events Research AI Assistant!

‍