CrewAI: Borrowed Assumptions, Not a Ready-Made Team

The model: four concepts, one contract

CrewAI’s model is simple. An Agent has a role, a goal and a backstory. A Task has a description, an expected output and the agent that does it. A Crew puts agents and tasks together. The Process says how the tasks move forward: sequential, or hierarchical, where a manager agent hands out the work.

Our job was simple: read the weekly customer complaints, group them into categories, flag the ones with regulatory risk, and write a one-page report for managers. Three roles came out naturally:

from crewai import Agent, Task, Crew, Process

analyst = Agent(
    role="Complaint analyst",
    goal="Group the weekly complaints into categories",
    backstory="You work in the customer experience team of a brokerage.",
    allow_delegation=False,   # do not leave it to the default, write it
)
compliance = Agent(
    role="Compliance expert",
    goal="Flag complaints that carry regulatory risk",
    backstory="You know capital markets regulation well.",
    allow_delegation=False,
)
writer = Agent(
    role="Report writer",
    goal="Write a one-page summary for managers",
    backstory="You write short and clear.",
    allow_delegation=False,
)

classify = Task(
    description="Group these complaints into categories: {complaints}",
    expected_output="Count per category and a short explanation",
    agent=analyst,
)
risk = Task(
    description="Find the complaints that carry regulatory risk",
    expected_output="List of risky complaints with the reason",
    agent=compliance,
    context=[classify],       # CAREFUL: this is the analyst's OUTPUT, not the raw complaints
)
report = Task(
    description="Write the weekly report",
    expected_output="A one-page report",
    agent=writer,
)

crew = Crew(
    agents=[analyst, compliance, writer],
    tasks=[classify, risk, report],
    process=Process.sequential,
)
result = crew.kickoff(inputs={"complaints": complaint_text})

If you read the code, you see the bug right away: {complaints} appears only in the first task’s description. The compliance agent does not see the complaints. It sees the analyst’s summary. I did not see this while writing it, because I had a team in my head, and in a team everyone looks at the same folder.

Fast prototypes: really fast

First, credit where it is due. I started on Tuesday 14 July in the morning, and by the evening the first report was with the compliance team. Writing the roles also helped me think about the work. We asked “what does a compliance expert look for?” while defining the agent, and we got the answer from the compliance team. With plain code I could not have reached that point in one day. I would have had to decide how many calls each step needs and how to parse the output.

For two weeks, on 16 and 23 July, the report came out fine. Managers were happy. The 30 July report looked fine too.

From the field: the lost sentence

On Monday 3 August, after the compliance team called, we found the complaint. The customer was unhappy that a withdrawal request had waited three days. The last sentence was: “If this is not solved this week, I will file a complaint with the regulator.” The analyst agent had classified it correctly: “withdrawal delay, customer unhappy”. Correct, but incomplete. The word “regulator” was not in the summary.

The compliance agent made the right decision based on the text in front of it: delay complaints are not a risk. No agent made the mistake. The mistake was in the data flow, and the data flow was decided by the default of the sequential process, not by me. Finding it took a day and a half, because first we looked for the problem in the prompt. We made the compliance agent’s backstory longer and added “be careful”. None of it worked. We were asking the agent to read carefully a sentence it never saw.

Where it gets in your way

1. Passing context

In a sequential process, each task gets the previous task’s output as context. That means the information is compressed once at every step. In a three-step chain, the raw data stays with the first agent. The next ones see a summary of a summary. You can put the raw data into every task yourself, but then the tokens double, and the convenience of the “team” model is gone.

2. Hidden retries

An agent does not make one call per task. It thinks, calls a tool if needed, reads the result, and thinks again. If the output does not match the expected format, it tries again. There is an iteration limit (max_iter). If you did not set it, you chose the default. In one of our runs the compliance agent could not get the format right, and the run went from 38 calls to 52, and to 241k tokens. We only noticed from the total at the end of the run.

Our tools were read-only. If one of them wrote something — opened a ticket, sent a message to a customer — a hidden retry would have meant a duplicate ticket. This is the same problem as in the at-least-once post: in a system that retries, every step with a side effect has to be idempotent.

3. Tokens lost between agents

In every LLM call, the agent’s role, goal, backstory, format instructions and the previous task’s output are sent again. In one call that is small. In 38 calls it is big. Of the 186k tokens in an average run, 108k — that is 58% — was this repeated text, not the work itself. The complaint texts themselves were about 30k tokens in total.

4. Debugging

With verbose on, the text on the screen is readable, but you cannot query it. The answer to “what text did the compliance agent see in its third call?” is somewhere in that stream, but you have to scroll and read to find it. Most of the day and a half went here.

Monitoring CrewAI: who spent what?

When a run ends, the crew gives you a usage summary: total tokens, total requests. That is enough for the cost per task I described in the token dashboard post. But it does not answer “why so much?”. To get the breakdown, we moved monitoring under the framework, to the level of each LLM call:

# wrap the LLM client: whatever the framework does, every call passes here
def traced_call(messages, **options):
    start = now()
    response = llm_client.call(messages, **options)
    record({
        "run_id"    : current_run(),         # one weekly report = one run
        "agent"     : current_agent_role(),  # set when a task starts
        "task"      : current_task(),
        "iteration" : iteration_counter(),   # which call inside the same task
        "input_tok" : response.usage.input,
        "output_tok": response.usage.output,
        "time_ms"   : now() - start,
        "input"     : messages,              # the real win: what the agent SAW
    })
    return response

CrewAI accepts callbacks at the end of tasks and steps. We used them only to mark the current agent and task. The real information came from recording the input of each call. When we replayed the missed complaint with these records, one query showed that the word “regulator” was not in the text the compliance agent saw. Two minutes instead of a day and a half.

The same job, twice: CrewAI and plain code

After we found the bug, I wrote the same job in plain code. No agents, no roles: three steps and a clear data contract between them:

# 1) classify: batches of 20, JSON output, schema validation
categories = [classify(batch) for batch in batches(complaints, 20)]   # 6 calls

# 2) risk: the RAW complaint text goes in, not a summary
#    first a cheap pre-filter (regulator, lawyer, lawsuit, court...), then the model
candidates = pre_filter(complaints) + risky_categories(categories)
risks = [assess_risk(batch) for batch in batches(candidates, 10)]    # ~6 calls

# 3) report: one call, the input is structured data
report = write_report(categories, risks)                            # 1 call

Then we ran both versions on four weeks of complaints that the compliance team had labelled by hand. There were 11 risky complaints in those four weeks:

One honest warning about this table: I wrote the plain code version after I knew the problem. Those three days include what I learned from CrewAI. Starting from zero, plain code would probably have taken longer. The debugging row is not fair either: the two bugs were not equally hard, and CrewAI’s day and a half was before we had monitoring. Do not read the table as “CrewAI is bad”. Read it as “where do you pay for what”.

What did not work for me

• Fixing it with a longer backstory. I added three paragraphs of regulation to the compliance agent’s backstory. Because it is sent again in every call, the run went from 186k to 214k tokens. The regulator sentence still did not arrive, because the problem was not what the agent knew. It was the text in front of it.
• Switching to the hierarchical process. We said, “Let a manager agent hand out the work; maybe it will give the raw data to the right agent.” Calls went from 38 to 61 and the result did not change. You cannot fix a data flow problem by adding one more agent.
• Treating screen output as a log. For two weeks we said, “verbose is on, we can see it.” We could see it. But we did not store it, and we could not query it.

What to track

Checklist

Conclusion

That Thursday’s report was not wrong. It was incomplete. Every agent made the right decision based on the text in front of it. What was missing was a decision I did not write, but that the framework made for me: what does the compliance expert read?

We did not drop CrewAI. When a new job comes in, we still start with it on the first day, because it is the fastest way to find the steps. But once the steps are clear, we move to plain code, and we see that move not as a fix but as the job growing up.

The test: can you say exactly what text each agent in your crew saw in a given call? If you cannot, you do not have a team. You are working with someone whose assumptions you do not know.