Designing an Autonomous Cybersecurity Team of AIs with Mixture of Agents (MoA) — Part 1
We are now beginning to approach the performance of massive-parameter LLMs with models that are tens of times smaller. But of course, they cannot replace giant models in certain areas; they fall short on critical patterns such as Causal Reasoning and Chain-of-Thought. In particular, academic studies show that when we train small models on CR datasets for challenging patterns like Causal Reasoning, they exhibit significant drops on benchmarks. Therefore, while small models offer advantages, they also come with clear drawbacks. Likewise, with 235B+-parameter models, even running them — let alone fine-tuning — incurs very high costs; and even if you can budget for it, properly fine-tuning models at this scale demands extensive iteration and expertise. For this reason, AI system design is shifting from single, monolithic intelligences toward distributed, collaboratively operating communities of AIs. This is where architecture patterns like MoA come into play.
MoA is built on the core ideas of MoE (Mixture of Experts), but — crucially — implements the mixture at the system level across models rather than at the intra-model layer level. Therefore, instead of routing tokens to small expert sub-networks inside a single model, it routes them to independent LLMs that act agentically and are trained in different domains.
In the conventional approach we’re used to, we scale up a single model to expand its capabilities. In MoA, however, the problem-solving process itself is decomposed, and portions of the work are assigned to separate, individually fine-tuned LLMs. This illustrates the evolution of AI model architecture from a monolith to microservices — that is, MoA forms a microservice architecture in which each agent is a fully independent, specialized service. This structure suggests that as AI advances, its future lies in sophisticated methods that enable multiple, diverse models to collaborate effectively. You can liken it to the hierarchical division of labor in a research team: each agent behaves like a “specialist assistant” and produces solutions within its own area of expertise, while a top-level agent aggregates the assistants’ findings (like a lead or project manager) to determine the outcome and actions.
In the MoA approach, task allocation typically means assigning each agent a specific sub-task or domain and having it process data or information accordingly. For example, one agent might check code security from a cybersecurity perspective while another performs financial-statement calculations. Coordination is ensured by a hierarchy embedded in the system’s architecture: agents first work in parallel, generating different perspectives or partial solutions. These intermediate outputs are then either fed into higher-level agents or passed directly to an integrator agent, which synthesizes all outputs, resolves inconsistencies, and consolidates them into a single answer. This well-orchestrated flow resembles forming a shared inference and aligning around it.
Moreover, in some MoA designs, a referee (or judge) agent is inserted before the integrator. This referee evaluates the quality of the sub-agents’ solutions and forwards only the best — or the most diverse — ones to the integrator. For instance, in Sparse MoA (SMoA) studies, a jury model can reduce the number and cost of parallel queries by filtering some responses or even triggering their regeneration. Of course, reliably grading answer quality is a challenge in its own right. Still, it’s feasible — and with a well-designed MoA coordination scheme, multiple agents can produce solutions with a breadth and accuracy that a single agent cannot achieve. Indeed, several academic studies have observed that multiple LLM-based models yield higher-quality results when they leverage each other’s outputs.
Recent Academic Studies on MoA and Multi-Agent Systems (2023–2025)
In recent years — particularly between 2023 and 2025 — there has been growing interest in LLM-based multi-agent systems. Researchers show that orchestrating multiple artificial agents toward a shared goal opens new horizons for solving complex problems. The table below summarizes several notable academic works from this period:
RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation
This study proposes an LLM-based multi-agent architecture combining hierarchical layers, parallel agents, and an integrator. It provides higher flexibility and efficiency in multi-agent systems (MAS), delivering strong performance through parallel processing and aggregation.
SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agents
In this academic study, sparsification is added to the MoA architecture. A jury model supervises the models and filters out unnecessary answers, resulting in fewer tokens being processed. While the computational cost of MoA is reduced, the reliability of answer selection remains a challenge.
You can also review these studies. Among them, the work titled “Multi-Agent Collaboration in Incident Response with Large Language Models” is particularly relevant for us. Let’s now take a closer look at this study.
Multi-Agent Collaboration in Incident Response with Large Language Models Multi-Agent Collaboration in Incident Response with Large Language Models
In Liu’s study, virtual incident response teams composed of LLM agents are formed using Backdoors & Breaches, a cybersecurity incident-scenario game. The findings show that these LLM agents can accelerate decision-making and coordination processes, effectively supporting security teams.
I hope this has been helpful. In the next section, we will discuss how to design MoA to cover sub-domains of cybersecurity and how to train our models for these domain-specific roles.
