Hmm… In my case, the main agent I use is an off-the-shelf product, but I may still have a few things worth sharing:
I am not running the same kind of local multi-agent framework, so I cannot offer a drop-in implementation recipe. My setup is closer to one managed general-purpose agent, explicit external state, and a human kept deliberately inside the control loop.
Even so, it has handled some fairly long and complicated projects reasonably well. From that experience, I think several of the goals behind Phoenix are quite practical at the component level:
- carrying work across sessions without relying on the model’s personal memory;
- preserving why a decision was made, not just the final answer;
- distinguishing current, candidate, stable, superseded, and recovery artifacts;
- creating a recovery point before mutation;
- separating planning, implementation, testing, and security concerns;
- exposing enough of the execution trace that a human can see what happened.
The difficult jump is not any one of those pieces. It is composing them autonomously while keeping authority, state ownership, verification, permissions, recovery, and escalation unambiguous. The complexity tends to grow non-linearly as more agents can mutate shared state or trigger effects that are difficult to undo.
So, at a high level, Phoenix looks directionally sensible to me. In particular:
- Decision Memory is aimed at a real continuity problem.
- Backup before code mutation is a strong default.
- Planner / Coder / Tester / Security separation can be useful when the contracts really are different.
- A central event trace and live UI can make debugging much easier.
- Local/offline execution is a meaningful privacy boundary, although it does not by itself protect the host from local commands, malicious repository content, or over-broad credentials.
I would not read the current module count or database count as good or bad by itself. The more useful question is which boundaries those modules and databases enforce, and which ones participate in the same logical operation.
What has worked in my case
I use ChatGPT as the main runtime, with memory features deliberately not relied upon. Instead, each working session reconstructs the relevant state from several explicit layers:
- Small, stable behavioral defaults — general rules such as separating observations from inference, checking unstable facts, and inspecting actual outputs rather than trusting a completion claim.
- Mission-specific workflow instructions — how this particular class of work should be researched, tested, documented, and handed off.
- Routed reusable knowledge — reference material selected through indexes and routers rather than one undifferentiated memory dump.
- Project-specific current-state / handoff material — the current canonical state, decisions, evidence, artifacts, unresolved questions, and next actions.
- Versioned artifacts — candidate, stable, superseded, and recovery versions have different authority.
- A human gate — I run the external action, check the real behavior, decide what becomes canonical, and perform publication or irreversible operations.
The key mechanism is:
It does not recreate personal memory. It reconstructs the working state, evidence, and authority order at the beginning of a session.
That is intentionally lower-autonomy than Phoenix. I am a usable component in the workflow: I can read some code, run things, notice when a result is implausible, and decide whether a candidate should be adopted. I also have no economic reason to build and maintain a separate agent runtime while a managed product is cheaper for my use case.
This does not demonstrate that Phoenix’s event-driven multi-agent integration is already reliable. It does suggest that continuity, decision records, layered knowledge, checkpoints, and explicit promotion of artifacts can work well before full autonomy is solved.
A public example of one source-guide component is this Hugging Face Spaces debugging guide. It is large, but it is organized around multiple entry paths, search aliases, freshness labels, cross-references, and selective use rather than treating the whole document as active memory. Anthropic’s long-running agent harness write-up independently reports a related pattern: fresh contexts can continue useful work when they receive explicit progress artifacts, repository history, clean intermediate states, and end-to-end checks.
The boundary I would make explicit first
For every component, I would define something like this:
component | may read | may propose | may mutate | output status | may approve | escalation
For example, a Planner might propose a plan but not modify project files; a Coder might modify only a workspace branch; a Tester might execute tests but not rewrite the oracle; a Security agent might classify risk but not be the only mechanism that decides whether a command can run; and a human or deterministic policy might remain the final authority for external or irreversible actions.
The important questions are not only “Is this agent capable?” but:
- Who owns the current state?
- Who may modify it?
- Is an output advisory, proposed, candidate, verified, or authoritative?
- Who may trigger a side effect?
- Who may approve, retry, stop, or escalate?
- What happens when two agents disagree or partially succeed?
That same table connects the memory, security, event-bus, and rollback questions. It also makes later automation incremental: one can broaden one authority boundary at a time instead of changing the whole system at once.
Three checks that seem especially high-value
1. Treat Decision Memory as a lifecycle, not only a text store
The most useful part may not be storing more prose, but preserving the status and lineage of a decision. A compact record could include fields such as:
decision_id
scope / project
source_or_evidence
target_artifact_version
status: proposed | candidate | verified | superseded | invalid
confidence
created_by
verified_by / verified_at
supersedes / invalidates
reversal_condition
This helps distinguish three different questions:
- Is this memory relevant?
- Was it supported when it was written?
- Is it still authoritative now?
The existing human practice of Architecture Decision Records is useful here: accepted records are treated as an append-only history, and a changed decision is represented by a linked superseding record rather than silently rewriting the old rationale.
A very close recent project is PROJECTMEM, which stores issues, attempts, fixes, decisions, and notes as an append-only plain-text event log and deterministically projects a compact current view for coding agents. Its published evaluation is still a small self-study, so I would treat it as a useful design example rather than an established standard.
If Phoenix already has decision status, provenance, and supersession, that sounds like a strong design choice. If it currently stores mostly narrative explanations, adding lifecycle relations may provide more value than simply filling the memory with more data.
2. Define exactly what belongs to one recovery point
“Rollback” can mean at least three different things:
- Restore: put a saved state back exactly.
- Replay: resume from a checkpoint and run later steps again.
- Compensate: perform another action that repairs or offsets an effect that cannot be undone exactly.
Git-style checkpoints are excellent for version-controlled files. They do not automatically restore:
- SQLite state;
- queued or acknowledged events;
- installed packages;
- running processes;
- external API calls;
- sent messages or published content;
- exposed credentials.
This matters because checkpoint replay can execute later model calls and APIs again, potentially producing different results. LangGraph documents that distinction explicitly in its persistence and time-travel behavior.
For effects that cannot be restored exactly, the older distributed-systems vocabulary is useful. The Compensating Transaction pattern treats recovery as a recorded sequence of application-specific undo or repair operations; those operations can also fail and therefore need progress tracking, idempotency, and sometimes manual intervention.
The central event bus adds another small but important test surface. Message systems may deliver duplicates, so a processed-event ledger or stable operation ID is often more realistic than assuming exactly-once execution. AWS’s Transactional Outbox guidance explicitly recommends idempotent consumers that track processed messages.
A compact failure test could be:
- send one operation with a stable ID;
- stop the process after the state mutation but before completion is acknowledged;
- restart and redeliver the same operation;
- verify that the mutation exists exactly once;
- repeat with a file change plus a database change, then intentionally fail a test;
- inspect whether both return to the intended recovery point.
The eight SQLite databases only become a special concern if one logical operation must update several of them atomically. If that is the case, connection and journal configuration matter; SQLite notes that in WAL mode, changes across multiple attached databases are atomic per database but not necessarily as one set. The relevant official notes are in WAL limitations and ATTACH DATABASE. If the databases are independent domains with independent recovery points, that warning may not apply.
3. Keep the Security agent, enforcement, and isolation as separate layers
A Security agent can be useful as a contextual reviewer. It can notice that a command is unusual for the current task, explain risk, or classify a proposed action.
I would still keep it separate from the mechanisms that actually enforce boundaries:
- filesystem permissions;
- command policy;
- network scope;
- secret and credential scope;
- sandbox or container isolation;
- confirmation and escalation policy;
- audit logs;
- human approval for high-risk or irreversible operations.
OpenHands makes this separation visible in its security and action-confirmation design: a security analyzer assigns risk, while a separate confirmation policy decides whether approval is required. Likewise, Hugging Face’s smolagents secure-execution guide warns that restricted local Python execution is not a complete security sandbox and recommends stronger isolation such as Docker or remote executors for untrusted code.
A useful default is therefore:
- low-risk work proceeds inside a narrow sandbox;
- crossing the sandbox boundary requires a deterministic policy decision;
- uncertain risk is not treated as safe;
- irreversible or credential-bearing actions require explicit approval;
- the Security agent can add context, but it is not the only lock on the door.
For longer autonomous runs, action-by-action review may also be insufficient. OpenAI recently described cases where individually acceptable-looking actions formed an unacceptable sequence, and responded with trajectory-level monitoring, pause capability, incident-derived evaluations, and greater user visibility. Phoenix is at a very different scale, but the general lesson is useful: the UI should ideally expose the evolving objective, events, tools, state transitions, approvals, retries, and recovery actions—not claim to reveal a perfectly faithful internal “thought process.”
A gradual route that preserves the project you already have
I would not replace the current design with a different framework. A practical route is to use the existing Phoenix modules as an experimental map and widen autonomy only where it produces measurable value.
-
Establish a simple baseline.
Use one capable model or one central manager, a constrained workspace, a repeatable task set, and human approval for high-impact actions.
-
Separate state classes.
Keep current execution checkpoints, reusable knowledge, decisions/evidence, and superseded history distinct.
-
Define recovery classes.
Mark operations as read-only, safely repeatable, compensable, approval-required, or irreversible.
-
Define component authority.
Record who may read, propose, mutate, verify, approve, stop, retry, and escalate.
-
Add specialist agents where the contract is genuinely narrower.
A Tester is useful when it owns a stable test contract; a Security reviewer is useful when it has a clear review surface; a Planner is useful when its structured output improves downstream execution.
-
Compare the simple and full routes.
Run the same small tasks through one agent, a Planner + Coder/Tester route, and full Phoenix. Measure not only pass/fail, but human interventions, duplicate actions, recovery success, runtime, model calls, and premature completion.
-
Convert real failures into regression fixtures.
Save the initial state, trigger, bad trajectory, expected stop or recovery, and final result. That gradually turns “it once broke this way” into a reusable safety and reliability test.
This is close to the incremental route in OpenAI’s practical guide to building agents: first establish accuracy with the strongest suitable model, then substitute smaller models or introduce multiple agents where the evaluation shows that the extra structure helps.
What the external-state structure looks like
The exact contents are project-specific, but the generic shape is roughly:
workflow/
├── 00_START_HERE.md
├── CURRENT_STATE.md
├── DECISIONS_AND_RATIONALE.md
├── ARTIFACTS_AND_VERSIONS.md
├── SOURCES_AND_EVIDENCE.md
├── OPEN_QUESTIONS.md
└── history/
and a reusable knowledge layer might look like:
knowledge_pack/
├── START_HERE.md
├── router.md
├── topic_index.md
├── aliases.md
├── manifest.md
├── lineage.md
└── guides/
The important rules are more important than the filenames:
- read current state before history;
- treat routers and indexes as navigation, not evidence;
- make the current canonical artifact explicit;
- retain old versions without allowing them to regain authority merely because they contain more detail;
- record decisions separately from observations and hypotheses;
- promote temporary chat work into explicit artifacts only when it becomes useful;
- verify actual files, tests, diffs, and runtime behavior rather than relying on the agent saying it finished.
That last point has been particularly important. The model can summarize what it intended to do, but adoption should depend on observable results.
Memory: current state, knowledge, decisions, and history
I would avoid one global category called “memory.” At least four kinds behave differently:
| Kind |
Main purpose |
Typical authority |
Common failure |
| Current execution state |
Resume the active task |
High but temporary |
Stale or partial checkpoint |
| Reusable knowledge |
Help many projects |
Reference only |
Outdated or over-generalized guidance |
| Decision record |
Preserve rationale and status |
Project-scoped |
Old decision remains active after conditions change |
| History / evidence |
Audit and reconstruction |
Immutable evidence |
Too much detail overwhelms the current view |
LangGraph similarly distinguishes thread-scoped checkpoints from longer-term stores. Recent memory systems such as Letta also expose layered context and Git-tracked memory concepts, but the storage technology is less important than the authority rules around writing, promotion, correction, and deletion.
A cautious write path could be:
raw observation
↓
candidate memory with source and scope
↓
contradiction / duplication check
↓
verified project state or reusable knowledge
↓
later superseded or invalidated, not silently overwritten
If external documents, repository instructions, or tool output can become durable memory automatically, I would also attach a trust class. Retrieved material can contain mistakes or instructions that should remain data rather than become system policy.
PROJECTMEM’s append-only event log is interesting because it keeps the historical substrate separate from the compact projection shown to the agent. That is close to what has worked for me: history is preserved, but the agent normally starts from a small current-state view and follows links only when more evidence is needed.
Recovery: files, databases, events, processes, and external effects
A practical classification is:
| Operation |
Default treatment |
| Read-only inspection |
Log it; apply time and scope limits |
| Version-controlled file edit |
Branch or checkpoint, then syntax/unit/E2E tests |
| Local database mutation |
Transaction or snapshot plus migration/recovery test |
| Event handling |
Stable operation ID, idempotent consumer, retry limit |
| Package or process change |
Isolated environment or reproducible environment manifest |
| Network/API mutation |
Approval plus idempotency key where available |
| Publish/send/delete/credential exposure |
Treat as irreversible or explicitly compensable |
The distinction between restore and replay is important for LLMs. A replayed model call is not guaranteed to reproduce the original text or tool choice, so a checkpoint is not automatically a deterministic snapshot of the external world.
Useful low-cost fault injections include:
- duplicate the same event;
- terminate between mutation and acknowledgement;
- terminate during compensation;
- corrupt or remove one expected artifact;
- make the test runner unavailable;
- make one database temporarily read-only;
- exhaust the retry limit;
- restore an old checkpoint while a newer artifact exists;
- verify that the system stops rather than guessing which version is canonical.
For each injected failure, the UI or log should make it possible to answer:
What was the intended operation?
What state changed?
Which changes were committed?
Which were retried?
Which were compensated?
What remains uncertain?
Who must decide the next step?
Security: reviewer, policy, sandbox, credentials, and trajectory
A compact permission matrix might be more useful than a general “security score”:
| Component |
Filesystem |
Shell |
Network |
Secrets |
External accounts |
Approval |
| Planner |
Read limited context |
None |
None or search-only |
None |
None |
Not applicable |
| Coder |
Workspace branch |
Build/test allowlist |
Off by default |
Scoped build credentials only |
None |
Boundary crossings |
| Tester |
Build artifacts and tests |
Test commands |
Usually off |
Test-only |
None |
Test-environment changes |
| Security reviewer |
Read proposed action and context |
None |
None |
Metadata only |
None |
Advisory/classification |
| Orchestrator |
State/event metadata |
Dispatch only |
Narrow |
Minimal |
Minimal |
Escalation policy |
| Human |
Explicitly chosen scope |
Explicit |
Explicit |
Explicit |
Explicit |
Final authority |
The exact rows will differ, but the exercise reveals whether a role boundary is architectural or merely a prompt label.
Other practical boundaries:
- keep untrusted repository text separate from system instructions;
- do not expose credentials merely because the agent runs locally;
- keep network access off unless the task needs it;
- prefer short-lived and task-scoped credentials;
- log approvals and boundary crossings;
- give a reviewer enough context to understand cumulative changes;
- preserve a hard stop that does not depend on the same model agreeing to stop.
A local system removes one provider boundary, but it also means the agent may be operating directly beside personal files, browser sessions, SSH keys, tokens, and development credentials. Least privilege remains useful even when every component is on one machine.
When multiple agents or multiple model sizes are likely to help
The small-planner / large-coder idea is plausible, but I would validate the routing contract separately from the worker.
A structured plan might contain:
objective
constraints
input artifact versions
allowed tools
expected outputs
acceptance tests
risk class
fallback / escalation condition
Then distinguish:
- planner chose the wrong route;
- plan omitted a required constraint;
- coder failed despite a good plan;
- test oracle failed to detect the problem;
- orchestrator ended the task too early.
A safe model-substitution route is:
strongest model establishes the baseline
↓
collect successful and failing traces
↓
replace one bounded step with a smaller model
↓
compare quality, cost, latency, and escalation rate
↓
retain the smaller model only if the contract still holds
Multi-agent systems appear strongest when work can be decomposed into genuinely independent branches. Anthropic’s multi-agent research system found strong benefits for parallel research, but also much higher token use and weaker fit for tasks with heavy shared context or serial dependencies.
There are also positive examples of highly autonomous coding teams. Anthropic’s parallel-agent C compiler experiment produced a large working artifact, but the enabling environment is informative: isolated containers, per-agent Git clones, task locks, progress documents, strong test oracles, many sessions, and a substantial compute budget. Parallelism helped when failures were independent and stopped helping when every agent reached the same serial bottleneck.
So I would not use “number of agents” as the target variable. I would use:
- independent parallel work available;
- context that can be safely separated;
- clear merge and ownership rules;
- a strong oracle;
- acceptable coordination and inference cost.
Evaluation and observability
A Tester agent is useful only to the extent that the test oracle represents the intended result. A system can become very efficient at passing the wrong test.
I would evaluate at three levels:
- Outcome — did the resulting software actually satisfy the task?
- Trajectory — were the right tools, handoffs, approvals, and recovery paths used?
- State integrity — did the project, databases, artifacts, and event ledger end in a coherent state?
For a small recurring task set, compare:
| Metric |
Single agent |
Partial specialization |
Full Phoenix |
| Acceptance-test success |
|
|
|
| End-to-end manual success |
|
|
|
| Human interventions |
|
|
|
| Duplicate/invalid actions |
|
|
|
| Successful recovery |
|
|
|
| Runtime / model calls |
|
|
|
| Premature completion |
|
|
|
OpenAI’s agent-evaluation guidance recommends using traces because they preserve the end-to-end sequence of model calls, tool calls, guardrails, and handoffs. Anthropic’s agent eval guide likewise recommends combining outcome checks, transcript inspection, code-based graders, model-based graders, and human review according to the task.
A useful regression fixture can be very small:
initial state
trigger
action or trajectory that previously failed
expected stop / confirmation / recovery
expected final artifacts and state
actual result
This lets Phoenix’s own accidents become assets over time.
For the UI, I would prioritize observable facts:
- task and operation IDs;
- current plan summary;
- selected agent and model;
- input artifact versions;
- tool calls and results;
- state transitions;
- approvals and denied operations;
- retries and compensations;
- resulting artifact and test status.
That provides strong debugging value without requiring the UI to claim that a displayed narrative is a complete or faithful window into hidden model reasoning.
Compact decision flow
Does an operation touch only version-controlled files?
├─ Yes → checkpoint/branch + syntax/unit/E2E tests
└─ No
├─ Does it change local DB, event, package, or process state?
│ └─ define a common recovery boundary + operation IDs + crash/resume tests
└─ Does it affect a network, account, publication, or secret?
└─ approval + compensation plan, or classify it as irreversible
What kind of memory is being written?
├─ Reusable knowledge → routed reference store with source and freshness
├─ Current project state → checkpoint/current-state record
├─ Decision or inference → provenance + status + supersession
└─ Untrusted observation → candidate only until verified
Does a specialist measurably improve a bounded step?
├─ Yes → retain it and define its authority and fallback
├─ Not yet measured → compare against the simple route
└─ No → keep the specialist optional or remove that handoff
My overall impression is therefore positive but conditional: the problems Phoenix is trying to solve are real, and the modular direction gives you places to improve them independently. The hardest part is likely not making each module smarter; it is preserving clear ownership and safe recovery when those modules begin acting for one another.
I would increase autonomy one boundary at a time: make the authority explicit, observe the trace, inject a failure, verify recovery, and only then broaden the tool or decision scope. That route still leaves room for a highly autonomous Phoenix later, without requiring the whole system to become fully autonomous before its individual parts can already be useful.