Hmm… based on what I could find, I think it looks roughly like this:
Direct answer: if I were choosing the next single endpoint to wrap, I would probably start with oauth-watchlist.
The practical reason is that this is already exposed by the public relayshield-mcp package, so there is an existing contract to reuse and compare against. The current public implementation takes an email address rather than a raw OAuth token, which also makes it a relatively low-friction next wrapper.
After that, I would group oauth-watchlist, nhi-exposure, and session-risk as one agent authority / credential exposure family rather than presenting them as three unrelated tools:
oauth-watchlist: risk inherited from connected SaaS applications and delegated accessnhi-exposure: API keys, service accounts, PATs, machine identities, and other non-human credentialssession-risk: active or reusable session material that may bypass normal authentication controls
That seems closer to the actual agent-security question: can someone else exercise the authority this agent currently holds?
My next choice after that would probably be supply-chain. I would not necessarily convert every remaining endpoint into an MCP or smolagents.Tool, though. Some appear more natural as scheduled checks, local hooks, gateways, or batch integrations.
| Use case |
Default delivery surface I would consider |
| An agent or analyst wants enrichment during an investigation |
Advisory MCP / smolagents.Tool |
| A check must occur before sending, purchasing, deleting, connecting, or executing |
Host-side pre-action gate |
| The input may contain raw secrets, private source, or local credentials |
Local/user-run hook or scanner |
| The signal changes slowly or should be monitored continuously |
Scheduled job / webhook |
| Large IOC or fleet-wide processing |
Batch / SIEM integration |
The main architectural distinction I would make explicit is:
A security tool that the model may call is not the same control as a security gate that the system must pass.
A smolagents.Tool makes a capability available to the agent. That is useful for investigation and self-checking, but it does not by itself guarantee that the model will invoke the check before a sensitive action. For the stronger claim—“check before acting”—I think a second reference workflow would be valuable: put the check in a host hook, gateway, or composite tool that controls access to the side-effecting operation. CrewAI’s before_tool_call hooks, for example, can inspect an action and return False to block it.
So I would suggest publishing two clearly named examples:
- Advisory investigation tool — the agent can request risk context and show the evidence to the user.
- Mandatory pre-action gate — the raw high-impact action is not directly available until the host has run the required check and made an allow/review/deny decision.
The other high-value improvement, especially for the second workflow, would be a small machine-readable result contract. Something along these lines would already make integration much easier:
{
"outcome": "finding | no_known_finding | unknown | error",
"recommended_action": "allow | review | deny | defer",
"reason_codes": [],
"evidence": [],
"coverage": {
"complete": false,
"scope": "what was actually checked"
},
"freshness": {
"observed_at": "timestamp or null",
"expires_at": "timestamp or null"
},
"error": {
"kind": "timeout | rate_limited | auth | upstream | malformed_response | other",
"retryable": true
}
}
I would deliberately use no_known_finding, not safe. A threat-intelligence lookup can report that it found nothing in the sources and scope queried; that is not generally the same statement as proving that the target is safe.
MCP already has the relevant protocol pieces: outputSchema and structuredContent, plus isError for tool-execution failures. smolagents also supports object output types, so this does not need to remain a human-formatted string-only interface.
I did a small contract-level sanity check using the current public wrappers and synthetic local upstream responses only—not the live detector and not a detection-accuracy test. In the public MCP package, synthetic upstream 401, 429, 500, HTTP 200 with {"ok": false}, malformed/empty responses, and a network disconnect were returned as ordinary tool content with isError: false; missing required arguments and an unknown tool name did produce isError: true. That matches the current public response-handling path in server.py.
I would not interpret that as anything about the threat-intel backend. It is just an integration-contract observation. But if these checks are ever used as policy gates, distinguishing clean result, unknown result, and failed check becomes important; otherwise a client can accidentally treat “the check did not complete” as “nothing was found.”
A fairly small next release could therefore be:
- Add the
oauth-watchlist smolagents.Tool.
- Add one host-side mandatory-gate example around an existing high-impact action.
- Add a shared structured result/error schema.
- Publish redacted fixtures for positive, negative, stale, rate-limited, malformed, and upstream-error cases.
- Add a short data-flow note for each endpoint.
That would make the project easier to evaluate and integrate even for people who cannot inspect the proprietary backend or call the live API.
Why OAuth, NHI, and session risk seem like one useful agent-security bundle
Agents increasingly act through delegated and machine-held authority rather than through a human entering a password for every operation. That authority can exist in several forms:
- OAuth access or refresh tokens
- service-account credentials
- API keys and personal access tokens
- session cookies or other bearer session artifacts
- credentials inherited from the host environment
- connected third-party applications with delegated account access
This is why I think the three endpoints form a coherent family.
The OWASP Non-Human Identities Top 10 provides useful surrounding vocabulary here: secret leakage, overprivileged identities, long-lived secrets, insecure authentication, reuse, and lifecycle/offboarding problems. It is not specific to agents, but those risks map naturally to agents because agents often operate as machine identities or exercise machine credentials.
For OAuth, the current check_oauth_watchlist contract is especially convenient because it asks for an email identity and looks for connected-app exposure; it does not ask the tool caller to submit a raw access token. That distinction is worth keeping visible in the documentation.
The broader OAuth security context is also useful when deciding what a positive finding should cause. RFC 9700 emphasizes token replay prevention, refresh-token rotation or sender constraints, audience restriction, and least privilege. A watchlist hit may therefore suggest actions such as reviewing connected applications, revoking grants, rotating credentials, or reducing scopes—but the tool should probably return evidence and recommended actions rather than autonomously revoking access unless that separate authority has been explicitly granted.
For session-risk, I would keep “session evidence” separate from “authorization to act.” A restored process or agent state does not prove that its previous authorization remains valid. Session expiry, revocation, credential rotation, and already-executed external actions may all have changed since the state was captured.
A compact lifecycle mapping might be:
| Lifecycle point |
Useful checks |
| Agent/service onboarding |
NHI exposure, supply-chain posture, least-privilege review |
| Before connecting to an external server/tool |
MCP registry risk, domain/reputation, supply-chain |
| Before a high-impact action |
session risk, OAuth/identity exposure, policy-specific checks |
| Periodic operation |
OAuth watchlist, NHI exposure, public-repository secret posture |
| Incident or suspicious behavior |
session risk, infostealer/breach correlation, credential rotation |
| Offboarding |
token revocation, session invalidation, NHI removal, audit reconciliation |
This also provides a more scalable roadmap than ranking all 24 endpoints in one flat list.
Suggested decision tree
- If the result is only advisory context: expose it as an MCP or framework tool.
- If a protected action must not proceed without the check: enforce it outside the model with a hook, gateway, or composite tool.
- If the check requires raw secrets or private code: prefer local processing and send only the minimum derived value needed.
- If the signal is posture or monitoring data: prefer scheduled or event-driven delivery.
- If the result is
unknown, stale, partial, rate-limited, or malformed: route to review/defer rather than silently treating it as allow.
- If the result is a finding: keep remediation as a separately authorized action unless the deployment has explicitly opted into automatic response.
Minimal reference workflows
Advisory investigation tool
- The agent or user requests a check.
- The adapter calls RelayShield.
- The adapter normalizes the response into
finding, no_known_finding, unknown, or error.
- It returns evidence, coverage, freshness, and suggested next actions.
- It does not silently convert an unavailable check into a clean result.
- Any remediation is a separate, explicit operation.
Mandatory pre-action gate
- The model proposes a side-effecting action.
- The raw action is not directly executable by the model.
- A host hook/gateway extracts the least-sensitive identifiers needed for the security check.
- The gate receives a typed result.
- A fresh, sufficiently covered
no_known_finding may allow the action.
- A finding follows the deployment policy: deny or require review.
- Timeout, rate limit, stale data, partial coverage, malformed response, or upstream failure follows an explicit degraded-mode policy—normally review/defer for high-impact actions.
- The audit record stores the decision and reason codes without unnecessarily storing secrets.
This is also where a small “degraded mode” can help. If the security service is temporarily unavailable, the agent might still be allowed to read local information or prepare a draft, while external sending, purchasing, deletion, or execution remains deferred.
Integration contract, credentials, documentation, and validation details
Credential placement and tool visibility
The public local relayshield-mcp package keeps the RelayShield API key or x402 proof in environment variables and sends them as headers. That is a good separation from ordinary tool arguments.
The hosted Agentic Attack Surface Space, however, currently includes api_key in the public function/tool inputs in app.py. Depending on the MCP client and observability setup, ordinary tool arguments may be included in model-visible context, traces, replay logs, or debug output.
A useful hosted reference configuration would therefore keep service credentials outside the model-generated argument schema—for example, server-side environment configuration, a gateway, or an authenticated request header. Gradio’s gr.Request can access request headers inside the prediction function, so the Space does not necessarily need to expose the API key as a semantic tool parameter.
This is mainly a deployment-boundary point, not a claim that a credential has leaked.
For future OAuth/session/NHI tools, the documentation could state whether the request contains:
- an email or account identifier
- a domain
- provider/client/application identifiers
- scopes or token metadata
- a hash/fingerprint
- a raw token, cookie, API key, or secret
- repository names, paths, snippets, or contents
The current OAuth watchlist tool uses an email, not a raw OAuth token. I would preserve that distinction. If a future check can operate from issuer/client ID/scope, a prefix, or a one-way fingerprint, that may be preferable to transmitting bearer material.
If raw downstream tokens ever do cross an MCP boundary, the MCP security guidance on token passthrough is relevant: accepting a token intended for another service and forwarding it without proper audience validation is an anti-pattern. This is not necessarily what RelayShield currently does; it is simply a useful boundary to document before adding token-oriented checks.
Data-flow note that would help adoption
For a security API, the data-handling page is part of the integration contract. A short endpoint-specific table could cover:
| Question |
Example field |
| What leaves the caller? |
email, domain, URL, repository metadata, token fingerprint |
| Is the value raw, truncated, hashed, or normalized? |
raw email, SHA-256 fingerprint, registrable domain |
| What is used only in memory? |
request payload |
| What may be retained? |
result, billing record, abuse-prevention record |
| What may enter logs or traces? |
identifiers, response codes, request IDs |
| How long is it retained? |
endpoint-specific retention |
| Can the caller request deletion? |
procedure and limits |
| Is the result cached? |
cache key and TTL |
| How is tenant isolation handled? |
account/project boundary |
This is especially relevant because the security checks themselves may receive the exact identifiers and credentials that an adopter is trying to protect.
Clarifying prompt-injection-breach
There appears to be a terminology/documentation difference worth resolving gently.
The HF blog post describes prompt-injection-breach as checking whether incoming content matches known prompt-injection patterns. The public MCP/PR material describes an email-based check for credential or session exposure associated with prompt-injection attacks against agents; the CrewAI PR calls it “credential breach exposure sourced from prompt-injection attacks.”
Both are potentially useful, but they are different integration contracts:
- Content scanner: input is untrusted text/document/tool output; result concerns the content being ingested.
- Breach-source correlation: input is an identity such as an email; result concerns previously observed compromise evidence.
If both exist, I would expose them as two separately named tools. If only the second exists today, aligning the Forum/blog wording with the email-based input and evidence model would prevent downstream users from placing it at the wrong point in an agent pipeline.
x402 and API-key routes
The API-key and x402 paths should remain distinct in examples. The discussion in CrewAI PR #6550 correctly notes that omitting an API-key header does not by itself implement x402: the client also needs the appropriate payment negotiation/signing flow and must call the intended route.
That argues for separate examples or adapters:
- API-key/metered client
- x402-capable client
- discovery response when neither is configured
This avoids making a wrapper appear keyless when it would simply receive an authentication failure.
Fixtures and controls
A redacted fixture pack would let outside users review the integration contract without requiring production access or sensitive data. I would include at least:
- confirmed/synthetic positive
- clean negative
- no known finding with explicit scope
- partial coverage
- stale result
- authentication failure
- timeout
- rate limit with retry information
- upstream
5xx
- malformed or empty upstream response
- asynchronous/pending result, where applicable
For each fixture, include the expected normalized outcome and action:
| Fixture |
Expected normalized result |
Typical gate action |
| Positive with usable evidence |
finding |
deny or review |
| Fresh, sufficiently covered negative |
no_known_finding |
allow according to policy |
| Partial or stale |
unknown |
review/defer |
| Timeout / 429 / upstream failure |
error |
retry or review/defer |
| Malformed response |
error |
do not treat as clean |
The most useful end-to-end controls are at the protected-action boundary:
- Does a malicious/synthetic-positive fixture actually prevent the protected action?
- Does a benign control still pass?
- Does service failure avoid becoming a false allow?
- Is automatic remediation impossible without separate authorization?
- Are logs useful without recording unnecessary secrets?
- Can the same policy be replayed against a saved fixture after a wrapper upgrade?
Versioning the input schema, output schema, evidence vocabulary, and policy recommendation separately would also make migrations less surprising.
ATT&CK dataset positioning
The MITRE ATT&CK group/technique dataset looks useful as a transparent taxonomy and join layer: group IDs, aliases, descriptions, technique IDs, software IDs, and source links are easy for others to inspect and reuse.
I would position it as:
- public taxonomy and normalization
- retrieval or enrichment material
- an explainability layer connecting findings to known groups/techniques
- test data for joins, schemas, and UI paths
I would not use the existence of the dataset itself as evidence of proprietary detector precision or recall; that would require a separate labeled evaluation with controls and a documented sampling method.
For future readers searching specifically for AI-system adversary techniques, MITRE ATLAS is a useful complementary vocabulary. ATT&CK remains relevant for the conventional infrastructure, identity, credential, and post-compromise parts of the chain; ATLAS can supplement it where the behavior is specifically about AI-enabled systems.
Overall, I think the strongest near-term route is:
Ship oauth-watchlist as the next low-friction wrapper, define NHI/OAuth/session as the next coherent bundle, and publish one advisory workflow plus one mandatory-gate workflow using a shared typed failure contract.
That gives people something immediately usable while also making the boundary between threat-intel enrichment and enforceable policy much clearer.