Skip to main content
AI Agents Are Running Code You Can't SeeGeneral
4 min readFor Security Engineers

AI Agents Are Running Code You Can't See

What Changed

Asymptote Labs has released Agent Beacon under the MIT license—an open-source telemetry layer that collects and normalizes activity data from AI coding agents across local endpoints, CI pipelines, and cloud environments. The tool installs on macOS through a Homebrew tap and extends OpenTelemetry standards to capture what AI agents actually do when they modify your codebase.

This matters because your existing endpoint detection and response (EDR) tools weren't built to track AI agent behavior. Tools like CrowdStrike excel at catching malicious binaries and suspicious process execution, but they don't tell you when an AI agent refactored your authentication logic, modified database queries, or pushed code that introduces new dependencies. Beacon fills that gap by creating a dedicated telemetry path for agent actions.

Key Findings

1. AI agents operate across three distinct telemetry paths that traditional EDR doesn't cover

Beacon supports agent runtimes in local development environments, continuous integration systems, and cloud-based agent platforms. Each path requires different collection strategies. A local agent running in your IDE generates different telemetry than one executing in GitHub Actions or operating through a cloud API. Without unified collection, you're monitoring fragments instead of the complete picture.

2. Full telemetry collection creates privacy concerns that require policy-based controls

Beacon records full telemetry by default, with configurable retention modes. This design choice acknowledges a fundamental tension: comprehensive visibility often conflicts with developer trust. Your team needs to see what agents changed, but developers don't want every keystroke logged. The tool addresses this through redaction capabilities and policy-based controls that let you define what gets captured versus what gets filtered.

3. Open-source governance tools enable faster adaptation than vendor solutions

The MIT license means you can modify Beacon's collection logic to match your specific compliance requirements. When SOC 2 Type II auditors ask how you track AI-generated code changes, you can point to your telemetry pipeline and show them the actual data model. When PCI DSS v4.0.1 Requirement 6.2.4 demands tracking of custom code changes, you can extend Beacon to tag AI-generated modifications differently than human commits.

4. Telemetry normalization solves the multi-agent problem

Different AI coding tools use different formats for describing their actions. One might log "refactored function X," while another records "modified lines 47-93 in file Y." Beacon normalizes these variations into a consistent schema, which matters when you're trying to correlate agent activity with security events. If a vulnerability appears in production, you need to know which agent introduced it and when—regardless of which tool was used.

What This Means for Your Team

You're probably running at least one AI coding assistant already. Your developers use GitHub Copilot, Cursor, or similar tools. Some teams have deployed agents that automatically fix dependency vulnerabilities or refactor code based on static analysis findings. Each of these creates a governance blind spot.

Your current security stack doesn't answer basic questions: Which AI agent modified the payment processing code last Tuesday? Did any agent introduce new API calls to third-party services? When the agent "improved" error handling, did it accidentally log sensitive data?

Beacon-style telemetry layers address these questions, but they also create new operational requirements. You need storage for telemetry data, retention policies that balance compliance needs with privacy concerns, and integration points with your existing security information and event management (SIEM) platform. The tool itself is open source, but the infrastructure around it requires planning.

Action Items by Priority

Immediate: Map your current AI agent usage

Survey your development teams to identify which AI coding tools they're actually using. Don't rely on procurement records—developers install tools directly. You need the real inventory before you can design telemetry collection. Document whether each tool runs locally, in CI, or through cloud APIs.

Week 1: Define your telemetry requirements

Decide what you actually need to capture. Start with compliance-driven requirements: if you're in scope for PCI DSS v4.0.1, you need to track code changes in your cardholder data environment. If you're pursuing SOC 2 Type II, you need evidence of change management controls. Map these requirements to specific telemetry fields before you deploy collection infrastructure.

Week 2: Establish privacy boundaries

Work with your development leadership to define what constitutes acceptable monitoring. Some teams will accept full telemetry with redaction of sensitive strings. Others will only agree to metadata collection (which agent, which files, when—but not the actual code changes). Document these boundaries in writing before you start collecting data.

Month 1: Deploy collection in a controlled environment

Start with a single team or project. Install Beacon (or build your own collection layer) and validate that it captures the telemetry you defined in week 1. Test your redaction rules. Verify that the data flows to your SIEM or log aggregation platform. Measure the storage and processing overhead.

Month 2: Build correlation capabilities

Telemetry only helps if you can connect it to security events. When your vulnerability scanner flags a new SQL injection risk, you should be able to query your telemetry to see whether a human or an AI agent introduced the vulnerable code. Build these correlation queries before you need them in an incident.

Ongoing: Update telemetry as agents evolve

AI coding tools change faster than traditional development tools. New capabilities mean new telemetry requirements. Schedule quarterly reviews of your collection strategy to ensure you're capturing relevant data as agent capabilities expand.

OpenTelemetry standards

GitHub Actions

Topics:General

You Might Also Like