The Executive Order on AI Safety, issued on October 30 by President Biden, isn't legally binding. Compliance teams often use this fact to justify inaction. However, this EO signals future regulatory directions, and many organizations are making predictable mistakes that will cost them later.
I've observed teams mishandle similar regulatory transitions—GDPR, CCPA, PCI DSS v4.0.1—and the pattern is consistent. Organizations that treat advance guidance as optional scramble when enforcement arrives. Those that use it as a roadmap build compliance into their systems from the start.
Why These Mistakes Keep Happening
The EO's language is broad. It mentions "frontier AI models" without a precise definition and sets a threshold for reporting at models trained with tens of millions of compute hours, without explaining how to measure that for fine-tuned models or ensemble systems. This ambiguity leads teams to make assumptions instead of building frameworks.
Your compliance team might be waiting for "real" regulations before acting or treating the EO as a checklist to satisfy. Both approaches miss the point. The EO indicates where agencies will focus their rulemaking efforts. Use this information to prepare.
Mistake 1: Treating Open Source AI Models as Someone Else's Problem
Why it happens: Your team downloads Llama 2, fine-tunes it on internal data, and deploys it. The model came from Meta, so compliance is Meta's responsibility, right?
The consequence: The EO's language around open source models is vague, but the direction is clear—deployers are responsible for how they use AI systems, regardless of origin. If your fine-tuned model produces biased outputs or leaks training data, saying "we just used an open source model" won't satisfy regulators.
The fix: Create an internal policy for open source AI adoption now. Document which models you're using, how you've modified them, what data you've used for fine-tuning, and what controls you've implemented. If you can't answer "what would we need to report about this model?" for each system, you're not ready.
Specifically, maintain a register that tracks the base model, training compute (even if estimated), intended use case, and responsible team. This isn't overhead—it's the foundation for the reporting requirements that are coming.
Mistake 2: Ignoring the AI Bill of Materials Because It's Not Required Yet
Why it happens: AIBOMs aren't mentioned in most compliance frameworks your team already follows. There's no SOC 2 Type II control for "document your AI supply chain." So it gets deprioritized.
The consequence: When agencies formalize AIBOM requirements—and the EO strongly suggests they will—you'll be starting from zero. Worse, you won't have historical records of which model versions were deployed when, making incident response and audit trails nearly impossible to reconstruct.
The fix: Start building your AIBOM incrementally. For each AI system in production, document:
- Model architecture and version
- Training data sources (even if summarized)
- Third-party APIs or services integrated
- Compute resources used for training and inference
- Known limitations or failure modes
You don't need a perfect system. You need a consistent practice. A spreadsheet that your team actually maintains beats a sophisticated tool that nobody updates.
Mistake 3: Separating AI Compliance from Your Existing Security Program
Why it happens: AI feels new, so teams create separate processes, separate documentation, separate review workflows. It's treated as a special case rather than another technology stack.
The consequence: You end up with compliance silos. Your vulnerability management process doesn't cover prompt injection. Your access controls don't address model weights. Your incident response plan doesn't account for model poisoning. When regulators ask about your AI security posture, you can't point to integrated controls—you have a patchwork.
The fix: Map AI risks to your existing control frameworks. If you're following NIST CSF v2.0, where do AI-specific risks fit? If you're maintaining ISO 27001 compliance, which controls need to be extended to cover AI systems?
For example, your access control policy (ISO 27001 A.9) should explicitly address who can modify model parameters, access training data, or deploy new model versions. Your change management process should include model updates alongside code deployments. Don't create parallel processes—extend what you already have.
Mistake 4: Waiting for Perfect Clarity on Compute Thresholds
Why it happens: The EO specifies reporting requirements for models trained with "tens of millions of compute hours," but your team doesn't know if your models hit that threshold. So you wait for more specific guidance.
The consequence: You're not building the measurement infrastructure needed to answer the question. When clarification arrives, you still won't know which of your models qualify because you haven't been tracking compute.
The fix: Start measuring now, even if your methodology isn't perfect. For each training run, log:
- GPU/TPU hours used
- Instance types and quantities
- Training duration and iterations
- Approximate total compute (even if estimated)
If you're using cloud providers, this data is already in your billing. Extract it. If you're running on-premises, instrument your training pipelines to capture it. The goal isn't precision—it's establishing a baseline so you can identify which systems might fall under future thresholds.
Mistake 5: Assuming Algorithmic Bias Testing Is Optional Until Mandated
Why it happens: Bias testing feels subjective. Your team doesn't have clear metrics for what constitutes "fair" outputs, so testing gets postponed until there's regulatory guidance on methodology.
The consequence: You deploy AI systems that embed bias, and you have no evidence that you attempted to detect or mitigate it. When regulations formalize bias testing requirements, you'll need to retrofit testing into production systems—far harder than building it in from the start.
The fix: Implement basic bias testing for any AI system that makes decisions about people. This doesn't require sophisticated fairness metrics. Start with:
- Demographic parity checks: do outcomes differ significantly across protected groups?
- Error rate analysis: are false positives/negatives distributed evenly?
- Edge case testing: how does the model perform on underrepresented groups?
Document your testing methodology and results. If you find bias and can't eliminate it, document the mitigation steps you took and the residual risk you accepted. This record demonstrates good faith effort—which matters when regulators evaluate compliance.
Prevention Checklist
Use this checklist quarterly to audit your AI compliance posture:
- AI system inventory is current and includes all production models
- Each AI system has a documented owner and intended use case
- Open source models are tracked with modification history
- AIBOM exists for each production AI system (even if incomplete)
- AI-related risks are mapped to existing security controls
- Training compute is logged for new models
- Bias testing is performed before production deployment
- Incident response plan includes AI-specific scenarios
- Privacy impact assessment covers AI data processing
- Regular review process exists for AI system performance and drift
The EO isn't enforcement. It's a preview. The teams that treat it as such will build compliance into their AI operations before it becomes a crisis. The ones that wait for mandatory requirements will spend the next two years retrofitting controls they could have implemented today.
