Your team is evaluating AI coding agents. The vendor shows you impressive benchmark scores on SWE-Bench. You're ready to sign. But those benchmarks don't test what you actually need: the ability to refactor 50 files across your authentication layer or migrate your entire API from REST to gRPC.
Understanding the Scope
This guide is for security and DevOps engineers evaluating AI coding agents for refactoring work. You'll learn which capabilities current AI models lack, how to test them properly, and how to structure refactoring tasks so AI can actually help.
This isn't about abandoning AI tools. It's about using them where they work and compensating where they don't.
Key Concepts
Large-scale refactoring involves changes that span multiple files, preserve behavior, and require understanding how components interact. Examples include dependency updates, API migrations, and security pattern enforcement.
Cross-file context is the ability to track how a change in one module affects imports, type definitions, and function calls in other modules. Current AI models struggle here because they process files independently.
Benchmark gaming occurs when AI models optimize for test performance without developing the structural understanding needed for real work. Nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, allowing models to pass by exploiting test weaknesses rather than solving problems correctly.
SWE-Bench ProMax is a multilingual benchmark of 170 instances across seven programming languages, designed to test cross-file refactoring. The best model achieved only a 41.2% resolve rate, compared to 50-60% on traditional benchmarks.
What AI Models Can't Do Yet
Current AI coding agents fail at three specific tasks you probably need:
Dependency graph traversal: When you update a shared authentication function, the AI should identify every caller and update the signature consistently. Instead, it updates some files and misses others, leaving your codebase in a broken state.
Type system reasoning: Changing a return type from Optional<User> to Result<User, AuthError> requires updating every caller to handle the error case. AI models don't track these relationships reliably across files.
Test-driven validation: The AI should run tests after each change, detect failures, and backtrack. Most agents either skip validation or can't interpret test failures to guide corrections.
How to Test AI Capabilities Yourself
Don't rely on vendor benchmarks. Run these tests with your actual codebase:
Multi-file dependency update: Pick a library you use in 10+ files. Ask the AI to update it to the next major version. Check whether it updated all import statements, handled API changes consistently, and caught breaking changes in tests.
Security pattern enforcement: Ask the AI to add input validation to every API endpoint. Verify it identified all endpoints, applied the same validation logic, and didn't introduce new vulnerabilities like double-encoding.
Cross-module refactoring: Request a change that touches your data layer, business logic, and API layer. For example: "Add audit logging to all database writes." See if the AI maintains consistency across architectural boundaries.
If the AI completes these tasks without human intervention, you've found a capable tool. If it requires you to fix broken imports and type errors, you're still doing the hard work.
Implementation Guidance
Partition your refactoring work: AI handles single-file changes well. Break large refactors into isolated chunks the AI can complete independently, then review the integration points yourself.
Use AI for discovery, not execution: Ask the AI to identify all files that need changes. Review the list, then either make the changes manually or feed them to the AI one file at a time with explicit instructions.
Implement validation gates: After any AI-generated change, run your full test suite and static analysis. Don't merge until both pass. This catches the cross-file inconsistencies AI models miss.
Maintain a human review layer: For security-critical refactoring (authentication, authorization, cryptography), treat AI output as a first draft. A senior engineer should verify the logic, not just the syntax.
Common Pitfalls
Trusting benchmark scores: A model with 60% on SWE-Bench might achieve 20% on your actual refactoring tasks. The benchmark tests don't reflect cross-file complexity.
Skipping incremental validation: Running tests only at the end means you can't tell which AI-generated change broke your build. Validate after each file or module.
Assuming structural understanding: The AI doesn't "know" your architecture. It pattern-matches based on training data. If your codebase uses uncommon patterns, expect failures.
Over-scoping AI tasks: "Migrate our auth system to OAuth 2.1" is too broad. The AI will make inconsistent decisions across files. Scope down to "Update the token validation function" and expand from there.
Quick Reference Table
| Task Type | AI Capability | Your Action |
|---|---|---|
| Single-file bug fix | High | Let AI handle with review |
| Dependency version bump (breaking) | Low | Generate file list with AI, update manually |
| Security pattern enforcement | Medium | AI identifies locations, you verify logic |
| API migration (cross-module) | Low | Break into file-level tasks, validate incrementally |
| Type signature changes | Low | Manual or use language-specific refactoring tools |
| Test generation | Medium | AI writes tests, you verify coverage and edge cases |
| Code style consistency | High | Safe to automate with validation |
What This Means for Your Team
If you're evaluating AI coding agents, test them on your hardest refactoring tasks, not your easiest. The 41.2% resolve rate on SWE-Bench ProMax tells you these tools aren't ready to replace human judgment on complex changes.
Use AI where it's strong: generating boilerplate, suggesting fixes for isolated bugs, writing tests for well-defined behavior. For large-scale refactoring, you still need engineers who understand your system's structure.
The gap between benchmark performance and real-world capability matters for security work. An AI that passes tests by exploiting flawed test cases won't catch the edge cases that become vulnerabilities. Until models demonstrate reliable cross-file reasoning, keep a human in the loop for anything that touches authentication, authorization, or data validation.
