The problem
Changes to an AI operating framework need repeatable tests and evidence, not just informal prompts and impressions.
Prompt-driven tests are difficult to compare when the tested revision, prompt, and evidence are not recorded together.
The underlying question is not simply whether the project can perform a task, but how that task can remain understandable, repeatable, and safe as conditions change.
The solution
Keep versioned prompts, fixtures, runners, and captured evidence in a separate testing repository.
Keep prompts, fixtures, runners, and captured results under test/ai, and record immutable framework and test-source revisions with each run.
Use it to assess a framework revision across normal paths, edge cases, failures, and recovery before treating a change as validated.
Who it's for
Framework contributors and reviewers validating behavior before changes are accepted.
A practical fit is a reader or maintainer who wants a defined workflow and documented boundaries rather than an unexplained one-off operation.
Use it to assess a framework revision across normal paths, edge cases, failures, and recovery before treating a change as validated.
How it works
Key capabilities and concepts. See the project documentation or source code for implementation details.
Versioned prompts, fixtures, and test runners
Versioned prompts, fixtures, and test runners.
Keep prompts, fixtures, runners, and captured results under test/ai, and record immutable framework and test-source revisions with each run.
Normal, boundary, failure, and recovery test scenarios
Normal, boundary, failure, and recovery test scenarios.
Separating tests from the distributed framework adds repository boundaries but reduces the risk of coupling test scaffolding to user installations.
Durable evidence linked to exact test and framework revisions
Durable evidence linked to exact test and framework revisions.
Use it to assess a framework revision across normal paths, edge cases, failures, and recovery before treating a change as validated.
Design decisions that matter
- Separate test assets from the installable framework.
Separate test assets from the installable framework.
Separating tests from the distributed framework adds repository boundaries but reduces the risk of coupling test scaffolding to user installations.
- Tie outcomes to immutable source revisions.
Tie outcomes to immutable source revisions.
Prompt-driven tests are difficult to compare when the tested revision, prompt, and evidence are not recorded together.
- Include boundary, failure, and recovery scenarios.
Include boundary, failure, and recovery scenarios.
Keep prompts, fixtures, runners, and captured results under test/ai, and record immutable framework and test-source revisions with each run.
Explore the implementation
The testing repository records current fixtures, test methods, and published evidence.
Separating tests from the distributed framework adds repository boundaries but reduces the risk of coupling test scaffolding to user installations.
For exact current usage, prerequisites, and release status, consult the source repository.
Related articles
Engineering context and lessons behind this project.