Documentation

How TestyAI works

TestyAI is an AI-powered web QA platform that transforms natural language specifications into verified test evidence. The system decomposes complex test scenarios into atomic subtasks, executes them through browser automation, and validates outcomes against expected results.

Each test case follows a structured workflow: the LLM agent plans the steps needed to satisfy the requirement, executes those steps through Playwright browser controls, collects evidence at each checkpoint, and reports success or failure with actionable artifacts.

Governed execution and versioned plans

Test plans are versioned and validated before reuse. When a new plan is generated from a feature description, it is recorded as Draft. The system then attempts a replay validation run—executing the plan without LLM involvement—to confirm the steps are deterministic and reproducible.

If replay succeeds, the plan is promoted to Validated status and becomes the canonical execution path for future runs. If replay fails, the plan is regenerated automatically, and the previous version is soft-deleted. This ensures that only deterministic, replay-safe plans are reused, reducing LLM cost and execution variability over time.

When test cases have dependencies or lifecycle requirements (setup, cleanup), the execution engine coordinates those phases automatically. The lifecycle flow is: Setup → Dependencies → Main → Verification → Cleanup. Quality gates monitor execution for repeated failures, zero progress, or strict mode violations, and bounded recovery strategies can replan limited subtask ranges when failures occur.

Replay validation and evidence artifacts

Every execution generates structured evidence that survives beyond the test run:

• result.json — structured execution result with status, duration, and error details
• llm-conversation.md — full reasoning and tool call sequence from the LLM agent
• Screenshots — timestamped browser screenshots captured at key execution points
• Screenshot manifest — JSON index of all screenshots with metadata for post-run review

These artifacts are stored per execution and retrievable through the API, making it possible to audit test behavior, debug failures, and provide compliance evidence tied to the original requirement.

CI/CD-ready suite runs

TestyAI exposes REST API endpoints for integration with CI/CD pipelines:

• Trigger a suite or feature-level run with optional environment selection
• Poll for completion status until all test cases finish
• Retrieve JUnit XML reports for pipeline-native test result consumption
• Cancel active runs if a deployment is aborted or a pipeline is interrupted

All CI/CD endpoints require API key authentication scoped to company context. This makes TestyAI suitable for automated quality gates in Jenkins, GitHub Actions, GitLab CI, or similar orchestration systems.

LLM providers, environments, and variables

TestyAI supports multiple LLM providers (Azure OpenAI, DeepSeek) with runtime selection by company or explicit override. Execution runs can target different environments (staging, production, QA) within the same project, and variable overrides per environment let you configure base URLs, credentials, or test data scoped to that environment.

Variables use placeholder syntax ({{variableName}}) in test case definitions, and secret values are masked in logs and artifacts. Project-level variables provide defaults, and environment-specific overrides apply at runtime based on the selected environment for the test run.

Who it's for

TestyAI is designed for engineering teams who need:

• Faster test coverage for complex web workflows without manual script maintenance
• Verification evidence tied to natural language requirements
• Automated regression checks integrated into CI/CD pipelines
• Real-time visibility into test execution and failure root cause

Next steps

This overview helps you understand the product surface and core capabilities. For more technical details about execution lifecycle, implementation patterns, and pilot preparation, see Read the Docs.