IAnversic EVAL
AI evaluation with measurable quality
An AI evaluation platform built to make the quality of AI applications measurable, enabling structured evaluations, model and prompt comparisons, and regression testing through reproducible runs.
Project overview
A quality-engineering platform that helps developers evaluate, compare and improve LLM-based applications through structured, reproducible experiments.
It combines deterministic checks with LLM-as-a-Judge to measure output quality against explicit criteria — covering datasets, model/prompt comparison, evaluation runs and regression detection.
Visit live productArchitecture
Design decisions
- Deterministic graders first, LLM judges only when needed.
- Bias calibration for LLM judges with explicit rubrics.
- Reproducibility through temperature 0 and versioned prompts.
Technical challenges
- Avoiding false negatives in structured evaluations.
- Aligning automated scores with human judgment.
- Telling real regressions from noise.
Test Evidence & Validation
Multiple layers of verification across evaluation execution, grading, and regression detection.
IAnversic EVAL combines automated testing, real-provider integration, and regression checks to validate reproducible LLM evaluation workflows.
01Automated Testing
Tests cover evaluation logic, graders, metrics, input validation, and run state transitions.
Evidence: test reports and execution logs.
02Evaluation Pipeline
End-to-end checks validate dataset execution, model invocation, grading, and result persistence using simulated and OpenAI providers.
Evidence: evaluation reports, grader results, and execution records.
03Regression Detection
Baseline comparisons apply configurable thresholds to quality scores, latency, cost, token usage, and failures. The CLI can fail CI/CD pipelines when regressions are detected.
Evidence: baseline comparisons, regression verdicts, and CLI exit codes.
04Reproducibility & Benchmarks
Immutable dataset and prompt versions enable traceable evaluation runs. The simulated 500-sample benchmark completes in approximately 2–4 seconds on a developer laptop; this measures platform execution with a fake provider, not real-model performance.
Evidence: run manifests, benchmark reports, and versioned evaluation results.
Current status & next steps
Validated with simulated and OpenAI providers, supported by automated tests and regression checks.
Next Steps: Run multi-model benchmarks, measure throughput, assess statistical significance, and expand into human, RAG, and agent evaluation, followed by AI observability.
Contact about this project