LLM Evaluation Mini-Hack: build better benchmarks
A morning hack where the RMAIIG AI Engineering group built its own open-source LLM benchmark and evaluation suite.
The RMAIIG AI Engineering group spent a Thursday morning building our own open-source LLM benchmark and evaluation suite, testing for the things we thought were fun, interesting, and worth measuring. I brought the coffee.
The tools we started from live in the RMAIIG-Eval repo.