Can an Agent Debug a Web Corpus? Agentic Auditing of Slovak FineWeb2
Marek Šuppa, Ivan Vykopal, Jaroslav Kopčan
WaC-13 @ EMNLP 2026 · Workshop, non-archival poster
Multilingual pipelines such as FineWeb2 apply one recipe across a thousand languages, leaving the failure modes of individual languages unanticipated. We use an LLM once, as a corpus debugger rather than a classifier: a tool-using agent explores the corpus through a read-only, budgeted API and compiles each finding into an executable test, which a native speaker then validates. On the 26.5M-document Slovak portion of FineWeb2, three runs costing $2.21-$2.84 each found seven findings over four phenomena. The largest is a machine-generated content farm of 1,408,437 documents (5.32%) that passed the pipeline's filters, 86.6% of it passing even stricter thresholds. Blinded labels support 6 of the 7 findings and do not support one. The compiled suite scans the full corpus in 0.7 s of wall-clock time without a GPU, against $12k-$90k per LLM classification pass. We release the framework, Slovak test suite, validation data and agent trajectories.