Machine Learning Eval Engineer
AI summary of the role
This role builds the evaluation systems and benchmarks that drive model improvement at a fast-growing document intelligence API company.
No longer in the current catalog. Last included 2026-09-09. Check the employer’s posting for availability.
What you’ll do
- Design, build, and maintain evaluation benchmarks that reveal where our models perform well and where they fail.
- Develop metrics, heuristics, and workflows to automatically identify new failure modes across large and messy real-world datasets.
- Partner closely with other ML engineers to turn evaluation insights into model improvements and better training priorities.
- Work hands-on with unstructured enterprise data, including PDFs, spreadsheets, and other difficult document formats, to uncover edge cases and hard examples.
What you’ll bring
- Strong Python skills and ability to independently build clean, reliable technical solutions.
- Comfortable working with data infrastructure such as AWS S3 and OLAP or analytics systems like Tinybird.
- Experience getting hands-on with unstructured data and chasing down difficult failure cases.
- Ability to operate well in fast-changing, high-growth environments and collaborate across technical and non-technical teams.
Technologies
Python · Flask · AWS S3 · Tinybird · OLAP · vision models · LLMs · OCR
About Reducto
Reducto uses OCR plus vision-language models to convert complex unstructured documents into clean, LLM-ready inputs, sold as an enterprise API.
Series B · 50–200 people
Source and classification
Internal deployment & tooling · Evidence for this classification:
strong benchmarks, and create metrics and tooling that surface new failure modes as we scale. This is a high-impact role where you’ll help define how model quality is measured at Reducto and shape the systems we use to improve it. What You’ll Do Design, build, and maintain evaluation benchmarks that reveal where our models perform well and where they fail. Develop metrics, heuristics, and workflows to automatically identify new failure modes across large and messy real-world datasets. Partner closely with other ML engineers to turn evaluation insights into model improvements and better training priorities. Work hands-on with unstructured enterprise data, including PDFs, spreadsheets, and other difficult document formats, to uncover edge cases and hard examples. Build lightweight internal and user-facing tools, including simple interfaces in Python frameworks like Flask, to help
More from the job description
About Reducto Reducto is the agentic document platform for leading AI teams who demand enterprise performance at scale. We provide a comprehensive toolkit for working with documents the way a human would, combining custom in-house and leading frontier models to power efficient and accurate document workflows. We’ve grown rapidly, increasing revenue 8x year over year and partnering with hundreds of companies, from leading AI teams like Harvey, Vanta, and Scale, to enterprise customers across FAANG and top trading firms. Reducto has raised over $100M from world-class investors including a16z, Benchmark, and First Round Capital. The Opportunity As an ML Eval Engineer, you’ll play a key role in building the evaluation systems and benchmarks that make Reducto’s models better over time. You’ll collaborate closely with our ML, platform, and GTM teams to identify model weaknesses, design strong benchmarks, and create metrics and tooling that surface new failure modes as we scale. This is a high-impact role where you’ll help define how model quality is measured at Reducto and shape the systems we use to improve it. What You’ll Do Design, build, and maintain evaluation benchmarks that reveal where our models perform well and where they fail. Develop metrics, heuristics, and workflows to automatically identify new failure modes across large and messy real-world datasets. Partner [... source excerpt omitted ...] g tools, including simple interfaces in Python frameworks like Flask, to help teams inspect results, analyze model behavior, and communicate evaluation outcomes. Collaborate with customers and internal teams to understand real-world data needs and create bespoke benchmarks that highlight Reducto’s strengths. You’ll Thrive Here If You: Hold yourself to a high bar for quality and precision. Enjoy solving complex problems and building from first principles. Have strong Python skills and can independently build clean, reliable technical solutions. Bonus points for product and frontend experience! Are comfortable working with data infrastructure such as AWS S3 and OLAP or analyt [... source excerpt omitted ...] e at an early-stage or high-growth startup. Have some background in product thinking and can build simple, polished user-facing interfaces. Are comfortable working directly with customers to understand their workflows and data needs. Have experience in AI/ML, data infrastructure, enterprise software, or document understanding systems. Care deeply about combining technical excellence with business impact. This is an in person role at our office in SF. We’re an early stage company which means that the role requires working hard and moving quickly. Please only apply if that excites you. Benefits at Reducto At Reducto, we’re invested in the well-being and growth of our team. H
Employer postings · Data from · Sources