Applied Engineer, Evaluations Role

About the Role

We are seeking software engineers to help design and improve high-level assessment criteria for cutting-edge coding agents. In this role, you will help shape top-tier industry benchmarks by converting finished pull requests into structured engineering assignments. You will use a proprietary evaluation platform to measure the effectiveness of powerful large language models, such as Codex and Claude Code, within diverse development settings. Working in tandem with a research group, you will confirm that these benchmarks precisely demonstrate the strengths and constraints of current AI programming assistants. Success in this position demands a dedication to excellence as you advance the methods used to quantify machine intelligence in software engineering. Through diligent testing and continuous improvement, you will develop reliable and significant assessments that help determine how state-of-the-art models compare to human coding proficiency.

Requirements

• You must have robust software engineering foundations and extensive experience working within sophisticated, expansive codebases.
• You need proficiency in at least one of these programming languages: Python, TypeScript, JavaScript, Java, Kotlin, Go, Ruby, PHP, C++, or Rust.
• You must be capable of rapidly grasping new code, diagnosing nuanced behaviors, and evaluating whether different implementations resolve the intended problems.
• You should have experience developing high-quality tests that cover regression scenarios and edge cases, alongside comfort using CI tools, test runners, and Git.
• You need practical exposure to AI-powered coding tools and the ability to critically assess AI-generated results.
• You must demonstrate strong communication abilities, with the capacity to manage your own tasks and workflows while clearly documenting your insights and evaluation choices.

Responsibilities

• You will select appropriate code repositories and high-standard pull requests to act as the basis for rigorous testing scenarios.
• You will conduct deep-dive analyses into specific issues, their solutions, and the testing practices and architecture of the originating project.
• You will draft technical prompts and construct evaluations that incorporate shell commands, automated test suites, and LLM-based grading instructions.
• You will verify the accuracy of your evaluations against both correct solutions and problematic code to identify any flaws or insufficient criteria.
• You will execute multiple rounds of testing across various models to determine whether failures are caused by the agent, the system environment, or the evaluation logic itself.
• You will refine your benchmarks through ongoing assessment to effectively highlight the limitations in current AI agent capabilities.