Applied Engineer, Evaluations Role
Posted 2 hours ago
You'll apply on the company's website.
About the Role
Mercor is seeking skilled software engineers to join the team as Applied Engineers in an evaluations-focused role. The work involves building and refining sophisticated benchmarks to measure the performance of frontier coding agents such as Claude Code and Codex. By collaborating closely with an in-house research team, you will help produce industry-leading benchmarks that push the boundaries of large language model capabilities. You will be responsible for turning completed pull requests into rigorous engineering tasks, ensuring that every evaluation accurately reflects the complexity of real-world software development. In this position, you will operate within a dynamic environment where you must understand unfamiliar codebases and investigate subtle behaviors to determine if various implementations effectively solve a given problem. You will be expected to work independently, taking ownership of the entire lifecycle of task selection, evaluation development, and iterative refinement. Success in this role requires a high degree of technical precision and the ability to maintain consistent, high-quality results while working with automated testing frameworks and LLM-based grading systems.
Requirements
- You need strong programming fundamentals and practical experience working within substantial and complex codebases.
- The team works across a wide range of languages including TypeScript, JavaScript, Python, Java, Kotlin, Go, Ruby, PHP, C++, and Rust.
- Candidates must possess the ability to quickly grasp new code, write meaningful tests that cover edge cases, and utilize Git, test runners, and CI tooling effectively.
- Practical experience with AI coding tools is essential, as is the judgment required to verify outputs and identify errors.
- Excellent written and verbal communication skills are necessary, along with the patience to engage in repeated investigation and refinement of your work.
Responsibilities
- Your primary responsibilities include identifying repositories that support challenging evaluations and selecting suitable completed pull requests.
- You will investigate repository architecture, tests, and conventions to develop task prompts using automated tests, shell commands, and LLM grading prompts.
- It is vital to validate these evaluations against reference solutions and flawed implementations to ensure accuracy.
- You will run evaluations across multiple models, investigate the root cause of failures, and document your grading decisions while incorporating feedback to continuously refine the quality and meaningfulness of the benchmarks produced.
Benefits & Perks
- The role includes comprehensive onboarding to the company's evaluation framework and ongoing professional review feedback to support your growth.
Note: This listing is summarized and paraphrased by WorkNow from a publicly posted role. For the original wording and full details, please refer to the employer's official posting.
Similar Jobs
Customer Service and Inside Sales Representative
CrewBloom
This remote staffing agency connects organizations with elite global professionals across diverse sectors. The organization simplifies hiring by identifying, vetting, and integrating candidates aligned with specific business operational requirements. Through global talent solutions, organizations can optimize efficiency, lower expenses, and remain compliant with HR standards. Your role contributes to these objectives by delivering exceptional client support and facilitating business growth through dedicated sales and service initiatives. You will join an inclusive, collaborative, and fast-paced environment that fosters innovation and values individual input. The position provides full remote flexibility, allowing you to work from anywhere and achieve a healthy work-life balance while eliminating daily travel.
Commercial Property Administrator
Boomering Inc
In this role, you function as a key support professional within the commercial real estate industry, focusing on the operational maintenance of property portfolios. You will handle documentation, client correspondence, and marketing materials, operating within an international, remote-first, and transparent team environment. This position suits a professional with an eye for detail who can manage administrative workflows alongside marketing responsibilities. You will utilize specialized property management software and digital tools to maintain accurate data and enhance property listing visibility, all while benefiting from a culture that emphasizes employee development and well-being.
Senior Salesforce Developer
DysrupIT
We are looking for a Senior Salesforce Developer based in the Philippines to work remotely. Your focus will be on maintaining, improving, and supporting sophisticated Salesforce platforms to provide secure, scalable solutions for enterprise customers. Operating within a fast-paced support and business-as-usual framework, you will partner with business analysts and onshore stakeholders to adhere to service level agreements and advance platform excellence. You will be instrumental in optimizing and stabilizing the Salesforce environment, with a specific focus on outcomes related to legacy migrations. By applying your architectural and development skills, you will support team success through technical documentation, high-standard coding, and the implementation of industry-standard practices at this global organization focused on long-term growth and skill development.