Applied Engineer, Evaluations Role
Posted 1 day ago
You'll complete your application on Mercor's career site.
About the Role
We are seeking software engineers to serve as Applied Engineers within our evaluation division. Your primary focus involves constructing and enhancing advanced systems designed to stress-test cutting-edge autonomous coding agents. You will convert finished pull requests into demanding technical assignments and employ our proprietary assessment framework to gauge how different large language models perform. By working alongside our research staff, you will help create top-tier benchmarks that reliably differentiate the capabilities of modern AI technologies. You will dive deeply into intricate software projects to guarantee your assessments are precise and impactful. Acting with autonomy, you will choose relevant tasks, analyze existing solutions, and create automated testing suites that accurately quantify agent proficiency. Success in this position demands an analytical perspective to ensure your testing environments are reliable and your scoring logic perfectly captures the caliber of the agent's work. You will also refine these benchmarks through consistent trial, error, and critical analysis of the outcomes.
Requirements
- You need a solid foundation in programming and demonstrated experience working within large, intricate codebases using languages like Python, TypeScript, Java, Rust, Go, or similar.
- You must demonstrate the capability to navigate unfamiliar code, detect nuanced behaviors, and determine if various solutions are technically correct.
- It is essential that you have experience designing high-quality tests that account for edge cases.
- Expertise in using Git, CI tools, and test runners is required.
- You should feel comfortable leveraging AI coding assistants while exercising the necessary critical thinking to validate their work.
- Effective communication skills are vital for managing your own projects and seeking clarification when instructions are unclear.
Responsibilities
- You will locate repositories containing significant technical contributions and pick appropriate pull requests for the assessment process.
- Your tasks include studying project structures and existing solutions to craft precise task prompts, alongside creating assessments that utilize shell commands and LLM-based grading instructions.
- You will verify your testing criteria by comparing them against standard solutions and broken code to ensure accurate scoring.
- Additionally, you will execute evaluations across various models, troubleshoot performance issues, and improve your processes through continuous iterations.
- You are also expected to record your observations and apply feedback to enhance the overall effectiveness of the benchmarking system.
Benefits & Perks
- You will receive thorough training on our internal evaluation framework, coupled with continuous professional guidance to foster your growth.
Note: This listing is summarized and paraphrased by WorkNow from a publicly posted role. For the original wording and full details, please refer to the employer's official posting.
Similar Jobs
Senior Shopify Plus Developer
Hunt St
We are looking for a Senior Shopify Plus Developer to work remotely from the Philippines. The position centers on advanced storefront development, theme tailoring, and sophisticated integrations, with a commitment to maintaining rigorous technical excellence and meeting project requirements. You will handle diverse client demands, ranging from ongoing operational maintenance and retainer engagements to comprehensive project deployments. You will serve as a primary technical authority, guiding projects through the full lifecycle from early estimation to final launch validation. Your role involves generating reliable projections, addressing technical issues, and ensuring tasks align with established budgets and scopes. Success requires consistent communication with internal teams and leads to manage project hurdles and risks. This role suits an expert developer comfortable in a decentralized agency setting. You are encouraged to use advanced AI development tools as part of our forward-thinking strategy and contribute to internal R&D initiatives. The position blends deep technical execution with project oversight, demanding high-quality coding standards and transparent collaboration with clients and management.
Freelance Website Developer, Graphic Designer
Pramira
We are looking for a multi-talented freelance Web Developer and Graphic Designer to work remotely. This position suits a creative expert who can integrate technical web work with superior visual design. Your main tasks include managing our WordPress presence to keep the site fast, secure, and attractive, while also creating various marketing materials. You will assist multiple departments by delivering custom design solutions that match our standards in the telecommunications and engineering sectors. This role provides great flexibility, as you can organize your own hours within a freelance structure. You will partner with our team to run marketing initiatives, keep branding consistent across all channels, and complete various digital and print production assignments. We need someone who can work autonomously, communicate effectively, and document their hours, making this a great fit for a self-driven person interested in varied project work.
Senior Art Director
Construct Digital
This agency focuses on building data-backed brand stories that produce tangible commercial success. Through a mix of artistic flair and sophisticated technical solutions, the firm drives growth in sales, lead acquisition, and ROI. The service offering includes data analysis, UX design, content marketing, and modern AI integration to optimize engagement and performance. As a Senior Art Director, you will work alongside creative technologists, planners, and strategists to deliver impactful solutions for clients throughout the Asia-Pacific territory. This pivotal position requires a blend of aesthetic leadership, hands-on production, and strategic guidance within an environment that promotes experimentation and continuous workflow improvement.