Software Engineer – LLM Evaluation & Repository Validation
Posted 2 days ago
You'll complete your application on Rex's career site.
About the Role
As a software engineer, you will focus on verifying authentic repositories used to benchmark large language models. Your objective is to guarantee that the datasets and environments employed for AI testing remain reliable, precise, and rigorous. By establishing reproducible environments and verifying the integrity of testing suites, you will have a direct impact on how accurately AI models are assessed against practical programming challenges. Your work will involve identifying technical problems and analyzing the proficiency of language models in resolving defects within open-source codebases. You must apply a critical lens to differentiate between superior test cases and those not fit for assessment. Through your exploration of intricate projects and the reporting of your results to research personnel, you will contribute significantly to the improvement of processes used to build and validate advanced AI systems.
Requirements
- Minimum of three years of work experience in software engineering.
- Advanced skills in one or more of the following: Python, C++, C#, Go, Rust, or Ruby.
- Proficiency with Docker, Git, and typical CI/CD pipeline structures.
- Capable of investigating and running code within sophisticated, real-world development environments.
- High level of written English proficiency for documentation and interaction.
- Reliable and precise individual who can operate autonomously from a remote location and adhere to project schedules.
- Previous exposure to developer tooling, open-source projects, or the evaluation of large language models is a major plus.
Responsibilities
- Oversee the complete lifecycle of readying open-source repositories from GitHub for benchmarking, encompassing container deployment and the automation of environment configuration.
- Review and categorize technical bugs while evaluating the robustness and coverage of unit tests.
- Run code in local environments to determine how well models address specific software defects.
- Support the process of choosing difficult repositories suitable for evaluation.
- Record your observations clearly for research staff, engage in alignment meetings, and integrate critique to guarantee uniform and high-quality results.
Benefits & Perks
- The pay rate ranges from USD $50 to $100 per hour, varying depending on your geographic location.
- This is a fully remote position.
Note: This listing is summarized and paraphrased by WorkNow from a publicly posted role. For the original wording and full details, please refer to the employer's official posting.
Similar Jobs
Senior Shopify Plus Developer
Hunt St
We are looking for a Senior Shopify Plus Developer to work remotely from the Philippines. The position centers on advanced storefront development, theme tailoring, and sophisticated integrations, with a commitment to maintaining rigorous technical excellence and meeting project requirements. You will handle diverse client demands, ranging from ongoing operational maintenance and retainer engagements to comprehensive project deployments. You will serve as a primary technical authority, guiding projects through the full lifecycle from early estimation to final launch validation. Your role involves generating reliable projections, addressing technical issues, and ensuring tasks align with established budgets and scopes. Success requires consistent communication with internal teams and leads to manage project hurdles and risks. This role suits an expert developer comfortable in a decentralized agency setting. You are encouraged to use advanced AI development tools as part of our forward-thinking strategy and contribute to internal R&D initiatives. The position blends deep technical execution with project oversight, demanding high-quality coding standards and transparent collaboration with clients and management.
Freelance Website Developer, Graphic Designer
Pramira
We are looking for a multi-talented freelance Web Developer and Graphic Designer to work remotely. This position suits a creative expert who can integrate technical web work with superior visual design. Your main tasks include managing our WordPress presence to keep the site fast, secure, and attractive, while also creating various marketing materials. You will assist multiple departments by delivering custom design solutions that match our standards in the telecommunications and engineering sectors. This role provides great flexibility, as you can organize your own hours within a freelance structure. You will partner with our team to run marketing initiatives, keep branding consistent across all channels, and complete various digital and print production assignments. We need someone who can work autonomously, communicate effectively, and document their hours, making this a great fit for a self-driven person interested in varied project work.
Senior Art Director
Construct Digital
This agency focuses on building data-backed brand stories that produce tangible commercial success. Through a mix of artistic flair and sophisticated technical solutions, the firm drives growth in sales, lead acquisition, and ROI. The service offering includes data analysis, UX design, content marketing, and modern AI integration to optimize engagement and performance. As a Senior Art Director, you will work alongside creative technologists, planners, and strategists to deliver impactful solutions for clients throughout the Asia-Pacific territory. This pivotal position requires a blend of aesthetic leadership, hands-on production, and strategic guidance within an environment that promotes experimentation and continuous workflow improvement.