Benchmark & Evaluation Engineer
Posted 23 days ago
You'll complete your application on Expedock's career site.
About the Role
This AI-driven automation provider seeks to revolutionize global supply chain operations through sophisticated data infrastructure. By automating manual logistics and freight data tasks with 99.99% accuracy, the organization plays a vital role in industry efficiency. As a Benchmark and Evaluation Engineer, you will facilitate this goal by developing and overseeing the frameworks that measure and improve machine learning outcomes. Your work will involve building scalable, enterprise-grade environments and comprehensive benchmark datasets that replicate real-world conditions, such as synthetic data scenarios, seasonal shifts, and operational noise. You will collaborate with logistics and AI professionals to maintain high standards of evaluation integrity as the automated platform expands into new sectors.
Requirements
- You need advanced proficiency in production-level SQL, specifically regarding window functions, CTEs, and query optimization.
- You must be highly skilled in Python, particularly with libraries such as NumPy, Polars, and pandas.
- You should have documented experience in data modeling with star and snowflake schemas, along with practical experience importing data into databases or warehouses like Postgres, BigQuery, Redshift, or Snowflake.
- You must show a history of building intricate benchmark tasks, confirming SQL solutions, and calibrating LLM evaluation judges.
- You are required to have experience managing large-scale semi-structured data and building realistic multi-system architectures.
- Background knowledge in healthcare, finance, or supply chain, or experience with LLM evaluation frameworks, is a strong advantage.
Responsibilities
- You will architect and deploy complex multi-system enterprise environments involving data lakes, warehouses, APIs, and databases.
- You will produce calibrated datasets for benchmarking that integrate realistic elements like drift, noise, and synthetic personally identifiable information.
- You will draft and validate benchmark assignments, manage verified SQL datasets, and build rubrics for reasoning assessments.
- You will identify and remove data leakage and ambiguity to ensure evaluation quality.
- You will upgrade grading mechanisms by implementing deterministic checks and LLM-based judges.
- You will manage scalability testing for queries and oversee leaderboards with regression tracking to guarantee evaluation precision.
Benefits & Perks
- This is a remote position located in the Philippines that follows a US-based work schedule.
Note: This listing is summarized and paraphrased by WorkNow from a publicly posted role. For the original wording and full details, please refer to the employer's official posting.
Similar Jobs
Senior Shopify Plus Developer
Hunt St
We are looking for a Senior Shopify Plus Developer to work remotely from the Philippines. The position centers on advanced storefront development, theme tailoring, and sophisticated integrations, with a commitment to maintaining rigorous technical excellence and meeting project requirements. You will handle diverse client demands, ranging from ongoing operational maintenance and retainer engagements to comprehensive project deployments. You will serve as a primary technical authority, guiding projects through the full lifecycle from early estimation to final launch validation. Your role involves generating reliable projections, addressing technical issues, and ensuring tasks align with established budgets and scopes. Success requires consistent communication with internal teams and leads to manage project hurdles and risks. This role suits an expert developer comfortable in a decentralized agency setting. You are encouraged to use advanced AI development tools as part of our forward-thinking strategy and contribute to internal R&D initiatives. The position blends deep technical execution with project oversight, demanding high-quality coding standards and transparent collaboration with clients and management.
Freelance Website Developer, Graphic Designer
Pramira
We are looking for a multi-talented freelance Web Developer and Graphic Designer to work remotely. This position suits a creative expert who can integrate technical web work with superior visual design. Your main tasks include managing our WordPress presence to keep the site fast, secure, and attractive, while also creating various marketing materials. You will assist multiple departments by delivering custom design solutions that match our standards in the telecommunications and engineering sectors. This role provides great flexibility, as you can organize your own hours within a freelance structure. You will partner with our team to run marketing initiatives, keep branding consistent across all channels, and complete various digital and print production assignments. We need someone who can work autonomously, communicate effectively, and document their hours, making this a great fit for a self-driven person interested in varied project work.
Senior Art Director
Construct Digital
This agency focuses on building data-backed brand stories that produce tangible commercial success. Through a mix of artistic flair and sophisticated technical solutions, the firm drives growth in sales, lead acquisition, and ROI. The service offering includes data analysis, UX design, content marketing, and modern AI integration to optimize engagement and performance. As a Senior Art Director, you will work alongside creative technologists, planners, and strategists to deliver impactful solutions for clients throughout the Asia-Pacific territory. This pivotal position requires a blend of aesthetic leadership, hands-on production, and strategic guidance within an environment that promotes experimentation and continuous workflow improvement.