Requirements
Key Responsibilities
1. Engineering Problem Design
▪Develop original, self-contained engineering design problems with realistic technical requirements, competing constraints, and clearly defined optimization objectives.
▪Create challenging tasks that require AI models to interpret specifications, evaluate design trade-offs, configure simulation environments, and iteratively improve engineering solutions.
▪Establish validated reference solutions and objective automated grading mechanisms to assess correctness, performance, and compliance with engineering requirements.
2. Simulation and Computational Engineering
▪Build, configure, execute, and validate simulation-based problem environments using relevant open-source engineering tools.
▪Develop Python scripts, custom test benches, and supporting utilities to automate simulations and verify expected results.
▪Ensure that simulation outputs satisfy physical plausibility, dimensional consistency, boundary conditions, convergence requirements, and other domain-specific constraints.
▪Troubleshoot simulation failures, numerical instability, dependency issues, and computational errors.
3. AI Evaluation and Failure Analysis
▪Evaluate the performance of large language models (LLMs) and coding agents across repeated engineering problem-solving trials.
▪Analyze execution trajectories, tool interactions, simulator feedback, and intermediate outputs to identify systematic reasoning failures.
▪Investigate issues such as incorrect interpretation of engineering specifications, premature convergence, invalid geometries, inconsistent units, and physically impossible solutions.
▪Apply evaluation concepts such as pass@k, failure-mode classification, and analysis of nondeterministic model behavior.
4. Benchmark Development and Refinement
▪Calibrate task difficulty using empirical model performance and evaluation results.
▪Refine engineering problems to improve their ability to distinguish between successful and unsuccessful AI solutions.
▪Ensure that tasks remain unambiguous, technically rigorous, reproducible, and objectively gradable.
▪Collaborate with AI researchers, technical leads, and subject-matter experts to integrate validated benchmarks into AI evaluation workflows.
Required Qualifications and Skills
▪Master's degree or PhD in Electrical Engineering, Mechanical Engineering, Aerospace Engineering, or a closely related technical discipline.
▪At least 10 years of practical engineering design experience.
▪Strong Python programming and scripting skills.
▪Hands-on proficiency with at least one relevant open-source engineering simulation or design tool, such as:
▪Electrical simulation: ngspice, PySpice
▪Mechanical and computational engineering: OpenFOAM, FEniCSx, CalculiX
▪Control systems: python-control, OpenModelica
▪CAD and design automation: CadQuery, build123d
▪Thermodynamics and meshing: Cantera, Gmsh
▪Experience developing, executing, debugging, and validating simulation-based engineering workflows.
▪Familiarity with modern LLMs, AI coding agents, and AI evaluation methodologies.
▪Ability to analyze model execution logs and identify underlying reasoning, planning, and tool-use failures.
▪Strong understanding of physical constraints, units, boundary conditions, numerical convergence, and engineering validation.
▪Excellent analytical, problem-solving, and technical documentation skills.
▪Ability to work independently and deliver technically accurate results within defined timelines.
Preferred Qualifications
▪Experience in AI model evaluation, benchmark development, automated grading, or coding-agent assessment.
▪Familiarity with repeated-trial evaluation, pass@k metrics, failure taxonomies, and nondeterministic model behavior.
▪Experience building automated testing frameworks and reproducible computational environments.
▪Knowledge of optimization methods, numerical analysis, simulation automation, and engineering design validation.
▪Experience collaborating with multidisciplinary engineering and AI research teams.
Engineering Domains
Candidates should have strong expertise in one or more of the following areas:
▪Electrical Engineering
▪Mechanical Engineering
▪Aerospace Engineering
Work Commitment and Contract Terms
▪Expected commitment of approximately 40 hours per week.
▪At least 4 hours of daily overlap with US Pacific Time.
▪Weekend on-call availability is required; part-time engagement may be acceptable where agreed.
▪The engagement is contractor-based and may last up to 24 weeks.
▪Candidates must have a personal desktop or laptop and a stable, high-speed internet connection.
▪Compensation is USD 500 per task, with an estimated 30–40 hours of effort per task. Actual task allocation and payment terms should be confirmed before engagement.