AI Applied Scientist, AI Evaluation Science - Jobs - Careers at Apple
- Develop and adapt methods that raise the quality of the evaluations engineers run on LLMs and agentic systems.
- Build and improve the ML capabilities behind the team's developer-facing evaluation tools, such as rubric and test-set generation, automated judging, and failure analysis. Work with measurement scientists to integrate validity frameworks and psychometric methods, so the tools measure what they claim to measure and give developers the right signals.
- Develop ways for engineers to understand what their evaluation results mean and what to do next, such as uncertainty, likely failure causes, and when a result shouldn't be trusted. Partner with measurement scientists on how results are reported.
- Learn where engineers get stuck when evaluating AI features, and turn their recurring questions ("is this better?", "why did this fail?", "is it ready to ship?") into capabilities the tools can answer.
- Design and run large-scale experiments that test whether methods hold up on production data, including robustness to distribution shift across domains and languages, applying the team's statistical and validity standards for reliability and uncertainty.
- Prototype methods in well-structured Python, and partner with research and platform engineers to productionize them into evaluation SDKs, APIs, and services, owning the scientific correctness of what ships.
- When existing methods fall short, develop new ones with research and measurement scientists. Turn recurring gaps into open problems for the team's research agenda, and share findings through publications at top-tier venues and internal research reports.
- Communicate findings, limitations, and recommendations clearly to technical and non-technical audiences.
- Ph.D. in Computer Science, Machine Learning, Statistics, or a closely related field, or a Master's degree in one of these fields with 3+ years of applied research experience.
- Deep expertise in evaluating AI systems, gained through research on evaluation methods, substantial hands-on experience evaluating LLMs or agentic systems in real products, or both.
- Strong proficiency in Python and the modern ML ecosystem, with demonstrated ability to implement complex methods from recent papers and run large-scale experiments.
- Hands-on experience with large language models, whether fine-tuning, inference, prompting pipelines, or building LLM-powered applications.
- Demonstrated research ability through publications at peer-reviewed venues, preprints, or equivalent research output.
- Track record of applying research to practical problems: methods, prototypes, or tools adopted by other teams or used to inform real product decisions.
- Solid software engineering fundamentals for research code: version control, testing, reproducibility, and code others can build on.
- Excellent written and verbal communication skills, including the ability to explain complex results, and their limitations, to diverse audiences.
- Comfort with ambiguity and new problem spaces; you thrive when building something that doesn't yet have a playbook.
- First-author publications at top-tier venues (NeurIPS, ICML, ICLR, ACL, EMNLP), especially on evaluation methodology: papers about how to evaluate, not just papers that use evaluation to demonstrate model improvements.
- Specific experience in one or more of: LLM-as-judge approaches (calibration, rubric design, bias mitigation); benchmark design and validation (IRT, contamination detection); human evaluation methodology (protocol design, quality control); experimentation and comparisons between system versions (A/B testing, uncertainty quantification); or agentic and multi-agent system evaluation, including failure diagnosis and automated red-teaming.
- Familiarity with theoretical foundations of evaluation: measurement theory and validity frameworks, statistical learning theory (calibration, reliability, decision theory), causal inference, or preference aggregation.
- Experience with the practical challenges of evaluating generative AI and agentic systems: non-deterministic outputs, token economics, multi-turn agent trajectories, and tool use.
- Experience building methods or tools used directly by non-experts, such as developer-facing ML tooling or self-service analytics.
- Familiarity with modern evaluation tools and frameworks (e.g., DeepEval, Ragas, TruLens, LangSmith, or agent tracing and observability tools).
- Experience working with distributed computing or cloud-native tooling (e.g., Ray, Spark, Docker/Kubernetes) to run evaluations at scale.
- Experience with cross-disciplinary work, such as collaboration with social scientists, psychometricians, or domain experts.
- Experience in fast-moving, early-stage teams where you helped define technical direction from the ground up.
Apple is an equal opportunity employer that is committed to inclusion and diversity. We seek to promote equal opportunity for all applicants without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, Veteran status, or other legally protected characteristics. Learn more about your EEO rights as an applicant
At Apple, we believe accessibility is a fundamental human right. You’ll find that idea reflected in everything here — in our culture, our benefits and our digital tools. By welcoming as many perspectives as possible, we help you build a career where you feel like you belong.
Learn about accessibility in Apple’s workplace
Learn about reasonable accommodations for job applicants
Apple accepts applications to this posting on an ongoing basis.
Required Skills
Required Languages
🇬🇧 English