AI Assurance Engineer - MLE Focus
TikTok USDS Joint Venture
3 hours ago
Full-time
On-site
San Jose, California, United States
$187,040 - $438,000 USD yearly
Engineer
Responsibilities
About the Team
Detection Engineering is part of Content Assurance and is aligned with USDS JV Security & Privacy. The team provides product-independent technical assurance for recommender and generative AI systems, testing whether system behavior remains within declared safety, quality, and governance boundaries before and after release.
AI Assurance Engineering builds and operates shared evaluation infrastructure, including test harnesses, benchmark pipelines, replay systems, adversarial-testing tools, safety-metric telemetry, and assurance CI/CD. The team is independent from the product teams whose models it evaluates.
About the Role
As an AI Assurance Engineer, you will design rigorous evaluation methods and production-quality capabilities for measuring and diagnosing recommender and generative AI behavior. You will develop benchmark datasets, define reproducible test protocols, implement safety, quality, robustness, and drift metrics, and investigate behavioral regressions through adversarial testing and counterfactual replay. This role focuses on evaluation engineering and technical assurance, not product model training, policy setting, moderation, or release ownership.
Responsibilities
- Evaluation Methods and Benchmarks: Define reproducible protocols, benchmark suites, sampling strategies, datasets, and statistical analyses for recommender and generative AI evaluations.
- Behavioral Testing and Diagnosis: Test safety, ranking quality, consistency, adversarial robustness, refusal behavior, leakage, factuality within scope, drift, and regressions.
- Evaluation Platforms and Signals: Implement evaluation code, metrics, replay workflows, and assurance signals on shared infrastructure while preserving traceability and repeatability.
- Design Evaluation Methodologies: Define measurable evaluation objectives, metrics, sampling plans, baselines, acceptance criteria, and reproducible protocols for pre-release and post-release assurance of recommender and generative AI systems.
- Engineer Benchmark Datasets: Curate, version, document, and quality-check benchmark and adversarial datasets, including harm-exposure samples, ranking-quality cases, jailbreak prompts, leakage tests, and counterfactual replay inputs.
- Implement Behavioral Tests: Build evaluation code for recommender objectives and GenAI behavior, including safety-category coverage, ranking quality, fairness and consistency, refusal correctness, factuality within scope, adversarial robustness, provenance signals, and drift detection.
- Diagnose Evaluation Results: Apply statistical and experimental rigor to distinguish model-behavior regressions from data, sampling, or infrastructure effects; analyze post-release drift against release-time baselines and produce traceable technical findings.
- Contribute Production-Quality Platforms: Integrate benchmarks, metrics, replay tests, and red-team assets with shared evaluation runtimes and assurance CI/CD; improve reproducibility, telemetry quality, and failure diagnosis without assuming ownership of the underlying platform methodology validation.
Qualifications
Minimum Qualifications:
- Education: Bachelor degree or higher in Computer Science, Machine Learning, Statistics, or a related technical field, or equivalent practical experience.
- Machine Learning Evaluation: Experience evaluating machine learning systems with clearly defined metrics, datasets, baselines, and reproducible test protocols.
- Software Engineering: Experience developing production-quality evaluation or data-processing software in Python, Go, C++, or a comparable language.
- Statistical and Experimental Rigor: Strong knowledge of sampling, hypothesis testing, uncertainty, experiment design, error analysis, and regression diagnosis.
Preferred Qualifications:
- Recommender-System Evaluation: Experience with ranking metrics, counterfactual replay, harm-exposure sampling, cohort consistency, or ranking-quality benchmarks.
- Generative AI Evaluation: Experience evaluating refusal behavior, jailbreak resistance, leakage, factuality within scope, provenance, or multimodal generative systems.
- Adversarial and Robustness Testing: Experience designing red-team tests, adversarial datasets, robustness analyses, or continuous post-release evaluations.
- Evaluation Infrastructure: Experience integrating benchmarks and metrics with test harnesses, replay pipelines, experiment systems, or CI/CD workflows.
- Cross-Functional Assurance: Ability to communicate evidence, limitations, and technical findings clearly to product engineering, Trust & Safety, business owners, and governance partners while maintaining ownership boundaries.
About USDS
TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. The Joint Venture has been established in compliance with the Executive Order signed by President Trump on September 25, 2025. Our foundation is a comprehensive data privacy and cybersecurity program we operate under defined safeguards to protect national security and secure U.S. user data, apps and the algorithm. We safeguard the U.S. content ecosystem, holding decision-making authority for trust and safety policies and moderation. USDS Joint Venture helps ensure Americans can continue to express their creativity, discover new hobbies and interests, and build thriving communities and businesses on a global scale.
On-site presence across teams allows the company to operate with greater speed, alignment, and agility — especially in areas like real-time decision-making, team development, and integrated execution. As such, the company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week.
Why Join Us
Inspiring creativity is at the core of TikTok's mission. Our innovative product is built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and bring joy - a mission we work towards every day.
We strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. Every challenge is an opportunity to learn and innovate as one team. We're resilient and embrace challenges as they come. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our company, and our users. When we create and grow together, the possibilities are limitless. Join us.
Diversity & Inclusion
TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.
USDS Reasonable Accommodation
USDS is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://tinyurl.com/USDS-RA
Job Information
【For Pay Transparency】Compensation Description (Annually)
The base salary range for this position in the selected city is $187040 - $438000 annually.
Compensation may vary outside of this range depending on a number of factors, including a candidate’s qualifications, skills, competencies and experience, and location. Base pay is one part of the Total Package that is provided to compensate and recognize employees for their work, and this role may be eligible for additional discretionary bonuses/incentives, and restricted stock units.
Benefits may vary depending on the nature of employment and the country work location. Employees have day one access to medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short-term and long-term disability coverage, life insurance, wellbeing benefits, among others. Employees also receive 10 paid holidays per year, 10 paid sick days per year and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).
The Company reserves the right to modify or change these benefits programs at any time, with or without notice.
For Los Angeles County (unincorporated) Candidates:
Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state, and local laws including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Our company believes that criminal history may have a direct, adverse and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment:
1. Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues;
2. Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems; and
3. Exercising sound judgment.
About the Team
Detection Engineering is part of Content Assurance and is aligned with USDS JV Security & Privacy. The team provides product-independent technical assurance for recommender and generative AI systems, testing whether system behavior remains within declared safety, quality, and governance boundaries before and after release.
AI Assurance Engineering builds and operates shared evaluation infrastructure, including test harnesses, benchmark pipelines, replay systems, adversarial-testing tools, safety-metric telemetry, and assurance CI/CD. The team is independent from the product teams whose models it evaluates.
About the Role
As an AI Assurance Engineer, you will design rigorous evaluation methods and production-quality capabilities for measuring and diagnosing recommender and generative AI behavior. You will develop benchmark datasets, define reproducible test protocols, implement safety, quality, robustness, and drift metrics, and investigate behavioral regressions through adversarial testing and counterfactual replay. This role focuses on evaluation engineering and technical assurance, not product model training, policy setting, moderation, or release ownership.
Responsibilities
- Evaluation Methods and Benchmarks: Define reproducible protocols, benchmark suites, sampling strategies, datasets, and statistical analyses for recommender and generative AI evaluations.
- Behavioral Testing and Diagnosis: Test safety, ranking quality, consistency, adversarial robustness, refusal behavior, leakage, factuality within scope, drift, and regressions.
- Evaluation Platforms and Signals: Implement evaluation code, metrics, replay workflows, and assurance signals on shared infrastructure while preserving traceability and repeatability.
- Design Evaluation Methodologies: Define measurable evaluation objectives, metrics, sampling plans, baselines, acceptance criteria, and reproducible protocols for pre-release and post-release assurance of recommender and generative AI systems.
- Engineer Benchmark Datasets: Curate, version, document, and quality-check benchmark and adversarial datasets, including harm-exposure samples, ranking-quality cases, jailbreak prompts, leakage tests, and counterfactual replay inputs.
- Implement Behavioral Tests: Build evaluation code for recommender objectives and GenAI behavior, including safety-category coverage, ranking quality, fairness and consistency, refusal correctness, factuality within scope, adversarial robustness, provenance signals, and drift detection.
- Diagnose Evaluation Results: Apply statistical and experimental rigor to distinguish model-behavior regressions from data, sampling, or infrastructure effects; analyze post-release drift against release-time baselines and produce traceable technical findings.
- Contribute Production-Quality Platforms: Integrate benchmarks, metrics, replay tests, and red-team assets with shared evaluation runtimes and assurance CI/CD; improve reproducibility, telemetry quality, and failure diagnosis without assuming ownership of the underlying platform methodology validation.
Qualifications
Minimum Qualifications:
- Education: Bachelor degree or higher in Computer Science, Machine Learning, Statistics, or a related technical field, or equivalent practical experience.
- Machine Learning Evaluation: Experience evaluating machine learning systems with clearly defined metrics, datasets, baselines, and reproducible test protocols.
- Software Engineering: Experience developing production-quality evaluation or data-processing software in Python, Go, C++, or a comparable language.
- Statistical and Experimental Rigor: Strong knowledge of sampling, hypothesis testing, uncertainty, experiment design, error analysis, and regression diagnosis.
Preferred Qualifications:
- Recommender-System Evaluation: Experience with ranking metrics, counterfactual replay, harm-exposure sampling, cohort consistency, or ranking-quality benchmarks.
- Generative AI Evaluation: Experience evaluating refusal behavior, jailbreak resistance, leakage, factuality within scope, provenance, or multimodal generative systems.
- Adversarial and Robustness Testing: Experience designing red-team tests, adversarial datasets, robustness analyses, or continuous post-release evaluations.
- Evaluation Infrastructure: Experience integrating benchmarks and metrics with test harnesses, replay pipelines, experiment systems, or CI/CD workflows.
- Cross-Functional Assurance: Ability to communicate evidence, limitations, and technical findings clearly to product engineering, Trust & Safety, business owners, and governance partners while maintaining ownership boundaries.
About USDS
TikTok USDS Joint Venture LLC is dedicated to the safety and security of millions of Americans who create, discover, and connect with what they love on the apps we operate. The Joint Venture has been established in compliance with the Executive Order signed by President Trump on September 25, 2025. Our foundation is a comprehensive data privacy and cybersecurity program we operate under defined safeguards to protect national security and secure U.S. user data, apps and the algorithm. We safeguard the U.S. content ecosystem, holding decision-making authority for trust and safety policies and moderation. USDS Joint Venture helps ensure Americans can continue to express their creativity, discover new hobbies and interests, and build thriving communities and businesses on a global scale.
On-site presence across teams allows the company to operate with greater speed, alignment, and agility — especially in areas like real-time decision-making, team development, and integrated execution. As such, the company is shifting from a hybrid work model to a fully in-person schedule up to 5 days a week.
Why Join Us
Inspiring creativity is at the core of TikTok's mission. Our innovative product is built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and bring joy - a mission we work towards every day.
We strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. Every challenge is an opportunity to learn and innovate as one team. We're resilient and embrace challenges as they come. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our company, and our users. When we create and grow together, the possibilities are limitless. Join us.
Diversity & Inclusion
TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.
USDS Reasonable Accommodation
USDS is committed to providing reasonable accommodations in our recruitment processes for candidates with disabilities, pregnancy, sincerely held religious beliefs or other reasons protected by applicable laws. If you need assistance or a reasonable accommodation, please reach out to us at https://tinyurl.com/USDS-RA
Job Information
【For Pay Transparency】Compensation Description (Annually)
The base salary range for this position in the selected city is $187040 - $438000 annually.
Compensation may vary outside of this range depending on a number of factors, including a candidate’s qualifications, skills, competencies and experience, and location. Base pay is one part of the Total Package that is provided to compensate and recognize employees for their work, and this role may be eligible for additional discretionary bonuses/incentives, and restricted stock units.
Benefits may vary depending on the nature of employment and the country work location. Employees have day one access to medical, dental, and vision insurance, a 401(k) savings plan with company match, paid parental leave, short-term and long-term disability coverage, life insurance, wellbeing benefits, among others. Employees also receive 10 paid holidays per year, 10 paid sick days per year and 17 days of Paid Personal Time (prorated upon hire with increasing accruals by tenure).
The Company reserves the right to modify or change these benefits programs at any time, with or without notice.
For Los Angeles County (unincorporated) Candidates:
Qualified applicants with arrest or conviction records will be considered for employment in accordance with all federal, state, and local laws including the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act. Our company believes that criminal history may have a direct, adverse and negative relationship on the following job duties, potentially resulting in the withdrawal of the conditional offer of employment:
1. Interacting and occasionally having unsupervised contact with internal/external clients and/or colleagues;
2. Appropriately handling and managing confidential information including proprietary and trade secret information and access to information technology systems; and
3. Exercising sound judgment.