Principal Agentic AI Operations Engineer

About the job

Claritev is seeking a Principal Agentic AI Operations Engineer to provide technical leadership for the operationalization, evaluation, governance, and reliable production performance of agentic AI systems powering the next generation of healthcare solutions.

In this role, you will define what high-quality, trustworthy AI looks like in complex healthcare workflows and build the evaluation frameworks, observability capabilities, operational tooling, and deployment standards that ensure agents perform safely, efficiently, and consistently at scale.

Working closely with Product, Engineering, AI Science, Security, and business leaders, you will establish the operational foundation for Claritev's agentic AI platform while influencing architecture, mentoring teams, and driving measurable improvements in reliability, quality, governance, and cost efficiency.

What You'll Do ****

• Lead the end-to-end operational lifecycle of production agentic AI systems, including deployment, rollout strategies, monitoring, incident response, governance, and continuous improvement.

• Design and implement evaluation frameworks, benchmarking platforms, golden datasets, simulation environments, regression testing, and online evaluation methodologies.

• Define, monitor, and optimize agent quality metrics including task completion, tool-call accuracy, grounding quality, hallucination rates, latency, safety, and operational costs.

• Build observability and distributed tracing solutions across LLM interactions, retrieval workflows, tool invocations, memory operations, and orchestration layers.

• Establish LLMOps and AgentOps best practices, including CI/CD pipelines, prompt and model versioning, deployment gates, drift detection, and incident management.

• Operate and optimize AI, agentic, and RAG workloads across cloud platforms, vector databases, enterprise data systems, and containerized environments.

• Implement guardrails, auditability, human-in-the-loop controls, and policy enforcement frameworks for responsible AI operations.

• Manage AI infrastructure utilization, throughput, capacity planning, model routing, caching strategies, and cost governance.

• Ensure compliance with privacy, HIPAA, PHI/PII protection, security, access control, and enterprise governance requirements.

• Develop reusable operational frameworks, runbooks, dashboards, and evaluation harnesses that enable efficient enterprise-wide adoption of agentic AI.

• Provide technical leadership and architectural guidance across complex cross-functional initiatives.

• Mentor engineers and data scientists while fostering a culture of operational excellence, measurement-driven decision making, and continuous innovation.

What You Will Bring ****

Required Qualifications ****

• Bachelor's degree in Computer Science, Engineering, Data Science, a quantitative discipline, or a related field required.

• 10+ years of experience in software engineering, ML engineering, platform engineering, SRE, or a related technical discipline.

• 5+ years operating production AI or machine learning systems in MLOps, LLMOps, or ML platform environments.

• 3+ years of experience with Generative AI, LLMs, RAG architectures, and agentic AI systems in production.

• Experience building evaluation and benchmarking capabilities for LLM and agentic systems with measurable quality outcomes.

• Proven success leading complex technical initiatives from concept through production deployment and business impact.

• Expert-level Python skills and experience designing scalable services, APIs, and distributed systems.

• Deep expertise in AI evaluation methodologies including evaluation harnesses, golden datasets, LLM-as-judge frameworks, statistical testing, and regression detection.

• Experience with observability and evaluation platforms such as LangSmith, Langfuse, Arize Phoenix, Ragas, DeepEval, Braintrust, OpenAI Evals, or equivalent tools.

• Experience with agentic AI frameworks including LangGraph, LangChain, AutoGen, CrewAI, and related orchestration patterns.

• Experience with cloud environments, large-scale data platforms, vector databases, embeddings, and RAG systems.

• Experience implementing MLOps and LLMOps practices including CI/CD, deployment strategies, experimentation frameworks, monitoring, and troubleshooting.

• Experience with Kubernetes, containerization, Terraform, and infrastructure-as-code practices.

• Working knowledge of deep-learning frameworks such as PyTorch or TensorFlow.

• Strong leadership, communication, organizational, and problem-solving skills.

Preferred Qualifications ****

• Master's degree or PhD preferred.

• Experience with Oracle Cloud Infrastructure (OCI), including Generative AI, Data Science, Database, and Observability services.

• Healthcare, insurance, claims, payment integrity, healthcare technology, or other regulated-industry experience.

• Experience operating AI systems that process PHI or PII within HIPAA-regulated environments.

• Experience with AI red-teaming, adversarial testing, safety evaluation, and responsible AI governance.

• Experience driving process automation and enterprise workflow integration.

• Site Reliability Engineering experience including SLOs, SLIs, error budgets, and reliability practices for AI systems.

Compensation ****

The salary range for this position is $165,000 - $185,000. Actual compensation is based on experience, skills, education, work location, and internal equity. This position may also be eligible for incentive compensation and the comprehensive benefits outlined in the My Total Value section.

#LI-MC2

About Us

Why Claritev?

Healthcare is complex. We help make it clearer.

At Claritev, you'll do work that matters. Together, we're helping make healthcare more transparent and affordable for all through the power of data, technology, and expertise. We offer meaningful opportunities to grow your career, collaborate with talented colleagues, and make an impact on the clients and communities we serve. If you're looking for purpose, growth, and a team that succeeds together, you'll find it here.

What Guides Us

At Claritev, innovation, agility, and a focus on results drive our success. We embrace bold thinking, work as one team, take ownership, and strive for excellence in everything we do — creating meaningful impact for our clients, communities, and each other.

My Total Value

Claritev’s global total rewards philosophy — My Total Value — is grounded in investing in our associates and rewarding performance. This includes:

• Competitive compensation and incentive opportunities (where eligible)

• Medical, dental, and vision coverage

• Life and disability coverage

• 401(k) with company match

• Employee stock purchase plan

• Flexible spending accounts and health savings accounts

• Paid time off and company holidays

• Paid parental leave

• Mental health resources

• Tuition reimbursement

• Financial planning resources

• Professional development opportunities

Our Commitment to Inclusion

At Claritev, we believe diverse perspectives strengthen our teams and lead to better outcomes. We are committed to fostering an inclusive workplace where every associate feels respected, valued, and empowered to succeed.

Claritev is an Equal Opportunity Employer and complies with all applicable laws and regulations. Qualified applicants will receive consideration for employment without regard to age, race, color, religion, sex, sexual orientation, gender identity, national origin, disability, protected veteran status, or any other characteristic protected by law.

Additional Information

• Job Identification: 399
• Job Category: AI
• Posting Date: 2026-09-02
• Job Schedule: Full time
• Locations: 7900 Tysons Corner Pl., McLean, VA, 22102, US (Remote)