Principal Software Engineering - Reliability and FinOps Engineer

About the job

This is a full-time onsite role based in the location(s) posted and is performed from the designated work location five days per week. No relocation is offered. ****

AT&T will not hire any applicants for this position who require employer sponsorship now or in the future.

Join AT&T and reimagine the communications and technologies that connect the world. The Technology Services organization is responsible for advancing information technology performance and delivering solutions with a focus on maximizing ROI, increasing efficiency and enhancing the experience of end users. Guided by experienced leaders, Corporate Systems seamlessly integrate with advanced Technology and Operations to drive our enterprise forward. Our Systems Reliability and Software Delivery teams are unwavering in their commitment to excellence, ensuring every solution is robust and efficient. When you step into a career with AT&T, you won't just imagine the future, you'll create it.

About The Team

AT&T Technology Services is modernizing the technology ecosystem through secure, scalable, cloud-enabled, and AI-powered platforms. Within Reliability & FinOps Engineering, our purpose is to elevate the customer experience by strengthening the reliability and resiliency of critical systems, maximizing the value of technology investments, and accelerating responsible automation.

Our organization brings together reliability engineering, FinOps and infrastructure optimization, AI and automation enablement, and customer-centric business intelligence. We build reusable engineering patterns, policy-driven controls, self-service capabilities, and intelligent optimization services that improve operational outcomes at enterprise scale.

What You'll Do

As a Principal System Engineer, you will serve as a senior technical leader across FinOps, cloud optimization, system reliability, and AI-driven automation. You will shape and execute enterprise strategies that improve the reliability of critical services, reduce cloud waste, strengthen financial accountability, and automate operational workflows across Azure, AWS, Kubernetes, and hybrid-cloud environments.

In this role, you will translate business priorities into durable technical solutions, influence cross-organizational roadmaps, and move optimization opportunities from insight through governed execution and measurable value realization.

Key responsibilities include:

• Provide principal-level technical leadership for resilient, secure, scalable, and cost-efficient platforms across Azure, AWS, Kubernetes, and hybrid-cloud environments.

• Lead FinOps and infrastructure optimization capabilities, including cost allocation, showback, anomaly detection, forecasting, rightsizing, workload scheduling, commitment optimization, waste reduction, and validated savings realization.

• Develop policy-driven, closed-loop optimization workflows that move recommendations through identification, application-owner engagement, approval, ServiceNow change control, execution, validation, evidence capture, and reporting.

• Apply an automation-by-default approach using Python, PowerShell, Bash, REST APIs, Azure Functions, Power Automate, and event-driven services to reduce manual handoffs and operational toil.

• Design practical generative AI, AI-fueled coding, and agentic automation solutions for engineering operations, cost analysis, anomaly investigation, remediation guidance, executive reporting, knowledge retrieval, and decision support.

• Establish cloud-governance guardrails, policy-as-code controls, exception processes, auditability, and risk-based automation patterns aligned with security, architecture, compliance, and change-management requirements.

• Engineer secure identity and authorization models using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, and time-bound permissions.

• Integrate optimization and automation capabilities with Terraform, GitHub, Azure DevOps, CI/CD pipelines, ServiceNow, monitoring platforms, cost-management tools, and enterprise data sources while preventing configuration drift.

• Advance reliability engineering practices, including observability, service-level objectives, capacity management, performance analysis, incident prevention, operational readiness, post-change validation, rollback planning, and continuous improvement.

• Build executive dashboards and operational scorecards that communicate cloud spend, reliability, automation coverage, optimization pipeline health, risk, realized savings, and customer or business outcomes.

• Lead proofs of concept, production readiness, phased adoption, and complex technical issue resolution in partnership with internal teams and strategic vendors.

• Mentor engineers, create reusable reference architectures and standards, and influence priorities across Reliability, FinOps, Security, Architecture, Finance, Operations, and application teams.

• Demonstrate AT&T leadership behaviors by serving customers first, moving faster, acting boldly, winning as one, and developing talent.

What You'll Need

• 7+ years of progressive experience in systems engineering, cloud engineering, site reliability engineering, platform engineering, DevOps, FinOps, or enterprise automation.

• Demonstrated ability to lead complex, cross-functional engineering initiatives and influence technical direction without relying solely on direct authority.

• Deep hands-on experience with Azure and/or AWS architecture, networking, compute, storage, identity, monitoring, governance, and cost-management services.

• Strong knowledge of FinOps practices, including allocation, tagging, showback or chargeback, budgeting, forecasting, unit economics, commitment management, anomaly detection, optimization, and savings validation.

• Experience implementing cloud optimization for virtual machines, storage, databases, platform services, Kubernetes, and idle or underutilized resources.

• Advanced scripting and automation experience using Python, PowerShell, Bash, REST APIs, JSON, and event-driven integration patterns.

• Strong Infrastructure as Code and CI/CD experience using Terraform, GitHub Actions, Azure DevOps, or comparable enterprise tooling.

• Hands-on experience with Kubernetes, AKS, EKS, or OpenShift, including workload rightsizing, Helm, observability, node lifecycle management, and container cost allocation.

• Experience applying generative AI, prompt engineering, AI assistants, AI-fueled coding, or agentic automation to engineering and operational use cases.

• Strong understanding of reliability engineering, observability, incident management, capacity planning, performance optimization, recovery, change control, and operational readiness.

• Experience designing secure automation using role-based access control, managed identities, service principals, privileged identity management, just-in-time access, least privilege, secrets management, and auditable approval controls.

• Ability to analyze complex cloud cost, utilization, telemetry, and operational data and convert findings into prioritized actions and clear executive insights.

• Excellent written and verbal communication skills, including the ability to present complex technical and financial information to engineers, application owners, finance partners, vendors, and senior leaders.

What You'll Bring

• Bachelor's or master's degree in Computer Science, Computer Engineering, Information Systems, Data Science, Finance preferred.

• FinOps Certified Practitioner or certifications in Azure, AWS, Kubernetes, Terraform, ITIL, site reliability engineering, security, data, or artificial intelligence.

• Experience with IBM Turbonomic, IBM Cloudability, Kubecost, Flexera SPOT/Ocean, Flexera ElasticGroup, Azure Cost Management, AWS Cost Explorer, or comparable FinOps and optimization platforms.

• Experience building intelligent automation with approved enterprise AI services such as Microsoft 365 Copilot, GitHub Copilot, Azure OpenAI, Azure AI Foundry, Power Platform, or comparable platforms.

• Experience with Power BI, Kusto Query Language, Log Analytics, Snowflake, data engineering, anomaly detection, forecasting, or machine-learning pipelines.

• Knowledge of the FinOps Open Cost and Usage Specification (FOCUS), reservation and savings-plan economics, and enterprise cost-allocation models.

• Experience integrating cloud automation with ServiceNow change management, configuration management databases, approval workflows, audit evidence, and operational controls.

• Experience designing safe remediation workflows with policy validation, human-in-the-loop approvals, post-change monitoring, rollback, evidence capture, and drift detection.

• Knowledge of container security, software supply-chain security, vulnerability management, policy as code, and compliance automation.

• Demonstrated success delivering measurable cost savings, reliability improvements, risk reduction, or productivity gains at enterprise scale.

• Knowledge of AT&T technology governance, public-cloud architecture patterns, internal change-management processes, or application ownership practices.

#LI-Onsite Full-time office role.

Ready to join our team?

Apply today.

Our Principal Software Engineering jobs earn between $155,400.00 - $261,100.00 USD Annual. Not to mention all the other amazing rewards that working at AT&T offers. Individual starting salary within this range may depend on geography, experience, expertise, and education/training. Joining our team comes with amazing perks and benefits:

• Medical/Dental/Vision coverage
• 401(k) plan
• Tuition reimbursement program
• Paid Time Off and Holidays (based on date of hire, at least 23 days of vacation each year and 9 company-designated holidays)
• Paid Parental Leave
• Paid Caregiver Leave
• Additional sick leave beyond what state and local law require may be available but is unprotected
• Adoption Reimbursement
• Disability Benefits (short term and long term)
• Life and Accidental Death Insurance
• Supplemental benefit programs: critical illness/accident hospital indemnity/group legal
• Employee Assistance Programs (EAP)
• Extensive employee wellness programs
• Employee discounts up to 50% off on eligible AT&T mobility plans and accessories, AT&T internet (and fiber where available) and AT&T phone

Weekly Hours:

40

Time Type:

Regular

Location:

Alpharetta, Georgia, Atlanta, Georgia, Bothell, Washington, Middletown, New Jersey, Plano, Texas, USA:TX:Dallas / Two AT&T Plaza (211 S Akard St) - Dat:211 S Akard St

Salary Range:

$155,400.00 - $261,100.00 AT&T and its subsidiaries are committed to equal employment opportunity. All hiring, promotion, and other employment decisions remain merit-based and free from discrimination on the basis of race, color, religion, religious creed, national origin, ancestry, age, sex, sexual orientation, gender, gender identity, gender expression, physical disability, mental disability, pregnancy, medical condition, genetic information, marital status, citizenship status, military status, veteran status, or any other characteristic protected by federal, state, or local laws. In addition, AT&T will provide reasonable accommodations to qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made. Click here to learn more or request an application accommodation here.