Principal Site Reliability Engineer
We’re looking for talented professionals to join us in bringing smart money management and payment solutions to everyone’s fingertips.
This position is classified as structured hybrid, with an expectation of three (3) days per week working in the office and flexibility to work remotely on the remaining days. On-site expectations may evolve over time to support business needs, with clear communication provided in advance.
<<>><<>><<>><<>><<>><<>><<>><<>><<>><<>>
JOB DESCRIPTION
Location Requirement
This position is open only to candidates who currently reside in one of the following approved locations:
West Los Angeles, CA
About Green Dot
We're looking for talented professionals, anywhere in the United States, to join us in bringing smart money management and payment solutions to everyone's fingertips. Green Dot is a financial technology and registered bank holding company. Our platforms support millions of customers and enterprise partners, and they must run securely, reliably, and in compliance with banking and payment-industry standards.
Role Overview
As a Principal Site Reliability Engineer, you will be the senior technical leader for reliability, release management, and database operations across Green Dot's cloud and hybrid platforms. You will set the strategy and standards for how we build, deploy, observe, and recover our services. You will own the end-to-end release process from code merge to production, and you will make sure our data tier (SQL Server, Azure SQL, Cosmos DB, and related data stores) is highly available, performant, and resilient.
This is a hands-on, high-impact individual contributor role. It requires deep expertise in software and systems engineering, sound judgment in a regulated environment, and the ability to influence engineering teams across the organization.
Key Responsibilities
Site Reliability Engineering
- Define and drive the SRE strategy, standards, and roadmap for availability, latency, scalability, and resilience across critical payment and banking services.
- Establish and govern SLIs, SLOs, and error budgets with product and engineering stakeholders, and report on reliability KPIs regularly to leadership.
- Architect and maintain highly available, fault-tolerant infrastructure across Azure and AWS, including multi-region and disaster recovery (DR) designs.
- Govern and improve the observability platform (logging, metrics, tracing, alerting) with Datadog as the primary tool, supplemented by Prometheus/Grafana and native cloud monitoring (Azure Monitor, CloudWatch).
- Lead incident management: act as Incident Commander for high-severity events, run blameless post-incident reviews (RCAs), and track corrective actions to completion.
- Take part in and improve the on-call rotation, including runbooks, alert quality, and escalation paths.
- Make sure messaging and event-driven systems (e.g., Azure Service Bus, RabbitMQ) process messages efficiently, and monitor them for lag, dead-lettering, and throughput.
Proactive Reliability & Resilience
- Lead capacity planning, performance testing, and cost optimization for cloud infrastructure.
- Reduce toil through automation and self-service tooling.
- Run chaos engineering, resilience testing, and DR/business continuity exercises to validate RTO/RPO targets.
Release Management
- Own the enterprise release management process: release planning, scheduling, release calendars, change windows, and go/no-go decisions.
- Design and implement predictable, automated deployment strategies (blue/green, canary, rolling, feature flags) across Dev, QA, UAT, Staging, and Production.
- Architect, standardize, and maintain CI/CD pipelines in Azure DevOps and GitHub Actions (Jenkins where applicable), with automated testing, security scanning, and quality gates built in.
- Drive GitOps practices (e.g., Argo CD) so that infrastructure and application deployments are declarative, version-controlled, and auditable.
- Lead Change Advisory Board (CAB) participation and change management, and make sure releases have the approvals, evidence, and audit trails needed for regulated environments (e.g., SOX, PCI DSS).
- Define branching, versioning, and environment-promotion strategies, and oversee rollback and hotfix procedures.
- Coordinate cross-team releases with Development, QA, Product, Security, and Operations, and communicate release status, risks, and outcomes to stakeholders.
- Track and improve DORA metrics (deployment frequency, lead time for changes, change failure rate, mean time to restore).
- Maintain release documentation, release notes, and post-release reviews in Jira and Confluence.
Database Reliability Engineering
- Act as the reliability lead for the data tier: Microsoft SQL Server, Azure SQL Database / Managed Instance, Cosmos DB, and related stores (e.g., PostgreSQL, Redis).
- Design and maintain high-availability and DR solutions such as Always On Availability Groups, failover clustering, geo-replication (including Cosmos DB multi-region configuration), and backup/restore strategies with tested recovery procedures.
- Lead database change management inside CI/CD: automated, version-controlled schema migrations (e.g., SSDT/DACPAC, Flyway, Liquibase) with safe rollback paths.
- Drive database performance tuning: query optimization, indexing strategy, execution plan analysis, blocking/deadlock resolution, Cosmos DB partitioning and RU/s capacity, and capacity forecasting.
- Build database observability (Datadog Database Monitoring, Query Store, Extended Events), with proactive alerts for replication lag, resource pressure, and long-running queries.
- Partner with Information Security on database security controls, including encryption at rest and in transit, least-privilege access, auditing, and patching.
- Automate routine DBA tasks (maintenance, patching, health checks, provisioning) using PowerShell, Python, and IaC.
Leadership & Collaboration
- Serve as a technical authority and mentor to SRE, DevOps, and release engineers, and help build a culture of continuous learning and operational excellence.
- Influence architecture decisions across engineering teams so that new services are designed for reliability, observability, and deployability from the start.
- Partner with Security, Compliance, and Audit teams to support control requirements and audits with evidence.
- Evaluate emerging tools and practices, and lead adoption where they add measurable value.
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent experience.
- 10+ years of experience in Site Reliability Engineering, DevOps, Release Engineering, or Infrastructure Engineering, with a record of running complex, high-availability production systems.
- 5+ years of hands-on experience leading release management or CI/CD for multiple teams or services.
- 5+ years of hands-on database experience with Microsoft SQL Server and/or Azure SQL, and hands-on experience with Azure Cosmos DB, including HA/DR, performance tuning, and production troubleshooting.
- Deep expertise in Azure (required) and working knowledge of AWS.
- Production experience with Kubernetes (AKS/EKS), Docker, and Helm.
- Strong proficiency with Infrastructure as Code and GitOps: Terraform (required), plus Ansible and Argo CD or similar.
- Expertise with CI/CD tooling: Azure DevOps pipelines, GitHub Actions, and Git-based workflows.
- Expert-level observability experience with Datadog or comparable platforms (logging, APM, metrics, alerting).
- Strong scripting and automation skills in PowerShell, Python, and Bash; strong T-SQL skills.
- Experience with messaging and event bus technologies (Azure Service Bus, RabbitMQ, or similar).
- Experience running incident response, RCAs, and on-call programs.
- Strong communication skills, with the ability to explain complex technical topics to engineers and executives.
Preferred Qualifications
- Experience in financial services, banking, payments, or fintech, including working under PCI DSS, SOX, or similar regulatory controls.
- Experience with progressive delivery or feature-flag platforms.
- Experience with caching technologies (Redis) and PostgreSQL.
- Familiarity with service mesh (Istio, Linkerd) and Kubernetes security best practices.
- Experience with .NET application environments and Windows Server/IIS as well as Linux.
- Certifications such as Azure Solutions Architect Expert, Azure DevOps Engineer Expert, Microsoft Certified: Azure Database Administrator Associate, CKA/CKAD, AWS Solutions Architect, or ITIL.
**Ability to work in the U.S. without sponsorship**
**Ability to meet the location requirement outlined above**
POSITION TYPE
RegularPAY RANGE
The targeted base salary for this position is $129,000 to $193,400 per year. The final compensation will be determined by a number of factors such as qualifications, expertise, and the candidate’s geographical location.<<>><<>><<>><<>><<>><<>><<>><<>><<>><<>>
We’re Here to Support You—Accommodations Upon Request
Green Dot is committed to providing an inclusive and accessible hiring experience for all candidates. If you require a reasonable accommodation during any part of the application or interview process, we encourage you to let us know. We will work with you to meet your needs in a way that respects your privacy and ensures equal opportunity. Our goal is to support every applicant in showcasing their talents and potential.
Work Authorization Requirement
At Green Dot Corporation, we value diversity and strive for fair and inclusive hiring practices. However, we are currently unable to offer visa sponsorship. All applicants must be legally authorized to work in the United States at the time of application and throughout the duration of employment, without the need for current or future sponsorship.
Important Notice on Application Accuracy
We value integrity in our hiring process. Please ensure that all information provided in your resume and application is accurate and authentic. Submissions found to be fraudulent or misleading will result in disqualification from consideration, and any offers extended may be rescinded.
Required Skills
Required Languages
🇬🇧 English