Senior Data Engineer, Platform

Senior Data Engineer, Platform

Salary range: $180,000 - $220,000 per year

About the Role

Two kinds of data decide whether we can do our job. What a health plan tells us happened to a member, in eligibility files, claims, pharmacy and hospital event feeds. And what our own software records about what we did for them. When a file is late, malformed, or quietly incomplete, care teams work from a stale list. When the two do not line up, nobody can say whether the work is helping.

You will own the infrastructure our data runs on: where and how our jobs run and whether they run efficiently, including compute choices, serverless versus clusters, cost and performance; how work is orchestrated and deployed; how we connect to partners and systems, over SFTP, S3, private networking and partner feeds; and whether all of it is reliable and observable. You will set the technical direction for that platform, make the architecture calls, and build it as software, with tests, version control and infrastructure as code. This is a senior individual contributor role, with a clear path to leading data engineering as the team grows.

This role is complementary to analytics engineering. Analytics owns the questions and the answers, analytics engineering builds the gold tier (curated tables) they work from, and you own the platform underneath it, so those tables arrive complete, on time and at a sensible cost.

We are a small, AI-forward team. The whole team builds with AI coding agents, and part of this platform's job is to make our data usable by AI. We build and evaluate AI systems on real data, where a wrong answer has consequences, so evaluation is part of the work rather than something bolted on at the end.

We want the person who looks at a 384 GB Spark driver and asks why. We are adding markets and data quickly, and we want a platform that grows by adding machines rather than by buying bigger ones. You will know what belongs on a single machine and what does not, and you will keep our pipelines fast, predictable and economical to run as they grow.

Our partners send and expect files on fixed monthly schedules. Eligibility, claims and attribution files come in; engagement, roster and service files go out. Part of the job is making those exchanges boring: replacing recurring manual work with automation, so routine exchanges run themselves and people are pulled in only when something needs judgment. And you will not become the person everything depends on. We treat that as a design flaw, not a badge.

In your first 90 days, you might take on one or two of these:

• Take over the performance work already underway on our heaviest batch jobs, carry it from single-machine processing onto distributed compute, and prove the output did not change

• Move our legacy Databricks jobs onto our newer platform, with the orchestration, deploys and runbooks to operate them

• Connect a new payer data feed end to end, from the first file over SFTP or S3 to members live in our care platform, monitored from day one

• Review where each of our scheduled jobs runs, serverless or cluster, and right-size the ones that cost more than they should

You will also be close to the people whose work this data describes: coaches, clinicians and researchers who spend their days with our members. When you cannot tell what a field means, or whether a number is plausible, you can go and ask the person who generated it. Most data engineers never get to do that. Our members are among the most underserved people in American healthcare, and these data problems are not abstract.

About Vanna Health

At Vanna Health, we believe that a sense of belonging and engagement in the community is the foundation for improving overall health and wellbeing for individuals with serious mental illness. We help people living with serious mental illness thrive by strengthening the community ecosystem around them. Through innovative payment models and technology, Vanna Health promotes collaboration across existing behavioral health and primary care providers, hospitals, payers, clubhouses and other community based organizations. Additionally, Vanna Health can deploy our own top quality providers and case managers, as needed, in areas where there may be a shortage of resources.

Vanna Health was founded in 2021 by a team of mission-driven entrepreneurs, working in close partnership with payers, providers, and community based organizations to promote collaboration and improve outcomes while driving down costs. We are committed to the goal of supporting all individuals with culturally competent care regardless of their race, ethnicity, religion, sexual orientation, or gender identity. With modern technology, extensive development opportunities, diverse careers paths, and a culture that values diversity, equity, inclusion, and belonging, we are redefining what it means to work in community health.

Responsibilities

In your first year, most of your time goes to compute, ingestion and orchestration; the AI and governance work grows with the team.

• Shape the data platform roadmap with data leadership, own delivery of the work you lead, and keep stakeholders informed of progress, risks and trade-offs

• Set technical direction for data engineering: design and code-review standards, mentoring, and unblocking the engineers and contractors who work on the platform

• Own the platform architecture on Databricks and AWS, including ingestion, storage, compute, access control and environments. Write the design docs and make the build-versus-buy calls

• Own connectivity to partners and systems, built as tested, versioned software: payer claims, eligibility and attribution files over SFTP and S3, private networking, clinical and event feeds (FHIR, HL7), and data exchanged with our care platform

• Decide where and how jobs run, and make them fast, memory-predictable and economical: compute choices, serverless versus clusters, right-sizing, and moving single-machine Python onto distributed compute

• Own orchestration and deploys, and treat infrastructure and pipelines as code: Terraform, GitLab CI/CD, automated tests and release practices

• Make the platform reliable and observable, and turn failures into fixes that hold. Know that a job failed, quietly skipped, or ran twice as long as usual before anyone downstream notices, then ask what would have caught it earlier and what could have recovered it without you

• Make our data usable by AI: governed metric definitions, Genie spaces and MCP servers, so colleagues and AI agents work from trusted data rather than guesses

• Validate what arrived against what was promised: row counts, date spans, expected fields, and whether history reaches as far back as it should

• Replace recurring manual work with automation and AI agents, so routine data operations run themselves and people are pulled in only when something needs judgment

• Take part in a shared data operations rotation, which we keep light through automation

• Protect PHI by design: access controls, auditing, de-identification and secure data exchange, in partnership with our security leadership

• Give analytics engineering a platform to build on: they build the gold tier the analytics team uses, and you make sure it builds reliably, on time and at a sensible cost

• Represent data engineering with partners, vendors and colleagues, and share what you learn

About You

• Typically seven or more years building production software or data systems, including several years owning data infrastructure or pipelines end to end, from design through operation

• Hands-on production experience with at least one distributed data processing engine at scale, such as Spark or Databricks, Snowflake, BigQuery, Flink, Dask or Ray, rather than single-machine pandas: you know what belongs on one machine and what does not, have made large jobs fast and memory-predictable, and can describe one with before and after numbers

• Strong Python, and SQL you can defend: not just that the query runs, but why you believe the answer it gives you, and what you checked before you believed it

• Experience designing systems on a major cloud, AWS preferred (storage, compute, networking, IAM and cost), with production engineering discipline: automated testing, CI/CD, infrastructure as code, code review, observability and incident response

• You have led technical work: set the direction for a project or a team, and mentored other engineers

• You have chased a data problem that turned out to be upstream of you, and you kept going until you found where it actually started

• You use AI coding agents in your daily engineering work, and can tell us about an AI result you caught being wrong on real data and how you caught it. Knowing when an answer is wrong, and checking before you act on it, is the harder half of that skill

Bonus Points

• Databricks, Delta Lake, Unity Catalog and dbt

• Databricks Genie, MCP servers, or other ways of putting governed data in front of AI agents

• Healthcare data: claims (837/835), eligibility (834), FHIR, HL7 v2 and hospital event feeds, or another regulated setting such as HIPAA or SOC 2

• Workflow orchestration (Databricks Workflows, Temporal), event-driven ingestion and serverless compute

• Experience leading or managing engineers, or a clear interest in growing into a data engineering lead