Lead Data Engineer

Title: Lead Data Engineer

Lead Data Engineer

 

We are seeking an experienced Lead Data Engineer to join our healthcare technology team with a primary focus on healthcare interoperability. This role will be instrumental in designing, building, and maintaining our Azure-based data infrastructure, including databases, data pipelines, analytics platforms, and business intelligence solutions — with an emphasis on integrating with Electronic Health Record (EHR) systems and healthcare data exchange standards (FHIR, C-CDA, HL7). The ideal candidate will have deep expertise in the Microsoft data stack, strong architectural thinking, and experience with healthcare data compliance requirements. This senior-level position will also provide technical leadership, mentorship to the broader engineering team and reports to the Director of Data Engineering.

 

Duties & Responsibilities:

• Design and implement scalable, secure data architectures on Azure to support business intelligence, analytics, and operational systems
• Build and maintain data pipelines using Azure Data Factory, Azure Synapse and related services

Contribute to database designs across multiple platforms (Azure SQL Database, SQL Managed Instance, PostgreSQL, CosmosDB)

Optimize data storage strategies across Azure Blob Storage, Data Lake, and structured databases

Design data models and structures that enable efficient analytics and reporting

Design, develop, and maintain complex stored procedures, functions, triggers, and views

Implement database DevOps processes including schema versioning, deployment automation, and rollback procedures

Perform database performance tuning, indexing strategies, and query optimization

Ensure data integrity, security, and compliance with HIPAA/HITRUST requirements for PHI data

Manage database lifecycle across development, pre-production, and production environments utilizing RedGate Flyway

Design and implement ETL/ELT processes to integrate data from multiple sources

Build event-driven data processing solutions including message producers, consumers, and workers using Azure Service Bus, Event Grid, and related messaging services

Integrate with external REST APIs for data ingestion and develop internal data APIs using Python (FastAPI/Flask) on Azure Container Apps or Functions

Develop data transformation logic and orchestration workflows

Ensure data quality, consistency, and lineage across the data platform

Integrate with Azure AI services and machine learning endpoints as needed

Review designs of Power BI solutions including semantic models, reports, & dashboards

Collaborate with stakeholders to translate business requirements into technical solutions

Optimize data structures for reporting and analytics performance

Provide technical guidance and mentorship to junior developers and engineers

Contribute to data engineering best practices, standards, and patterns

Participate in architecture reviews and technical decision-making

Collaborate with DevOps and Infrastructure teams on platform improvements

Drive continuous improvement in data quality, performance, and reliability

Create and maintain comprehensive technical documentation including data dictionaries, architecture diagrams, and process flows

Document data lineage, transformation logic, and business rules

Ensure all data handling practices meet HIPAA/HITRUST compliance requirements

Maintain audit trails and documentation for compliance purposes

Troubleshoot and resolve complex data-related issues across the platform

Provide escalation support for production incidents

Collaborate with cross-functional teams to support business operations

Participate in on-call rotation as needed

Preferred:

• Build and maintain integrations with EHR systems using FHIR resources (e.g., Patient, DocumentReference) and C-CDA document parsing/transformation
• Support HL7 v2/MDM/ADT message flows where needed, including message parsing and acknowledgment handling
• Define and maintain OpenAPI specifications and API versioning; write contract tests to prevent breaking changes
• Lead the document templating framework and server-side rendering pipeline for HTML-to-PDF output; ensure accessibility and compliance requirements

 

Experience & Qualifications:

Required:

• Bachelor’s degree in computer science, Information Systems, Data Engineering, or related field, or equivalent work experience
• 8+ years of experience in data engineering, database development, or related roles
• Expert-level proficiency in T-SQL and Microsoft SQL Server
• Expert-level proficiency of Python

Experience building REST APIs and background workers in Python (FastAPI/Flask)

Understanding of event-driven architectures and asynchronous processing patterns

Strong experience with Azure data services (Azure SQL, Data Factory, Synapse, Blob Storage, etc.)

Hands-on experience building and maintaining data pipelines and ETL/ELT processes

Knowledge of Power BI development skills including DAX, data modeling, and performance optimization

Solid understanding of database design principles, normalization, and data modeling

Experience with version control systems (Git) and CI/CD practices

Strong analytical and problem-solving abilities with attention to detail

Excellent written and verbal communication skills

Ability to work both independently and collaboratively in a team environment

Microsoft certification (Azure Data Engineer Associate, Azure Database Administrator, Power BI Data Analyst)

Knowledge of Infrastructure as Code (Terraform, Bicep, ARM templates)

Knowledge of additional Azure services (Container Apps, Functions, Logic Apps)

Experience with other database platforms (PostgreSQL, CosmosDB)

Understanding of HIPAA/HITRUST compliance requirements for healthcare data

Experience PowerShell for automation and scripting

Experience with Azure DevOps or similar platforms for CI/CD

Experience with data governance, cataloging, and metadata management

Background in healthcare technology or regulated industries

Experience with OpenAPI/Swagger specification design and contract testing

Proficient in advanced usage of AI development tooling (e.g., Claude Code, GitHub Copilot)

Azure API Management and private networking patterns

Travel up to 15%.

 

Preferred:

• Practical experience with FHIR (resources, search parameters, OAuth2/SMART on FHIR) and C-CDA parsing/transformation
• Experience integrating with Epic, Cerner, Athena, or other EHR/EMR systems
• Experience with HL7 v2 (MLLP/MDM/ADT) message handling