Senior Databricks Migration Engineer (remote)

Tier One Technologies | Information technology services

11d 20h ago

Remote In-Country

Contract

United States of America

Tier One Technologies is seeking a Databricks Migration Engineer to support our US Government client.
This remote Contract-to-Hire position will be originated in Rosslyn, VA. East Coast time zone candidates preferred.
Must be a US Citizen.
SELECTED CANDIDATES WITHOUT REQUIRED CLEARANCE WILL BE SUBJECT TO A FEDERAL GOVERNMENT BACKGROUND INVESTIGATION TO RECEIVE IT.

Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
Establish best practices for partition strategies and file size management within Delta Lake.
Design and implement a robust data security model using Unity Catalog for centralized governance.
Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
Communicate effectively verbally and in written form to both technical and non-technical audience
Work in an organized fashion, completing tasks timely while paying close attention to details

Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field.
5+ years of experience in Data Engineering, Data System Development or related roles.
5+ years of experience with Cloud platforms (e.g. Azure, AWS, GCP).
1+ year leading complex, cross-functional data projects and technical teams.
Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.
Excellent communication skills.
Must be a US Citizen and be able to obtain a Position of Public Trust Clearance.
Must have resided in the US for the last 5 years and not have traveled outside the US for a combined total of 6 months or more in last 5 years.

Required Skills

databrickspysparksparkmssqlazurepower_bi

Required Languages

🇬🇧 English

Senior Databricks Migration Engineer (remote)

Tier One Technologies | Information technology services

11d 20h ago

Remote In-Country

Contract

United States of America

Tier One Technologies is seeking a Databricks Migration Engineer to support our US Government client.
This remote Contract-to-Hire position will be originated in Rosslyn, VA. East Coast time zone candidates preferred.
Must be a US Citizen.
SELECTED CANDIDATES WITHOUT REQUIRED CLEARANCE WILL BE SUBJECT TO A FEDERAL GOVERNMENT BACKGROUND INVESTIGATION TO RECEIVE IT.

Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
Establish best practices for partition strategies and file size management within Delta Lake.
Design and implement a robust data security model using Unity Catalog for centralized governance.
Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
Communicate effectively verbally and in written form to both technical and non-technical audience
Work in an organized fashion, completing tasks timely while paying close attention to details

Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field.
5+ years of experience in Data Engineering, Data System Development or related roles.
5+ years of experience with Cloud platforms (e.g. Azure, AWS, GCP).
1+ year leading complex, cross-functional data projects and technical teams.
Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms.
Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark.
Excellent communication skills.
Must be a US Citizen and be able to obtain a Position of Public Trust Clearance.
Must have resided in the US for the last 5 years and not have traveled outside the US for a combined total of 6 months or more in last 5 years.

Required Skills

databrickspysparksparkmssqlazurepower_bi

Required Languages

🇬🇧 English