Sr. Databricks Engineer - Information Technology Jobs Near Me

Capstone IT is helping our client to find a Senior Data Architect – Databricks who will lead the design, implementation, and modernization of enterprise data platforms using Databricks and cloud technologies.

This individual will be responsible for building scalable, secure, and high-performing data environments from initial architecture and configuration through production deployment. They will design Databricks Lakehouse solutions, develop data pipelines, implement data governance, and help migrate legacy data systems into modern cloud platforms.

The ideal candidate combines strategic architecture experience with strong hands-on engineering skills. They must be comfortable making technical decisions, writing code, configuring environments, troubleshooting complex issues, and leading engineering teams.

The biggest differentiator: Our client needs someone who has personally led a complete Databricks implementation—not just someone who has worked within an established Databricks environment.

Top 4 Required Skills

1. Databricks Architecture & End-to-End Implementation

3+ years of hands-on Databricks experience, including at least one enterprise implementation from initial architecture and environment setup through production.

2. Python, PySpark, Apache Spark & SQL

Advanced data engineering skills, including building and optimizing distributed data pipelines and ETL/ELT solutions.

3. Cloud Data Architecture & Lakehouse Design

Strong Azure, AWS, or GCP experience with Delta Lake, medallion architecture, cloud storage, and enterprise platform modernization.

4. Unity Catalog, Governance & Technical Leadership

Experience implementing governance, security, access controls, data quality, architectural standards, and guiding engineering teams.

Main Responsibilities

• Architect and deploy enterprise Databricks Lakehouse platforms from the ground up.

• Lead environment configuration, data integration, migration, testing, and production deployment.

• Design data pipelines using Python, PySpark, SQL, Apache Spark, and Delta Lake.

• Implement Bronze, Silver, and Gold data layers.

• Establish governance and security using Unity Catalog.

• Modernize legacy data warehouses and ETL platforms.

• Optimize compute performance, scalability, reliability, and cloud costs.

• Implement CI/CD pipelines and infrastructure automation.

• Develop architecture diagrams, standards, and technical roadmaps.

• Provide hands-on technical leadership to engineering, DevOps, analytics, and BI teams.

Required Qualifications

Qualification	Requirement
Overall experience	8+ years in data architecture, engineering, or warehousing
Databricks	3+ years hands-on
Implementation	At least one complete Databricks implementation
Programming	Advanced Python, PySpark, and SQL
Cloud	Azure, AWS, or GCP
Lakehouse	Delta Lake, Unity Catalog, Spark
Data engineering	ETL/ELT, integration, modeling, optimization
Modernization	Legacy-to-cloud migration experience
Leadership	Architecture ownership, technical mentoring, stakeholder communication

Preferred: Databricks certifications, Terraform, CI/CD, Azure Data Factory, AWS Glue, dbt, Airflow, Power BI, Tableau, streaming, and AI/ML workloads.

What Success Looks Like

The successful candidate delivers a production-ready Databricks platform, establishes secure and reusable architecture patterns, improves data processing performance, and enables engineering teams to build reliable analytics solutions independently.