Senior Data Engineer

SigmaSoftware2

1h 13m ago

On-site

Full Time

Poland

Design and build scalable cloud-native data platforms from greenfield to production
Design and implement near-real-time ingestion pipelines using event-driven patterns
Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts
Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability
Introduce best practices for code quality, testing, and CI/CD across data pipelines
Drive adoption of AI tooling and agentic workflows within the data engineering team
Ensure data quality, observability, and reliability across pipelines and platforms
Design and deliver self-service tooling and microservices that abstract platform complexity
Collaborate directly with the Customer’s scientific and engineering teams
Monitor platform performance, reliability, and cloud infrastructure costs

5+ years of experience in data engineering
3+ years of production experience with Databricks and Delta Lake
Strong expertise in medallion architecture and event-driven asynchronous pipeline design
Advanced proficiency in Python, PySpark, and SQL
Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization
Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway
Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles
Practical experience with Git workflows, pull request reviews, and automated testing
Experience configuring secure CI pipelines using OIDC or service principals
Practical MLflow experience including experiment tracking, model registration, and reproducibility practices
Strong understanding of data quality, lineage, provenance, schema contracts, and versioning
Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX
Experience with monitoring, observability, and cloud cost optimization
Upper-Intermediate English level or higher

WILL BE A PLUS

Experience packaging workloads in Docker containers and running batch jobs on AWS
Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy
Experience building feature sets and train/test splits that avoid data leakage
Experience with regulated or high-security environments including data residency requirements
Experience disabling or restricting AI features in secured environments

PERSONAL PROFILE

databrickspysparkpythonsqlawsterraformmlflow

🇬🇧 English

SigmaSoftware2

1h 13m ago

On-site

Full Time

Poland

Design and build scalable cloud-native data platforms from greenfield to production
Design and implement near-real-time ingestion pipelines using event-driven patterns
Define and enforce data platform standards including Data Lake/Lakehouse principles, medallion architecture, and data contracts
Refactor, optimize, and modernize Spark and PySpark scripts for performance and maintainability
Introduce best practices for code quality, testing, and CI/CD across data pipelines
Drive adoption of AI tooling and agentic workflows within the data engineering team
Ensure data quality, observability, and reliability across pipelines and platforms
Design and deliver self-service tooling and microservices that abstract platform complexity
Collaborate directly with the Customer’s scientific and engineering teams
Monitor platform performance, reliability, and cloud infrastructure costs

5+ years of experience in data engineering
3+ years of production experience with Databricks and Delta Lake
Strong expertise in medallion architecture and event-driven asynchronous pipeline design
Advanced proficiency in Python, PySpark, and SQL
Deep hands-on experience with Databricks, including Unity Catalog, Lakeflow Jobs/Workflows, and job optimization
Strong AWS expertise including S3, IAM, KMS, and services such as AWS Batch, ECR, EventBridge, Lambda, or API Gateway
Experience with infrastructure as code and CI/CD using Terraform and Databricks Asset Bundles
Practical experience with Git workflows, pull request reviews, and automated testing
Experience configuring secure CI pipelines using OIDC or service principals
Practical MLflow experience including experiment tracking, model registration, and reproducibility practices
Strong understanding of data quality, lineage, provenance, schema contracts, and versioning
Experience with data quality frameworks such as Great Expectations, Soda, pyDeequ, or Databricks Labs DQX
Experience with monitoring, observability, and cloud cost optimization
Upper-Intermediate English level or higher

WILL BE A PLUS

Experience packaging workloads in Docker containers and running batch jobs on AWS
Familiarity with experiment optimization tools such as scikit-optimize, BoTorch, Ax, and scipy
Experience building feature sets and train/test splits that avoid data leakage
Experience with regulated or high-security environments including data residency requirements
Experience disabling or restricting AI features in secured environments

PERSONAL PROFILE

databrickspysparkpythonsqlawsterraformmlflow

🇬🇧 English