Senior Data Engineer / Data Architect (Databricks & Snowflake) urgent Need

Tech Mirrors; Boston (MA); Hybrid; USD 100,000 - 130,000; Full time

Job summary

Tech Mirrors is seeking a highly experienced Senior Data Engineer / Data Architect with deep expertise in Databricks, Snowflake, and Azure cloud data platforms. The ideal candidate will implement scalable data pipelines, Lakehouse architectures, and real-time data processing solutions, especially in Life Sciences or Healthcare sectors.

The role involves designing data engineering pipelines, optimizing Spark workloads, and managing CI/CD processes. Knowledge of cloud-native data practices is essential, as well as collaboration with cross-functional teams to support analytics and reporting needs.

Qualifications

• Extensive experience designing and implementing scalable data pipelines.
• Strong proficiency in Spark (PySpark) and Delta Lake.
• Background in regulated domains such as Life Sciences or Healthcare.

Responsibilities

• Design and implement end-to-end data engineering pipelines.
• Develop scalable ETL/ELT pipelines using PySpark and Spark SQL.
• Optimize Spark workloads for improved performance.

Skills

Databricks

Snowflake

Azure

PySpark

Spark SQL

Tools

Talend

Azure DevOps

Jenkins

Git

Job description

Position

Senior Data Engineer / Data Architect (Databricks & Snowflake)

Location

Boston, MA – Hybrid

Job Type

Contract

Duration

Long term

Job Summary

We are seeking a highly experienced Senior Data Engineer / Data Architect with deep expertise in Databricks, Snowflake, and Azure cloud data platforms. The ideal candidate will have extensive experience designing and implementing scalable data pipelines, Lakehouse architectures, and real-time data processing solutions, particularly in regulated domains such as Life Sciences or Healthcare. This role requires strong proficiency in Spark (PySpark), Delta Lake, Medallion architecture, and cloud-native data engineering practices, along with a solid background in data warehouse modernization and performance optimization.

Key Responsibilities

• Design and implement end-to-end data engineering pipelines using Azure Databricks, ADLS Gen2, and Snowflake.
• Develop scalable ETL/ELT pipelines using PySpark, Spark SQL, Python, and Talend.
• Build and maintain Lakehouse architecture using Delta Lake and Medallion (Bronze, Silver, Gold) layers.
• Implement real-time and batch data ingestion pipelines, including streaming using Spark Structured Streaming.
• Design and enforce data governance, access control, and lineage using Unity Catalog.
• Optimize Spark workloads through partitioning, caching, broadcast joins, and cluster tuning to improve performance and reduce cloud costs.
• Architect and manage CI/CD pipelines using Azure DevOps, Jenkins, and Git for automated deployments.
• Integrate multiple data sources and systems, ensuring high-quality, reliable, and scalable data delivery.
• Collaborate with cross-functional teams including data analysts, scientists, and business stakeholders to support analytics and reporting needs.
• Support data warehouse modernization initiatives, including migration from legacy systems to cloud platforms.