Senior Data Engineer- Remote, India ( Immediate Joiner)

Leewayhertz Technologies; United States; Remote; USD 140,000 - 180,000; Full time

Job summary

Leewayhertz Technologies is seeking an experienced Data Engineer to design, build, and optimize scalable data pipelines and data architectures across Lakehouse and dimensional models. You will integrate SAP data, maintain CI/CD for pipelines, and ensure data quality and security.

You will work closely with data scientists and analysts to meet business needs, monitor pipelines for high availability, and document data processes and flows.

Qualifications

• 7–8 years of hands-on data engineering experience building and running production data pipelines at scale.
• Strong expertise in Azure data services (ADLS Gen2, Azure Databricks, Azure DevOps).
• Deep Databricks experience: PySpark, Spark SQL, Delta Lake, Lakehouse architecture, and Workflows.
• CI/CD for data engineering using Databricks Asset Bundles (DAB) and Azure DevOps.
• Strong Snowflake experience (modeling, tuning, loading, optimization).
• Proficiency in SQL and Python.
• Experience integrating SAP and other enterprise ERP data into a data lake/warehouse.
• Solid data modeling (dimensional, star/snowflake, Lakehouse) and ETL/ELT design.
• Building data-quality, validation, reconciliation and pipeline observability.

Responsibilities

• Design, develop, and maintain scalable data pipelines for data ingestion, processing and storage.
• Build and optimize data architectures and data models (Lakehouse / medallion, dimensional) for efficient data storage and retrieval.
• Develop ETL/ELT processes to transform and load data from various sources into data warehouses and data lakes.
• Build and orchestrate pipelines on Azure Databricks using PySpark, Spark SQL and Delta Lake, orchestrated with Databricks Workflows.
• Integrate data from enterprise source systems including SAP (ABAP/CDS extracts, RPA/CSV or connectors) and load into Snowflake and Databricks.
• Own end-to-end CI/CD for data pipelines using Databricks Asset Bundles (DAB) and Azure DevOps (Git repositories, YAML build and release pipelines), promoting code across dev, QA and production.
• Implement data quality, validation, freshness and reconciliation checks with pipeline observability.
• Ensure data integrity, quality, and security across all data systems.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.
• Document data processes, architectures, and data flow diagrams.

Skills

SQL

Python

Azure Databricks

Delta Lake

Data Modeling

Spark SQL

ETL/ELT

Snowflake

CI/CD

Databricks Workflows

Tools

Azure DevOps

Snowflake

Databricks

DAB (Databricks Asset Bundles)

Job description

Role & responsibilities

• Design, develop, and maintain scalable data pipelines for data ingestion, processing and storage.
• Build and optimize data architectures and data models (Lakehouse / medallion, dimensional) for efficient data storage and retrieval.
• Develop ETL/ELT processes to transform and load data from various sources into data warehouses and data lakes.
• Build and orchestrate pipelines on Azure Databricks using PySpark, Spark SQL and Delta Lake, orchestrated with Databricks Workflows.
• Integrate data from enterprise source systems including SAP (ABAP/CDS extracts, RPA/CSV or connectors) and load into Snowflake and Databricks.
• Own end-to-end CI/CD for data pipelines using Databricks Asset Bundles (DAB) and Azure DevOps (Git repositories, YAML build and release pipelines), promoting code across dev, QA and production.
• Implement data quality, validation, freshness and reconciliation checks with pipeline observability.
• Ensure data integrity, quality, and security across all data systems.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Monitor and troubleshoot data pipelines and workflows to ensure high availability and performance.
• Document data processes, architectures, and data flow diagrams.

Preferred candidate profile

• 7 - 8 years of hands-on data engineering experience building and running production data pipelines at scale.
• Strong expertise in Azure and Azure data services (ADLS Gen2, Azure Databricks, Azure DevOps).
• Deep hands-on experience with Databricks: PySpark, Spark SQL, Delta Lake, Lakehouse / medallion architecture and Databricks Workflows.
• CI/CD for data engineering using Databricks Asset Bundles (DAB) and Azure DevOps (Git, YAML build/release pipelines, multi-environment promotion). (Must-have)
• Strong Snowflake experience (data modeling, performance tuning, loading and optimization).
• Proficiency in SQL and Python.
• Experience integrating data from SAP and other enterprise ERP / source systems into a data lake or warehouse.
• Solid data modeling (dimensional, star/snowflake, Lakehouse) and ETL/ELT design.
• Building data-quality, validation, reconciliation and pipeline monitoring / observability.