Azure Databricks Data Engineer: Lakehouse & PySpark

Saransh Inc; Missouri; On-site; USD 110,000 - 165,000; Full time

Job summary

Saransh Inc. is seeking an experienced data engineer to architect and implement production-grade batch data pipelines on Azure Databricks using PySpark. You will design bronze/silver/gold data models, ensure data quality and lineage, and collaborate with HR data systems and business stakeholders.

You will optimize Spark workloads, implement governance with Unity Catalog, and participate in CI/CD and DevOps practices to ensure reliable, scalable analytics solutions.

Qualifications

• Hands-on experience with Azure Databricks and PySpark pipelines.
• Advanced SQL development and performance tuning for large datasets.
• Expertise with Delta Lake table design and schema evolution.
• Experience with Medallion architecture and layered data models.
• Strong data quality, testing, and reconciliation practices.
• Familiarity with Unity Catalog and governance tooling.

Responsibilities

• Design and develop scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
• Build bronze, silver, and gold data structures including certified facts and dimensions.
• Translate business metrics into robust technical designs and data contracts.
• Implement data-quality checks, lineage, and monitoring across pipelines.
• Work with HR data sources (Workday, Dayforce, PeopleSoft) for analytics readiness.
• Coordinate with DevOps and Power BI teams for deployment and stakeholder enablement.
• Apply security governance, least privilege access, and row-level security patterns.
• Contribute to CI/CD, testing, and production-readiness activities.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

Medallion architecture

Data quality

Testing

CI/CD

DevOps

Power BI

Unity Catalog

Data governance

HR data systems

SQL optimization

Tools

Databricks Lakeflow

Delta Live Tables

Informatica

Workday/Dayforce/PeopleSoft

Azure Data Lake Storage

Job description

Saransh Inc. is seeking an experienced data engineer to architect and implement production-grade batch data pipelines on Azure Databricks using PySpark. You will design bronze/silver/gold data models, ensure data quality and lineage, and collaborate with HR data systems and business stakeholders.

You will optimize Spark workloads, implement governance with Unity Catalog, and participate in CI/CD and DevOps practices to ensure reliable, scalable analytics solutions.