Azure Databricks Data Engineer | PySpark & Delta Lake

Prodapt Solutions Private Limited; Irving (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Prodapt Solutions Private Limited seeks an experienced Azure Databricks Engineer in Dallas, TX to design and build scalable data pipelines with PySpark, Python and SQL. Lead automated workflows and mentor junior engineers while collaborating across data architects, engineers, scientists, and business stakeholders.

Responsibilities include ETL/ELT development, Delta Lake architectures, and integrating Databricks with Azure services.

Qualifications

• 10+ years of Data Engineering experience including ETL/ELT pipelines.
• Bachelor’s degree in Computer Science, Engineering, or related field.
• Strong hands-on experience with Azure Databricks; 4+ years across Azure services.
• 7+ years of Python development with reusable libraries.
• 4+ years of Snowflake or SQL experience.
• Expert PySpark and Spark SQL knowledge.
• Genuine Python and SQL programming strength.
• Delta Lake and Lakehouse architecture experience.
• ADLS Gen2 and other Azure data services expertise.
• Git, CI/CD, and DevOps practices familiarity.
• Experience with performance tuning in Databricks/Spark.
• Ability to collaborate with cross-functional teams.

Responsibilities

• Design, develop, and implement scalable data pipelines using Azure Databricks.
• Build and optimize ETL/ELT pipelines with PySpark, Python, and SQL.
• Develop data solutions using Bronze/Silver/Gold Medallion architecture.
• Work with Delta Lake, Delta tables, and Databricks optimization techniques.
• Integrate Databricks with ADLS Gen2, ADF, Synapse, and Key Vault.
• Develop and manage Databricks Workflows, Jobs, and Notebooks.
• Implement data quality, monitoring, error handling, and performance optimization.
• Collaborate with Data Architects, Engineers, Scientists, and stakeholders.
• Establish CI/CD, Git/version control, code reviews, and automated deployments.
• Lead technical discussions and guide junior engineers.
• Analyze Python OOP apps and redesign for distributed Spark execution.
• Design, develop, and deploy enterprise-scale pipelines on Databricks.
• Implement Delta Lake solutions across Bronze/Silver/Gold.
• Build robust ETL/ELT pipelines with Azure Data Factory and Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs with partitioning, caching, and Delta optimization.

Skills

Data engineering

Python

SQL

PySpark

Spark SQL

ETL/ELT design

CI/CD

DevOps

Git

Distributed processing

Education

Bachelor's degree in Computer Science or Engineering

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

ADLS Gen2

Snowflake

Delta Lake

Git

Terraform

Azure DevOps

Unity Catalog

Job description

Prodapt Solutions Private Limited seeks an experienced Azure Databricks Engineer in Dallas, TX to design and build scalable data pipelines with PySpark, Python and SQL. Lead automated workflows and mentor junior engineers while collaborating across data architects, engineers, scientists, and business stakeholders.

Responsibilities include ETL/ELT development, Delta Lake architectures, and integrating Databricks with Azure services.