DataBricks Data Engineer

Prodapt Solutions Private Limited; Irving (TX); On-site; USD 140,000 - 190,000; Full time

Job summary

Prodapt Solutions Private Limited seeks an experienced Azure Databricks Engineer in Dallas, TX to design and build scalable data pipelines with PySpark, Python and SQL. Lead automated workflows and mentor junior engineers while collaborating across data architects, engineers, scientists, and business stakeholders.

Responsibilities include ETL/ELT development, Delta Lake architectures, and integrating Databricks with Azure services.

Qualifications

• 10+ years of Data Engineering experience including ETL/ELT pipelines.
• Bachelor’s degree in Computer Science, Engineering, or related field.
• Strong hands-on experience with Azure Databricks; 4+ years across Azure services.
• 7+ years of Python development with reusable libraries.
• 4+ years of Snowflake or SQL experience.
• Expert PySpark and Spark SQL knowledge.
• Genuine Python and SQL programming strength.
• Delta Lake and Lakehouse architecture experience.
• ADLS Gen2 and other Azure data services expertise.
• Git, CI/CD, and DevOps practices familiarity.
• Experience with performance tuning in Databricks/Spark.
• Ability to collaborate with cross-functional teams.

Responsibilities

• Design, develop, and implement scalable data pipelines using Azure Databricks.
• Build and optimize ETL/ELT pipelines with PySpark, Python, and SQL.
• Develop data solutions using Bronze/Silver/Gold Medallion architecture.
• Work with Delta Lake, Delta tables, and Databricks optimization techniques.
• Integrate Databricks with ADLS Gen2, ADF, Synapse, and Key Vault.
• Develop and manage Databricks Workflows, Jobs, and Notebooks.
• Implement data quality, monitoring, error handling, and performance optimization.
• Collaborate with Data Architects, Engineers, Scientists, and stakeholders.
• Establish CI/CD, Git/version control, code reviews, and automated deployments.
• Lead technical discussions and guide junior engineers.
• Analyze Python OOP apps and redesign for distributed Spark execution.
• Design, develop, and deploy enterprise-scale pipelines on Databricks.
• Implement Delta Lake solutions across Bronze/Silver/Gold.
• Build robust ETL/ELT pipelines with Azure Data Factory and Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs with partitioning, caching, and Delta optimization.

Skills

Data engineering

Python

SQL

PySpark

Spark SQL

ETL/ELT design

CI/CD

DevOps

Git

Distributed processing

Education

Bachelor's degree in Computer Science or Engineering

Tools

Azure Databricks

Azure Data Factory

Azure Synapse

ADLS Gen2

Snowflake

Delta Lake

Git

Terraform

Azure DevOps

Unity Catalog

Job description

Overview

Prodapt is the largest specialized player in the Connectedness industry. As an AI-first strategic technology partner, Prodapt provides consulting, business reengineering, and managed services for the largest telecom and tech enterprises building networks and digital experiences of tomorrow. A ServiceNow-invested company, Prodapt has been recognized by Gartner as a Large, Telecom-Native, Regional IT Service Provider. A Great Place To Work Certified company, Prodapt employs over 5,000 technology and domain experts across the Americas, Europe, India, Africa, & Japan. Prodapt is part of the 130-year-old business conglomerate The Jhaver Group, which employs over 32,000 people across 80+ locations globally.

We are seeking an experienced Azure Databricks Engineer to join our team in Dallas, TX. The ideal candidate has strong hands-on experience in designing and building scalable data engineering solutions using Azure Databricks, PySpark, Python, SQL, and Azure Data Services including leading the modernization and migration of existing Python object-oriented applications into scalable PySpark and Spark SQL data-processing solutions on Azure Databricks.

Responsibilities

• Design, develop, and implement scalable data pipelines using Azure Databricks.
• Build and optimize ETL/ELT pipelines using PySpark, Python, and SQL.
• Develop data solutions using the Medallion Architecture (Bronze, Silver, and Gold layers).
• Work with Delta Lake, Delta tables, and advanced Databricks optimization techniques.
• Integrate Databricks with Azure services such as ADLS Gen2, Azure Data Factory, Azure Synapse, and Azure Key Vault.
• Develop and manage Databricks Workflows, Jobs, and Notebooks.
• Implement data quality, monitoring, error handling, and performance optimization.
• Collaborate with Data Architects, Data Engineers, Data Scientists, and business stakeholders.
• Establish best practices for CI/CD, Git/version control, code reviews, and automated deployments.
• Lead technical discussions and provide guidance to junior engineers.
• Analyze existing Python OOP applications and redesign single-node processing logic for distributed Spark execution.
• Design, develop, and deploy enterprise-scale data pipelines on Azure Databricks; build reusable PySpark frameworks and utility modules.
• Implement Delta Lake solutions using the Bronze, Silver, Gold architecture.
• Build robust ETL/ELT pipelines with Azure Data Factory, ADLS Gen2, and Azure Synapse Analytics.
• Implement data quality, reconciliation, validation, and monitoring frameworks.
• Optimize Spark jobs using partitioning, bucketing, caching, broadcast joins, Adaptive Query Execution, and Delta optimization.

Requirements

• 10+ years of overall Data Engineering experience, including designing and implementing ETL/ELT pipelines with Azure Data Factory and other Azure services.
• Bachelor’s degree in Computer Science, Engineering, or a related field; OR equivalent combination of education and relevant experience.
• Strong hands-on experience with Azure Databricks; 4+ years of experience across Azure services and Databricks (ADLS, ADF, Azure DevOps, etc.).
• 7+ years of Python development experience, with the ability to design and build reusable libraries.
• 4+ years of experience with Snowflake or SQL (No-SQL experience is a plus).
• Expert-level knowledge of PySpark and Spark SQL.
• Strong programming experience in Python and SQL.
• Experience with Delta Lake and Lakehouse architecture.
• Strong experience with Azure Data Lake Storage (ADLS Gen2).
• Experience with Azure Data Factory (ADF) and other Azure data services.
• Strong understanding of ETL/ELT, data modeling, and large-scale data pipelines.
• Experience with performance tuning and optimization in Databricks/Spark.
• Experience with Git, CI/CD, and DevOps practices.

Preferred Skills

• Databricks certifications.
• Experience with Unity Catalog and Databricks governance/security.
• Experience with Terraform or Infrastructure as Code.
• Knowledge of Azure DevOps.
• Experience designing APIs and integrating with React JS within a cloud platform