Azure Databricks Engineer

Robotic Technologies Automation; Dallas (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Robotic Technologies Automation in Dallas, TX is seeking an experienced Azure Databricks Engineer to design and optimize large-scale data pipelines and lakehouse architectures.

You will build ETL/ELT workflows with PySpark, Python, Spark SQL, and orchestrate with ADF, while ensuring data quality and cost-efficient Spark jobs. Collaborate with data architects and stakeholders and implement CI/CD deployment processes.

Qualifications

• Experience designing, developing, and optimizing large-scale data pipelines.
• Hands-on knowledge of Lakehouse architectures and Delta Lake.
• Strong SQL, Python, and Spark skills.
• Experience with Azure Data Factory and ADLS Gen2.
• Knowledge of CI/CD pipelines and deployment processes.
• Familiarity with Delta Lake and data governance concepts.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT workflows with PySpark, Python, and Spark SQL.
• Develop and optimize SQL queries and data transformations.
• Orchestrate pipelines with Azure Data Factory.
• Work with ADLS Gen2 and Lakehouse architectures.
• Implement CI/CD pipelines and deployment processes.
• Ensure data quality checks, monitoring, and error handling.
• Collaborate with data architects, analysts, and business stakeholders.

Skills

Azure Databricks

PySpark

Python

Apache Spark

Spark SQL

Delta Lake

SQL

Azure DevOps/CI-CD

Data pipeline optimization

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Terraform

Databricks Unity Catalog

Databricks Workflows

Kafka/Event Hubs

Job description

US Citizen, Green Card, H-1B, OPT-EAD, CPT, Other

Work Authorization: US Citizen, Green Card, OPT-EAD, CPT, H-4-EAD, Other

Job Description

We are looking for an experienced Azure Databricks Engineer with strong expertise in building scalable data engineering solutions using Azure Databricks, Apache Spark, Python, and Azure Data Services. The ideal candidate should have hands-on experience in designing, developing, and optimizing large-scale data pipelines and Lakehouse architectures.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT workflows using PySpark, Python, and Spark SQL.
• Develop and optimize complex SQL queries and data transformations.
• Work with Azure Data Factory (ADF) for orchestration and workflow automation.
• Work with Azure Data Lake Storage Gen2 (ADLS).
• Perform data modelling, data integration, and data migration.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Implement data quality checks, monitoring, and error handling.
• Collaborate with data architects, analysts, and business stakeholders.
• Implement CI/CD pipelines and deployment processes.

Required Technical Skills

• Strong hands-on experience with Azure Databricks.
• Excellent programming skills in Python and PySpark.
• Strong knowledge of Apache Spark and Spark SQL.
• Experience with Azure Data Factory (ADF).
• Hands-on experience with Azure Data Lake Storage Gen2.
• Strong understanding of Delta Lake and Lakehouse architecture.
• Expertise in SQL and data warehousing concepts.
• Experience with Azure DevOps and CI/CD.
• Knowledge of data pipeline optimization and performance tuning.

Good to Have

• Experience with Unity Catalog and Databricks Workflows.
• Knowledge of Databricks Asset Bundles.
• Experience with Azure Synapse Analytics.
• Knowledge of Terraform and infrastructure automation.
• Experience with streaming technologies such as Kafka or Event Hubs.
• Understanding of data governance and security.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.