Azure Databricks Data Engineer - Lakehouse & Pipelines

Robotic Technologies Automation; Dallas (TX); On-site; USD 120,000 - 160,000; Full time

Job summary

Robotic Technologies Automation in Dallas, TX is seeking an experienced Azure Databricks Engineer to design and optimize large-scale data pipelines and lakehouse architectures.

You will build ETL/ELT workflows with PySpark, Python, Spark SQL, and orchestrate with ADF, while ensuring data quality and cost-efficient Spark jobs. Collaborate with data architects and stakeholders and implement CI/CD deployment processes.

Qualifications

• Experience designing, developing, and optimizing large-scale data pipelines.
• Hands-on knowledge of Lakehouse architectures and Delta Lake.
• Strong SQL, Python, and Spark skills.
• Experience with Azure Data Factory and ADLS Gen2.
• Knowledge of CI/CD pipelines and deployment processes.
• Familiarity with Delta Lake and data governance concepts.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Build ETL/ELT workflows with PySpark, Python, and Spark SQL.
• Develop and optimize SQL queries and data transformations.
• Orchestrate pipelines with Azure Data Factory.
• Work with ADLS Gen2 and Lakehouse architectures.
• Implement CI/CD pipelines and deployment processes.
• Ensure data quality checks, monitoring, and error handling.
• Collaborate with data architects, analysts, and business stakeholders.

Skills

Azure Databricks

PySpark

Python

Apache Spark

Spark SQL

Delta Lake

SQL

Azure DevOps/CI-CD

Data pipeline optimization

Tools

Azure Data Factory

Azure Data Lake Storage Gen2

Terraform

Databricks Unity Catalog

Databricks Workflows

Kafka/Event Hubs

Job description

Robotic Technologies Automation in Dallas, TX is seeking an experienced Azure Databricks Engineer to design and optimize large-scale data pipelines and lakehouse architectures.

You will build ETL/ELT workflows with PySpark, Python, Spark SQL, and orchestrate with ADF, while ensuring data quality and cost-efficient Spark jobs. Collaborate with data architects and stakeholders and implement CI/CD deployment processes.