Senior Azure Databricks Data Engineer: Scalable Pipelines

Robotic Technologies Automation; Houston (TX); On-site; USD 120,000 - 180,000; Full time

Job summary

ROBOTIC PROCESS AUTOMATION LLC seeks a Specialist with 7 years of experience in designing and developing scalable data solutions using Azure Databricks. You will lead the design and optimization of enterprise-grade data pipelines, writing PySpark and Python code for ingestion, transformation, and analytics.

You will troubleshoot pipelines, collaborate with data architects and analysts, document processes, and mentor junior teammates to uphold best practices in data engineering.

Qualifications

• 7 years of experience designing and developing scalable data solutions using Azure Databricks.
• Experience building enterprise-level data pipelines and data platforms with strong focus on quality and reliability.

Responsibilities

• Lead design, development and enhancement of complex datapipelines using Azure Databricks.
• Write clean, maintainable PySpark and Python code for data ingestion, transformation and analytics.
• Troubleshoot and resolve issues in datapipelines to ensure minimal downtime and high availability.
• Collaborate with data architects, analysts and stakeholders to deliver scalable data solutions.
• Mentor and guide junior team members on Azure Databricks best practices in data engineering.
• Document data processes, architectures and workflows for knowledge sharing and compliance.

Skills

Azure Databricks

PySpark

Python

ETL

Tools

SQL

Airflow

Job description

ROBOTIC PROCESS AUTOMATION LLC seeks a Specialist with 7 years of experience in designing and developing scalable data solutions using Azure Databricks. You will lead the design and optimization of enterprise-grade data pipelines, writing PySpark and Python code for ingestion, transformation, and analytics.

You will troubleshoot pipelines, collaborate with data architects and analysts, document processes, and mentor junior teammates to uphold best practices in data engineering.