Data Engineer — Databricks & ADF Pipeline Architect

Cloud Analytics Technologies, LLC

Atlanta (GA)

On-site

USD 110,000 - 150,000

Full time

3 hours ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Cloud Analytics Technologies, LLC is seeking a Data Engineer to design and optimize data pipelines using Databricks and Azure Data Factory (ADF). The role involves SQL-based data transformation, PySpark/Python processing, and integrating data from multiple sources into data lakes or data warehouses.

You will monitor pipelines, troubleshoot issues, and collaborate with teams to ensure data quality and reliability in Azure cloud environments. Experience with ETL/ELT concepts is essential.

Qualifications

Experience with Databricks and Azure Data Factory (ADF).
Strong understanding of ETL/ELT concepts.
Good knowledge of SQL and data transformations.
Basic experience with Python or PySpark.
Understanding of Azure cloud services and data lake concepts.

Responsibilities

Develop and maintain ETL/ELT pipelines using Azure Data Factory (ADF).
Work with Databricks for data processing and transformation.
Write and optimize SQL queries for data extraction and transformation.
Work with PySpark or Python for large-scale data processing.
Integrate data from multiple sources into data warehouses or data lakes.
Monitor, troubleshoot, and optimize data pipelines.
Collaborate with teams to ensure data quality and reliability.

Skills

ETL/ELT concepts

SQL

Python

PySpark

Databricks

ADF / Azure Data Factory

Azure cloud services

Data lake concepts

Tools

Databricks

Azure Data Factory

Job description

You will monitor pipelines, troubleshoot issues, and collaborate with teams to ensure data quality and reliability in Azure cloud environments. Experience with ETL/ELT concepts is essential.

Get your free, confidential resume review.

or drag and drop your file here.