Lead Data Engineer – Databricks

Apptad; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

Apptad is seeking a Lead Data Engineer to design and implement scalable data solutions using Databricks, Advanced SQL, and AWS. You will own end-to-end data pipelines, optimize performance, and drive data governance.

The role emphasizes Medallion Architecture (Bronze-Silver-Gold), Delta Tables, Unity Catalog, and integration with AWS services like S3, SQS, and SNS. You will collaborate with cross-functional teams and mentor juniors.

Qualifications

• 5+ years of hands-on experience with Databricks.

Responsibilities

• Build and optimize data pipelines on Databricks with Delta Lake.

Skills

Databricks

SQL

PySpark

Performance tuning

Data pipelines

DBU optimization

Ingestion pipelines

JSON parsing

XML parsing

APIs ingestion

Delta tables

Databricks Auto Loader

Unity Catalog

AWS services

S3

SQS

SNS

Medallion Architecture

Tools

Databricks

Databricks Auto Loader

Unity Catalog

Delta Tables

Medallion Architecture

AWS

S3

SQS

SNS

Job description

We are looking for an experienced Lead Data Engineer with strong hands-on expertise in Databricks, Advanced SQL, and AWS to join our team!

• 5+ years of hands-on experience with Databricks.
• Strong expertise in Medallion Architecture (Bronze, Silver & Gold).
• Advanced SQL, PySpark, and performance tuning.
• Pipeline optimization and DBU cost optimization across All-Purpose and Job Clusters.
• Data ingestion from multiple source systems.
• Parsing and processing JSON/XML data.
• Ingesting data from APIs into Delta Tables.
• Hands-on experience with Databricks Auto Loader.
• Publishing data to AWS services such as S3, SQS, and SNS.
• Experience working with relational and non-relational databases.
• Strong proficiency in Unity Catalog.

Good to Have:

• DBUtils commands.
• Knowledge of broader AWS services.

Looking for candidates with strong hands-on Databricks experience and expertise in building, optimizing, and managing scalable data engineering solutions.