Lead Data Engineer

TechDigital Group

Bethlehem Township (NJ)

Hybrid

USD 150,000 - 210,000

Full time

3 days ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

TechDigital Group in New Jersey seeks a Databricks-focused Data Engineer to design scalable data pipelines, build ETL/ELT workflows on AWS, and optimize analytics with PySpark and Delta Lake. Strong SQL and shell scripting are required.

You will lead migrations from Informatica to Databricks, collaborate with business and architecture teams, perform data quality checks, and support production monitoring in a hybrid delivery model.

Qualifications

  • 10+ years of experience in Data Engineering & ETL.
  • Experience coordinating onsite and offshore teams with business stakeholders.

Responsibilities

  • Design and develop scalable data pipelines using Databricks (PySpark, Delta Lake).
  • Build and optimize ETL/ELT workflows in cloud data platforms.
  • Perform data analysis, transformation, and integration across systems.
  • Collaborate with business and architecture teams to translate requirements into solutions.
  • Lead ETL design, optimization, and migration (Informatica → Databricks).
  • Perform data quality checks, reconciliation, and performance tuning.
  • Support deployment, monitoring, and production support activities.
  • Support production issues, incident resolution, and performance tuning.

Skills

Databricks
PySpark
Delta Lake
SQL
ETL/ELT design
Cloud AWS
Informatica
Unix shell scripting
Data modeling
Data warehousing concepts

Tools

Informatica IDMC
Informatica PowerCenter
AWS

Job description

Seeking a strong Databricks-focused Data Engineer with expertise in ETL design, data pipeline development, and analytics, along with strong SQL skills. Informatica experience is preferred to support legacy integration and migration initiatives.

Key Responsibilities
  • Design and develop scalable data pipelines using Databricks (PySpark, Delta Lake)
  • Build and optimize ETL/ELT workflows in cloud data platforms
  • Perform data analysis, transformation, and integration across systems
  • Collaborate with business and architecture teams to translate requirements into solutions
  • Lead ETL design, optimization, and migration (Informatica → Databricks)
  • Perform data quality checks, reconciliation, and performance tuning
  • Support deployment, monitoring, and production support activities
  • Support production issues, incident resolution, and performance tuning
Required Skills
  • 10+ years of experience in Data Engineering & ETL
  • 8+ Years of experience in coordinating with onsite and offshore teams and business stakeholders on requirements and delivery

Strong expertise in:

  • Databricks (PySpark, Notebooks, Delta tables)
  • SQL (advanced querying, optimization, analytics)

Strong understanding of:

  • ETL/ELT design and data warehousing concepts
  • Data modeling and pipeline design

Good exposure to cloud platforms (AWS)

Unix shell scripting

Experience working in hybrid/onshore-offshore delivery model

Good-to-Have
  • Experience in:
    • Informatica IDMC & Powercenter
    • Migration from legacy ETL tools to Databricks
Core Competencies
  • Strong analytical thinking and problem-solving ability
  • Ability to work with large datasets and complex transformations
  • Strong communication and stakeholder management skills
  • Experience working in Agile teams
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer - Databricks
Senior Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 160,000
Senior Data Engineer
Senior Data Engineer

Veriipro • Atlanta (GA)

On-site
USD 140,000 - 180,000
Databricks Data Engineer
Databricks Data Engineer

Henderson Scott • Irving (TX)

On-site
USD 100,000 - 130,000
Senior Data Engineer
Senior Data Engineer

Bitful • United States

Remote
USD 120,000 - 180,000
Data Engineer
Data Engineer

TechDigital Group • Holmdel Township (NJ)

On-site
USD 130,000 - 180,000
Data Engineer - Databricks
Data Engineer - Databricks

UNAVAILABLE • McLean (VA)

On-site
USD 120,000 - 170,000
Lead Databricks Data Engineer
Lead Databricks Data Engineer

Fa Ewjt Saasfaprod1 • New York (NY)

On-site
USD 140,000 - 170,000
Senior Consultant (Databricks)
Senior Consultant (Databricks)

Unison Group • San Francisco (CA)

On-site
USD 180,000 - 250,000
Databricks Data Engineer
Databricks Data Engineer

Compunnel Inc. • Spring (TX)

On-site
USD 110,000 - 140,000
Senior Data Engineer
Senior Data Engineer

Bitful Inc. • Wadsworth (OH)

On-site
USD 110,000 - 160,000