Lead Big Data Engineer - Cloud Pipelines & Analytics

Compunnel Inc.

Atlanta (GA)

On-site

USD 120,000 - 150,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

A technology firm is seeking a Lead Data Engineer - Big Data to design and maintain cloud-based data pipelines for enterprise applications. This role involves leading data-driven solutions and collaborating closely with business stakeholders to translate their requirements into effective data strategies. Ideal candidates should have extensive experience in Data Engineering, proficiency with Palantir Foundry, and a strong background in Hadoop technologies. A Bachelor's degree in Computer Science or similar is required.

Responsibilities

  • Design and maintain cloud-based data pipelines to process enterprise data.
  • Establish best practices for data analytics technologies.
  • Lead delivery of data-driven solutions.
  • Act as liaison between business units and technical teams.
  • Support enterprise-wide reporting and analytics.
  • Guide teams on efficient data management.
  • Provide leadership and contribution in cross-functional teams.
  • Participate in the Agile SDLC.

Skills

Data Engineering experience
Subject Matter Expert (SME)
Palantir Foundry
Hadoop ecosystem technologies
Python
ETL concepts
cloud-based data platforms
NoSQL databases
relational databases
Git
analytical and problem-solving skills
communication skills

Education

Bachelor’s degree in Computer Science or related field

Job description

The Lead Data Engineer - Big Data will develop and support enterprise-scale business applications using modern Big Data technologies in a cloud-based environment. This role combines technical leadership with hands-on data engineering and close collaboration with business stakeholders. The ideal candidate will architect, build, and manage complex data pipelines while translating business requirements into actionable data solutions that drive insights, efficiency optimization, and predictive analytics.

Key Responsibilities:

  • Design, architect, and maintain cloud-based data pipelines to process and curate large-scale enterprise data
  • Establish best practices for cloud-based data analytics technologies to analyze high-volume manufacturing and financial data
  • Lead end-to-end delivery of data-driven and big data solutions
  • Act as a liaison between business units and technical teams to gather, analyze, and translate requirements
  • Support business analysts by enabling enterprise-wide reporting and analytics for leadership
  • Guide teams on efficient data and resource management using cloud-based platforms and services
  • Provide technical leadership and hands-on contribution within cross-functional teams of data engineers and analysts
  • Participate in the full Agile SDLC including design, development, unit testing, and peer reviews
Required Skills:
  • 8–10 years of strong hands-on Data Engineering experience
  • Experience working as a Subject Matter Expert (SME)
  • Hands-on experience with Palantir Foundry for designing and implementing scalable data solutions
  • Strong experience with Hadoop ecosystem technologies such as Hive and PySpark
  • Proficiency in Python or Java
  • Solid understanding and practical experience with ETL concepts and at least one ETL technology
  • Experience with cloud-based data platforms and services
  • Strong experience with NoSQL databases (HBase, MongoDB, etc.)
  • Strong experience with relational databases (SQL Server, Oracle, MySQL, etc.)
  • Experience with source control systems, specifically Git
  • Strong analytical, critical thinking, and problem-solving skills
  • Excellent communication skills for both technical and non-technical audiences
Education:

Bachelor’s degree in Computer Science or a related field

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer - Big Data
Lead Data Engineer - Big Data

Compunnel Inc. • Atlanta (GA)

On-site
USD 120,000 - 150,000
Lead Big Data Engineer: Real-Time Pipelines & Cloud
Lead Big Data Engineer: Real-Time Pipelines & Cloud

Compunnel Inc. • Plano (TX)

Hybrid
USD 140,000 - 180,000
Senior Data Engineer: Cloud Pipelines, Analytics & ML Data
Senior Data Engineer: Cloud Pipelines, Analytics & ML Data

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Senior Data Engineer - Scalable Pipelines & Cloud Analytics
Senior Data Engineer - Scalable Pipelines & Cloud Analytics

Initvalue • Fremont (CA)

Hybrid
USD 130,000 - 190,000
Lead Data Engineer: Architect Scalable Pipelines
Lead Data Engineer: Architect Scalable Pipelines

Compunnel Inc. • Columbus (OH)

On-site
USD 100,000 - 130,000
Lead Data Engineer
Lead Data Engineer

Compunnel Inc. • Columbus (OH)

On-site
USD 100,000 - 130,000
Lead Data Engineer
Lead Data Engineer

Good co India • United States

Remote
USD 120,000 - 180,000
Lead Data Engineer
Lead Data Engineer

Harnham • United States

On-site
USD 150,000 - 210,000
Senior / Lead Big Data Engineer
Senior / Lead Big Data Engineer

Compunnel Inc. • Plano (TX)

On-site
USD 140,000 - 180,000
Data Engineer: Scalable ETL & Analytics Pipelines
Data Engineer: Scalable ETL & Analytics Pipelines

Compunnel Inc. • Pennsylvania

Hybrid
USD 85,000 - 125,000