Lead Data Engineer - Big Data

Compunnel, Inc.; Atlanta (GA); On-site; USD 120,000 - 150,000; Full time

Job summary

A technology firm is seeking a Lead Data Engineer - Big Data to design and maintain cloud-based data pipelines for enterprise applications. This role involves leading data-driven solutions and collaborating closely with business stakeholders to translate their requirements into effective data strategies. Ideal candidates should have extensive experience in Data Engineering, proficiency with Palantir Foundry, and a strong background in Hadoop technologies. A Bachelor's degree in Computer Science or similar is required.

Responsibilities

• Design and maintain cloud-based data pipelines to process enterprise data.
• Establish best practices for data analytics technologies.
• Lead delivery of data-driven solutions.
• Act as liaison between business units and technical teams.
• Support enterprise-wide reporting and analytics.
• Guide teams on efficient data management.
• Provide leadership and contribution in cross-functional teams.
• Participate in the Agile SDLC.

Skills

Data Engineering experience

Subject Matter Expert (SME)

Palantir Foundry

Hadoop ecosystem technologies

Python

ETL concepts

cloud-based data platforms

NoSQL databases

relational databases

Git

analytical and problem-solving skills

communication skills

Education

Bachelor’s degree in Computer Science or related field

Job description

The Lead Data Engineer - Big Data will develop and support enterprise-scale business applications using modern Big Data technologies in a cloud-based environment. This role combines technical leadership with hands‑on data engineering and close collaboration with business stakeholders. The ideal candidate will architect, build, and manage complex data pipelines while translating business requirements into actionable data solutions that drive insights, efficiency optimization, and predictive analytics.

Key Responsibilities

• Design, architect, and maintain cloud-based data pipelines to process and curate large-scale enterprise data
• Establish best practices for cloud-based data analytics technologies to analyze high-volume manufacturing and financial data
• Lead end-to-end delivery of data-driven and big data solutions
• Act as a liaison between business units and technical teams to gather, analyze, and translate requirements
• Support business analysts by enabling enterprise-wide reporting and analytics for leadership
• Guide teams on efficient data and resource management using cloud-based platforms and services
• Provide technical leadership and hands‑on contribution within cross‑functional teams of data engineers and analysts
• Participate in the full Agile SDLC including design, development, unit testing, and peer reviews

Required Skills

• 8–10 years of strong hands‑on Data Engineering experience
• Experience working as a Subject Matter Expert (SME)
• Hands‑on experience with Palantir Foundry for designing and implementing scalable data solutions
• Strong experience with Hadoop ecosystem technologies such as Hive and PySpark
• Proficiency in Python or Java
• Solid understanding and practical experience with ETL concepts and at least one ETL technology
• Experience with cloud-based data platforms and services
• Strong experience with NoSQL databases (HBase, MongoDB, etc.)
• Strong experience with relational databases (SQL Server, Oracle, MySQL, etc.)
• Experience with source control systems, specifically Git
• Strong analytical, critical thinking, and problem‑solving skills
• Excellent communication skills for both technical and non‑technical audiences

Education

Bachelor’s degree in Computer Science or a related field