Lead Data Engineer

Celebal Technologies; New York (NY); Remote; USD 100,000 - 200,000; Full time

Job summary

A leading technology company seeks a Lead Data Engineer to spearhead data architecture initiatives. This remote position requires hands-on expertise in Databricks and advanced data solutions while collaborating with cross-functional teams. The ideal candidate will play a vital role in optimizing data delivery and ensuring quality management across products.

Qualifications

• Proven experience in data engineering roles focusing on Databricks and AWS.
• Strong knowledge of Medallion Architecture and data lakehouse design.
• Ability to mentor junior engineers and lead data projects.

Responsibilities

• Lead the design and implementation of scalable data pipelines.
• Architect and optimize data solutions leveraging Medallion Architecture.
• Collaborate with teams to meet and translate business requirements.

Skills

Databricks

PySpark

ETL frameworks

SQL

NoSQL technologies

Job description

Get AI-powered advice on this job and more exclusive features.

Sign in to access AI-powered advices

Continue with Google Continue with Google

Continue with Google Continue with Google

Continue with Google Continue with Google

Continue with Google Continue with Google

Continue with Google Continue with Google

Continue with Google Continue with Google

Direct message the job poster from Celebal Technologies

Senior Talent Acquisition Leader | Architect of High-Performing Global Teams | Software Development & Tech Hiring Strategist

Dear Applicant,

Please find following job description.

Position - Full Time

Start Date - Immediate

Location - Remote

Position Overview

The Lead Data Engineer is responsible for leading the design, development, and implementation of robust data pipelines and architectures that support the organization’s data-driven decision-making. This role requires deep expertise in modern data platforms, particularly Databricks, and the ability to architect scalable solutions using Medallion Architecture principles. The Lead Data Engineer will collaborate closely with data scientists, analysts, and business stakeholders to ensure the availability, reliability, and quality of enterprise data.

Key Responsibilities

• Lead the design and implementation of scalable data pipelines using Databricks, PySpark, and ETL frameworks.
• Architect and optimize data solutions leveraging Medallion Architecture to ensure data quality, security, and governance throughout the data lifecycle.
• Develop and maintain data models, schemas, and databases using SQL and NoSQL technologies.
• Oversee the integration of data from various sources into cloud-based platforms, primarily AWS, ensuring efficient and reliable data ingestion and processing.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
• Mentor and guide junior data engineers in best practices, code quality, and data engineering methodologies.
• Monitor and optimize the performance of data pipelines and storage systems, identifying and resolving bottlenecks and inefficiencies.
• Ensure adherence to data standards, security policies, and compliance requirements across all data engineering activities.
• Develop and maintain dashboards and reporting tools to provide actionable insights for stakeholders7.
• Stay current with emerging technologies and industry trends in data engineering and cloud platforms.

Required Skills and Qualifications

• Proven experience as a Lead Data Engineer or similar role with hands-on expertise in Databricks, PySpark, and ETL tools.
• Strong knowledge of Medallion Architecture and best practices in data lakehouse design.
• Extensive experience with AWS cloud services (e.g., S3, Glue, Lambda, Redshift, Athena).
• Advanced proficiency in SQL and experience with relational and non-relational databases.
• Excellent problem-solving, analytical, and communication skills.

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Information Technology

Referrals increase your chances of interviewing at Celebal Technologies by 2x

Get notified about new Data Engineer jobs in New York, NY.

New York, NY $115,000 - $145,000 4 days ago

Manhattan, NY $110,000 - $160,000 3 weeks ago

New York, NY
$110,000.00
-
$140,000.00
2 weeks ago

New York, NY
$140,000.00
-
$190,000.00
1 week ago

New York, NY
$150,000.00
-
$180,000.00
1 day ago

New York, NY
$160,000.00
-
$200,000.00
1 week ago

New York City Metropolitan Area
$135,000.00
-
$155,000.00
1 week ago

Yonkers, NY
$110,000.00
-
$120,000.00
4 days ago

New York, NY
$145,000.00
-
$260,000.00
7 months ago

New York, NY
$100,000.00
-
$150,000.00
4 weeks ago

Brooklyn, NY
$80,000.00
-
$100,000.00
15 hours ago

New York, NY
$120,000.00
-
$240,000.00
2 weeks ago

New York, NY $35,000 - $46,000 1 month ago

New York, NY $150,000 - $190,000 3 months ago

Newark, NJ $80,000 - $100,000 15 hours ago

New York, United States $115,000 - $120,000 1 week ago

New York, NY $70,000 - $150,000 4 months ago

Data and Machine Learning Engineer- Fully Remote!

New York, NY $110,000 - $140,000 1 day ago

New York, NY $117,300 - $125,500 2 days ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.