Lead Data Engineer: Scale Data Lakes & Real-Time Pipelines

Good co India

United States

Remote

USD 120,000 - 180,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Good co India seeks an experienced Data Engineer to design, build, and optimize scalable data pipelines and platforms in a modern cloud environment.

You will work with Spark, Kafka, Airflow, and cloud services to deliver reliable ETL/ELT processes, data warehouses, and data lakes. Collaborate with data scientists and software engineers to enable data-driven decisions, while promoting best practices in governance and CI/CD.

Qualifications

  • Bachelor's or master's degree in computer science, information technology, data science, software engineering, or a related technical field.
  • 5–10 years of professional experience in data engineering, big data development, or data platform engineering.
  • Strong proficiency in Python, SQL, PySpark, and large-scale data processing.
  • Hands-on experience designing and developing complex ETL/ELT pipelines and data integration workflows.
  • Strong knowledge of Apache Spark, Kafka, Airflow, Hadoop, and distributed data processing frameworks.
  • Experience with modern data platforms such as Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
  • Proven expertise in cloud technologies, including AWS, Azure, or Google Cloud Platform.
  • Strong understanding of data warehousing, data lakes, lakehouse architecture, data modeling, and database optimization.
  • Experience with real-time streaming, batch processing, and workflow orchestration.
  • Knowledge of data governance, data quality frameworks, metadata management, and data security practices.
  • Experience with Docker, Kubernetes, Git, CI/CD pipelines, Terraform, and infrastructure automation.
  • Ability to lead technical projects, define data architecture, and establish engineering best practices.
  • Proven experience mentoring engineers, conducting code reviews, and collaborating with cross-functional teams.
  • Strong analytical, problem-solving, communication, and stakeholder-management skills.
  • Ability to translate business requirements into scalable and efficient data engineering solutions.

Responsibilities

  • Lead the design, development, and implementation of scalable data engineering solutions.
  • Architect and maintain ETL/ELT pipelines for batch and real-time processing.
  • Lead data warehouse, data lake, and lakehouse architecture development.
  • Design and manage data ingestion, transformation, and orchestration workflows.
  • Work with large-scale datasets using Spark, PySpark, Kafka, Hadoop, and similar tech.
  • Build cloud-based data platforms on AWS, Azure, or GCP.
  • Develop and optimize data models, schemas, and processing frameworks.
  • Ensure data quality, security, governance, and regulatory compliance across platforms.
  • Implement workflow orchestration and automation using Airflow, dbt, and related tools.
  • Monitor pipelines, troubleshoot failures, and optimize performance and costs.
  • Establish data engineering standards, documentation, and CI/CD practices.
  • Collaborate with data scientists, analysts, software engineers, and stakeholders.
  • Lead architecture reviews, code reviews, and technology evaluations.
  • Mentor junior/mid-level data engineers and provide guidance.

Skills

Python
SQL
PySpark
Big data
Spark
Kafka
Airflow
Hadoop
Docker
Kubernetes
Git
CI/CD
Terraform
Cloud platforms
Data modeling
Data warehousing
Data lakes

Education

Bachelor's/Master's in CS/Data Science

Tools

Snowflake
Databricks
Redshift
BigQuery
Azure Synapse

Job description

Good co India seeks an experienced Data Engineer to design, build, and optimize scalable data pipelines and platforms in a modern cloud environment.

You will work with Spark, Kafka, Airflow, and cloud services to deliver reliable ETL/ELT processes, data warehouses, and data lakes. Collaborate with data scientists and software engineers to enable data-driven decisions, while promoting best practices in governance and CI/CD.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Lead Data Engineer: Real-Time Pipelines & Spark/Kafka
Lead Data Engineer: Real-Time Pipelines & Spark/Kafka

Mode.Inc • United States

Remote
USD 140,000 - 210,000
Senior Data Engineer - Scalable Pipelines & Cloud Analytics
Senior Data Engineer - Scalable Pipelines & Cloud Analytics

Initvalue • Fremont (CA)

Hybrid
USD 130,000 - 190,000
Lead Data Engineer
Lead Data Engineer

TechDigital Group • Warsaw (IN)

On-site
USD 80,000 - 120,000
Senior Data Engineer: Real-Time Pipelines & Cloud Analytics
Senior Data Engineer: Real-Time Pipelines & Cloud Analytics

Cloudndata • United States

Hybrid
USD 140,000 - 180,000
Lead Data Platform Engineer - Architect & Scale Pipelines
Lead Data Platform Engineer - Architect & Scale Pipelines

Mirion Technologies • United States

On-site
USD 120,000 - 160,000
Lead Data Engineer
Lead Data Engineer

Good co India • United States

Remote
USD 120,000 - 180,000
Senior Data Engineer: Lead Scalable ETL & AI Pipelines
Senior Data Engineer: Lead Scalable ETL & AI Pipelines

Iscale Solutions • United States

Remote
USD 120,000 - 180,000
Salary package
Vacation and sick leave
Health coverage
+6
Lead Data Engineer
Lead Data Engineer

Harnham • United States

On-site
USD 150,000 - 210,000
Senior Data Engineer - Scalable Pipelines & ML-Ready Data
Senior Data Engineer - Scalable Pipelines & ML-Ready Data

Compunnel Inc. • Jersey City (NJ)

On-site
USD 100,000 - 130,000
Lead Data Platform Engineer
Lead Data Platform Engineer

Scale AI, Inc. • New York (NY)

On-site
USD 252,000 - 315,000
Health coverage
Dental coverage
Vision coverage
+3