Lead Data Engineer

Good co India

United States

Remote

USD 120,000 - 180,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Good co India seeks an experienced Data Engineer to design, build, and optimize scalable data pipelines and platforms in a modern cloud environment.

You will work with Spark, Kafka, Airflow, and cloud services to deliver reliable ETL/ELT processes, data warehouses, and data lakes. Collaborate with data scientists and software engineers to enable data-driven decisions, while promoting best practices in governance and CI/CD.

Qualifications

  • Bachelor's or master's degree in computer science, information technology, data science, software engineering, or a related technical field.
  • 5–10 years of professional experience in data engineering, big data development, or data platform engineering.
  • Strong proficiency in Python, SQL, PySpark, and large-scale data processing.
  • Hands-on experience designing and developing complex ETL/ELT pipelines and data integration workflows.
  • Strong knowledge of Apache Spark, Kafka, Airflow, Hadoop, and distributed data processing frameworks.
  • Experience with modern data platforms such as Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
  • Proven expertise in cloud technologies, including AWS, Azure, or Google Cloud Platform.
  • Strong understanding of data warehousing, data lakes, lakehouse architecture, data modeling, and database optimization.
  • Experience with real-time streaming, batch processing, and workflow orchestration.
  • Knowledge of data governance, data quality frameworks, metadata management, and data security practices.
  • Experience with Docker, Kubernetes, Git, CI/CD pipelines, Terraform, and infrastructure automation.
  • Ability to lead technical projects, define data architecture, and establish engineering best practices.
  • Proven experience mentoring engineers, conducting code reviews, and collaborating with cross-functional teams.
  • Strong analytical, problem-solving, communication, and stakeholder-management skills.
  • Ability to translate business requirements into scalable and efficient data engineering solutions.

Responsibilities

  • Lead the design, development, and implementation of scalable data engineering solutions.
  • Architect and maintain ETL/ELT pipelines for batch and real-time processing.
  • Lead data warehouse, data lake, and lakehouse architecture development.
  • Design and manage data ingestion, transformation, and orchestration workflows.
  • Work with large-scale datasets using Spark, PySpark, Kafka, Hadoop, and similar tech.
  • Build cloud-based data platforms on AWS, Azure, or GCP.
  • Develop and optimize data models, schemas, and processing frameworks.
  • Ensure data quality, security, governance, and regulatory compliance across platforms.
  • Implement workflow orchestration and automation using Airflow, dbt, and related tools.
  • Monitor pipelines, troubleshoot failures, and optimize performance and costs.
  • Establish data engineering standards, documentation, and CI/CD practices.
  • Collaborate with data scientists, analysts, software engineers, and stakeholders.
  • Lead architecture reviews, code reviews, and technology evaluations.
  • Mentor junior/mid-level data engineers and provide guidance.

Skills

Python
SQL
PySpark
Big data
Spark
Kafka
Airflow
Hadoop
Docker
Kubernetes
Git
CI/CD
Terraform
Cloud platforms
Data modeling
Data warehousing
Data lakes

Education

Bachelor's/Master's in CS/Data Science

Tools

Snowflake
Databricks
Redshift
BigQuery
Azure Synapse

Job description

Role & responsibilities

  • Lead the design, development, and implementation of scalable, reliable, and high-performance data engineering solutions.
  • Architect, build, and maintain robust ETL/ELT pipelines for batch and real-time data processing.
  • Lead the development and optimization of data warehouses, data lakes, and lakehouse architectures.
  • Design and manage data ingestion, transformation, integration, and orchestration workflows.
  • Work with large-scale datasets using Apache Spark, PySpark, Kafka, Hadoop, and other big data technologies.
  • Implement and manage cloud-based data platforms using AWS, Microsoft Azure, or Google Cloud Platform.
  • Develop and optimize data models, database schemas, and data processing frameworks.
  • Ensure data quality, consistency, security, governance, and regulatory compliance across data platforms.
  • Implement workflow orchestration and automation using Apache Airflow, dbt, and related tools.
  • Monitor data pipelines, troubleshoot failures, and optimize system performance and infrastructure costs.
  • Establish data engineering standards, best practices, documentation, and CI/CD processes.
  • Collaborate with data scientists, data analysts, software engineers, and business stakeholders to deliver data-driven solutions.
  • Lead technical discussions, architecture reviews, code reviews, and technology evaluations.
  • Mentor junior and mid-level data engineers and provide technical guidance to the engineering team.
  • Drive continuous improvements in data infrastructure, scalability, reliability, and operational efficiency.

Preferred candidate profile

  • Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, Software Engineering, or a related technical field.
  • 5 to 10 years of professional experience in data engineering, big data development, or data platform engineering.
  • Strong proficiency in Python, SQL, PySpark, and experience with large-scale data processing.
  • Hands-on experience designing and developing complex ETL/ELT pipelines and data integration workflows.
  • Strong knowledge of Apache Spark, Kafka, Airflow, Hadoop, and distributed data processing frameworks.
  • Experience with modern data platforms such as Snowflake, Databricks, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
  • Proven expertise in cloud technologies, including AWS, Azure, or Google Cloud Platform.
  • Strong understanding of data warehousing, data lakes, lakehouse architecture, data modeling, and database optimization.
  • Experience with real-time streaming, batch processing, and workflow orchestration.
  • Knowledge of data governance, data quality frameworks, metadata management, and data security practices.
  • Experience with Docker, Kubernetes, Git, CI/CD pipelines, Terraform, and infrastructure automation.
  • Ability to lead technical projects, define data architecture, and establish engineering best practices.
  • Proven experience mentoring engineers, conducting code reviews, and collaborating with cross-functional teams.
  • Strong analytical, problem-solving, communication, and stakeholder-management skills.
  • Ability to translate business requirements into scalable and efficient data engineering solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Big Data Engineer
Big Data Engineer

Good co India • United States

Remote
USD 120,000 - 180,000
Lead Big Data Engineer: Real-Time Pipelines & Cloud
Lead Big Data Engineer: Real-Time Pipelines & Cloud

Compunnel Inc. • Plano (TX)

Hybrid
USD 140,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Gennte Technologies • Pittsburgh

Hybrid
USD 100,000 - 140,000
Senior Data Engineer: Cloud Pipelines, Analytics & ML Data
Senior Data Engineer: Cloud Pipelines, Analytics & ML Data

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Insomniac Design • United States

On-site
USD 120,000 - 160,000
Data Engineering Team Lead
Data Engineering Team Lead

UNAVAILABLE • McLean (VA)

On-site
USD 150,000 - 190,000
Senior / Lead Big Data Engineer
Senior / Lead Big Data Engineer

Compunnel Inc. • Plano (TX)

On-site
USD 140,000 - 180,000
Senior Data Engineer
Senior Data Engineer

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Senior Data Engineer
Senior Data Engineer

Tap Growth ai • Pittsburgh

On-site
USD 140,000 - 180,000
Senior Data Engineer - Scalable Pipelines & Cloud Analytics
Senior Data Engineer - Scalable Pipelines & Cloud Analytics

Initvalue • Fremont (CA)

Hybrid
USD 130,000 - 190,000