Big Data Engineer

Good co India; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Good co India is seeking a senior data engineer to design, develop, and maintain scalable big data pipelines for batch and real-time processing across cloud platforms.

The role requires 5–10 years of experience, strong Python and SQL, Spark, Kafka, and expertise in data lakes/lakehouse architectures, plus collaboration with cross-functional teams.

Qualifications

• Minimum 5–10 years in big data engineering or distributed data processing.
• Strong Python and SQL skills with exposure to Scala or Java.
• Hands-on experience with Spark, PySpark, Hadoop, Hive, Kafka, and related tools.
• Proven ability to design scalable batch and real-time data pipelines.
• Familiarity with data lakes, lakehouse architectures, and cloud platforms.

Responsibilities

• Design, develop, and maintain scalable big data pipelines for batch and real-time processing.
• Build and optimize data ingestion, transformation, and ETL/ELT workflows from diverse sources.
• Develop distributed data processing solutions using Spark, PySpark, Hadoop, Hive, and Kafka.
• Implement real-time streaming via Kafka and other platforms.
• Design data lakes, data warehouses, and lakehouse architectures for analytics.
• Work with cloud services across AWS, Azure, or GCP for data engineering tasks.
• Optimize processing performance, storage usage, and pipeline reliability.
• Implement data quality, governance, security, and monitoring standards.
• Collaborate with data scientists, analysts, engineers, and stakeholders.
• Automate orchestration with Airflow and CI/CD; troubleshoot production issues.
• Document architectures, workflows, and operational procedures.

Skills

Python

SQL

Scala/Java

Apache Spark

PySpark

Hadoop

Kafka

Airflow

Git

CI/CD

Linux

Snowflake

Delta Lake

Databricks

Data governance

Data quality

Communication

Problem solving

Education

Bachelor's degree in Computer Science / Information Technology / Software Engineering / Data Engineering

Tools

Databricks

Snowflake

Delta Lake

Apache Airflow

Git

Linux

CI/CD practices

Job description

Role & responsibilities

• Design, develop, and maintain scalable big data pipelines for batch and real-time data processing.
• Build and optimize data ingestion, transformation, and ETL/ELT workflows from multiple structured and unstructured data sources.
• Develop distributed data processing solutions using Apache Spark, PySpark, Hadoop, and related big data technologies.
• Implement real-time streaming solutions using Apache Kafka and other streaming platforms.
• Design and manage data lakes, data warehouses, and lakehouse architectures to support analytics and business intelligence.
• Develop and maintain data pipelines on cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Optimize data processing performance, storage utilization, query execution, and pipeline reliability.
• Implement data quality checks, validation, monitoring, security, and governance standards.
• Collaborate with data scientists, data analysts, software engineers, and business stakeholders to deliver reliable data solutions.
• Automate pipeline orchestration, scheduling, testing, and deployment using tools such as Apache Airflow and CI/CD frameworks.
• Troubleshoot production issues, resolve data processing bottlenecks, and ensure data availability and consistency.
• Document data architectures, workflows, technical specifications, and operational procedures.

Preferred candidate profile

• Bachelors degree in Computer Science, Information Technology, Software Engineering, Data Engineering, or a related technical discipline.
• 5 to 10 years of experience in big data engineering, data engineering, distributed computing, or large-scale data processing.
• Strong proficiency in Python, SQL, and preferably Scala or Java.
• Hands-on experience with Apache Spark, PySpark, Hadoop, Hive, Kafka, and distributed data processing frameworks.
• Proven experience designing and maintaining scalable batch and real-time data pipelines.
• Strong understanding of ETL/ELT processes, data modeling, data warehousing, data lakes, and lakehouse architectures.
• Experience with cloud data engineering services across AWS, Microsoft Azure, or Google Cloud Platform.
• Familiarity with Databricks, Snowflake, Delta Lake, Apache Airflow, Git, Linux, and CI/CD practices.
• Knowledge of data quality, data governance, security, access control, and pipeline monitoring.
• Strong analytical, debugging, problem-solving, and performance optimization skills.
• Ability to collaborate effectively with cross-functional teams and communicate complex technical concepts clearly.
• Relevant cloud or data engineering certifications are an advantage; a Master’s degree is preferred but not mandatory.