Principal Data Engineer

Good co India; United States; À distance; USD 130 000 - 170 000; Plein temps

Résumé du poste

Good co India is seeking a data engineering lead to design scalable, secure data architectures and modern data platforms on cloud providers. You will drive standards, build batch and streaming pipelines, and mentor engineers across teams to deliver data-driven solutions.

You will work with Python/SQL, Spark, Kafka, and orchestration tools, ensuring robust data delivery, governance, and cost-effective cloud infrastructure across AWS, Azure, or GCP.

Qualifications

• 5 to 10 years of experience in data engineering or distributed data processing, with demonstrated technical leadership and architecture ownership.
• Strong proficiency in Python and SQL, with experience in Scala or Java.
• Extensive hands-on experience with Apache Spark, PySpark, Kafka, Hadoop, and large-scale data processing frameworks.

Responsabilités

• Lead the design and implementation of scalable, secure, and high-performance enterprise data architectures.
• Define data engineering standards, architectural guidelines, and best practices across data platforms and pipelines.
• Design and optimize batch and real-time data processing solutions using Apache Spark, Kafka, and other distributed computing technologies.
• Architect and manage modern data platforms, including data lakes, data warehouses, and lakehouse solutions.
• Drive cloud-based data engineering initiatives across AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Develop and optimize complex ETL/ELT pipelines to ensure reliable, efficient, and high-quality data delivery.
• Establish data governance, security, privacy, quality, and compliance frameworks across enterprise data systems.
• Lead technical design reviews, evaluate emerging technologies, and recommend solutions aligned with business objectives.
• Mentor and guide data engineers, promote engineering excellence, and support technical capability development.
• Collaborate with data scientists, analytics teams, architects, product managers, and business stakeholders to deliver data-driven solutions.
• Optimize data platform performance, scalability, availability, and cloud infrastructure costs.
• Implement automated testing, CI/CD, monitoring, orchestration, and infrastructure-as-code practices.
• Troubleshoot complex production issues and drive continuous improvement in data reliability and operational efficiency.

Connaissances

Python

SQL

Scala

Java

Apache Spark

Kafka

Hadoop

Databricks

Snowflake

Delta Lake

Airflow

CI/CD

Infrastructure as Code

AWS

Azure

GCP

data lakes

data warehouses

lakehouse

ETL/ELT

data governance

security

monitoring

Formation

Bachelor's degree in Computer Science, Information Technology, Software Engineering, or a related technical discipline

Outils

Databricks

Snowflake

Delta Lake

Apache Airflow

Description du poste

Role & responsibilities

• Lead the design and implementation of scalable, secure, and high-performance enterprise data architectures.
• Define data engineering standards, architectural guidelines, and best practices across data platforms and pipelines.
• Design and optimize batch and real-time data processing solutions using Apache Spark, Kafka, and other distributed computing technologies.
• Architect and manage modern data platforms, including data lakes, data warehouses, and lakehouse solutions.
• Drive cloud-based data engineering initiatives across AWS, Microsoft Azure, or Google Cloud Platform (GCP).
• Develop and optimize complex ETL/ELT pipelines to ensure reliable, efficient, and high-quality data delivery.
• Establish data governance, security, privacy, quality, and compliance frameworks across enterprise data systems.
• Lead technical design reviews, evaluate emerging technologies, and recommend solutions aligned with business objectives.
• Mentor and guide data engineers, promote engineering excellence, and support technical capability development.
• Collaborate with data scientists, analytics teams, architects, product managers, and business stakeholders to deliver data-driven solutions.
• Optimize data platform performance, scalability, availability, and cloud infrastructure costs.
• Implement automated testing, CI/CD, monitoring, orchestration, and infrastructure-as-code practices.
• Troubleshoot complex production issues and drive continuous improvement in data reliability and operational efficiency.

Preferred candidate profile

• Bachelors degree in Computer Science, Information Technology, Software Engineering, or a related technical discipline.
• 5 to 10 years of experience in data engineering, big data platforms, or distributed data processing, with demonstrated technical leadership and architecture ownership.
• Strong proficiency in Python and SQL, with experience in Scala or Java.
• Extensive hands-on experience with Apache Spark, PySpark, Kafka, Hadoop, and large-scale data processing frameworks.
• Proven expertise in designing enterprise-grade data pipelines, data models, data warehouses, data lakes, and lakehouse architectures.
• Strong experience with cloud data platforms and services across AWS, Microsoft Azure, or Google Cloud Platform.
• Familiarity with Databricks, Snowflake, Delta Lake, Apache Airflow, and modern data orchestration frameworks.
• Strong understanding of distributed systems, system design, data governance, security, and data quality management.
• Experience with CI/CD, automated testing, infrastructure as code, observability, and production deployment practices.
• Demonstrated ability to lead technical initiatives, establish engineering standards, mentor team members, and influence architectural decisions.
• Excellent analytical, problem-solving, communication, and stakeholder management skills.
• Relevant cloud, data engineering, or architecture certifications are advantageous; a Master’s degree is preferred but not mandatory.