Data Engineer (Databrick + Pyspark)

Capco; India; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

Capco is seeking a skilled Data Engineer for their Pune office, offering hybrid working arrangements. The role requires strong expertise in PySpark and Databricks, focusing on developing scalable data pipelines and utilizing cloud-based data platforms.

The ideal candidate should have 5-9 years of experience, with proficiency in SQL and ETL processes, including using Apache Airflow for workflow management. Opportunities for growth in a dynamic team await the right individual.

Qualifications

• 5–9 years of experience in data engineering roles.
• Strong hands-on experience with PySpark and Apache Spark internals.
• Proficiency in SQL and experience with Data Warehouse (DWH) systems.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Work extensively with PySpark and Apache Spark for large-scale data processing.
• Build and manage workflows using Apache Airflow.

Skills

PySpark

Apache Spark

Databricks

SQL

Apache Airflow

ETL/ELT pipelines

Cloud data lakes

Job description

Job Title: Data Engineer (PySpark / Databricks)

Experience: 5–9 Years

Location: Pune (Hybrid – Capco Office)

Job Summary

We are looking for a skilled Data Engineer with strong expertise in PySpark, Databricks, and modern data engineering practices. The ideal candidate will have hands‑on experience in building scalable data pipelines, working with large datasets, and leveraging cloud‑based data platforms.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Work extensively with PySpark and Apache Spark for large‑scale data processing.
• Build and manage workflows using Apache Airflow.
• Develop and optimize data solutions on Databricks (Jobs, Delta Lake).
• Work with cloud‑based data lakes (S3 or equivalent).
• Write efficient and complex SQL queries for data transformation and analysis.
• Run and manage Spark workloads on EMR Serverless or other managed Spark platforms.
• Ensure data quality, reliability, and performance optimization of pipelines.

Must Have Skills

• Strong hands‑on experience with PySpark and Apache Spark internals.
• Experience with Databricks (Jobs, Delta Lake).
• Proficiency in Apache Airflow for workflow orchestration.
• Solid experience building ETL/ELT pipelines at scale.
• Strong SQL skills and experience with Data Warehouse (DWH) systems.
• Experience running Spark workloads on EMR Serverless or managed Spark platforms.
• Hands‑on experience with cloud data lakes (S3 or equivalent).

Good to Have Skills

• Experience with Delta Lake / Apache Iceberg.
• Exposure to streaming frameworks (Spark Structured Streaming, Kafka).
• Familiarity with CI/CD pipelines for data engineering workflows.
• Knowledge of data governance, cataloging, and lineage tools.