Big Data | Data Engineer | Python+SQL

Capgemini; Hyderabad, Chennai District, Bengaluru; On-site; INR 900,000 - 1,500,000; Full time

Job summary

Capgemini in Hyderabad, India is seeking a Data Engineer with strong Python expertise to design, build, and optimize scalable data pipelines and architectures, collaborating with data scientists and business teams to ensure reliable data availability.

You will design, develop, and maintain robust data pipelines using Python, work with Pandas, PySpark, or Dask, and deploy pipelines with Airflow on cloud platforms such as AWS, Azure, or GCP.

Qualifications

• Strong proficiency in Python with solid coding skills.
• Experience with SQL and relational databases.
• Hands-on ETL development and data integration.
• Familiarity with Spark or Hadoop ecosystems.
• Knowledge of data warehousing concepts.
• Experience consuming REST APIs.
• Understanding of data structures and algorithms.

Responsibilities

• Design, develop, and maintain robust data pipelines using Python.
• Build scalable ETL/ELT workflows.
• Work with large datasets using Pandas, PySpark, or Dask.
• Integrate data from multiple sources.
• Optimize data storage and retrieval.
• Ensure data quality and validation standards.
• Collaborate with cross-functional teams.
• Deploy and monitor pipelines using Airflow.
• Work on cloud platforms like AWS, Azure, or GCP.

Skills

Python

SQL

ETL

Spark

Hadoop

Data warehousing

REST APIs

DS & algorithms

Tools

Airflow

Pandas

PySpark

Docker

Kubernetes

Job description

We are seeking a skilled Data Engineer with strong Python expertise to design, build, and optimize scalable data pipelines and architectures. The ideal candidate will work closely with data scientists, analysts, and business teams to ensure reliable data availability and quality.

Key Responsibilities

• Design, develop, and maintain robust data pipelines using Python
• Build scalable ETL/ELT workflows
• Work with large datasets using Pandas, PySpark, or Dask
• Integrate data from multiple sources
• Optimize data storage and retrieval
• Ensure data quality and validation standards
• Collaborate with cross-functional teams
• Deploy and monitor pipelines using Airflow
• Work on cloud platforms like AWS, Azure, or GCP

Required Skills & Qualifications

• Strong proficiency in Python
• Experience with SQL and relational databases
• Hands-on ETL experience
• Familiarity with Spark or Hadoop
• Knowledge of data warehousing
• Experience with REST APIs
• Understanding of data structures and algorithms

Preferred Skills

• Cloud platform experience
• Docker/Kubernetes
• Kafka or stream processing
• CI/CD for data workflows
• Basic ML pipeline understanding