Python_Data Engineering

Cognizant; Gurugram District; On-site; INR 600,000 - 1,200,000; Full time

Job summary

Cognizant is seeking a Data Engineer in Gurugram District, Haryana, to build and maintain ETL/ELT pipelines using Python. You will handle data ingestion from various sources and optimize pipelines for performance and scalability.

The ideal candidate must have experience in cloud platforms like AWS, Azure, or GCP, as well as proficiency in handling both structured and unstructured data. Strong knowledge of big data technologies and data warehousing practices will be essential.

Qualifications

• Experience in building and maintaining ETL/ELT pipelines.
• Strong knowledge of data ingestion from various sources.
• Proficient in cloud platforms like AWS, Azure, GCP.

Responsibilities

• Build and maintain ETL/ELT pipelines using Python.
• Ingest data from multiple sources including APIs and databases.
• Optimize pipelines for performance and scalability.

Skills

ETL/ELT pipelines using Python

Data ingestion from APIs, databases, files

Handling structured and unstructured data

Database optimization

Cloud platforms (AWS, Azure, GCP)

Big data technologies (Apache Spark, Hadoop)

Tools

Pandas

PySpark

Dask

SQL (PostgreSQL, MySQL, SQL Server)

MongoDB

Snowflake

Apache Airflow

AWS

Azure

GCP

Job description

Responsibilities

• Build and maintain ETL/ELT pipelines using Python
• Ingest data from multiple sources (APIs, databases, files, streaming systems)
• Optimize pipelines for performance and scalability
• Clean, transform, and validate raw datasets
• Handle structured and unstructured data
• Use frameworks like:
  • Pandas
  • PySpark
  • Dask
• Database & Data Warehousing
  • SQL (PostgreSQL, MySQL, SQL Server)
  • NoSQL (MongoDB, Cassandra)
  • Design schemas and optimize queries
  • Build data warehouses using:
    • Snowflake
    • Redshift
    • BigQuery
• Big Data Technologies
  • Apache Spark
  • Hadoop
  • Process large-scale datasets efficiently
• Workflow Orchestration
  • Apache Airflow
• Cloud Platforms
  • Work on cloud environments:
    • AWS (S3, Glue, Lambda, EMR)
    • Azure (Data Factory, Synapse)
    • GCP (Dataflow, BigQuery)
• Data Quality & Monitoring
  • Implement data validation checks
  • Monitor pipeline failures and fix bugs
  • Ensure data reliability and integrity