Data Analytics Engineer

Monocept; Gurugram District; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

Monocept is looking for a skilled Data Engineer with 4-5 years of experience to develop and maintain ETL/ELT pipelines using PySpark and orchestration tools.

The ideal candidate will be well-versed in cloud platforms (AWS or Azure) and have expertise in SQL, data modeling, and the Big Data ecosystem.

This role entails optimizing Spark jobs, ensuring data quality, and working with CI/CD workflows.

Qualifications

• 4-5 years of experience as a Data Engineer.
• Strong understanding of ETL/ELT processes and data modeling.
• Experience with cloud platforms (AWS or Azure).

Responsibilities

• Develop and maintain ETL/ELT pipelines using PySpark.
• Optimize Spark jobs for performance and cost.
• Integrate Kafka/Kinesis for near real-time data processing.

Skills

Big Data Ecosystem: Hadoop, HDFS, Hive/Delta Lake, Spark fundamentals

Cloud: AWS or Azure

PySpark: RDD/DataFrame APIs

SQL & Databases: Advanced SQL

ETL & DWH Concepts

Orchestration: Airflow, ADF, Step Functions

Version Control & CI/CD: Git

Data Quality & Testing

Tools

Kafka

Streaming: Kinesis, Event Hub

Job description

Role Title: Data Engineer

Experience: 4–5 years

Key Responsibilities

• Develop and maintain ETL/ELT pipelines using PySpark and orchestration tools (Airflow, AWS Step Functions, Azure Data Factory).
• Implement data models (staging, core, mart) aligned with Data Warehouse concepts (star/snowflake).
• Optimize Spark jobs (partitioning, caching, broadcast joins) for performance and cost.
• Build and manage data lakes/warehouses (S3/ADLS + Hive/Delta Lake/Redshift/Synapse).
• Author robust SQL (CTEs, window functions) and stored procedures/functions for complex transformations.
• Integrate Kafka/Kinesis/Event Hub for near real‑time ingestion and streaming pipelines.
• Ensure data reliability via unit tests, data quality checks (Great Expectations, Deequ), and monitoring.
• Manage CI/CD for data workflows; collaborate on code reviews and versioning (Git).
• Implement security, governance, and metadata management (IAM, Key Vault, Lake Formation, Unity Catalog).
• Create operational documentation, runbooks, and contribute to production support (on‑call rotations).

Mandatory Skills

• Big Data Ecosystem: Hadoop, HDFS, Hive/Delta Lake, Spark fundamentals.
• Cloud: AWS (S3, EMR/Glue, Athena, Redshift) or Azure (ADLS, Databricks/Synapse, ADF).
• PySpark: RDD/DataFrame APIs, UDFs, Spark SQL, optimization.
• SQL & Databases: Advanced SQL, indexing, query tuning, relational modeling.
• ETL & DWH Concepts: Slowly Changing Dimensions, CDC, partitioning, schema evolution.
• Orchestration: Airflow, ADF, Step Functions (at least one).
• Version Control & CI/CD: Git, branching, pipelines.
• Data Quality & Testing: Unit tests, validation frameworks, logging & observability.

Good‑to‑Have Skills

• NoSQL: MongoDB, Cassandra, DynamoDB, Cosmos DB.
• Streaming: Kafka, Kinesis, Event Hub, Spark Structured Streaming.
• Procedures & Functions: PL/pgSQL, T‑SQL.
• Performance Tuning: Cost‑aware design, cluster sizing, job profiling.
• Security: IAM/Role‑based access, encryption at rest/in transit.
• Exposure: dbt, Lakehouse patterns, data governance tools.