Data Engineering Lead

WebSenor Ltd; Mumbai; On-site; INR 1,500,000 - 2,500,000; Full time

Job summary

WebSenor Ltd is hiring a Data Engineering Lead in Mumbai. The ideal candidate will design and build scalable AWS data platforms, lead architecture discussions, and mentor team members. With 7-10 years of experience in data engineering, proficiency in Python, AWS services, and a strong analytics background, you will play a key role in integrating various systems and ensuring data quality and security. Competitive salary as per industry standards and a chance to work in a dynamic environment.

Qualifications

• 7-10 years in Data Engineering or Platform Engineering.
• Proven experience building cloud-native data platforms.
• Hands-on with batch and real-time data processing.
• Experience in financial services or wealth management preferred.

Responsibilities

• Design and implement AWS-based enterprise data lake architecture.
• Build scalable frameworks for various data types.
• Develop real-time and batch ingestion pipelines.
• Integrate internal and external systems and data sources.
• Implement security and compliance using AWS services.
• Monitor pipeline performance and system health.
• Build and manage CI/CD pipelines.

Skills

Python

PySpark

SQL

AWS (S3, Glue, Athena, Lambda, AppFlow)

Databricks

CloudWatch

Grafana

GitLab

Job description

We’re Hiring: Data Engineering Lead – Mumbai

Job Title: Data Engineering Lead
Location: Mumbai
Experience: 7+ Years
Shift: General IST

About the Role

We are looking for a highly skilled Data Engineering Lead to design, build, and operate a scalable, cloud-native enterprise data platform on AWS.

This is a hands‑on leadership role combining architecture design, data pipeline development, and team mentorship in an Agile environment. The role focuses on enabling secure and reliable ingestion of structured and unstructured data for analytics and downstream consumption on Databricks.

Key Responsibilities

A. Enterprise Data Platform Architecture

• Design and implement AWS-based enterprise data lake architecture
• Build scalable frameworks for all data types (structured, semi-structured, unstructured)
• Define standards for ingestion, transformation, storage, and access
• Ensure seamless integration with analytics workloads

B. Data Ingestion & Integration

• Develop real‑time and batch ingestion pipelines
• Integrate internal systems (CRM, ERP, KYC, OMS, PMS)
• Integrate external data sources (market data, research feeds, documents)
• Utilize AWS services like AppFlow, Lambda, Glue, S3, and Athena

C. Real‑Time Data Processing

• Build event‑driven pipelines for near real‑time analytics
• Support use cases such as:
  • Trading & operational analytics
  • Compliance and surveillance monitoring

D. Security, Governance & Compliance

• Implement security using AWS KMS, Secrets Manager, and Security Hub
• Use AWS Config and CloudTrail for governance and audit
• Ensure strong data protection, encryption, and access control

E. Monitoring & Observability

• Implement monitoring using CloudWatch and Grafana
• Track pipeline performance, failures, data freshness, and system health

F. DevOps & Automation

• Build and manage CI/CD pipelines using GitLab
• Automate testing and deployment processes
• Maintain code quality and version control standards

G. Data Quality & Metadata

• Implement validation, reconciliation, and monitoring frameworks
• Manage metadata and lineage using AWS Glue Data Catalog

H. Technical Leadership & Collaboration

• Lead architecture discussions and troubleshooting
• Mentor team members through code reviews and guidance
• Collaborate with analytics teams and business stakeholders

Skills & Competencies

Technical Skills

• Programming: Python, PySpark, SQL
• Cloud: AWS (S3, Glue, Athena, Lambda, AppFlow)
• Analytics Platform: Databricks
• Monitoring: CloudWatch, Grafana
• DevOps: GitLab

Behavioral Competencies

• Strong analytical and problem‑solving skills
• Ownership mindset with delivery focus
• Ability to lead in fast‑paced Agile environments

Experience Required

• 7–10 years in Data Engineering / Platform Engineering
• Proven experience in building cloud-native data platforms
• Hands‑on experience with batch and real‑time data processing
• Experience in financial services / wealth management preferred

Salary

As per industry standards