Senior Data Engineer - PAN India

Concentrix Catalyst; India; Remote; INR 10,869,565 - 13,586,956; Full time

Job summary

A tech consulting firm is seeking a Sr. Data Engineer to develop data pipelines and migrate data to AWS. This role involves collaborating with a Data Architect to ensure data accuracy and performance, leading streaming application development, and mentoring other engineers. The ideal candidate will have 7+ years of data engineering experience, proficiency in Python and Java, and hands-on experience with AWS services such as S3 and Glue. This is a remote full-time opportunity.

Qualifications

• 7+ years of experience in data engineering with at least 3+ years in senior or lead roles.
• Strong proficiency in Python and Java.
• Hands-on experience with AWS data services.

Responsibilities

• Lead the development of data pipelines for AWS.
• Implement data migration from legacy systems.
• Build and optimize streaming data applications.

Skills

Data pipeline development

AWS data services

Python

Java

SQL

Apache Kafka

ETL/ELT design patterns

Data governance

Education

Bachelor's or Master's degree in Computer Science or related field

Tools

Docker

Kubernetes

Terraform

CloudFormation

Job description

We are seeking a Sr. Data Engineer to lead the build and maintenance of data pipelines supporting our Operational Data Store (ODS) modernization and migration to AWS. Working in close partnership with the Data Architect, this role is responsible for implementing the migration of data from legacy systems, developing real-time streaming applications, and supporting the data needs of microservices and shared service teams (CIAM, Notifications, Configuration). The ideal candidate will translate data model designs into production pipelines, ensuring data accuracy, reliability, and performance at scale. This is a remote, work‑at‑home opportunity in the US.

Key Responsibilities

• Lead the development of data pipelines for batch and real‑time data ingestion into AWS.
• Implement the migration of data from legacy systems to target AWS data stores (Aurora, DynamoDB, S3, Redshift).
• Build and optimize streaming data applications using Kafka, Kafka Connect, and/or AWS Kinesis/EventBridge.
• Implement data validation, cleansing, and transformation logic based on specifications from the Data Architect.
• Develop data quality checks, exception handling, and reconciliation processes to ensure system of record accuracy.
• Create and manage data APIs and services to expose data securely to consuming microservices.
• Work with development teams to implement optimal data storage solutions (SQL/NoSQL) for their services.
• Optimize pipeline performance, monitoring, and alerting for reliability and scalability.
• Partner with the Data Architect to translate logical data models into physical implementations and working pipelines.
• Develop and maintain data lineage documentation and pipeline metadata.
• Troubleshoot data quality issues across the ingestion and transformation layers.
• Collaborate with domain teams to understand source system data structures and integration requirements.
• Collaborate with DevOps to containerize data applications (Docker) and deploy them on Kubernetes (EKS).
• Support the integration of data platforms with CIAM for security and API Gateway for management.
• Mentor Data Engineers on pipeline development best practices and AWS data services.
• Participate in architecture reviews and provide input on pipeline feasibility and performance implications.

Required Qualifications

• Bachelor's or Master's degree in Computer Science, Information Systems, Data Science, or related field.
• 7+ years of experience in data engineering with at least 3+ years in senior or lead roles.
• Strong proficiency in Python and Java. Java required for legacy system migration work, Python for new AWS pipeline development.
• Hands‑on experience with AWS data services (S3, Glue, Lambda, RDS, DynamoDB, Redshift, Kinesis, Aurora).
• Proven expertise building data pipelines for real‑time and batch ingestion at scale.
• Practical experience building and maintaining streaming applications using Apache Kafka or similar platforms.
• Solid SQL skills and experience working with relational databases.
• Experience with data quality frameworks, validation patterns, and exception handling.
• Hands‑on experience with ETL/ELT design patterns and data transformation logic.
• Understanding of data modeling concepts and ability to translate logical models into physical implementations.
• Experience with infrastructure‑as‑code tools (Terraform, CloudFormation) and CI/CD pipelines.
• Solid understanding of data governance, lineage, and metadata management practices.

Preferred Qualifications

• Demonstrates judgment and flexibility – positively deals with shifting priorities and rapid change of environments.
• Experience in financial services, insurance, or banking technology environment.
• AWS Certified Developer, DevOps Engineer, or Data Analytics Specialty.
• Experience with data orchestration tools (Apache Airflow, AWS Step Functions).
• Knowledge of containerization (Docker, Kubernetes) for data workloads.
• Familiarity with Node.js given the broader team ecosystem.
• Knowledge of Data Mesh principles and domain‑oriented data architectures.
• Understanding of regulatory requirements (SOX, GDPR, CCPA) as they apply to data pipelines.
• Experience mentoring engineers and establishing team standards.

Seniority level

• Mid‑Senior level

Employment type

• Full‑time

Job function

• Information Technology

Industries

• IT Services and IT Consulting
• Technology, Information and Media