Senior Data Engineer

Vertage Global; Chennai District; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

Vertage Global is seeking an experienced Data Engineer to support a Global Capital Markets modernization program. Design, build and optimize scalable data pipelines using Databricks, Python, PySpark and AWS to underpin front-office stabilization and data-lake platforms.

Responsibilities include developing data workflows, ETL/ELT solutions, APIs with FastAPI, CI/CD and Agile delivery, and collaborating with stakeholders across Risk, Quant, and Technology to deliver production-ready data

Qualifications

• 8+ years of experience in Data Engineering, Data Science or Analytics Engineering roles.

Responsibilities

• Partner with Front Office, Product Owners, Quantitative Analysts, Risk and Technology teams to understand business requirements and transform them into scalable data solutions.
• Design, develop and maintain data pipelines and analytical workflows using Databricks, Python, PySpark and AWS.
• Implement scalable ETL/ELT solutions leveraging AWS services such as S3, Glue, Lambda, CloudWatch and Databricks, and develop reusable data models and data-quality controls.
• Develop and optimize PySpark applications and distributed-processing workloads to improve performance and scalability.
• Develop APIs and data services using Python and FastAPI to enable consumption of analytics across applications.
• Analyze structured and unstructured datasets to derive actionable insights and support business decision-making.
• Support migration and modernization initiatives involving data platforms and cloud transformation.
• Collaborate with stakeholders to define data-quality standards, governance controls and monitoring frameworks.
• Contribute to CI/CD, DevOps, testing and release-management activities, and participate in Agile delivery including estimation and sprint planning.
• Identify risks and bottlenecks and recommend solutions; contribute to documentation and operational readiness.

Skills

Data engineering

ETL/ELT

Distributed processing

Spark architecture

Python programming

Communication

Education

Bachelor's degree in Computer Science/Engineering/IT

Master's degree (preferred)

Tools

Databricks

Python

PySpark

AWS

S3

Glue

Lambda

IAM

CloudWatch

API Gateway

FastAPI

Job description

We are seeking an experienced Data Engineer to support a large-scale Global Capital Markets platform-modernization program. The role focuses on designing, building and optimizing scalable data pipelines and analytical workflows using Databricks, Python, PySpark and AWS cloud services, underpinning the stabilization of Front Office Trading systems and the build-out of target-state capabilities across event-driven integration, data-lake / Databricks platforms, reporting, migration and the front-to-back trade lifecycle for FX Cash, Cleared IRS and future product onboarding. The ideal candidate combines strong hands-on data-engineering and distributed-processing expertise with a solid grasp of ETL/ELT and data lake architectures, and partners closely with Front Office, Product Owners, Risk, Quantitative Analysts and Technology teams to deliver production-ready, high-performance data solutions in a complex capital-markets setting.

Key responsibilities

• Partner with Front Office, Product Owners, Quantitative Analysts, Risk and Technology teams to understand business requirements and transform them into scalable data and analytics solutions.
• Design, develop and maintain data pipelines and analytical workflows using Databricks, Python, PySpark and AWS cloud services.
• Implement scalable ETL/ELT solutions leveraging AWS services such as S3, Glue, Lambda, CloudWatch and Databricks, and develop reusable data models, transformation frameworks and data-quality controls.
• Develop and optimize PySpark applications and distributed-processing workloads to improve performance, scalability and operational efficiency.
• Develop APIs and data services using Python and FastAPI to enable consumption of analytics and business data across multiple applications.
• Analyze structured and unstructured datasets to derive actionable insights and support business decision-making, aligned to business and regulatory requirements.
• Support migration and modernization initiatives involving data platforms, cloud transformation and distributed-processing architectures, optimizing data architecture, processing performance and cloud resource utilization.
• Collaborate with stakeholders to define data-quality standards, reconciliation processes, governance controls and monitoring frameworks.
• Contribute to CI/CD, DevOps, testing and release-management activities, and participate in Agile delivery including estimation, sprint planning, refinement, deployment and production support.
• Identify risks, dependencies, bottlenecks and performance challenges and proactively recommend solutions; contribute to documentation, knowledge sharing and operational readiness.
• Bachelor’s / Master’s degree with 8+ years of experience in Data Engineering, Data Science or Analytics Engineering roles.
• Strong hands-on experience with Databricks on AWS and distributed data processing frameworks, with a strong understanding of Spark architecture, performance tuning and optimization.
• Advanced programming skills in Python and PySpark.
• Experience implementing data-engineering solutions using AWS services including S3, Glue, Lambda, IAM, CloudWatch and API Gateway.
• Expertise in building scalable ETL/ELT pipelines and modern data lake architectures, with strong data-warehousing, dimensional-modeling and large-scale data-management knowledge.
• Strong SQL skills across relational and NoSQL databases, and experience developing APIs using FastAPI or similar Python frameworks.
• Experience with CI/CD pipelines, Git, DevOps and automated deployment practices, working in Agile environments with JIRA and sprint-based execution.
• Excellent analytical, problem-solving and troubleshooting skills, with strong stakeholder-management and communication skills across business and technology teams.

Preferred qualifications

• Databricks and/or AWS data-engineering certifications.
• Prior experience in regulated, security-controlled financial-services environments and platform-modernization / migration programs.