Data Engineer

LPL Financials; Hyderabad; On-site; INR 1,500,000 - 2,800,000; Full time

Job summary

LPL Financials Hyderabad seeks a skilled Data Engineer to design and maintain distributed data pipelines on AWS, using Spark, Python, and modern DevOps practices. You will craft event-driven data solutions, APIs, and scalable data services, collaborating across teams to ensure reliability and performance.

The role emphasizes end-to-end data quality, monitoring, and auditability, with opportunities to implement near real-time data processing and cloud-native architectures.

Qualifications

• BE in Computer Science, Engineering, or equivalent practical experience.
• 5+ years of experience in data engineering or distributed systems development.
• Strong hands-on experience with AWS cloud services.
• Advanced proficiency in Python.
• Hands-on experience with Apache Spark and AWS services such as Glue, Lambda, Step Functions, EventBridge, S3, Athena, RDS Aurora PostgreSQL.
• Experience building event-driven data pipelines.

Responsibilities

• Design, develop, and maintain distributed data pipelines on AWS at scale.
• Ingest, process, and deliver data using batch and event-driven patterns.
• Build event-driven data solutions with asynchronous architectures for near real-time processing.
• Ensure end-to-end data quality with validation, monitoring, lineage, and orchestration.
• Design and develop RESTful and event-driven APIs for data exposure.
• Build and deploy data services using Docker and manage workloads on Kubernetes.
• Collaborate with data consumers, architects, platform teams, and business stakeholders.
• Contribute to monitoring, logging, alerting, and incident response for data platforms.

Skills

Python

AWS Cloud

Apache Spark

Event-driven architectures

Education

BE in Computer Science

Tools

Docker

Kubernetes

AWS Glue

AWS Lambda

AWS Step Functions

AWS EventBridge

Amazon S3

Amazon Athena

RDS Aurora PostgreSQL

Job description

Responsibilities:

• Design, develop, and maintain distributed data pipelines on AWS that ingest, process, and deliver data at scale. Implement both batch and event-driven data processing patterns using AWS-native services.
• Build and support event-driven data solutions using asynchronous, decoupled architectures to enable near real-time processing and system scalability.
• Provide hands‑on engineering using Python and AWS Glue with pyspark to process large datasets efficiently. Apply best practices in distributed systems, performance optimization, and fault tolerance.
• Ensure end-to-end data quality, implement validation and monitoring checks, establish data lineage, and manage orchestration workflows to ensure reliable and auditable data movement.
• Design and develop RESTful and event-driven APIs to expose data and data services to internal and external consumers.
• Build and deploy data services using Docker containers and manage workloads on Kubernetes following cloud-native and DevOps best practices.
• Collaborate with data consumers, architects, platform teams, and business stakeholders to gather requirements, design scalable solutions, and deliver high-quality outcomes.
• Contribute to monitoring, logging, alerting, and incident response for data platforms. Ensure systems meet reliability, performance, and security standards.

What are we looking for?

We are looking for strong data engineers who can deliver reliable, scalable, and high-quality data solutions. The ideal candidate thrives in a fast-paced environment, is passionate about data engineering, and is comfortable working across teams to solve complex data problems.

Requirements:

• B.E in Computer Science, Engineering, or equivalent practical experience
• 5+ years of experience in data engineering or distributed systems development
• Strong hands‑on experience with AWS cloud services
• Advanced proficiency in Python
• Hands‑on experience with Apache Spark and AWS AWS services such as Glue, Lambda, Step functions, Event Bridge, S3, Athena, RDS Aurora Postgresql
• Experience building event‑driven data pipelines