Senior Data Engineer

LPL Financials; Hyderabad; Presencial; INR 1.200.000 - 1.800.000; Jornada completa

Descripción de la vacante

LPL Financials in Hyderabad is seeking an experienced Data Engineer to design, develop, and maintain large-scale data pipelines on AWS, mixing batch and streaming processing approaches. You will apply Python and Apache Spark to transform data, ensure quality and lineage, and build RESTful and event-driven APIs.

You’ll containerize services with Docker and run on Kubernetes, collaborating with stakeholders for scalable solutions.

Formación

• BS in Computer Science, Engineering, or equivalent practical experience
• 8+ years of experience in data engineering or distributed systems development
• Strong hands-on experience with AWS cloud services
• Advanced proficiency in Python
• Hands-on experience with Apache Spark
• Experience building event-driven data pipelines
• Experience designing distributed data processing architectures
• Strong knowledge of data quality, data lineage, and orchestration
• Experience in API development and hands-on experience with Docker and Kubernetes

Responsabilidades

• Design, develop, and maintain distributed data pipelines on AWS that ingest, process, and deliver data at scale.
• Implement both batch and event-driven data processing patterns using AWS-native services.
• Build and support event-driven data solutions using asynchronous, decoupled architectures to enable near real-time processing and system scalability.
• Provide hands-on engineering using Python and Apache Spark to process large datasets efficiently.
• Ensure end-to-end data quality, implement validation and monitoring checks, establish data lineage, and manage orchestration workflows to ensure reliable and auditable data movement.
• Design and develop RESTful and event-driven APIs to expose data and data services to internal and external consumers.
• Build and deploy data services using Docker containers and manage workloads on Kubernetes following cloud-native and DevOps best practices.
• Collaborate with data consumers, architects, platform teams, and business stakeholders to gather requirements, design scalable solutions, and deliver high-quality outcomes.
• Contribute to monitoring, logging, alerting, and incident response for data platforms. Ensure systems meet reliability, performance, and security standards.

Conocimientos

Python

Data pipelines

Event-driven architecture

Distributed systems

Data quality

Data lineage

API development

Educación

BS in Computer Science or equivalent

Herramientas

Docker

Kubernetes

AWS cloud services

Apache Spark

Descripción del empleo

Responsibilities:

Data Pipeline Development:

Design, develop, and maintain distributed data pipelines on AWS that ingest, process, and deliver data at scale. Implement both batch and event-driven data processing patterns using AWS-native services.

Event-Driven Architecture:

Build and support event-driven data solutions using asynchronous, decoupled architectures to enable near real-time processing and system scalability.

Technical Expertise:

Provide hands-on engineering using Python and Apache Spark to process large datasets efficiently. Apply best practices in distributed systems, performance optimization, and fault tolerance.

Data Quality, Lineage & Orchestration:

Ensure end-to-end data quality, implement validation and monitoring checks, establish data lineage, and manage orchestration workflows to ensure reliable and auditable data movement.

API Development:

Design and develop RESTful and event-driven APIs to expose data and data services to internal and external consumers.

Containers & Cloud-Native Development:

Build and deploy data services using Docker containers and manage workloads on Kubernetes following cloud-native and DevOps best practices.

Stakeholder Collaboration:

Collaborate with data consumers, architects, platform teams, and business stakeholders to gather requirements, design scalable solutions, and deliver high-quality outcomes.

Operational Excellence:

Contribute to monitoring, logging, alerting, and incident response for data platforms. Ensure systems meet reliability, performance, and security standards.

Requirements:

• BS in Computer Science, Engineering, or equivalent practical experience
• 8+ years of experience in data engineering or distributed systems development
• Strong hands-on experience with AWS cloud services
• Advanced proficiency in Python
• Hands-on experience with Apache Spark
• Experience building event-driven data pipelines
• Experience designing distributed data processing architectures
• Strong knowledge of data quality, data lineage, and orchestration
• Experience in API developmentHands-on experience with Docker and Kubernetes