AWS & Airflow Developer

LPL Financials

Hyderabad

Híbrido

INR 1 200 000 - 1 800 000

Tempo integral

14 dias+

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

LPL Financials is seeking a Data Engineer to design, build and maintain scalable data pipelines using Python, SQL, Airflow and AWS services. You will ingest, transform and process data from diverse sources, ensure data quality, and support production deployments while collaborating with cross-functional teams.

The role emphasizes ETL/ELT workflows, data modeling and cloud modernization initiatives, with a strong focus on observability, CI/CD and Agile ceremonies.

Qualificações

  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
  • 4-6 years of experience in Data Engineering, ETL Development, or Data Integration.
  • Strong proficiency in Python programming.
  • Strong hands-on experience with Apache Airflow for workflow orchestration and scheduling.
  • Experience with AWS cloud services including S3, Glue, Lambda, Step Functions, Athena, EventBridge, RDS/Aurora PostgreSQL, CloudWatch.
  • Strong SQL skills with relational databases such as PostgreSQL, Oracle, or SQL Server.
  • Experience developing and supporting batch and event-driven data pipelines.
  • Understanding of data modeling, data warehousing and ETL/ELT concepts.
  • Experience with Git.
  • Knowledge of monitoring, logging and troubleshooting distributed data workflows.
  • Strong analytical, problem-solving and debugging skills.
  • Excellent verbal and written communication skills.

Responsabilidades

  • Develop data transformation logic using Python, SQL and PySpark where applicable.
  • Design, develop and maintain scalable data pipelines using Apache Airflow and AWS services.
  • Build and optimize ETL/ELT workflows to ingest, transform and process data from various source systems.
  • Develop and manage Airflow DAGs, task dependencies, scheduling, monitoring and error handling.
  • Implement data ingestion solutions using AWS services such as S3, Glue, Lambda, Step Functions, EventBridge, Athena and RDS/Aurora.
  • Monitor data pipeline performance and troubleshoot workflow failures, data quality issues and operational incidents.
  • Collaborate with architects, business analysts and application teams to understand data requirements and deliver scalable solutions.
  • Support production deployments, incident management, root cause analysis and performance tuning activities.
  • Implement data validation, reconciliation and quality checks to ensure accurate and reliable data delivery.
  • Participate in cloud modernization and migration initiatives to AWS-based platforms.
  • Contribute to CI/CD implementation, automation, documentation and engineering best practices.
  • Develop and maintain operational dashboards, monitoring and alerting solutions for pipeline observability.
  • Participate in Agile ceremonies, sprint planning and technical design discussions.

Conhecimentos

Python
SQL
Airflow
AWS
Git

Formação académica

Bachelor's degree in Computer Science/IT/Engineering

Ferramentas

Docker
Kubernetes
AWS EMR
Redshift
Snowflake

Descrição da oferta de emprego

Role & responsibilities
  • Develop data transformation logic using Python, SQL and PySpark where applicable.
  • Design, develop and maintain scalable and reliable data pipelines using Apache Airflow and AWS services.
  • Build and optimize ETL/ELT workflows to ingest, transform and process data from various source systems.
  • Develop and manage Airflow DAGs, task dependencies, scheduling, monitoring and error handling.
  • Implement data ingestion solutions using AWS services such as S3, Glue, Lambda, Step Functions, EventBridge, Athena and RDS/Aurora.
  • Monitor data pipeline performance and troubleshoot workflow failures, data quality issues and operational incidents.
  • Collaborate with architects, business analysts and application teams to understand data requirements and deliver scalable solutions.
  • Support production deployments, incident management, root cause analysis and performance tuning activities.
  • Implement data validation, reconciliation and quality checks to ensure accurate and reliable data delivery.
  • Participate in cloud modernization and migration initiatives to AWS-based platforms.
  • Contribute to CI/CD implementation, automation, documentation and engineering best practices.
  • Develop and maintain operational dashboards, monitoring and alerting solutions for pipeline observability.
  • Participate in Agile ceremonies, sprint planning and technical design discussions.
  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
  • 4-6 years of experience in Data Engineering, ETL Development, or Data Integration.
  • Strong proficiency in Python programming.
  • Strong hands-on experience with Apache Airflow for workflow orchestration and scheduling.
  • Experience with AWS cloud services, including:
    • Amazon S3, Glue, Lambda, Step Functions, Athena, EventBridge, RDS/Aurora PostgreSQL, CloudWatch
  • Strong SQL skills with experience in relational databases such as PostgreSQL, Oracle, or SQL Server.
  • Experience developing and supporting batch and event-driven data pipelines.
  • Understanding of data modeling, data warehousing and ETL/ELT concepts.
  • Experience with source control systems such as Git.
  • Knowledge of monitoring, logging and troubleshooting distributed data workflows.
  • Strong analytical, problem-solving and debugging skills.
  • Excellent verbal and written communication skills.
Preferred candidate profile
  • Experience with PySpark and distributed data processing frameworks.
  • Knowledge of AWS EMR, Redshift, or Snowflake.
  • Experience working with Docker and containerized applications.
  • Exposure to Kubernetes or Amazon EKS.
  • Familiarity with REST API integration and event-driven architectures.
  • Experience implementing CI/CD pipelines using GitHub Actions, GitLab CI, or AWS CodePipeline.
  • Understanding of data governance, metadata management and data lineage concepts.
  • Experience working in Agile/Scrum environments.
  • Exposure to infrastructure-as-code tools such as Terraform or AWS CloudFormation.
Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

AWS Big Data Engineer
AWS Big Data Engineer

Zorba AI • Hyderabad

Presencial
INR 1 500 000 - 2 500 000
Aws Data Engineer
Aws Data Engineer

Qentelli LLC • Bengaluru

Presencial
INR 1 400 000 - 2 200 000
Data Engineer
Data Engineer

AppSierra • Dadri

Presencial
INR 1 200 000 - 2 400 000
Airflow Python Developer
Airflow Python Developer

Sutherland Company • Hyderabad

Presencial
INR 1 200 000 - 1 800 000
AWS Data Engineer
AWS Data Engineer

Durapid Technologies Pvt Ltd • Hyderabad

Presencial
INR 900 000 - 1 350 000
Airflow Python Developer
Airflow Python Developer

Sutherland • Hyderabad

Presencial
INR 1 000 000 - 1 600 000
Python Data Engineer
Python Data Engineer

Wissen Technology Pte Ltd • Pune District, Mumbai, Bengaluru

Híbrido
INR 1 200 000 - 1 800 000
Data Engineer- Apache Airflow
Data Engineer- Apache Airflow

Astek Group • Ernakulam

Presencial
INR 1 200 000 - 1 800 000
Aws Data Engineer
Aws Data Engineer

Astrosoft Technologies • Hyderabad

Presencial
INR 1 800 000 - 3 000 000
Big Data Engineer(AWS, Python,Airflow)
Big Data Engineer(AWS, Python,Airflow)

Zorba AI • Hyderabad

Presencial
INR 800 000 - 1 500 000