Senior Data Engineer

Procorp Systems; Hyderabad; On-site; INR 2,500,000 - 4,000,000; Full time

Job summary

Procorp Systems is seeking a seasoned Data Engineer to design and manage enterprise data platforms, build scalable data pipelines in Python, and orchestrate analytics workloads with Airflow. The role requires deep AWS experience (IAM, S3, Lambda, Redshift, Glue) and IaC proficiency with Terraform/CloudFormation.

You will contribute to data warehousing, BI, data security and data quality initiatives, while supporting both modern and legacy systems.

Qualifications

• 6+ years of experience with enterprise data platforms, data lakes and big data technologies.
• 5+ years of experience with relational and NoSQL databases; strong SQL.
• 4+ years with AWS including IAM, S3, API Gateway, Lambda, Glue, Lake Formation, Redshift, DynamoDB, RDS.
• 4+ years using Python for data engineering and building scalable data pipelines.
• 2+ years Apache Airflow for orchestrating analytics workloads.
• 2+ years IaC solutions like Terraform, CloudFormation.
• Preference for ML/AI use cases in AWS (SageMaker, Bedrock).
• Strong understanding of data warehousing, BI, data security, quality and profiling.
• Experience with ETL tools like Talend Open Studio, Informatica, Pentaho.
• Cloud DevOps experience with GitLab, Bitbucket.
• Experience with AWS EKS for Kubernetes.

Responsibilities

• Build and manage enterprise data platforms including data lakes.
• Develop scalable data pipelines using Python.
• Orchestrate analytics workloads with Apache Airflow.
• Implement IaC with Terraform and CloudFormation.
• Leverage AWS services for ML/AI use cases.
• Advise on data warehousing, security, quality and profiling.
• Utilize ETL tools such as Talend, Informatica or Pentaho.
• Support DevOps with GitLab/Bitbucket integration.
• Manage Kubernetes clusters on AWS EKS.
• Collaborate with teams including legacy system support.

Skills

Data engineering

SQL & NoSQL

Python

Airflow

Kubernetes

AWS cloud

CI/CD (GitLab/Bitbucket)

Data lakes / big data

Tools

Terraform

CloudFormation

Talend / Informatica / Pentaho

SageMaker

Bedrock

S3

Lambda

API Gateway

Redshift

DynamoDB

RDS

EKS

Job description

Duration: Contract Long term / Full time

Salary : DOE

Working Hours : 8:30 PM IST to 4:30 AM IST

JD:

• 6+ years of experience working with enterprise data platforms, building and managing data lakes and using big data technologies
• 5+ years of experience working with relational and non-relational / NoSQL databases, advanced knowledge of SQL
• 4+ years of experience working with AWS platform. Experience with solutioning on AWS infrastructure using services like AWS IAM, S3, API Gateway, Lambda, Glue, Lake formation, Redshift, DynamoDB, RDS
• 4+ years of experience with using Python for data engineering and developing high performance and scalable data pipeline
• 2+ years of experience working with Apache Airflow for orchestrating analytics workload
• 2+ years of experience with IaC solutions like terraform, cloud formation
• Preference to candidates with MLOps experience and exposure to executing ML/AI use cases in AWS cloud utilizing services like Sagemaker, Bedrock
• Strong understanding of concepts around data warehousing, BI, data security, data quality and profiling
• Prior experience with traditional ETL tools like Talend Open Studio, Informatica, Pentaho or something similar
• Cloud DevOps experience utilizing products like Gitlab, Bitbucket or similar ones
• Prior experience with AWS EKS service, setting up and managing kubernetes clusters and services is preferred
• Have to be a team player and open to working with newer technologies as well as supporting legacy systems