Senior Data Engineer

INFOSLAB CONSULTANCY PRIVATE LIMITED; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

A leading consultancy firm in Hyderabad is looking for a skilled Senior Data Engineer to develop scalable data platforms and pipelines. The role requires expertise in Python, Scala, AWS, and Docker. Responsibilities include designing data pipelines, collaborating with cross-functional teams, and managing CI/CD processes. Ideal candidates will have a solid background in data architecture and cloud technologies.

Qualifications

• Strong hands-on experience with Python, Scala, SQL, and Amazon Redshift.
• Proven expertise in AWS cloud services (EC2, S3, Redshift, Glue, Lambda).
• Experience implementing Infrastructure as Code (IaC) with Terraform.
• Proficient in managing and deploying Docker containers.
• Hands-on experience with CI/CD pipelines using Jenkins.
• Strong understanding of data architecture and ETL pipelines.

Responsibilities

• Design, build, and optimize scalable data pipelines and platform components.
• Collaborate with teams to ensure seamless data flow and integration.
• Develop infrastructure as code using Terraform to automate provisioning.
• Manage containerized services using Docker.
• Set up and optimize CI/CD pipelines using Jenkins.

Skills

Python

Scala

SQL

Amazon Redshift

AWS

Terraform

Docker

CI/CD pipelines

Tools

Jenkins

Apache Airflow

Spark

Databricks

Job description

About the Role

We are looking for a highly skilled and experienced Senior Data Engineer to join our team in Hyderabad. The ideal candidate will bring strong technical expertise in building scalable data platforms and pipelines using modern technologies such as Python, Scala, AWS, Redshift, Terraform, Jenkins, and Docker. This role demands a hands-on professional who thrives in a fast-paced, collaborative environment and is eager to solve complex data problems.

Key Responsibilities

• Design, build, and optimize robust, scalable, and secure data pipelines and platform components.
• Collaborate with data scientists, analysts, and engineering teams to ensure seamless data flow, integration, and availability across systems.
• Develop infrastructure as code using Terraform to automate provisioning and environment management.
• Manage containerized services and workflows using Docker.
• Set up, manage, and optimize CI/CD pipelines using Jenkins for continuous integration and deployment.
• Optimize performance, scalability, and reliability of large-scale data systems on AWS.
• Write clean, modular, and efficient code in Python and Scala to support ETL, data transformation, and processing tasks.
• Support data architecture planning and participate in technical reviews and design sessions.

Must-Have Skills

• Strong hands-on experience with Python,Scala,SQL, and Amazon Redshift.
• Proven expertise in AWS cloud services and ecosystem (EC2, S3, Redshift, Glue, Lambda, etc.).
• Experience implementing Infrastructure as Code (IaC) with Terraform.
• Proficient in managing and deploying Docker containers in development and production environments.
• Hands-on experience with CI/CD pipelines using Jenkins.
• Strong understanding of data architecture, ETL pipelines, and distributed data processing systems.
• Excellent problem-solving skills and ability to mentor junior engineers.

Nice-to-Have

• Experience working in regulated domains like healthcare or finance.
• Exposure to Apache Airflow,Spark, orDatabricks.
• Familiarity with data quality frameworks and observability tool