Aws Data Engineer

Astrosoft Technologies LLC; Hyderabad; On-site; INR 1,800,000 - 3,000,000; Full time

Job summary

Astrosoft Technologies LLC is seeking an experienced Data Platform Engineer to design, build, and maintain cloud-native data pipelines supporting batch and streaming workloads. You will implement Medallion Architecture, develop lakehouse solutions on AWS (S3, Athena, Lake Formation), and build scalable data platform services with Terraform/OpenTofu.

The role emphasizes governance, security, cost optimization, CI/CD for data infrastructure, and collaboration with platform engineering,

Qualifications

• 5+ years of experience in Data Engineering, Cloud Engineering, or Data Platform Engineering.
• 3+ years of hands-on AWS experience building cloud-native data platforms.
• Experience implementing Medallion Architecture or similar lakehouse patterns.
• Strong experience with S3, Athena, Lake Formation, Iceberg.
• Experience building scalable ETL/ELT pipelines.
• Strong SQL skills and optimization of analytical workloads.
• IaC experience with Terraform/Open Tofo.
• CI/CD pipelines for infrastructure and data deployments.
• Git and modern source control workflows.
• Automation using Python/Java/Go/TypeScript.
• Linux environments and cloud networking basics.
• Agile development with Jira/Azure DevOps.
• Experience with AWS Glue, Spark, and Redshift is a plus.
• Container tech including Docker and Kubernetes (EKS preferred).
• Data quality, observability, and monitoring experience.

Responsibilities

• Design, build, and maintain cloud-native data pipelines for batch and incremental processing.
• Implement Medallion Architecture (Bronze, Silver, Gold) for enterprise data products.
• Develop and optimize data lake solutions using AWS services such as S3, Athena, Lake Formation.
• Build scalable data platform services using Terraform/OpenTofu and CI/CD pipelines.
• Develop data ingestion, transformation, and orchestration across multiple sources.
• Design data models and storage patterns for analytics and downstream apps.
• Governance, access controls, metadata management, and security with Lake Formation.
• Optimize query performance, partitioning, formats, and cost across the platform.
• Automate infrastructure and data pipelines; maintain operational tooling.
• Collaborate with platform engineering, architecture, security, and app teams.
• Triage production pipelines and platform issues; write architecture docs and standards.
• Mentor junior engineers and promote engineering best practices.

Skills

Data Engineering

Cloud Engineering

Medallion Architecture

ETL/ELT pipelines

SQL

CI/CD

Python

Java

Go

TypeScript

Linux

AWS

Docker

Kubernetes

Spark

Tools

Amazon S3

Amazon Athena

AWS Lake Formation

Apache Iceberg

Terraform

OpenTofu

Git

Jira

Azure DevOps

Docker

Kubernetes

AWS Glue

Job description

Role & responsibilities

• Design, build, and maintain cloud-native data pipelines supporting batch and incremental data processing.
• Implement and support a Medallion Architecture (Bronze, Silver, Gold) for enterprise data products.
• Develop and optimize data lake solutions using Amazon S3, Apache Iceberg, AWS Lake Formation, and Amazon Athena.
• Build scalable, secure, and reusable data platform services using Infrastructure as Code (Terraform/Open Tofu).
• Develop data ingestion, transformation, and orchestration processes across multiple data sources.
• Design data models and storage patterns that support analytics, reporting, and downstream applications.
• Implement data governance, access controls, metadata management, and security using AWS Lake Formation.
• Optimize query performance, partitioning strategies, storage formats, and cost efficiency across the data platform.
• Build and maintain CI/CD pipelines for infrastructure, data pipelines, and platform components.
• Develop automation and operational tooling using modern programming languages.
• Collaborate with platform engineering, architecture, security, and application development teams to deliver cloud-native data solutions.
• Troubleshoot production data pipelines, infrastructure, and platform issues.
• Produce technical documentation, architecture diagrams, and implementation standards.
• Mentor junior engineers and promote engineering best practices across the team.

Preferred candidate profile

• 5+ years of experience in Data Engineering, Cloud Engineering, or Data Platform Engineering.
• 3+ years of hands-on AWS experience building cloud-native data platforms.
• Experience implementing Medallion Architecture (Bronze/Silver/Gold) or similar lakehouse patterns.
• Strong experience with:
• Amazon S3
• Amazon Athena
• AWS Lake Formation
• Apache Iceberg
• Experience building scalable ETL/ELT pipelines.
• Strong SQL skills and experience optimizing analytical workloads.
• Experience with Infrastructure as Code using Terraform/Open Tofu.
• Experience implementing CI/CD pipelines for infrastructure and application deployments.
• Experience with Git and modern source control workflows.
• Experience developing automation using one or more of the following:
• Python (preferred)
• Java
• Go
• TypeScript
• Experience with Linux environments (Ubuntu, Red Hat, or CentOS).
• Knowledge of cloud networking and AWS security best practices.
• Experience working in Agile development environments using Jira or Azure DevOps.
• Experience with AWS Glue.
• Experience with Apache Spark or distributed data processing frameworks.
• Experience with event-driven data ingestion architectures.
• Experience with Amazon Redshift and Redshift Spectrum.
• Experience building reusable data platform capabilities and shared engineering frameworks.
• Experience with container technologies including Docker and Kubernetes (Amazon EKS preferred).
• Experience implementing data quality, observability, and monitoring solutions.
• Experience supporting enterprise data governance initiatives.