Data Engineer (AWS | Spark | Python)

Trigent Software - Professional Services; India; On-site; INR 1,500,000 - 2,300,000; Full time

Job summary

Trigent Software - Professional Services is seeking an expert-level Data Engineer to design, build, and optimize scalable cloud data solutions using AWS, Apache Spark, and Python. You will develop robust data pipelines, implement CI/CD practices, and collaborate with cross-functional teams to power analytics and business decisions.

The role emphasizes automation, performance tuning, data quality, and scalable architectures, with mentorship responsibilities for junior engineers.

Qualifications

• Expert-level experience in data engineering with a track record of production-grade solutions.
• Hands-on expertise with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, and IAM.
• Strong working knowledge of Apache Spark, including performance tuning and optimization.
• Advanced proficiency in Python.
• Experience building and maintaining CI/CD pipelines (e.g., Jenkins, GitHub Actions, GitLab CI, AWS CodePipeline).
• Solid understanding of data modeling, ETL/ELT patterns, and data warehousing concepts.
• Excellent problem-solving, communication, and stakeholder management skills.

Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines on AWS.
• Build and optimize large-scale data processing workloads using Apache Spark (PySpark).
• Write clean, efficient Python code for data transformation and automation.
• Implement and manage CI/CD pipelines for data engineering workflows.
• Ensure data quality, reliability, security, and performance across data platforms.
• Troubleshoot and tune pipeline performance and cost on cloud infrastructure.
• Collaborate with data scientists, analysts, architects, and product teams to translate requirements into technical solutions.
• Mentor junior engineers and contribute to code reviews, standards, and best practices.

Skills

Data engineering

AWS

Spark

Python

CI/CD

Data modeling

ETL/ELT

Communication

Stakeholder management

Tools

Airflow

Terraform

CloudFormation

Docker

Kubernetes

Delta Lake

Iceberg

Hudi

Job description

We're looking for an expert-level Data Engineer to design, build, and optimize scalable cloud data solutions for our Cloud Data Engineering team. You'll work with AWS, Apache Spark, and Python to deliver robust data pipelines and platforms that power analytics and business decisions, while championing automation and CI/CD best practices.

Key Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines on AWS
• Build and optimize large-scale data processing workloads using Apache Spark (PySpark)
• Write clean, efficient, and well-tested Python code for data transformation and automation
• Implement and manage CI/CD pipelines for data engineering workflows
• Ensure data quality, reliability, security, and performance across data platforms
• Troubleshoot and tune pipeline performance and cost on cloud infrastructure
• Collaborate with data scientists, analysts, architects, and product teams to translate requirements into technical solutions
• Mentor junior engineers and contribute to code reviews, standards, and best practices

Required Skills & Qualifications

• Expert-level experience in data engineering, with a strong track record of delivering production-grade solutions
• Hands-on expertise with AWS services such as S3, Glue, EMR, Lambda, Redshift, Athena, and IAM
• Strong working knowledge of Apache Spark, including performance tuning and optimization
• Advanced proficiency in Python
• Experience building and maintaining CI/CD pipelines (e.g., Jenkins, GitHub Actions, GitLab CI, AWS CodePipeline)
• Solid understanding of data modeling, ETL/ELT patterns, and data warehousing concepts
• Excellent problem-solving, communication, and stakeholder management skills

Good to Have

• Experience with orchestration tools such as Airflow or Step Functions
• Familiarity with Infrastructure as Code (Terraform, CloudFormation)
• Exposure to Docker, Kubernetes, and containerized deployments
• Knowledge of Delta Lake, Iceberg, or Hudi
• AWS certifications (e.g., AWS Certified Data Analytics / Solutions Architect)