Senior AWS Data Engineer

Persistent; Pune District; Hybrid; INR 5,000,000 - 8,000,000; Full time

Benefits offered by this job

Competitive salary

Education and certifications support

Cutting-edge tech stacks

Flexible hours

Annual health check-ups

Job summary

Persistent is seeking a Senior AWS Data Engineer to design and optimize enterprise-scale data platforms. You will develop scalable pipelines on AWS and Databricks, implement PySpark-based ETL/ELT workflows, and drive AI/ML initiatives with Lakehouse architectures and Delta Lake.

The role requires 10+ years of IT experience with a strong Data Engineering background, proficiency in Python/SQL, and hands-on AWS/Databricks expertise.

Qualifications

• Bachelor’s or Master’s degree in CS/IT/Data Science/Engineering or related field.
• 10+ years of IT experience with strong Data Engineering background.
• Hands-on with AWS services (S3, Glue, EMR, Lambda, Redshift, Athena, IAM, Step Functions).
• Extensive experience with Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong development in PySpark and Spark SQL and Spark performance tuning.
• Experience implementing MLOps frameworks using MLflow and CI/CD.
• Databricks Mosaic AI, LLMs, RAG, and Generative AI solutions experience.
• Proficiency in Python and SQL; Git for version control.
• Knowledge of Terraform, Docker, Kubernetes, and Airflow is a plus.

Responsibilities

• Design, develop, and optimize enterprise-scale data solutions for AI/ML initiatives.
• Build and maintain scalable data pipelines using AWS and Databricks.
• Create ETL/ELT workflows using PySpark and Spark SQL.
• Develop Lakehouse-based data solutions with Delta Lake.
• Tune Spark/Databricks performance to reduce costs and improve throughput.
• Implement and maintain MLOps workflows for model deployment and monitoring.
• Leverage Databricks Mosaic AI for Generative AI use cases.
• Collaborate with Data Scientists, ML Engineers, Architects, and stakeholders.
• Ensure data quality, governance, security, and compliance.

Skills

AWS

Databricks

PySpark

Spark SQL

Python

SQL

MLOps

Databricks Mosaic AI

Education

Bachelor's or Master's in CS/IT/Data Science/Engineering

Tools

Terraform

Docker

Kubernetes

Airflow

Git

Job description

About Position:

We are looking for a highly skilled AWS Data Engineer with strong expertise in AWS, Databricks, and PySpark to build and optimize scalable data platforms and AI-driven solutions. The ideal candidate will have extensive experience in designing modern data architectures, performance tuning, implementing MLOps practices, and leveraging Databricks Mosaic AI capabilities.

• Role: Senior AWS Data Engineer
• Location: All Persistent Location
• Experience: 10+ Years
• Job Type: Full-Time Employment

What You'll Do:

• As a Senior AWS Data Engineer, you will be responsible for designing, developing, and optimizing enterprise-scale data solutions while enabling advanced AI/ML initiatives.
• Design, develop, and maintain scalable data pipelines using AWS and Databricks.
• Build and optimize ETL/ELT workflows using PySpark and Spark SQL.
• Develop data solutions based on Lakehouse architecture and Delta Lake.
• Perform Spark and Databricks performance tuning to improve processing efficiency and reduce costs.
• Implement and maintain MLOps workflows for model deployment, monitoring, and lifecycle management.
• Leverage Databricks Mosaic AI capabilities for Generative AI and ML use cases.
• Collaborate with Data Scientists, ML Engineers, Architects, and business stakeholders.
• Ensure data quality, governance, security, and compliance standards are followed.
• Automate data engineering processes using CI/CD pipelines and Infrastructure as Code.
• Troubleshoot and resolve complex production issues across data platforms.

Expertise You'll Bring:

• The ideal candidate should possess strong technical expertise in cloud-based data engineering, big data technologies, and AI/ML ecosystems.
• 10+ years of overall IT experience with strong Data Engineering background.
• Hands-on experience with AWS services including S3, Glue, EMR, Lambda, Redshift, Athena, IAM, and Step Functions.
• Extensive experience working with Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong development experience using PySpark and Spark SQL.
• Expertise in Spark Performance Tuning, Query Optimization, Partitioning, and Cluster Management.
• Experience implementing MLOps frameworks using MLflow and CI/CD methodologies.
• Hands-on experience with Databricks Mosaic AI, LLMs, RAG, and Generative AI solutions.
• Strong proficiency in Python and SQL.
• Experience with version control tools such as Git.
• Knowledge of Terraform, Docker, Kubernetes, and Airflow will be an added advantage.
• Strong analytical, problem-solving, and communication skills.
• Ability to work independently and collaborate with cross-functional teams.
• Bachelor's or master's degree in computer science, Information Technology, Data Science, Engineering, or a related field.
• Relevant AWS and Databricks certifications are highly preferred.
• AWS Certified Data Engineer Associate
• AWS Certified Solutions Architect
• Databricks Certified Data Engineer Professional
• Databricks Generative AI Engineer Certification
• Databricks Machine Learning Professional Certification

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."