Senior AWS Data Engineer

Persistent Systems; Pune District; Hybrid; INR 4,000,000 - 7,000,000; Full time

Benefits offered by this job

Hybrid work

Education sponsorship

Flexible hours

Long Service awards

Insurance coverage

Job summary

Persistent Systems is seeking an experienced AWS Data Engineer to design, build, and optimize enterprise-scale data platforms that enable AI-driven initiatives.

You'll leverage AWS, Databricks, and PySpark to implement Lakehouse architectures, perform performance tuning, and drive CI/CD and MLOps practices across data pipelines in a hybrid work environment.

Qualifications

• Minimum 10+ years in IT with strong Data Engineering background.
• Hands-on AWS services: S3, Glue, EMR, Lambda, Redshift, Athena, IAM, Step Functions.
• Proficient in PySpark and Spark SQL; experienced with Spark performance tuning.

Responsibilities

• Design, develop, and optimize enterprise-scale data solutions for AI/ML initiatives.
• Build scalable data pipelines using AWS and Databricks.
• Develop ETL/ELT workflows with PySpark and Spark SQL.
• Implement Lakehouse architecture and Delta Lake.
• Tune Spark/Databricks for performance and cost efficiency.
• Implement and maintain MLOps workflows for model deployment and monitoring.
• Leverage Databricks Mosaic AI for Generative AI use cases.
• Collaborate with Data Scientists, ML Engineers, and stakeholders.
• Ensure data quality, governance, security, and compliance.
• Automate data engineering with CI/CD and IaC.
• Troubleshoot complex production issues across data platforms.

Skills

AWS

Databricks

PySpark

Spark SQL

MLOps

Terraform

Docker

Kubernetes

Airflow

Python

SQL

Education

Bachelor's or Master's in CS/IT/Data Science

Tools

Databricks

Delta Lake

Unity Catalog

Databricks Workflows

Job description

We are looking for a highly skilled AWS Data Engineer with strong expertise in AWS, Databricks, and PySpark to build and optimize scalable data platforms and AI-driven solutions. The ideal candidate will have extensive experience in designing modern data architectures, performance tuning, implementing MLOps practices, and leveraging Databricks Mosaic AI capabilities.

• Location: All Persistent Location
• Experience: 10+ Years
• Job Type: Full-Time Employment

What You\'ll Do:

• As a Senior AWS Data Engineer, you will be responsible for designing, developing, and optimizing enterprise-scale data solutions while enabling advanced AI/ML initiatives.
• Design, develop, and maintain scalable data pipelines using AWS and Databricks.
• Build and optimize ETL/ELT workflows using PySpark and Spark SQL.
• Develop data solutions based on Lakehouse architecture and Delta Lake.
• Perform Spark and Databricks performance tuning to improve processing efficiency and reduce costs.
• Implement and maintain MLOps workflows for model deployment, monitoring, and lifecycle management.
• Leverage Databricks Mosaic AI capabilities for Generative AI and ML use cases.
• Collaborate with Data Scientists, ML Engineers, Architects, and business stakeholders.
• Ensure data quality, governance, security, and compliance standards are followed.
• Automate data engineering processes using CI/CD pipelines and Infrastructure as Code.
• Troubleshoot and resolve complex production issues across data platforms.

Expertise You\'ll Bring:

• The ideal candidate should possess strong technical expertise in cloud-based data engineering, big data technologies, and AI/ML ecosystems.
• 10+ years of overall IT experience with strong Data Engineering background.
• Hands-on experience with AWS services including S3, Glue, EMR, Lambda, Redshift, Athena, IAM, and Step Functions.
• Extensive experience working with Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
• Strong development experience using PySpark and Spark SQL.
• Expertise in Spark Performance Tuning, Query Optimization, Partitioning, and Cluster Management.
• Experience implementing MLOps frameworks using MLflow and CI/CD methodologies.
• Hands-on experience with Databricks Mosaic AI, LLMs, RAG, and Generative AI solutions.
• Strong proficiency in Python and SQL.
• Experience with version control tools such as Git.
• Knowledge of Terraform, Docker, Kubernetes, and Airflow will be an added advantage.
• Strong analytical, problem-solving, and communication skills.
• Ability to work independently and collaborate with cross-functional teams.
• Bachelor\'s or master\'s degree in computer science, Information Technology, Data Science, Engineering, or a related field.
• Relevant AWS and Databricks certifications are highly preferred.
• AWS Certified Solutions Architect
• Databricks Machine Learning Professional Certification
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let\'s unleash your full potential at Persistent - persistent.com/careers

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."