AWS Databricks Architect

Persistent; Bengaluru; On-site; INR 3,500,000 - 7,000,000; Full time

Benefits offered by this job

Competitive salary

Talent development

Cutting-edge technologies

Employee engagement initiatives

Annual health check-ups

Insurance coverage

Job summary

Persistent is seeking an experienced Lead Data Engineer to design, build, and optimize enterprise-scale data platforms supporting analytics, ML, and GenAI initiatives in Bengaluru.

You will lead cloud-native data pipelines on AWS and Databricks, drive DataOps practices, ensure data quality and security, mentor engineering teams, and partner with data scientists to enable AI-driven business outcomes.

Qualifications

• Extensive experience in building enterprise-scale data platforms for analytics and ML.
• Proficient in designing scalable data pipelines on AWS and Databricks.
• Strong understanding of data governance, security, and compliance.
• Proven ability to lead, mentor, and collaborate with AI/data science teams.

Responsibilities

• Design, build, and optimize scalable data pipelines for analytics, ML, and GenAI workloads.
• Develop cloud-native solutions using AWS and Databricks.
• Translate business needs into scalable, secure data architectures.
• Design modern data lake/lakehouse and enterprise data platform solutions.
• Support EDP architecture and roadmap initiatives.
• Enable AI-ready data ecosystems for advanced analytics and ML use cases.
• Build and optimize distributed data processing with PySpark and Databricks.
• Mentor engineering teams and collaborate with data scientists.
• Ensure data quality, governance, lineage, security, and compliance.

Skills

Data engineering

AWS

Databricks

GenAI

PySpark

Data modeling

MLOps

Data governance

Tools

Amazon S3

AWS Glue

AWS Lambda

Amazon Redshift

Amazon EMR

Amazon Athena

Amazon SageMaker

AWS IAM

AWS CloudWatch

Job description

About Position:

We are looking for an experienced Lead Data Engineer with strong expertise in AWS, Databricks, Data Engineering, and Generative AI technologies. The ideal candidate will be responsible for designing, building, and optimizing enterprise-scale data platforms that support analytics, machine learning, and AI-driven business initiatives.

• Role: Lead Data Engineer AWS Databricks & GenAI
• Location: Bengaluru
• Experience: 12 to 14 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, build, and optimize scalable data pipelines for analytics, machine learning, and Generative AI workloads.
• Develop cloud-native data engineering solutions using AWS and Databricks.
• Translate business requirements into scalable, secure, and high-performance data architectures.
• Design and implement modern data lake, lakehouse, and enterprise data platform solutions.
• Support Enterprise Data Platform (EDP) architecture and roadmap initiatives.
• Develop data ingestion, transformation, and integration frameworks for structured and unstructured data.
• Enable AI-ready data ecosystems that support advanced analytics and machine learning use cases.
• Build and optimize distributed data processing solutions using PySpark and Databricks.
• Collaborate with data scientists and AI teams to support ML, GenAI, and analytics initiatives.
• Design and implement data models supporting reporting, analytics, AI, and operational workloads.
• Ensure data quality, governance, lineage, security, and compliance across the data platform.
• Participate in architecture reviews and technology roadmap planning.
• Drive DataOps, DevOps, MLOps, and LLMOps practices across data and AI programs.
• Monitor platform performance and optimize solutions for scalability, reliability, and cost efficiency.
• Support deployment, operationalization, and lifecycle management of AI and ML solutions.
• Provide technical leadership and mentoring to engineering teams.
• Work closely with stakeholders to define data strategy and deliver business-focused solutions.
• Contribute to continuous improvement initiatives and adoption of emerging data and AI technologies.

Expertise You'll Bring:

• 12-14 years of experience in Data Engineering and enterprise-scale data platform implementation.
• Strong expertise in designing and building large-scale data pipelines and modern data architectures.
• Experience working with enterprise data lakes, lakehouses, and analytical platforms.
• Strong understanding of data modeling, data warehousing, and distributed data processing concepts.
• Experience handling high-volume, high-velocity, and large-scale data workloads.
• Extensive hands‑on experience with AWS services and cloud-native architectures.
• Experience working with services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, Amazon EMR, Amazon Athena, Amazon SageMaker, AWS IAM, AWS CloudWatch.
• Strong understanding of cloud security, governance, and operational best practices.
• Experience designing highly available and scalable cloud solutions.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."