AWS Databricks Architect

Persistent Systems; Bengaluru; Hybrid; INR 4,000,000 - 6,000,000; Full time

Benefits offered by this job

Insurance coverage

Job summary

Persistent Systems is seeking an experienced Lead Data Engineer in Bengaluru to design, build, and optimize enterprise-scale data platforms supporting analytics, ML, and GenAI initiatives. You will lead cloud-native data solutions leveraging AWS and Databricks, ensuring scalable, secure architectures.

The role requires deep expertise in PySpark, data lakehouse paradigms, and collaboration with AI teams. Hybrid work setup and a competitive compensation package are offered.

Qualifications

• 12-14 years of experience in Data Engineering and enterprise-scale data platform implementation.
• Strong expertise in designing and building large-scale data pipelines and modern data architectures.
• Experience with enterprise data lakes, lakehouses, and analytical platforms.
• Hands-on experience with AWS services and cloud-native architectures.
• Experience with PySpark and Databricks.

Responsibilities

• Design, build, and optimize scalable data pipelines for analytics, ML, and Generative AI workloads.
• Develop cloud-native data engineering solutions using AWS and Databricks.
• Translate business requirements into scalable, secure, and high-performance data architectures.
• Design and implement modern data lake, lakehouse, and enterprise data platform solutions.
• Support Enterprise Data Platform (EDP) architecture and roadmap initiatives.
• Develop data ingestion, transformation, and integration frameworks for structured and unstructured data.
• Enable AI-ready data ecosystems that support advanced analytics and ML use cases.
• Build and optimize distributed data processing solutions using PySpark and Databricks.
• Collaborate with data scientists and AI teams to support ML, GenAI, and analytics initiatives.
• Design and implement data models supporting reporting, analytics, AI, and operational workloads.
• Ensure data quality, governance, lineage, security, and compliance across the data platform.
• Participate in architecture reviews and technology roadmap planning.
• Drive DataOps, DevOps, MLOps, and LLMOps practices across data and AI programs.
• Monitor platform performance and optimize solutions for scalability, reliability, and cost efficiency.
• Support deployment, operationalization, and lifecycle management of AI and ML solutions.
• Provide technical leadership and mentoring to engineering teams.
• Work closely with stakeholders to define data strategy and deliver business-focused solutions.
• Contribute to continuous improvement initiatives and adoption of emerging data and AI technologies.

Skills

AWS

Databricks

PySpark

Generative AI

Data Engineering

Tools

Amazon S3

AWS Glue

AWS Lambda

Amazon Redshift

Amazon EMR

Amazon SageMaker

Job description

We are looking for an experienced Lead Data Engineer with strong expertise in AWS, Databricks, Data Engineering, and Generative AI technologies. The ideal candidate will be responsible for designing, building, and optimizing enterprise-scale data platforms that support analytics, machine learning, and AI-driven business initiatives.

• Experience: 12 to 14 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, build, and optimize scalable data pipelines for analytics, machine learning, and Generative AI workloads.
• Develop cloud-native data engineering solutions using AWS and Databricks.
• Translate business requirements into scalable, secure, and high-performance data architectures.
• Design and implement modern data lake, lakehouse, and enterprise data platform solutions.
• Support Enterprise Data Platform (EDP) architecture and roadmap initiatives.
• Develop data ingestion, transformation, and integration frameworks for structured and unstructured data.
• Enable AI-ready data ecosystems that support advanced analytics and machine learning use cases.
• Build and optimize distributed data processing solutions using PySpark and Databricks.
• Collaborate with data scientists and AI teams to support ML, GenAI, and analytics initiatives.
• Design and implement data models supporting reporting, analytics, AI, and operational workloads.
• Ensure data quality, governance, lineage, security, and compliance across the data platform.
• Participate in architecture reviews and technology roadmap planning.
• Drive DataOps, DevOps, MLOps, and LLMOps practices across data and AI programs.
• Monitor platform performance and optimize solutions for scalability, reliability, and cost efficiency.
• Support deployment, operationalization, and lifecycle management of AI and ML solutions.
• Provide technical leadership and mentoring to engineering teams.
• Work closely with stakeholders to define data strategy and deliver business-focused solutions.
• Contribute to continuous improvement initiatives and adoption of emerging data and AI technologies.

Expertise You'll Bring:

• 12-14 years of experience in Data Engineering and enterprise-scale data platform implementation.
• Strong expertise in designing and building large-scale data pipelines and modern data architectures.
• Experience working with enterprise data lakes, lakehouses, and analytical platforms.
• Strong understanding of data modeling, data warehousing, and distributed data processing concepts.
• Experience handling high-volume, high-velocity, and large-scale data workloads.
• Extensive hands-on experience with AWS services and cloud-native architectures.
• Experience working with services such as Amazon S3, AWS Glue, AWS Lambda, Amazon Redshift, Amazon EMR, Amazon Athena, Amazon SageMaker, AWS IAM, AWS CloudWatch.
• Strong understanding of cloud security, governance, and operational best practices.
• Experience designing highly available and scalable cloud solutions.
• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalization for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment

Let's unleash your full potential at Persistent - persistent.com/careers

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."