Data Engineer- LLM

Persistent Systems; Pune District; Hybrid; INR 2,500,000 - 4,500,000; Full time

Benefits offered by this job

Competitive salary

Hybrid work model

Career development

Job summary

Persistent Systems is seeking a hands-on Data Engineer with 6–10 years of experience in Data Engineering, Data Warehousing, and Big Data development. The role emphasizes Azure cloud technologies, Databricks, SQL, Python, and modern data platforms to deliver scalable data pipelines and analytics-ready data products.

You will design ETL/ELT pipelines, build data warehouses and data lakes, and partner with analytics and AI/ML teams to enable enterprise data initiatives.

Qualifications

• 6–10 years in Data Engineering, Data Warehousing or Big Data development.
• Proficiency in SQL and Python for data workloads.
• Hands-on experience building scalable ETL/ELT pipelines.
• Strong expertise with Azure cloud data services and enterprise platforms.
• Experience with Databricks, Spark, and Lakehouse architectures.
• Knowledge of data governance, quality, and metadata management.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines and integration frameworks.
• Build and optimize data warehouses, data lakes, and cloud-native platforms.
• Develop data processing solutions using Azure, Databricks, SQL, Python, and Spark.
• Ensure data quality, integrity, availability, reliability, and performance.
• Collaborate with analytics, AI/ML, reporting, and business teams to deliver data products.
• Support AI/ML initiatives through feature engineering and data prep.

Skills

Data engineering

SQL

Python

Databricks

Azure

Big data

LLM/AI data

Data modeling

Tools

Azure Data Factory

Azure Synapse Analytics

Databricks

Snowflake

Spark

Job description

About Position:

We are seeking a hands-on Data Engineer with 6 to 10 years of experience in Data Engineering, Data Warehousing, and Big Data development. The ideal candidate will have strong expertise in Azure cloud technologies, Databricks, SQL, Python, and modern data platform architectures. This role involves building scalable data pipelines, enabling analytics and AI/ML initiatives, and delivering reliable, high-performance data solutions that support enterprise business objectives.

• Role: Data Engineer- LLM
• Location: All Persistent Locations
• Experience: 6 to 10 Years
• Job Type: Full Time Employment

What You'll Do:

• Design, develop, and maintain scalable ETL/ELT data pipelines and integration frameworks.
• Build and optimize enterprise data warehouses, data lakes, and cloud-native data platforms.
• Develop robust data processing solutions using Azure, Databricks, SQL, Python, and Spark.
• Ensure data quality, integrity, availability, reliability, and performance across enterprise data environments.
• Partner with analytics, AI/ML, reporting, and business teams to deliver trusted data products.
• Support AI/ML initiatives through feature engineering, data preparation, and pipeline development.
• Work with Azure Data Factory, Synapse Analytics, Databricks, Snowflake, and related enterprise data platforms.
• Implement data governance, metadata management, security, and compliance best practices.
• Build monitoring, automation, and operational processes to improve platform reliability and efficiency.
• Optimize data models, queries, and processing frameworks for performance and scalability.
• Support structured and unstructured data processing across cloud-based data ecosystems.
• Participate in Agile ceremonies and collaborate with cross-functional stakeholders.
• Implement CI/CD pipelines, DevOps practices, Infrastructure as Code, and platform automation strategies.
• Troubleshoot and resolve complex data platform, integration, and performance issues.
• Contribute to data architecture decisions and best practices for enterprise data engineering.
• AI/ML Data Enablement and Feature Engineering
• Contact Center Analytics and Reporting Platforms
• Operational Data Mart and KPI Enablement
• Data Pipeline Automation and Optimization Programs
• Enterprise Data Modernization and Cloud Migration Initiatives

Expertise You'll Bring:

• 6 to 10 years of experience in Data Engineering, Data Warehousing, or Big Data development.
• Strong proficiency in SQL and Python for data engineering, transformation, and analytics workloads.
• Hands-on experience building and supporting scalable ETL/ELT pipelines.
• Strong expertise with Azure cloud data services and enterprise data platforms.
• Experience working with Databricks, Spark, and modern Lakehouse architectures.
• Strong knowledge of data modeling, dimensional modeling, and data warehouse design principles.
• Experience implementing data governance, data quality, lineage, and metadata management practices.
• Hands-on experience with AI/ML data engineering, feature engineering, and data preparation workflows.
• Ability to manage large-scale structured and unstructured datasets.
• Experience building highly available, scalable, and performance-optimized data solutions.
• Enterprise Data Engineering and Data Platform development best practices.
• Business Intelligence, Advanced Analytics, AI/ML, and data-driven solution delivery.
• Azure Data Factory, Azure Synapse Analytics, Databricks, Snowflake, and related cloud technologies.
• Healthcare, payer/provider, claims, operational, or contact center data domains.
• Data Lake, Lakehouse, and Enterprise Data Warehouse architectures.
• CI/CD practices, DevOps methodologies, Infrastructure as Code, and platform automation.
• Data security, privacy, regulatory compliance, and governance frameworks.
• Agile delivery methodologies and cross-functional team collaboration.
• Data lifecycle management, monitoring, observability, and operational excellence.
• Modern AI and GenAI data platform requirements and architecture.
• Maintain a minimum of 90% weekly usage of GitHub Copilot, Microsoft 365 Copilot, and other approved enterprise AI tools.
• Leverage AI tools to enhance software development, data engineering, documentation, and analytical workflows.
• Utilize GenAI and LLM-based capabilities to improve productivity, code quality, and delivery velocity.
• Stay current with emerging AI technologies, enterprise AI governance, and best practices.
• Apply AI-assisted engineering approaches while maintaining security, compliance, and quality standards.

Benefits:

• Competitive salary and benefits package
• Culture focused on talent development with quarterly growth opportunities and company-sponsored higher education and certifications
• Opportunity to work with cutting-edge technologies
• Employee engagement initiatives such as project parties, flexible work hours, and Long Service awards
• Annual health check-ups
• Insurance coverage: group term life, personal accident, and Mediclaim hospitalisation for self, spouse, two children, and parents

Values-Driven, People-Centric & Inclusive Work Environment:

Persistent is dedicated to fostering diversity and inclusion in the workplace. We invite applications from all qualified individuals, including those with disabilities, and regardless of gender or gender preference. We welcome diverse candidates from all backgrounds.

• We support hybrid work and flexible hours to fit diverse lifestyles.
• Our office is accessibility-friendly, with ergonomic setups and assistive technologies to support employees with physical disabilities.
• If you are a person with disabilities and have specific requirements, please inform us during the application process or at any time during your employment.

"Persistent is an Equal Opportunity Employer and prohibits discrimination and harassment of any kind."