Lead Data Engineer

Algoworks; India; On-site; INR 2,500,000 - 4,500,000; Full time

Job summary

Algoworks in India is seeking a hands-on Senior Data Engineer with strong expertise in Azure Databricks and Azure Data Factory to build, optimize, and maintain scalable enterprise data pipelines.

You will design and implement Bronze, Silver, and curated layers, optimize Delta Lake, and collaborate with DWH and reporting teams to enable downstream analytics.

The role emphasizes reliability, performance, data quality, CI/CD practices, and adherence to coding standards in an Agile environment.

Qualifications

• Hands-on data engineering experience in enterprise data platforms.
• Strong expertise with Azure Databricks and Azure Data Factory.
• Proficient in PySpark, SQL, and Delta Lake optimization.
• Experience designing Bronze/Silver/Curated data layers and ETL/ELT patterns.
• Familiar with Git and basic CI/CD for data pipelines.
• Ability to work in Agile delivery environments.

Responsibilities

• Build and maintain scalable data pipelines using Azure Databricks and Data Factory.
• Develop Bronze, Silver, and curated data layers with Delta Lake.
• Optimize storage formats and performance of Delta Lake tables.
• Collaborate with DWH and reporting teams for downstream analytics.
• Implement CI/CD practices and maintain reusable pipeline code.
• Troubleshoot production data issues and ensure data quality.

Skills

PySpark

SQL

ETL/ELT

Delta Lake

Azure Databricks

Batch Processing

Incremental Processing

Parquet

Git & Version Control

CI/CD pipelines

Tools

Azure Data Factory

Delta Lake

Spark tuning

Microsoft Fabric (nice to have)

Job description

Algoworks is an award-winning artificial intelligence, engineering services and experience transformation firm with offices across the United States, Europe, South America and India. We bring together a global team of engineers, architects, designers, researchers and operators united by rigor, accountability, and a commitment to delivering measurable results.

For over 20 years, Algoworks has partnered with Fortune 500 organizations across the Americas, Europe and Asia to define, build and run technology that drives meaningful business outcomes. Our work combines human-centered design, engineering excellence and AI-powered capabilities to solve complex challenges with clarity and precision. Innovation, particularly in the responsible application of AI, is embedded in how teams approach problem-solving and continuous improvement.

At Algoworks, growth is continuous and closely tied to impact. Teams collaborate across geographies and disciplines, strengthening outcomes through shared insight and collective expertise. The culture values transparency, open dialogue and an environment where every voice is heard and contribution is recognized.

Through collaboration, accountability and a focus on results, Algoworks operates at the intersection of technology and people, building not only advanced systems but strong global teams that elevate performance and create lasting impact.

Follow the video below to know about us!

Clipchamp

Role overview

We are seeking a hands-on Senior Data Engineer with strong expertise in Azure Databricks and Azure Data Factory to build, optimize, and maintain scalable enterprise data pipelines.

The role will focus on high-performance ETL/ELT development, Delta Lake optimization, data processing across Bronze, Silver, and curated layers, and close collaboration with DWH and reporting teams for downstream consumption.

The ideal candidate will act as a senior technical contributor, ensuring reliability, performance, data quality, and maintainability across the data platform.

Key responsibilities:

• Build and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Implement ingestion and transformation logic across Bronze and Silver data layers.
• Develop batch and incremental data-processing patterns.
• Design reliable and reusable pipeline components for enterprise workloads.
• Monitor and troubleshoot pipeline execution and data-processing issues.

2.Curated Layer & Delta Lake Development

• Implement hydration, merge, and upsert logic using Delta Lake.
• Build and maintain curated datasets aligned with data quality and business requirements.
• Handle late-arriving data and incremental updates.
• Implement reliable data transformation and reconciliation processes.
• Ensure curated datasets are optimized for downstream consumption.

3.Performance & Storage Optimization

• Optimize Delta Lake tables for performance and cost efficiency.
• Select and tune appropriate storage formats such as Parquet and Delta.
• Apply partitioning, compaction, and file-sizing strategies.
• Identify and resolve performance bottlenecks across data pipelines and storage layers.
• Work closely with DWH and reporting teams to support downstream data consumption.
• Provide optimized datasets for reporting and analytical workloads.
• Support data validation and reconciliation with Gold-layer outputs.
• Collaborate with downstream teams to understand data requirements and optimize delivery.
• Ensure consistency and reliability of data consumed by reporting and analytics platforms.
• Implement basic CI/CD practices for data pipelines.
• Follow coding standards, documentation, and version-control practices.
• Maintain reusable, scalable, and maintainable pipeline code.
• Support production troubleshooting and performance tuning.
• Participate in Agile delivery processes and technical discussions.

6.Data Quality & Production Support

• Implement data validation and quality checks across ingestion and transformation processes.
• Investigate data discrepancies and pipeline failures.
• Perform root-cause analysis and implement corrective actions.
• Support production deployments and resolve data-processing issues.
• Maintain reliability and consistency across enterprise data pipelines.

Required technical skills and competencies:

• Strong hands-on experience in data engineering and enterprise data platforms.
• Strong experience building data pipelines on Azure.
• Advanced proficiency in PySpark.
• Hands-on experience with Azure Databricks.
• Strong experience with Azure Data Factory.
• Deep knowledge of Delta Lake tuning and optimization.
• Strong understanding of storage optimization using Parquet and Delta.
• Strong SQL skills for data transformation, validation, and reconciliation.
• Experience working with large datasets and distributed processing.
• Experience implementing batch and incremental processing patterns.
• Experience with hydration, merge, and upsert logic.
• Experience with Git and basic CI/CD pipelines.
• Familiarity with data quality and validation techniques.
• Experience working in Agile delivery environments.

Must have skills:

• PySpark.
• SQL.
• ETL/ELT.
• Bronze/Silver/Curated Data Layers.
• Delta Lake Performance Optimization.
• Spark Performance Tuning.
• Parquet.
• Batch and Incremental Processing.
• Git and Version Control.
• Strong analytical and problem-solving skills.

Good to have skills:

• Microsoft Fabric.
• Streaming or near real-time data pipelines.
• Data governance tools.
• Metadata management tools.
• Advanced CI/CD practices.
• Experience with Gold-layer development.
• Experience supporting enterprise reporting and DWH platforms.

Desired attributes:

• Strong analytical and problem-solving capabilities.
• Ability to independently design and develop complex data pipelines.
• Strong focus on performance, scalability, reliability, and data quality.
• Ability to troubleshoot complex production data issues.
• Strong attention to detail and coding discipline.
• Good communication and collaboration skills.
• Ability to work effectively with DWH, reporting, and cross-functional teams.
• Proactive approach to performance optimization and continuous improvement.
• Strong ownership of data engineering deliverables.