Lead Data Engineer

Apexon; Bengaluru; On-site; INR 1,400,000 - 2,000,000; Full time

Job summary

Apexon in Bengaluru, India is seeking a Data Engineer with 6-12 years of experience to build and support scalable data pipelines across Azure and Big Data platforms. You will work with Azure Databricks, PySpark, Hadoop, Talend, Delta Lake, and Azure Data Factory to enable analytics, reporting, and AI/ML initiatives.

You will collaborate with data architects, engineers, and business stakeholders to design, implement, and optimize end-to-end data solutions while ensuring data quality, security,

Qualifications

• 6-12 years of experience in Data Engineering / Big Data Engineering.
• Hands-on experience with Azure Databricks and PySpark.
• Strong experience with Hadoop and its ecosystem.
• Hands-on experience with Talend for ETL/data integration.
• Good experience with Delta Lake / Lakehouse architecture.
• Strong experience with Azure Data Factory (ADF).
• Strong SQL and data transformation skills.
• Experience working with large-scale datasets and distributed data processing.
• Understanding of cloud-based data engineering and data integration concepts.

Responsibilities

• Develop and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build data transformation and processing solutions using Delta Lake and Lakehouse architecture.
• Develop and optimize ETL/ELT pipelines using Talend for large-scale data ingestion and transformation.
• Work with Hadoop ecosystem technologies for distributed data processing and storage.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Build and manage data orchestration workflows using Azure Data Factory (ADF).
• Optimize Spark jobs, Databricks clusters, and data pipelines for performance and cost efficiency.
• Implement data quality, validation, security, and governance practices.
• Collaborate with architects, business stakeholders, and data teams to understand requirements and deliver technical solutions.
• Participate in code reviews, technical discussions, and implementation of engineering best practices.
• Work with DevOps teams on CI/CD and deployment automation for data engineering solutions.
• Troubleshoot and resolve data pipeline and production issues.

Skills

Azure Databricks

PySpark

Hadoop

Delta Lake

Azure Data Factory

SQL

Distributed processing

Big Data Engineering

Tools

Talend

Hadoop Ecosystem

Delta Lake

Azure Data Factory (ADF)

Job description

We are looking for a Data Engineer with 6-12 years of experience in building and supporting scalable data engineering solutions across Azure and Big Data platforms. The ideal candidate should have strong hands-on experience with Azure Databricks, PySpark, Hadoop, Talend, Delta Lake, and Azure Data Factory (ADF).

The role will involve developing, integrating, and optimizing data pipelines that support analytics, reporting, and AI/ML initiatives, while working closely with data architects, engineers, and business stakeholders.

Key Responsibilities

• Develop and maintain scalable data pipelines using Azure Databricks and PySpark.
• Build data transformation and processing solutions using Delta Lake and Lakehouse architecture.
• Develop and optimize ETL/ELT pipelines using Talend for large-scale data ingestion and transformation.
• Work with Hadoop ecosystem technologies for distributed data processing and storage.
• Integrate data from multiple sources including databases, APIs, files, and streaming platforms.
• Build and manage data orchestration workflows using Azure Data Factory (ADF).
• Optimize Spark jobs, Databricks clusters, and data pipelines for performance and cost efficiency.
• Implement data quality, validation, security, and governance practices.
• Collaborate with architects, business stakeholders, and data teams to understand requirements and deliver technical solutions.
• Participate in code reviews, technical discussions, and implementation of engineering best practices.
• Work with DevOps teams on CI/CD and deployment automation for data engineering solutions.
• Troubleshoot and resolve data pipeline and production issues.

Must-Have Skills

• 6-12 years of experience in Data Engineering / Big Data Engineering.
• Strong hands-on experience with Azure Databricks and PySpark.
• Strong experience with Hadoop and its ecosystem.
• Hands-on experience with Talend for ETL/data integration.
• Good experience with Delta Lake / Lakehouse architecture.
• Strong experience with Azure Data Factory (ADF).
• Strong SQL and data transformation skills.
• Experience working with large-scale datasets and distributed data processing.
• Understanding of cloud-based data engineering and data integration concepts.