Lead Data Engineer

Epiroc Mining India; Bengaluru; Hybrid; INR 4,500,000 - 6,000,000; Full time

Job summary

Epiroc Mining India in Bengaluru is seeking a Lead Data Engineer to architect and steward scalable data pipelines, ensuring reliability, governance, and data quality across domains. You will collaborate with AI/ML engineers, data scientists, and product teams to design data models, optimize processing, monitor SLA adherence, and integrate new data sources from global platforms.

Steering cloud-based data architecture and mentoring peers, you will drive performance improvements and align data

Qualifications

• Bachelor's or Master's degree in Computer Science Engineering or related field.
• 10+ years of professional experience in data engineering.
• Proficiency with Python, PySpark and SQL for scalable data solutions.

Responsibilities

• Continuously monitor data pipeline performance and reliability, optimizing workflows and proactively resolving issues.
• Integrate and connect various data sources from global platforms.
• Identify, evaluate, and integrate new data sources to broaden data scope and quality.
• Optimize indexing strategies for large-scale, multi-domain datasets.
• Ensure adherence to data quality, consistency, and enterprise compliance standards.
• Develop monitoring and alerting to track data pipeline health and SLA compliance.
• Collaborate with AI/ML engineers, data scientists, and product teams to deliver data-driven solutions.
• Work with cross-functional teams to align data initiatives with project objectives and deadlines.

Skills

Python

PySpark

SQL

REST APIs

Generative AI concepts

LLM integration

Education

Bachelor's or Master's degree in Computer Science Engineering or related field

Tools

Azure DataBricks

Azure DevOps

Visual Studio Code

Job description

Job Title : Lead Data Engineer

Location : Bangalore

Responsibilities :

• Continuously monitor data pipeline performance and reliability, optimizing workflows and proactively resolving issues. requirements.
• Continuously monitor data pipeline performance and reliability, optimizing workflows and proactivelyresolving issues.
• Integrate and connect various data sources, including structured and unstructured data from global platforms.
• Identify, evaluate, and integrate new data sources to broaden the scope and improve the quality of available data sets.
• Optimize indexing strategies for handling large-scale, multi-domain datasets efficiently.
• Ensure strict adherence to data quality, consistency, and enterprise compliance standards throughout data processing.
• Develop and implement monitoring and alerting mechanisms to track data pipeline health and SLA compliance.
• Partner with AI/ML engineers, data scientists, and product teams to enable seamless delivery of data-driven solutions.
• Collaborate with cross-functional teams to ensure data engineering initiatives align with strategic project objectives and deadlines.

Qualifications & Skills :

• Bachelor's or Masters degree in Computer Science Engineering or related field, accompanied by 10+years of professional experience in data engineering.
• Proficiency with programming languages including Python, PySpark, and SQL for developing efficient andscalable data solutions.
• Hands-on experience with leading data engineering tools such as Azure DataBricks, Azure DevOps, andVisual Studio Code.
• Strong knowledge of REST APIs, facilitating effective data integration and automation.
• Familiarity with Generative AI concepts and experience integrating Large Language Models (LLM) ishighly desirable.
• Solid understanding of cloud-based data platforms and designing scalable data architectures.