Lead Big Data Engineer

Mployee.me; Pune District; In loco; INR 4.000.000 - 7.000.000; Tempo pieno

Descrizione del lavoro

Mployee.me in Pune, India is seeking a senior Data Engineer to design, build, and maintain scalable data solutions that power analytics across the enterprise. You will mentor Data & Analytics talent and drive delivery excellence in large programs.

The role requires deep expertise in Big Data, cloud platforms, and modern data engineering practices, with hands-on Azure and on‑premises ETL/ELT work.

Competenze

• 12–14+ years of Data Engineering, Big Data and Cloud technologies.
• Strong SQL, Python/Scala and enterprise data pipelines.
• Expertise in Azure Data Factory, Synapse, Databricks and cloud data services.
• Experience with Medallion Architecture and data governance practices.

Mansioni

• Design, develop, and maintain ETL/ELT pipelines on Azure and on‑premises.
• Build scalable data ingestion and processing frameworks for diverse datasets.
• Address complex business problems with data engineering best practices.
• Monitor, optimize, and troubleshoot pipelines for reliability and performance.
• Ensure adherence to data quality, governance, security, and compliance.
• Collaborate with stakeholders, architects, data scientists, governance, DevOps and analytics partners.
• Lead solution design discussions, architecture reviews, and estimates.
• Mentor engineering teams and drive capability development.
• Promote reusable frameworks and engineering standards across programs.

Conoscenze

SQL

Python

Scala

Spark

Hadoop

Hive

Spark Streaming

Azure Data Factory

Azure Synapse Analytics

Azure Databricks

Cosmos DB

HBase

Data Warehouse

Data Lake

Medallion Architecture

Leadership

Communication

Formazione

Bachelor's degree in Computer Science/Information Technology/Engineering

Strumenti

MongoDB

NiFi

Git

Azure DevOps

Descrizione del lavoro

Data Engineer

The Opportunity

As a Data Engineer, you will be responsible for designing, building, and maintaining scalable data solutions that support enterprise-wide analytics and business intelligence initiatives. The role requires deep expertise in Big Data technologies, cloud platforms, and modern data engineering practices. In addition to technical leadership, you will play a key role in mentoring and developing Data & Analytics talent while driving delivery excellence across complex programs.

Key Responsibilities

• Design, develop, and maintain robust ETL/ELT pipelines on Azure cloud and on-premises platforms for batch and real-time data processing.
• Build scalable data ingestion and processing frameworks to handle structured, semi-structured, and unstructured datasets.
• Solve complex business problems by leveraging data engineering best practices and advanced technologies.
• Monitor, optimize, and troubleshoot data pipelines to ensure high availability, reliability, performance, and scalability.
• Ensure adherence to data quality, governance, security, privacy, and compliance standards.
• Collaborate with business stakeholders, architects, data scientists, governance teams, DevOps teams, and analytics partners to deliver business value.
• Lead solution design discussions, technical estimations, architecture reviews, and implementation strategies.
• Provide technical guidance and mentorship to engineering teams while fostering continuous learning and capability development.
• Drive adoption of engineering standards, reusable frameworks, and best practices across programs.

Primary Requirements

• Technical Skills
• Strong experience with SQL, Python, and/or Scala.
• Hands-on expertise in Big Data technologies including Apache Spark, Hadoop, and Hive.
• Experience designing and developing enterprise-scale ETL/ELT pipelines.
• Expertise in Azure Data Engineering ecosystem:
  • Azure Data Factory (ADF)
  • Azure Synapse Analytics
  • Azure Databricks
  • Azure Event Hub
  • Azure Functions
• Experience with NoSQL and Distributed Databases such as Cosmos DB and HBase.
• Strong understanding of Data Warehouses, Data Lakes, and Data Marts.
• Experience implementing Medallion Architecture and modern data design patterns.
• Expertise in data performance tuning, optimization, and quality validation.
• Experience building both real-time and batch processing pipelines, including Spark Streaming.
• Additional Requirements
• Excellent communication and stakeholder management skills.
• Strong analytical and structured problem-solving capabilities.
• Proven experience in solution architecture, estimation, and technical leadership.
• Ability to effectively communicate technical concepts through storytelling and presentations.
• Experience managing relationships with business partners and senior stakeholders.
• Demonstrated people leadership experience, including coaching, mentoring, talent development, and engagement.

Good to Have

• Experience with DevOps and CI/CD practices using Git and Azure DevOps.
• Unix/Linux shell scripting expertise.
• Experience with MongoDB and Apache NiFi.
• Understanding of Lambda and Kappa architecture patterns.
• Exposure to Generative AI technologies, tools, and use cases.
• Banking, Financial Services, and Insurance (BFSI) domain experience.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or equivalent educational qualification.
• 12-14+ years of relevant experience in Data Engineering, Big Data, and Cloud technologies.