Data Engineer

Master Mind Consultancy

Bengaluru

On-site

INR 1,500,000 - 3,000,000

Full time

1 hour ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Master Mind Consultancy in Bengaluru seeks an experienced Data Engineer to design, build, and maintain scalable batch and streaming data pipelines using Azure Data Factory, Databricks, Spark, and Event Hubs. You will ingest data from diverse sources and develop reusable frameworks for analytics and data science applications.

The role requires strong Python/SQL/Spark skills and hands-on Azure data platform experience, with a focus on data quality, security, and scalable architecture.

Qualifications

  • Bachelor's degree in Computer Science, Data Engineering, IT, Engineering, or related field.
  • 5–8 years of technology experience, including at least 3 years in Data Engineering, ETL/ELT, or cloud data platforms.
  • Strong proficiency in Python, SQL, Spark/PySpark, and distributed data processing.
  • Hands-on experience with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage.
  • Knowledge of Azure SQL, Synapse Analytics, or Microsoft Fabric.
  • Experience developing batch/streaming data pipelines and data integration frameworks.
  • Strong understanding of data modelling, Lakehouse architecture, and Delta Lake.
  • Knowledge of data quality, metadata, data lineage, and data governance.

Responsibilities

  • Design, develop, and maintain scalable batch and streaming data pipelines using Azure Data Factory, Azure Databricks, Apache Spark, and Event Hubs.
  • Ingest data from databases, APIs, files, SaaS applications, IoT sources, and operational systems.
  • Build reusable data ingestion, transformation, validation, and processing frameworks using Python, SQL, and PySpark.
  • Develop and maintain curated data products for analytics, reporting, and data science applications.
  • Implement incremental loading, CDC, schema evolution, partitioning, and error-handling mechanisms.
  • Build cloud data solutions using Azure Data Lake Storage, Databricks, Synapse Analytics, Azure SQL, and Microsoft Fabric.
  • Apply Medallion Architecture and optimize data pipelines for performance, scalability, and cost efficiency.
  • Implement data-quality checks, metadata management, data lineage, and governance standards.
  • Ensure data security through access controls, encryption, secrets management, and compliance practices.
  • Develop CI/CD pipelines using Azure DevOps or GitHub Actions for automated deployments.
  • Monitor pipeline performance, troubleshoot failures, and perform root-cause analysis.
  • Collaborate with Data Architects, Data Scientists, Analysts, and business teams to deliver reliable data solutions.
  • Maintain technical documentation, operational runbooks, and support production releases.

Skills

Python
SQL
Spark/PySpark
Data modeling
Analytical thinking

Education

Bachelor's degree in CS/IT/Eng

Tools

Azure Data Factory
Azure Databricks
Azure Data Lake Storage
Delta Lake
Git / CI/CD

Job description

  • Design, develop, and maintain scalable batch and streaming data pipelines using Azure Data Factory, Azure Databricks, Apache Spark, and Event Hubs.
  • Ingest data from databases, APIs, files, SaaS applications, IoT sources, and operational systems.
  • Build reusable data ingestion, transformation, validation, and processing frameworks using Python, SQL, and PySpark.
  • Develop and maintain curated data products for analytics, reporting, and data science applications.
  • Implement incremental loading, Change Data Capture (CDC), schema evolution, partitioning, and error-handling mechanisms.
  • Build cloud data solutions using Azure Data Lake Storage, Databricks, Synapse Analytics, Azure SQL, and Microsoft Fabric.
  • Apply Medallion Architecture and optimize data pipelines for performance, scalability, and cost efficiency.
  • Implement data-quality checks, metadata management, data lineage, and governance standards.
  • Ensure data security through access controls, encryption, secrets management, and compliance practices.
  • Develop CI/CD pipelines using Azure DevOps or GitHub Actions for automated deployments.
  • Monitor pipeline performance, troubleshoot failures, and perform root-cause analysis.
  • Collaborate with Data Architects, Data Scientists, Analysts, and business teams to deliver reliable data solutions.
  • Maintain technical documentation, operational runbooks, and support production releases.
Requisites
  • Bachelor's degree in Computer Science, Data Engineering, IT, Engineering, or a related field.
  • 5–8 years of technology experience, including at least 3 years in Data Engineering, ETL/ELT, or cloud data platforms.
  • Strong proficiency in Python, SQL, Spark/PySpark, and distributed data processing.
  • Hands-on experience with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage.
  • Knowledge of Azure SQL, Synapse Analytics, or Microsoft Fabric.
  • Experience developing batch/streaming data pipelines and data integration frameworks.
  • Strong understanding of data modelling, Lakehouse architecture, and Delta Lake.
  • Knowledge of data quality, metadata, data lineage, and data governance.
  • Experience with Git, CI/CD, automated testing, and Azure DevOps.
  • Understanding of cloud security, access management, monitoring, and production support.
  • Strong analytical, troubleshooting, documentation, and communication skills.
Skills :
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

Techcarrot • Chennai

On-site
INR 1,500,000 - 2,100,000
Data Engineer-Information Technology
Data Engineer-Information Technology

Emcure Pharmaceuticals Limited • Pune District

On-site
INR 1,400,000 - 2,000,000
Azure Databricks Data Engineer
Azure Databricks Data Engineer

WTW group • Kolkata District

On-site
INR 1,800,000 - 3,400,000
Azure Data Engineer
Azure Data Engineer

Saisystems Technology • Pune District, Mumbai, Gurugram District

Hybrid
INR 2,400,000 - 4,200,000
Senior Data Engineer
Senior Data Engineer

SOTIIND SOTI India Private Ltd. • Ernakulam

On-site
INR 1,500,000 - 2,000,000
Data Engineer
Data Engineer

Intact Insurance • India

On-site
INR 1,500,000 - 2,300,000
Data Engineer
Data Engineer

Intact India • Gurugram District

On-site
INR 1,200,000 - 2,400,000
Azure Data Engineer
Azure Data Engineer

Einfochips • Indore District, Ahmedabad District

On-site
INR 2,000,000 - 4,000,000
Data Engineer
Data Engineer

V Edge Ai Business Consulting • Visakhapatnam District

On-site
INR 1,200,000 - 2,400,000
Data Engineer (Azure & Databricks)
Data Engineer (Azure & Databricks)

Lufthansa Technik Services India Pvt Ltd • Bengaluru

On-site
INR 1,000,000 - 1,500,000