Specialist - Data Engineering

Vivantify Technology Solutions India Private Limited; Bengaluru; Sur place; INR 1 800 000 - 3 200 000; Plein temps

Avantages offerts par ce poste

Competitive salary and benefits

Opportunities for professional growth

Collaborative and inclusive work环境

Challenging projects with leading клиє

Bellandur, Bengaluru

Résumé du poste

Vivantify Technology Solutions India Private Limited in Bengaluru (Bellandur) is seeking an experienced Senior Data Engineer to design, develop, and optimize large-scale data pipelines and analytics platforms using Apache Spark, Databricks, SQL, Python, and Azure data services.

You will implement ETL/ELT workflows, build data warehouses and lakehouse architectures, ensure data quality and governance, and collaborate with data scientists and business stakeholders to deliver scalable, reliable

Qualifications

• Bachelor's or Master's degree in CS/Engineering/Data Science or related field.
• Experience with CI/CD, DevOps, and cloud-based data platforms.
• Azure Data Engineer Associate certification is preferred.

Responsabilités

• Design, develop, and optimize ETL/ELT pipelines for large-scale data processing.
• Build scalable data engineering solutions using Spark, Databricks, SQL, and Python.
• Develop batch and real-time data ingestion frameworks.
• Design and implement data warehouses, data marts, and lakehouse architectures.
• Create and maintain data models to meet business requirements.
• Ensure data quality, governance, security, and compliance.
• Develop streaming data solutions and real-time processing frameworks.
• Use ADF, Synapse, and ADLS for orchestration and transformation.
• Collaborate with data scientists and analysts to deliver analytics solutions.
• Monitor and optimize data platform performance and reliability.

Connaissances

Data Engineering and Data Architecture

ETL/ELT Pipeline Development

Apache Spark and Databricks

SQL and Python

Data Warehousing and Data Modeling

Data Quality and Governance

Streaming Data Processing

Azure Data Factory (ADF)

Azure Synapse Analytics

Azure Data Lake Storage (ADLS)

Performance Tuning and Optimization

Azure IoT

Formation

Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field

Outils

Azure Data Factory (ADF)

Azure Synapse Analytics

Azure Data Lake Storage (ADLS)

Apache Spark

Databricks

Description du poste

Location: Bellandur, Bengaluru

Experience: 7–8 Years
Employment Type: Permanent
Openings: 1
Shift: 1:30 PM – 10:30 PM IST

About the Role

We are seeking an experienced Senior Data Engineer with strong expertise in data engineering, ETL/ELT development, Apache Spark, Databricks, SQL, Python, and Azure Data Services. The role focuses on designing and maintaining scalable data pipelines, enterprise data platforms, and analytics solutions that support business intelligence, advanced analytics, and AI/ML initiatives.

Job Description

The Senior Data Engineer will design, develop, and optimize high-performance data pipelines for large-scale batch and real-time data processing. The candidate will work with Apache Spark, Databricks, SQL, and Python to build reliable data processing solutions and implement enterprise data warehouses, data marts, and lakehouse architectures.

The role involves developing data ingestion frameworks, creating logical and physical data models, and implementing data quality, governance, security, and compliance practices. The engineer will also leverage Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake Storage (ADLS) to orchestrate data workflows and transform data efficiently.

The ideal candidate will collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions. Responsibilities also include monitoring data platform performance, troubleshooting issues, and optimizing reliability and scalability.

Key Responsibilities

• Design, develop, and optimize ETL/ELT pipelines for large-scale data processing.
• Build and maintain scalable data engineering solutions using Apache Spark, Databricks, SQL, and Python.
• Develop robust batch and real-time data ingestion frameworks.
• Design and implement enterprise data warehouses, data marts, and lakehouse architectures.
• Create and maintain logical and physical data models to meet business requirements.
• Ensure data quality, governance, security, and compliance across data platforms.
• Develop streaming data solutions using event-driven architectures and real-time processing frameworks.
• Use Azure Data Factory (ADF), Azure Synapse Analytics, and Azure Data Lake Storage (ADLS) for data orchestration and transformation.
• Collaborate with data scientists, analysts, and business stakeholders to deliver analytics solutions.
• Monitor, troubleshoot, and optimize data platform performance, scalability, and reliability.

Required Skills

• Data Engineering and Data Architecture
• ETL/ELT Pipeline Development
• Apache Spark and Databricks
• SQL and Python
• Data Warehousing and Data Modeling
• Data Quality and Governance
• Streaming Data Processing
• Azure Data Factory (ADF)
• Azure Synapse Analytics
• Azure Data Lake Storage (ADLS)
• Performance Tuning and Optimization

Mandatory Skills

• Azure IoT

Preferred Skills and Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, Data Science, or a related field.
• Experience with CI/CD, DevOps, and cloud-based data platforms.
• Azure Data Engineer Associate certification.
• Strong analytical, problem-solving, and stakeholder management skills.

Candidate Profile

The ideal candidate should have strong hands-on experience building scalable data pipelines and cloud-based data platforms. They should possess excellent problem-solving skills and the ability to collaborate with cross-functional teams to deliver reliable, high-performance data solutions.

Benefits Package

• Competitive salary and benefits package.
• Opportunities for professional growth and development.
• A collaborative and inclusive work environment.
• The opportunity to work on exciting and challenging projects with leading clients.