Senior Data Engineer

NewVision Software; Pune District; Hybrid; INR 2,000,000 - 4,200,000; Full time

Job summary

New Vision Softcom & Consultancy Pvt. Ltd in Pune is seeking a Senior Data Engineer with 5+ years of experience to design, develop, and optimize ETL pipelines on Azure data platforms.

You will implement data quality checks, schedule workflows with ADF or Apache Airflow, and tune SQL and Spark performance. The role requires strong SQL, Python, Spark, PySpark, and Azure skills, with exposure to notebooks for development and performance tuning.

Qualifications

• Bachelor’s degree in computer science, information technology, or a related field.
• 5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
• Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
• Proven experience in performance tuning and optimization of SQL and ETL processes.
• Strong understanding of Data Architecture and Data Modeling.
• Experience in healthcare and pharmaceutical industries is preferred.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Responsibilities

• Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
• Optimize ETL processes for performance and scalability.
• Implement data quality and validation checks within ETL processes.
• Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
• Schedule and monitor ETL jobs to ensure timely data processing and availability.
• Optimize SQL queries and data processing tasks for performance.
• Utilize notebooks for development and performance tuning of ETL processes.
• Develop and maintain data models and schemas; ensure data integrity, consistency, and security.
• Collaborate with data engineers, solution architects, and stakeholders to translate business requirements into technical solutions.

Skills

Databricks

SQL

Python

Spark

PySpark

Azure

ADF / Apache Airflow

Fabric

Education

Bachelor’s degree in computer science / IT or related field

Tools

Databricks

SQL

Python

Spark

PySpark

Azure

Azure Data Factory

Apache Airflow

Fabric

Job description

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice AI & Analytics

Region APAC

Country India

Base office location Pune

Working model Hybrid

Weekly Off Pune office Standard

State Maharashtra

Job Description

Senior Data Engineer with 5+ years of experience

Mandatory Skills:

• Databricks
• performance tuning using SQL.
• SQL, Python, Spark, PySpark, and Azure.
• Fabric experience preferred.

Key Responsibilities:

• Design, develop, and maintain ETL pipelines to extract, transform, and load data from various sources into Azure data platforms.
• Optimize ETL processes for performance and scalability.
• Implement data quality and validation checks within ETL processes.
• Scheduling and Orchestration:
• Develop and manage data integration workflows using Azure Data Factory (ADF) and/or Apache Airflow.
• Schedule and monitor ETL jobs to ensure timely data processing and availability.
• Performance Tuning:
• Optimize SQL queries and data processing tasks for performance.
• Utilize notebooks for development and performance tuning of ETL processes.
• Data Integration and Management:
• Develop and maintain data models and schemas.
• Ensure data integrity, consistency, and security across data platforms.
• Collaborate with data engineers, solution architects, and other stakeholders to understand business requirements and translate them into technical solutions.

Qualifications:

• Bachelor’s degree in computer science, Information Technology, or a related field.
• 5-6 years of experience as a Data Engineer in developing ETL pipelines and scheduling using ADF and/or Apache Airflow.
• Strong technical skills in SQL, Python, Spark, PySpark, and Azure.
• Proven experience in performance tuning and optimization of SQL and ETL processes.
• Strong understanding of Data Architecture and Data Modeling.
• Experience in healthcare and pharmaceutical industries is preferred.
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

Preferred Skills:

• Knowledge of big data technologies (Hadoop, Hive, Kafka).
• Familiarity with data visualization tools (Power BI, Tableau).