Senior Software Engineer- Data

NewVision; South Carolina; Hybrid; USD 120,000 - 180,000; Full time

Job summary

NewVision Software in the United States is seeking a Senior Data Engineer with 6+ years of experience to design and build scalable data pipelines on the Azure platform. You will work with Azure Data Factory, Databricks, PySpark, and ADLS to deliver robust data solutions and collaborate with architects, analysts, and stakeholders.

Responsibilities include designing ETL/ELT processes, implementing data quality checks, and optimizing pipelines for performance and cost.

Qualifications

• 6+ years of professional experience in Data Engineering.
• Strong hands-on experience with Microsoft Azure data services.
• Proficiency in Python, PySpark, and Databricks.
• Experience with ADLS, SQL and data warehousing concepts.

Responsibilities

• Design, build, and maintain scalable data pipelines using Azure Data Factory and Databricks.
• Develop data processing and transformation solutions using Python and PySpark.
• Build and optimize data pipelines for batch and incremental processing.
• Develop ingestion frameworks to integrate data from databases, APIs, files, and cloud storage.
• Implement and manage ADF pipelines, datasets, and integration runtimes.
• Optimize Databricks notebooks, jobs, and Spark-based applications.
• Ensure data quality and governance through validation and monitoring.
• Collaborate with data architects, analysts and stakeholders to meet data requirements.

Skills

Analytical skills

Communication skills

Tools

Microsoft Azure

Azure Data Factory

Python

PySpark

Databricks

Azure Data Lake Storage

SQL

Git

Azure DevOps

Job description

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice AI & Analytics

Region USA

Country US

Working model Hybrid

Weekly Off US standard

State Georgia

Skills

Skill

Graduation/Equivalent Course

CERTIFICATION

Working Language

Job Description

Senior Data Engineer with 6+ years of experience

Mandatory Skills -

-designing, developing, and maintaining scalable data solutions.

- strong hands-on experience with Microsoft Azure, Azure Data Factory (ADF),

Role Summary

The Data Engineer will be responsible for building robust data pipelines, transforming and processing large datasets, implementing data integration solutions, and working closely with data architects, analysts, application teams, and business stakeholders to deliver reliable and high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure Data Factory and Databricks.
• Develop data processing and transformation solutions using Python and PySpark.
• Build and optimize data pipelines for batch and incremental data processing.
• Develop data ingestion frameworks to integrate data from databases, APIs, files, cloud storage, and other data sources.
• Develop and manage Azure Data Factory pipelines, datasets, linked services, triggers, parameters, and integration runtimes.
• Develop and optimize Databricks notebooks, jobs, workflows, and Spark-based data processing applications.
• Implement data transformations, cleansing, validation, aggregation, and enrichment processes.
• Work with Azure Data Lake Storage (ADLS) and other Azure data services.
• Design and implement scalable data lake and data warehouse solutions.
• Optimize PySpark jobs and Databricks workloads for performance and cost efficiency.
• Implement incremental data loading and change data capture (CDC) patterns where required.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Implement data quality checks and validation frameworks to ensure data accuracy and completeness.
• Work closely with data architects and technical leads to implement enterprise data architecture standards.
• Collaborate with business analysts and stakeholders to understand data requirements and translate them into technical solutions.
• Participate in code reviews and follow established coding, testing, documentation, and deployment standards.
• Implement CI/CD processes for data engineering solutions using Azure DevOps or similar tools.
• Maintain technical documentation for data pipelines, data flows, transformations, and integration processes.
• Support production deployments and resolve data-related incidents and issues.
• Mentor junior data engineers and contribute to improving data engineering best practices.

Required Qualifications

• 6+ years of professional experience in Data Engineering, ETL development, or related roles.
• Strong hands-on experience with Microsoft Azure data services.
• Strong experience with Azure Data Factory (ADF).
• Strong programming experience in Python.
• Hands-on experience with PySpark and Apache Spark.
• Strong hands-on experience with Databricks.
• Experience working with Azure Data Lake Storage (ADLS).
• Strong SQL skills and experience working with relational databases.
• Experience designing and developing complex ETL/ELT pipelines.
• Experience working with large-volume and complex datasets.
• Strong understanding of data warehousing and data lake concepts.
• Experience with batch and incremental data processing.
• Experience with Git and source control.
• Experience with CI/CD and deployment processes, preferably using Azure DevOps.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.