Senior Software Engineer- Data

NewVision Software; South Carolina; Hybrid; USD 120,000 - 160,000; Full time

Job summary

New Vision Softcom & Consultancy Pvt. Ltd in the USA (Georgia) seeks a Senior Data Engineer with 6+ years of experience for a hybrid role. You will design and implement robust data pipelines, ETL/ELT processes, and data integrations across cloud data services.

You will develop with Azure Data Factory, Databricks, PySpark, and Python, ensuring scalable data lake and warehousing solutions while collaborating with analysts and stakeholders.

Qualifications

• 6+ years of professional experience in Data Engineering, ETL development, or related roles.
• Strong hands-on experience with Microsoft Azure data services.
• Strong experience with Azure Data Factory (ADF).
• Strong programming experience in Python.
• Hands-on experience with PySpark and Apache Spark.
• Strong hands-on experience with Databricks.
• Experience working with Azure Data Lake Storage (ADLS).
• Strong SQL skills and experience working with relational databases.
• Experience designing and developing complex ETL/ELT pipelines.
• Experience working with large-volume and complex datasets.
• Strong understanding of data warehousing and data lake concepts.
• Experience with batch and incremental data processing.
• Experience with Git and source control.
• Experience with CI/CD and deployment processes, preferably using Azure DevOps.
• Strong analytical and problem-solving skills.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure Data Factory and Databricks.
• Develop data processing and transformation solutions using Python and PySpark.
• Build and optimize data pipelines for batch and incremental data processing.
• Develop data ingestion frameworks to integrate data from databases, APIs, files, cloud storage, and other data sources.
• Develop and manage Azure Data Factory pipelines, datasets, linked services, triggers, parameters, and integration runtimes.
• Develop and optimize Databricks notebooks, jobs, workflows, and Spark-based data processing applications.
• Implement data transformations, cleansing, validation, aggregation, and enrichment processes.
• Work with Azure Data Lake Storage (ADLS) and other Azure data services.
• Design and implement scalable data lake and data warehouse solutions.
• Optimize PySpark jobs and Databricks workloads for performance and cost efficiency.
• Implement incremental data loading and change data capture (CDC) patterns where required.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Implement data quality checks and validation frameworks to ensure data accuracy and completeness.
• Work closely with data architects and technical leads to implement enterprise data architecture standards.
• Collaborate with business analysts and stakeholders to understand data requirements and translate them into technical solutions.
• Participate in code reviews and follow established coding, testing, documentation, and deployment standards.
• Implement CI/CD processes for data engineering solutions using Azure DevOps or similar tools.
• Maintain technical documentation for data pipelines, data flows, transformations, and integration processes.
• Support production deployments and resolve data-related incidents and issues.
• Mentor junior data engineers and contribute to improving data engineering best practices.

Skills

Analytical thinking

Communication skills

Team collaboration

Problem-solving

Tools

Azure Data Factory (ADF)

Databricks

PySpark

Apache Spark

Azure Data Lake Storage (ADLS)

Git

Azure DevOps

SQL

Job description

Company Name New Vision Softcom & Consultancy Pvt. Ltd

Function Business Units (BU)

Department/Practice AI & Analytics

Region USA

Country US

Working model Hybrid

Weekly Off US standard

State Georgia

Skills

Skill

Graduation/Equivalent Course

CERTIFICATION

Working Language

Job Description

Senior Data Engineer with 6+ years of experience

Mandatory Skills -

-designing, developing, and maintaining scalable data solutions.

- strong hands-on experience with Microsoft Azure, Azure Data Factory (ADF),

Role Summary

The Data Engineer will be responsible for building robust data pipelines, transforming and processing large datasets, implementing data integration solutions, and working closely with data architects, analysts, application teams, and business stakeholders to deliver reliable and high-quality data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL/ELT processes using Azure Data Factory and Databricks.
• Develop data processing and transformation solutions using Python and PySpark.
• Build and optimize data pipelines for batch and incremental data processing.
• Develop data ingestion frameworks to integrate data from databases, APIs, files, cloud storage, and other data sources.
• Develop and manage Azure Data Factory pipelines, datasets, linked services, triggers, parameters, and integration runtimes.
• Develop and optimize Databricks notebooks, jobs, workflows, and Spark-based data processing applications.
• Implement data transformations, cleansing, validation, aggregation, and enrichment processes.
• Work with Azure Data Lake Storage (ADLS) and other Azure data services.
• Design and implement scalable data lake and data warehouse solutions.
• Optimize PySpark jobs and Databricks workloads for performance and cost efficiency.
• Implement incremental data loading and change data capture (CDC) patterns where required.
• Monitor data pipelines and troubleshoot failures, performance issues, and data quality problems.
• Implement data quality checks and validation frameworks to ensure data accuracy and completeness.
• Work closely with data architects and technical leads to implement enterprise data architecture standards.
• Collaborate with business analysts and stakeholders to understand data requirements and translate them into technical solutions.
• Participate in code reviews and follow established coding, testing, documentation, and deployment standards.
• Implement CI/CD processes for data engineering solutions using Azure DevOps or similar tools.
• Maintain technical documentation for data pipelines, data flows, transformations, and integration processes.
• Support production deployments and resolve data-related incidents and issues.
• Mentor junior data engineers and contribute to improving data engineering best practices.

Required Qualifications

• 6+ years of professional experience in Data Engineering, ETL development, or related roles.
• Strong hands-on experience with Microsoft Azure data services.
• Strong experience with Azure Data Factory (ADF).
• Strong programming experience in Python.
• Hands-on experience with PySpark and Apache Spark.
• Strong hands-on experience with Databricks.
• Experience working with Azure Data Lake Storage (ADLS).
• Strong SQL skills and experience working with relational databases.
• Experience designing and developing complex ETL/ELT pipelines.
• Experience working with large-volume and complex datasets.
• Strong understanding of data warehousing and data lake concepts.
• Experience with batch and incremental data processing.
• Experience with Git and source control.
• Experience with CI/CD and deployment processes, preferably using Azure DevOps.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration skills.