Sr. Data Engineer

NexTurn; Hyderabad; On-site; INR 1,200,000 - 1,800,000; Full time

Job summary

NexTurn is seeking a data engineer to design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python for enterprise data-processing needs. The role emphasizes building Lakehouse solutions with Delta Lake, handling diverse data sources, and optimizing Spark workloads.

The candidate will manage workflow orchestration, ensure data quality, and collaborate with architects and stakeholders to deliver secure, Azure-based data solutions at scale.

Qualifications

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python to support enterprise data-processing requirements.
• Build and optimize Lakehouse solutions using Databricks and Delta Lake, ensuring scalability, performance, reliability, and maintainability.
• Ingest and process data from diverse enterprise sources, including structured, semi-structured, and unstructured data, while handling different formats and data complexities.
• Develop and manage workflow orchestration for data pipelines, including scheduling, dependency management, monitoring, failure handling, and operational support.
• Implement data-quality and validation frameworks to perform data profiling, reconciliation, completeness checks, accuracy validation, and anomaly detection.
• Optimize Spark/PySpark and SQL workloads by applying appropriate partitioning, caching, query optimization, Delta Lake optimization, and other performance-tuning techniques.
• Collaborate with data engineers, architects, analysts, and business stakeholders to understand data requirements and deliver secure, reliable, and scalable Azure-based data solutions.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python to support enterprise data-processing requirements.
• Build and optimize Lakehouse solutions using Databricks and Delta Lake, ensuring scalability, performance, reliability, and maintainability.
• Ingest and process data from diverse enterprise sources, including structured, semi-structured, and unstructured data, while handling different formats and data complexities.
• Develop and manage workflow orchestration for data pipelines, including scheduling, dependency management, monitoring, failure handling, and operational support.
• Implement data-quality and validation frameworks to perform data profiling, reconciliation, completeness checks, accuracy validation, and anomaly detection.
• Optimize Spark/PySpark and SQL workloads by applying appropriate partitioning, caching, query optimization, Delta Lake optimization, and other performance-tuning techniques.
• Collaborate with data engineers, architects, analysts, and business stakeholders to understand data requirements and deliver secure, reliable, and scalable Azure-based data solutions.

Skills

Databricks

Apache Spark

PySpark

SQL

Python

Delta Lake

Lakehouse architecture

Data pipelines

Data quality

Azure data services

EDI knowledge

Education

Bachelor's degree in computer science/IT/Engineering

Tools

Databricks

Delta Lake

Azure Data Factory

Airflow

Spark SQL

Job description

• 2+ years of hands-on experience with Databricks in designing, developing, and maintaining scalable data engineering solutions.
• Strong expertise in Apache Spark / PySpark, SQL, and Python for large-scale data processing and transformation.
• Strong understanding of Delta Lake and Lakehouse architecture, including data modeling, optimization, and reliable data storage patterns.
• Experience with workflow orchestration, including designing and managing data pipelines, job scheduling, dependencies, and monitoring.
• Strong knowledge of data quality, validation, reconciliation, and error-handling practices to ensure data accuracy and reliability.
• Experience working with heterogeneous enterprise data sources, including structured, semi-structured, and unstructured data.
• Azure data-platform experience is strongly preferred, particularly with Azure-based data engineering and analytics services.
• EDI knowledge is not mandatory and will be considered an added advantage.

Qualifications

Bachelor's degree in computer science, Information Technology, Engineering, or equivalent.

Job Description

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python to support enterprise data-processing requirements.
• Build and optimize Lakehouse solutions using Databricks and Delta Lake, ensuring scalability, performance, reliability, and maintainability.
• Ingest and process data from diverse enterprise sources, including structured, semi-structured, and unstructured data, while handling different formats and data complexities.
• Develop and manage workflow orchestration for data pipelines, including scheduling, dependency management, monitoring, failure handling, and operational support.
• Implement data-quality and validation frameworks to perform data profiling, reconciliation, completeness checks, accuracy validation, and anomaly detection.
• Optimize Spark/PySpark and SQL workloads by applying appropriate partitioning, caching, query optimization, Delta Lake optimization, and other performance-tuning techniques.
• Collaborate with data engineers, architects, analysts, and business stakeholders to understand data requirements and deliver secure, reliable, and scalable Azure-based data solutions.