Senior Data Engineer

LatentView; Hyderabad; Hybrid; INR 1,800,000 - 2,400,000; Full time

Job summary

LatentView is seeking a strong Databricks Engineer with hands-on experience designing scalable data engineering solutions using Databricks, Apache Spark, and cloud platforms such as Azure or AWS.

The candidate should excel in PySpark, SQL, data pipelines, Delta Lake, ETL/ELT, and cloud-based data platforms, with the ability to work independently on complex data problems.

Qualifications

• Hands-on experience with Databricks and Spark.
• Strong PySpark and SQL skills for data pipelines and transformations.
• Experience with Delta Lake, partitioning, and data lake design.
• Proficiency with cloud platforms (Azure or AWS) for end-to-end data solutions.
• Knowledge of ETL/ELT patterns and data integration.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Spark.
• Develop high‑performance ETL/ELT workflows with PySpark and SQL.
• Build and optimize Delta Lake tables and data processing frameworks.
• Collaborate with cloud services on Azure or AWS for end‑to‑end solutions.
• Develop ingestion pipelines from databases, APIs, files, and streaming sources.
• Optimize Spark jobs and queries for performance and cost.
• Implement data quality, monitoring, and error handling.

Skills

PySpark

SQL

Data pipelines

ETL/ELT

Cloud platforms

Spark

Tools

Databricks

Apache Spark

Delta Lake

Azure

AWS

Python

Job description

Job Summary

We are looking for a strong Databricks Engineer with 47 years of hands‑on experience in designing and building scalable data engineering solutions using Databricks, Apache Spark, and cloud platforms such as Azure or AWS.

The ideal candidate should have strong expertise in PySpark, SQL, data pipelines, Delta Lake, ETL/ELT, and cloud‑based data platforms, with the ability to work independently on complex data engineering problems.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Develop high‑performance ETL/ELT workflows using PySpark and SQL.
• Build and optimize Delta Lake tables and implement reliable data processing frameworks.
• Work with cloud services on Microsoft Azure or AWS to build end‑to‑end data solutions.
• Develop data ingestion pipelines from various sources including databases, APIs, files, and streaming systems.
• Optimize Spark jobs, SQL queries, and Databricks workloads for performance and cost.
• Implement data quality, validation, monitoring, and error‑handling mechanisms.
• Work with large and complex datasets and ensure data accuracy, consistency, and availability.
• Implement incremental data processing and CDC‑based data pipelines where required.
• Collaborate with data architects, analysts, data scientists, and application teams.
• Follow engineering best practices including version control, CI/CD, testing, documentation, and code reviews.
• Troubleshoot production data pipelines and resolve performance and data‑quality issues.

Required Skills

Databricks & Spark

• Strong hands‑on experience with Databricks.
• Strong knowledge of Apache Spark and PySpark.
• Experience with Delta Lake, Delta tables, optimization, and partitioning.
• Understanding of Spark performance tuning, joins, caching, partitioning, and file optimization.
• Experience developing production‑grade notebooks, workflows/jobs, and data pipelines.

Programming & Data

• Strong Python/PySpark programming skills.
• Strong SQL skills, including complex queries, joins, CTEs, window functions, and query optimization.
• Strong understanding of data warehousing and dimensional modeling concepts.
• Experience with batch and, preferably, streaming data processing.
• Understanding of ETL/ELT frameworks and data integration patterns.