Data Engineer

LatentView Analytics Ltd.; Pune District; On-site; INR 2,400,000 - 4,200,000; Full time

Job summary

LatentView Analytics Ltd. is seeking an experienced Data Engineer to design, build, and deploy enterprise-grade data solutions on Databricks with Azure. The role focuses on scalable pipelines, metadata-driven ingestion, and robust data quality, collaborating with global teams and mentoring junior engineers.

You will lead technical discussions, optimize Spark workloads, and implement Unity Catalog for secure data access while contributing to end-to-end delivery and governance across projects.

Qualifications

• 5–7 years of experience in Data Engineering with hands-on Databricks on Azure.
• Proficient in building metadata-driven ingestion and data quality frameworks using PySpark.
• Strong understanding of Lakehouse architecture and data platform concepts.
• Experience with Unity Catalog and fine-grained data access controls.
• Ability to lead technical discussions and mentor junior engineers.
• Experience designing scalable data pipelines and data governance.

Responsibilities

• Data Engineering & Architecture: Design and develop scalable data solutions aligned with technical architecture, integration standards, and established engineering practices.
• Pipeline Development: Build and optimize enterprise-grade data pipelines using Databricks, PySpark, Delta Lake, Delta Live Tables, Auto Loader, Databricks Workflows, and/or Apache Airflow.
• Metadata & Data Quality: Develop metadata-driven ingestion frameworks and robust data quality (DQ) frameworks using PySpark to ensure reliable and governed data processing.
• Lakehouse & Data Platforms: Design and implement modern Lakehouse architectures using Apache Spark, Delta Lake, Azure Data Lake Storage, and related big data technologies.
• Technical Leadership: Lead technical discussions, clarify requirements, resolve ambiguities, conduct design and code reviews, and provide technical guidance and mentorship to junior team members.
• Performance & Optimization: Perform performance tuning and optimization across Databricks and Apache Spark workloads, pipelines, queries, and data processing jobs.
• Security & Governance: Implement Databricks Unity Catalog and fine-grained access controls to support enterprise data governance and security requirements.
• Client & Cross-Functional Collaboration: Work closely with business analysts, functional teams, onsite clients, architects, and global delivery teams to translate requirements into effective technical solutions.
• Automation & Delivery: Develop reusable templates, frameworks, and scripts to automate development and operational activities. Support project estimation, planning, execution, tracking, and continuous improvement.

Tools

Databricks

Azure

Python

SQL

PySpark

Apache Spark

Delta Lake

Lakehouse

Azure Data Lake Storage Gen2

Azure Data Factory

Azure Synapse

Azure Key Vault

Cosmos DB

Unity Catalog

Delta Live Tables

Auto Loader

Databricks Workflows

Apache Airflow

Job description

Design, develop, and deploy enterprise-scale data engineering solutions using Databricks and Azure. The role focuses on building robust and scalable data pipelines, metadata-driven ingestion frameworks, data quality solutions, and Lakehouse architectures. The ideal candidate will collaborate with global cross-functional teams, provide technical leadership, mentor junior engineers, and drive high-quality delivery across the full project lifecycle.

Roles and Responsibilities

• Data Engineering & Architecture: Design and develop scalable data solutions aligned with technical architecture, integration standards, and established engineering practices.
• Pipeline Development: Build and optimize enterprise-grade data pipelines using Databricks, PySpark, Delta Lake, Delta Live Tables, Auto Loader, Databricks Workflows, and/or Apache Airflow.
• Metadata & Data Quality: Develop metadata-driven ingestion frameworks and robust data quality (DQ) frameworks using PySpark to ensure reliable and governed data processing.
• Lakehouse & Data Platforms: Design and implement modern Lakehouse architectures using Apache Spark, Delta Lake, Azure Data Lake Storage, and related big data technologies.
• Technical Leadership: Lead technical discussions, clarify requirements, resolve ambiguities, conduct design and code reviews, and provide technical guidance and mentorship to junior team members.
• Performance & Optimization: Perform performance tuning and optimization across Databricks and Apache Spark workloads, pipelines, queries, and data processing jobs.
• Security & Governance: Implement Databricks Unity Catalog and fine-grained access controls to support enterprise data governance and security requirements.
• Client & Cross-Functional Collaboration: Work closely with business analysts, functional teams, onsite clients, architects, and global delivery teams to translate requirements into effective technical solutions.
• Automation & Delivery: Develop reusable templates, frameworks, and scripts to automate development and operational activities. Support project estimation, planning, execution, tracking, and continuous improvement.

Required Skills

• Databricks, Azure, Python, SQL, PySpark, Apache Spark, Delta Lake, Lakehouse Architecture, Azure Data Lake Storage Gen2 (ADLS Gen2), Azure Data Factory, Azure Synapse, Azure Key Vault, Cosmos DB, Unity Catalog, Delta Live Tables, Auto Loader, Databricks Workflows, and Apache Airflow.
• 5–7 years of experience in Data Engineering with significant hands-on expertise in Databricks on Azure.
• Strong experience building metadata-driven ingestion and data quality frameworks using PySpark.
• Strong understanding of Lakehouse architecture, data lakes, data warehouses, data marts, 3NF, dimensional modeling, and modern data platforms.
• Hands-on experience with Databricks/Spark performance tuning, data pipeline optimization, and scalable data processing.
• Experience implementing Unity Catalog and fine-grained data access controls.
• Strong problem-solving, analytical, communication, stakeholder management, and team collaboration skills.

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status.