Databricks+Sql

LTM; Chennai District; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

LTM in Chennai, India seeks a Data Engineer to design and implement robust ETL/ELT pipelines using PySpark, Spark SQL, and Python. The role focuses on performance optimization of Databricks clusters, Delta Lake data lakehouse management, and scalable data modeling.

You will also integrate Databricks with ADLS and orchestrate workflows with Airflow and Azure Data Factory. The candidate should have 3-5 years of data engineering experience, strong skills in PySpark, Python, SQL, and Spark

Qualifications

• 3-5 years of data engineering experience.
• Proficiency in PySpark, Python and SQL.
• Hands-on Databricks experience and Delta Lake knowledge.

Responsibilities

• Design and implement robust ETL/ELT pipelines using PySpark and Spark SQL.
• Optimize Databricks clusters and code for large-scale data processing.
• Manage Delta Lake architecture for high-quality data storage and performance.
• Create scalable data models and analytical structures.
• Integrate Databricks with data lakes (ADLS) and orchestrate workflows (Airflow, Azure Data Factory).
• Monitor job performance and ensure data quality.

Skills

Experience 3-5 years in data eng

Data engineering

Analytical mindset

Tools

PySpark

Python

SQL

Spark optimization

Azure Databricks

AWS Databricks

GCP Databricks

Delta Lake

Unity Catalog

Databricks SQL

CI/CD tools

Git

Airflow

Azure Data Factory

Job description

Role Description

Core Responsibilities

• Data Pipeline Development Design and implement robust ETLELT pipelines using PySpark Spark SQL and Python
• Performance Optimization Optimize Databricks clusters and code for efficient largescale data processing
• Data Lakehouse Management Implement Delta Lake architecture for highquality data storage management and performance
• Data Modeling Architecture Create scalable data models and analytical structures
• System Integration Automation Integrate Databricks with data lakes eg ADLS and orchestrate workflows eg Azure Data Factory Airflow
• Monitoring Support Monitor job performance troubleshoot failures and ensure data quality

Key Qualifications Skills

• Experience 35 years in data engineering or related roles
• Technical Proficiency Strong skills in PySpark Python SQL and Spark optimization
• Platforms Handson experience with Azure Databricks preferred AWS Databricks or GCP Databricks
• Tools Technologies Knowledge of Delta Lake Unity Catalog Databricks SQL and CICD tools Git
• Databases Experience with Data Warehousing and Big Data technologies