Data Engineer

Tekskills; Chennai District; On-site; INR 2,000,000 - 4,000,000; Full time

Job summary

Tekskills in Chennai seeks a highly experienced Data Engineer with 10+ years of expertise in data engineering, Databricks, Azure, SQL, PySpark and Python. The role centers on building scalable data pipelines, applying CICD and DevOps practices, and optimizing data transformations.

You will collaborate with business and functional teams, govern data quality, implement Unity Catalog, and contribute to platform architecture and performance tuning for large-scale batch and streaming workloads.

Qualifications

• Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).
• Deep understanding of Unity Catalog for governance and lineage tracking.
• Proficiency in SQL for large-scale data manipulation and analysis.
• Solid understanding of CI/CD, infrastructure automation, and DevOps.
• Familiarity with cloud platforms (Azure) and data services.
• Strong problem-solving, debugging, and system design skills.
• Excellent communication and collaboration abilities in cross-functional teams.
• Hands-on experience with Delta Lake management, schema evolution, and ACID-compliant pipelines.
• Knowledge of performance tuning techniques in Spark, including job optimization, caching, and partitioning.

Responsibilities

• Apply CICD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).
• Optimize query performance and data transformations using advanced SQL.
• Implement and uphold data governance, quality, and access control policies.
• Support production data pipelines and respond to issues and performance bottlenecks.
• Contribute to architectural decisions around data strategy and platform scalability.
• Develop and maintain Databricks notebooks and jobs for large-scale batch and streaming data processing.
• Write modular, production-grade PySpark and Python code, including reusable functions and libraries for data transformation.
• Implement streaming data ingestion and Structured Streaming in Databricks for near real-time data solutions.
• Apply performance tuning techniques in Spark, including job optimization, caching, and partitioning strategies.
• Utilize data quality frameworks and testing practices.
• Manage data governance, access controls, and lineage tracking using Unity Catalog.

Skills

Databricks

Delta Lake

Spark (PySpark)

SQL

Python

Azure

Unity Catalog

CI/CD

DevOps

GitHub Actions

Jenkins

Terraform

Tools

Databricks

GitHub Actions

Jenkins

Terraform

Delta Lake

Unity Catalog

Job description

• We are seeking a highly experienced Data Engineer with over 10+ years of expertise in Data Engineering, Databricks, Azure, SQL, PySpark, and Python.
• The ideal candidate will collaborate with Business and functional teams and create the data pipelines, apply CICD and DevOps practices, and optimize query performance and data transformations.
• This role requires a deep understanding of data governance, quality, and access control policies, as well as the ability to support production data pipelines and contribute to architectural decisions.

Key Responsibilities

• Apply CICD and DevOps practices to automate data workflows and deployments (e.g., with GitHub Actions, Jenkins, Terraform).
• Optimize query performance and data transformations using advanced SQL.
• Implement and uphold data governance, quality, and access control policies.
• Support production data pipelines and respond to issues and performance bottlenecks.
• Contribute to architectural decisions around data strategy and platform scalability.
• Develop and maintain Databricks notebooks and jobs for large-scale batch and streaming data processing.
• Write modular, production-grade PySpark and Python code, including reusable functions and libraries for data transformation.
• Implement streaming data ingestion and Structured Streaming in Databricks for near real-time data solutions.
• Apply performance tuning techniques in Spark, including job optimization, caching, and partitioning strategies.
• Utilize data quality frameworks and testing practices.
• Manage data governance, access controls, and lineage tracking using Unity Catalog.

Requirements

• Proven expertise in Databricks, Delta Lake, and Apache Spark (PySpark preferred).
• Deep understanding of Unity Catalog for fine-grained data governance and lineage tracking.
• Proficiency in SQL for large-scale data manipulation and analysis.
• Solid understanding of CICD, infrastructure automation, and DevOps principles.
• Familiarity with cloud platforms (Azure) and data services.
• Strong problem-solving, debugging, and system design skills.
• Excellent communication and collaboration abilities in cross-functional teams.
• Strong hands‑on experience with Delta Lake, including table management, schema evolution, and implementing ACID‑compliant pipelines.
• Knowledge of performance tuning techniques in Spark, including job optimization, caching, and partitioning strategies.

Basic understanding of Unity Catalog for managing data governance, access controls, and lineage tracking from a developers perspective. le & responsibilities

Preferred candidate profile