Lead Data Engineer

Flexi Careers India; Chennai District; On-site; INR 2,800,000 - 5,200,000; Full time

Job summary

Flexi Careers India is seeking a Senior Data Engineer to design, implement, and automate data infrastructure while optimizing pipelines for reliability and performance. You will build tools to monitor ETLs and ensure scalable, efficient data processing across systems.

The role requires 6+ years in data engineering, strong SQL and Python skills, and hands-on experience with distributed processing frameworks and cloud platforms (Azure/AWS/GCP). Databricks knowledge is a plus.

Qualifications

• 6+ years of experience in data engineering and building scalable data pipelines.
• Strong SQL skills and ability to write optimized queries for large-scale workloads.
• Experience with distributed data processing frameworks such as Apache Spark for batch/streaming data.
• Experience with cloud data platforms (Azure/AWS/GCP).

Responsibilities

• Design, implementation, and improvement of processes and automation of data infrastructure.
• Tuning of data pipelines for reliability and performance.
• Building tools and scripts to develop, monitor, and troubleshoot ETLs.
• Perform scalability, latency, and availability tests regularly.
• Code reviews and QA of data imported by various processes.
• Investigate, analyze, correct and document reported data defects.
• Create and maintain technical specification documentation.

Skills

SQL

Python

Apache Spark

Data pipelines

ETL/ELT

Databricks

Cloud platforms

Data modeling

Performance tuning

QA/data defects

Education

B.Tech/BS in CS/CE/Statistics or Engineering

Tools

Databricks

Job description

Role & responsibilities

• Design, implementation, and improvement of processes & automation of Data infrastructure
• Tuning of Data pipelines for reliability & performance
• Building tools and scripts to develop, monitor, and troubleshoot ETLs
• Perform scalability, latency, and availability tests on a regular basis.
• Perform code reviews and QA data imported by various processes.
• Investigate, analyze, correct and document reported data defects.
• Create and maintain technical specification documentation.

Preferred candidate profile

• B.Tech/BS degree in Computer Science, Computer Engineering, Statistics, or other Engineering disciplines
• 6+ years of experience in data engineering, building scalable and reliable data pipelines in production environments.
• Strong experience with any cloud data platforms such as Azure/ AWS/ GCP.
• Hands-on expertise with distributed data processing frameworks such as Apache Spark for large-scale batch and/or streaming data processing.
• Expert-level SQL skills with the ability to write optimized queries, perform complex transformations, and tune queries for large-scale analytical workloads.
• Proficiency in Python for data engineering tasks including pipeline development, automation, data transformations, and integration with APIs or cloud services.
• Strong knowledge of data modeling techniques, including dimensional modeling, star/snowflake schemas, and designing data models optimized for analytics and reporting.
• Experience designing scalable ETL/ELT architectures, ensuring high data quality, reliability, and performance for large-scale data platforms.

Strong experience or knowledge of Databricks will be a plus.