Databricks, PySpark, Python - Data Engineer

Infosys; Thiruvananthapuram; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

Infosys is seeking a Team Lead - Data Engineering with 7-9 years of IT experience to design and implement scalable data solutions using Databricks, PySpark, and Python.

You will lead a small team, drive technical delivery, and build high-performance data pipelines for enterprise analytics platforms, while mentoring team members and coordinating with stakeholders.

Qualifications

• 7–9 years of IT experience.
• Strong hands-on with Databricks and PySpark.
• Proficiency in Python programming and SQL.
• Experience building enterprise-scale data pipelines and data lake solutions.
• Strong understanding of Spark architecture and distributed computing concepts.
• Experience in performance tuning and optimization.
• Excellent stakeholder management and team coordination.

Responsibilities

• Lead the design, development, and deployment of data engineering solutions using Databricks and PySpark.
• Develop scalable ETL/ELT pipelines and data transformation frameworks.
• Write efficient Python and SQL code for data processing and analytics.
• Work closely with stakeholders to understand business requirements and translate them into technical solutions.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Ensure data quality, governance, and security standards are followed.
• Review code, mentor team members, and provide technical guidance.
• Support production deployments, incident troubleshooting, and root cause analysis.
• Participate in Agile ceremonies and drive sprint deliverables.

Skills

Team leadership

Stakeholder management

Agile ceremonies

Distributed computing

Education

MBA

MCA

ME

MSc

MTech

Bachelor of Engineering

BCA

BSc

BTech

Tools

Databricks

PySpark

Python

SQL

Job description

We are seeking a highly skilled Team Lead - Data Engineering with 7-9 years of experience in designing and developing scalable data solutions using Databricks, PySpark, and Python. The candidate will be responsible for leading a small team, driving technical delivery, and building high-performance data pipelines for enterprise analytics platforms.

Responsibilities

• Lead the design, development, and deployment of data engineering solutions using Databricks and PySpark.
• Develop scalable ETL/ELT pipelines and data transformation frameworks.
• Write efficient Python and SQL code for data processing and analytics.
• Work closely with stakeholders to understand business requirements and translate them into technical solutions.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Ensure data quality, governance, and security standards are followed.
• Review code, mentor team members, and provide technical guidance.
• Support production deployments, incident troubleshooting, and root cause analysis.
• Participate in Agile ceremonies and drive sprint deliverables.

Technical Requirements

• 7-9 years of overall IT experience.
• Strong hands-on expertise in Databricks and PySpark.
• Proficiency in Python programming and SQL.
• Experience building enterprise-scale data pipelines and data lake solutions.
• Strong understanding of distributed computing concepts and Spark architecture.
• Experience in performance tuning and optimization.
• Excellent stakeholder management and team coordination skills.

Education

• MBA
• MCA
• ME
• MSc
• MTech
• Bachelor of Engineering
• BCA
• BSc
• BTech

Preferred Skills

• Technology->Data Engineering->Databricks
• Technology->AI-Data science->PYTHON
• Technology->Big Data - Data Processing->PySpark

Service Line

Data & Analytics Unit