Python Data Engineer

Dexian India; Chennai District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A reputable IT consulting firm is seeking a Data Engineer in Chennai, Tamil Nadu. You will design and build optimized data pipelines, lead the implementation of algorithms, and collaborate with cross-functional teams. Required is expertise in Python and proven experience with big data technologies. This full-time role is suited for a mid-senior level candidate with a strong background in data processing and management.

Qualifications

• Minimum 8+ years of hands-on experience as a Python and Data Engineer.
• Experience working in distributed environments with global teams.
• Obsession for service observability, instrumentation, monitoring, and alerting.

Responsibilities

• Designing and building optimized data pipelines using cutting-edge technologies.
• Leading the implementation of algorithms and prototypes to transform raw data.
• Collaborating closely with Product and Business Managers to design technical solutions.

Skills

Python

Scala

Spark

SQL

Data Lifecycle Management

Agile methodologies

Monitoring and alerting

Data Lake

Delta Lake

Hive

Education

Bachelor’s/Master’s Degree

Tools

MySQL

Snowflake

Job description

Responsibilities

• Designing and building optimized data pipelines using cutting‑edge technologies in a cloud environment to drive analytical insights.
• Constructing infrastructure for efficient ETL processes from various sources and storage systems.
• Collaborating closely with Product Managers and Business Managers to design technical solutions aligned with business requirements.
• Leading the implementation of algorithms and prototypes to transform raw data into useful information.
• Architecting, designing, and maintaining database pipeline architectures, ensuring readiness for AI/ML transformations.
• Creating innovative data validation methods and data analysis tools.
• Ensuring compliance with data governance and security policies.
• Interpreting data trends and patterns to establish operational alerts.
• Developing analytical tools, programs, and reporting mechanisms.
• Conducting complex data analysis and presenting results effectively.
• Preparing data for prescriptive and predictive modeling.
• Continuously exploring opportunities to enhance data quality and reliability.
• Applying strong programming and problem‑solving skills to develop scalable solutions.
• Passion for testing strategy, problem‑solving, and continuous learning.
• Willingness to acquire new skills and knowledge.
• Possess a product/engineering mindset to drive impactful data solutions.
• Experience working in distributed environments with global teams.

Technical Skills and Experience Requirements

• Minimum 8+ years of hands‑on experience designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.
• High proficiency in Python, Scala and Spark for applied large‑scale data processing.
• Expertise with big data technologies, including Spark, Data Lake, Delta Lake, and Hive.
• Solid understanding of batch and streaming data processing techniques.
• Proficient knowledge of the Data Lifecycle Management process, including data collection, access, use, storage, transfer, and deletion.
• Expert‑level ability to write complex, optimized SQL queries across extensive data volumes.
• Experience with RDBMS and OLAP databases like MySQL, Snowflake.
• Familiarity with Agile methodologies.
• Obsession for service observability, instrumentation, monitoring, and alerting.
• Knowledge or experience in architectural best practices for building data lakes.

Qualifications

• Bachelor’s/Master’s Degree.
• Relevant Experience: 8+ Years (as Python and Data Engineer)
• Overall IT Experience: 8 to 12 Years

Seniority Level

• Mid‑Senior level

Employment Type

• Full-time

Job Function

• IT Services and IT Consulting

Get notified about new Data Engineer jobs in Chennai, Tamil Nadu, India.