Python Expert with AWS Data Engineering

Contus LLC; Chennai District; On-site; INR 1,000,000 - 1,500,000; Full time

Job summary

A technology company is seeking a Python Data Engineering expert based in Chennai, India. The ideal candidate will have over 8 years of experience in Data Engineering and strong hands-on expertise in Python development. Responsibilities include developing scalable data solutions, migrating data to NoSQL databases, and utilizing AWS services such as EMR, S3, and Lambda. This role also requires the ability to collaborate effectively within a team and implement cost-optimized cloud strategies.

Qualifications

• 8+ years of overall experience in Data Engineering with expertise in Python development.
• Strong proficiency in RDBMS and SQL for managing relational databases.
• Experience in migrating data sets from RDBMS to NoSQL databases.

Responsibilities

• Develop scalable, high-performance data engineering solutions using Python.
• Lead efforts to migrate data from RDBMS to NoSQL databases.
• Design and build data lakes, managing configurations for Delta tables.
• Implement cost optimization strategies in cloud environments like AWS.
• Utilize Apache Airflow to manage and monitor workflows efficiently.
• Work with AWS services to build and manage data solutions.

Skills

Python Expertise

RDBMS Expertise

NoSQL

Data Lakes & Delta Tables

AWS Services

Apache Airflow

Team Collaboration

Job description

We are seeking a Python Data Engineering expert with 8+ years of overall experience in Data Engineering and at least 8+ years of hands‑on expertise in Python development. The ideal candidate should have strong proficiency in Python, RDBMS, and a solid understanding of AWS services.

Key Responsibilities:

• Python Expertise: Leverage 8+ years of hands‑on experience with Python for developing scalable, high‑performance data engineering solutions.
• RDBMS to NoSQL Conversion: Lead the effort in migrating large data sets from RDBMS to NoSQL databases, ensuring seamless and efficient data storage and retrieval.
• Data Lake & Delta Tables: Design and build data lakes and manage configurations on Delta tables to optimize data processing and storage workflows.
• Cost Optimization: Implement computing strategies that minimize costs while maximizing data processing efficiency within cloud environments like AWS.
• Holistic Framework Development: Analyze current environments and use cases to develop holistic data engineering frameworks that address both short‑term and long‑term business needs.
• Airflow Framework: Utilize Apache Airflow to manage, schedule, and monitor workflows, ensuring data pipelines are running efficiently.
• AWS Service Integration: Work extensively with AWS services such as EMR, S3, Lambda, Step Functions, and Aurora (RDS) to build, manage, and optimize data solutions on the cloud.

Required Skills:

• 8+ Years of Data Engineering Experience: A strong background in Data Engineering, with 8+ years of Python development experience.
• RDBMS Expertise: Proficiency in RDBMS and SQL for managing large‑scale relational databases.
• NoSQL: Familiarity with converting data from RDBMS to NoSQL databases to enable flexible, scalable data storage solutions.
• Data Lakes & Delta Tables: Experience in building data lakes and working with Delta table configurations for optimized data storage and processing.
• AWS Services: Hands‑on experience with AWS technologies such as EMR, S3, Lambda, Step Functions, and Aurora (RDS) for cloud‑based data solutions.
• Apache Airflow: Strong understanding of Apache Airflow to automate, schedule, and manage data workflows.

Soft Skills:

• Team Collaboration: Demonstrated ability to work as part of a team, with a positive attitude and commitment to collaborative problem‑solving.