Phyton with Spark Developer

Infotel India; Chennai District; On-site; INR 800,000 - 1,200,000; Full time

Job summary

Infotel India is seeking a skilled Data Engineer in Chennai to design and develop robust data pipelines using Python, PySpark, and SQL. Collaborating closely with data architects, you will translate business requirements into technical specifications.

The ideal candidate will have a Bachelor’s degree, extensive experience in data engineering, and familiarity with Agile practices. The role emphasizes continuous learning and innovation in data transformation and analytics.

Join us for a dynamic opportunity to enhance data processes within a collaborative team.

Qualifications

• Hands-on experience with Python, PySpark, and SQL is required.
• Knowledge of front-end technologies preferably Flask is good to have.
• Experience in Agile and DevOps environments is preferred.

Responsibilities

• Design and develop ingestion, transformation, and enrichment pipelines.
• Collaborate with data architects and business analysts for requirements.
• Maintain CI/CD pipelines for automated testing and deployment.

Skills

Python

PySpark

SQL

ETL

Education

Bachelor’s degree or equivalent

Tools

Git

Jenkins

Docker

Job description

Responsibilities

Direct Responsibilities

• Design & develop robust ingestion, transformation, and enrichment pipelines with Python, PySpark, and SQL.
• Write and optimize complex SQL queries, analytical UDFs, and window functions for data aggregation and reporting.
• Collaborate with CEFS data architects, data scientists, and business analysts to translate functional requirements into technical specifications.
• Unit‑test, integrate‑test, and review code.
• Maintain CI/CD pipelines (Git, Jenkins, Docker) for automated build, test, and deployment of jobs.
• Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
• Document data lineage, pipeline design, and operational run‑books in Confluence/SharePoint.
• Keep up to date with latest technologies and trends and provide input, expertise and recommendations.

Contributing Responsibilities

• Contribute towards innovation (e.g. AI/ML); suggest new technical practices for efficiency improvement.
• Participate in Agile ceremonies (sprint planning, daily standups, retrospectives) and help groom the backlogs.
• Mentor junior engineers and champion best practices in Python coding, Spark optimization, and data‑engineering patterns.
• Evaluate emerging technologies and deliver proof‑of‑concepts for CEFS.

Technical & Behavioral Competencies

• Resourceful to quickly understand complexities involved and provide the way forward.
• Good experience in technical analysis of n-tier applications with multiple integrations using object-oriented, APIs & Microservices approaches.
• Strong knowledge about design patterns and development principles.
• Inclination and prior experience of working across SQL, Python and ETL.
• Hands‑on experience with Python (NumPy, pandas, Python Frameworks, Restful APIs, MS‑SQL or Oracle).
• PySpark – DataFrames, Spark SQL, Structured Streaming, performance tuning (partitioning, caching, broadcast joins).
• Advanced SQL – complex queries, stored procedures, query optimization.
• Good knowledge and experience to use Python packages such as Pandas, NumPy, etc. for cleaning up data, data wrangling, analysis of data and visualization of data.
• Good experience in development and maintenance of code/scripts in both functional and technical specifications of all applications components, bug fixing and production support.
• Good knowledge on Linux/Unix environment (basic commands, shell scripting, etc.), testing phases, documentation and new framework.
• Some experience with build tools like Maven & DevOps tools like Bitbucket, Jenkins.
• Knowledge of Agile, Scrum, DevOps.
• Development experience in Data Engineering environment.
• Ability & willingness to learn & work on diverse technologies (languages, frameworks, and tools).
• Self‑motivated, good interpersonal skills and inclination to constantly upgrade new technologies and frameworks.
• Good communication and coordination skills.

Specific Qualifications

Good to have knowledge of front‑end technologies preferably Flask.

Skills Referential (Required knowledge, skills and abilities)

Technical Skills

• Python
• Pyspark
• SQL
• ETL

Behavioral Skills

• Ability to synthesize / simplify
• Ability to collaborate / Teamwork
• Attention to detail / rigor
• Ability to deliver / Results driven

Education Level

Bachelor’s degree or equivalent

Location

Chennai