Big Data

Affine Analytics; India; À distance; INR 900 000 - 1 700 000; Plein temps

Résumé du poste

Affine Analytics is seeking a data engineering professional with 3-5 years of hands-on experience in Scala, Spark, SQL, and AWS. The role focuses on building scalable data pipelines, optimizing data quality, and supporting both batch and streaming processing.

You will work with Kafka/Kinesis and Hive, contribute to ETL/ELT design, and mentor junior engineers while designing robust data solutions for cloud-first platforms.

Qualifications

• Strong programming and data engineering capabilities with hands-on experience in Spark and SQL.
• Proficient in Cloud data platforms (AWS) and streaming services (Kafka/Kinesis).
• Experience designing ETL/ELT pipelines, data modeling, and performance tuning.

Responsabilités

• Design, develop, and optimize data pipelines for batch and streaming workloads.
• Migrate on-prem data warehouses to cloud platforms (AWS preferred).
• Mentor new team members and contribute to solution design.

Connaissances

Scala

Spark

SQL

AWS

Kafka

Hive

ETL/ELT design

CI/CD

Data Modeling

Description du poste

Roles & Responsibilities:

1. Strong SQL Skills: Candidate should be proficient in writing and executing complex SQL queries (no PL/SQL) and be able to handle real-world scenarios confidently.
2. Hands-on Functional Programming (Scala): Strong hands-on experience in Scala is essential.
3. Solid Data Engineering Fundamentals: Emphasis is on strong foundational understanding rather than tool-specific expertise.
4. Experience in Data Quality & Optimization: Candidate should have a sound understanding of data quality practices, performance optimization techniques, and solution design in both event-driven and batch processing contextsincluding data modeling.
5. Sound knowledge in Spark architecture and distributed computing and Spark streaming, including RDD and Data frames core functions, troubleshooting and performance tuning.
6. Experience in analyzing, re-architecting, and re-platforming on-premises data warehouses to data platforms on any cloud (AWS is preferred)
7. Experience in any cloud (ETL/ELT pipelines, Datalake and Data streaming) is preferred.
8. Understand the client requirement and able to design, develop from scratch and deliver.
9. Able to mentor new members for onboarding to the project.

SKILLS AND EXPERTISE

Experience: 3-5 Years

• Scala
• Spark
• SQL
• AWS
• Kafka /Kinesis/Any streaming services
• Hive, ETL/ELT design experience
• CI/CD experience (ETL pipeline deployment)
• Data Modeling experience