Big Data Engineer

Affine Analytics; Bengaluru; On-site; INR 1,400,000 - 2,100,000; Full time

Job summary

Affine Analytics in Bengaluru is seeking a skilled Big Data Engineer to expand and optimize our data architecture, pipelines, and data flows for cross-functional teams. You will build robust data wrangling solutions, support software developers, data scientists, and analysts, and mentor junior team members.

The role emphasizes strong fundamentals, cloud readiness, and experience with Spark streaming and data quality practices across event-driven and batch processing contexts.

Qualifications

• Strong SQL skills and ability to write complex queries.
• Hands-on Scala programming experience.
• Solid data engineering fundamentals across batch and streaming.
• Experience with data quality, optimization, and data modeling.

Responsibilities

• Write and execute complex SQL queries and optimize data flows.
• Develop data pipelines using Scala and Spark.
• Design and optimize data architectures and ETL/ELT processes.
• Work on on-prem to cloud data platform migrations (AWS preferred).
• Support data initiatives for cross-functional teams and mentor new members.

Skills

Scala

Spark

SQL

AWS

Kafka/Kinesis

Hive

ETL/ELT design

CI/CD

Data Modeling

Job description

Role Objective:

Big Data Engineer will be responsible for expanding and optimizing our data and database architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing data systems and building. The Data Engineer will support our software developers, database architects, data analysts and data scientists on data initiatives and will ensure optimal data delivery architecture is consistent throughout ongoing projects. They must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products

Roles & Responsibilities:

1. Strong SQL Skills: Candidate should be proficient in writing and executing complex SQL queries (no PL/SQL) and be able to handle real-world scenarios confidently.
2. Hands-on Functional Programming (Scala): Strong hands-on experience in Scala is essential.
3. Solid Data Engineering Fundamentals: Emphasis is on strong foundational understanding rather than tool-specific expertise.
4. Experience in Data Quality & Optimization: Candidate should have a sound understanding of data quality practices, performance optimization techniques, and solution design in both event-driven and batch processing contextsincluding data modeling.
5. Sound knowledge in Spark architecture and distributed computing and Spark streaming, including RDD and Data frames core functions, troubleshooting and performance tuning.
6. Experience in analyzing, re-architecting, and re-platforming on-premises data warehouses to data platforms on any cloud (AWS is preferred)
7. Experience in any cloud (ETL/ELT pipelines, Datalake and Data streaming) is preferred.
8. Understand the client requirement and able to design, develop from scratch and deliver.
9. Able to mentor new members for onboarding to the project.

Must have Skills:

• Scala
• Spark
• SQL
• AWS
• Kafka /Kinesis/Any streaming services
• Hive, ETL/ELT design experience
• CI/CD experience (ETL pipeline deployment)
• Data Modeling experience