Big Data

Affine Analytics; United States; Remote; USD 90,000 - 140,000; Full time

Job summary

Affine Analytics is seeking a data engineer with strong SQL and hands-on Scala skills to drive data pipelines and architecture. The role emphasizes data quality, performance optimization, and scalable design across batch and streaming contexts.

The candidate should be proficient in Spark, cloud platforms (preferably AWS), and ETL/ELT design, with experience in data modeling and mentoring teammates on complex data projects.

Qualifications

• Strong SQL skills for complex queries and data manipulation.
• Hands-on Scala programming with practical data processing tasks.
• Solid data engineering fundamentals and architectural thinking.
• Experience with data quality practices, optimization, and scalable design.

Responsibilities

• Strong SQL query writing and optimization for real-world scenarios.
• Hands-on Scala development for data pipelines.
• Apply solid data engineering fundamentals across batch and streaming workloads.
• Design and optimize ETL/ELT pipelines and data models for cloud platforms.

Skills

Scala

Spark

SQL

AWS

Kafka

Hive

ETL/ELT design

CI/CD (ETL pipelines)

Data modeling

Job description

Roles & Responsibilities:

1. Strong SQL Skills: Candidate should be proficient in writing and executing complex SQL queries (no PL/SQL) and be able to handle real-world scenarios confidently.
2. Hands-on Functional Programming (Scala): Strong hands-on experience in Scala is essential.
3. Solid Data Engineering Fundamentals: Emphasis is on strong foundational understanding rather than tool-specific expertise.
4. Experience in Data Quality & Optimization: Candidate should have a sound understanding of data quality practices, performance optimization techniques, and solution design in both event-driven and batch processing contextsincluding data modeling.
5. Sound knowledge in Spark architecture and distributed computing and Spark streaming, including RDD and Data frames core functions, troubleshooting and performance tuning.
6. Experience in analyzing, re-architecting, and re-platforming on-premises data warehouses to data platforms on any cloud (AWS is preferred)
7. Experience in any cloud (ETL/ELT pipelines, Datalake and Data streaming) is preferred.
8. Understand the client requirement and able to design, develop from scratch and deliver.
9. Able to mentor new members for onboarding to the project.

SKILLS AND EXPERTISE

Experience:

3-5 Years

Must have Skills:

• Scala
• Spark
• SQL
• AWS
• Kafka /Kinesis/Any streaming services
• Hive, ETL/ELT design experience
• CI/CD experience (ETL pipeline deployment)
• Data Modeling experience