Senior Big Data Engineer - Spark/Scala on AWS

Pinterest; United States; Remote; USD 124,000 - 255,000; Full time

Job summary

tvScientific is seeking a Data Engineer to design and implement scalable data infrastructure in AWS using Spark with Scala, and to evolve data pipelines to handle massive growth. You will store data efficiently and collaborate across cross-functional teams to meet business needs.

The role includes designing knowledge graphs exposed via batch processing and APIs, with a focus on data quality and AI-assisted workflows within a fast-paced environment.

Qualifications

• Bachelor's degree in Computer Science or related field.
• Experience with Spark and Scala in data infrastructure.
• Experience delivering APIs backed by large datasets.
• Experience with cloud data lakes and AWS.
• Ability to work with AI in workflows and ensure data quality.

Responsibilities

• Design and implement robust data infrastructure in AWS, using Spark with Scala.
• Evolve core data pipelines to scale with growth.
• Store data in optimal engines and formats for performance and cost.
• Collaborate with Data Science and Product teams to meet business needs.
• Design knowledge graphs, expose via Batch Processing and APIs.
• Leverage AWS resources while designing for scale.

Skills

Data engineering

Spark

Scala

APIs

AWS

SQL

Communication

AI in workflow

Data quality checks

Education

Bachelor's degree in Computer Science or related field

Tools

Delta Lake

Apache Iceberg

AWS S3

Job description

tvScientific is seeking a Data Engineer to design and implement scalable data infrastructure in AWS using Spark with Scala, and to evolve data pipelines to handle massive growth. You will store data efficiently and collaborate across cross-functional teams to meet business needs.

The role includes designing knowledge graphs exposed via batch processing and APIs, with a focus on data quality and AI-assisted workflows within a fast-paced environment.