Senior Data Engineer

Talentstream; United States; Remote; USD 110,000 - 160,000; Full time

Job summary

Talentstream is seeking a data engineer to design and maintain real-time and batch data pipelines in a fast-paced environment. You will build scalable data ingestion, transformation, and processing frameworks to support analytics and ML workloads.

You will ensure data reliability, availability, and performance while developing APIs and data models. Strong Python skills and experience with Kafka, Spark Streaming, or Flink are required, along with ETL optimisation.

Qualifications

• Experience in data engineering or large-scale data systems.
• Strong Python skills; Java/Scala is a plus.
• Experience with Kafka, Spark Streaming, or Flink.
• Strong knowledge of data modelling, distributed systems, and APIs.
• Experience with web scraping tools such as Scrapy, Selenium, or Playwright.
• Experience optimising ETL pipelines and handling large datasets.

Responsibilities

• Design and maintain real-time and batch data pipelines.
• Build scalable data ingestion, transformation, and processing frameworks.
• Ensure data reliability, availability, and performance.
• Develop APIs and data models for analytics and ML applications.
• Optimise ETL workflows, queries, and storage systems.

Skills

Python

APIs

Data modelling

Distributed systems

ETL pipelines

Tools

Kafka

Spark Streaming

Flink

Scrapy

Selenium

Playwright

Java

Scala

Job description

Responsibilities

• Design and maintain real-time and batch data pipelines.
• Build scalable data ingestion, transformation, and processing frameworks.
• Ensure data reliability, availability, and performance.
• Develop APIs and data models for analytics and ML applications.
• Optimise ETL workflows, queries, and storage systems.

Requirements

• Experience in data engineering or large-scale data systems.
• Strong Python skills; Java/Scala is a plus.
• Experience with Kafka, Spark Streaming, or Flink.
• Strong knowledge of data modelling, distributed systems, and APIs.
• Experience with web scraping tools such as Scrapy, Selenium, or Playwright.
• Experience optimising ETL pipelines and handling large datasets.