Senior Data Engineer

Next Ventures; San Francisco (CA); On-site; USD 120,000 - 160,000; Full time

Job summary

A Real Time Intelligence Company is seeking a Senior Data Engineer to build and maintain data pipelines for large scale data ingestion. Responsibilities include developing AI applications, writing CI/CD pipelines, and designing analytics infrastructure. Ideal candidates will have over 5 years of experience in Data Engineering and proficiency in tools like Apache Storm, Kafka, and Databricks.

Qualifications

• Professional experience in using Apache Storm, Kafka, or Flink for real time data or streaming.
• 5+ years of experience in Data Engineering and building pipelines using Databricks and Spark.
• Demonstrates experience in building CI/CD pipelines and DevOps best practices.
• Experience with real time data streaming.

Responsibilities

• Build and maintain data pipelines for large scale data ingestion, real time analytics and Machine Learning Services.
• Develop and optimize NLP, LLMs and other AI powered applications.
• Write CI/CD pipelines and conduct code reviews.
• Design and build a data analytics infrastructure.

Skills

Apache Storm

Kafka

Flink

Databricks

Spark

Real time data streaming

CI/CD pipelines

DevOps best practices

Job description

I am partnered with a Real Time Intelligence Company, who have built a platform that uses AI to transform publicly available data into real time operational insights.

With a huge part of their business growing, they are now looking to hire a Senior Data Engineer, who will be an integral part of the team, working on new and exciting projects, building new pipelines, migrating from one service to another service and building a data and analytics infrastructure.

Responsibilities

• Build and maintain data pipelines for large scale data ingestion, real time analytics and Machine Learning Services
• Develop and optimize NLP, LLMs and other AI powered applications
• Write CI/CD pipelines and have the ability to do code reviews
• Design and build a data analytics infrastructure

Qualifications

• Professional Experience in using Apache Storm, Kafka or Flink for real time data or streaming
• 5+ years experience in Data Engineering and building pipelines using Databricks and Spark
• Can demonstrate experience in building CI/CD pipelines and DevOps best practises
• Experience with real time data streaming