Senior Data Engineer

Optomi; New York (NY); Hybrid; USD 140,000 - 180,000; Full time

Job summary

Optomi, in partnership with a leader in the entertainment industry, seeks a Senior Data Engineer to join a high-impact Product Performance team. Build and optimize large-scale data pipelines powering product analytics, streaming performance, search, and customer experience for a world-class digital platform.

The role offers a 4-day hybrid schedule in Bristol, CT or New York City, NY, with collaboration across Product Managers, Data Analysts, and Engineering teams to deliver scalable,

Qualifications

• 5+ years of software engineering experience developing large-scale data pipelines.
• Strong software engineering experience with Scala and Python.
• Hands-on production experience with distributed processing technologies such as Spark and Kafka.
• Strong SQL skills with experience analyzing and optimizing complex datasets.
• Experience with workflow orchestration tools such as Airflow.
• Strong understanding of CI/CD, SDLC, version control, and software engineering best practices.
• Experience with cloud platforms such as AWS (S3 or similar cloud services).
• Strong analytical, troubleshooting, and communication skills with the ability to solve complex engineering problems.

Responsibilities

• Design, build, and deploy scalable batch and streaming data pipelines using Scala, Python, Spark, and Kafka.
• Develop and maintain core data models and algorithms that support product analytics and business reporting.
• Build reliable, production-ready data solutions supporting large-scale analytics and customer-facing products.
• Investigate production issues, troubleshoot pipeline failures, and implement long-term improvements for reliability and scalability.
• Maintain high standards for software development through code reviews, testing, CI/CD, and deployment best practices.
• Partner with Product Managers, Data Analysts, and engineering teams to deliver new data capabilities and features.
• Continuously optimize data processing performance while ensuring data quality, reliability, and maintainability.
• Contribute to an Agile engineering environment while learning and adopting new technologies as the platform evolves.

Skills

Scala

Python

SQL

Spark

Kafka

Airflow

CI/CD

SDLC

AWS

Tools

Airflow

Spark

Kafka

Job description

Senior Data Engineer (Bristol, CT or New York City, NY – Hybrid)

Optomi, in partnership with a leader in the entertainment industry, is seeking a Senior Data Engineer to join a high-impact Product Performance team! This is an exciting opportunity to build and optimize large-scale data pipelines that power product analytics, streaming performance, search, and customer experience for a world-class digital platform.

What the right candidate will enjoy:

• 4-day hybrid opportunity in Bristol, CT or New York City, NY!
• Building high-scale batch and streaming data pipelines using Scala, Python, Spark, Kafka, and Airflow!
• Developing data solutions that power product analytics, streaming performance, and customer insights!
• Collaborating with Product Managers, Data Analysts, and Engineering teams to deliver scalable, production-grade data solutions!
• Joining a collaborative engineering team that values software engineering best practices, problem solving, and continuous learning!

Experience the right candidate will have:

• 5+ years of software engineering experience developing large-scale data pipelines.
• Strong software engineering experience with Scala and Python.
• Hands-on production experience with distributed processing technologies such as Spark and Kafka.
• Strong SQL skills with experience analyzing and optimizing complex datasets.
• Experience with workflow orchestration tools such as Airflow.
• Strong understanding of CI/CD, SDLC, version control, and software engineering best practices.
• Experience with cloud platforms such as AWS (S3 or similar cloud services).
• Strong analytical, troubleshooting, and communication skills with the ability to solve complex engineering problems.

Responsibilities of the right candidate:

• Design, build, and deploy scalable batch and streaming data pipelines using Scala, Python, Spark, and Kafka.
• Develop and maintain core data models and algorithms that support product analytics and business reporting.
• Build reliable, production-ready data solutions supporting large-scale analytics and customer-facing products.
• Investigate production issues, troubleshoot pipeline failures, and implement long-term improvements for reliability and scalability.
• Maintain high standards for software development through code reviews, testing, CI/CD, and deployment best practices.
• Partner with Product Managers, Data Analysts, and engineering teams to deliver new data capabilities and features.
• Continuously optimize data processing performance while ensuring data quality, reliability, and maintainability.
• Contribute to an Agile engineering environment while learning and adopting new technologies as the platform evolves.