Senior Data Engineer

YO IT Group; New York (NY); On-site; USD 120,000 - 143,000; Full time

Job summary

A leading tech company seeks a Senior Data Engineer to design and implement high-performance data pipelines for large-scale time series data. This role requires at least 8 years of experience in data engineering with a focus on building streaming and batch data pipelines. You will be collaborating with data scientists and ML engineers to ensure data accessibility and quality while mentoring junior staff.

Qualifications

• 8+ years of experience in data engineering.
• Deep experience working with time series and storage systems.
• Proficiency in Python and integrating data pipelines with ML workflows.

Responsibilities

• Design and optimize high-performance data pipelines.
• Collaborate with data scientists to ensure data quality for machine learning.
• Mentor junior engineers and establish best practices.

Skills

Data Engineering

Python

Time Series Data

Streaming Data

Batch Data Pipelines

Cloud-native Architectures

Communication Skills

Job description

This range is provided by YO IT Group. Your actual pay will be based on your skills and experience — talk with your recruiter to learn more.

Base pay range

$120,000.00/yr - $143,000.00/yr

Experience: 8 - 15 Years

Must Have

• 8+ years of experience in data engineering, with a focus on large-scale and high-throughput systems
• Deep experience working with time series data and purpose-built storage systems (e.g., KDB+, TimeSet, Kronos)
• Strong experience building streaming and batch data pipelines using tools like Glue, Kafka, Flink, or Spark
• Proficiency in Python and integrating data pipelines with machine learning workflows and libraries (e.g., pandas, NumPy, scikit-learn, PyTorch)
  
  

Requirements

You’ll bring:

• 8+ years of experience in data engineering, with a focus on large-scale and high-throughput systems
• Deep experience working with time series data and purpose-built storage systems (e.g., KDB+, TimeSet, Kronos)
• Strong experience building streaming and batch data pipelines using tools like Glue, Kafka, Flink, or Spark
• Proficiency in Python and integrating data pipelines with machine learning workflows and libraries (e.g., pandas, NumPy, scikit-learn, PyTorch)
• Experience designing efficient, scalable data models and partitioning strategies for time series data
• Knowledge of distributed systems, columnar databases, and parallel processing
• Familiarity with cloud-native data architectures (AWS, GCP, or Azure) and containerized data infrastructure
• Strong understanding of data quality, lineage, monitoring, and observability tools
• Excellent communication skills and a proactive, consultative mindset in client-facing environments
• Bonus: Experience with multiple time series systems (e.g., KDB+ and Kronos) or contributing to open-source data infrastructure projects
  
  

What You Do

• Design, build, and optimize high-performance data pipelines for large-scale time series data
• Implement scalable data infrastructure using tools such as KDB+, TimeSet (Google’s large time series database), or Kronos
• Develop efficient data ingestion and transformation workflows that handle real-time and historical time series data
• Connect time series data systems with Python-based model pipelines to support machine learning training and inference
• Collaborate with data scientists and ML engineers to ensure data availability, quality, and accessibility for experimentation and production
• Design data models and schemas optimized for time series use cases, including downsampling, aggregation, and indexing strategies
• Ensure system reliability, scalability, and performance through monitoring, testing, and tuning
• Establish data governance, lineage, and observability best practices in large-scale environments
• Mentor junior engineers on large-scale data design, distributed processing, and real-time system architecture
• Partner with product, engineering, and infrastructure teams to align data systems with business goals
  
  

Skills: pipelines,storage systems,azure,observability tools,data pipelines,distributed systems,kdb+,parallel processing,client-facing communication,gcp,data infrastructure,large-scale systems,python,pytorch,time series data,kronos,pandas,data engineering,containerized infrastructure,streaming data pipelines,cloud-native architectures,scikit-learn,cloud-native data architectures,glue,flink,aws,data lineage,monitoring tools,spark,numpy,timeset,communication skills,data models,batch data pipelines,columnar databases,high-throughput systems,machine learning libraries,data quality,kafka,machine learning,data

Seniority level

• Seniority level

  Mid-Senior level

Employment type

• Employment type

  Full-time

Job function

• Job function

  Information Technology

• Industries

  IT Services and IT Consulting

Referrals increase your chances of interviewing at YO IT Group by 2x

Get notified about new Senior Data Engineer jobs in New York, NY.

Fairfield, NJ $160,000.00-$195,000.00 1 week ago

New York, NY $150,000.00-$160,000.00 1 month ago

New York, NY $153,000.00-$210,000.00 6 months ago

Senior Data Architect/Engineer - Leading Systematic Hedge Fund - New York

New York, NY $147,000.00-$216,000.00 1 week ago

Azure Data Architect - Remote With Travel

Senior Data Analyst cum Business Analyst

Sr Data Engineer – P&C Insurance / Reinsurance

New York, NY $130,000.00-$150,000.00 4 days ago

Jersey City, NJ $153,000.00-$196,000.00 1 week ago

Software or Data Engineers, or Business Analyst

New York, NY $170,000.00-$223,650.00 2 days ago

Senior Data Solutions Developer - Private Markets (VP)

New York, NY $125,000.00-$165,000.00 2 weeks ago

Jersey City, NJ $142,320.00-$213,480.00 2 weeks ago

Healthcare Data Architect / Sr. Data Engineer

New York, NY $200,000.00-$222,000.00 1 week ago

New York, NY $124,323.31-$128,959.50 9 hours ago

Senior Software Engineer - Partner Experience Team, Bloomberg Connects

New York, NY $160,000.00-$240,000.00 1 day ago

Rutherford, NJ $176,720.00-$265,080.00 6 days ago

Senior Data Analyst – Middle Office & Investment Systems

New York City Metropolitan Area 1 week ago

Data Analyst - TC - Data and Analy - Data Arch and Eng - FSO - Sr Mgr - Mul Positions - 1615992

Senior Software Engineer - BLAW Contracts Drafting

New York, NY $160,000.00-$240,000.00 1 day ago

Senior / Staff Software Engineer - Computational Chemistry / Molecular Dynamics

Jersey City, NJ $150,000.00-$205,000.00 4 months ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.