Sr. Data Pipeline Engineer

TekStream Solutions

Baltimore (MD)

On-site

USD 90,000 - 120,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

A leading IT services firm in Baltimore is seeking an experienced Data Engineer specializing in the design and optimization of data pipelines. You will collaborate with Data Scientists and Analysts to ensure the delivery of quality data solutions. The ideal candidate should have at least 5 years of experience in data engineering and proficiency in cloud-based technologies. This full-time position requires strong problem-solving and communication skills.

Qualifications

  • 5+ years of experience in data engineering, with a focus on building and maintaining data pipelines.
  • 1-2 years experience with building NiFI data flows or similar for Kafka and Hadoop-based NoSQL databases.
  • Experience with API-led design.
  • Strong problem-solving skills and attention to detail.

Responsibilities

  • Design, develop, and optimize end-to-end data pipelines for efficient data ETL processes.
  • Collaborate with cross-functional teams to understand data requirements.
  • Implement best practices for data integration.

Skills

Data pipeline development
ETL processes
Sql
Cloud technologies
Data validation and cleansing

Education

Bachelor’s degree in Computer Science, Engineering, or a related field

Tools

Apache NiFi
AWS S3
Redshift
Google BigQuery
Python
Scala
ELK stack

Job description

  • Design, develop, and optimize end-to-end data pipelines for efficient data extraction, transformation, and loading (ETL) processes.
  • Collaborate with cross-functional teams to understand data requirements and translate them into scalable pipeline solutions.
  • Implement best practices for data integration, ensuring high performance, reliability, and scalability.
Data Transformation and Quality
  • Transform raw data into usable formats, ensuring data quality, consistency, and accuracy.
  • Handle data validation, cleansing, and error handling to maintain data integrity.
  • Monitor and proactively maintain data pipelines to ensure high service availability.
Performance Optimization
  • Continuously improve pipeline performance by identifying bottlenecks and implementing optimizations.
  • Work with cloud-based technologies (e.g., AWS, GCP, Azure) to enhance scalability and efficiency.
Collaboration and Leadership
  • Partner with Data Scientists, Analysts, and other stakeholders to understand their data needs.
  • Lead discussions on system enhancements, process improvements, and data governance.
  • Mentor junior engineers and contribute to the growth of the data engineering team.
Requirements
  • Bachelor’s degree in Computer Science, Engineering, or a related field.
  • 5+ years of experience in data engineering, with a focus on building and maintaining data pipelines.
  • 1-2 years experience with building NiFI data flows or similar for Kafka and Hadoop-based NoSQL databases.
  • Proficiency in ETL tools, SQL, and scripting languages (Python, Scala, etc.).
  • Experience with Data Catalog and Accumulo indexes for information retrieval and discovery.
  • Experience with API-led design.
  • Experience with ELK stack a plus.
  • Experience with cloud-based data platforms (e.g., AWS S3, Redshift, Google BigQuery).
  • Strong problem-solving skills and attention to detail.
  • Excellent communication and collaboration abilities.
  • Security+ Certification
  • Active US Government Clearance at Secret level or higher
  • Effective written and verbal communications skills for collaboration with both customers and fellow team members.
  • Ability to sit for extended periods of time.
  • Ability to regularly lift at least 25 pounds.
  • Ability to commute to the designated onsite work location as required.
Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Information Technology

Industries

IT Services and IT Consulting

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Data Engineer — End-to-End Pipelines & Cloud
Senior Data Engineer — End-to-End Pipelines & Cloud

TechDigital Group • Warsaw (IN)

On-site
USD 80,000 - 120,000
Senior Data Engineer
Senior Data Engineer

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Senior Data Engineer - Scalable Pipelines & Cloud Analytics
Senior Data Engineer - Scalable Pipelines & Cloud Analytics

Initvalue • Fremont (CA)

Hybrid
USD 130,000 - 190,000
Senior Data Engineer: Cloud Pipelines, Analytics & ML Data
Senior Data Engineer: Cloud Pipelines, Analytics & ML Data

Compunnel Inc. • Charlotte (NC)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

VTG Defense • McLean (VA)

On-site
USD 100,000 - 130,000
Senior Data Engineer: Scalable Pipelines & Data Quality
Senior Data Engineer: Scalable Pipelines & Data Quality

TechDigital Group • Burbank (CA)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

Aptdata Solutions Inc. • Farmington Hills (MI)

On-site
USD 90,000 - 115,000
Data Engineer
Data Engineer

TechDigital Group • Burbank (CA)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

TechDigital Group • New Jersey

Hybrid
USD 100,000 - 130,000
ME00559-Data Engineer
ME00559-Data Engineer

Momentum Engineering, Inc • Washington

On-site
USD 120,000 - 165,000
11 paid holidays
Minimum of 3 weeks PTO
Company sponsored group medical plan
+4