Data Engineer with IRS MBI Clearance

3M Consultancy; Washington; On-site; USD 100,000 - 130,000; Full time

Job summary

A leading consultancy firm in the United States is looking for an experienced Data Engineer to design, build, and maintain scalable data processing pipelines. The ideal candidate will have over 5 years of software development experience with strong proficiency in Python, Java, or Scala, and extensive hands-on experience with Apache Spark. Key responsibilities include optimizing data workflows and supporting data scientists' needs. This is an excellent opportunity for those passionate about large-scale data systems.

Qualifications

• 5+ years of software development experience.
• Extensive hands-on experience with Apache Spark.
• Experience with CI/CD pipelines.

Responsibilities

• Design and implement scalable data pipelines.
• Collaborate with data scientists and analysts to support their data needs.
• Maintain documentation for data processes and architectures.

Skills

Python

Java

Scala

Apache Spark

Data Quality Frameworks

Distributed Computing

Git

SQL

NoSQL

Education

Bachelor's degree in Computer Science, Engineering, or related technical field

Tools

Apache Spark

AWS EMR

Databricks

AWS Glue

KAFKA

Job description

Washington, United States | Posted on 01/11/2026

We are seeking an experienced Data Engineer to design, build, and maintain scalable data processing pipelines using modern big data technologies. The ideal candidate will have strong development skills and experience with distributed computing frameworks.

Required Skills & Qualifications:

• Bachelor's degree in Computer Science, Engineering, or related technical field
• 5+ years of software development experience
• Strong proficiency in Python, Java or Scala programming languages
• Hands on Experience working with cloud-based Spark platforms (Databricks, AWS EMR, AWS Glue)
• Strong understanding of Hive, Unity catalog, Glue catalog
• Hands on Experience working with any data quality framework/tools
• Hands on Experience working with Observability and monitoring for data processing pipelines
• Strong understanding of distributed computing concepts
• Proficiency in version control systems (Github/GitLab/Bitbucket)

Extensive hands-on experience with Apache Spark, including:

• Spark SQL
• Spark Streaming and its data sources KAFKA, CDC feed, etc
• Performance optimization and tuning
• Data transformation and processing

Technical Requirements:

• Experience building and maintaining high-volume data processing pipelines
• Knowledge of data modeling and ETL / ELT best practices
• Familiarity with SQL and NoSQL databases
• Understanding of data warehouse concepts and dimensional modeling

Preferred Qualifications:

• Experience with real-time data processing and streaming architectures
• Knowledge of Delta Lake or similar data lakehouse technologies
• Experience with CI/CD pipelines
• Contributions to open-source projects

Key Responsibilities:

• Design and implement scalable data pipelines
• Optimize existing data workflows for performance and reliability
• Collaborate with data scientists and analysts to support their data needs
• Implement data quality checks and monitoring
• Maintain documentation for data processes and architectures

Soft Skills:

• Strong problem-solving abilities
• Team collaboration
• Ability to work independently
• Strong attention to detail

The ideal candidate should have a proven track record of developing and maintaining production-grade data pipelines and be passionate about working with large-scale data systems.