Senior Data Engineer: Scalable Spark Pipelines & Real-Time

3M Consultancy; Washington; On-site; USD 100,000 - 130,000; Full time

Job summary

A leading consultancy firm in the United States is looking for an experienced Data Engineer to design, build, and maintain scalable data processing pipelines. The ideal candidate will have over 5 years of software development experience with strong proficiency in Python, Java, or Scala, and extensive hands-on experience with Apache Spark. Key responsibilities include optimizing data workflows and supporting data scientists' needs. This is an excellent opportunity for those passionate about large-scale data systems.

Qualifications

• 5+ years of software development experience.
• Extensive hands-on experience with Apache Spark.
• Experience with CI/CD pipelines.

Responsibilities

• Design and implement scalable data pipelines.
• Collaborate with data scientists and analysts to support their data needs.
• Maintain documentation for data processes and architectures.

Skills

Python

Java

Scala

Apache Spark

Data Quality Frameworks

Distributed Computing

Git

SQL

NoSQL

Education

Bachelor's degree in Computer Science, Engineering, or related technical field

Tools

Apache Spark

AWS EMR

Databricks

AWS Glue

KAFKA

Job description

A leading consultancy firm in the United States is looking for an experienced Data Engineer to design, build, and maintain scalable data processing pipelines. The ideal candidate will have over 5 years of software development experience with strong proficiency in Python, Java, or Scala, and extensive hands-on experience with Apache Spark. Key responsibilities include optimizing data workflows and supporting data scientists' needs. This is an excellent opportunity for those passionate about large-scale data systems.