Big Data Engineer – Python/Scala, Spark, Hadoop

Programmers.io; Arlington (VA); On-site; USD 120,000 - 180,000; Full time

Job summary

Programmers.io in Arlington, VA is seeking a seasoned Data Engineer with 10+ years of experience to design and implement end-to-end data pipelines using Python/Scala, Spark and Hadoop. You will work closely with business, data management, and technical teams to translate business use cases into scalable data solutions.

Experience delivering production-grade data-driven applications and ML workflows is preferred; cloud know-how with Databricks or AWS is a plus, though not required.

Qualifications

• 10+ years of experience in data engineering or related field.
• Proficiency with Python or Scala and Spark.
• Experience building robust, scalable data pipelines and ensuring data quality.

Responsibilities

• Design and implement end-to-end data pipelines.
• Collaborate with business, data management, and tech teams to identify solutions.
• Build and deploy production-ready data-driven applications and analytics workflows.

Skills

Data pipelines

Data quality

Cross-functional

Cloud knowledge

Tools

Python

Scala

Spark

Hadoop

Hive

Airflow

NiFi

Sqoop

SQL

Job description

Location: Arlington, VA (3 Days onsite/week)

Hire Type: Full Time

Experience: 10+ Years

Job Summary:

Ability to easily move between business, data management, and technical teams; ability to quickly intuit the business use case and identify technical solutions to enable it

Experience building robust and efficient data pipelines end-to-end with a strong focus on data quality

High proficiency in using Python or Scala, Spark, Hadoop platforms & tools (Hive, Airflow, NiFi, Scoop), SQL to build Big Data products & platforms

Experience in building and deploying production-level data-driven applications and data processing workflows/pipelines and/or

Implementing machine learning systems at scale in Java, Scala, or Python and deliver analytics involving all phases like data ingestion, feature engineering, modeling, tuning, evaluating, monitoring, and presenting

Cloud knowledge (Databricks or AWS ecosystem) is a plus but not required