Big Data Engineer: End-to-End Pipelines, ML & Analytics

Programmers.io; Arlington (VA); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Programmers.io in Arlington, VA is seeking a data engineer who can bridge business, data, and technical teams to uncover use cases and design scalable data solutions.

You will build robust data pipelines end-to-end with a strong emphasis on data quality, using Python or Scala, Spark and Hadoop, plus SQL for analytics.

Experience with Hive, Airflow, NiFi, Scoop and cloud platforms like Databricks or AWS is a plus as you deploy production-grade data-driven applications.

Qualifications

• Experience building end-to-end data pipelines with a focus on data quality.
• Proficiency in Python or Scala, Spark, and Hadoop ecosystems.
• Ability to translate business use cases into scalable data solutions with cross-functional teams.
• Experience deploying production data-driven applications and processing workflows.
• Cloud knowledge (Databricks or AWS) is a plus, but not required.
• Strong SQL skills for analytics and data products.

Responsibilities

• Move between business, data management, and technical teams to map use cases to solutions.
• Build robust data pipelines end-to-end with data quality as a priority.
• Develop data products using Spark, Hadoop, and SQL.
• Deliver analytics across ingestion, transformation, modeling, monitoring, and evaluation.
• Implement ML systems at scale in Python/Scala/Java when needed.

Skills

Python/Scala

Spark

SQL

Data pipelines

Data quality

Tools

Hive

Airflow

NiFi

Scoop

Job description

Programmers.io in Arlington, VA is seeking a data engineer who can bridge business, data, and technical teams to uncover use cases and design scalable data solutions.

You will build robust data pipelines end-to-end with a strong emphasis on data quality, using Python or Scala, Spark and Hadoop, plus SQL for analytics.

Experience with Hive, Airflow, NiFi, Scoop and cloud platforms like Databricks or AWS is a plus as you deploy production-grade data-driven applications.