Big Data Engineer

Programmers.io; Arlington (VA); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Programmers.io in Arlington, VA is seeking a data engineer who can bridge business, data, and technical teams to uncover use cases and design scalable data solutions.

You will build robust data pipelines end-to-end with a strong emphasis on data quality, using Python or Scala, Spark and Hadoop, plus SQL for analytics.

Experience with Hive, Airflow, NiFi, Scoop and cloud platforms like Databricks or AWS is a plus as you deploy production-grade data-driven applications.

Qualifications

• Experience building end-to-end data pipelines with a focus on data quality.
• Proficiency in Python or Scala, Spark, and Hadoop ecosystems.
• Ability to translate business use cases into scalable data solutions with cross-functional teams.
• Experience deploying production data-driven applications and processing workflows.
• Cloud knowledge (Databricks or AWS) is a plus, but not required.
• Strong SQL skills for analytics and data products.

Responsibilities

• Move between business, data management, and technical teams to map use cases to solutions.
• Build robust data pipelines end-to-end with data quality as a priority.
• Develop data products using Spark, Hadoop, and SQL.
• Deliver analytics across ingestion, transformation, modeling, monitoring, and evaluation.
• Implement ML systems at scale in Python/Scala/Java when needed.

Skills

Python/Scala

Spark

SQL

Data pipelines

Data quality

Tools

Hive

Airflow

NiFi

Scoop

Job description

Location: Arlington, VA (3 Days onsite/week)

Job Summary:

• Ability to easily move between business, data management, and technical teams; ability to quickly intuit the business use case and identify technical solutions to enable it
• Experience building robust and efficient data pipelines end-to-end with a strong focus on data quality
• High proficiency in using Python or Scala, Spark, Hadoop platforms & tools (Hive, Airflow, NiFi, Scoop), SQL to build Big Data products & platforms
• Experience in building and deploying production-level data-driven applications and data processing workflows/pipelines and/or
• Implementing machine learning systems at scale in Java, Scala, or Python and deliver analytics involving all phases like data ingestion, feature engineering, modeling, tuning, evaluating, monitoring, and presenting
• Cloud knowledge (Databricks or AWS ecosystem) is a plus but not required