Senior Data Engineer — Remote with Spark & Databricks

ICF; Reston (VA); On-site; USD 90,000 - 152,000; Full time

Job summary

ICF is seeking a seasoned Senior Data Engineer in the United States to drive data pipeline design and delivery. You will build scalable workflows with Spark, Hive, and Airflow, deploy Databricks pipelines, and create API services for data access.

Strong AWS QuickSight visuals and data security practices are required for success. Candidates should have 7+ years of software development, 4+ years in data pipelines, and be eligible for a Public Trust clearance, residing in the US.

Qualifications

• Bachelor's degree required.
• 7+ years hands-on software development experience.
• 4+ years building data pipelines with Python/Java and cloud tech; Spark/Hive for large-scale processing.
• Must obtain and maintain a Public Trust clearance.
• Must reside in the US and be authorized to work in the US; work performed in the US.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Spark, Hive, and Airflow.
• Develop and deploy data processing workflows on Databricks.
• Develop API services for data access and integration.
• Create interactive data visualizations and reports using AWS QuickSight.
• Ensure data security and privacy for data at rest and in transit.

Skills

7+ years software development

4+ years data pipelines

Public Trust clearance

US work authorization

Education

Bachelor's degree

Tools

Databricks

Airflow

Terraform

Github Actions

Jupyter Notebooks

Luigi

Azkaban

Spark-Streaming

Cassandra

Postgres

Job description

ICF is seeking a seasoned Senior Data Engineer in the United States to drive data pipeline design and delivery. You will build scalable workflows with Spark, Hive, and Airflow, deploy Databricks pipelines, and create API services for data access.

Strong AWS QuickSight visuals and data security practices are required for success. Candidates should have 7+ years of software development, 4+ years in data pipelines, and be eligible for a Public Trust clearance, residing in the US.