Data Engineer: ETL, Data Lake & Cloud Pipelines (Clearance)

Jobtailor; Vienna (VA); On-site; USD 130,000 - 180,000; Full time

Job summary

Jobtailor is seeking a senior data engineer to build and optimize batch and real-time data ingestion pipelines in a cloud-first environment. You will design scalable ETL processes across distributed systems and manage data lake architecture for structured and unstructured data.

You will utilize dbt, Informatica, Azure Data Factory, Databricks, AWS Glue, and related services to transform data, orchestrate workflows, and ensure data quality.

Qualifications

• Bachelor's degree in Computer Science or a related field; minimum 3 years of relevant work experience.
• U.S. citizenship; active security clearance or ability to obtain one.

Responsibilities

• Build, optimize, and maintain batch and real-time data ingestion pipelines.
• Lead design and implementation of scalable ETL processes across distributed systems.
• Develop and manage data lake architecture for structured and unstructured data.
• Perform data transformation using dbt, Informatica, Azure Data Factory, Databricks, AWS Glue, and related tools.
• Write performant SQL queries and Python/JavaScript scripts for data parsing and cleansing.
• Conduct data profiling, linkage, validation, and quality checks.

Skills

ETL Frameworks

Data Ingestion

Data Transformation

SQL Proficiency

Python Programming

Pandas

PySpark

AWS Glue

Azure Data Factory

Databricks

Education

Bachelor's degree in Computer Science or related field

Tools

Dbt

Informatica

Azure Data Factory

Databricks

AWS Glue

AWS EventBridge

S3 Event Notifications

Pandas

PySpark

Azure Blob

Job description

Jobtailor is seeking a senior data engineer to build and optimize batch and real-time data ingestion pipelines in a cloud-first environment. You will design scalable ETL processes across distributed systems and manage data lake architecture for structured and unstructured data.

You will utilize dbt, Informatica, Azure Data Factory, Databricks, AWS Glue, and related services to transform data, orchestrate workflows, and ensure data quality.