Data Engineer: Databricks & Azure Pipelines (Remote)

United States Digital Space LLC; United States; Remote; USD 100,000 - 120,000; Full time

Benefits offered by this job

401(k)

401(k) matching

Dental insurance

Flexible spending account

Health insurance

Health savings account

Life insurance

Paid time off

Professional development assistance

Retirement plan

Tuition reimbursement

Vision insurance

Job summary

United States Digital Space LLC is seeking a Data Engineer to design, develop, and maintain scalable data solutions supporting a federal healthcare and operations mission. You will collaborate with data analysts, developers, and government stakeholders to optimize data pipelines, ensure secure data handling, and enable analytics across platforms.

The role emphasizes data architecture, ETL/ELT processes, cloud data platforms, and adherence to federal security and privacy standards.

Qualifications

• 3+ years of experience as a Data Engineer or in a similar data-focused role.
• Hands-on experience with Databricks.
• Strong experience building ETL/ELT pipelines.
• Proficiency in Python and SQL.
• Experience with Apache Spark / PySpark.
• Experience with Azure, Azure Synapse Analytics, and Azure Data Factory.
• Solid understanding of data modeling, data warehousing, and analytics use cases.
• Design, develop, and maintain ETL/ELT pipelines to ingest, transform, and load data from multiple sources.
• Build scalable batch and near real-time data pipelines using Databricks and Apache Spark (PySpark / SQL).
• Implement data transformation logic following best practices for performance, reliability, and reusability.

Responsibilities

• Develop and optimize pipelines using Delta Lake and medallion architecture patterns.
• Use Databricks Workflows/Jobs or similar orchestration tools to schedule and monitor pipelines.
• Optimize Spark jobs for performance and cost (partitioning, caching, file sizing, query tuning).
• Collaborate on data governance initiatives using Unity Catalog, access controls, and lineage where applicable.
• Work with data architects and analytics teams to define data requirements.
• Troubleshoot pipeline failures and data quality issues; implement long-term fixes.
• Produce documentation for pipelines, datasets, and operational runbooks.
• Participate in CI/CD practices using Git-based version control for notebooks and code deployments.

Skills

Python

SQL

Databricks

ETL/ELT pipelines

Azure

Azure Synapse Analytics

Azure Data Factory

Spark / PySpark

Data modeling / data warehousing

APIs ingestion

Batch & near real-time pipelines

Data governance

Tools

Delta Lake

Unity Catalog

Airflow

Databricks Workflows/Jobs

Job description

United States Digital Space LLC is seeking a Data Engineer to design, develop, and maintain scalable data solutions supporting a federal healthcare and operations mission. You will collaborate with data analysts, developers, and government stakeholders to optimize data pipelines, ensure secure data handling, and enable analytics across platforms.

The role emphasizes data architecture, ETL/ELT processes, cloud data platforms, and adherence to federal security and privacy standards.