Databricks Data Engineer Lakehouse & Cloud Pipelines

Big Impact Tech (BIT); Washington (District of Columbia); Hybrid; USD 130,000 - 190,000; Full time

Job summary

Big Impact Tech (BIT) is seeking a Data Engineer with Databricks expertise to support Coast Guard data modernization, analytics, and cloud transformation initiatives. The role is hybrid in Washington, DC, 2-3 days on site per week, collaborating with data architects, analysts, and stakeholders to deliver reliable data platforms.

You will design and maintain scalable data pipelines, Delta Lake lakehouse solutions, and cloud-based platforms, with PySpark/Spark SQL/SQL transformations for

Qualifications

• 5+ years of data engineering experience.
• Strong Databricks and cloud data platform expertise.
• Proficient in PySpark, Spark SQL, and SQL.
• Experience building scalable data pipelines and ETL/ELT solutions.
• Knowledge of data warehousing and lakehouse architectures.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and cloud-native technologies.
• Build and optimize ETL/ELT processes for structured, semi-structured, and unstructured data.
• Develop and manage data ingestion frameworks from various source systems, APIs, databases, and files.
• Implement and maintain Delta Lake architecture and Lakehouse solutions.
• Develop PySpark, Spark SQL, and SQL-based transformations for large-scale data processing.
• Support data warehousing, data lakes, and enterprise analytics initiatives.
• Monitor and optimize data pipeline performance, reliability, and scalability.
• Ensure data quality, governance, lineage, and security across data platforms.
• Collaborate with business stakeholders and technical teams to understand requirements and deliver data solutions.
• Implement CI/CD practices and automated deployment processes for data engineering solutions.
• Troubleshoot and resolve production issues related to data ingestion, transformation, and reporting.
• Create and maintain technical documentation, data dictionaries, and architecture diagrams.

Skills

Databricks

PySpark

Spark SQL

SQL

ETL/ELT

Data Pipelines

Cloud Platforms

Education

Bachelor’s degree in Computer Science, Information Systems, Engineering, Data Science, or related field

Tools

Azure

AWS

GCP

Delta Lake

Databricks Workflows

Job description

Big Impact Tech (BIT) is seeking a Data Engineer with Databricks expertise to support Coast Guard data modernization, analytics, and cloud transformation initiatives. The role is hybrid in Washington, DC, 2-3 days on site per week, collaborating with data architects, analysts, and stakeholders to deliver reliable data platforms.

You will design and maintain scalable data pipelines, Delta Lake lakehouse solutions, and cloud-based platforms, with PySpark/Spark SQL/SQL transformations for