Databricks Data Engineer

Brite Consulting; San Antonio (TX); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Brite Consulting is seeking an experienced Databricks Data Engineer to support the War Data Platform in a complex federal financial management environment.

You will design and implement scalable data solutions with Databricks, Python, PySpark, and SQL, collaborating with data engineers, analysts, SMEs, and stakeholders to deliver trusted data products for downstream analytics and reporting. Hybrid work with DoD-focused data projects.

Qualifications

• 5+ years of experience in data engineering or related technical field.
• Strong hands-on experience with Databricks.
• Proficiency in Python and advanced SQL.

Responsibilities

• Support the continued development of the War Data Platform (WDP).
• Design, develop, test, and maintain scalable data solutions in Databricks.
• Build ETL/ELT pipelines using Python, PySpark, Spark SQL, and Databricks SQL.
• Integrate data from multiple sources into trusted analytical datasets.
• Develop notebooks, jobs, and workflows for automated data processing.
• Collaborate with stakeholders to translate requirements into technical solutions.
• Support downstream reporting and visualization using BI tools.

Skills

Databricks

Python

PySpark

SQL

ETL/ELT

Data modeling

Data warehousing

Lakehouse

Spark SQL

Git version control

Education

Bachelor's degree in Computer Science, Information Systems, Data Science, Engineering, Mathematics, Statistics

Tools

Databricks

SQL

Python

Spark

Tableau

Qlik

Power BI

Git

AWS S3

Job description

Brite Consulting is a management and technology consulting firm dedicated to delivering value-added services that impact the public and private sectors. We work closely with our clients to optimize their operational needs and provide innovative solutions that help them achieve their missions.

We are a small business with a big heart. We are passionate about helping our clients succeed, and we are committed to providing the highest quality of service. Join us in building a brighter future.

Job Overview:

We are seeking an experienced Databricks Data Engineer to support the continued development and advancement of the War Data Platform (WDP) within a complex federal financial management environment.

The ideal candidate will have strong hands-on experience developing scalable data solutions using Databricks, Python, PySpark, Databricks SQL, and SQL. This individual will serve as a key technical resource helping the team navigate the WDP environment, design and build scalable data solutions, transform large and complex datasets, and develop trusted data products for downstream analytics and reporting.

The successful candidate should be comfortable working through ambiguous data challenges, evaluating existing data sources and processes, recommending a technical path forward, and then building the solution. The developer will work closely with data engineers, analysts, functional SMEs, developers, and business stakeholders to translate requirements into reliable, maintainable, and efficient WDP solutions.

Responsibilities:

• Serve as a key technical resource supporting the continued development and maturation of the War Data Platform (WDP).
• Help the team navigate the WDP data environment, architecture, source systems, development processes, and technical dependencies.
• Design, develop, test, and maintain scalable data solutions within Databricks.
• Build ETL/ELT pipelines using Python, PySpark, Spark SQL, Databricks SQL, and SQL.
• Integrate data from multiple enterprise data sources into consolidated and trusted analytical datasets.
• Perform complex joins, aggregations, transformations, reconciliation logic, and business-rule implementation.
• Develop reusable Databricks notebooks, jobs, workflows, and scheduled data processes.
• Develop data models and curated datasets to support downstream reporting, analytics, and decision-making.
• Support migration of existing reporting and analytical processes into more automated and scalable WDP-based solutions.
• Analyze and reconcile data across multiple systems to identify discrepancies, missing data, business-rule issues, and data-quality concerns.
• Develop automated data validation and quality-control processes to ensure outputs are accurate, complete, reliable, and traceable.
• Troubleshoot data pipeline failures, performance issues, and complex data-quality problems.
• Optimize Spark workloads, Databricks compute resources, and SQL queries for performance and scalability.
• Collaborate with business stakeholders and functional SMEs to understand requirements and translate them into technical solutions.
• Support downstream reporting and visualization solutions using Qlik, Tableau, Power BI, or similar platforms.
• Document data pipelines, transformation logic, source-to-target mappings, dependencies, business rules, and technical processes.
• Follow development standards and best practices for source control, testing, deployment, code review, and maintainability.
• Help establish development standards and best practices supporting the long‑term scalability and maintainability of the WDP environment.
• Communicate technical issues, risks, dependencies, and recommendations clearly to technical and non-technical stakeholders.
• Provide technical guidance to team members as WDP capabilities continue to mature.

Qualifications:

• 5+ years of experience in data engineering, software development, analytics engineering, data architecture, or a related technical field.
• Strong hands‑on experience developing production data solutions in Databricks.
• Strong proficiency in Python and advanced SQL.
• Experience using PySpark and Apache Spark to process large datasets.
• Experience with Spark SQL and Databricks SQL.
• Experience building and maintaining ETL/ELT pipelines and data transformation processes.
• Experience developing and maintaining Databricks notebooks, jobs, workflows, and scheduled workloads.
• Strong understanding of relational data, complex joins, data modeling, and data transformation.
• Experience working with large and complex enterprise datasets.
• Understanding of data warehousing, data lakes, and lakehouse architecture.
• Ability to troubleshoot and optimize data pipelines, Spark workloads, and SQL queries.
• Demonstrated ability to assess an existing data environment and help determine an appropriate technical solution or architecture.
• Experience gathering technical requirements and translating business needs into technical solutions.
• Strong analytical, troubleshooting, and root‑cause analysis capabilities.
• Strong communication and collaboration skills.
• Ability to work independently while supporting a larger development team.
• Must be able to pass a government security background check.
• Authorization to work in the United States is required; visa sponsorship is not available at this time.

Preferred Skills:

• Previous experience working within the War Data Platform (WDP) or another Department of Defense data environment.
• Experience supporting DoD, DHA, or another federal financial management organization.
• Experience developing in R (including sparklyr, dplyr, or similar R‑based frameworks) or Scala within a Databricks/Spark environment.
• Experience with Delta Lake and Delta tables.
• Experience implementing Bronze, Silver, and Gold data architectures.
• Experience with Databricks Workflows / Lakeflow Jobs.
• Experience configuring and optimizing Databricks compute resources.
• Familiarity with Unity Catalog, data governance, permissions, and lineage.
• Experience with AWS and Amazon S3.
• Experience with Git‑based version control, collaborative development, testing, and CI/CD practices.
• Experience integrating Databricks data products with Qlik, Tableau, Power BI, or other BI platforms.
• Experience supporting financial, accounting, budget execution, acquisition, contract, invoice, operational, logistics, healthcare, or other federal government datasets.
• Experience developing solutions requiring strong data lineage, auditability, governance, and traceability.
• Experience helping teams transition from legacy or manual reporting processes to modern data platforms.
• Databricks certification, such as Databricks Certified Data Engineer Associate or Professional, is a plus.

Location:

• Hybrid, based on client requirements.

Travel Required:

• Minimal

Minimum Required Years of Experience:

• 5+ years of relevant professional experience, including hands‑on data engineering or development.
• Significant hands‑on development experience with Databricks, SQL, and Python is required.

Education Requirements:

• Bachelor's degree in Computer Science, Information Systems, Data Science, Engineering, Mathematics, Statistics, or a related technical field. Relevant professional experience may be considered in place of a degree.

Security Clearance/Investigation:

• T-3 Investigation/Tier II or ability to obtain the required government background investigation.