Databricks Data Engineer

NewGen Technologies; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

NewGen Technologies seeks a data engineer to build and operate Databricks pipelines across bronze/silver/gold layers, migrating legacy logic and data. You will gather requirements, monitor jobs, and document workflows to support critical national-security data initiatives.

The role emphasizes batch and streaming ingestion, Delta Lake, and Medallion Architecture within a DoeD-aware AWS Gov Cloud environment, with strong collaboration across teams.

Qualifications

• US Citizenship required.
• Secret Clearance required.
• Excellent interpersonal and prioritization skills.
• Excellent written and oral communication skills.
• Strong data engineering/ETL/ELT or data warehousing background.
• Solid PySpark/Spark SQL and SQL fundamentals.
• Knowledge of Medallion Architecture and Delta Lake.

Responsibilities

• Build and maintain Databricks pipelines (60% of time).
• Gather and analyze requirements (20% of time).
• Monitor and troubleshoot data jobs (10% of time).
• Document processes and workflows (10% of time).

Skills

PySpark

Spark SQL

SQL fundamentals

ETL/ELT development

Data warehousing

Medallion Architecture

Delta Lake

Batch processing

Streaming ingestion

Agile process

Customer service

DoD IT architectures

Tools

Databricks platform

AWS Gov Cloud

Delta Lake

Job description

Build and operate the ingestion, bronze, silver, and gold pipelines on Databricks, migrating logic and data out of legacy.

Responsibilities

• Build and maintain Databricks pipelines (60% of the time)
• Gather and analyze requirements (20% of the time)
• Monitor and troubleshoot data jobs (10% of the time)
• Document processes and workflows (10% of the time)
• Perform additional assigned tasks

Requirements

• US Citizenship
• Secret Clearance
• Excellent interpersonal skills and ability in overseeing/prioritizing work requests
• Excellent written and oral communication skills
• Strong data engineering, ETL/ELT development, or data warehousing
• Strong PySpark and/or Spark SQL skills; solid SQL fundamentals
• Solid understanding of the Medallion Architecture (Bronze/Silver/Gold layering)
• Proficiency with Delta Lake (ACID transactions, schema evolution, time travel, MERGE/Upsert patterns)
• Experience designing both batch and incremental/streaming ingestion pipelines
• Familiarity with the Agile process

Desired Skills

• Databricks Certification/experience/familiarity preferred
• Knowledge of and experience with the DoD
• Familiar with Customer and/or DoD IT architectures
• Federal experience
• Experience in post go-live production support and break/fix
• Focus on customer service and responsiveness, with sound customer handling skills
• Experience working within AWS Gov Cloud
• Experience working migrating code from non-accredited to accredited environments within AWS

About Us

For more than 20 years, NewGen Technologies has solved our clients’ toughest IT challenges with integrity, security, and outstanding service by delivering both technology and talent. We have helped secure borders, have used artificial intelligence (AI) to fight terror, aided the identification of criminals, and have helped to prevent crime through the introduction of biometrics.Our team of Highly Cleared Specialists have hard-to-find skills and expertise in a wide spectrum of technologies to provide solutions that transform business processes and solve problems of national significance. #CJ