Data Engineer

Gridiron IT; Washington (District of Columbia); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

401(k)

HSA/FSA

Disability & ADD insurance

Life insurance

Pet insurance

Job summary

Gridiron IT is seeking an experienced Databricks Data Engineer to design and maintain scalable data pipelines in a federal program. You will work with Databricks, Spark, Python, and SQL to enable analytics and operations across large, secure datasets.

Collaboration with data engineers, scientists, and stakeholders is essential, with emphasis on data quality and governance. You will build Delta Lake architectures, use CI/CD for pipelines, and apply best practices in security and information

Qualifications

• Bachelor's degree or equivalent professional experience in a technical field.
• Hands-on experience with Databricks and Spark.
• Proficiency in Python and SQL for data processing.
• Experience with large-scale distributed datasets and data pipelines.
• Knowledge of data modeling, warehousing, and lakehouse architectures.
• Familiarity with version control and CI/CD.
• Security and compliance awareness for government programs.
• DoD or Intelligence Community experience is a plus.

Responsibilities

• Design, develop, and maintain scalable data pipelines with Databricks and Spark.
• Build ETL/ELT workflows for ingesting transforming data.
• Develop processing solutions using Python, PySpark, and SQL.
• Build and optimize Delta Lake / Lakehouse architectures.
• Integrate data from APIs, databases, and cloud storage.
• Implement data quality, governance, and monitoring.
• Optimize Spark workloads and Databricks jobs for performance.
• Enable CI/CD deployment of data pipelines.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

Data modeling

Data warehousing

CI/CD practices

Git

Collaboration

Security/compliance

Education

Bachelor's degree in Computer Science, Engineering, IT, Data Science, or related field, or equivalent experience

Tools

Databricks Delta Lake

Unity Catalog

Databricks Workflows/Jobs

AWS (S3, Glue, Lambda, Redshift, EMR, IAM)

Azure (Data Lake Storage, Data Factory, Synapse, DevOps)

Airflow

Terraform

Docker

Kubernetes

Job description

We are seeking an experienced Databricks Data Engineer to support a mission-critical federal program. The ideal candidate will have hands-on experience building and maintaining scalable data pipelines and data platforms using Databricks, Apache Spark, Python, and SQL.

This role will work with large, complex datasets in a secure environment and collaborate with data engineers, software engineers, data scientists, and mission stakeholders to make data accessible, reliable, and usable for analytics and operational requirements.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark
• Build ETL/ELT workflows for ingesting, transforming, and processing structured and unstructured data
• Develop data processing solutions using Python, PySpark, and SQL
• Build and optimize Delta Lake / Lakehouse architectures
• Integrate data from APIs, databases, cloud storage, and other enterprise data sources
• Implement data quality, validation, governance, and monitoring processes
• Optimize Spark workloads and Databricks jobs for performance and scalability
• Support automated deployment of data pipelines through CI/CD
• Work with cloud services and modern data architectures in AWS or Azure
• Collaborate with data scientists and analysts to prepare datasets for analytics, reporting, and machine learning
• Troubleshoot data pipeline, integration, and performance issues
• Maintain technical documentation for data pipelines, architectures, and processes
• Operate within federal security and information assurance requirements
• Required Qualifications
• Active TS/SCI clearance
• Bachelor's degree in Computer Science, Engineering, Information Technology, Data Science, or a related technical field, or equivalent professional experience
• 3+ years of professional data engineering or software engineering experience
• Hands-on experience with Databricks
• Strong experience with Apache Spark / PySpark
• Strong programming experience with Python
• Experience working with large-scale or distributed datasets
• Knowledge of data modeling, data warehousing, and modern data lake/lakehouse architectures
• Experience with Git and modern software development practices
• Preferred Qualifications
• Experience with Databricks Delta Lake
• Experience with Unity Catalog
• Experience with Databricks Workflows or Jobs
• Experience with AWS services such as S3, Glue, Lambda, Redshift, EMR, or IAM
• Experience with Azure services such as Azure Data Lake Storage, Azure Data Factory, Synapse, or Azure DevOps
• Experience with orchestration tools such as Apache Airflow
• Experience with Terraform or other Infrastructure-as-Code tools
• Experience with Docker and Kubernetes
• Experience implementing CI/CD pipelines
• Knowledge of data governance, lineage, access controls, and data security
• Databricks certifications such as Databricks Certified Data Engineer Associate or Databricks Certified Data Engineer Professional
• Previous experience supporting DoD, Intelligence Community, or other federal government customers

Clearance:

Applicants selected will be subject to a security investigation and may need to meet eligibility requirements for access to classified information. TS/SCI clearance required.

Benefits: Gridiron offers a comprehensive benefits package including medical, dental, vision insurance, HSA, FSA, 401(k), disability & ADD insurance, life and pet insurance to eligible employees. Full-time and part-time employees working at least 30 hours per week on a regular basis are eligible to participate in Gridiron's benefits programs.

Gridiron IT Solutions is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, protected veteran status or disability status.

Gridiron IT is a Women Owned Small Business (WOSB) headquartered in the Washington, D.C. area that supports our clients' missions throughout the United States. Gridiron IT specializes in providing comprehensive IT services tailored to meet the needs of federal agencies. Our capabilities include IT Infrastructure & Cloud Services, Cyber Security, Software Integration & Development, Data Solution & AI, and Enterprise Applications. These capabilities are backed by Gridiron IT's experienced workforce and our commitment to ensuring we meet and exceed our clients' expectations.