Data Engineer (Databricks)

Cogent People; Columbia (MD); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health, Dental, and Vision Insurance:

401(k) Plan: Matching contributions

Life Insurance & Disability Coverage

Paid Time Off: 3 weeks PTO + 10 days

Job summary

Cogent People Inc. is seeking a skilled Data Engineer with Databricks experience to join our team. You will build and maintain data pipelines for a mission-critical, data-centric system supporting a large-scale Federal Government IT program in the healthcare domain.

In this role, you will translate business requirements into scalable pipelines, design Delta Lake tables, and ensure security boundaries. You will collaborate with developers, project managers, business analysts and testers in an

Qualifications

• Bachelor’s degree or equivalent experience in CS/engineering or related field.
• 5+ years of hands-on data engineering, with at least 2 years on Databricks.
• Strong Python, PySpark and SQL skills for data pipelines.
• Experience with Delta Lake, Unity Catalog, and Databricks workflows.

Responsibilities

• Understand requirements and translate them into data pipelines within security boundaries.
• Build, test, and maintain pipelines on Databricks using Python, PySpark, Spark SQL.
• Design Delta Lake tables and ingestion patterns for batch and reports.
• Migrate Oracle SQL/PL/SQL logic to Databricks and verify results.
• Automate recurring Databricks reports and monitor jobs.
• Manage Databricks Workflows, tuning for performance and cost.
• Administer Databricks workspace: clusters, permissions, and secrets.
• Apply data governance using Unity Catalog in federal security boundaries.
• Follow DevSecOps: Git, CI/CD, and environment promotion.
• Collaborate in an Agile team and document builds.

Skills

Python

PySpark

SQL

Delta Lake

Databricks

AWS

DevSecOps

Agile

Education

Bachelor’s degree in computer science, software engineering, data engineering or a related field

Tools

Databricks

Unity Catalog

Delta Lake

Spark SQL

Python notebooks/repos

Job description

Description

Columbia, MD (Remote/Hybrid if local to Maryland; MON & WED in the office)

Residency Requirement

Candidates must have lived in the U.S. for at least 3 of the past 5 years and be authorized to work in the U.S. (Citizen, Permanent Resident, or EAD).

Sponsorship

This position does not offer any type of sponsorship. Candidates must already be authorized to work in the U.S. (e.g., Citizen, Permanent Resident, or EAD).

Clearance

Public Trust Clearance (or ability to obtain)

Job Overview

Cogent People Inc. is seeking a skilled and motivated Data Engineer with Databricks experience to join our team. You will build and maintain data pipelines for a mission-critical, data-centric system supporting a large-scale Federal Government IT program in the healthcare domain.

Responsibilities

• Understand business requirements and turn them into working pipelines that fit the overall architecture and security boundaries.
• Build, test, and maintain data pipelines on Databricks using Python, PySpark, and Spark SQL.
• Design Delta Lake tables and ingestion patterns that work for both batch loads and recurring reports.
• Convert existing Oracle SQL and PL/SQL logic to Databricks, and check that the results match the source system.
• Automate recurring reports on Databricks.
• Schedule and monitor jobs with Databricks Workflows, and tune them for performance, cost and availability.
• Handle day-to-day Databricks workspace administration: compute and cluster policies, users, groups and service principals, permissions, and secrets.
• Apply data governance and access controls using Unity Catalog inside federal security boundaries.
• Follow DevSecOps practices: code in Git, CI/CD for pipelines, and promoting changes through environments.
• Work with developers, project managers, business analysts and testers in an Agile team, and document what you build.

Requirements

• Bachelor’s degree in computer science, software engineering, data engineering or a related field (or equivalent experience).
• 5+ years of hands-on data engineering, with at least 2 years building production pipelines on Databricks.
• Technical Skills:
• Strong Python and PySpark
• Strong SQL
• Delta Lake, Databricks Workflows/Jobs, and Databricks notebooks or Repos
• Basic Databricks administration: creating and configuring clusters and SQL warehouses, cluster policies, managing users, groups and service principals, workspace and job permissions, Unity Catalog grants (catalogs, schemas, tables), secret scopes, and monitoring usage and cost
• AWS data services (S3, IAM, Glue or Lambda)
• Good understanding of software development processes
• Experience with Agile methodologies (Scrum; SAFe a plus)
• Working knowledge of DevSecOps practices, including CI/CD and release management
• Ability to apply best practices for data security, performance tuning of Spark jobs at scale, and system availability.
• Strong communication and teamwork skills to work with cross-functional teams, including developers, project managers and business analysts.

Preferred Qualifications

• Database background (Oracle preferred): data modeling, query tuning, indexing and partitioning
• Oracle PL/SQL development (stored procedures, packages, functions)
• Hands-on experience migrating Oracle databases or PL/SQL logic to Databricks or Spark
• Databricks Certified Data Engineer (Associate or Professional)
• Experience in regulated or Government environments, especially healthcare
• Understanding of AI/ML principles, or exposure to ML workflows on Databricks (for example, MLflow)
• Experience with analytical and BI tools

Why Cogent People Inc.

At Cogent People, we foster a collaborative, growth-driven environment where innovation thrives. You’ll be part of a team working on mission-critical technology solutions that make a real impact. We offer competitive salaries, career development opportunities, and the chance to contribute to meaningful projects that shape the future.

As a Certified Great Place to Work, we prioritize employee satisfaction and provide a supportive, inclusive culture where every team member is valued and empowered to succeed.

Benefits

• Health, Dental, and Vision Insurance: Comprehensive coverage.
• 401(k) Plan: Matching contributions.
• Life Insurance & Disability Coverage: Company-paid life, short-term, and long-term disability insurance.
• Paid Time Off: 3 weeks of PTO annually, plus 10 holidays.

EEO Employer

Cogent People Inc. is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, religion, color, national origin, gender, gender identity, sexual orientation, age, disability status, or any other status protected by law.