Data Engineer (Databricks)

Cogent People Inc; Columbia (MD); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Health insurance

401(k) Plan

Life Insurance & Disability Coverage

Paid Time Off

Job summary

Cogent People Inc. is seeking a Data Engineer with Databricks experience to build and maintain data pipelines for a large federal healthcare IT program. You’ll work on Delta Lake designs, convert Oracle logic to Databricks, and automate reports within a secure, agile environment.

You will join an Agile team, collaborate with developers, PMs, and testers, and participate in DevSecOps practices, including Git and CI/CD, within federal security boundaries.

Qualifications

• Bachelor's degree in computer science, software engineering, data engineering or related field, or equivalent experience.
• 5+ years hands-on data engineering with at least 2 years on Databricks.
• Experience with Delta Lake, Databricks workflows, and Spark SQL.
• Experience in Agile environments and DevSecOps practices.

Responsibilities

• Translate business needs into scalable Databricks pipelines.
• Build, test, and maintain data pipelines using Python, PySpark, Spark SQL.
• Design Delta Lake tables and ingestion patterns for batch and reports.
• Migrate Oracle SQL/PLSQL logic to Databricks and verify results.
• Automate recurring Databricks reports.
• Schedule and monitor Databricks Workflows for performance and cost.
• Manage Databricks workspace: clusters, policies, users, permissions, secrets.
• Apply data governance with Unity Catalog within security boundaries.
• Follow DevSecOps: Git, CI/CD, environments.
• Collaborate with cross-functional teams in Agile and document work.

Skills

Python

PySpark

SQL

Delta Lake

Databricks

AWS

Agile/Scrum

DevSecOps

Data governance

Education

Bachelor’s degree in CS/SE

Tools

Git

Unity Catalog

CI/CD

Job description

Description

Location: Columbia, MD (Remote/Hybrid if local to Maryland; MON & WED in the office)

Residency Requirement: Candidates must have lived in the U.S. for at least 3 of the past 5 years and be authorized to work in the U.S. (Citizen, Permanent Resident, or EAD).

Sponsorship: This position does not offer any type of sponsorship. Candidates must already be authorized to work in the U.S. (e.g., Citizen, Permanent Resident, or EAD).

Clearance: Public Trust Clearance (or ability to obtain)

Job Overview:

Cogent People Inc. is seeking a skilled and motivated Data Engineer with Databricks experience to join our team. You will build and maintain data pipelines for a mission-critical, data-centric system supporting a large-scale Federal Government IT program in the healthcare domain.

Responsibilities:

• Understand business requirements and turn them into working pipelines that fit the overall architecture and security boundaries.
• Build, test, and maintain data pipelines on Databricks using Python, PySpark, and Spark SQL.
• Design Delta Lake tables and ingestion patterns that work for both batch loads and recurring reports.
• Convert existing Oracle SQL and PL/SQL logic to Databricks, and check that the results match the source system.
• Automate recurring reports on Databricks.
• Schedule and monitor jobs with Databricks Workflows, and tune them for performance, cost and availability.
• Handle day-to-day Databricks workspace administration: compute and cluster policies, users, groups and service principals, permissions, and secrets.
• Apply data governance and access controls using Unity Catalog inside federal security boundaries.
• Follow DevSecOps practices: code in Git, CI/CD for pipelines, and promoting changes through environments.
• Work with developers, project managers, business analysts and testers in an Agile team, and document what you build.

Requirements

• Bachelor’s degree in computer science, software engineering, data engineering or a related field (or equivalent experience).
• 5+ years of hands‑on data engineering, with at least 2 years building production pipelines on Databricks.
• Technical Skills:
  • Strong Python and PySpark
  • Strong SQL
  • Delta Lake, Databricks Workflows/Jobs, and Databricks notebooks or Repos
  • Basic Databricks administration: creating and configuring clusters and SQL warehouses, cluster policies, managing users, groups and service principals, workspace and job permissions, Unity Catalog grants (catalogs, schemas, tables), secret scopes, and monitoring usage and cost
  • AWS data services (S3, IAM, Glue or Lambda)
  • Good understanding of software development processes
• Experience with Agile methodologies (Scrum; SAFe a plus)
• Working knowledge of DevSecOps practices, including CI/CD and release management
• Ability to apply best practices for data security, performance tuning of Spark jobs at scale, and system availability.
• Strong communication and teamwork skills to work with cross‑functional teams, including developers, project managers and business analysts.

Preferred Qualifications:

• Database background (Oracle preferred): data modeling, query tuning, indexing and partitioning
• Oracle PL/SQL development (stored procedures, packages, functions)
• Hands‑on experience migrating Oracle databases or PL/SQL logic to Databricks or Spark
• Databricks Certified Data Engineer (Associate or Professional)
• Experience in regulated or Government environments, especially healthcare
• Understanding of AI/ML principles, or exposure to ML workflows on Databricks (for example, MLflow)
• Experience with analytical and BI tools

Why Cogent People Inc.?

At Cogent People, we foster a collaborative, growth‑driven environment where innovation thrives. You’ll be part of a team working on mission‑critical technology solutions that make a real impact. We offer competitive salaries, career development opportunities, and the chance to contribute to meaningful projects that shape the future.

As a Certified Great Place to Work, we prioritize employee satisfaction and provide a supportive, inclusive culture where every team member is valued and empowered to succeed.

Benefits:

• Health, Dental, and Vision Insurance: Comprehensive coverage.
• 401(k) Plan: Matching contributions.
• Life Insurance & Disability Coverage: Company‑paid life, short‑term, and long‑term disability insurance.
• Paid Time Off: 3 weeks of PTO annually, plus 10 holidays.

EEO Employer:

Cogent People Inc. is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, religion, color, national origin, gender, gender identity, sexual orientation, age, disability status, or any other status protected by law.