Senior Databricks Data Engineer – Lakehouse & Streaming

Open Systems Inc.; Atlanta (GA); Hybrid; USD 140,000 - 170,000; Full time

Job summary

Open Systems Inc. in Atlanta is seeking a Senior Big Data Engineer to own architecture and delivery of scalable analytics and streaming solutions.

You will design and operate production ingestion and transformation pipelines using Databricks, Delta Lake, Kafka, and AWS services, focusing on data quality, observability, scalability, and maintainability. You will collaborate with Data Science and BI teams to deliver data models, pipelines, and governance across the lakehouse and streaming stack.

Qualifications

• 5+ years of professional data engineering experience in production environments.
• Databricks platform with jobs/workflows, notebooks, and pipelines.
• Delta Lake tables incl. incremental processing, schema evolution, merges/upserts.
• Delta Live Tables (DLT) design, implementation, and monitoring.
• Unity Catalog governance and secured sharing patterns.
• 4+ years of Spark engineering for analytics workloads.
• 3+ years of Kafka or Confluent Kafka for high-volume streams.
• 3+ years of AWS usage supporting analytics platforms.

Responsibilities

• Define data requirements across structured and unstructured data.
• Ingest, validate, normalize, and enrich data for downstream use.
• Implement data quality controls and observability.
• Design and operate lakehouse and streaming pipelines.
• Collaborate with Data Science and BI teams on models and reports.
• Build pipelines for data cleansing, transformation, and publishing for BI/analytics.
• Use Databricks, Spark, scripting, and AWS services to integrate systems.
• Balance correctness, performance, cost, and maintainability.

Skills

Databricks platform

Apache Spark (PySpark)

SQL proficiency

Data pipeline design

Data quality observability

Agile delivery

AWS services

Data governance

Education

Bachelor's degree in CS/IS or related

Tools

Databricks

Delta Lake

Delta Live Tables (DLT)

Unity Catalog

Apache Spark

Kafka/Confluent

AWS

S3

Job description

Open Systems Inc. in Atlanta is seeking a Senior Big Data Engineer to own architecture and delivery of scalable analytics and streaming solutions.

You will design and operate production ingestion and transformation pipelines using Databricks, Delta Lake, Kafka, and AWS services, focusing on data quality, observability, scalability, and maintainability. You will collaborate with Data Science and BI teams to deliver data models, pipelines, and governance across the lakehouse and streaming stack.