Big Data Engineer: Various levels (Lead, Senior, Intermediate)

Open Systems Inc.; Atlanta (GA); Hybrid; USD 140,000 - 170,000; Full time

Job summary

Open Systems Inc. in Atlanta is seeking a Senior Big Data Engineer to own architecture and delivery of scalable analytics and streaming solutions.

You will design and operate production ingestion and transformation pipelines using Databricks, Delta Lake, Kafka, and AWS services, focusing on data quality, observability, scalability, and maintainability. You will collaborate with Data Science and BI teams to deliver data models, pipelines, and governance across the lakehouse and streaming stack.

Qualifications

• 5+ years of professional data engineering experience in production environments.
• Databricks platform with jobs/workflows, notebooks, and pipelines.
• Delta Lake tables incl. incremental processing, schema evolution, merges/upserts.
• Delta Live Tables (DLT) design, implementation, and monitoring.
• Unity Catalog governance and secured sharing patterns.
• 4+ years of Spark engineering for analytics workloads.
• 3+ years of Kafka or Confluent Kafka for high-volume streams.
• 3+ years of AWS usage supporting analytics platforms.

Responsibilities

• Define data requirements across structured and unstructured data.
• Ingest, validate, normalize, and enrich data for downstream use.
• Implement data quality controls and observability.
• Design and operate lakehouse and streaming pipelines.
• Collaborate with Data Science and BI teams on models and reports.
• Build pipelines for data cleansing, transformation, and publishing for BI/analytics.
• Use Databricks, Spark, scripting, and AWS services to integrate systems.
• Balance correctness, performance, cost, and maintainability.

Skills

Databricks platform

Apache Spark (PySpark)

SQL proficiency

Data pipeline design

Data quality observability

Agile delivery

AWS services

Data governance

Education

Bachelor's degree in CS/IS or related

Tools

Databricks

Delta Lake

Delta Live Tables (DLT)

Unity Catalog

Apache Spark

Kafka/Confluent

AWS

S3

Job description

Title: Big Data Engineer: Various levels (Lead, Senior, Intermediate)

Location: Atlanta, GA 30308 (Hybrid: 2x/Week)

Duration: 6+ Months, Long-term. Renewable.

Description

Big Data Engineers own the architecture, design, and delivery of scalable analytics and streaming solutions that transform enterprise data into trustworthy, governed datasets for BI and advanced analytics.

Data Engineers participate across the full data engineering lifecycle—from gathering and clarifying requirements with business partners and data modelers, to shaping lakehouse and pipeline designs, and building and operating production ingestion and transformation pipelines using Databricks, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Apache Spark (Spark SQL/PySpark), Kafka (including managed patterns such as Confluent Cloud), AWS services (such as S3 and supporting integration patterns), and SQL.

The role requires strong engineering practices focused on data quality, observability, scalability, performance, and maintainability. Success depends on advanced hands‑on proficiency with these technologies and the ability to communicate clearly in both written and verbal formats, partnering across the organization to translate business questions and data needs into durable requirements and measurable outcomes.

Client Corporation is seeking a Senior Data Engineer who enjoys collaborating across the organization to deliver reliable, scalable data solutions. Join a skilled team focused on modern data platforms, where you will help shape and operate our Databricks-based lakehouse and streaming analytics capabilities.

This role blends strong business and analytical judgment with hands‑on engineering. You will partner with stakeholders to understand problems, translate them into clear requirements, design pragmatic architectures, and deliver production‑grade ingestion and transformation pipelines—with ownership through deployment and steady‑state operations.

You will work with teams across the company to clarify data and BI needs, perform structured requirements discovery alongside Data Modelers, align with BI development on scope and estimates, and help guide delivery through completion while promoting engineering quality and consistency.

Responsibilities

• Define data requirements; discover, integrate, and wrangle large volumes of structured and semi‑structured/unstructured data; and validate outcomes using appropriate tooling in our data environment.
• Support standardized datasets and ad hoc analysis needs; build mechanisms to ingest, validate, normalize, cleanse, and enrich data for downstream consumption.
• Contribute to data policies and technical controls where needed, including access patterns, retention considerations, and synthesizing/anonymizing sensitive attributes in line with enterprise standards.
• Implement rigorous data quality approaches for new and evolving sources; iterate with analytics partners to improve sourcing, preparation, and trust in datasets used for insights and modeling.
• Champion data engineering best practices, including testing, observability, and operational readiness, and contribute practical guidance on analytics preparation and visualization‑friendly dataset design where applicable.
• Partner closely with Data Science and Business Intelligence teams to deliver data models and pipelines that support reporting, research, and machine learning workflows.
• Build pipelines that clean, transform, aggregate, and publish data from disparate systems into curated layers suitable for BI and advanced analytics.
• Use Databricks, SQL, Spark, scripting, and AWS services to integrate systems reliably and efficiently.
• Apply solid knowledge of data architecture principles and help lead initiatives from requirements through implementation, balancing correctness, performance, cost, and maintainability.

Qualifications

Required

• 5+ years of professional experience in data engineering, or equivalent depth, including manipulating, processing, and extracting value from large datasets in production settings.
• Databricks (Must Have): Hands‑on delivery on the Databricks platform, including developing and operating Databricks Jobs/Workflows, notebooks, and production pipelines using Apache Spark (Spark SQL and/or PySpark) on Databricks.
• Delta Lake on Databricks (Must Have): Experience building and maintaining Delta Lake tables, including incremental processing, merges/upserts, schema evolution, and performance patterns aligned with Delta best practices.
• Delta Live Tables / DLT (Must Have): Experience designing, implementing, and operating DLT pipelines using Python or SQL, including pipeline dependencies, incremental processing, and operational monitoring as supported by DLT.
• Databricks Governance (Must Have): Practical experience with Unity Catalog or equivalent Databricks governance patterns strongly aligned to Unity Catalog, including secured sharing concepts and catalog/table governance as implemented in the environment.
• 4+ years of hands‑on Apache Spark engineering for analytics workloads, including batch and/or streaming, with the ability to implement reliable transformations and troubleshoot performance issues.
• 3+ years of experience working with Apache Kafka or managed equivalents such as Confluent Kafka for high‑volume event/stream processing, including operational considerations such as throughput, lag, scaling, and replay scenarios.
• 3+ years of experience building and operating AWS capabilities supporting analytics platforms, including services such as S3, IAM‑aligned access patterns, and integration with compute/storage patterns common to lakehouse architectures. Familiarity with services such as AWS Glue, Lambda, and MSK is applicable.
• Strong SQL: Ability to write and optimize intermediate‑to‑advanced queries and translate business logic into reliable datasets.
• Demonstrated experience delivering ETL/ELT pipelines in a Databricks lakehouse environment, with comfort working with incremental loads, schema evolution, and data quality checks.
• Experience working in Agile delivery environments, including Scrum, Kanban, SAFe, or similar methodologies, with experience in backlog refinement, iterative delivery, and dependency coordination.

Preferred

• Experience with Snowflake or other large‑scale analytical databases, helpful for hybrid/platform comparisons.
• NoSQL exposure, including Cassandra or similar technologies, where operational systems feed analytics platforms.
• Experience with enterprise messaging/integration ecosystems such as TIBCO EMS, IBM MQ, or similar technologies, in addition to Kafka, helpful when bridging operational feeds into analytics landing zones.

Education

Bachelor’s degree preferred in Information Systems, Computer Science, Computer Information Systems, or a related field, or equivalent practical experience.

Senior Data Engineer responsible for designing, building, and operating scalable Databricks-based lakehouse and streaming data solutions for rail transportation, including ingestion, transformation, data quality, governance, and analytics‑ready datasets. Requires 5+ years of data engineering experience, hands‑on Databricks, Delta Lake, Delta Live Tables (DLT), Unity Catalog, Apache Spark (Spark SQL/PySpark), SQL, Kafka/Confluent Kafka, and AWS experience; 4+ years of Spark, 3+ years each of Kafka and AWS, and strong ETL/ELT expertise with incremental processing, merges/upserts, schema evolution, performance optimization, testing, and observability. Responsibilities include requirements discovery, large‑scale structured/semi‑structured/unstructured data integration, pipeline development, data cleansing/enrichment, governance and access controls, and partnering with Data Science, BI, and Data Modeling teams. Experience with Agile/Scrum/Kanban/SAFe is required. Preferred qualifications include Snowflake, NoSQL/Cassandra, and enterprise messaging such as TIBCO EMS or IBM MQ. Bachelor’s degree in Information Systems, Computer Science, Computer Information Systems, or a related field is preferred, or equivalent practical experience.