DataOps Engineer

SBT Global, Inc.; Englewood Cliffs (NJ); On-site; USD 140,000 - 220,000; Full time

Job summary

SBT Global, Inc. is seeking an experienced Data Platform Engineer to sustain Iceberg-based tables, manage schema changes and catalog syncing, and build robust data pipelines.

You will create Docker images, test pipelines with Docker‑Compose, and drive security scans. You will automate CI/CD and implement observability with Prometheus, Grafana, and OpenTelemetry, while supporting on-call incidents and compliance efforts.

Qualifications

• Bachelor’s degree in Computer Science, IT, Data Engineering, or related field (Master’s a plus).
• ~5 years of hands-on experience building and operating large-scale data platforms (lake-house, data-warehouse, or big-data ecosystems).
• Proven production experience with Apache Iceberg (table creation, partition management, schema evolution, catalog integration).
• Strong Docker skills: multi-stage builds, Docker-Compose testing, routine image security scanning.
• Experience with Spark, Flink, or Presto/Trino and its connection to Iceberg tables.
• Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
• Experience building CI/CD pipelines that include Docker linting, vulnerability scanning, and automated deployment of data-pipeline code.
• Familiarity with observability tooling (Prometheus, Grafana, OpenTelemetry, Loki) and ability to create useful alerts and dashboards.
• Ability to respond to incidents, write clear root‑cause analysis reports, and contribute to post‑mortem actions.
• Willingness to participate in an on‑call rotation as a first‑line responder.
• Availability to work on-site in New Jersey for the initial assignment and relocate to Dallas by October 2026.

Responsibilities

• Iceberg operations: support tables, manage schema changes, partitions, and catalog sync.
• Docker image creation & testing: write multi‑stage Dockerfiles, run local test environments with Docker‑Compose, and conduct vulnerability scans.
• Data pipeline development: build ETL/ELT jobs that ingest raw data and write to Iceberg tables; add simple streaming components using Kafka, Pulsar, or Kinesis when needed.
• CI/CD automation: configure pipelines (GitHub Actions, GitLab CI, Azure DevOps, …) to lint Dockerfiles, scan images, version Iceberg metadata, and deploy pipelines without downtime.
• Automation with Ansible/Python: script cluster provisioning, catalog configuration, vacuum/compaction, and other routine housekeeping tasks.
• Observability: instrument services with OpenTelemetry, Prometheus, Grafana, and Loki; create dashboards showing pipeline latency, resource usage, table health, and error rates; set up basic alerts.
• SLA monitoring: measure data freshness, job success rates, and query response times against agreed‑upon targets and report deviations.
• Incident response: join the on‑call rotation, perform first‑line diagnosis and resolution of pipeline failures, Iceberg metadata issues, or container crashes; write concise root‑cause analyses and suggest improvements.
• Security & compliance support: help enforce image signing, mTLS, IAM roles, and bucket policies; collaborate with the security team to meet GDPR, HIPAA, or ISO 27001 requirements.
• Knowledge sharing: keep internal documentation up to date and run short tech demos or brown‑bag sessions on Iceberg, Docker best practices, and automation techniques.

Skills

Python

Ansible

Iceberg knowledge

Spark/Flink/Presto

CI/CD automation

On-call readiness

Education

Bachelor’s degree in Computer Science/IT/Data Engineering

Master’s degree is a plus

Tools

Docker

Docker Compose

Trivy

Snyk

GitHub Actions

GitLab CI

Azure DevOps

OpenTelemetry

Prometheus

Grafana

Loki

Kafka

Pulsar

Kinesis

Iceberg

Job description

Job Description

• Iceberg operations: support tables, manage schema changes, partitions, snapshot retention, and keep the catalog (Hive Metastore, AWS Glue, Nessie, …) synchronized.
• Docker image creation & testing: write multi‑stage Dockerfiles for Spark/Flink/Presto, run local test environments with Docker‑Compose, and conduct vulnerability scans (Trivy, Snyk, …).
• Data pipeline development: build ETL/ELT jobs that ingest raw data and write to Iceberg tables; add simple streaming components using Kafka, Pulsar, or Kinesis when needed.
• CI/CD automation: configure pipelines (GitHub Actions, GitLab CI, Azure DevOps, …) to lint Dockerfiles, scan images, version Iceberg metadata, and deploy pipelines without downtime.
• Automation with Ansible/Python: script cluster provisioning, catalog configuration, vacuum/compaction, and other routine housekeeping tasks.
• Observability: instrument services with OpenTelemetry, Prometheus, Grafana, and Loki; create dashboards showing pipeline latency, resource usage, table health, and error rates; set up basic alerts.
• SLA monitoring: measure data freshness, job success rates, and query response times against agreed‑upon targets and report deviations.
• Incident response: join the on‑call rotation, perform first‑line diagnosis and resolution of pipeline failures, Iceberg metadata issues, or container crashes; write concise root‑cause analyses and suggest improvements.
• Security & compliance support: help enforce image signing, mTLS, IAM roles, and bucket policies; collaborate with the security team to meet GDPR, HIPAA, or ISO 27001 requirements.
• Knowledge sharing: keep internal documentation up to date and run short tech demos or brown‑bag sessions on Iceberg, Docker best practices, and automation techniques.

Requirements

• Bachelor’s degree in Computer Science, IT, Data Engineering, or a related field (Master’s a plus).
• ~5 years of hands‑on experience building and operating large‑scale data platforms (lake‑house, data‑warehouse, or big‑data ecosystems).
• Proven production experience with Apache Iceberg (table creation, partition management, schema evolution, catalog integration).
• Strong Docker skills: multi‑stage builds, Docker‑Compose testing, routine image security scanning.
• Experience with at least one major data‑processing engine (Spark, Flink, or Presto/Trino) and its connection to Iceberg tables.
• Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
• Experience building CI/CD pipelines that include Docker linting, vulnerability scanning, and automated deployment of data‑pipeline code.
• Familiarity with observability tooling (Prometheus, Grafana, OpenTelemetry, Loki) and ability to create useful alerts and dashboards.
• Ability to respond to incidents, write clear root‑cause analysis reports, and contribute to post‑mortem actions.
• Willingness to participate in an on‑call rotation as a first‑line responder.
• Availability to work on‑site in New Jersey for the initial assignment and relocate to Dallas by October 2026.

Preferred Qualifications

• Experience with cloud‑native data services on AWS, Azure, or GCP (EMR, Dataproc, Synapse, etc.).
• Familiarity with other lake‑house formats such as Delta Lake or Apache Hudi and ability to evaluate trade‑offs against Iceberg.
• Knowledge of streaming platforms (Kafka, Pulsar, Kinesis) and real‑time processing patterns.
• Relevant certifications (Databricks Lakehouse Associate, Google Professional Data Engineer, AWS Certified Data Analytics – Specialty, etc.).
• Background supporting data platforms in regulated industries (pharma, finance, healthcare) and understanding of associated compliance frameworks.

Additional Information

All your information will be kept confidential according to EEO guidelines.