Data Engineer - Scalable Pipelines & AI-Driven Analytics

Optomi; Austin (TX); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading technology consulting firm in Austin, Texas, is seeking an experienced Data Engineer to design scalable data pipelines and develop ETL workflows optimized for performance. The role requires over 6 years of experience in data engineering, proficiency in Python, Scala, or Java, and hands-on experience with tools like Spark and Kafka. Candidates should have a strong understanding of data modeling and cloud platforms such as AWS, Azure, or GCP. This position offers the opportunity to work on advanced use cases, focusing on AI-driven automation and compliance.

Qualifications

• 6+ years of experience in data engineering for analytics or ML systems.
• Hands-on experience with Spark, Kafka, and Airflow (or similar).
• Strong understanding of data modeling and lakehouse architectures like Iceberg.
• Experience with Snowflake, Databricks, and OLAP/NRT systems.

Responsibilities

• Design and implement scalable batch and near-real-time data pipelines.
• Develop ETL/ELT workflows optimized for performance and cost.
• Ensure data integrity, governance, privacy, and compliance.

Skills

Data Engineering

Python

Scala

Java

Spark

Kafka

Airflow

Data Modeling

AWS

Azure

GCP

Snowflake

Databricks

Tableau

Docker

Kubernetes

MLOps

Tools

CI/CD

GitLab

Grafana

Spinnaker

Datadog

Job description

A leading technology consulting firm in Austin, Texas, is seeking an experienced Data Engineer to design scalable data pipelines and develop ETL workflows optimized for performance. The role requires over 6 years of experience in data engineering, proficiency in Python, Scala, or Java, and hands-on experience with tools like Spark and Kafka. Candidates should have a strong understanding of data modeling and cloud platforms such as AWS, Azure, or GCP. This position offers the opportunity to work on advanced use cases, focusing on AI-driven automation and compliance.