Data Engineer

Optomi; Austin (TX); On-site; USD 100,000 - 130,000; Full time

Job summary

A leading technology consulting firm in Austin, Texas, is seeking an experienced Data Engineer to design scalable data pipelines and develop ETL workflows optimized for performance. The role requires over 6 years of experience in data engineering, proficiency in Python, Scala, or Java, and hands-on experience with tools like Spark and Kafka. Candidates should have a strong understanding of data modeling and cloud platforms such as AWS, Azure, or GCP. This position offers the opportunity to work on advanced use cases, focusing on AI-driven automation and compliance.

Qualifications

• 6+ years of experience in data engineering for analytics or ML systems.
• Hands-on experience with Spark, Kafka, and Airflow (or similar).
• Strong understanding of data modeling and lakehouse architectures like Iceberg.
• Experience with Snowflake, Databricks, and OLAP/NRT systems.

Responsibilities

• Design and implement scalable batch and near-real-time data pipelines.
• Develop ETL/ELT workflows optimized for performance and cost.
• Ensure data integrity, governance, privacy, and compliance.

Skills

Data Engineering

Python

Scala

Java

Spark

Kafka

Airflow

Data Modeling

AWS

Azure

GCP

Snowflake

Databricks

Tableau

Docker

Kubernetes

MLOps

Tools

CI/CD

GitLab

Grafana

Spinnaker

Datadog

Job description

• Design and implement scalable batch and near-real-time data pipelines.
• Develop ETL/ELT workflows optimized for performance and cost.
• Implement dimensional data models and standardize business metrics.
• Instrument APIs and user journeys to capture behavioral and transactional data.

Data Governance & Quality

• Ensure data integrity , governance, privacy , and compliance.
• Maintain reliability and availability of mission-critical systems.

ML & Advanced Use Cases

• Enable RAG-based data preparation and AI-driven automation.

Required Qualifications

• 6+ years of experience in data engineering for analytics or ML systems.
• Experience in Python, Scala, or Java.
• Hands-on experience with Spark, Kafka, and Airflow (or similar).
• Strong understanding of data modeling and lakehouse architectures (e.g., Iceberg).
• Experience with AWS, Azure, or GCP .
• Experience with Snowflake, Databricks, Trino, OLAP/NRT systems, Superset or Tableau.
• Familiarity with CI/CD, data observability , infrastructure-as-code.
• Exposure to MLOps and GenAI/RAG pipelines.
• Hands-on experience with LLMs (prompt engineering, fine-tuning, RAG).
• Experience in FinTech, Wallet, or Payments domain.
• Nice-to-haves: Docker, Kubernetes, Splunk, Grafana, Scala, GitLab, Spinnaker, Datadog, Rust, GO, or MLOps/GenAI experience.