IT Data Engineering JG3

United Global Technologies; Houston (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

A leading technology firm in Houston is seeking a deeply technical Data Engineer to enhance a large-scale cloud data platform. Responsibilities include diagnosing performance issues, optimizing architecture, and mentoring junior engineers. Candidates should possess deep knowledge of Databricks, strong Python expertise, and advanced SQL skills. This hands-on role requires the ability to analyze complex data pipelines and propose improvements. The ideal candidate thrives in a collaborative environment and is comfortable tackling ambiguous challenges.

Qualifications

• Strong technical depth and hands-on individual contributor ability.
• Ability to analyze pipelines end-to-end.
• Comfortable challenging Databricks as a product.

Responsibilities

• Diagnose performance, latency, and cost issues in cloud data platforms.
• Propose and prove better approaches via POCs.
• Collaborate with and mentor other data engineers.

Skills

Deep understanding of Databricks internals

Performance tuning and optimization

Strong Python

Advanced SQL

PySpark (advanced)

Tools

Qlik Replicate

Azure Data Lake Storage

Databricks

Job description

This is not a standard data engineer role.

Looking for a deeply technical, hands-on individual contributor who can:

• Diagnose performance, latency, and cost issues in a large-scale cloud data platform
• Take a top-down, platform-level view across multiple projects
• Improve architecture, efficiency, and cost optimization, not just write Spark code
• Act as a technical problem-solver and mentor, guiding other data engineers

This person is expected to make the platform better, not just execute tasks.

Current Platform & Architecture (Very Important)

Data Flow:

• On-premise systems → Cloud (Azure)
• Streaming ingestion → Azure Data Lake Storage (ADLS)

• Data processed into two separate containers:
  • Crude trading
  • Product trading

Technologies in Use:

• Qlik Replicate (formerly Attunity)
  • Streaming data from on-prem to Azure
• Azure Data Lake Storage (ADLS)
• Databricks
  • Delta Live Tables (DLT)
  • Spark / PySpark
• Python
• SQL (complex queries and procedures)
  • Complex queries
  • Stored procedures
  • Analytical logic

Key Challenges the Role Is Meant to Solve

1. Data Latency

• High-volume streaming data
• End-to-end latency issues that need root-cause analysis

2. Databricks / DLT Cost Spikes

• DLT costs are far higher than expected
• Known contributors:
  • Very high data volume (expected)
  • Inefficient lookup logic used to split data into separate containers
• The current solution works but is not optimal

This role exists because generic recommendations are not enough.

What we doNot Want

• Someone who has:
  • Only written Spark notebooks
  • Only followed architectural guidance
  • Only worked at a surface level
• Someone who:
  • Needs strict 9–5 boundaries
  • Avoids ambiguity or deep technical investigation
• Someone whose resume was “AI-polished” but not real

What WE DO WANT

Technical Depth

• Deep understanding of:
  • Databricks internals
  • Spark engine behavior
  • Performance tuning and optimization
• Ability to:
  • Analyze pipelines end-to-end
  • Identify architectural inefficiencies
  • Propose and prove better approaches via POCs
• Comfortable challenging Databricks as a product
  • Gather evidence
  • Support escalation discussions with Databricks engineers

Programming & Data Skills

• Strong Python (mandatory)
• PySpark (advanced, not basic)
• Advanced SQL
  • Complex queries
  • Stored procedures
  • Analytical logic

Working Style

• Hands-on individual contributor
• Collaborative with data engineers
• Willing to:
  • Review others’ solutions
  • Build POCs independently
  • Demonstrate better outcomes (performance, cost, scalability)

Role Scope

• Will work across multiple projects
• Acts as a cross-platform technical expert
• Evaluates:
  • Architecture
  • Cost drivers
  • Scalability
  • Reusability for future programs