Senior Data Engineer

Peyton Resource Group; Houston (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

Peyton Resource Group in Houston, Texas is seeking a Senior Data Engineer to design and operate cloud-scale data platforms. You will lead the development of enterprise data solutions, ensuring quality, performance, and security. The ideal candidate has strong experience in data engineering, cloud data platforms, and excellent collaboration skills. This position requires a Bachelor’s degree and significant experience in leading data architecture at scale, alongside strong technical proficiencies in Python and SQL.

Qualifications

• 5–10+ years of experience in data engineering or platform roles delivering enterprise-grade systems.
• 2+ years leading or architecting cloud data platforms at scale.
• Hands-on experience with cloud-native data services, orchestration frameworks, and distributed processing.

Responsibilities

• Lead design, development, and operation of enterprise data platforms.
• Provide technical leadership and guidance to engineering teams.
• Coach engineers on standards, observability, and operational excellence.

Skills

Cloud data engineering

Data platform architecture

ELT/ETL and streaming pipelines

Python

SQL

Collaboration skills

Education

Bachelor’s degree in Computer Science or related field

Master’s degree

Tools

Airflow

Kafka

Azure Data Lake

AWS S3

Terraform

Job description

Position Overview

We are seeking a Senior Data Engineer to design, build, and operate cloud-scale data platforms and pipelines that support analytics, machine learning, and data products across the organization. This role collaborates with Product, Architecture, and Data Science teams to deliver secure, reliable, and cost-efficient batch and streaming data services. The ideal candidate brings deep technical expertise in cloud data engineering, data platform architecture, and best practices for governance, reliability, and observability.

Key Responsibilities

• Lead the design, development, and operation of enterprise data platforms, including lakehouse, streaming, and centralized data hubs, while managing dependencies and integrations with enterprise systems.
• Provide technical leadership and guidance to engineering teams, including planning, scoping, risk management, and delivery accountability.
• Coach engineers on standards, observability, operational excellence, and best practices; conduct design and code reviews.
• Define and evolve cloud data architectures for ingestion, storage, compute, cataloging, and access across real-time and batch use cases.
• Build and maintain robust ELT/ETL and streaming pipelines using orchestration frameworks (Airflow, Dagster), event platforms (Kafka, Kinesis, Event Hub), and distributed processing frameworks (Spark, Databricks, Flink, Beam).
• Implement lakehouse/lake patterns with medallion layering, CDC, and schema evolution; ensure quality gates, testing, and lineage.
• Establish and monitor service-level objectives for data freshness, availability, and quality; implement observability and runbooks for operations.
• Optimize storage and compute costs and performance; manage capacity, autoscaling, and governance.
• Ensure privacy, security, and compliance (IAM, tokenization, row/column security, data retention); work closely with enterprise security teams.
• Translate product and stakeholder requirements into data contracts and SLAs; document architecture and operational procedures.
• Evaluate emerging data technologies and provide proof-of-concepts and reference implementations.
• Anticipate future platform needs, including multi-cloud, IoT/edge ingestion, and AI/ML workloads.
• Modernize existing data architectures using current best practices: lakehouse, CDC, medallion layering, data mesh principles, and streaming pipelines.
• Develop software engineering solutions in Python/SQL and one additional language (Scala/Java); familiarity with REST/gRPC and microservices patterns.
• Utilize cloud data services (Azure Data Lake/Databricks/Synapse; AWS S3/EMR/Glue/Redshift; GCP BigQuery/Dataflow) and distributed stores (Delta/Iceberg/Hudi, Kafka, Redis).
• Work with SQL/NoSQL databases (Postgres, DynamoDB, MongoDB, Cassandra) and CI/CD pipelines, containerization (Docker/Kubernetes), and infrastructure as code (Terraform).
• Ensure proper data governance and security across catalogs, lineage, RBAC/ABAC, encryption, and key management.
• Communicate effectively with stakeholders, translating business needs into scalable, reliable data solutions.

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, or related field; Master’s preferred.
• 5–10+ years of experience in data engineering or platform roles delivering enterprise-grade systems (SaaS or large-scale internal platforms).
• 2+ years leading or architecting cloud data platforms at scale.
• Proven experience with batch and streaming pipelines, lakehouse implementations, and production ML data flows.
• Experience operating and supporting data platforms with SLAs/SLOs, cost governance, and on‑call responsibilities.
• Hands‑on experience with cloud‑native data services, orchestration frameworks, and distributed processing.

Preferred / Additional Experience

• Experience with real‑time data systems and IoT ingestion.
• Background in large‑scale SaaS or multi‑cloud platforms.
• Strong collaboration skills with cross‑functional teams, including product, architecture, and analytics.