Data Engineer: Batch & Streaming Analytics

GM Financial; Irving (TX); On-site; USD 120,000 - 160,000; Full time

Benefits offered by this job

401K matching

Bonding leave for new parents (12+ wks

Tuition assistance

Training

GM employee auto discount

Community service pay

Nine company holidays

Job summary

GM Financial is expanding its data engineering team to design and deploy scalable data processing pipelines in cloud environments. The role focuses on batch and streaming transformations to support data science and analytics, while collaborating with scientists, architects, and IT partners to deliver features and datasets for ML training and model execution.

Candidates should have hands-on experience with Hadoop, Spark, Kafka and cloud technologies, plus strong SQL and Python skills.

Qualifications

• Experience with processing large data sets using Hadoop, HDFS, Spark, Kafka, Flume or similar distributed systems.
• Experience with ingesting various source data formats such as JSON, Parquet, SequenceFile, Cloud Databases, MQ, Relational Databases such as Oracle.
• Experience with Cloud technologies (Azure, AWS, GCP) and native toolsets such as Azure ARM Templates, Hashicorp Terraform, AWS Cloud Formation.
• Understanding of cloud computing technologies, business drivers and emerging computing trends.
• Thorough understanding of Hybrid Cloud Computing: virtualization technologies, IaaS, PaaS and SaaS, and current landscape.
• Knowledge of Object Storage technologies (Data Lake Storage Gen2, S3, Minio, Ceph, ADLS).
• Experience with containerization (Docker, Kubernetes, Spark on Kubernetes, Spark Operator).
• Working knowledge of Agile development / SAFe, Scrum, ALM.
• Strong background with source control (GIT/Subversion); Build Systems (Maven/Gradle/Webpack); Code Quality (Sonar); Artifact Repos (Artifactory); CI/CD (Azure DevOps).
• Experience with NoSQL stores such as CosmosDB, MongoDB, Cassandra, Redis, Riak or NoSQL search like MarkLogic/Lily.

Responsibilities

• Code, test, deploy, orchestrate, monitor, document and troubleshoot cloud-based data engineering processing and automation per best practices and security standards.
• Collaborate with data scientists, data architects, ETL developers, and business partners to extract features from data sources.
• Evaluate, research, and experiment with batch and streaming data engineering technologies and assess business impact.
• Showcase capabilities of emerging technologies and enable adoption across teams.
• Contribute to defining and refining data engineering processes and procedures.
• Educate ETL developers on cloud-based initiatives to enable transition to data engineering.

Skills

Distributed data processing

Data ingestion

Python

SQL

Spark

Hadoop

Kafka

Cloud computing

REST APIs

Education

Bachelor’s Degree in related field

Tools

Hadoop

Spark

Kafka

Terraform

Azure

AWS

GCP

Docker

Kubernetes

Azure DevOps

Job description

GM Financial is expanding its data engineering team to design and deploy scalable data processing pipelines in cloud environments. The role focuses on batch and streaming transformations to support data science and analytics, while collaborating with scientists, architects, and IT partners to deliver features and datasets for ML training and model execution.

Candidates should have hands-on experience with Hadoop, Spark, Kafka and cloud technologies, plus strong SQL and Python skills.