Cloud Data Engineer: Spark Pipelines & Real-Time Analytics

Shamrock Trading Corp.

Overland Park (KS)

On-site

USD 110,000 - 165,000

Full time

3 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Shamrock Trading Corporation is looking for a Data Engineer to design and build modern data platforms for analytics and business intelligence. You will gather data from internal and external sources, design cloud-focused pipelines, and ensure reliable data delivery to analysts.

The ideal candidate has strong data engineering skills on cloud platforms, experience with Databricks and AWS, and the ability to collaborate with cross-functional teams to turn business needs into scalable data solutions.

Qualifications

  • Bachelor’s degree in a related technical field or equivalent experience.
  • 2-4 years of data engineering experience on cloud platforms.
  • Hands-on Databricks experience (Spark, Delta Lake, MLflow).
  • Experience building and maintaining AWS-based data pipelines.

Responsibilities

  • Develop, optimize, and maintain ETL/ELT pipelines with Databricks, Spark, Delta Lake.
  • Tune Spark-based workloads for performance.
  • Design cloud data architectures using AWS services.
  • Create streaming data solutions with Kafka and Databricks Structured Streaming.
  • Implement data validation, observability, and governance with Unity Catalog or equivalents.
  • Collaborate with analysts, data scientists, and engineers to meet data needs.
  • Apply IaC and CI/CD for data pipelines using Terraform and GitHub Actions.

Skills

Python
SQL
Data Modeling
ETL/ELT
Cloud Data Platforms
Cross-functional Collaboration

Education

Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience

Tools

Databricks
Spark
Delta Lake
MLflow
AWS (S3, ECS, Lambda, Glue)
Kafka
Kafka Connect (Schema Registry, Avro)
Terraform
GitHub Actions
Unity Catalog

Job description

Shamrock Trading Corporation is looking for a Data Engineer to design and build modern data platforms for analytics and business intelligence. You will gather data from internal and external sources, design cloud-focused pipelines, and ensure reliable data delivery to analysts.

The ideal candidate has strong data engineering skills on cloud platforms, experience with Databricks and AWS, and the ability to collaborate with cross-functional teams to turn business needs into scalable data solutions.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

Shamrock Trading Corp. • Overland Park (KS)

On-site
USD 110,000 - 165,000
Cloud Data Engineer — Real-Time Pipelines & Spark
Cloud Data Engineer — Real-Time Pipelines & Spark

Jobo • United States

Remote
USD 90,000 - 130,000
Private medical care
Dental care
Life insurance
+2
Senior Data Engineer: Real-Time Pipelines & Cloud Analytics
Senior Data Engineer: Real-Time Pipelines & Cloud Analytics

Cloudndata • United States

Hybrid
USD 140,000 - 180,000
Cloud Data Engineer: Spark, Snowflake & Azure Pipelines
Cloud Data Engineer: Spark, Snowflake & Azure Pipelines

Quantum Technologies. LLC • San Jose (CA)

On-site
USD 120,000 - 180,000
Senior Data Engineer - Real-Time Pipelines (Spark/Kafka)
Senior Data Engineer - Real-Time Pipelines (Spark/Kafka)

UST • Sunnyvale (CA)

On-site
USD 72,000 - 108,000
Medical Insurance
401(k) with employer match
Paid time off
Cloud Data Engineer: AWS, Databricks & PySpark Pipelines
Cloud Data Engineer: AWS, Databricks & PySpark Pipelines

Advanced Tech Placement • Atlanta (GA)

On-site
USD 120,000 - 150,000
Senior Data Engineer: Real-Time Pipelines on AWS
Senior Data Engineer: Real-Time Pipelines on AWS

eOne Infotech • Fort Mill (SC)

On-site
USD 90,000 - 120,000
Senior Data Engineer: Real-Time Pipelines & Snowflake
Senior Data Engineer: Real-Time Pipelines & Snowflake

UMB Financial Corporation • Kansas City (MO)

On-site
USD 118,000 - 173,000
401(k) matching
Annual incentive pay
Health insurance
+2
Senior Cloud Data Engineer: Real-Time Pipelines
Senior Cloud Data Engineer: Real-Time Pipelines

Recru • Houston (TX)

On-site
USD 130,000 - 180,000
Senior Data Engineer - Scalable Pipelines & Spark Expert
Senior Data Engineer - Scalable Pipelines & Spark Expert

Ethereum Technologies LLC • Charlotte (NC)

On-site
USD 120,000 - 180,000