Data Engineer

Shamrock Trading Corporation

Overland Park (KS)

On-site

USD 110,000 - 165,000

Full time

2 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Shamrock Trading Corporation is looking for a Data Engineer to design and build modern data platforms for analytics and business intelligence. You will gather data from internal and external sources, design cloud-focused pipelines, and ensure reliable data delivery to analysts.

The ideal candidate has strong data engineering skills on cloud platforms, experience with Databricks and AWS, and the ability to collaborate with cross-functional teams to turn business needs into scalable data solutions.

Qualifications

  • Bachelor’s degree in a related technical field or equivalent experience.
  • 2-4 years of data engineering experience on cloud platforms.
  • Hands-on Databricks experience (Spark, Delta Lake, MLflow).
  • Experience building and maintaining AWS-based data pipelines.

Responsibilities

  • Develop, optimize, and maintain ETL/ELT pipelines with Databricks, Spark, Delta Lake.
  • Tune Spark-based workloads for performance.
  • Design cloud data architectures using AWS services.
  • Create streaming data solutions with Kafka and Databricks Structured Streaming.
  • Implement data validation, observability, and governance with Unity Catalog or equivalents.
  • Collaborate with analysts, data scientists, and engineers to meet data needs.
  • Apply IaC and CI/CD for data pipelines using Terraform and GitHub Actions.

Skills

Python
SQL
Data Modeling
ETL/ELT
Cloud Data Platforms
Cross-functional Collaboration

Education

Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience

Tools

Databricks
Spark
Delta Lake
MLflow
AWS (S3, ECS, Lambda, Glue)
Kafka
Kafka Connect (Schema Registry, Avro)
Terraform
GitHub Actions
Unity Catalog

Job description

About the Role

Shamrock Trading Corporation is looking for a Data Engineer who wants to utilize their expertise in modern data platforms, microservices architectures, application programming interfaces, data pipeline development/support and analytical reporting skills by joining our Data Services team. This role is responsible for gathering and analyzing data from several internal and external sources, designing a cloud-focused data platform for analytics and business intelligence, reliably providing data to our analysts.

This role requires a significant understanding of data engineering and analytical techniques. An ideal candidate will have strong technical capabilities, business acumen, and the ability to work effectively with cross-functional teams

What You’ll Do
  • Develop & Maintain Scalable Data Pipelines: Build, optimize, and maintain ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
  • Optimize Data Processing: Implement performance tuning techniques to improve Spark-based workloads.
  • Cloud Data Engineering: Work with AWS services (S3, ECS, Lambda, Glue, etc.) to design and implement robust data architectures.
  • Real-time & Streaming Data: Develop streaming solutions using Kafka and Databricks Structured Streaming.
  • Data Quality & Governance: Implement data validation, observability, and governance best practices using Unity Catalog or other tools.
  • Cross-functional Collaboration: Partner with analysts, data scientists, and application engineers to ensure data meets business needs.
  • Automation & CI/CD: Implement infrastructure-as-code (IaC) and CI/CD best practices for data pipelines using tools like Terraform, and GitHub Actions.
What You’ll Bring
  • Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience
  • 2-4 years of experience in data engineering, with a focus on cloud-based platforms.
  • Strong hands-on experience with Databricks (including Spark, Delta Lake, and MLflow).
  • Experience building and maintaining AWS based data pipelines: currently utilizing AWS Lambda, Docker / ECS, MSK, Airflow, Databricks, Unity Catalog
  • Development experience utilizing one or more of the following:
  • Git ecosystem (GitHub, GitLab, etc.)
  • Python
  • Databricks (PySpark, SQL, Spark Declarative Pipelines/DLT, Databricks Asset Bundles)
  • Kafka and the Kafka Connect ecosystem (Schema Registry,Avro, etc.)
  • Familiarity with CI/CD for data pipelines and infrastructure as code (GitHub Actions, Terraform)
  • Strong SQL skills for data transformation and performance tuning.
  • Understanding of data security and governance best practices.
  • Enthusiasm for working directly with customer teams (Business units and internal IT)
What Will Set You Apart
  • Proven experience with relational and NoSQL databases (e.g. Postgres, Microsoft SQL Server, MongoDB, etc.)
  • Familiarity with data lakehouse architectures and optimization strategies.
  • Familiarity with API architectures such as FastAPI, AWS API Gateways, and OpenAPI Specifications
  • Familiarity with data visualization techniques using tools such as Databricks Dashboards, Grafana, PowerBI, and Excel.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Engineer
Data Engineer

Advanced Tech Placement • Atlanta (GA)

On-site
USD 120,000 - 150,000
Data Engineer
Data Engineer

The Judge Group • New York (NY)

On-site
USD 150,000 - 190,000
Data Engineer
Data Engineer

The Phoenix Group • United States

On-site
USD 90,000 - 120,000
Data Engineer
Data Engineer

GSquared Group • Atlanta (GA)

On-site
USD 120,000 - 160,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Alephys • Houston (TX)

On-site
USD 120,000 - 180,000
Data Engineer
Data Engineer

Evlo AI • New York (NY)

On-site
USD 130,000 - 170,000
Data Engineer
Data Engineer

Compunnel Inc. • San Francisco (CA)

On-site
USD 100,000 - 130,000
Data Engineer
Data Engineer

nimblegravity • United States

Remote
USD 120,000 - 170,000
Senior Databricks Data Engineer
Senior Databricks Data Engineer

Alephys LLC • United States

On-site
USD 120,000 - 190,000
Data Engineer
Data Engineer

Compunnel Inc. • Columbus (OH)

On-site
USD 85,000 - 115,000