About the Role
Shamrock Trading Corporation is looking for a Data Engineer who wants to utilize their expertise in modern data platforms, microservices architectures, application programming interfaces, data pipeline development/support and analytical reporting skills by joining our Data Services team. This role is responsible for gathering and analyzing data from several internal and external sources, designing a cloud-focused data platform for analytics and business intelligence, reliably providing data to our analysts.
This role requires a significant understanding of data engineering and analytical techniques. An ideal candidate will have strong technical capabilities, business acumen, and the ability to work effectively with cross-functional teams
What You’ll Do
- Develop & Maintain Scalable Data Pipelines: Build, optimize, and maintain ETL/ELT pipelines using Databricks, Apache Spark, and Delta Lake.
- Optimize Data Processing: Implement performance tuning techniques to improve Spark-based workloads.
- Cloud Data Engineering: Work with AWS services (S3, ECS, Lambda, Glue, etc.) to design and implement robust data architectures.
- Real-time & Streaming Data: Develop streaming solutions using Kafka and Databricks Structured Streaming.
- Data Quality & Governance: Implement data validation, observability, and governance best practices using Unity Catalog or other tools.
- Cross-functional Collaboration: Partner with analysts, data scientists, and application engineers to ensure data meets business needs.
- Automation & CI/CD: Implement infrastructure-as-code (IaC) and CI/CD best practices for data pipelines using tools like Terraform, and GitHub Actions.
What You’ll Bring
- Bachelor’s degree in computer science, data science or related technical field, or equivalent practical experience
- 2-4 years of experience in data engineering, with a focus on cloud-based platforms.
- Strong hands-on experience with Databricks (including Spark, Delta Lake, and MLflow).
- Experience building and maintaining AWS based data pipelines: currently utilizing AWS Lambda, Docker / ECS, MSK, Airflow, Databricks, Unity Catalog
- Development experience utilizing one or more of the following:
- Git ecosystem (GitHub, GitLab, etc.)
- Python
- Databricks (PySpark, SQL, Spark Declarative Pipelines/DLT, Databricks Asset Bundles)
- Kafka and the Kafka Connect ecosystem (Schema Registry,Avro, etc.)
- Familiarity with CI/CD for data pipelines and infrastructure as code (GitHub Actions, Terraform)
- Strong SQL skills for data transformation and performance tuning.
- Understanding of data security and governance best practices.
- Enthusiasm for working directly with customer teams (Business units and internal IT)
What Will Set You Apart
- Proven experience with relational and NoSQL databases (e.g. Postgres, Microsoft SQL Server, MongoDB, etc.)
- Familiarity with data lakehouse architectures and optimization strategies.
- Familiarity with API architectures such as FastAPI, AWS API Gateways, and OpenAPI Specifications
- Familiarity with data visualization techniques using tools such as Databricks Dashboards, Grafana, PowerBI, and Excel.