Senior Data Engineer – Databricks | Spark (Entertainment Company

TechLink Resources, Inc; Glendale (CA); Sur place; USD 120 000 - 180 000; Plein temps

Résumé du poste

TechLink Resources, Inc. is seeking a Senior Data Engineer to design and maintain scalable data platforms on AWS, leveraging Databricks, Spark, Python, and SQL to support batch and streaming workloads.

You will own data pipelines, optimize performance, and collaborate with engineering teams to translate business requirements into scalable technical solutions while ensuring governance and reliability across cloud-based architectures.

Qualifications

• 5+ years of professional data engineering experience.
• Hands-on Databricks experience required.
• Strong Python/Scala programming.
• Advanced SQL and performance tuning.
• Experience with AWS data infrastructure.
• Familiarity with Docker, Kubernetes, and cloud-based data architectures.
• Experience with Databricks governance, Unity Catalog, and access management.

Responsabilités

• Design, develop, and optimize scalable data pipelines using PySpark, Scala, Python, and SQL.
• Build and maintain batch and streaming pipelines using Databricks and Apache Spark.
• Manage Databricks platform administration, governance, access controls, and Unity Catalog.
• Develop and maintain workflow orchestration using Apache Airflow (MWAA).
• Optimize Spark performance, troubleshoot pipeline failures, and improve platform reliability.
• Design and support cloud-based data architectures across AWS, Kubernetes, and Databricks.
• Implement automation, CI/CD processes, monitoring, and data quality solutions.
• Collaborate with engineering teams and stakeholders to translate business requirements into scalable technical solutions.

Outils

Databricks

Apache Spark

Python

SQL

Airflow (MWAA)

AWS

Docker

Kubernetes

Unity Catalog

Description du poste

We are seeking an experienced Senior Data Engineer with deep expertise in Databricks, Apache Spark, Python, SQL, and Apache Airflow to design, develop, and maintain scalable data platforms and pipelines.

In this role, you will play a key part in building and optimizing enterprise data solutions, supporting both batch and streaming pipelines, and managing Databricks platform operations within an AWS cloud environment.

Key Responsibilities

• Design, develop, and optimize scalable data pipelines using PySpark, Scala, Python, and SQL.
• Build and maintain batch and streaming pipelines using Databricks and Apache Spark.
• Manage Databricks platform administration, governance, access controls, and Unity Catalog.
• Develop and maintain workflow orchestration using Apache Airflow (MWAA).
• Optimize Spark performance, troubleshoot pipeline failures, and improve platform reliability.
• Design and support cloud-based data architectures across AWS, Kubernetes, and Databricks.
• Implement automation, CI/CD processes, monitoring, and data quality solutions.
• Collaborate with engineering teams and stakeholders to translate business requirements into scalable technical solutions.

Required Qualifications

• 5+ years of professional data engineering experience.
• Strong hands-on experience with Databricks (primary requirement).
• Advanced knowledge of Apache Spark / PySpark and Spark architecture.
• Strong programming experience with Python and Scala.
• Advanced SQL skills, including query optimization and performance tuning.
• Production experience with Apache Airflow for workflow orchestration.
• Experience designing, developing, and maintaining ETL/ELT pipelines.
• Experience with AWS cloud services and data infrastructure.
• Familiarity with Docker, Kubernetes, and cloud-based data architectures.
• Experience with Databricks governance, Unity Catalog, and access management.

Preferred Qualifications

• Experience with Snowflake or similar cloud data platforms.
• Familiarity with Databricks APIs, SDKs, and CLI tools.
• Experience with CI/CD, DevOps, and Git-based workflows.
• Knowledge of data governance, lineage, monitoring, and platform security.

If you have strong hands-on Databricks experience and enjoy building scalable, high-performance data solutions, this job is for you!