Senior PySpark Data Engineer - Cloud & ETL Specialist

Highbrow Technology Inc; Dallas (TX); Vor Ort; USD 140.000 - 200.000; Vollzeit

Zusammenfassung

Highbrow Technology Inc. in Dallas, TX seeks an experienced Data Engineer with strong PySpark to design and maintain scalable data processing solutions. The role focuses on building end-to-end data pipelines and optimizing Spark workloads within a cloud-first environment.

You will collaborate with data scientists and analytics teams, work with Delta Lake, Hive, and Databricks, and implement robust ETL/ELT processes using Airflow and CI/CD practices.

Qualifikationen

• 8+ years of data engineering experience.
• Strong Python and PySpark hands-on experience.
• Expertise in Apache Spark, Spark SQL, and distributed processing.
• Proficient in SQL with complex queries and performance tuning.
• Experience with ETL/ELT pipelines, data warehousing, and data lake architecture.
• Experience with cloud data platforms (AWS, Azure, or GCP).
• Familiarity with Databricks, Delta Lake, Hive or similar big data tech.
• Knowledge of workflow orchestration tools like Airflow, Data Factory, or Glue.

Aufgaben

• Design, develop, and maintain scalable data processing solutions.
• Optimize Spark workloads and build robust data pipelines.
• Collaborate with cross-functional teams to deliver reliable data products.
• Work with cloud data platforms and big data toolchains.

Kenntnisse

PySpark

Python

Spark SQL

SQL

ETL pipelines

Data warehousing

Cloud platforms

Databricks

Delta Lake

Airflow

Git

CI/CD

Agile

Tools

Hadoop ecosystem

Jobbeschreibung

Highbrow Technology Inc. in Dallas, TX seeks an experienced Data Engineer with strong PySpark to design and maintain scalable data processing solutions. The role focuses on building end-to-end data pipelines and optimizing Spark workloads within a cloud-first environment.

You will collaborate with data scientists and analytics teams, work with Delta Lake, Hive, and Databricks, and implement robust ETL/ELT processes using Airflow and CI/CD practices.