Data Engineer – GCP, AWS, Databricks

Jobtailor; Deutschland; Remote; EUR 90.000 - 130.000; Vollzeit

Zusammenfassung

Jobtailor in Germany is seeking an experienced Data Engineer to design, develop, and maintain scalable ETL/ELT data pipelines across cloud environments. You will work with GCP and AWS, leveraging Databricks, Apache Spark, and PySpark for large-scale processing.

Responsibilities include integrating diverse data sources, optimizing batch and real-time workflows, and ensuring data quality, security, and governance. Collaboration with data architects, analysts, and business stakeholders is essential.

Qualifikationen

• 6+ years experience in data engineering.
• 5+ years of professional data engineering experience.
• Strong hands-on experience with both GCP and AWS.
• Expertise in Databricks, Spark, and PySpark.
• Strong programming skills in Python and SQL.
• Proven experience developing ETL/ELT pipelines and cloud data platforms.
• Experience with data warehouses, data lakes, and dimensional modeling.
• Experience with orchestration tools such as Apache Airflow or Google Cloud Composer.
• Understanding of data security, governance, monitoring, and quality frameworks.
• Strong analytical, troubleshooting, and problem-solving skills.
• Excellent communication and cross-functional collaboration skills.

Aufgaben

• Design, build, and maintain scalable ETL/ELT data pipelines.
• Develop cloud data solutions on GCP and AWS.
• Use Databricks, Spark, and PySpark for large-scale processing.
• Integrate structured, semi-structured, and unstructured data from multiple sources.
• Develop and optimize batch and real-time data-processing workflows.
• Improve pipeline performance, reliability, and cost efficiency.
• Implement data-quality checks, monitoring, security, and governance standards.
• Design and support cloud data warehouses and data lakes.
• Troubleshoot production issues and perform root-cause analysis.
• Collaborate with data architects, analysts, application teams, and business stakeholders.
• Create and maintain technical documentation for pipelines, data models, and workflows.

Kenntnisse

Python

SQL

ETL/ELT

Data Pipelines

Data Modeling

Airflow

Tools

Apache Airflow

Google Cloud Composer

BigQuery

Cloud Storage

Dataflow

Dataproc

Pub/Sub

S3

Glue

EMR

Jobbeschreibung

• Design, develop, and maintain scalable ETL/ELT data pipelines
• Build cloud-based data solutions using GCP and AWS services
• Use Databricks, Apache Spark, and PySpark for large-scale data processing
• Integrate structured, semi-structured, and unstructured data from multiple sources
• Develop and optimize batch and real-time data-processing workflows
• Improve pipeline performance, reliability, scalability, and cost efficiency
• Implement data-quality checks, monitoring, security, and governance standards
• Design and support cloud data warehouses and data lakes
• Troubleshoot production issues and perform root-cause analysis
• Collaborate with data architects, analysts, application teams, and business stakeholders
• Create and maintain technical documentation for pipelines, data models, and workflows

Requirements

• 6+ Years experience required in the position overview
• 5+ years of professional data engineering experience
• Strong hands-on experience with both GCP and AWS
• Expertise in Databricks, Apache Spark, and PySpark
• Strong programming skills in Python and SQL
• Proven experience developing ETL/ELT pipelines and cloud data platforms
• Experience with data warehouses, data lakes, and dimensional data modeling
• Experience with orchestration tools such as Apache Airflow or Google Cloud Composer
• Understanding of data security, governance, monitoring, and quality frameworks
• Strong analytical, troubleshooting, and problem-solving skills
• Excellent communication and cross-functional collaboration skills
• Preferred: experience with BigQuery, Cloud Storage, Dataflow, Dataproc, Pub/Sub, Cloud Composer, S3, Glue, EMR, Redshift, Lambda, Kinesis, Delta Lake, Databricks Lakehouse, Apache Kafka, CI/CD, Git, Terraform, cloud infrastructure automation, and Agile development environments

Core Competencies

Demonstrates expertise in designing and developing scalable ETL/ELT data pipelines using GCP and AWS, with strong proficiency in Databricks, Apache Spark, and PySpark. Capable of optimizing data workflows and ensuring data quality, security, and governance across cloud data solutions.

Highest-signal resume keywords

• ETL/ELT Pipeline Development
• GCP and AWS Expertise
• Databricks, Apache Spark, and PySpark
• Data Warehouse and Data Lake Design
• Data Security and Governance

Hard Skills

• Python
• SQL
• Data Engineering
• Data Modeling
• Batch and Real-Time Processing
• Data Quality Checks
• Root-Cause Analysis
• Orchestration Tools
• Cloud Data Solutions
• Performance Optimization

Soft Skills

• Analytical Skills
• Troubleshooting Skills
• Problem-Solving Skills
• Communication Skills
• Cross-Functional Collaboration

Industry Keywords

• Cloud Infrastructure Automation
• Agile Development
• Data Governance
• Data Security
• Data Monitoring

Tools & Technologies

• Apache Airflow
• Google Cloud Composer
• BigQuery
• Cloud Storage
• Dataflow
• Dataproc
• Pub/Sub
• S3
• Glue
• EMR