Senior Data Engineer

Jobtailor; Santa Clara (CA); On-site; USD 150,000 - 230,000; Full time

Job summary

Jobtailor is seeking a senior data/analytics engineer to design and maintain scalable data pipelines using Apache Spark and PySpark. The role involves migrating on-prem Cloudera environments to cloud platforms like Snowflake or Databricks, and building Python microservices to serve data-driven features in production.

You will collaborate with data scientists, analysts, and engineers to support analytics workflows, data quality, and secure access control in cloud-native environments.

Qualifications

• 5+ years of professional experience in software or data engineering.
• Strong Python skills for building scalable data pipelines.
• Hands-on experience migrating data platforms (Cloudera to Snowflake/Databricks).
• Deep understanding of distributed systems and data architectures.

Responsibilities

• Design, develop, and maintain scalable data pipelines and workflows with Spark and PySpark.
• Build microservices in Python to serve data-driven features in production.
• Develop internal tools to support CI/CD, experiment tracking, and data versioning.
• Ingest and integrate large datasets from databases, files, and APIs.
• Ensure data quality through validation and monitoring; optimize performance.

Skills

Python

Spark/PySpark

Distributed Systems

Data Platform Migration

Cloud-native Platforms

Tools

Apache Beam

Databricks

Snowflake

Cloudera

Job description

• Design, develop, and maintain scalable data processing pipelines and workflows using frameworks such as Apache Spark, PySpark, and Apache Beam.
• Build and maintain microservices in Python that serve data-driven features in production.
• Develop internal tools to support CI/CD pipelines, experiment tracking, and data versioning.
• Collect, process, and integrate large datasets from multiple sources, including databases, file systems, and APIs.
• Ensure data integrity, consistency, and quality through robust validation and monitoring processes.
• Optimize data systems for performance, scalability, and high availability.
• Implement best practices for data security, access control, and privacy.
• Collaborate with data scientists, analysts, and engineers to support analytics and ML workflows.
• Lead complex migration initiatives involving the transition from on-premise Cloudera environments to cloud-native platforms, ensuring zero data loss and minimal downtime.

Requirements

• Strong understanding of distributed systems and modern data architectures.
• Proven, hands‑on experience with large-scale data platform migrations, specifically transitioning from Cloudera (CDH/HDP) to either Snowflake or Databricks.
• Deep technical expertise in building and orchestrating a high‑performance data platform from scratch.
• 5+ years of professional experience in software engineering or data engineering.
• Strong software engineering skills with Python in large‑scale, high‑performance production environments.
• Hands‑on experience with Spark/PySpark and other big data frameworks.

Core Competencies

Demonstrates expertise in designing and maintaining scalable data processing pipelines using frameworks like Apache Spark and PySpark, with a strong focus on data integrity and performance optimization. Proven ability to lead data platform migrations and collaborate effectively with cross‑functional teams to support analytics and machine learning workflows.

Highest-signal resume keywords

• Apache Spark
• PySpark
• Data Platform Migration
• Python Software Engineering
• Distributed Systems

ATS Optimization Keywords

Hard Skills

• Data Processing Pipelines
• Microservices Development
• CI/CD Pipelines
• Data Integration
• Data Validation
• Performance Optimization
• Data Security
• Access Control
• Cloud‑Native Platforms
• Big Data Frameworks

Soft Skills

• Collaboration
• Problem‑Solving
• Communication

Industry Keywords

• Data Architecture
• Data Quality
• Data Versioning
• Analytics Workflows
• Machine Learning

Tools & Technologies

• Apache Beam
• Cloudera
• Snowflake
• Databricks