Data Engineer II

Jobtailor; California (MO); On-site; USD 110,000 - 150,000; Full time

Job summary

Jobtailor seeks a data engineer to design and own large-scale data pipelines and foundational datasets for ML-driven experiences.

You will collaborate with ML engineers to create features for recommendation systems, validate diverse data sources, and push for higher data quality through testing and monitoring. A strong background in Python and cloud platforms is essential.

Qualifications

• Bachelor's degree in Computer Science (or related field) or equivalent work experience.
• 3 years of experience with large-scale data workloads, building and curating datasets and developing data pipelines and processing jobs.
• Strong programming skills in Python, Java, or comparable object-oriented language.
• Experience with Databricks or equivalent large-scale data processing platforms.
• Demonstrated knowledge of operating within a Public Cloud Provider (AWS, Azure, Google Cloud).
• Experience with source control systems and CI/CD pipelines.
• Strong grasp of computer science fundamentals (data structures, algorithms, design patterns).

Responsibilities

• Design, craft and own high quality foundational datasets for ML models and subscriber experiences.
• Source, analyze and validate diverse upstream data sources from partner teams and raw signals using exploratory data analysis.
• Identify and build new user and content data features for recommendation systems with ML engineers.
• Enhance existing feature quality through experimentation and iterative development.
• Build, optimize and maintain large scale data processing jobs, pipelines and workloads.
• Improve data quality and reliability through checks, anomaly detection, testing, monitoring and alerting.
• Participate in on-call rotation to support critical production data pipelines and observability best practices.

Skills

Python Programming

Data Pipeline Development

Large Scale Data Processing

Cloud Computing (AWS, Azure, Google)

Databricks Experience

Education

Bachelor’s degree in Computer Science or related field

Tools

CI/CD Pipelines

Source Control Systems

Data Processing Platforms

Job description

Responsibilities

• Design, craft and own high quality foundational datasets to produce reliable, meaningful inputs for machine learning models and subscriber experiences
• Source, analyze and validate diverse upstream data sources from upstream partner teams and raw signals using exploratory data analysis
• Identify and build new user and content data features for recommendation systems through collaboration with machine learning engineers
• Enhance existing feature quality through experimentation, analysis and iterative development
• Build, optimize and maintain large scale data processing jobs, pipelines and workloads
• Improve data quality and reliability through implementing checks, anomaly detection, testing, monitoring and alerting
• Contribute to operational excellence by following observability best practices and participating in an on-call rotation to support critical production data pipelines

Requirements

• Bachelor’s degree in Computer Science (or related field), or equivalent work experience
• 3 years of experience working with large scale data workloads, building and curating datasets and developing data pipelines and processing jobs
• Strong programming skills in Python, Java, or comparable object-oriented language
• Experience working with Databricks or equivalent large-scale data processing platforms
• Demonstrated knowledge of operating within a Public Cloud Provider (e.g. AWS, Microsoft Azure, Google Cloud)
• Experience working with source control systems and CI/CD pipelines
• Strong grasp of computer science fundamentals (data structures, algorithms, design patterns, etc.)

Core Competencies

Demonstrates expertise in designing and maintaining large-scale data processing pipelines and datasets, with a strong foundation in programming and cloud technologies. Proficient in enhancing data quality and reliability through rigorous testing and monitoring practices.

Highest-signal resume keywords

• Python Programming
• Data Pipeline Development
• Large Scale Data Processing
• Cloud Computing (AWS, Azure, Google Cloud)
• Databricks Experience

Hard Skills

• Data Analysis
• Exploratory Data Analysis
• Feature Engineering
• Anomaly Detection
• Data Quality Improvement
• Data Structures
• Algorithms
• Design Patterns

Industry Keywords

• Machine Learning Models
• Recommendation Systems
• Observability Best Practices
• Operational Excellence

Tools & Technologies

• CI/CD Pipelines
• Source Control Systems
• Data Processing Platforms