Senior Data Engineer

1872 Consulting; Mountain View (CA); Remote; USD 120,000 - 150,000; Full time

Job summary

A leading company is seeking a Sr. Data Engineer to join their expanding Data Platform and Engineering teams. The ideal candidate will have extensive experience in developing scalable data platforms and must be adept at collaborating with product managers and data scientists. Responsibilities include designing data processing pipelines, integrating data, and ensuring data quality.

Qualifications

• Over 5 years of experience in object-oriented programming with Python.
• Proficiency with cloud computing, especially AWS services.

Responsibilities

• Design and develop data processing pipelines.
• Integrate data from multiple sources and develop cross-platform ETL processes.
• Collaborate with Engineering and Data Science teams.

Skills

Reporting

Analytics

Database Management

Data Ingestion

ETL

Data Mining

Statistical Analysis

Cloud Computing

Object-Oriented Programming

SQL

Tools

AWS

Apache Spark

Docker

Job description

Sr. Data Engineer
MOUNTAIN VIEW, CA / DENVER, CO / REMOTE
FULL-TIME

Position Summary:

We are seeking a Sr. Data Engineer to join our expanding Data Platform and Engineering teams. The ideal candidate will have extensive experience in developing scalable data platforms that support business intelligence, analytics, data science, and data products. They should possess strong, hands-on technical skills across various technologies and demonstrate the ability to craft robust, scalable solutions. Comfort working in an agile environment with minimal supervision is essential. The candidate must be adept at collaborating with product managers, data scientists, and business stakeholders to translate vague requirements into effective code. A passion for continuous improvement and data quality is vital for success in this role.

Responsibilities:

1. Design and develop data processing pipelines
2. Integrate data from multiple sources and develop cross-platform ETL processes
3. Perform data validation and verification
4. Analyze data, troubleshoot issues, and implement solutions to ensure data quality and delivery
5. Create systems for data acquisition and cleaning
6. Develop new tools and processes for managing data workflows and infrastructure
7. Collaborate with Engineering and Data Science teams to build, maintain, and monitor database infrastructure
8. Work with product managers, data scientists, business users, and engineers to define requirements and design solutions
9. Extract and analyze data from web sources (census, open data, commercial vendors)

Skills and Requirements:

• Expertise in reporting, analytics, and database management
• Experience with data ingestion, ETL, and storage solutions
• Interest in aggregating data from various sources
• Experience with big data, data mining, and statistical analysis
• Proficiency with cloud computing, especially AWS services (S3, EC2, etc.)
• Ability to select appropriate technologies for specific applications (e.g., MySQL vs. PostgreSQL, Hadoop vs. Cassandra)
• Over 5 years of experience in object-oriented programming with Python
• Knowledge of additional languages such as Scala, C++, Java is a plus
• Experience with Apache Spark
• Strong SQL skills
• Familiarity with Docker
• Knowledge of machine learning libraries/frameworks like scikit-learn, TensorFlow, PyTorch is advantageous
• Experience deploying algorithms at scale