Sr Data Engineer

TechDigital Group; St. Louis (MO); On-site; USD 80,000 - 120,000; Full time

Job summary

An established industry player is seeking a talented Data Engineer to join their innovative team. In this role, you will leverage Google Cloud Platform to design and implement scalable data solutions, ensuring high performance and measurable business value. You will manage deployments using Kubernetes and Helm, automate processes with Jenkins, and optimize data workflows using Python and SQL. The ideal candidate will have a strong understanding of data engineering principles and excellent communication skills to collaborate across teams. This is a fantastic opportunity to make a significant impact in a dynamic environment focused on cutting-edge technology and data solutions.

Qualifications

• 3+ years of experience as a data engineer with strong skills in Python and SQL.
• Proficient in GCP tools and container orchestration using Kubernetes and Helm.

Responsibilities

• Design and implement data pipelines using GCP services.
• Oversee deployments of data applications on Kubernetes Engine using Helm.

Skills

Google Cloud Platform (GCP)

Kubernetes

Helm

Jenkins

Python

SQL

Cloud Monitoring

Data Engineering

Communication Skills

Education

Bachelor's Degree in Computer Science or related field

Tools

GCP Services

Cloud Composer (Apache Airflow)

Terraform

Datadog

Job description

The Data Engineer will work iteratively on the cloud platform to design, develop and implement scalable, high performance solutions that offer measurable business value to customers.

Responsibilities

1. Data Solutions: Design and implementation of data pipelines using GCP services.
2. Manage Deployments with Helm and Kubernetes: Oversee containerized deployments of data applications on Kubernetes Engine using Helm for package management, ensuring efficient orchestration of services.
3. Automation with Jenkins: Implement CI/CD pipelines using Jenkins to automate data pipeline deployment, testing, and integration with other services, ensuring quick iterations and deployments.
4. Develop and Optimize Pipelines: Write efficient Python and SQL scripts to build data pipelines and ETL/ELT processes. Continuously monitor and optimize data workflows for performance and cost-effectiveness.
5. Data Integration and Orchestration: Design workflows to integrate data from various sources using GCP services, and orchestrate complex tasks with Cloud Composer (Apache Airflow).
6. Security and Compliance: Ensure data security and compliance by implementing IAM policies, encryption, and other security measures in GCP, adhering to best practices for handling sensitive data.
7. Collaborate Across Teams: Work closely with application developers, data architects, and business stakeholders to define and deliver robust data-driven solutions. Provide technical leadership and ensure alignment between business and technical objectives.
8. Monitoring and Logging: Set up monitoring, logging, and alerting using Cloud Monitoring (formerly Stackdriver), Datadog, or other tools to ensure visibility into pipeline performance and quickly identify and resolve issues.

Skills

1. Google Cloud Platform (GCP) Tools: (Preferred) - BigQuery, Cloud Storage, Dataflow, Cloud Functions, Pub/Sub, Cloud Run, Cloud Composer (Airflow), Cloud Spanner, Bigtable.
2. Container Orchestration: Kubernetes (preferred on GKE) and Helm for managing and deploying containerized applications.
3. CI/CD and Automation: Jenkins for building CI/CD pipelines to automate deployment and testing of data pipelines.
4. Programming Languages: Proficient in Python for data processing and automation, SQL for querying and data manipulation. Experience with Java is a plus.
5. DevOps Tools: Familiarity with Terraform or Deployment Manager for Infrastructure as Code (IaC) to manage GCP resources.
6. Monitoring and Logging: Experience with Cloud Monitoring, Datadog, or other monitoring solutions to track pipeline performance and ensure operational efficiency.
7. Data Engineering Skills: Expertise in ETL/ELT pipelines, data modeling, and data integration across large datasets.
8. Strong understanding of data warehousing and real-time data processing workflows.
9. Strong communication skills to work effectively with cross-functional teams and mentor junior developers. Proven ability to lead in an Agile environment.
10. 3+ years of experience as a data engineer, with hands-on experience in Kubernetes, Helm, Python and Jenkins.
11. Strong experience building and optimizing data pipelines and services in any cloud platform.
12. Proficiency in Python and SQL. Familiarity with Java and Docker is a plus.