Sr. Data Engineer

iitjobs; United States; Remote; USD 90,000 - 135,000; Full time

Job summary

A leading company is seeking a Sr. Data Engineer to develop innovative data solutions. This role involves designing and maintaining robust data pipelines, implementing feature engineering for machine learning, and collaborating with cross-functional teams to drive data-driven decisions.

Qualifications

• Proficient in GCP and Vertex AI.
• Experience with Kafka, Databricks, and machine learning workflows.
• Strong background in data engineering principles.

Responsibilities

• Provide technical leadership and collaboration across teams.
• Design and maintain efficient data pipelines.
• Mentor team members on data engineering best practices.
• Implement automated testing for data quality.

Skills

GCP

Vertex AI

Feature Engineering

Kafka

Databricks

SQL

NoSQL

Python

Job description

• As a Sr. Data Engineer, you will have the opportunity to lead the development of innovative data solutions, enabling the effective use of data across the organization.
• You will be responsible for designing, building, and maintaining robust data pipelines and platforms to meet business objectives, focusing on data as a strategic asset.
• Your role will involve collaboration with cross-functional teams, leveraging cutting-edge technologies, and ensuring scalable, efficient, and secure data engineering practices.
• A strong emphasis will be placed on expertise in GCP, Vertex AI, and advanced feature engineering techniques.

Key Responsibilities:

• Provide Technical Leadership: Offer technical leadership to ensure clarity between ongoing projects and facilitate collaboration across teams to solve complex data engineering challenges.
• Build and Maintain Data Pipelines: Design, build, and maintain scalable, efficient, and reliable data pipelines to support data ingestion, transformation, and integration across diverse sources and destinations, using tools such as Kafka, Databricks, and similar toolsets.
• Drive Digital Innovation: Leverage innovative technologies and approaches to modernize and extend core data assets, including SQL-based, NoSQL-based, cloud-based, and real-time streaming data platforms.
• Implement Feature Engineering: Develop and manage feature engineering pipelines for machine learning workflows, utilizing tools like Vertex AI, BigQuery ML, and custom Python libraries.
• Implement Automated Testing: Design and implement automated unit, integration, and performance testing frameworks to ensure data quality, reliability, and compliance with organizational standards.
• Optimize Data Workflows: Optimize data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
• Mentor Team Members: Mentor team members in data principles, patterns, processes, and practices to promote best practices and improve team capabilities.
• Draft and Review Documentation: Draft and review architectural diagrams, interface specifications, and other design documents to ensure clear communication of data solutions and technical requirements.
• Cost/Benefit Analysis: Present opportunities with cost/benefit analysis to leadership, guiding sound architectural decisions for scalable and efficient data solutions