Lead Data Engineer

Univedge Consulting LLC; EE. UU.; Presencial; USD 150.000 - 210.000; Jornada completa

Descripción de la vacante

Univedge Consulting LLC is seeking an experienced Lead Data Engineer to design and implement enterprise-scale data platforms, pipelines, and integrations. The role emphasizes Python, PySpark, SQL, Spark, Databricks, Snowflake, and modern Lakehouse architectures.

You will guide engineering teams and align with data architects and stakeholders. You will architect data structures, ensure governance and security, and drive delivery across multiple teams and platforms, with a focus on reliability and

Formación

• 12+ years IT/data engineering experience with data modeling and pipelines.
• Hands-on Python, PySpark, SQL, Spark experience.
• Experience with ETL/ELT pipelines and enterprise platforms.
• Knowledge of Databricks, Snowflake, Delta Lake, and data warehouses.
• Strong collaboration with analytics, architecture, and business teams.

Responsabilidades

• Design, develop, and maintain scalable data pipelines (batch/real-time).
• Build ingestion, transformation, and processing workflows across systems.
• Develop enterprise data warehouses, data lakes, and processing systems.
• Write complex SQL and optimize performance across pipelines.
• Troubleshoot production data issues and improve reliability.
• Data Architecture & Platform Engineering responsibilities including schemas and models.
• Lead teams, translate requirements into technical plans, and ensure delivery.
• Ensure governance, quality, security, and HIPAA considerations.
• Analytics, AI & Data Science enablement collaboration and data preparation.
• DevOps & DataOps practices to improve deployment and monitoring.

Conocimientos

Python

PySpark

SQL

Spark

Data Modeling

ETL/ELT pipelines

Cloud data platforms

Leadership

Data governance

Collaboration

Educación

Bachelor's degree in CS/IT or related

Herramientas

Databricks

Snowflake

Delta Lake

OCI

Descripción del empleo

We are seeking an experienced, technically strong Lead Data Engineer to design, develop, implement, and support enterprise-scale data engineering platforms and integration solutions.

The ideal candidate will bring deep hands-on experience in Python, PySpark, SQL, Apache Spark, Databricks, Snowflake, ETL/ELT, and modern Data Lakehouse architectures, along with the ability to guide engineering teams and drive technical delivery.

The role involves close collaboration with Data Architects, Data Engineering teams, AI/ML teams, business stakeholders, and technology leadership to deliver scalable, reliable, secure, and high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines using Python, PySpark, SQL, Spark, and ETL/ELT technologies.
• Build and optimize data ingestion, transformation, integration, and processing workflows across enterprise systems.
• Develop and maintain enterprise data warehouses, data lakes, databases, tables, and data processing systems.
• Write complex SQL queries and optimize database and pipeline performance.
• Troubleshoot production data issues and improve pipeline reliability, efficiency, and quality.

2. Data Architecture & Platform Engineering

• Design and implement data structures, schemas, data models, and integration frameworks.
• Work with modern cloud data platforms including Databricks, Snowflake, OCI, and similar technologies.
• Build and support Delta Lake, Lakehouse, and enterprise data warehouse solutions.
• Apply data engineering best practices to ensure scalability, performance, maintainability, and reliability.
• Evaluate technical designs, identify potential issues, and recommend improvements.

3. Technical Leadership & Delivery

• Provide technical guidance to data engineers and collaborate with Data Architects and engineering teams.
• Translate business requirements into technical specifications, implementation plans, and test plans.
• Coordinate technical activities across multiple teams and enterprise platforms.
• Support implementation, testing, deployment, and production stabilization.
• Manage competing technical priorities and ensure timely delivery of engineering solutions.

4. Data Governance, Quality & Security

• Collaborate with data governance, data quality, and information security teams.
• Implement data quality checks, validation processes, and monitoring frameworks.
• Support metadata management, data modeling, and data security requirements.
• Ensure pipelines meet applicable governance, security, and certification standards before production deployment.
• Ensure compliance with HIPAA regulations and appropriate handling of Protected Health Information (PHI).

5. Analytics, AI & Data Science Enablement

• Work closely with analytics, AI/ML, and data science teams to develop reliable datasets and data processing solutions.
• Prepare structured and unstructured data for predictive and prescriptive analytics.
• Support data transformation and integration requirements for machine learning and AI applications.
• Identify opportunities to improve automation, data processing, and analytical capabilities.

6. DevOps & DataOps

• Apply Agile, DevOps, and DataOps principles to data engineering development and delivery.
• Improve deployment processes, pipeline monitoring, validation, and production reliability.
• Support automation of engineering workflows and continuous improvement initiatives.

Required Qualifications

• 12+ years of relevant IT/data engineering experience, including at least 3 years in data modeling, schema design, and building and maintaining data processing systems.
• Strong hands-on programming experience with Python, PySpark, SQL, and Spark.
• Extensive experience designing, developing, and optimizing ETL/ELT pipelines.
• Experience with Databricks, Snowflake, Delta Lake, data warehouses, or comparable enterprise data platforms.
• Strong knowledge of data modeling, database development, data transformation, and large-scale data processing.
• Experience troubleshooting end-to-end data pipelines and resolving production data issues.
• Understanding of data governance, data quality, metadata, and security controls.
• Ability to collaborate with engineering, architecture, analytics, and business teams.
• Strong analytical, problem-solving, communication, and technical leadership skills.
• Bachelor's degree in Computer Science, Information Technology, or a related discipline preferred.