Lead Software Engineer

Virtusa; New York (NY); Presencial; USD 150.000 - 210.000; Jornada completa

Descripción de la vacante

Virtusa seeks a senior Core Data Engineer to design, build, and optimize scalable ETL/ELT pipelines in Python and PySpark, with strong SQL. You will work with Snowflake, BigQuery or Redshift and Databricks to model data for self‑service analytics. Emphasis on data quality, testing, and reliable datasets.

The role involves collaboration with analysts, data scientists, and business teams, leveraging Airflow, dbt, and CI/CD tooling to deliver robust data solutions.

Formación

• 5+ years designing robust data pipelines (ETL/ELT).
• 4+ years Python coding and advanced SQL.
• 3+ years with Spark (PySpark) and Databricks.
• Cloud data platforms: Snowflake, BigQuery or Redshift.
• Experience with data quality, automated testing, and error handling.

Responsabilidades

• Design, build, and maintain scalable ETL/ELT pipelines using Python, PySpark, and Databricks.
• Implement Medallion Architecture and optimized data models in Snowflake/BigQuery/Redshift.
• Tune performance, troubleshoot data workflows, and ensure reliability.
• Collaborate with analysts, data scientists, and business teams to deliver high‑quality datasets.

Conocimientos

Python

SQL

PySpark

Databricks

Snowflake

BigQuery

Redshift

Airflow

Prefect

dbt

Terraform

Kafka

Kinesis

Spark

Herramientas

Databricks

Snowflake

BigQuery

Redshift

Airflow

Prefect

dbt

GitHub Actions

GitLab CI

Jenkins

Terraform

Kafka

Kinesis

Spark

Descripción del empleo

Core Data Engineering & Architecture: 5+ years of experience designing, building, and optimizing robust, scalable data pipelines (ETL/ELT).Programming Languages: 4+ years of strong, production-grade coding experience in Python and advanced proficiency writing complex, optimized SQL queries.

Big Data & Lakehouse Platforms: 3+ years of experience working with distributed computing frameworks such as Apache Spark (PySpark) and modern lakehouse platforms such as Databricks.

Data Warehousing & Dimensional Modeling: Advanced proficiency in cloud data platforms—specifically Snowflake (or Google BigQuery / AWS Redshift)—including schema design, partitioning strategies, and building optimized dimensional models.

Problem-Solving & Data Quality: Proven track record of implementing rigorous data validation, data cleansing, automated testing, and error-handling frameworks across pipeline stages.

Preferred Skills & Attributes

• Cloud Infrastructure: 2+ years of hands‑on experience with cloud platforms (AWS, GCP, or Azure), particularly with managed data services and serverless computing.
• Orchestration Tools: Experience managing and monitoring complex workflows using tools like Apache Airflow, Prefect, or dbt.
• CI/CD & DevOps: Experience deploying pipelines via CI/CD frameworks (GitHub Actions, GitLab CI, or Jenkins) and infrastructure-as-code (Terraform).
• Streaming Architecture: Familiarity with real-time data streaming technologies such as Apache Kafka, AWS Kinesis, or Spark Structured Streaming is a strong plus.

Primary Responsibilities

• Pipeline Development: Design, build, and maintain automated, scalable ETL/ELT pipelines ingesting high-volume data using Python, PySpark, and Databricks.
• Data Modeling & Architecture: Implement Medallion Architecture standards and construct clean relational and dimensional data models in Snowflake to support self-service analytics and BI workloads.
• Performance Tuning: Monitor, troubleshoot, and optimize data workflows, Spark jobs, and Snowflake queries for speed, cost‑efficiency, and reliability.
• Collaboration & Integration: Partner with analysts, data scientists & business teams to translate complex needs into structured, high‑quality datasets..

Preferred Experience & Domain

Enterprise tech background; skilled in data value chains, cross‑functional Agile delivery.