Senior Data Engineer

Tap Growth ai; Pittsburgh (Allegheny County); On-site; USD 140,000 - 180,000; Full time

Job summary

Tap Growth ai is seeking a Senior Data Engineer to design, develop, and maintain secure, scalable enterprise data platforms. You will build reliable data pipelines, optimize processing workflows, and implement governance and security best practices to support analytics and reporting initiatives.

Responsibilities include ETL/ELT development with Databricks, Delta Lake, and Azure Data Factory, as well as data modeling, API integration, and streaming solutions with Kafka.

Qualifications

• Bachelor's degree or equivalent in a related technical field.
• 6+ years in data engineering or enterprise data platforms.
• Strong SQL development and performance tuning abilities.
• Experience with Databricks ecosystem and cloud data services.

Responsibilities

• Design, develop, test, deploy, and maintain enterprise data engineering solutions.
• Build and optimize ETL/ELT pipelines using Databricks, Spark, Delta Lake, and ADF.
• Develop data ingestion, transformation, and integration frameworks for analytics.
• Write and optimize complex SQL queries and data transformation logic.
• Design scalable data models for BI, analytics, and AI initiatives.
• Develop and integrate REST APIs and event-driven architectures.
• Maintain streaming solutions with Apache Kafka and monitor pipelines.

Skills

Databricks

Spark PySpark

Python

SQL

Scala

Education

Bachelor's degree or equivalent

Tools

Databricks

Delta Lake

Azure Data Factory

Apache Kafka

Job description

We are seeking an experienced Senior Data Engineer to design, develop, and maintain secure, scalable, and high-performance enterprise data platforms. The ideal candidate will have strong hands‑on experience with Databricks, Apache Spark (PySpark), Python, SQL, Azure Data Factory, Delta Lake, and Apache Kafka.

The candidate will be responsible for building reliable data pipelines, optimizing data processing workflows, implementing data governance and security best practices, and supporting enterprise analytics and reporting initiatives.

Key Responsibilities

• Design, develop, test, deploy, and maintain enterprise-scale data engineering solutions using modern cloud and big data technologies.

• Build and optimize ETL/ELT pipelines using Databricks, Apache Spark (PySpark), Delta Lake, and Azure Data Factory.

• Develop scalable data ingestion, transformation, and integration frameworks for enterprise analytics and reporting.

• Write, optimize, and maintain complex SQL queries, stored procedures, and data transformation logic.

• Design scalable data models to support business intelligence, analytics, and AI initiatives.

• Develop and integrate REST APIs, event-driven architectures, and SOAP services for enterprise data exchange.

• Build and maintain streaming and messaging solutions using Apache Kafka.

• Monitor and troubleshoot production data pipelines, perform root cause analysis, and implement long-term solutions.

• Implement data quality, governance, security, and performance optimization best practices.

• Manage source code using Git and follow CI/CD and DevOps practices.

• Collaborate with architects, product owners, business analysts, and cross-functional engineering teams to deliver technical solutions.

• Participate in Agile ceremonies, architecture discussions, code reviews, and sprint planning.

• Maintain technical documentation, deployment artifacts, testing documentation, and operational runbooks.

• Mentor junior engineers and contribute to engineering standards and continuous improvement initiatives.

Required Skills and Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical discipline, or equivalent professional experience.

• 6+ years of experience in data engineering, enterprise data platforms, cloud analytics, or large-scale data processing.

• Strong hands‑on experience with:
  • Databricks
  • Apache Spark and PySpark
  • Python
  • SQL
  • Scala

• Experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, Delta Live Tables (DLT), Databricks SQL, MLflow, Databricks Workspaces, and Databricks Jobs.

• Strong SQL development skills, including query optimization and performance tuning.

• Experience designing and implementing ETL/ELT pipelines using Azure Data Factory or comparable cloud integration tools.

• Experience with Apache Kafka or other event-streaming technologies.

• Experience integrating enterprise applications using REST APIs, JSON, XML, and SOAP web services.

• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2) or comparable cloud storage platforms.

• Proficiency with Git and collaborative source code management.

• Experience working in Linux environments, including shell scripting and command-line utilities.

• Strong analytical, troubleshooting, and problem‑solving skills.

• Experience working in Agile environments using Scrum, Kanban, or SAFe methodologies.

• Ability to work independently, manage multiple priorities, and mentor team members.

Preferred Qualifications

• Experience with Microsoft Azure cloud services.

• Familiarity with Terraform or other Infrastructure as Code (IaC) tools.

• Experience implementing CI/CD pipelines using Azure DevOps, GitHub Actions, or similar platforms.

• Exposure to Azure Synapse Analytics, Microsoft Fabric, or Power BI.

• Knowledge of data governance, metadata management, and data quality frameworks.

• Experience with DataOps and MLOps practices.

• Knowledge of enterprise data warehousing and Master Data Management (MDM).

• Experience working in financial services or public sector data environments.

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related discipline, or equivalent professional experience.