Senior Data Engineer

Rapsys Technologies Pte Ltd.; Pittsburgh (Allegheny County); On-site; USD 140,000 - 190,000; Full time

Job summary

Rapsys Technologies Pte Ltd. in Pittsburgh is seeking an experienced Senior Data Engineer to design, develop, and maintain secure data platforms for enterprise analytics.

You will build end-to-end data pipelines using Databricks, Spark, Python, SQL, and Azure Data Factory, while ensuring governance and security. You will contribute to BI and AI initiatives, implement streaming architectures with Kafka, and collaborate with cross-functional teams to deliver scalable data solutions in a fast-paced

Qualifications

• Bachelor's degree or higher in a technical field or equivalent experience.
• 6+ years of data engineering experience in enterprise platforms.
• Hands-on with Databricks, Spark, Python, SQL, and Azure Data Factory.
• Experience with Kafka or similar event streaming and data integration tools.

Responsibilities

• Design, develop, test, deploy, and maintain enterprise-scale data solutions.
• Build and optimize ETL/ELT pipelines with Databricks, Spark, Delta Lake, and ADF.
• Develop scalable ingestion, transformation, and integration frameworks for analytics.
• Write and optimize SQL queries, stored procedures, and transformations.
• Design scalable data models for BI, analytics and AI.
• Develop and integrate REST APIs, event-driven architectures, and SOAP services.
• Build and maintain streaming and messaging solutions with Kafka.
• Monitor production pipelines, perform root cause analysis, and implement solutions.
• Implement data quality, governance, security, and performance optimization.
• Manage source code with Git and follow CI/CD practices.
• Collaborate with architects, product owners, and cross-functional teams.
• Participate in Agile ceremonies and maintain runbooks and docs.
• Mentor junior engineers and contribute to engineering standards.

Skills

Databricks

Spark PySpark

Python

SQL

Azure Data Factory

Delta Lake

Kafka

Data governance

DevOps

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or related field

Tools

Delta Live Tables

Unity Catalog

MLflow

Databricks SQL

Git

Linux

Job description

We are seeking an experienced Senior Data Engineer to design, develop, and maintain secure, scalable, and high-performance enterprise data platforms. The ideal candidate will have strong hands‑on experience with Databricks, Apache Spark (PySpark), Python, SQL, Azure Data Factory, Delta Lake, and Apache Kafka.

The candidate will be responsible for building reliable data pipelines, optimizing data processing workflows, implementing data governance and security best practices, and supporting enterprise analytics and reporting initiatives.

Key Responsibilities

• Design, develop, test, deploy, and maintain enterprise-scale data engineering solutions using modern cloud and big data technologies.

• Build and optimize ETL/ELT pipelines using Databricks, Apache Spark (PySpark), Delta Lake, and Azure Data Factory.

• Develop scalable data ingestion, transformation, and integration frameworks for enterprise analytics and reporting.

• Write, optimize, and maintain complex SQL queries, stored procedures, and data transformation logic.

• Design scalable data models to support business intelligence, analytics, and AI initiatives.

• Develop and integrate REST APIs, event-driven architectures, and SOAP services for enterprise data exchange.

• Build and maintain streaming and messaging solutions using Apache Kafka.

• Monitor and troubleshoot production data pipelines, perform root cause analysis, and implement long-term solutions.

• Implement data quality, governance, security, and performance optimization best practices.

• Manage source code using Git and follow CI/CD and DevOps practices.

• Collaborate with architects, product owners, business analysts, and cross-functional engineering teams to deliver technical solutions.

• Participate in Agile ceremonies, architecture discussions, code reviews, and sprint planning.

• Maintain technical documentation, deployment artifacts, testing documentation, and operational runbooks.

• Mentor junior engineers and contribute to engineering standards and continuous improvement initiatives.

Required Skills and Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical discipline, or equivalent professional experience.

• 6+ years of experience in data engineering, enterprise data platforms, cloud analytics, or large-scale data processing.

• Strong hands‑on experience with:
  • Databricks
  • Apache Spark and PySpark
  • Python
  • SQL
  • Scala

• Experience with the Databricks ecosystem, including Delta Lake, Unity Catalog, Delta Live Tables (DLT), Databricks SQL, MLflow, Databricks Workspaces, and Databricks Jobs.

• Strong SQL development skills, including query optimization and performance tuning.

• Experience designing and implementing ETL/ELT pipelines using Azure Data Factory or comparable cloud integration tools.

• Experience with Apache Kafka or other event-streaming technologies.

• Experience integrating enterprise applications using REST APIs, JSON, XML, and SOAP web services.

• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2) or comparable cloud storage platforms.

• Proficiency with Git and collaborative source code management.

• Experience working in Linux environments, including shell scripting and command-line utilities.

• Strong analytical, troubleshooting, and problem‑solving skills.

• Experience working in Agile environments using Scrum, Kanban, or SAFe methodologies.

• Ability to work independently, manage multiple priorities, and mentor team members.

Preferred Qualifications

• Experience with Microsoft Azure cloud services.

• Familiarity with Terraform or other Infrastructure as Code (IaC) tools.

• Experience implementing CI/CD pipelines using Azure DevOps, GitHub Actions, or similar platforms.

• Exposure to Azure Synapse Analytics, Microsoft Fabric, or Power BI.

• Knowledge of data governance, metadata management, and data quality frameworks.

• Experience with DataOps and MLOps practices.

• Knowledge of enterprise data warehousing and Master Data Management (MDM).

• Experience working in financial services or public sector data environments.

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related discipline, or equivalent professional experience.