Senior Data Engineer

Quantumtech Inc; United States; On-site; USD 180,000 - 250,000; Full time

Job summary

Quantumtech Inc. is seeking a highly experienced Senior Data Engineer to design, develop, and implement scalable data pipelines and architectures.

You will lead enterprise data initiatives, optimize workflows, and collaborate with cross-functional teams to deliver reliable data solutions. The role requires 15+ years in data engineering, strong Python/SQL/PySpark skills, and deep expertise in Spark, Hadoop, and cloud platforms (AWS/Azure/GCP).

Qualifications

• 15+ years in Data Engineering/Big Data/Software Engineering
• Strong Python, SQL, PySpark
• Expertise in Spark, Hadoop, distributed computing
• Experience with ETL/ELT tools and data integration
• Cloud platforms: AWS, Azure, GCP
• Knowledge of data modeling, warehousing, and data governance
• Experience with Airflow, AWS Glue, Azure Data Factory or similar orchestration tools
• Knowledge of CI/CD, Git, Docker, Kubernetes

Responsibilities

• Design, develop, and maintain scalable data pipelines and architectures.
• Lead development of complex ETL/ELT workflows.
• Build and optimize large-scale data processing systems.
• Develop data solutions using Python, SQL, PySpark, Spark.
• Design and implement data lakes, warehouses, and lakehouse architectures.
• Work with AWS, Azure, and GCP.
• Implement real-time and batch data processing.
• Optimize performance, reliability, security.
• Lead technical discussions and mentor engineers.
• Collaborate with Data Scientists, Analysts, Architects, and stakeholders.
• Establish data engineering best practices and coding standards.
• Monitor pipeline performance and troubleshoot production issues

Skills

Python

SQL

PySpark

Apache Spark

Hadoop

Kafka

Airflow

CI/CD

Docker

Kubernetes

Cloud AWS/Azure/GCP

Education

Bachelor's or Master's in Computer Science/IT

Tools

Snowflake

Databricks

Redshift/BigQuery

Job description

We are looking for a highly experienced Senior Data Engineer with 15+ years of experience in designing, developing, and implementing scalable data engineering solutions. The ideal candidate should have strong expertise in big data technologies, cloud platforms, data architecture, ETL/ELT pipelines, and distributed computing.

The candidate will be responsible for leading data engineering initiatives, building enterprise-level data platforms, optimizing data workflows, and collaborating with cross-functional teams to deliver reliable and high-performance data solutions.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data architectures.
• Lead the development of complex ETL/ELT workflows.
• Build and optimize large-scale data processing systems.
• Develop data solutions using Python, SQL, PySpark, and Spark.
• Design and implement data lakes, data warehouses, and lakehouse architectures.
• Work with cloud platforms such as AWS, Azure, and Google Cloud Platform.
• Implement real-time and batch data processing solutions.
• Optimize data performance, reliability, scalability, and security.
• Lead technical discussions and provide guidance to junior and senior engineers.
• Collaborate with Data Scientists, Data Analysts, Architects, and business stakeholders.
• Establish data engineering best practices and coding standards.
• Monitor data pipeline performance and troubleshoot production issues.

Required Technical Skills

• 15+ years of experience in Data Engineering / Big Data / Software Engineering.
• Strong programming experience in Python, SQL, and PySpark.
• Expertise in Apache Spark, Hadoop, and distributed computing.
• Strong experience with ETL/ELT tools and data integration.
• Experience with cloud platforms: AWS, Azure, or GCP.
• Knowledge of Snowflake, Databricks, Redshift, or BigQuery.
• Experience with Apache Kafka and real-time data streaming.
• Strong understanding of data modeling, data warehousing, and database concepts.
• Experience with Airflow, Azure Data Factory, AWS Glue, or similar orchestration tools.
• Knowledge of CI/CD, Git, Docker, and Kubernetes.
• Strong understanding of data governance, data quality, and security.

Preferred Qualifications

• Bachelor's or Master's degree in Computer Science, Information Technology, or a related field.
• Experience leading enterprise-level data engineering projects.
• Strong problem-solving and analytical skills.
• Excellent communication and stakeholder management skills.
• Ability to manage multiple projects and lead technical teams.

Python | SQL | PySpark | Apache Spark | Hadoop | AWS | Azure | GCP | Snowflake | Databricks | Kafka | Airflow | ETL | Data Warehousing | Data Lakes | Data Architecture

Candidate Expectations

• Strong technical leadership and architecture capabilities.
• Ability to drive end-to-end data engineering solutions.
• Excellent communication and collaboration skills.

About us

Quantumtech Inc. is a US-based IT services and consulting company providing technology talent and solutions to enterprises, mid-market organizations, and government contractors. We specialize in IT staffing, consulting, technology solutions.

Why Quantum ?

Day 1 NOC Stay because you choose to.

Low Payroll Margin & Transparent Payroll

Refer a consultant facing payroll issues to our team and earn after a successful transition.

Check out the Payroll Calculator: https://quantumtechus.com/payroll-calculator/