Data Engineer

The Value Maximizer; United States; On-site; USD 90,000 - 120,000; Full time

Job summary

A leading data-driven company in the United States is seeking a skilled Data Engineer to design and maintain scalable data pipelines. The ideal candidate will have strong expertise in PySpark and SQL, along with extensive experience in big data technologies. This full-time role offers the chance to collaborate with cross-functional teams and contribute to data-driven initiatives.

Qualifications

• 6+ years of experience as a Data Engineer or in a similar role.
• Strong skills in PySpark and SQL.
• Hands-on experience with big data tools like Apache Spark, Kafka, Hadoop, or Hive.

Responsibilities

• Develop and maintain scalable, robust data pipelines using Scala and big data technologies.
• Collaborate with Data Scientists and Analysts to understand data requirements.
• Monitor production jobs, troubleshoot issues, and ensure system reliability.

Skills

PySpark

SQL

Apache Spark

Kafka

Hadoop

Hive

Education

Bachelor's or Master's degree in Computer Science, Engineering, or related field

Tools

AWS

Azure

GCP

Docker

Airflow

Kubernetes

Job description

We are seeking a skilled and motivated Data Engineer with strong expertise in PySpark, SQL to join our Data Engineering team. You will be responsible for designing, building, and maintaining scalable data pipelines and processing systems that support our business intelligence, analytics, and machine learning initiatives.

Key Responsibilities:

• Develop and maintain scalable, robust data pipelines using Scala and big data technologies
• Work with large datasets from multiple sources to ingest, transform, and make data available for analytics and reporting
• Collaborate with Data Scientists, Analysts, and other engineers to understand data requirements and deliver efficient solutions
• Optimize ETL jobs for performance and cost
• Ensure data quality, governance, and consistency across all environments
• Monitor production jobs, troubleshoot issues, and ensure system reliability
• Implement best practices for data engineering, including code reviews, testing, and documentation

Required Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, or related field
• 6+ years of experience as a Data Engineer or in a similar role
• Strong skills in PySpark and SQL (experience with functional programming is a plus)
• Hands-on experience with big data tools like Apache Spark, Kafka, Hadoop, or Hive
• Proficiency in building ETL pipelines and working with structured and unstructured data
• Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services
• Familiarity with version control systems (e.g., Git), CI/CD, and DevOps practices
• Solid understanding of data warehousing and data modeling concepts

Preferred Qualifications:

• Knowledge of SQL and database systems such as Postgres, Redshift, or Snowflake
• Exposure to containerization technologies like Docker and orchestration tools like Airflow or Kubernetes

Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Other

Industries

IT Services and IT Consulting