Data Engineer: Spark, Python & Cloud Data Pipelines

Machine Learning Technologies, LLC; San Jose (CA); Vor Ort; USD 140.000 - 190.000; Vollzeit

Zusammenfassung

Machine Learning Technologies, LLC is seeking a data engineer to design, develop, and maintain scalable data pipelines that support BI and analytics. You will work with Python, SQL, PySpark, and cloud data services to extract, transform, and load data from diverse sources.

Responsibilities include building ETL/ELT workflows, optimizing Spark-based processing, and implementing data quality and governance practices.

Qualifikationen

• Bachelor's degree in Computer Science, Information Technology, or a related field.
• Experience in data engineering, ETL development, and data pipeline implementation.
• Strong programming knowledge in Python, SQL, and PySpark.
• Experience working with cloud platforms and data warehousing technologies.
• Understanding of data modeling, database concepts, and data integration.
• Strong analytical, problem-solving, debugging, and communication skills.

Aufgaben

• Designed, developed, and maintained scalable data pipelines to extract, transform, and load data from multiple sources.
• Developed ETL/ELT workflows using Python, SQL, PySpark, and Azure Data Factory.
• Built and optimized data processing solutions using Apache Spark and Databricks.
• Worked with structured and unstructured data from relational databases, APIs, flat files, and cloud storage.
• Designed and implemented data integration solutions to support business intelligence and analytics requirements.
• Developed complex SQL queries, stored procedures, views, and functions to transform and process large datasets.

Kenntnisse

Python

SQL

PySpark

Analytical thinking

Team collaboration

Ausbildung

Bachelor's degree in Computer Science or related field

Tools

Azure Data Factory

Azure Data Lake Storage

Snowflake

Power BI

Databricks

Apache Spark

Hadoop

Jobbeschreibung

Machine Learning Technologies, LLC is seeking a data engineer to design, develop, and maintain scalable data pipelines that support BI and analytics. You will work with Python, SQL, PySpark, and cloud data services to extract, transform, and load data from diverse sources.

Responsibilities include building ETL/ELT workflows, optimizing Spark-based processing, and implementing data quality and governance practices.