Data Engineer

Full Scope; San Francisco (CA); On-site; USD 120,000 - 170,000; Full time

Job summary

Full Scope is seeking a Data Engineer in San Francisco to design and optimize data pipelines for AI and machine learning initiatives. You will work with data scientists, ML engineers, and software teams to ensure robust data architecture and seamless model deployment.

The role requires strong programming skills, experience with big data stacks, and cloud data services. You will contribute to data governance and scalable infrastructure across AWS, GCP, and Azure, staying ahead of industry trends.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer with a focus on AI and machine learning projects.
• Strong proficiency in programming languages such as Python, Scala, Java, or SQL.
• Experience with big data technologies and frameworks such as Hadoop, Spark, Kafka, etc.
• Solid understanding of data modeling, database design, and data warehousing concepts.
• Experience with cloud-based data services (e.g., AWS S3, Redshift, EMR, Glue, Google BigQuery, Azure Data Lake).
• Familiarity with AI and machine learning frameworks and libraries such as TensorFlow, PyTorch, scikit‑learn, etc.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork skills.

Responsibilities

• Design, build, and maintain scalable data pipelines and ETL processes to support AI and machine learning workflows.
• Collaborate with data scientists and machine learning engineers to understand data requirements and translate them into technical solutions.
• Implement data processing, transformation, and integration solutions to ensure high-quality, clean, and usable data for model training and evaluation.
• Optimize data storage and retrieval processes for performance and efficiency.
• Develop and maintain data infrastructure on cloud platforms (AWS, Google Cloud, Azure) to support AI model deployment and scaling.
• Implement monitoring and alerting mechanisms to ensure data pipeline reliability and availability.
• Ensure compliance with data governance and security standards throughout the data lifecycle.
• Stay current with industry trends and advancements in data engineering, AI, and machine learning.
• Participate in code reviews, team meetings, and contribute to a collaborative development environment.
• Document data architecture, processes, and workflows comprehensively.

Skills

Python

Scala/Java/SQL

Big Data

TensorFlow/ PyTorch

Data Modeling

Cloud Data Services

Communication

Problem-Solving

Education

Bachelor's or Master's in CS/Engineering

Tools

Hadoop

Spark

Kafka

Docker

Kubernetes

AWS S3/Redshift/EMR

Google BigQuery

Job description

Job Title: Data Engineer
Location: San Francisco, CA
Required Clearance: Secret
Salary: Competitive

We are looking for a skilled Data Engineer with a strong focus on AI and machine learning to join our dynamic team. The ideal candidate will play a critical role in designing, implementing, and optimizing data pipelines and infrastructure to support our AI-driven initiatives. You will collaborate closely with data scientists, machine learning engineers, and software developers to ensure robust data architecture and seamless integration of AI models.

Key Responsibilities:

• Design, build, and maintain scalable data pipelines and ETL processes to support AI and machine learning workflows.
• Collaborate with data scientists and machine learning engineers to understand data requirements and translate them into technical solutions.
• Implement data processing, transformation, and integration solutions to ensure high-quality, clean, and usable data for model training and evaluation.
• Optimize data storage and retrieval processes for performance and efficiency.
• Develop and maintain data infrastructure on cloud platforms (e.g., AWS, Google Cloud, Azure) to support AI model deployment and scaling.
• Implement monitoring and alerting mechanisms to ensure data pipeline reliability and availability.
• Ensure compliance with data governance and security standards throughout the data lifecycle.
• Stay current with industry trends and advancements in data engineering, AI, and machine learning.
• Participate in code reviews, team meetings, and contribute to a collaborative development environment.
• Document data architecture, processes, and workflows comprehensively.

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer with a focus on AI and machine learning projects.
• Strong proficiency in programming languages such as Python, Scala, Java, or SQL.
• Experience with big data technologies and frameworks such as Hadoop, Spark, Kafka, etc.
• Solid understanding of data modeling, database design, and data warehousing concepts.
• Experience with cloud-based data services (e.g., AWS S3, Redshift, EMR, Glue, Google BigQuery, Azure Data Lake).
• Familiarity with AI and machine learning frameworks and libraries such as TensorFlow, PyTorch, scikit‑learn, etc.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork skills.

Preferred Qualifications:

• Experience with containerization technologies such as Docker and orchestration tools like Kubernetes.
• Knowledge of DevOps practices and CI/CD pipelines.