Senior Engineer - Data Engineering

Ingrain Systems Inc; San Francisco (CA); On-site; USD 180,000 - 260,000; Full time

Job summary

Ingrain Systems Inc. is seeking a Senior Data Engineer with deep data engineering expertise and hands-on AI skills in San Francisco, CA.

You will build scalable data infrastructure across ingestion, transformation, orchestration, and delivery while leveraging AI tooling as a core part of engineering and shipping. You will collaborate with backend, BI, and ML teams to design event-driven pipelines, maintain data quality, and participate in on-call reliability, agile processes, and modern tooling

Qualifications

• 5+ years of data engineering experience in large-scale data platforms.
• Strong Python and SQL with Spark experience (PySpark).
• Proficient in building scalable ETL/ELT pipelines and data orchestration.
• Experience with CI/CD and monitoring data workflows.

Responsibilities

• Design, build, and maintain scalable data ingestion and processing pipelines.
• Optimize ETL/ELT workflows and resolve production bottlenecks.
• Integrate multiple structured and unstructured data sources.
• Design event-driven architectures using Kafka, EventBridge, or SNS/SQS.
• Develop and maintain schema, validation, and backward compatibility.
• Collaborate with backend, BI, and ML teams to deliver data products.
• Implement CI/CD pipelines for data workflows and tests.

Skills

Python

SQL

PySpark/Scala

Unix scripting

Airflow

Kafka

DBT

Git/Jenkins/GitHub Actions

Docker

APIs

Tools

Airflow

SnapLogic

DBT

Git

Jenkins

Docker

Job description

Job Description: Senior Data Engineer with AI

About the Role

We're hiring an exceptional Senior Data Engineer with deep data engineering expertise and strong,

hands-on AI skills. You'll build scalable data infrastructure across ingestion, transformation,

orchestration, and delivery — collaborating with backend, BI, and ML teams — while actively using AI tooling (Claude, Cursor, and beyond) as a core part of how you engineer and ship. Strong ownership

mindset, data quality focus, on-call reliability, and a genuine drive to adopt modern AI tooling are a must.

Responsibilities

• Design, build, and maintain scalable data ingestion and processing pipelines using SQL , Spark, Hadoop, Airflow, SnapLogic, or similar tools
• Optimize existing ETL/ELT workflows and resolve bottlenecks in production jobs
• Integrate multiple structured and unstructured data sources from internal and external systems
• Design and implement event-driven data pipelines using event bus architectures (Kafka, AWS EventBridge, SNS/SQS, or similar)
• Build and maintain event producers, consumers, and stream processors for real-time and near-
• Manage event schema design, schema registry, and versioning to ensure backward compatibility across consumers
• Monitor event bus health, lag, throughput, and reliability; respond to failures before SLA breach
• Implement and manage CI/CD pipelines for data workflows using Git, Jenkins, or GitHub Actions
• Ensure robust job scheduling, monitoring, logging, and end-to-end testing of data pipelines
• Debug and resolve production issues before SLA breach; participate in on-call rotation
• Collaborate with backend and BI teams for API and data integration
• Design and develop schema definitions; support data warehouse/mart to integrate disparate data sources and make them available for analysis
• Support large data volumes and flexible provisioning of new data sources
• Contribute to data architecture design across the data landscape
• Write clean, reusable, well-tested Python and Shell code across the full software development lifecycle
• Gather functional requirements, develop technical specifications, and own project and test planning
• Work in Agile/Scrum environments; help align team priorities across the organization
• Leverage AI-assisted dev tools (Claude, Cursor .. etc) to accelerate delivery and code quality

Required Skills

• Strong Python, Advanced SQL, PySpark or Scala, Unix scripting
• Data orchestration: Airflow, SnapLogic, DBT, or equivalent
• Event bus & streaming: Kafka, AWS EventBridge, SNS/SQS, Kinesis, or similar
• Data lake & warehouse: Snowflake, Delta Lake, Redshift, BigQuery
• CI/CD tooling: Git, Jenkins / GitHub Actions, Docker
• Experience with APIs for data exchange and integration
• Data quality mindset, ownership, and SLA awareness
• NoSQL, Neo4j, GraphQL is plus

Preferred / Nice to Have

• Experience with AI/ML models and feature pipelines
• Exposure to BI platforms (Tableau, Power BI) or data serving for dashboards
• Spark Streaming or Flink for stateful stream processing

Skills

Data Engineer, Spark, AWS, Airflow, ETL, Amazon Web Services Datalake, Python PySpark, SNS, SQS,