Bigdata Engineer with Agentic AI Workflows

IMR Soft LLC; Mountain View (CA); On-site; USD 180,000 - 230,000; Full time

Job summary

IMR Soft LLC is seeking a Senior Software Engineer to design, build, and operate large-scale data platforms and agentic AI workflows. You will work across the Data Development Lifecycle, delivering scalable pipelines and data products while applying LLM-driven automation.

You will collaborate in an Agile/Scrum environment, develop ETL/ELT solutions, and optimize data processing on AWS, Hive, and Spark-based systems. Strong SQL, Python, and cloud experience are essential.

Qualifications

• BS/MS in Computer Science or equivalent work experience.
• Experience building large-scale data pipelines.
• Proficiency in Python and SQL.
• Hands-on with AWS (EC2, S3, EMR) and Redshift.
• Experience with Hive and Spark.
• Experience with JSON/REST services.

Responsibilities

• Design, build, and maintain scalable data pipelines.
• Develop schemas and ETL/ELT processes.
• Operate on MPP/Hadoop systems with Hive/Hive on Spark.
• Leverage AWS to build data infrastructure.
• Write automation scripts in Python or Shell.
• Build agentic workflow solutions using LLM tools.
• Write analytical SQL for data warehousing.
• Create and consume JSON/REST services.
• Own production pipeline health and SLA management.
• Participate in Agile/Scrum ceremonies.

Skills

Python

SQL

ETL/ELT

Hive

Spark

AWS

LLM tools

Data Pipelines

Agile/Scrum

Data Warehousing

Education

BS/MS in Computer Science

Job description

Senior Software Engineer – Big Data & Agentic AI Workflows

Location: Mountainview, CA

12 months

Onsite Only

About the Role

We are looking for a Software Engineer in Data with deep experience building large-scale data platforms, data products and pipelines to join our team. You will design, build, and operate highly scalable, fault-tolerant data processing systems, and help modernize how we deliver data by building agentic workflow solutions that use LLM tools to automate steps across the Data Development Lifecycle (DDLC). This role blends strong traditional big-data engineering with hands-on application of AI agents to accelerate data design, development, testing, and operations.

What You'll Do

• Design, build, and maintain highly scalable, robust, and fault-tolerant data processing pipelines from the ground up.
• Develop database schemas and build ETL/ELT pipelines that process large data volumes reliably and efficiently.
• Build and operate solutions on MPP/Hadoop-based systems, with hands-on work in Hive and Hive on Spark.
• Leverage AWS services (EC2, S3, EMR, Redshift, or equivalent cloud platforms) to build and scale data infrastructure.
• Write advanced scripts and automation in Python, Shell, or similar languages to support pipeline development and operations.
• Build agentic workflow solutions using LLM tools to automate and accelerate tasks across the Data Development Lifecycle (DDLC) — including schema design, code generation, data quality checks, testing, and documentation.
• Design and write analytical SQL for data marts, data warehousing, and broader analytic architecture.
• Create and consume JSON/REST web services to integrate with upstream and downstream systems.
• Own operational health of production pipelines, including problem management, SLA management, and incident response.
• Participate actively in Agile/Scrum teams — sprint planning, estimation, code reviews, and retrospectives.
• Data Maturity Data Products using Data Capabilities with Data Quality ( Data Completeness and Accuracy) and Observability for SLA
• Cost and Performance Optimization of the data pipelines.

What You'll Bring

• Deep software development experience, including building database schemas, developing ETLs, and working with MPP/Hadoop systems.
• Advanced proficiency in a scripting language such as Python or Shell.
• Experience with AWS (EC2, S3, EMR) and Redshift, or equivalent cloud computing platforms.
• Hands-on experience with the Hadoop stack, particularly Hive and Hive on Spark.
• Proven experience designing, building, and maintaining large-scale, fault-tolerant data pipelines.
• Experience working with large data volumes in production environments.
• Experience creating and consuming JSON/REST web services and integrating across systems.
• Practical experience building agentic workflow solutions or applying LLM tools to automate parts of the Data Development Lifecycle.
• Strong grounding in software development methodologies and best practices.
• Experience working in Agile development teams, with working knowledge of Scrum.
• An operational mindset, comfortable with Problem, SLA, and Incident Management.
• Strong expertise writing analytical SQL, and experience with data marts, data warehousing, and analytic architecture.

Education

• BS/MS in Computer Science, or equivalent work experience.