Senior Data Engineer

AI Tech Suite; Snowflake (AZ); Remote; USD 80,000 - 120,000; Full time

Job summary

Join a forward-thinking company at the forefront of AI technology, specializing in Reinforcement Learning and Computer Vision. This role offers the chance to work with cutting-edge technologies, contributing to innovative AI solutions that bridge the gap between research and industry. You'll collaborate with a talented team to design scalable processing pipelines, implement data structures, and enhance machine learning algorithms. With a strong emphasis on personal growth, flexible work models, and a commitment to continuous learning, this position is perfect for those eager to make a significant impact in the AI field. Embrace the opportunity to grow professionally while working on groundbreaking projects in a collaborative environment.

Benefits

Continuous training and education opportunities

Work From Home Days

Individual KPI access

Weekly team events

Technical and soft-skills trainings

Flexible work arrangements

Company share programme

Health, dental and vision insurance

401(k) plan

Annual leave plus national holidays

Qualifications

• 3+ years in data-driven environments with a focus on AI and ML.
• Proficiency in Python and experience with cloud environments.

Responsibilities

• Collaborate to design and manage scalable processing pipelines.
• Implement robust processing flows for large datasets.

Skills

Python programming

Data engineering

Machine Learning

Computer Vision

Kubernetes

Docker

SQL

Cloud environments (AWS, Gcloud, Azure)

CI/CD

ETL processes

Education

Bachelor's degree in Computer Science

Master's degree in Computer Science

Tools

Postgres

SQLalchemy

Airflow

Flask

Terraform

Apache Spark

Job description

Find the latest job opportunities in AI and tech.

enliteAI is a technology provider for Artificial Intelligence specialized in Reinforcement Learning and Computer Vision/geoAI. They offer AI solutions and services.

International product and innovation-driven team with rich expertise in Computer Vision and Reinforcement Learning as well as distributed training, data engineering, ML ops and cloud architectures.

Working with the latest technologies at the interface between research and industry (enliteAI is an ELISE EU research network Organizing Node).

Personal growth: Receive continuous training and education opportunities. Budget and time allotment for the pursuit of individual R&D projects, training or conference participations.

Flexible work models: Remote work, an office in Vienna's 1st district and minimal core hours.

Experience Requirements:

1. 3+ years of work experience in data-driven environments.
2. Passionate about everything related to AI, Machine Learning and Computer Vision.

Other Requirements:

1. Python programming skills, emphasis on data engineering and distributed processing (e.g. Flask, Postgres, SQLalchemy, Airflow).
2. Proficiency in working with databases and data storage solutions.
3. Experience with Kubernetes and Docker (Helm, Terraform, Amazon Kubernetes Service).
4. Familiarity with cloud environments (AWS, Gcloud, Azure).
5. Used to mature workflows in software development (Git, issue management, documentation, unit testing, CI/CD).
6. Fluent in English both spoken and in written language.
7. Valid work permit for Austria.

Responsibilities:

1. Collaborate with our machine learning and backend engineers to design and manage scalable processing pipelines in productive environments.
2. Implement robust processing flows and I/O-efficient data structures, powering use cases such as road surface analysis, sign detection and localization on large volumes of point cloud and imagery data.
3. Design and manage relevant database schemes in close collaboration with backend engineering.
4. Create and maintain comprehensive documentation of the processing pipelines, database schemes, configuration and software architecture.

Scylla is an AI-powered video analytics platform for security, offering gun detection, face recognition, and more.

Individual Benefit package: serves for your work and life balance.

Work From Home Days.

Individual KPI (Key Performance Indicator); Access to advisers including advisory board member at NASA Health Institute and serial founders to help us grow, personally and professionally; Weekly team events; Technical and soft-skills trainings; Variety of knowledge-sharing and self-development opportunities.

Education Requirements:

1. Degree in Computer Science, IT, or similar field; a Master’s is a plus.

Experience Requirements:

1. 4+ years of experience in designing and developing Python-based solutions.

Responsibilities:

1. Build high-performance databases, and improve data models.
2. Machine learning algorithm deployment.
3. Manage data and meta-data.
4. Perform data analysis and implement solutions to improve processes; Development of data-related instruments/instances; Write unit/integration tests to maintain accuracy in the data models; Track pipeline stability; Drive the collaboration process with other team members.

Shaped helps increase engagement, conversion, and retention with GenAI search & recommendations, using a configurable system adapting in real-time for various platforms.

Arturo is an AI-powered property intelligence platform for the insurance industry, offering solutions for underwriting, risk management, and claims.

Willingness to learn: You have an insatiable desire to continue growing, a fearless approach to the unknown, and love a challenge.

Teamwork/Collaboration: You like working with others; you participate actively and enjoy sharing the responsibilities and rewards. You pro-actively work to strengthen our team. And you definitely have a sense of humor.

Critical Thinking: You incorporate analysis, interpretation, inference, explanation, self-regulation, open-mindedness, and problem-solving in everything you do.

Drive for Results: You keep looking forward, solve problems and participate in the success of our growing organization.

Experience Requirements:

1. Good to Expert level understanding of geospatial systems, concepts, patterns, and software, including both legacy formats and software, as well as the hottest newest open-source packages and tools.
2. Professional experience writing production-ready Python code that leverages modern software development best practices (automated testing, CICD, observability).
3. Experience working on a team of developers, maintaining a shared codebase, and having your code reviewed before merging.
4. Strong DB Expertise in an Amazon environment (RDS, Postgres, and DynamoDB).
5. Strong ETL Experience (especially in extraction and ingestion of 3rd party data).

Other Requirements:

1. Familiarity with machine learning concepts.
2. Familiarity with asynchronous programming.

Responsibilities:

1. Onboard new geospatial datasets into our systems: imagery, parcels, extracted polygons, and build ETL pipelines to automate these processes.
2. Manage the interface between our AI systems and the input data they need to operate via streaming systems, storage systems, and caching systems.
3. Be aware of, and manage 3rd party provider rate limits, and architect our systems to gracefully deal with these limits while maximizing our throughput.
4. Participate in an agile product development process, where collaboration with stakeholders is a vital step to building what is needed.
5. Challenge and be challenged on a diverse, collaborative, and brilliant team.
6. Write automated test suites to ensure the quality of your code.
7. Contribute to open-source geospatial software.
8. Build solutions that enable new products -- typically involving large scale or intricate geospatial techniques.
9. Build-in system quality from the beginning by writing unit & integration tests and integrating with logging, metrics, and observability systems.

Voice.ai is a free AI-powered voice changer software that allows you to change your voice in real-time or transform any audio with a collection of thousands of AI voices. Works with Windows, Discord, Skype, Zoom, and many games.

Experience Requirements:

1. Proven experience as a data engineer, with a strong track record in data pipeline development and data warehousing.
2. Proficiency in data engineering technologies, including ETL tools, database systems, and data processing frameworks (e.g. Apache Spark).

Responsibilities:

1. Data Pipeline Development: Design and develop robust and scalable data pipelines for collecting, processing, and storing voice data from various sources.
2. Data Warehousing: Build and maintain data warehouses and data lakes to store and organize structured and unstructured voice data.
3. ETL Processes: Develop and optimize ETL processes to transform raw data into actionable insights, supporting the work of data scientists and machine learning engineers.

Akkio is an AI analytics platform built for advertising agencies, offering AI-powered insights and campaign optimization. It offers solutions for audience discovery, forecasting, and performance analysis.

Experience Requirements:

1. 5+ years of experience as a data engineer.

Other Requirements:

1. Must be authorized to work in the US.

Responsibilities:

1. Work with large datasets from a variety of customers, all with privacy concerns.
2. Operate in a fast-paced environment to bring customers value quickly.
3. Continuously optimize, standardize, and improve code and processes.
4. Build systems that run reliably, and report/alert reliably when they fail.

LXT provides high-quality AI training data solutions, including data annotation, collection, evaluation, and generative AI services, to power global AI innovation.

Experience Requirements:

1. Bachelor’s degree in computer science, Data Engineering, or a related field.
2. 4-6 years of hands-on experience in data pipeline design, data transformation, and data integration.

Other Requirements:

1. Proficiency in programming languages such as Python, SQL, or Scala, and experience with data manipulation libraries and frameworks.
2. Solid knowledge of data storage and database management systems, including relational and NoSQL databases.
3. Familiarity with data visualization tools and techniques to facilitate data understanding and analysis.
4. Experience with cloud-based data platforms, such as AWS, GCP, or Azure, is a plus.
5. Solid understanding of data quality and data governance principles.

Rockerbox unifies marketing measurement with MTA, MMM, and incrementality testing, centralizing data for a complete performance view.

Experience Requirements:

1. 7+ years of experience as a Data Engineer.

Other Requirements:

1. Deep expertise in AWS services (Redshift preferred).
2. Proficient with Python and have experience working with Kubernetes (K8s).
3. Comfortable ingesting data from third-party APIs and familiar with modern orchestration tools (e.g., Airflow).
4. Comfortable with Infrastructure as Code (e.g., Terraform).
5. Experience mentoring engineers and enjoy sharing your knowledge, particularly in areas like data warehousing, pipelines, and architecture.
6. Product mindset, balancing technical excellence with business needs to deliver scalable solutions.
7. Communicate complex topics clearly, whether you’re writing technical specs or aligning stakeholders around a project plan.

Responsibilities:

1. Scope and own projects to build, improve, and maintain scalable data ingests and pipelines.
2. Design and implement solutions that are resilient, well-monitored, and cost-effective.
3. Lead technical initiatives, working closely with Product Managers and cross-functional teams to develop clear, detailed technical specifications.
4. Share knowledge with your peers through design and code reviews, fostering a culture of continuous learning.
5. Stay curious—evaluate emerging technologies and tools to improve efficiency and effectiveness in our engineering processes.

AppZen uses AI to automate accounts payable and expense management, improving efficiency and reducing fraud.

Education Requirements:

1. Bachelor's or master's degree in Computer Science, Engineering, or a related field.

Experience Requirements:

1. Strong proficiency in Python and SQL for data manipulation, analysis, and scripting.
2. Extensive experience with cloud platforms, particularly AWS, and working knowledge of services like EMR, Redshift, and S3.
3. Solid understanding of data warehousing concepts and experience with relational databases like PostgreSQL.

Other Requirements:

1. Experience with building and maintaining data pipelines using tools like Airflow.
2. Knowledge of Python web frameworks like Flask or Django for building data-driven applications.
3. Strong problem-solving and analytical skills, with a keen attention to detail.
4. Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Responsibilities:

1. Design, develop, and implement scalable and efficient data pipelines in the cloud using Python, SQL, and relevant technologies.
2. Build and maintain data infrastructure on platforms such as AWS, leveraging services like EMR, Redshift, and others.
3. Collaborate with data scientists, analysts, and other stakeholders to understand their requirements and provide the necessary data solutions.
4. Develop and optimize ETL (Extract, Transform, Load) processes to ensure the accuracy, completeness, and timeliness of data.
5. Create and maintain data models, schemas, and database structures using PostgreSQL and other relevant database technologies.

Harmonya uses AI to enrich product data, provide insights, and manage attribution for CPG and retail companies.

Experience Requirements:

1. At least 6 years of experience in software engineering in Python or an equivalent language.
2. At least 3 years of experience with data engineering products from early-stage concept to production rollouts.
3. Experience with cloud platforms (GCP, AWS, Azure), working on production payloads at large scale and complexity.
4. Hands-on experience with data pipeline building and tools (Luigi, Airflow etc), specifically on cloud infrastructure.

Other Requirements:

1. Advantage: Hands-on experience with relevant data analysis tools (Jupyter notebooks, Anaconda etc).
2. Advantage: Hands-on experience with data science tools, packages, and frameworks.
3. Advantage: Hands-on experience with ETL Flows.
4. Advantage: Hands-on experience with Docker / Kubernetes.

Responsibilities:

1. Design and build data acquisition pipelines that acquire, clean, and structure large datasets to form the basis of our data platform and IP.
2. Design and build data pipelines integrating many different data sources and forms.
3. Define architecture, evaluate tools and open source projects to use within our environment.
4. Develop and maintain features in production to serve our customers.
5. Collaborate with product managers, data scientists, data analysts and full-stack engineers to deliver our product to top tier retail customers.

DataNimbus provides AI-powered data management and integration tools, including a cloud-native ETL designer and a comprehensive data platform, designed for seamless Databricks integration and accelerated AI adoption.

Opportunity to work on cutting-edge data and AI projects.

Collaborative and innovative work environment.

Competitive salary and benefits package.

Professional growth and development opportunities.

Experience Requirements:

1. 4+ years experience in data engineering, data architecture, data platforms & analytics.
2. At least 3+ years experience with Databricks, PySpark, Python, and SQL.

Other Requirements:

1. Consulting / customer facing experience, working with external clients across a variety of industry markets.
2. Comfortable writing code in both Python and SQL.
3. Proficiency in SQL and experience with data warehousing solutions.
4. Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one.
5. Strong understanding of data modeling, ETL processes, and data architecture principles.
6. Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals.
7. Familiarity with CI/CD for production deployments – GitHub, Azure DevOps, Azure Pipelines.
8. Working knowledge of MLOps methodologies.
9. Design and deployment of performant end-to-end data architectures.
10. Experience with technical project delivery – managing scope and timeline.
11. Experience working with clients and managing conflicts.
12. Good to have Databricks Certifications.
13. Strong communication and collaboration skills.
14. Ability to travel up to 30% when needed.

Responsibilities:

1. Handle a variety of impactful customer technical projects which may include designing and building reference architectures, creating how-to’s and productionalizing customer use cases.
2. Work with engagement managers to scope variety of professional services work with input from the customer.
3. Guide strategic customers as they implement transformational big data projects, 3rd party migrations, including end-to-end design, build and deployment of industry-leading big data and AI applications.
4. Consult on architecture and design, bootstrap or implement customer projects which leads to a customers’ successful understanding, evaluation and adoption of Databricks.
5. Support customer operational issues with an escalated level of support.

Oxa develops self-driving software and services for businesses, focusing on safety, efficiency, and AI-powered solutions for autonomous transportation.

Competitive salary, benchmarked against the market and reviewed annually.

Company share programme.

Hybrid and/or flexible work arrangements.

An outstanding £3,000 flexible benefits including private medical insurance, critical illness coverage, life assurance, EAP, group income protection.

Funded relocation support.

Education Requirements:

1. Degree in Computer Science, Mathematics or a related field.

Experience Requirements:

1. 3+ years of professional experience developing behavioural machine learning technologies for autonomous vehicles or robotics.
2. Experience with production ML pipelines: data creation and curation, training frameworks, evaluation pipelines.
3. Fluency in Python and experience with data analysis libraries and packages.
4. Proven record of leading and delivering projects as part of a team.

Responsibilities:

1. Take a leading role within your team to develop and deploy state of the art data pipelines for our machine learning models.
2. Design and implement metrics for model validation and continuous monitoring in production.
3. Leverage the Oxa Metadriver platform to generate synthetic data, and train effective and robust driving policies.
4. Build cloud tooling and infrastructure in support of experimentation, evaluation and deployment workflows.
5. Engage with team members and colleagues throughout the business to create an environment that supports collaboration and mutual understanding.

Affine uses AI to solve complex business challenges, offering solutions across various industries and AI domains.

AI-powered keyboard enhancing creative expression and task completion, offering solutions for both consumers and businesses.

Education Requirements:

1. Bachelor's degree in Computer Science Technology, or a related field.
2. Master's degree in Computer Science Technology or computer application is an added advantage.

Experience Requirements:

1. 3-6 years of experience in Data Engineering, Database management, Data structures & ETL development.

Responsibilities:

1. Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL, EMR/Snowflake, Redshift/Bigquery and AWS/GCP ‘big data’ technologies.
2. Maintaining the high volume and compute data pipelines and ensuring that process is not interrupted.
3. Hands on Hadoop, SPARK and other relevant Big Data technologies.
4. Develop algorithms to transform data into useful, actionable information.
5. Build, test, and maintain database pipeline architectures.

AI-powered weather forecasting for energy trading, offering high-accuracy predictions up to 31 days in advance.

Time off (minimum 25 days paid vacation).

AI-powered news intelligence platform providing global news aggregation, NLP enrichment, and advanced analytics.

Annual leave, plus national holidays + your birthday off!

Volunteer Day off.

Experience Requirements:

1. 4+ years of industry experience in data engineering.
2. Proficiency in Scala, Java, Python, or similar.
3. Expertise building and deploying data processing systems.
4. Experience with modern development tooling and DevOps technologies.

Responsibilities:

1. Write efficient and fault-tolerant code.
2. Configure and deploy Quantexa software.
3. Act as a trusted source of knowledge for clients.
4. Collaborate with solution architects and R&D engineers.

VI is an AI-powered platform for health organizations that helps maximize member health outcomes and financial returns by improving acquisition, engagement, and retention.

Education Requirements:

1. BSC in a related technical field or equivalent practical experience.

Experience Requirements:

1. Over 5 years of experience in a Data Engineering role.

Other Requirements:

1. Coding experience with Python.
2. Experience with SQL and NoSQL databases.
3. Experience with data modeling, and building ELT/ETL pipelines.
4. Experience with common data warehouse and lakehouse technologies.
5. Experience with AWS platform.
6. Experience working with Git, CI/CD flows and Docker.
7. Experience building data pipelines with big data frameworks such as Spark.
8. Technologically diverse background and ability/willingness to learn new things quickly.

Responsibilities:

1. Design, develop and implement robust data pipelines using ETL/ELT to move data from various sources into our lakehouse.
2. Ensure quality and integrity of data by implementing validations, testing and monitoring.
3. End-to-end feature development and ownership, from design to production.

GoodNotes is an AI-powered note-taking app offering a seamless digital pen-and-paper experience across multiple platforms.

Meaningful equity in a profitable tech startup.

Budget for things like noise-cancelling headphones, setting up your home office, personal development, professional training, and health & wellness.

Sponsored visits to our Hong Kong or London office every 2 years.

Company-wide annual offsite (we met in Portugal in 2023 and Bali in 2024).

Flexible working hours and location. Medical insurance for you and your dependents.

Experience Requirements:

1. 7+ years of software engineering experience, notable part of it being in the data engineering space.
2. Expertise with designing and building data pipelines within the ETL and ELT paradigm.
3. Expertise in distributed systems, as well as different database systems and big data solutions.
4. Previous experience in designing, building and maintaining cloud infrastructure to support analytics operations.
5. Experience with working in Python, Spark and SQL.
6. Experience with some of the following tools: dbt, Airflow, Kafka, AWS Glue, delta lake.
7. Excellent problem-solving skills and ability to think critically.
8. Desire to work in a fast-paced, collaborative environment.
9. Excellent communication skills, both verbal and written.

Responsibilities:

1. Build and maintain end-to-end data pipelines.
2. Design and build components within data pipelines for collecting production data.
3. Data engineering for efficient ingestion of data.
4. Helping the team build and maintain analytics and insights infrastructure.
5. Monitoring the production and setting up alerting mechanisms.

Wizard is an AI-powered shopping assistant that uses natural language processing and machine learning to provide personalized product recommendations and streamline the shopping experience.

Competitive compensation packages, including equity.

Health, dental and vision insurance.

Mental healthcare support and services.

401(k) plan.

Education Requirements:

1. Bachelor's degree in Computer Science or a related field, or equivalent practical experience.

Experience Requirements:

1. 5+ years of professional experience in software development with a strong focus on data engineering.
2. Proficiency in Python with experience implementing software engineering best practices.
3. Strong expertise in building ETL pipelines using tools such as Apache Spark, Databricks, or Hadoop.
4. Solid understanding of distributed computing and data modeling for scalable systems.
5. Hands-on experience with NoSQL databases like MongoDB, Cassandra, DynamoDB, or CosmosDB.

Other Requirements:

1. Proficiency in real-time stream processing systems such as Kafka, AWS Kinesis, or GCP Data Flow.
2. Experience with Delta Lake, Parquet files, and cloud platforms (AWS, GCP, or Azure).
3. Familiarity with caching and search technologies such as Redis, Elasticsearch, or Solr.
4. Knowledge of message queuing systems like RabbitMQ, AWS SQS, or GCP Cloud Tasks.
5. Advocate for Test-Driven Development (TDD) and experienced in using version control tools like GitHub or Bitbucket.

Responsibilities:

1. Develop and maintain scalable data infrastructure to support batch and real-time data processing with high performance and reliability.
2. Build and optimize ETL pipelines for efficient data flow and accessibility.
3. Collaborate with data scientists and cross-functional teams to ensure accurate monitoring and insightful analysis of business processes.
4. Design backend data solutions to support a microservices architecture, ensuring seamless data integration and management.
5. Implement and manage integrations with third-party e-commerce platforms to enhance Wizard's data ecosystem.