Data Engineer

Acesoft Labs; Bengaluru; Hybrid; INR 1,200,000 - 2,100,000; Full time

Job summary

Acesoft Labs in Bengaluru, India, is seeking a Data Engineer to design, build, and maintain scalable data infrastructure across cloud platforms and big data technologies.

You will collaborate with data scientists and analysts to ensure data availability, quality, and governance, and work with Spark, Hadoop, Airflow, Kafka, and cloud services.

The role emphasizes ETL/ELT pipelines, data lakes and warehouses, real-time processing, and secure data APIs.

Qualifications

• Bachelor's degree in computer science, information systems, engineering, or related field.
• Proven experience in building and managing data pipelines and infrastructure.

Responsibilities

• Design, build, and maintain scalable data infrastructure across cloud platforms.
• Develop and maintain ETL/ELT pipelines for structured and unstructured data.
• Collaborate with Data Scientists and Analysts to ensure data availability and quality.

Skills

Data engineering

ETL/ELT design

Distributed systems

Cloud platforms

Big data

Education

Bachelor's degree in Computer Science/Information Systems/Engineering

Tools

Apache Spark

Hadoop

Airflow

Kafka

Flink

NiFi

Beam

Snowflake

Redshift

BigQuery

Azure Synapse

Job description

Job Description

Data Engineer

Job Summary: We are seeking a highly skilled and detail-oriented Data Engineer with expertise in data architecture, pipeline development, cloud platforms, and big data technologies. The ideal candidate will be responsible for designing, building, and maintaining scalable data infrastructure, ensuring efficient data flow across systems, and enabling advanced analytics and machine learning capabilities.

Key Responsibilities

• Good to Have Palantir Foundry Experience:
• Experience working with Foundry Ontology to model enterprise data and define semantic relationships.
• Building and maintaining Code Workbooks and Data Pipelines within Foundry for scalable ETL/ELT workflows.
• Familiarity with Foundry's Object Explorer and Data Lineage tools for tracking data transformations and dependencies.
• Integration of Foundry with external systems using Foundry APIs and Data Connections.
• Experience with Foundry's Operational Workflows for automating data-driven decision-making.
• Understanding of Foundry's security model and access control for enterprise-grade data governance.
• Design, develop, and maintain ETL/ELT pipelines for structured and unstructured data.
• Build and optimize data lakes, data warehouses, and real-time streaming systems.
• Collaborate with Data Scientists and Analysts to ensure data availability and quality for modeling and reporting.
• Implement data governance, security, and compliance protocols.
• Develop and maintain data APIs and services for internal and external consumption.
• Work with cloud platforms (AWS, Azure, GCP) to deploy scalable data solutions.
• Monitor and troubleshoot data workflows, ensuring high availability and performance.
• Automate data validation, transformation, and integration processes.
• Manage large-scale datasets using distributed computing frameworks like Spark and Hadoop.
• Stay updated with emerging data engineering tools and best practices.

Technical Skills

• Programming & Frameworks:
  • Languages: Python, SQL, Scala, Java
  • Frameworks & Tools: Apache Spark, Hadoop, Airflow, Kafka, Flink, NiFi, Beam
  • Libraries: Pandas, PySpark, Dask, FastAPI, SQLAlchemy
• Cloud & DevOps:
  • Platforms: AWS (Glue, Redshift, S3, EMR), Azure (Data Factory, Synapse), GCP (BigQuery, Dataflow)
  • DevOps Tools: Docker, Kubernetes, Jenkins, Terraform, Git, GitHub
• Databases:
  • Relational: MySQL, PostgreSQL, SQL Server
  • NoSQL: MongoDB, Cassandra, DynamoDB, Redis
  • Data Warehousing: Snowflake, Redshift, BigQuery, Azure Synapse
• Data Architecture & Processing:
  • ETL/ELT design and implementation
  • Batch and real-time data processing
  • Data modeling (Star, Snowflake schemas)
  • Data quality and lineage tools (Great Expectations, dbt, Amundsen)
• Monitoring & Visualization:
  • Prometheus, Grafana, CloudWatch
  • Integration with BI tools like Power BI, Tableau

Qualifications

• Bachelors or Master’s degree in Computer Science, Information Systems, Engineering, or related field.
• Proven experience in building and managing data pipelines and infrastructure.
• Strong understanding of data architecture, distributed systems, and cloud-native technologies.
• Excellent problem-solving and communication skills.