Requirements
- 3+ years of experience in Data Engineering, building and maintaining production-grade data pipelines and data processing solutions.
- Strong SQL skills and experience working with large datasets.
- Strong programming skills in Python or Java.
- Hands-on experience with Apache Spark and Apache Airflow.
- Experience with Dremio and modern enterprise data platforms.
- Experience working with Google Cloud Platform (GCP) and cloud-based data solutions.
- Solid understanding of data modeling concepts, data integration and data processing patterns.
- Experience designing and developing batch and real-time data pipelines.
- Understanding of ETL/ELT processes and data transformation techniques.
- Experience with data quality, monitoring and observability practices.
- Familiarity with Git, CI/CD pipelines and modern software engineering practices.
- Good understanding of DataOps principles and best practices.
- Ability to collaborate effectively with technical teams, architects and business stakeholders.
- Experience with Data Mesh, Scala, Kafka or other streaming technologies, as well as banking or insurance projects, would be an advantage.
- Professional English communication skills.
Responsibilities
- Design, develop, maintain and optimize scalable batch and real-time data pipelines.
- Build and enhance data processing workflows using modern data engineering frameworks and technologies.
- Develop and maintain ETL/ELT solutions to support data ingestion, transformation and integration across multiple source systems.
- Implement efficient data processing solutions using Apache Spark and other distributed processing technologies.
- Develop and maintain data workflows and orchestration processes using Apache Airflow.
- Work with Dremio to support data access, querying and data virtualization requirements.
- Contribute to the development and continuous improvement of an enterprise-grade data platform running in a cloud environment.
- Design and implement data integration solutions, ensuring consistency, reliability and data availability across different systems.
- Apply data modeling principles to structure and optimize data for analytical and operational use cases.
- Implement and maintain data quality checks, monitoring and observability mechanisms.
- Optimize data pipelines and processing workflows to improve performance, scalability and resource utilization.
- Collaborate with cloud and platform engineering teams to support cloud-native data platform capabilities.
- Follow CI/CD practices and contribute to automation, testing and deployment processes.
- Troubleshoot data pipeline issues, investigate failures and ensure the reliability of production data workflows.
- Collaborate with architects, engineers and business stakeholders to understand requirements and translate them into effective technical solutions.
- Contribute to technical documentation, engineering standards and DataOps best practices.
- Identify opportunities to improve existing data solutions, processes and platform capabilities.