• Maintaining and evolving our existing data platform;
• Building processes to ingest data from Kafka, APIs and Databases using AWS MSK Connect.
• Development and Maintenance of real-time data processing applications using frameworks like Spark Structured Streaming and Kafka Streams.
• Applying transformations to data streams;
• Being involved in data modelling following standards e.g. Inmon, Kimball, Data Vault;
• Ensuring data quality by verifying data consistency and accuracy;
• Keeping up to date on research and development of new technologies and techniques to enhance our data platform;
• Have an investigative mindset to be able to troubleshoot – thinking outside the box when it comes to troubleshooting problems and incident management;
• Full ownership on projects and tasks assigned together with being able to work within a Team;
• Able to document well processes and perform knowledge sharing sessions
###
Requirements
Must haves:
• Experience with modern cloud database technologies such as Snowflake;
• Experience with orchestrating data pipelines using Airflow;
• Experience with AWS Glue;
• Experience with Apache Iceberg;
• Experience with SQL, and Data Integration Tools
• Experience with any programming language like Python or Scala;
• Knowledge on AWS Services like S3/Lambda/API Gateways/DMS/RDS;
• Development experience in both Microsoft, and Linux/Cloud environments;
• Have strong analytical and problem-solving skills