Responsibilities
• Develop and maintain the data platform's infrastructure using Terraform.
• Implement and manage CI/CD pipelines for the automation of data and ai processes and pipelines.
• Automate the provisioning, configuration, and monitoring of resources in AWS.
• Apply GitOpsprinciples to manage configurations and deployments.
• Ensure the security, scalability, and resilience of the infrastructure through automation.
• Collaborate with Data Engineers to optimize the data development lifecycle (DDLC).
• Own the automation and infrastructure of orchestration tools like Airflow.
• Document the automation architecture and deployment processes.
### Must haves:
• Solid experience with Infrastructure as Code (IaC) tools, especially Terraform.
• Experience in creating and managing CI/CD pipelines (e.g., Github Action, AWS CodePipeline).
• Deep knowledge of version control tools and workflows with GIT.
• Experience in automation scripting with Python, AWS CLI, Spark.
• Knowledge of AWS services automation and management.|
### Nice to haves:
• Familiarity with the AWS data ecosystem (S3, Glue, Lambda, Redshift, Snowflake).
• Familiarity with orchestrators like Airflow.
• Familiarity with data streaming technologies such as Kafka.
• Knowledge of monitoring and visualization tools (Prometheus, Grafana, CloudWatch).
• Basic knowledge of containerization (Docker, Kubernetes).
• Excellent verbal and written English communication skills.
###