Python Fullstack Data Engineer

Project Description:

You'll keep the Platform AI - Recommender service running reliably, securely and in line with compliance requirements. We are a shared recommendation service that powers personalised suggestions for jobs, courses, careers and learning content across public-sector products, including MyCareersFuture, MySkillsFuture and CSC Learn.

The platform is already live and it runs both traditional ML models and an LLM-based recommendation. You'll also help agency teams build on top of our service and operate on the platform smoothly.

Key Responsibilities and Outcomes:

• Platform reliability: Maintain the recommendation APIs and services. Handle bug fixes, refactoring and small enhancements, review PRs, monitor production health, lead incident response and root-cause analysis, manage upgrades and patches, and keep GCC hosting costs under control.
• Data pipelines and batch jobs: Maintain the pipelines that feed the recommenders (ingestion, feature and embedding generation, scheduled jobs). Oversee batch jobs such as data refreshes, model retraining and bulk recommendation generation. Set up monitoring and data quality checks, and troubleshoot issues in SQL and PySpark on Databricks.
• Security and compliance: Own the platform's security posture under government ICT policies (e.g. IM8). Coordinate VAPT and code or dependency scans and remediate findings on time. Prepare documentation for security assessments, audits and AI governance clearances. Ensure sensitive personal data is handled correctly.
• Documentation and agency support: Keep runbooks, architecture docs and integration guides current, and act as the technical point of contact for agency engineering teams.

What we're looking for:

• 5+ years of professional software engineering experience, with strong backend skills in Python
• Experience running production APIs and services on cloud (AWS preferred) with Docker and Kubernetes, including monitoring, alerting and incident management
• Hands-on experience with data pipelines and scheduled batch jobs, with SQL and Spark/PySpark required
• Solid grounding in secure application practices, and experience remediating security findings (e.g. from VAPT or scans)
• Able to pick up an existing codebase quickly with little guidance, and actively uses agentic AI coding tools

Nice to have:

• Databricks or Airflow
• ML or LLM systems in production
• Recommender systems experience