Systems Software Engineer, Cluster Control and Automation

Employment: Full-time Experience: Mid-Senior level Education: Bachelor's Degree About CommonAI CommonAI CIC is a non-profit organisation built around collaborative engineering and is backed by a mix of UK Government and private funding in order to design, build and deploy innovative AI systems. Our Scaling Inference Lab (SIL) is developing simulation grounded in real measurements to help the UK invest in promising AI infrastructure. We connect proposed hardware designs to realistic workloads, helping investors and engineers understand their potential, their limitations and the next experiment worth doing. The Opportunity Build the software that lets SIL configure and control real AI infrastructure for reproducible experiments. You will turn the capabilities of our experimental clusters into programmatic controls for changes such as workload placement, resource allocation and inference-serving configuration. The engineering challenge is making those changes dependable: understanding the current state, checking whether an action can be carried out, confirming what actually happened and handling failures. Your software will make it possible to repeat experiments and understand which configuration produced a result. Working with measurement and simulation engineers and cluster operators, you will establish useful controls for the first experimental studies. Those interfaces will also provide the actions that future design-space optimisation software can call. What you will do - Identify the controls available on the selected clusters and expose them through documented, programmatic interfaces with clear supported actions. - Automate configuration and experiment setup across relevant parts of the stack, such as workload placement, serving settings, resource allocation or device configuration. - Build reliable execution with precondition checks, confirmation of the resulting state, error handling and recovery to a known configuration where supported. - Record requested actions, outcomes and resulting configuration so experiments can be reproduced and measurements interpreted correctly. - Coordinate access, permissions and changes with cluster operators, and test success, failure and recovery paths within an agreed experimental scope. - Design interfaces that future optimisation software can call, with clear capability limits and explicit outcomes when an action cannot be completed. An example early project would be to build an interface for changing serving configurations and demonstrate a repeatable experiment across two configurations, including checks of the applied settings and recovery from a failed change. Requirements - Experience building systems software or automation that configures and controls real computing infrastructure. - Strong programming and debugging skills, with practical understanding of Linux and the relevant hardware/software interfaces. - Depth in one or more of: cluster or HPC automation, distributed systems, orchestration, device control, networking, or experiment-control software. - The ability to reason about partial failure, state, permissions and the consequences of changing a running system. - Practical experience using AI coding agents, with the judgement to validate generated code and test changes that affect hardware or infrastructure. Experience with accelerators, job-control systems, Kubernetes, Slurm, operating systems (NixOS, CoreOS) or IaC tools (Terraform/OpenTofu) would be useful. Depth in a subset is sufficient; the initial controls and integration choices will follow the needs and capabilities of SIL's clusters. Why join us - Help build UK AI capability and improve how investment and compute serve the public good. - Build a network spanning UK hardware SMEs, major technology companies, venture and public investors, and leading universities. - Shape a growing team with startup pace and close academic links. - Use frontier models and coding agents with a substantial tooling budget. Benefits A collaborative and supportive work environment The opportunity to have a high impact in a growing organisation Competitive salary package, including pension and benefits Professional development opportunities Networking opportunities with influential figures from across the tech sector and academia A vibrant office environment located a few minutes’ walk away from Cambridge train station Working with us Hybrid, with regular time in Cambridge agreed individually around hardware work and collaboration; applicants across the UK are welcome. We are open to permanent employment, part-time arrangements and consultancy. We would like you to start as soon as practical and can be flexible for the right person. We particularly welcome experienced engineers but will also consider earlier-career candidates with strong evidence of ability. Relevant postgraduate study is useful, but qualifications are secondary to experience and the quality of your work. CommonAI CIC is an equal opportunity employer and is committed to creating an inclusive and diverse workplace.