UKCP History
[In conversation with Profs Volker Heine, Mike Payne and Paul Madden]
The UK Car-Parrinello Consortium (UKCP) owes its existence to the particle physicists and quantum chromo-dynamics – specifically the UKQCD consortium. In the very early 1990s, UKQCD saw the opportunity offered by parallel computing and a new generation of Intel processors and so made an application to SERC (the Science and Engineering Research Council – a single unified funding council and fore-runner to EPSRC, STFC, etc) for a 256 node Meiko machine. SERC thought this was a great proposal and agreed to fund UKQCD for 2/3 of 64 node machine! However, SERC also decided that (a) the remaining 1/3 of the machine should be paid for by its 'Science' Board, effectively physics and chemistry, and (b) this 1/3 should be used on some big spectacular project and not frittered away among a large number of users doing more or less what they were doing before. SERC originally asked Volker Heine to attend the meeting to discuss this, presumably anticipating that electronic structure might be a large project, and, maybe, having in mind the success of CCP9. However Volker was unable to go, and Dominic went instead. Apparently the work of Car and Parinello was featured on the cover of one of the most prestigious chemistry journals the previous month and so Dominic put that forward as something that would be of interest to both physicists and chemists (who could therefore both be expected to help produce the 1/3 cost from their budgets). Hence UKCP was born, and we all have a great deal to thank Dominic for on account of his wisdom.
The initial meeting of what was to become UKCP was hosted in Oxford by Paul Madden and was attended by Mike Gillan (then at Harwell), David Bird (Bath), Volker Heine and Mike Payne (Cambridge), and Dominic Tildesley (Southampton). Later this group was joined by Graeme Ackland (Edinburgh), partly in recognition of Edinburgh University funding 10% of the cost of the machine, and Adrian Sutton (Oxford) at the suggestion of Colin Humphreys who had put much effort into persuading the Materials programme in SERC to also invest in the project. I am sure our collective memories are rather hazy on all this but I seem to remember that significant parts of the first UKCP grant proposal £250,000 to fund the machine were written over a weekend at a time when none of us had any idea if Car-Parrinello calculations could be run on parallel machines. Not surprisingly, most of the proposed applications were wildly over-ambitious.
Volker adds: ‘You say "none of us had any idea if CP calculations could be run on parallel machines"; it was much worse than that! We thought desperately hard about how any electronic structure calculation could be run usefully on a parallel machine Indeed some of us, e.g. Walter Temmerman and myself, had already talked about it when the idea of parallel machines had appeared on the horizon. So we were somewhat in despair at the end of our first UKCP meeting when we could see no way forward: one could run a few k-points in parallel, but that was not going to get one far. Then at the start of the second meeting of UKCP Mike (Payne) announced "I see how to do it" and we were bowled over. The sigh of relief reverberated around Oxford's lofty spires. The trouble had been that in all our discussions we always thought of a Bloch function Ψk as one entity, and the conceptual breakthrough was to think of the plane wave coefficients as the entities to be parallelised.
As mentioned earlier, Edinburgh University paid for roughly 10% of the Meiko computer. The University already had an internationally recognised centre for parallel computing EPCC (Edinburgh Parallel Computer Centre) created and still led at that time by David Wallace. Once UKCP was created, he assigned an outstandingly talented young parallel programmer, Lyndon Clarke, to port the CASTEP code to parallel architectures – thus generating the CETEP code. The primary challenge in the parallelisation was finding an efficient implementation of the 3D- FFT (fast Fourier transform). Once this was done most of the operations in ‘Car-Parrinello’ calculations became localised in either real space or reciprocal space. One exception was the application of the non-local parts of the atomic pseudopotentials. Some reduction in cost had been achieved through the use of Kleinman-Bylander form for these potentials but, in their original reciprocal space implementation, the number of operations required scaled as the cube of the number of atoms in the unit cell and would have been the most costly part of any large calculation. Thus the development of real space projectors for non-local pseudoopotentials by Dominic King Smith was very important. His method made these parts of the calculation scale quadratically with the number of atoms in the system and, thus, orders of magnitude less expensive on large systems. Similarly, the use of stable, robust, efficient direct minimisation conjugate gradient schemes was important for the success of most of UKCP’s initial applications.
The 64 node Meiko i860 processor based machine, with communications handled by Inmos transputers, was purchased and installed in 1991. UKCP got access to the machine in the later part of the year after significant testing and re-engineering of the machine by Meiko, with considerable help from UKQCD. Ivan Stich was appointed as the first UKCP postdoc. There was no time sharing on the machine so UKCP had the machine for two slots during the week and UKQCD had the machine for the remainder of the time, including weekends. At the beginning of one of our productions slots, Mike Payne remembers sitting alongside Ivan Stich coding in real time to parallelise the Ewald routine – which without parallelisation was taking a prohibitive amount of time dealing with the 400 atoms in the unit cell used for our calculations on the silicon 7x7 Takayanagi reconstruction.
Interestingly, UKCP’s success was not due to the speed of The Meiko machine. It had only a 2.6GFlop theoretical ‘peak’ speed and delivered 1.3GFlop on the Top500 benchmark. It was ranked 220th in the first ever Top500 list published in June 1993, admittedly nearly two years after the machine was installed but the rate of decline down the Top500 list was much slower at that time than it is now. However, the machine had 1GByte of memory – something that was either impossible to provide or prohibitively expensive to put into the competing vector machines. The Portland Group BLAS libraries were outstandingly effective at getting good performance out of the Intel i860 processor and Lyndon Clarke developed a bespoke communications protocol that significantly enhanced the interprocessor communication speed which was crucial for UKCP’s work.
UKCP had a remarkably successful three years running on the Meiko machine. Indeed, the UKCP Consortium proved many of the capabilities of the plane wave pseudopotential methods which we now take for granted. The consortium has never received the international recognition of this success that it deserves. Furthermore, the consortium developed a general purpose parallel 3D FFT algorithm (or equally for 2D FFTs) – an operation that was previously believed to only possible on special purpose parallel machines such as the Connection Machine that had an interconnect topology that was designed specifically for this purpose. Amusingly, MCP discovered several years later that Thinking Machines (who made the Connection Machine) had started to use our algorithm for 3D FFTs as it was more efficient on their machines!! The success of UKCP contributed greatly to the realisation that parallel computers were (relatively) general purpose computers and by 1994 the debate was over and EPSRC chose to procure a parallel machine for the National (Supercomputer) Service.
Putting together this short account of the early days of UKCP gives us the opportunity to express our thanks to Dominic Tildesley. He guided UKCP through those early days, fought our political battles and continuously cajoled us to behave a bit more like professionals and less like academics and focus on results. He never claimed any personal credit for UKCP’s success and allowed those who ran the calculations to take all the glory – but all of us there in the early days know how much our productivity and success owed to Dominic’s influence and encouragement. We are truly grateful.
Volker Heine, Mike Payne and Paul Madden
February 2015