Somewhere in the Greater Toronto Area (GTA), nestled in a nondescript building, is one of Canada’s most powerful supercomputers.
Its presence is not advertised, and BetaKit was asked to avoid any details that might be used to specify its exact location, due to security concerns, and because it is home to close to $200 million CAD worth of chips and hardware.
Supercomputers are designed to handle massive datasets and complex calculations at very high speeds, far exceeding the computational capabilities of their general-purpose peers.
Daniel Gruner, SciNet
“There isn’t a single field of science today that does not need compute, and oftentimes, a lot of compute.”
This cluster, which is funded by the Canadian and Ontario governments, owned by the University of Toronto (U of T), and operated by SciNet, is one of five sites of its kind across Canada (the other four are in Burnaby, Victoria, Kitchener-Waterloo, and Montréal). The GTA hub’s mandate is “to be able to do the really big jobs,” SciNet CTO Daniel Gruner told BetaKit in an interview. It helps academics from coast to coast tackle large research analysis, modelling, and AI tasks in a wide range of domains, from astrophysics to aerospace, biomedicine, and climate science.
This specific supercomputer’s users range from the University of Victoria to Memorial University of Newfoundland. One group is using it to run simulations of the entire ocean, while another is leveraging it to model the internal dynamics of stars. Fitting and computing the data necessary to accomplish tasks like this requires lots of computers put together, Gruner noted.
“There isn’t a single field of science today that does not need compute, and oftentimes, a lot of compute,” Gruner said, noting that researchers need it to do everything from analyzing the human genome to ancient literature.
A rare glimpse inside
Concerns about Canada’s compute capacity and data sovereignty in the age of AI have made data centres and specialized computing systems a hot topic of discussion in recent years.
But despite all of the headlines, very few people have had the opportunity to see a supercomputer in person. So, in April, Gruner gave BetaKit a tour of the space, which he helped build, and which first came online in 2009.
The site itself is far from flashy. Its interior resembles a run-of-the-mill office, or school, replete with grey tiles, beige walls, and white ceilings. There is no decoration here, only utility. Gruner described it as like a factory.
The hub’s crown jewel—its power-hungry server racks—are found down a hall lined with cardboard boxes, past a small control centre where its 18-person team works and whose window is covered with various calculations scrawled in magic marker.
Inside a noisy, whirring room near the centre of the facility sit rows of hulking, black, fridge-sized, interconnected metal server racks that contain a bevy of densely-packed, interconnected nodes, chips, and colourful wires linking everything together. To the untrained eye, they may not look like much. But these are the tightly interconnected systems that make the magic happen.
Computing with them generates significant amounts of heat. Today, SciNet keeps them at the right temperature using direct liquid cooling. The surrounding mechanical systems send warm water directly down to a copper plate that sits above the chips, which can hit up to 90 degrees celsius, and extract the hot water that results.
This valuable hardware is located up a small set of stairs, and this placement proved critical a couple of years ago when a pipe burst and flooded the rest of the building.
RELATED: Canada opens applications to build a public AI supercomputer
The equipment requires substantial amounts of electricity to operate. This hub has four megawatts of power coming in—enough to power hundreds of homes at any given moment and a massive sum compared to the needs of a personal computer. Still, this is a pittance compared to the gigantic data centres run by the hyperscalers. Even so, at one point, Gruner said this was Canada’s most efficient data centre.
While the power supplying its systems is not as stable or as clean as SciNet would like, as a research supercomputing facility rather than one that caters to hospitals, banks, government, or other essential services, it can afford a sliver of downtime for maintenance so long as the data is preserved.
Whereas a laptop may have four to eight cores (or units of computing), SciNet’s latest system, Trillium, has more than 241,000. One current research group has tapped into it because it required access to 50,000.
Enabling bigger things
Trillium is three times more powerful than its predecessors. It came online last August to replace the facility’s aging Niagara and Mist clusters. It requires about the same amount of electricity despite having triple the computational power.
Gruner, who was born in Mexico, first came to Canada to obtain a PhD in theoretical chemical physics from U of T. Now, he and his team take satisfaction in making the work of other academic researchers possible.
His job is to keep Trillium operating at or near capacity at all hours of the day. “We’re always running flat out,” he said. During BetaKit’s visit, there were about 1,500 jobs sitting in the queue. AI research hub the Vector Institute locates most of its graphics processing units (GPUs) here. Demand for GPUs, the chips that support AI applications, is particularly high these days, Gruner said, and the hub is in the midst of a federally-funded upgrade designed to help address this need.
At the moment, Canada lags not only the US but many other countries when it comes to high-performance computing, and is the only nation in the G7 without a supercomputer ranked in the top 25 most powerful systems globally.
For context, Trillium ranks 142 globally, the second-fastest in Canada among academic supercomputers behind Simon Fraser University’s Fir at 87, and fifth overall. At 78, Telus’ Sovereign AI Factory in Rimouski, Québec, is the country’s most powerful supercomputer at the moment.
“Finland has a much bigger supercomputer than we do,” Gruner said. With a population of nearly six million, Finland is also “a much smaller economy” than Canada. “Admittedly, it’s part of the big European investment in compute, but we should be there, too.”
“If you can’t do research because you don’t have enough resources, it’s sad,” Gruner said. “It’s sad for a rich country. It’s sad for a country that touts itself as being advanced.”
However, Canada has changed its tune lately on this front, committing hundreds of millions of dollars towards bolstering existing infrastructure and establishing a new Canadian-owned public supercomputer. “It’s about time,” Gruner said.
RELATED: Queen’s University recruits Ian Karlin from Nvidia in bid to build Canadian supercomputer
With that supercomputer, the feds hope to improve domestic access to advanced computing for academic researchers and innovative businesses across the country at an even larger scale. Such a system would reduce the country’s dependence on the US and ensure sensitive data remains here, two driving factors amid rising geopolitical tensions. Applications for proposals to build and operate this supercomputer closed this Monday.
Gruner hopes that this appetite lasts and manifests in continued investment, noting that all computers eventually deteriorate and die, while faster, bigger replacements are developed. “You have to not just keep up, but you have to enable bigger things,” he said.
All images courtesy University of Toronto. Photos by David Lee.