Fast Just Got Faster: Researchers Launch Hermes Supercomputer Software
The world of high-performance computing can expect to see a boost in data flow efficiency with the launch of new input/output software developed by researchers at Illinois Institute of Technology, with collaborators from the HDF Group.
Hermes is a new deep memory and storage hierarchy software system designed to reduce bottlenecks in data movement from storage to memory in high-performance computing, which improves input/output performance. Improvements in microprocessors, data storage, and memory hardware have drastically increased the amount of data that supercomputers can process, but they also have resulted in traffic jams that occur when moving data from storage to memory and to the microprocessor.
“Different systems and different technologies all also run different software,” says Anthony Kougkas (Ph.D. CS ’19), Illinois Tech research assistant professor of computer science and lead researcher on the project. “Ideally the presence of those multiple tiers of storage should be transparent to applications without having to sacrifice I/O performance. With this problem in mind, we set out to create a team to address this problem.”
Kougkas led a webinar outlining Hermes’s capabilities as part of the software launch on March 26, with presentations from Hermes team collaborators Christopher Hogan, HDF Group software engineer; Kimmy Hu, HDF Group HPC lead engineer; Hariharan Devarajan, a Ph.D. student in Illinois Tech’s Scalable Computing Software Lab; and Gerd Heber, HDF Group lead engineer. Illinois Tech Distinguished Professor of Computer Science and Ron Hochsprung Endowed Chair Xian-He Sun is the principal investigator for Hermes, which was funded through a grant from the National Science Foundation.
The team described how Hermes organizes and accesses data from a variety of local and remote storage devices and moves it efficiently into memory, where the data can be processed and ultimately easily help people get increased performance from their data-analyzing applications.
“The problem is all of [the storage devices] are independently designed, deployed, and managed, making it hard for the end user to manage the complexity of hierarchical storage,” Kougkas says. “Hermes holistically manages this storage system and brings the benefit of each [storage component] to the end user.”
The Hermes software allows users to draw data from various storage hardware, organize it, and send it to memory using a single software package. This requires the user to write less code with a universal software package, rather than using different software for each storage device.
This improves the speed of moving data from storage to memory, reducing the input/output performance gap that exists with current systems. Although Hermes is fully functional and ready for researchers in high-performance computing to use, Kougkas says the open-source software is still “in development,” and hopes the community will continue to offer their feedback to improve the package.
NSF Program Director Seung-Jong Park attended this Webinar, with 75 other interested users.
