Performance Evaluation of Pseudospectral Ultrasound Simulations on a Cluster of Xeon Phi Accelerators

Ultrasound simulations, Local Fourier basis decomposition, Pseudospectral methods, Ultrasound, k-Wave toolbox, Intel Xeon Phi, Knight's Corner, MKL, MPI, OpenMP

The ability to perform large-scale ultrasound simulations has generated significant interest in medical ultrasonics, including for treatment planning in therapeutic ultrasound, and image reconstruction in photoacoustic tomography.
However, routine execution of such simulations using traditional computational methods, e.g., finite difference time domain, is considered intractable due to the computational and memory requirements. The k-Wave toolbox alleviates these requirements by employing a k-space corrected pseudospectral method. This significantly reduces the spatial and temporal grid resolution, however, at the cost of introducing global all-to-all communication through the use of the fast Fourier transform. To improve data locality, reduce data movements and allow efficient use of accelerators, we recently implemented a domain decomposition technique based on local Fourier basis.

Nowadays, the trend in parallel computing is towards the use of accelerated nodes where computationally intensive tasks are offloaded from processors to accelerators. In this paper, we investigate the performance aspects of the distributed k-Wave implementation running on the Salomon cluster equipped with 864 Intel Xeon Phi (Knight's Corner) accelerators.

The paper shows that running large simulations across many accelerators is not a trivial task. The main obstacle is the instability of Intel MPI on Infiniband interconnection once the number of accelerators exceeds 32. Beyond this limit, the use of service 1Gbps interconnection is the only solution. The second problem is low performance of the fast Fourier transforms ranging from 1% to 50% of a single 12-core CPU. Finally, there is no support for fast LustreFS file system. Despite these factors, the observed strong and weak scaling is comparable with a cluster of CPU, but the absolute running time is between 4.3x and 1.8x longer in the worst and best case, respectively. However, the accounting policy for Salomon's accelerators is much more favorable and thus their employment reduces the computational cost significantly.