I participated in the 4th Visualize This! challenge, a Canada-wide competition that “aims to celebrate the innovative ways visualization can help researchers explore datasets and answer important scientific questions”. It was hosted by Compute Canada regional partners and the Compute Canada Visualization Team (which I joined the following year). My submission, that won second place, explores the flow around an airfoil from a particularly large computational fluid dynamics simulation (performed by Joshua Brinkerhoff’s team at UBC Okanagan), in particular the separation of the laminar boundary layer from the airfoil and its transition to turbulence. My visualization was a demonstration of the VTK-m library (Visualization Toolkit–many cores), an open source C++ library for scientific visualization maintained by Kitware. I chose to experiment with this library because it was specifically designed for the GPU-accelerated HPC environment. This experiment was only partially successful, it turned out the library was not mature enough at the time to achieve my goals.
About VTK-m
VTK-m is similar to VTK in many respects but is a separate codebase and is derived from different older projects. The developers describe it as “algorithms for emerging processor architectures”; the idea is for the end user as well as the visualization expert to write hardware-agnostic code, and the library uses optimized algorithms at the backend. Some functionality of VTK-m could be used through ParaView and VisIt.
While VTK-m supports MPI through a class called PartitionedDataSet, many if not most of its builtin algorithms do not know how to handle this class. The solution was to use yet another library called VTK-h (h for hybrid) that is basically a wrapper around VTK-m that facilitates much of the inter-process communication without much user intervention, and importantly, creates a composite image from the data stored on each rank.
The visualization
The video visualizes four quantities from the dataset using four different rendering techniques.
- The pressure is shown using the slicing technique, where a two-dimensional plane scans through the volume, showing the pressure levels in different slices.
- The velocity field is shown using the particle advection method, where particles are randomly generated and their trajectories are integrated and shown in the rendered image as lines. In the first part, I let the initial position scan through the y-axis (the relevant parts of it, that produces a turbulent flow downstream) but only using the first snapshot. In the second part, the particles span a range along the y-axis, and each frame represents a different point in time. Interpolation is performed on the dataset such that each frame uses unique velocity data in the advection filter.
- The Q criterion is shown using isosurfaces with values of –500 and +500. Here again each frame (except the first) represents an interpolation between two snapshots in the dataset.
- Finally, the z-component of the vorticity vector is shown using volumetric ray casting. This technique is more frequently used to explore density fields, but as the density in this dataset is constant, I needed to find a different quantity to showcase this capability. The z-component of the vorticity vector shows nice coherent structures above and below the airfoil, it is especially interesting that there is a gap between the lower structure (pointing toward positive z) and the airfoil itself.
Technical issues
Unfortunately, my attempt was plagued with technical issues, and in combination with lack of time, I was not able to run my pipeline with GPUs on multiple nodes (which was the entire point of using these libraries in the first place). Currently VTK-m is used by a small community for seemingly very specific purposes, and it was very hard coming from outside and trying to apply it for my needs. More than that, VTK-h is completely undocumented (or was at the time) so there was a lot of guesswork and trial and error. I wholeheartedly do not recommend these except for development or curiosity purposes.
A relatively major part of this project was writing C++ code to read OpenFOAM data and convert them to the VTK-m native representation in memory. I had to write this code from scratch after delving into the documentation of both OpenFOAM and VTK-m. It is thanks to this control over the data that I was able to downsample the dataset and transfer it to my workstation, and get any visualization at all in the little remaining time.