Scientific visualization is a collaborative (multi-disciplinary) environment where the computer scientists (or computer engineers) work together with domain scientists to solve large and cutting-edge problems on various science-related issues in a collaborative manner. Throughout various projects that I worked on, I had the oppurtunity to work on the following different data types:
GPS streams,
Acoustic scans of underwater plumes,
MRI scans,
Computational fluid dynamics simulations (with up to 4TB total data size),
Ocean simulations,
In many of these different domains, there are common problems that the scientists focus on: extracting, tracking and visualizing the objects of interests. My PhD work focused on dealing with such different datasets and extract objects of interests first (mostly in time-varying volumetric domains), track them over time, model and extract their particular interactions. Scientists typically have to deal with hard to understand and hard to edit C/C++ codes that typically come with a very limited comments (documentations). In my PhD dissertation, I proposed Token-tracking PetriNets to allow scientists to model their activities of interests in a graphical environment first, and to extract all the instances of all those models from the data. At the final stage, we visualize all those extracted results in different ways too allowing scientist to further perform analysis on his/her dataset.
Below is a brief description of my PhD dissertation with highlights and with sample videos.
Activity detection has been an active research area in computer vision. Its goal is automatically detecting the instances of the activities of interest, especially in (large) video streams. Consider the video shown on the left. This surveillance video shows a sample activity in which a person appears first, leaves his/her bag and then leaves. Automated detection of such activities is important to a security personel. Similarly, while the automated detection of such human-behaviour based activities are important to security personel, the interactions of certain scientific objects in time-varying scientific data sets are important to the scientists for data analysis and hypothesis generation purposes.
An activity is essentially a spatial and temporal (spatio-temporal) pattern. It may include a combination of various types of objects, various types of object interactions and object states. The terms "activity detection" and "activity recognition" have been interchangably used to describe the same purpose: searching through the time-varying data to extract all the instances of the activity of interest(s).
2) Activity Detection in Scientific Visualization
Activity detection concept does not only apply to video data where the (primary) object of interest is human. Rather, it is a generic concept and it applies to many fields. In particular, time-varying scientific simulations or datasets also contain activities. The ability to detect activities (i.e., the detection of spatial & temporal patterns) in such scientific simulations would give the oppurtunity to analyze their data beyond focusing on only single time-steps. In activity detection framework, a scientist can model a spatial and temporal pattern as a time-sequence, and then can search in large time-varying data sets to extract all the instances of that pattern of interest. Once the activities are detected, they can be used for further data analysis or hypothesis refinement.
Providing such tools to the scientists would allow them to analyze their data and/or hypothesis in large time intervals. In this work, we introduce the concept of activity detection to the scientific visualization community and propose the use of Petri Nets for activity modeling, detection and visualization.
As an example, consider the videos shown on the left. The first video shows state-of-art approach before our activity detection framework. In this approach, time varying (individual) objects (features) are extracted,tracked, and their certain attrbiutes are computed for visualization.
The data set contains a Computational Fluid Dynamics (CFD) simulation formed of 100 time steps. The variable being visualized is the vorticity magnitude. The first video visualizes the evolution of all the extracted vortices over the time.
Now in this data set, assume that you are not interested in seeing all the vortices, and instead, you are looking for only the vortices performing a specific activity. Consider that activity is the "merge-split" activity in which individual vortices merge first, and then this new "combined feature" splits again within the next 5 time steps. Searching manually for all the instances (or maybe even for a single instance) of this activity is a hard task even in this small data set. An automated process would extract all the instances of the activity easily. Therefore, in this project, we look for a way to describe, model an activity first, and then extract all the instances of such activity within the data set. The second video highlights the instances of the "merge-split" activity found in the data set using our technique.
Similar to the computer vision field where the purpose is mainly analysing the video data of humans, there are groups in scientific data sets. A group is fundamentally a set of objects that are related "somehow". This relation can be only logical (such as a group of all green objects) or also physical (such as a flock of birds, a school of fish or a galaxy of stars). The definition of a group changes from domain to domain, and this makes it harder to find a generic framework that could help scientists to define, extract and follow the evolution of groups in their data sets. Moreover, as the data dimensions increase, it becomes more and more apparent that smarter and more meaningful abstractions are necessary in large data visualization.
In this work, to help with the above-mentioned problems, we propose using similarity functions to define a (physical) group in scientific visualization. Similarity functions can map the physical group definitions in scientific domains into the computational domain. Once a similarity function is set (defined), a clustering algorithm can be used to extract the groups in scientific simulations based on the similarity function. When groups are determined, it is necessary to track them to understand their evolution over time (in time varying data sets). Therefore in this study, we proposed a group tracking schema in which we track groups of objects (features) as well as the invidual objects. Group tracking allows us to define and detect new type of events such as cross-group, partial or full merge and partial or full split.
The older versions of the feature (and group) tracking code are available on the old feature-tracking website (the link is on the right). The more recent (and fast working) version is available through Vizlab, Rutgers or through me.
In this test example, the input data is the processed computational fluid dynamics simulation. The data is pre-processed to extract the necessary attributes for modelling. The pre-processed data is shown at the top right section of the video. The activity is modelled as a Petri Net (the model is shown at top-left section of the video). Every feature first falls in the first place (circle in the graphical model) named as "feature" the features that perform the "merge" action falls into the next place (circle) labelled as "merged feature". If this new merged feature splits into multiple features again within the next 5 time steps, then it falls into the red place (circle) which is the final state that defines the completed activity. All the features that fall into the red circle are the features that "performed" the entire activity. The features (objects) performing the modelled activity are highligted in the bottom right section of the video (the top right section shows all the extracted features).
The Petri Net model also visualizes the status of each feature at each time. The features are color coded tokens (dots) in the model.
5) Sample Results: Stable Eddy detection in ocean simulations "
In ocean simulations, scientists are interested in studying eddies and their behaviours. Among those, one particular interest of the scientist was detecting and studying the stable eddies. These eddies are the ones that exist in the data for more than a couple of days. This becomes a tedious task if the correct tools are not available. However, with the correct tools, this task can be achieved. Since this task, is a spatial and temporal pattern, this activity can be modelled and detected by our activity detection framework.
The video on the left shows the Petri Net model and its results in the ocean data. The Petri Net model is shown at the top and it detects the eddies that exist longer than 7 days. The detected eddies are highlighted in the bottom half of the video. All other eddies that do not exist longer than 7 days are shown transparent in the video.
Data Dimensions: 722x362x40, 256MB/day, 93GB/year (~365 time steps for each year since 1950)
Data owner- Collaborator: Dr. Enrique N. Curchitser (Rutgers Univ.)