In a previous post I presented a new tool developed to assess possible problems with datasets. But, you might ask, why is this tool necessary at all? Well, because without such a tool we risk making erroneous assumptions based on the data.

In this post I want to highlight a problem that can be caused by topological sparsity. When data is not internally well-connected, it can lead to inaccurate findings, even with well-specified statistical frameworks. Specifically, Bayesian and Frequentist models, due to their architecture, can smooth away existing signals or inflate isolated signals, leading to what I call the Topological Mirage: where models provide results that are artifacts of incorrect treatment of data. I go into more detail on this in a newly submitted paper, but here I highlight two experiments (which you can replicate for yourself) to illustrate the problems.

Experiment 1: Network degradation

As I was doing investigations for a different project, I noticed that models behave differently depending on the internal connections of the data in a given dataset. I observed that in datasets with low topological stability, there were estimation issues. So I decided to measure this using the Topological Stability Index (TSI) for different datasets, in order to see how connectivity affects estimation in standard statistical frameworks.

To do this systematically, I took a stratified sample of 1,000 nodes/species from a biological phylogeny (specifically the ‘amphibians’ tree) that was used in the Munstermann et al (2022) study. On a logarithmic scale, over 21 steps, I pruned the tree, cutting network connections to increasingly isolate nodes. I then ran two Bayesian models on the data with binary dependent and independent variables - one model that treats isolated nodes as missing information (NaN) and a “routed” framework that treats them as independent data points. At each step, I measured the TSI, energy signature (Energy Bayesian Fraction of Missing Information; E-BFMI), estimate uncertainty, and number of divergences for each model. Results are presented in the figure below.

This image shows a lot of different things. First, in the top two panels we can see that the unrouted framework becomes increasingly unstable as the network degrades, shown by the tracked divergences. We also see that as the TSI drops, the E-BFMI does too, with both crossing the Critical Horizon identified for this dataset at step 4, and dropping below the “safe” energy floor of 0.3 by step 7.

The second set of panels show estimates for all 1,000 nodes at critical steps, and violin plots for the same steps. Here we see that the unrouted framework waffles between uncertainty and strong certainty, collapsing estimates entirely toward the global mean by the end of the experiment. The routed framework preserves uncertainty longer, providing more stability across steps.

Experiment 2: Sparsity and low TSI

As I mentioned in my previous post, low TSI does not necessarily mean that your estimates will be wrong. But in specific datasets, the lack of network connectivity can result in a Topological Mirage, creating estimation artifacts. Bayesian practitioners are generally aware of the potential problems with shrinkage and funnels, and the fact that Bayesian models pull outliers toward a global mean is often viewed as a strength, since it allows them to provide results where datasets are sparse (as opposed to many Frequentist models, which tend to simply fail from lack of data/connectivity).

In order to investigate how data sparsity interacts with disconnected datasets, I implemented a second experiment. Here I kept the TSI of the 1,000 data points at the stage where it can trigger parameter collapse (step 7 from experiment #1). To make this even more similar to real-world data, I simulated world-based geographic coordinates for each data point, with corresponding geographically-based grouping of members in the same tree. Then I varied the relative number of data points in a binary dependent relationship, where X implies Y. This could be any situation where a counterexample for X would invalidate the implication.

As an example, we might observe that most cold-blooded animals lay eggs, so we form the hypothesis that if an animal is cold-blooded, it lays eggs (cold-blooded > egg-laying). To invalidate this hypothesis, we only need a few counterexamples, which are present in viviparous reptiles and some shark species among others. But can our statistical model detect this? After coding the data, we can systematically increase the number of counterexamples in the dataset, and run models at each step to see how the models perform. Importantly, this is a heavily skewed dataset, where the majority of the data points have no relationship (i.e. ~750 warm-blooded, live birth; 230 warm-blooded, egg-laying) a small number show the relationship (12 cold-blooded, egg-laying) and there are a small number of counterexamples (1-10 cold-blooded, live birth).

To compare models, I implemented the “routed” architecture from the tree pruning experiment (which preserves stability at low TSI) in two Bayesian models and a Frequentist model for each step. One of the Bayesian models uses non-centered parameterization, which theoretically mitigates problems with sampler traction and parameter collapse or explosion. All models also make use of spatial matrices based on the simulated geographical coordinates. I then measured estimate uncertainty at each step for each model.

This image shows the results of the experiment with between 1 and 10 counterexamples. Since the centered and non-centered Bayesian models did not differ significantly, I show just the non-centered parameterization results here. Essentially, the Bayesian model shrinks the estimates of the outliers toward the global mean, with extreme certainty, confirming our original hypothesis (i.e. that all cold-blooded animals lay eggs), which is obviously not correct. This is evidence of the Topological Mirage, with a clear false-positive, an artifact of how Bayesian models handle this kind of skewed data. Interestingly, the Frequentist model does not succumb to the same overconfidence, maintaining estimate uncertainty and preserving the signal of the outliers, leading to rejection of our original hypothesis.

Mitigating the Topological Mirage

What does this mean for our research? I conclude that we need to treat our data and how we model it with more awareness. We can’t simply throw data at a model and get some results if we want to be sure that those results are accurate. If we are not aware of how the shape of our data affects the way that our statistical models operate, we might end up in the Topological Mirage: mistaking true correlations for falsehoods and false correlations for reality. This is an epistemological concern, and can potentially explain why, in certain cases, models don’t work.

“Use a Bayesian model” is a common refrain in certain circles, but as I show here that doesn’t always suffice. In fact, sometimes a Frequentist model is better, or at least more honest. Although, I should note that the Frequentist model used here is a relatively recent development that integrates a Gaussian Process over a spatial matrix, which helps to stabilize estimates. As modeling techniques improve, we may be able to eliminate the Topological Mirage altogether.

Conclusion

This has been a bit of a deep dive into why we need tools such as pyTstability to measure properties of our data like the TSI. Importantly, measuring E-BFMI and estimates to identify stability, as I’ve done in the experiments above, is only something that can be done after a model has been run, provided you save the traces appropriately. In some cases (depending on the size of the dataset), it can take weeks to run a model, particularly a complex Bayesian MCMC specification. The TSI, on the other hand, is a measure that can be computed nearly instantly on a dataset before you being modeling, saving the headache of re-running analyses. Hopefully by using tools like these we can get closer to understanding the realities of the world around us, and not succumb to the Topological Mirage.