Mirrors are the new lenses: synthetic data, surrogates and the science to come
1000lungs dataset · Interactive viewer · SynAirG on Github · Project page
A few years ago, someone I care about was hospitalised with severe respiratory problems. I responded in the way technically inclined people often do when confronted with fear and uncertainty: I read far too much.
I looked particularly closely at bronchoscopy. The more I read, the more I came to realise a bronchoscopy is a lot less a neat diagnostic intervention and a lot more like spelunking through uncharted cave systems where the light is playing tricks on you and everything kind of looks the same and you have at best the rough outlines of a map. It is, in short, a navigation problem like any... except about a million times harder.
Experienced clinicians solve a great deal of this through skill and familiarity. Robots and machine-learning models are not so fortunate. We cannot reasonably expect bronchoscopy to become more consistent and repeatable until we have a stable way to determine where the scope is -- ideally without adding ionising radiation and without depending on external electromagnetic tracking hardware.
That leaves us with a rather awkward circle. We need models that can localise the bronchoscope from what it sees, but we need accurately localised data to train those models.
Going to war with the data we don't have
A real bronchoscopic video is enormously valuable, but it does not ordinarily arrive with metric depth, surface normals, precise six-degree-of-freedom camera pose, branch identity, per-pixel shading or a perfectly aligned representation of the surrounding anatomy.
Obtaining some of that ground truth would require the same localisation equipment we are hoping eventually not to require. The procedure itself is sensitive, expensive and governed by entirely appropriate constraints around consent, privacy and clinical burden. Annotation is difficult. Rare appearances remain rare. And we cannot simply repeat a procedure with the lighting shifted, the camera cadence altered or the secretion load increased by twenty per cent -- patients aren't domain randomisation APIs.
The result is that some of the data most important for training robust depth, pose and SLAM models is not merely expensive. In many cases, it is data that routine clinical practice does not naturally produce at all. In that, it is hardly in small company -- my colleagues, for instance, routinely deal with data that needs to be synthesised because it is costly or dangerous to obtain. Before their robots set foot in high explosive environments or traverse a dusty mine shaft, they will have been trained on intricately designed replicas of reality in sim. All of that rests on synthetic data.
Creating configurable worlds for reinforcement learning
1000lungs is a synthetic virtual-bronchoscopy dataset built with SynAirG, our open-source system for creating generated worlds for bronchoscopy and other endoluminal interventions. Each case begins with a synthetic chest CT generated using MAISI’s rflow-ct workflow finetuned some more on chest CTs, together with a paired label volume. SynAirG extracts the airway mask and converts it into a bronchoscopy-ready mesh. That process includes topology repair, accessibility filtering, centreline extraction, branch metadata and registration back to the source volume.
A renderer then samples a procedure-style camera path through the resulting airway and renders synchronised condition streams from the virtual bronchoscope’s point of view. The outputs include depth, surface normals, per-pixel shading, airway masks, pose data, TUM-format trajectories, mesh geometry, NIfTI volumes and episode metadata following LeRobot/Open-H-style conventions. Geometry, trajectory and appearance remain linked to the same case identifier and provenance.
That continuity matters. It means we can change a material, a light source, a camera path or a rendering model without losing track of which anatomy produced the result. What is remarkable is that we can do so on the fly.
Reinforcement learning fundamentally depends on the environment, and its efficiency depends on the ability of the environment to granularly adapt and challenge the learner. This lies at the heart of the new perspective, which I sought to allude to in the title. The history of science has mostly been about making better lenses: more granular ways to investigate what is. A new kind of science is emerging that puts an increasing emphasis on mirrors: better ways to replicate what could be. In that sense, SynAirG is a mirror par excellence, designed to be realistic yet configurable. Each of the meshes you can inspect using the interactive viewer can be adapted to your specific needs and rendered in your system of choice (out of the box, we support Omniverse RTX as the rendering engine as the best quality/speed compromise out there).
We chose a physics-first, mesh-based foundation because geometry is unforgiving. Transformer and diffusion-based world models have become remarkably good at generating visual appearance. They are not yet something we would trust to preserve metric geometry, branch identity, reachability and camera pose consistently throughout a long endoscopic rollout. A bifurcation cannot quietly change identity between frames, a distal branch cannot disappear because the model found a more aesthetically satisfying arrangement. But generative world models do have their place in what we refer to as double generativity. In SynAirG, they can provide realism through providing the texture layer.
Out of the box, SynAirG includes a deliberately simple and editable PBR material system. It provides structured controls for mucosal appearance, wetness, roughness, erythema, secretions, bump and displacement. This gives us a useful baseline and, more importantly, a clear interface between geometry and appearance. Above that interface, learned generative models can do something much more interesting: a comparatively small collection of real bronchoscopic cases can be used to train an appearance or texture-generating model. Rather than asking those real cases to supply every anatomy, every camera path and every label, we can use them to teach the model what real bronchoscopic appearance looks like: mucosal texture, illumination, secretions and the considerable visual messiness of an actual procedure can all be reflected.
Over the next few days, we will be completing the geometry release. Currently, we provide a reference renderer but no pre-rendered footage, which the viewer indicates using a deliberate black-and-white outline feed in the RGB window. We will also be providing notebooks for configuring the Omniverse renderer, deploying in Isaac Sim and fine-tuning depth and pose models for endobronchial localisation. The mesh generation workflow is also fully disclosed, and we will provide guidance on using segmentation transfer for the privacy-preserving replication of patient-specific anatomies.
Better mirrors
There is a new shape of science emerging, in which we probe nature not only with better lenses, but with better mirrors: models that reflect the systems we want to understand closely enough that we can perturb them, recombine them and ask questions that reality does not conveniently permit us to ask. In physics, that's surrogate models like PhysicsNeMo, which our lab is using extensively. In medicine, the implications can be much more consequential.
Medical datasets are not neutral samples of human biology. They reflect who could obtain care, who was referred, who was scanned, which institutions had the resources to preserve and annotate the result, and which conditions occurred frequently enough to attract sustained attention. Synthetic and generative methods do not magically erase those biases. They do, however, give us the choice to no longer be held captive by the privilege of the normal. They may become powerful tools of redressing systemic inequities.
We believe they should. We believe that the promise of AI in medicine should not be limited to the frequent and the fortunate but that this tool of considerable power should be used to put all on an equal footing. The widely documented issue of data set sourcing acting as an amplifier of socioeconomic biases is, in the age of synthetic data, not a necessity but a choice. We hope that our comparably small contribution will make the choice of a fairer, more equitable world that opens opportunity for better, longer and healthier lives to everyone -- not just those who were fortunate enough to make it into the training sets -- available to all.
