The 4th Dimension -- “A Window to the World”
Dr. Arie Kaufman shares his experiences discretizing the world through his ground-breaking, application-driven advancements in the field of virtual reality.
“So you ask me what brings me joy and fun in my work. 2D is basically a kind of boring image; it is when you add 3D when you start to have fun. It is analogous to the differences between painting and sculpting. And then if you go to 4D, you have 3D plus time - so this includes videos and animations - this is a real joy. And if you add total immersion, as I have with virtual reality, then this is really euphoria or dream. So that’s where we experience all the fun.”
***
Dr. Kaufman’s research interests and collaboration with industry in the field of volume visualization started in the 1980s. In 1987, after his technical presentations at two top computer graphics conferences, a large company, Hughes Aircraft, reached out to him: they were interested in his work on volume graphics, and wanted to license the technology. Throughout this collaboration, Dr. Kaufman and his team had to work out all mechanisms of patenting and licensing from scratch: it turned out at the time there was no organized process of protecting intellectual property and licensing patents at Stony Brook University.
“The beginning was very unorganized,” – says Dr. Kaufman. – “We had to invent all the mechanisms… To make the story short, Hughes Aircraft licensed a bunch of patents that we did on the fly, as we were talking to Hughes Aircraft and defining these patents.“
The contract with Hughes Aircraft was signed in 1989.
“This was a philosophical invention, because it was all about discretizing the world.” The technology involved converting geometric objects to discrete voxel-based form (voxel is a 3D pixel). And this was also the basis of hardware capable of handling very large data sets in a very short time frame that was developed by Dr. Kaufman’s team and licensed to Japan Radio Corporation and to Mitsubishi Electric in 1996. “It took 16 years , 4 PhD students – 4 generations of PhD students... to develop and demonstrate the concept”
Arie Kaufman continued working on volume visualization during the 1990s; and his seminal VolVis software package was licensed to different companies (such as Exxon in 1995), and disseminated free to academic institutions. By this time the whole process of SBU intellectual property management and licensing started to get more and more organized.
“In general my career has been focused on applied research,” - says Dr. Kaufman. – “Applied research lends itself to practical solutions, to useful patents, and ultimately to strong collaborations with industry. Often I call it “application-driven research.”
One of Dr. Kaufman’s most successful application-driven inventions is the Virtual Colonoscopy – a medical imaging technology that greatly increased colon screenings by transitioning away from the traditional, invasive colonoscopy to a much less invasive and more efficient method. This is an example where an invention came about directly in response to a problem that needed to be solved.
“I think part of being an inventor is having an imaginative memory as well as an associative memory.” – explains Dr. Kaufman – “Many of my inventions have to do with associating one thing with something else. For example, we developed one virtual application of “flying down” the corridor of the Old Computer Science building. This is probably the longest building on campus, and we did a flight through it in virtual reality. And then I said “If we can do it in the corridor we can do it in the human colon”. And that’s really the WOW moment.”
The development of the virtual colonoscopy technology began in 1994. And, 25 years later, this technology breakthrough has been commercialized by large CT Scanners manufacturers, entered the mainstream of colon cancer screening, and has saved a lot of lives. There was a portfolio of patents, multi-party non-exclusive licensing, 4 commercial licenses n all.
Virtual Colonoscopy gives radiologists a more complete view of the large intestine, and leads to earlier and better detection of colon polyps, the precursor of colon cancer. Still, it took many years to bring Virtual Colonoscopy from conception to medical practice. Though the product was ready in 1999, it really took a lot of time for the hospitals and diagnosis centers to start using it. “A hospital is very conservative and very slow,” – says Dr. Kaufman, – “and a medical device is very difficult to get into the market. But once you see it become mainstream, and people are using it, and then 30,000 lives are saved – that’s an extreme success.”
Immersive Virtual Colonoscopy: Dr. Kaufman inside the virtual colon
Among other advancements of virtual medical imaging is a recent technology of chest CT that helps to classify and confirm COVID-19 diagnosis. This involves lesion segmentation and classification technology that has previously been applied to Virtual Pancreatography – a new tool for detecting pancreatic cancer that Dr. Kaufman and his team are continuing to develop.
Future advancements in virtual and immersive technologies and specifically in medical
imaging and diagnosis are underway using Dr. Kaufman’s Reality DeckTM -- a 1.5 Billion-pixel immersive display which is the highest resolution immersive
visualization facility ever built -- enabling fine detail visual analytics of big
data. This is yet another example of an associative memory result: the idea was inspired
by Holodeck, the fictional device from the cult Star Trek movie. At the same time
it is also an application-driven development. There are currently more than 30 applications
running on the Reality DeckTM: it is used in advanced medical imaging, drug development, climate and weather modeling,
and in many other areas.
7-billion pixel NASA image of Milky Way displaying on the Reality DeckTM
It is also a basis for the “Aging in Place” project, Arie Kaufman’s long-term research goal. The Reality DeckTM or its derivatives, says Dr. Kaufman, may become a window to the world for the elderly, who would like to stay home instead of going to special facilities. It will enable interactions with medical providers, and communication with family and friends – all from home, with the help of the immersive wall.
Dr. Kaufman has a lot of experience working with industry; his inventions have been licensed by various companies. Of course working with industry might be really challenging.
“The challenge working with industry is that they want the solution as quickly as possible and it takes time.Still, working with industry is very rewarding, as it gives you direct access to technology needs and strategies.”
“The way I’ve been collaborating with industry [is to] involve them as soon as practicable.” - explains Dr. Kaufman. – “I would typically do a seed project with them, maximum one year, and explain to them that this is research, and it takes time.”
It is also important to build relationships with industry to understand the needs of the market and develop application-driven technology.
“If you want to be successful, you have to excel in your research” – this is Dr. Kaufman’s main advice to young inventors. “One should use imagination and association to advance research, while still being application-driven.”
“I work with imaging, and imaging is everywhere. There is a Chinese proverb that says ‘a picture is worth a thousand words.’ I think the Chinese didn’t appreciate that in a picture we have a million pixels -- so actually it should be more like ‘a picture is worth a million words’.”
Dr. Arie Kaufman is a Distinguished Professor of Computer Science, the Director of the Center of Visual Computing, and the Chief Scientist of the Center of Excellence in Wireless and Information Technology (CEWIT) at Stony Brook University. He was Chair of Computer Science from 1999 to 2017.