- Light Fields - Building the Core Immersive Photo and Video Format for VR and AR
by Ryan Overbeck (Google, USA)
Light fields provide transportive immersive experiences with a level of realism unsurpassed by any other imaging technology. Within a limited viewing volume, light fields accurately reproduce stereo parallax, motion parallax, reflections, refractions, and volumetric effects for real-world scenes. As such, many believe light fields will become the fundamental immersive media format for VR and AR. Although light fields have been explored in computer graphics since the mid-90’s, practical systems for recording, processing, and delivering high quality light field experiences have largely remained out of reach.
Our team at Google is developing hardware and software solutions that finally make it possible to acquire and view light fields on current and affordable hardware. In this talk, I will describe our work on Welcome to Light fields, the first immersive light field experience that can be downloaded and viewed on VR-ready Windows based computers, such as the HTC Vive, Oculus Rift, or Windows Mixed Reality HMDs. This piece won a Lumière Technology Award in 2018 and enables the user to step into a collection of panoramic light field still photographs and be guided through a number of light field environments, including the flight deck of Space Shuttle Discovery. I will also present our more recent work on light field video that uses machine learning to compress the light field data into a format that can be viewed even on low power mobile standalone headsets like the Oculus Quest.
- AI Powered Internet Video Streaming: Trends, Challenges and Practices
by Lifeng Sun (Tsinghua University, China)
The rapid development of networking technologies and multimedia services have imposed unprecedented demands on today's video streaming infrastructure. Tackling such tasks is challenging, as the real-world network conditions are getting increasingly complex and heterogeneous. The recent success of deep learning, however, provides new and powerful tools that can help taming the above problems. In this talk, I will explore the impacts of deep learning methods on today's Internet video streaming systems, including representative cases that deep learning methods outperform the traditional counterparts, challenges that we have encountered during real-world deployment, and recommended directions for further research.
Dr. Lifeng Sun is a full professor of the Department of Computer Science and Technology at Tsinghua University. His professional interests lie in the areas of Video Streaming, 3D video Signal Processing and Coding, Virtual Reality, Multimedia Big Data, Social Media and Multimedia Edge Computing. He has published over 200 papers in the above domain including IEEE JSAC, TMM, TPDS, TIP, TCSVT, TMC, CVPR, INFOCOM, ACM TOMM, Multimedia, AAAI, WWW. Prof. Sun got the Annual Best Paper Award of IEEE TCSVT (2010), Best Paper Award of ACM Multimedia (2012), Best Student Paper Award of Multimedia Modeling (2015), IEEE Multimedia Bigdata (2017) and ACM NOSSDAV (2019). He is a member of VSPC_TC (IEEE Visual Signal Processing and Communication Technical Committee in IEEE Circuits and Systems Society) and MMC_TC (IEEE Multimedia Communications Technical Committee in IEEE Communications Society), he served as Co-Chair of IEEE MMTC Media Streaming Interest Group (2010-2011), TPC Co-Chair of IEEE ICC Symposium on Communications Software, Services and Multimedia Applications (2018).
- Preserving Video Truth: an Anti-Deepfakes Narrative
by Roderick Hodgson (Amber Video, UK)
The prevalence of video, the ease to create fake video, and the speed and power to distribute these fakes at scale, globally, have come together to create the perfect landscape for the growth of malicious deepfakes. This talk will look at the state of synthetic media technology, highlight where the fear of it is overblown (on social media) and where we are critically not paying enough attention to it (evidence workflows in due process). The talk will also focus on the only durable technical approach to tackling deepfakes and why we need a shared framework/standard for trusted media.
Roderick Hodgson is co-founder and VP of Engineering for Amber Video, a startup creating a 'truth layer' for video. He holds a degree in Artificial Intelligence and Computer Science from the University of Edinburgh, and has spent the past decade working at the confluence of artificial intelligence, computer security and video engineering. His career in video engineering started in 2009 when he joined the BBC Research and Development department. He has lead the research teams of several startups and served as a Board Director of Secure Chorus, a not-for-profit membership organisation addressing data security requirements. He contributed to the writing of several standards, and holds several patents.
- Mind the Gap: Contextual Video Streaming and Transport
by Mohammad Alizadeh (MIT, USA)
Mohammad Alizadeh is an Associate Professor of Computer Science at MIT. His research interests are in the areas of computer networks and systems, and applied machine learning. His current research focuses on learning-augmented systems, video streaming, and congestion control algorithms for datacenter and wide-area networks. Mohammad's research has garnered significant industry interest. His work on datacenter transport protocols has been implemented in Linux and Windows, and has been deployed by large network operators; his work on adaptive network load balancing algorithms has been implemented in Cisco’s flagship datacenter switching products. Mohammad received his Ph.D. from Stanford University and then spent two years at Insieme Networks (a datacenter networking startup) and Cisco before joining MIT. He is a recipient of the Microsoft Research Faculty Fellowship, VMware Systems Research Award, NSF CAREER Award, SIGCOMM Rising Star Award, Alfred P. Sloan Research Fellowship, and multiple best paper awards.
- 360-Degree Video Streaming
by Yao Wang (NYU, USA)
360-degree video streaming is an emerging application that presents significant challenges to the bandwidth-limited and dynamically changing networks. This talk will present our recent work for 360-degree video streaming applications with different latency requirement, from on-demand streaming, to live streaming, to interactive streaming. We will first present the proposed system architecture for each application addressing the particular challenges as well as exploiting the unique opportunities of the underlying application. We will then discuss how to use deep learning to overcome two shared challenges in these systems: predicting where the viewer will look (FoV prediction), and adapting the total video rate based on the network condition and buffer status. For FoV prediction, we will present several LSTM-based prediction methods, ranging from using the viewer’s past FoV trajectory alone, to leveraging other viewers FOV distributions, to exploiting the actual video content. For rate adaptation, we will describe a deep reinforcement learning frame work, that aims to maximize the long-term average quality of experience.
Yao Wang is a Professor at New York University Tandon School of Engineering (formerly Polytechnic University, Brooklyn, NY), with joint appointment in Departments of Electrical and Computer Engineering and Biomedical Engineering. She is also Associate Dean for Faculty Affairs for NYU Tandon since June 2019. Her research areas include video coding and streaming, multimedia signal processing, computer vision, and medical imaging. She is the leading author of a textbook titled Video Processing and Communications, and has published over 250 papers in journals and conference proceedings. She received New York City Mayor's Award for Excellence in Science and Technology in the Young Investigator Category in year 2000. She was elected Fellow of the IEEE in 2004 for contributions to video processing and communications. She received the IEEE Communications Society Leonard G. Abraham Prize Paper Award in the Field of Communications Systems in 2004, and the IEEE Communications Society Multimedia Communication Technical Committee Best Paper Award in 2011. She was a keynote speaker at the 2010 International Packet Video Workshop, at INFOCOM Workshop on Contemporary Video in 2014, and at the 2018 Picture Coding Symposium. She received the NYU Tandon Distinguished Teacher Award in 2016.
- Preserving Video Truth: an Anti-Deepfakes Narrative