Overview
Multimedia streams from cameras are becoming increasingly popular as IoT data sources as they provide a visual of real-time events which can be used for various applications. Multimedia content possess lots of information that data streams and if synchronized properly, there a can be a lot of real word applications that could be built around it.
The intuition is to use multiple cameras to capture events from various positions and process them using image processing (OpenCV for eg.) to draw inferences. For Some applications for video streaming are listed and explained below:
Vehicular Ad Hoc Networks : Road junctions are important components of VANETS. Imagine an application built over a cross junction. Tow cross roads have separate cameras. The application uses the feeds from cameras, processes and detects vehicles that are travelling at a high speed and as result have a higher probability of colliding with each other. In such a scenario, the cameras on the different roads have to stream their videos in time and synchronized with each other to a common server that actually calculates possibility of a vehicle crash and alerts the vehicles to slow down. If the streams aren’t synchronized in this scenario, there is no way for the application to function properly since streams from one camera might end up much later than the other. With vehicles travelling pretty fast the situation can turn into a disaster.
Multi-Room Playback: Multi-room playback put simply, is the ability to listen to different media in several rooms. Depending on the system you choose, you'll be able to play different media in different rooms at the same time, or the same in all rooms, or just a couple simultaneously.
The media can be from streaming services like Spotify or Youtube or from your own personal collection whether that be on a network drive, your computer, a CD or even vinyl - with the right setup. Most of the multi-room systems tend to be wireless.
For all these applications, synchronization of multimedia streams is very important for real-time functionality. Some important aspects to consider regarding streaming applications are:
Timeliness of Action: Does an action need to be taken as the condition happens or can it wait for later for batch processing. Do the streams need to analyze and act as soon as it receives them?
Timeliness of Data(video streams): Is the current state of scene as depicted by a camera have any historic value that needs to be processed?
Synchronization involves measuring the delay caused by the network in delivery of the media frames and compensating for them, to ensure that the frames of different media streams arrive at a synchronized common time instant.
State of the Art
Much of the existing work focuses mainly on synchronizing video and audio streams of a single multimedia playback stream, also commonly termed the "lip-synch" problem. This is a relative easier problem to solve. Beyond that there will be the need to synchronize an arbitrary combination of audio and video, like for instance audio together with two video streams that show a single scene from different view-points. The continuous stream synchronization problem is in fact the combination of two problems, intra-stream synchronization and inter- stream synchronization, which are tightly correlated and cannot be solved separately. If continuous media are deployed in an interactive application, there is the additional requirement of keeping the end-to-end delay, i.e., the delay between acquisition and presentation of a medium sample, as low as possible.
As for as existing literature is concerned, there is still no established state of the art to perfectly synchronize 2 different video streams coming from independently connected video devices across different networks. Also, most of existing papers elaborate on possible theoretical solutions, while none have produced a practical implementation to establish a working concept of the same.
The intuition is to use multiple cameras to capture events from various positions and process them using image processing (OpenCV for eg.) to draw inferences. For Some applications for video streaming are listed and explained below:
Vehicular Ad Hoc Networks : Road junctions are important components of VANETS. Imagine an application built over a cross junction. Tow cross roads have separate cameras. The application uses the feeds from cameras, processes and detects vehicles that are travelling at a high speed and as result have a higher probability of colliding with each other. In such a scenario, the cameras on the different roads have to stream their videos in time and synchronized with each other to a common server that actually calculates possibility of a vehicle crash and alerts the vehicles to slow down. If the streams aren’t synchronized in this scenario, there is no way for the application to function properly since streams from one camera might end up much later than the other. With vehicles travelling pretty fast the situation can turn into a disaster.
Multi-Room Playback: Multi-room playback put simply, is the ability to listen to different media in several rooms. Depending on the system you choose, you'll be able to play different media in different rooms at the same time, or the same in all rooms, or just a couple simultaneously.
The media can be from streaming services like Spotify or Youtube or from your own personal collection whether that be on a network drive, your computer, a CD or even vinyl - with the right setup. Most of the multi-room systems tend to be wireless.
For all these applications, synchronization of multimedia streams is very important for real-time functionality. Some important aspects to consider regarding streaming applications are:
Timeliness of Action: Does an action need to be taken as the condition happens or can it wait for later for batch processing. Do the streams need to analyze and act as soon as it receives them?
Timeliness of Data(video streams): Is the current state of scene as depicted by a camera have any historic value that needs to be processed?
Synchronization involves measuring the delay caused by the network in delivery of the media frames and compensating for them, to ensure that the frames of different media streams arrive at a synchronized common time instant.
State of the Art
Much of the existing work focuses mainly on synchronizing video and audio streams of a single multimedia playback stream, also commonly termed the "lip-synch" problem. This is a relative easier problem to solve. Beyond that there will be the need to synchronize an arbitrary combination of audio and video, like for instance audio together with two video streams that show a single scene from different view-points. The continuous stream synchronization problem is in fact the combination of two problems, intra-stream synchronization and inter- stream synchronization, which are tightly correlated and cannot be solved separately. If continuous media are deployed in an interactive application, there is the additional requirement of keeping the end-to-end delay, i.e., the delay between acquisition and presentation of a medium sample, as low as possible.
As for as existing literature is concerned, there is still no established state of the art to perfectly synchronize 2 different video streams coming from independently connected video devices across different networks. Also, most of existing papers elaborate on possible theoretical solutions, while none have produced a practical implementation to establish a working concept of the same.