Timeline & Playback Handling
Content can be consumed in various ways, including live viewing or as Video On Demand (VOD) after an event has taken place. Additionally, it is possible to watch content concurrently on multiple devices, preferably with synchronized playback across these devices for both live and VOD content.
As a result of these diverse use cases, different requirements for playback handling arise. Each scenario demands specific behaviors to ensure a seamless experience, which we will explore in the next sections.
It is important to be mindful of terminology to avoid confusion. The context in which timeline and playback modes and states are used may vary, whether from a technical perspective, timeline management standpoint, or the end user's content consumption experience.
As an illustration, the term "live" can carry different interpretations depending on the context. From a player perspective, it may refer to the live edge or the current playback position. From an event standpoint, it could pertain to an ongoing real-time happening. Furthermore, it may signify that the timeline is in a live state or that the system is presently operating in live mode. Therefore, in the next sections we will try to be very specific when we talk about them.
The main experience timeline can be in 3 modes and 5 different states:
The system also uses two playback strategies in different states, namely:
The Live Mode enables users to play content at the live edge, which is defined in the EXP manifest. This manifest includes a duration property that describes the virtual timeline, typically for an event. The live edge latency is determined by the SMEP Service during the generation of the EXP Manifest. To achieve this, the SMEP system estimates the latencies of all relevant content and selects an optimal live edge latency.
When an experience is launched and the EXP manifest is still live, the system enters the Live Mode, switching to the LIVE state. Additionally, there might be a desire to offer users the option of watching without spoilers, allowing them to commence the experience from the event's beginning in the GO LIVE state.
It is important not to confuse the term "live" with the Live Mode or LIVE state in this context. The timeline can be in Sync Mode while the event is still live, potentially introducing some live capabilities into the experience.
When in the LIVE state, the Catch-Up Strategy is used for playback, where the EXP engine attempts to synchronize with the live edge specified in the manifest and automatically catches up if playback falls behind due to network conditions or buffering.
In the GO LIVE state, the Default Strategy is used for playback, where content is not continually synchronized against any specific target.
Additionally, in the LIVE state, non-seekable content (e.g., WebRTC-based streams) may be preferred over seekable HTTP-based streams with higher latency. Once the user seeks back, exits the LIVE state and switches to the GO LIVE state, the system starts using seekable content.
The system stays in the LIVE state until either the EXP manifest concludes, prompting a transition from Live Mode to VOD Mode and changing states, or until a user initiates manual interaction while in Live Mode, like seeking back for a replay during the live event. In the latter case, the system switches to the GO LIVE state.
Upon transitioning from Live Mode to VOD Mode, the system automatically shifts to the VOD state. When the user reaches the end of the timeline, a replay event button is displayed. Similarly, if the user is in VOD Mode and the event becomes live again (e.g., due to changes in the event's end time), the system switches to Live Mode but enters the GO LIVE state instead of the LIVE state to prevent skipping any content.
The Sync Mode and the respective SYNCED and GO SYNC states functions similarly to the Live Mode and its states, but instead of synchronizing the content against the live edge while in SYNCED state, it uses another target for synchronization. For instance, for a second screen audio sync, the EXP engine aligns with the audio being played on the TV.
Similiar to the Live Mode, the Catch-Up Strategy is used for playback while in SYNCED state and the Default Strategy while in GO SYNC state.
The system transitions to the Sync Mode through user interaction, such as performing a second screen audio sync, and can be exited by another interaction. The system won't leave the Sync Mode automatically only by manual interaction.
There is a key difference between the SYNCED state and the LIVE state when it comes to handling non-seekable content. When the synchronization target is in close proximity to the live edge, within a certain threshold, non-seekable content becomes a viable option. However, beyond this threshold, the system exclusively utilizes seekable content.
As mentioned above, the event may still be "live", introducing live capabilities to the system, despite the timeline being in Sync Mode. For example, live notifications could be active while the event is live and the timeline is in Sync Mode which will be disabled once the event ends. Hence, managing the "live" status of an event is an additional consideration alongside the timeline mode and is independent of the Live Mode.
The VOD mode is the default state when the timeline is neither in Live Mode or Sync Mode. This mode has only one state, the VOD state.
In the VOD state, the Default Strategy is used for playback, where content is not continually synchronized against any specific target.
This playback strategy is used when in the LIVE or SYNCED state. When using this strategy, the EXP engine attempts to synchronize against a specific target e.g. the live edge or a sync target e.g. the TV audio and automatically catches up if playback falls behind due to network conditions or buffering. Some content may have higher latency, but the EXP engine handles it by playing this content with an artificial delay.
When the network conditions consistently fail to support smooth content playback at the sync target for an extended duration, rather than just temporarily, the EXP system can respond by issuing a warning to the user. This notification informs the user that smooth playback is not feasible in the current state (e.g. LIVE or SYNCED) and recommends falling back for a better experience, resulting in a switch to the GO LIVE or GO SYNC state respectively.
To underline, it's important to note that while using this strategy the system doesn't automatically transition away from its current state, even during unfavorable network conditions. The system's aim is to remain synced with the target, even if this results in ongoing buffering.
Therefore, the above mentioned notification system is used, enabling users to easily transition from the current state to an alternative state that utilizes the Default Strategy for a more seamless playback experience.
This playback strategy is used when in the GO LIVE, GO SYNC or VOD state. When using this strategy, content playback is not consistently synchronized against any specific target. Consequently, we allow content to buffer, minimizing or even completely avoiding interruptions or jumps after playback starts following a seek.
To better illustrate this concept, consider the example of being in the LIVE state using the Catch-Up Strategy, where playback falls behind due to network request latencies. The system attempts to catch up and speed up the playback, which is the intended behavior. However, if the same catch-up mechanism is applied in the VOD state, seeking to a specific point in time would lead to an immediate start of the virtual playback timeline without waiting for the content. This could cause issues, like playback speed-ups, if the content requires additional time to initialize and buffer after a seek action or even re-seeks which can result in playback freezes.
To avoid this, we adopt a different approach in the VOD state by using the Default Strategy. We allow the content to buffer together and pause the virtual playback timeline during this buffering period until the content is fully ready to be played back. This strategy becomes particularly crucial in complex scenarios with multiple independent audio or video tracks e.g. for global audio or split view, where we aim to minimize any visible or audible catch-up mechanic for content synchronization especially after a seek action on the timeline by the user.