Please check Cluster Configuration as the next step in a cluster configuration.
Introduction to Clusters - Storm Streaming Server
Storm Streaming Server has been designed from the ground up with full scalability in mind, enabling it to handle tens of thousands of video streams, each of which can have thousands of simultaneous viewers. Thanks to the implementation of a cluster architecture, the system gains additional redundancy and resistance to serious network infrastructure failures.
Cluster building can begin with just one server instance - without the need to immediately invest in multiple physical machines or VPSs (Virtual Private Servers). As demand increases, adding subsequent Storm Streaming Server instances allows for gradually increasing the system's performance and reliability.
Instances and Applications
A key aspect of cluster operation in Storm Streaming Server is the distinction between a server instance and so-called applications.
-
Server Instance - refers to a running copy of the program on a
specific operating system. If we have two licenses, we can run two instances - for example,
on different physical or virtual servers (VPS).
- Application is an internal subcomponents operating within a single instance. The basic type of application in Storm Streaming Server is live - a universal application capable of receiving incoming streams, transcoding them, and delivering them to viewers. However, the Live application operates only within a single instance and cannot be scaled beyond its boundaries. This is where dedicated applications such as origin, transcode, edge, and the brain of the entire system - ClusterManager - come into play. These enable scaling, redundancy, and flexibility of the entire cluster's operation.
A single server instance can host multiple applications of different types and very different configurations. It is possible to configure a complete cluster on a single server machine, but also to split the whole system into individual servers.
Cluster Building Blocks
Each cluster consists of the following types of applications:
-
ClusterManager - the most important component of the cluster,
its "brain." Each of the remaining applications, immediately after startup, establishes a
direct connection with the ClusterManager and through this route sends or receives
information about active streams. Each cluster requires at least one application of this type.
-
Origin - an application responsible for accepting and
authorizing incoming (ingest) streams. It does not allow direct playback of streams. Origin
continuously reports the status and data of all handled streams to the ClusterManager.
Edge-type applications can connect to the Origin to copy active streams. Origin can also
transcode streams published on it. Configuration details can be found in the
Origin Application Configuration guide.
-
Transcode - an optional application in the cluster, whose
role is to transcode streams. The scope of transcoding is controlled by the Cluster Manager.
Each Transcode instance has a defined limit of simultaneous transcoding tasks - after
reaching it, it does not accept subsequent orders. Configuration details can be found in
the Transcode Application Configuration guide.
- Edge - an application responsible for copying streams from Origin or Transcode applications and redistributing them to viewers. It receives information about all changes regarding streams from the ClusterManager and updates its sources based on this information. Configuration details can be found in the Edge Application Configuration guide.
Functional and Requirements Schema for Applications
| Origin | Transcode | Edge | |
|---|---|---|---|
| Can it accept streams from external software? | Yes | No | No |
| Can it transcode streams? | Yes | Yes | No |
| Can it record streams? | Yes | No | No |
| Must it have a public/accessible IP address? | Yes | No | Yes |
| Required number of instances in a cluster | 1 | 0 | 1 |
Table 1. Functional and Requirements Schema for Applications table
Dynamic Allocation of Transcoding Resources
One of the functions of the Storm Streaming Server cluster is the ability to dynamically allocate transcoding resources. Each transcode application has a defined maximum number of simultaneous tasks (e.g., transcoding the original 1080p stream to 720p and 480p versions counts as two separate tasks).
Based on data provided by the ClusterManager, streams with the highest number of viewers - or those indicated based on a defined weight system - are automatically directed for transcoding. Streams can be withdrawn from transcoding if other transmissions are deemed more priority.
Management & Control Panel
A dedicated cluster module for the control panel is available at /cpanel/clustermanager. This module allows viewing all streams within the cluster, along with their statistics, status, and information on whether they are being transcoded or not. In the panel, we can also view information about applications connected to clusters and statistics of their server instances (usage of CPU/Memory resources or the number of transcoded streams).