SHA2017

Since there are very few recorded talks left from OHM2013, it seemed like a good idea to explain the fail-safes and redundancies C3VOC and Team Productiehuis are bringing to the track tents at SHA2017.

Starting with the hardware, there is always an encoding cube, a powerful workstation type computer, one or more cameras and various other hardware such as sound amplification equipment in every room. Both the cameras and the encoding cube record everything continuously. The cameras are equipped with two SD cards each which are large enough to record about 10 hours of content. Here we have our first fail-safe. If everything fails, we still have those video files to cut and publish. On average over the last Chaos events, C3VOC had to use the backup video from SD cards for just one talk.

Two SD cards in a camera

The plan is to have a camera in each tent which has, in addition to its power supply, a built-in battery providing power to record for at least two hours. So in the event of a power failure, we keep on recording. This is unlikely to happen because most speakers want to rely on their slides and don't have a voice loud enough to address all of the attendees in the track tent. During CCCamp15 there were no outages in the track tents, while they were running on diesel power, just like SHA2017 will do.

On to the next fail-safe. A clean audio feed is run from the central audio mixing desk of the tent into one of the cameras. If this clean audio feed fails, there is still the internal microphone of the camera. Both these tracks are recorded separately to the SD card. Using audio from the internal microphone isn't ideal, but most of the time, everything is still intelligible.

All recording up to here is done in hardware, so until now, there will have been no software (on computer-like hardware) that can fail. Let's have a look at the fail-safes built into the software side of a track recording.

The voctomix video mixing software used by C3VOC, which will also be used on SHA2017, is configured to be always running on the encoding cubes. Systemd acts as the watchdog and auto-start functionality here. All parts of voctomix are spread out over individual systemd units, which restart every second if they should ever fail. If one part, for example a video source like the camera capture-card fails, it's restarted immediately without pulling down everything else.

Voctomix GUI at Easterhegg 17, Andreas Sikkema

One ffmpeg script running on the encoder cube is responsible for recording everything to a RAID 1 array of two disks. This recording script is tied into a Mosquitto alerting chain. Connected to this alerting chain is a huge orange warning light, that will blink at the Productiehuis tent as soon as a recording script has died and restarted itself. We'll know immediately where and when that happened and can prepare, for example, to pull SD cards from the cameras to replace those few seconds that were just lost.

Warning light (C3VOC)

Streaming and recording are two separate output pipelines in voctomix. The stream can be blanked in breaks for example, but the recording can't. In the event that one of the video angels forgets to un-blank the stream, there will still be a useful recording on disk. Speaking of disks: they are known to fail. C3VOC has, so far, no failures during an event, but we are talking about fail-safes here. Everything that is recorded is rsynced to a central storage for further post-processing. After syncing all data is available in two places, on the encoding cubes in the track tents and on the central storage. If the network connection between cube and storage fails we can still go on with recording. The encoding cube only needs power and video signals to record something and all cubes in the track tents will be protected by a UPS. This is the next important feature: All recording related tasks are not dependent on the Internet up-link. Everything can and will be done locally. The complete up-link of the event can fail and all encoding cubes will still be recording happily, but unfortunately, live streams will fail. We hope you will believe us when we say that recording up to this point is stable and there will always be at least something usable left over. 

Let's go on to the post-processing chain. One might fear that someone accidentally presses delete and all recorded talks are gone. This is absolutely not the case. A person editing a talk can only see a virtual file spliced together from recording snippets. The underlying snippets are in a read-only state and are not altered during editing or post-processing, another one of our numerous fail-safes. Editing in this context is only searching for the exact beginning and end of a talk. Those timestamps are saved and used in the later, largely automated, post-processing chain. The whole post-processing chain never destroys any original recording snippets. They are only read once while encoding the HD master release. If something fails here, it can always be restarted over and over again, often with just a few clicks in a web UI. After being moved through the whole post-processing, sub-format encoding, audio mastering and so on pipeline, finished files get pushed to a worldwide distributed CDN.

This text got quite long, but we hope, that it is still understandable for someone who is not working in video production on a regular basis. If you want to have a look at the software used, you can check the C3VOC configuration management repository. Further info on all used hardware and more, can be found in the wiki.

felixs, Bix