💡🎬 Shared Storage & Collaboration

Multiple people will need access to the data at once in a post facility. They’ll need to do so in a way that prevents “branching” or multiple, separate copies of data that will need to later be reconciled. They’ll need it quickly and reliably. Making this all happen isn’t easy or cheap.

Shared Storage with Phil Crawley is an excellent video if you’re into shared storage and have the time for it.

Storage Space

Usually the first factor to consider is how much space you’ll need. HDD’s (magnetic spinning disk hard drives) are still cheaper than SSD (solid state flash media) for large amounts of storage, but remember HDD speeds are less than 200MBps while SSDs can achieve 3000MBps.

Stream Count

How many people need access to the shared storage simultaneously? 25 editors scrubbing through 25 different clips all being served from the same storage device simultaneously demands a high quality system. When comparing stream count numbers, consider the codec and frame rate used, how many separate devices requested those simultaneous streams, and that playback numbers aren’t artificially inflated by using RAM to store streams on local machines.

An easy way to calculate the bandwidth requirements of a file is to simply divide its size by its duration for the average data rate. Remember that 8 bits equal 1 byte:

DNx36 is 36Mbps or 4.5MBps.

AVID

When it comes to shared storage, Avid is the exemplar. Avid Unity was the first shared storage. SNFS is the common file system. This is one of the things that gave the Avid an early advantage in professional video workflow. But times change rapidly.

Shared storage has moved from fibre channel SANs to iSCSI via Ethernet-based SANs. This is an IP-based system. Clients run an iSCSI “initiator” which makes the storage look like a mounted disk.

TCP/IP based protocols like SMB (samba), AFP, aren’t optimized for video. A NAS is clever in how you can upgrade it’s size, but it’s connecting using these industry standard network protocols.

More efficient compression has made file sharing easier. We now use compression DNXhd 185 is 23MB/seconds and uncompressed SD was 21MB/s. Compression has allowed us to get better bang for buck video quality in smaller payload. Cat 5/6 ethernet can carry it.

Terrablock

Start here (from the video linked above) to get right to Terrablock, the first “cheap shared storage system”. It was a block-level SAN mounted as a fixed read/write disk on a single workstation. Nowadays it basically mimics an Avid media composer storage (unity or ISIS) system so you can use AVID’s shared storage features.

LumaForge Jellyfish

This is one of the first players in the “budget” shared storage arena. If setting up your own networked shared storage intimidates you, these guys have created a ready-to-go product. Though the price may seem intimidating, it’s pretty affordable compared to what shared storage cost previously. They also have rental options available.

Nowadays it’s most cost-effective (under $5k) to just configure a 10Gbe NAS for small teams of shared storage users. The bulk of the cost is then in the server-quality HDDs. Bob Zelin (bobzelin@icloud.com) has a good post on Creative Cow and it’s worth your time to simply contact him if you’re ever in the market for this.

EditShare

SNS EVO

Comes with “Share Browser”, a cool DAM tool.

Protocols

AFP, SMB, SMB2, SMB3, NFS, Fibre Channel, iSCSI

Bob Zelin on Shared Storage in 2019

The home-brewed solution. You don’t have to read through the following, but it’s a very good insight into some great specifics on how to make your own “affordable” shared storage with much less expensive hardware.

“The QNAP TVS-871T, TVS-1282T, and TS-1685 all come with the dual 10Gbase-T ports built into them, for free. You don’t need to second source anything – it’s all FREE from QNAP. The TVS-871T will work wonderfully for you, for your editors who are working in 4K. If you only have 2 computers, you don’t even need a switch – you can just go into the two built in 10G RJ45 ports and edit away. You must assign static IP addresses for these on two different subnets if you don’t use a switch. Do not get the TS-453B. You need EIGHT DRIVES to reliably do shared storage for multiple editors – particularly if you are doing 4K media, and ESPECIALLY if you are going to do a multiclip or multicam job. You are correct that thunderbolt 3 can only be a very short cable, and will reduce to thunderbolt 2 speeds if you extend the cable. And you will find that thunderbolt 2 for video editing is pretty unreliable (this connection is called thunderbolt bridging). You are much better off getting a thunderbolt to 10G adaptor for your computer (the Promise SanLink3 for thunderbolt 3 is only $299 US retail), and then you can plug directly into the 10G port on the TVS-871T, and you will have super fast speeds for 4K and 6K editing.  The price of the entire QNAP system for doing shared storage is a fraction of the price of any of your RED gear !!!!! Remember, you can only run a thunderbolt 3 cable 2 meters. Any longer and it drops in speed. With the 10G adaptor, you can have a 55 meter Cat6 cable on it that costs you 10 bucks ! In the past I only used AFP, but now I am only connecting with SMB. I turn off SMB Signing on every Mac with the nsmb.conf file. It’s a pain, but it makes a huge difference in performance. I used to use NFS before FCP X 10.3, because you could not write a library to the QNAP unless you used NFS, but now that Apple fixed this, I only use SMB.  For new systems, with Adobe Premiere, or Resolve, I use SMB. I face the reality that Apple wants AFP to go away. You do not need to buy a special cable (cross over cable) to go into the QNAP directly from your Mac or Win PC system. You plug directly into the QNAP ports (either the 1G or 10G ports) and assign static IP addresses on different subnets and you will have no problems. You can have 2 10G clients, and 4 1G clients with no switch, if you like. The switch just makes it easier, and you can keep everyone on the same subnet. 

https://www.reduser.net/forum/showthread.php?162272-Qnap-raid-system/page11

All in all it is a solid recommendation from my side. “QNAP’s work wonderfully as shared storage systems for RED workflow. To be clear, they work perfectly, and very fast for Adobe Premiere, FCP X, Davinci Resolve, and even AVID Media Composer (if you install Indiestor Mimiq on your editing computer). With a 1G connection, you will get 100 MB/sec bandwidth. With a 10G connection (using a thunderbolt to 10G adaptor from Promise or Sonnet), you will get 800 MB/sec – more than enough for 6K and 8K workflow. While the TVS-871T is a wonderful inexpensive product, and it’s “replacement” is the newer TVS-1282T, the best current value from QNAP is the incredible TS-1685, which is a 12 bay, that has built in 10GbE ports, and can be expanded with 4 additional expander chassis. To be clear, if you install 10TB HGST NAS drives in a TS-1685, after a RAID 6, you will have over 100 TB of usable storage in this tiny box, and all the editors will be able edit 4K, 6K and 8K. For multiple systems, as suggested above, get a small 10G switch from Netgear – specifically the XS708T or XS716T. Great products, easy to setup, and lifetime warranty. “

SSDs have shorter lifespans in server situations.

The Cloud

This is worth watching as more and more of our processes move into the cloud. While not an end-to-end solution for many productions today, this will likely be the future.