💡📷 🎬 Hard Drives, Transfer Protocols and Data Rates
Thunderbolt 3 uses a USB 3.1 Type C connector, providing 40 Gbps over a PCIe bus to the CPU.
Media files are large, so at some point we inevitably have to cover the technology that helps us transfer and store them. This will be a hopefully not-to-painful look at the terms involved in digital storage and handling.
What’s a bit?
A “bit” is the smallest data unit utilized by a computer. It’s a clever contraction of “binary digit”, so in reality, it’s just a 0 or 1 value. 8 of these combined can represent 256 different combinations (2^8). The following list should be useful. Keep in mind that a lowercase “b” means “bit” while an uppercase “B” means Byte.
Busses and Lanes?
A bus is simply a channel between components in a computer; it’s what allows you to connect external somethings to the brain of the computer, known as the CPU. You’re probably already familiar with the USB (universal serial bus); another common bus is the PCIe bus.
PCIexpress is the update to the older PCI. It’s the interface that connects the computer’s motherboard to external high bandwidth devices. They come in lane counts of 1x, 4x, 8x, and 16x. From the moment your computer starts up, PCIe figures out what’s connected to the motherboard, and then links all your devices together determining the width of each link. The entire bandwidth available depends on the version of PCIe supported.
- PCIe 1.0, each lane supports 250 MB/s (so a 4x slot transfer rate means 4*250 MB/s = 1 GBps max)
- PCIe 2.0, each lane supports 500 MB/s (4x slot’s transfer rate means 4*500 MB/s = 2 GBps)
- PCIe 3.0, each lane supports 1 GB/s (x16 15.75 GBps)
- PCIe 4.0, each lane supports up to 2 GB/s (x16 provides 31.5 GBps)
- PCIe 5.0 each lane supports 4GB/s (x16 provides 63 GBps)
You can use the “System Information” app on the Mac to see how many busses you have available to you on your system. It’s useful to know which physical ports on the back of your computer feed to the same bus. As we’re about to get into, the point of all this is to spread the traffic of your data connections across all the available paths.
In short, this means you need be conscious that you have limited bandwidth to work with. There are multiple points at which you can bottleneck the flow of data inside your computer. If you are transferring footage from a USB 2.0 card reader to an NVME SSD, the card reader will make the transfer so slow that your SSD will be wasted. If you load up all your camera cards and hard drives on the same bus, the bus will be saturated.
Also remember that information travels two directions when moving data around. With a total bandwidth of 500MBps, A RED Mini Mag card reader reading at 250MBps will have less than 250MBPs remaining bandwidth for writing to disks.
Speeds
Various standards exist in an attempt to create compatibility between devices. These standards govern cable type, connector, protocols for connection between two devices, power supply, etc. Below is a list of the most common in recent past and a quick estimate of their speed.
USB
Physical Connectors
- USB Type A
- USB Type B
- USB Type C
The most confusing of these is USB Type C. USB-C is a physical connector. Like USB Type A and B, C is just a new physical shape. It supports the Thunderbolt 3 protocol and 100 watts of power delivery. The USB-C spec even includes audio transmission, but so far it has not universally replaced the 3.5mm headphone jack.
USB 4 includes Thunderbolt compatibility if the manufacturer of the device elects to include it. Intel donated Thunderbolt specs to the USB implementers forum. In September of 2019, the USB Implementers Forum detailed the USB4 standard. It’s a 40Gbps and a two-lane interface. Devices incorporating the tech due to appear in 2020. In the past, many devices used the USB-C connection but lacked Thunderbolt 3 support. USB4 should mitigate that, meaning most USB4 devices should also support Thunderbolt (though I wouldn’t be surprised if phones don’t because while the tech is license-free the advertising of Thunderbolt support is still taxed by an Intel licensing fee.) All in all, the best news in my view is cheaper Thunderbolt accessories at last!
Thunderbolt
Thunderbolt is an Intel-developed transfer technology, initially licensed only to Apple. It can be daisy-chained. It is bi-directional, unlike USB, meaning it can transmit and receive data simultaneously. Thunderbolt 1 and 2 used the mini display port connector. Thunderbolt 3 uses the USB type-C connector. All Thunderbolt 3 cables will work as USB-C cables and all Thunderbolt 3 ports are compatible with USB 3.1 Type C devices. But, USB Type C ports can not be assumed to support Thunderbolt. Thunderbolt 3 is backward-compatible with previous Thunderbolt versions if you have the right adapter. Because Thunderbolt uses a PCIe bus, you can use it for external graphics cards; this was a pretty big deal for laptop users. Thunderbolt, as of version 2, uses x4 lanes (PCIe 3.0).
Physical Connectors
Popular External Storage Brands:
- G-drive 1TB G-drive
- OWC
- LaCie 4TB Rugged RAID
- CineRAID
- Cheaper alternatives include various USB 3 options: USB 3.0 RAID ($180); MediaSonic; External RAID box USB 3 – Makes a lot of sense. For $100 you get a drive using USB 3 which is enough for the speeds you’ll get from a spinning disk RAID.
- USB-C RAID enclosure – I’ve used one of these for 850MB+ transfer speeds.
- USB 3.0 enclosures are under $10
- Crucial 2TB SSD: A solid, $230 choice
- Seagate 2TB HDD: Same storage capacity but only $60
All of that said, there’s another potential bottleneck to the speed at which you can move those bits around, and that’s the physical hard drive technology you’re using. When it comes to speed, you get what you pay for. There’s a reason for the price disparity between the last two items on that list.
Physical Drives:
- An old-school spinning hard drive (HDD) gets around 80–120MB/s
- A RAID of HDDs can get around 200–300MB/s.
- Single SSD= around 200–500MB/s
- RAIDed SSDs get around 800 MB/s (this is what I work off as my primary drive)
- My iMac’s internal SSD gets around 1,000MB/s write and 2400 MB/sec read. It’s not limited by the SATA connection of external SSD RAID.
- SAS (Serial-Attached SCSI): 6Gb/s
If you want to get started with a simple solution for the least money without understanding most of this, just go with a bus powered (meaning no external power adapter required), lower storage capacity spinning drive for around $50:
NVMe and PCIe
PCIe was the next logical interface for flash memory after good old SATA which capped at around 600 MBps. This brings SSD performance to a new level.
This is a really cool development for the ultimate in speed. MysteryBox writes:
M.2 is a form factor. SATA is an interface. PCIe is an interface. NVMe is a transfer protocol. BUT NVMe is the standardized interface for PCIe SSDs.
“While eGPU and other graphic oriented Thunderbolt devices can utilize the full 40Gbps, this is the first storage device we’ve tested that comes close to actually using all of that glorious bandwidth.
All connections running through a SATA controller share a single PCIe lane, whereas NVMe drives can (depending on motherboard specs) completely bypass the SATA controller and run straight to the CPU over x4 PCIe lanes. In our particular case the SSD’s are connected via a Thunderbolt 3 connections, meaning the data will run though the TB3 controller with negligible latency before hitting the PCIe lanes directly to the CPU. Another benefit of NVMe over previous protocols like SATA (still held back by AHCI) is the command queueing. AHCI was not designed for the bandwidth of flash-based storage, it can only handle a single command at a time with 32 pending commands. No matter how “fast” the SSD hardware was, SATA controllers could never process more than 1 command at a time, creating a serious bottleneck. NVMe was purpose-built for flash storage and can theoretically handle 64,000 queues simultaneously, each with its own 64,000 commands.”
RAID
- RAID 0 – striping
- RAID 1 – mirroring
- RAID 5 – striping with parity (parity is fault protection, in the case of RAID 5 spread across all disks)
- RAID 6 – striping with double parity (very similar to 5, but you can lose two disks and keep going)
- RAID 10 – combining mirroring and striping
Don’t even worry about 2,3,4 and 7 🙂
In some cases, RAID 10 offers faster data reads and writes than RAID 5 because it does not need to manage parity. If you lose a disk, it’s also much faster to rebuild with RAID 10.
RAID is not backup. Physical failures with a drive enclosure can still occur. And it’s important to keep your backups physically separated.
Cloud backups at high frequency can be subject to malware which can mean your backup is also corrupted so make sure to keep adequate versioning.
Not differentiating between read and write speeds (the latter is quicker) but on average, here are some common video data rates:
Video Data Rates
At UHD Resolution:
- ProRes 422 Proxy: 145 Mbps, 65 GB/hr
- Pro Res 422 LT: 328 Mbps, 148 GB/hr
- Pro Res 422: 471 Mbps, 212 GB/hr
- Pro Res 422 HQ: 707 Mbps, 318 GB/hr
Hard Drives and File Systems
When you go to format a hard drive for use with your computer, if that computer’s a Mac, the simple answer is to choose “Mac OS Extended (journaled)” and you’ll likely be ok, but here’s the rundown of various “file system” used by hard drives.
FAT32
The old way to format a spinning disk. Works on Mac and Windows but has a 4GB file size limit. You shouldn’t need to use this option.
NTFS
Windows file system. A Mac can read a drive formatted as NTFS but can’t write to it without some sort of third party plug-in (Google “Paragon”). Bear in mind that you won’t be able to format an NTFS drive using DiskUtility (since you have no write access) so format as ExFAT from a PC if you need to get access to it on a Mac.
APFS
Optimized for flash or solid state but will work with spinning drives as well.
HFS+
AKA “MacOS extended” in newer versions of Mac OS. This is the one to pick if you’re using a spinning disk hard drive. This formats your drive using HFS+, the older of Mac’s native file systems, which won’t work on Windows computers which use the NTFS file system.
exFAT
Less reliable and robust than either NTFS or HFS+ and slower than a native file system. The lack of journaling can lead to data loss as well. It’s optimized for solid state, not spinning disks. It’s more vulnerable to file fragmentation than APFS or Mac OS Extended. For these reasons, software like BlackMagic’s Resolve will not allow its database to be placed on an ExFat drive.
In general, I’d recommend you DO use “Journaled” but DO NOT use case sensitivity. This means the OS will treat “File.dmg” and “file.dmg” the same.
