0% Complete
0/42 Steps

Hard Drives, Transfer Protocols and Data Rates

Thunderbolt 3 uses a USB 3.1 Type C connector, providing 40 Gbps over a PCIe bus to the CPU.

– Huh???

Media files are large, so at some point we inevitably have to cover the technology that helps us handle them. This will be a hopefully not-to-painful look at the terms involved in digital storage and handling.

What’s a bit?

A “bit” is the smallest data unit utilized by a computer. It’s a clever contraction of “binary digit”, so in reality, it’s just a 0 or 1 value. 8 of these combined can represent 256 different combinations (2^8). The following list should be useful. Keep in mind that a lowercase “b” means “bit” while an uppercase “B” means Byte.

  • 8 bits = 1 byte
  • 1000 bytes = 1 kilobyte (KB)
  • 1000 kilobytes = 1 megabyte (MB)
  • 1000 megabytes = 1 gigabyte (GB)
  • 1000 gigabytes = 1 terabyte (TB)
  • 1000 terabytes = 1 petabyte (PB)
  • 1000 petabytes = 1 exabyte (EB)
  • 1000 exabytes = 1 zettabyte (ZB)
  • 1000 zettabytes = 1 Yottabyte (YB)
  • We’ll stop there for now.

Busses and Lanes?

A bus is simply a channel between components in a computer; it’s what allows you to connect external somethings to the brain of the computer, known as the CPU. You’re probably already familiar with the USB (universal serial bus); another common bus is the PCIe bus.

PCIexpress is the update to the older PCI. It’s the interface that connects the computer’s motherboard to external high bandwidth devices. They come in lane counts of 1x, 4x, 8x, and 16x. From the moment your computer starts up, PCIe figures out what’s connected to the motherboard, and then links all your devices together determining the width of each link. The entire bandwidth available depends on the version of PCIe supported.

  • PCIe 1.0, each lane supports 250 MB/s (so a 4x slot transfer rate means 4*250 MB/s = 1 GBps max)
  • PCIe 2.0, each lane supports 500 MB/s (4x slot’s transfer rate means 4*500 MB/s = 2 GBps)
  • PCIe 3.0, each lane supports 1 GB/s (x16 15.75 GBps)
  • PCIe 4.0, each lane supports up to 2 GB/s (x16 provides 31.5 GBps)
  • PCIe 5.0 each lane supports 4GB/s (x16 provides 63 GBps)

You can use the “System Information” app on the Mac to see how many busses you have available to you on your system. It’s useful to know which physical ports on the back of your computer feed to the same bus. As we’re about to get into, the point of all this is to spread the traffic of your data connections across all the available paths.

In short, this means you need be conscious that you have limited bandwidth to work with. There are multiple points at which you can bottleneck the flow of data inside your computer. If you are transferring footage from a USB 2.0 card reader to an NVME SSD, the card reader will make the transfer so slow that your SSD will be wasted. If you load up all your camera cards and hard drives on the same bus, the bus will be saturated.

Also remember that information travels two directions when moving data around. With a total bandwidth of 500MBps, A RED Mini Mag card reader reading at 250MBps will have less than 250MBPs remaining bandwidth for writing to disks.

Speeds

Various standards exist in an attempt to create compatibility between devices. These standards govern cable type, connector, protocols for connection between two devices, power supply, etc. Below is a list of the most common in recent past and a quick estimate of their speed.

  • USB 1.0 = 100 Mbps
  • USB 2.0 = 500 Mbps
  • USB 3.0 (once termed USB 3.1 Gen 1 just to confuse you)= 5 Gbps
  • USB 3.1 (once termed 3.1 Gen 2, again, just to confuse you) = 10 Gb/s (100 Watts of power-check for PD symbol of a battery). Like TB, allows for data and DisplayPort signals.)
  • USB4.0 = 40Gbps. Finally, one connector type, Thunderbolt Compatible & USB PD (power delivery) is mandatory. At last, a “universal” connector.
  • Thunderbolt 1: 10 Gbps (Mini Display Port connector)
  • Thunderbolt 2: 20 Gbps ((Mini Display Port connector)
  • Thunderbolt 3: 40 Gbps (USB Type C connector)
  • Firewire 400: 50MBps
  • Firewire 800: 80MBps
  • Sata 3 = 600MBps (actual bandwidth though the interface runs at 6Gbps)
  • eSATA: 150 MBps
  • Gigabit Ethernet: 100MBps
  • 10 Gigabit Ethernet: 1000MBps

USB

Physical Connectors

  • USB Type A
  • USB Type B
  • USB Type C
Image result for usb type a
Image courtesy of conwire.com

The most confusing of these is USB Type C. USB-C is a physical connector. Like USB Type A and B, C is just a new physical shape. It supports the Thunderbolt 3 protocol and 100 watts of power delivery. The USB-C spec even includes audio transmission, but so far it has not universally replaced the 3.5mm headphone jack. 

USB 4 includes Thunderbolt compatibility if the manufacturer of the device elects to include it. Intel donated Thunderbolt specs to the USB implementers forum. In September of 2019, the USB Implementers Forum detailed the USB4 standard. It’s a 40Gbps and a two-lane interface. Devices incorporating the tech due to appear in 2020. In the past, many devices used the USB-C connection but lacked Thunderbolt 3 support. USB4 should mitigate that, meaning most USB4 devices should also support Thunderbolt (though I wouldn’t be surprised if phones don’t because while the tech is license-free the advertising of Thunderbolt support is still taxed by an Intel licensing fee.) All in all, the best news in my view is cheaper Thunderbolt accessories at last!

Thunderbolt

Thunderbolt is an Intel-developed transfer technology, initially licensed only to Apple. It can be daisy-chained. It is bi-directional, unlike USB, meaning it can transmit and receive data simultaneously. Thunderbolt 1 and 2 used the mini display port connector. Thunderbolt 3 uses the USB type-C connector. All Thunderbolt 3 cables will work as USB-C cables and all Thunderbolt 3 ports are compatible with USB 3.1 Type C devices. But, USB Type C ports can not be assumed to support Thunderbolt. Thunderbolt 3 is backward-compatible with previous Thunderbolt versions if you have the right adapter. Because Thunderbolt uses a PCIe bus, you can use it for external graphics cards; this was a pretty big deal for laptop users. Thunderbolt, as of version 2, uses x4 lanes (PCIe 3.0).

Physical Connectors

Image result for thunderbolt
Image result for thunderbolt
Image courtesy Intel

Popular External Storage Brands:

All of that said, there’s another potential bottleneck to the speed at which you can move those bits around, and that’s the physical hard drive technology you’re using. When it comes to speed, you get what you pay for. There’s a reason for the price disparity between the last two items on that list.

Physical Drives:

  • An old-school spinning hard drive (HDD) gets around 80–120MB/s
  • A RAID of HDDs can get around 200–300MB/s.
  • Single SSD= around 200–500MB/s
  • RAIDed SSDs get around 800 MB/s (this is what I work off as my primary drive)
  • My iMac’s internal SSD gets around 1,000MB/s write and 2400 MB/sec read. It’s not limited by the SATA connection of external SSD RAID.
  • SAS (Serial-Attached SCSI): 6Gb/s

If you want to get started with a simple solution for the least money without understanding most of this, just go with a bus powered (meaning no external power adapter required), lower storage capacity spinning drive for around $50:

Seagate 1TB USB 3.0

NVMe and PCIe

PCIe was the next logical interface for flash memory after good old SATA which capped at around 600 MBps. This brings SSD performance to a new level.

OWC Thunderblade

This is a really cool development for the ultimate in speed. MysteryBox writes:

M.2 is a form factor. SATA is an interface. PCIe is an interface. NVMe is a transfer protocol. BUT NVMe is the standardized interface for PCIe SSDs.

“While eGPU and other graphic oriented Thunderbolt devices can utilize the full 40Gbps, this is the first storage device we’ve tested that comes close to actually using all of that glorious bandwidth.

All connections running through a SATA controller share a single PCIe lane, whereas NVMe drives can (depending on motherboard specs) completely bypass the SATA controller and run straight to the CPU over x4 PCIe lanes. In our particular case the SSD’s are connected via a Thunderbolt 3 connections, meaning the data will run though the TB3 controller with negligible latency before hitting the PCIe lanes directly to the CPU. Another benefit of NVMe over previous protocols like SATA (still held back by AHCI) is the command queueing. AHCI was not designed for the bandwidth of flash-based storage, it can only handle a single command at a time with 32 pending commands. No matter how “fast” the SSD hardware was, SATA controllers could never process more than 1 command at a time, creating a serious bottleneck. NVMe was purpose-built for flash storage and can theoretically handle 64,000 queues simultaneously, each with its own 64,000 commands.”

RAID

  • RAID 0 – striping
  • RAID 1 – mirroring
  • RAID 5 – striping with parity (parity is fault protection, in the case of RAID 5 spread across all disks)
  • RAID 6 – striping with double parity (very similar to 5, but you can lose two disks and keep going)
  • RAID 10 – combining mirroring and striping

Don’t even worry about 2,3,4 and 7 🙂

In some cases, RAID 10 offers faster data reads and writes than RAID 5 because it does not need to manage parity. If you lose a disk, it’s also much faster to rebuild with RAID 10.

RAID is not backup. Physical failures with a drive enclosure can still occur. And it’s important to keep your backups physically separated.

Cloud backups at high frequency can be subject to malware which can mean your backup is also corrupted so make sure to keep adequate versioning.

Not differentiating between read and write speeds (the latter is quicker) but on average, here are some common video data rates:

Video Data Rates

At UHD Resolution:

  • ProRes 422 Proxy: 145 Mbps, 65 GB/hr
  • Pro Res 422 LT: 328 Mbps, 148 GB/hr
  • Pro Res 422: 471 Mbps, 212 GB/hr
  • Pro Res 422 HQ: 707 Mbps, 318 GB/hr

Hard Drives and File Systems

FAT32

The old way to format a spinning disk. Works on Mac and Windows but has a 4GB file size limit.

NTFS

Windows file system.

APFS

Optimized for flash or solid state but will work with spinning drives
HFS+ AKA “MacOS extended” in newer Mac OS

exFAT

Less reliable and robust than either NTFS or HFS+ and slower than a native file system. The lack of Journaling can lead to data loss as well. It’s optimized for solid state, not spinning disks. more vulnerable to file fragmentation than APFS or Mac OS Extended
Macs will read but not write NTFS, Windows often won’t even read HFS+.
DO use “Journaled”

DO NOT use case sensitivity. This means the OS will treat “File.dmg” and “file.dmg” the same.

Scroll to Top