Amateur productions are often very lax about proper digital asset protection so I encourage you to pay attention, even if some of this seems excessive.
On a bigger film set, a DIT or “Digital Imaging Technician” is responsible for media backup, possibly sync and transcode, and occasionally the more technical job of camera consultation for digital productions. They can play a role in helping communicate the intention of the cinematographer from set to post. When you consider the number of people involved on a higher end film, not to mention any stunts, effects, or performances that are not easy to repeat, the cost of the media captured is extremely high. For this reason, professional workflows and positions exist to safeguard the data.
Here’s the general workflow:Camera operators will eject the “exposed” camera media, mark it with red tape and write an incrementing number to identify the “mag” (AKA “memory card”) and the camera it came from. For example, if it’s the seventh mag of the day from camera A it might get labeled A007. The camera crew will put red tape over the contacts of the memory card. The DIT would receive the card at a dedicated place on their cart, often on the left side, put the red tape with the mag name on top of the card reader, put the ‘mag’ into the reader, copy the footage to a local hard drive array using special software that verifies the success of the copy, and then make more copies for safety as well as for the members of production and post production who will need the footage. This is the part of the job known as “Data Wrangling”. It’s also wise to keep a log of the cards you’ve processed so you can account for them. A simple Excel spreadsheet is sufficient. Accurate tracking of data and information is important in this job. Attention to detail, trustworthiness, focus and maintaining a clear head in high-pressure situations is all part of a good DIT.
The positions of DIT and assistant editor can also overlap. Sometimes a DIT will also be asked to make dailies/rushes and transcodes. “Dailies” are simply easy-to-share, smaller, video files that the creative team can receive daily to check the results of the previous day’s shoot. “Transcodes” are files created in a way that guarantees they will edit smoothly. These transcodes are a part of those above-mentioned files that will be put on a drive and sent to the post-production house. They’ll have a “one-light” grade applied to balance the footage into something that’s pleasant to view. In this way, the DIT can operate similar in function to the old film processing labs as they are first to ‘develop’ the footage.
Using your operating system’s file browser (AKA Mac’s “Finder” or Windows “Explorer”) to perform a file copy is generally frowned upon. Though the likelihood of anything going wrong is very slim, and the OS does a lot of file verification on its own, the industry standard, mandated by insurance companies, is to perform a checksum. A checksum looks at the data being copied and then performs a random bit verification comparison between source and destination, generating an alphanumeric hash as a unique descriptor of that data. If the hashes at the source and destination match, the data is assumed to be identical. This process makes the copy take longer, but adds a bit of security to the process of file transfer.
We cover sound sync later on. Sometimes that’s the domain of the DIT but it’s often not the case in modern productions. Sometimes the production isn’t even shooting dual system, the sound dept. is down mixing to camera so you don’t need to sync the slightly higher quality audio for dailies, and/or the post team (assistant editor) will handle sync for the show.
There are some limitations to Resolve for DIT work, but you can’t beat the price (free). Most of the limitations are speed-related. When you get an application dedicated to doing one thing it’s inherently more efficient. Resolve’s “Clone” feature on the first “Media” page can be used to generate a verified duplicate of the files on your camera media without purchasing any software. It may be slower than alternatives, but this whole idea of checksumming is debatable as your operating system does it on any file copy anyway.
Incidentally, if you’re transcoding files is that there’s no background rendering in Resolve so you may encounter a similar speed barrier. If you’re on a bigger production where you need to transcode a lot of material, having two computers available to you and using Resolve’s remote rendering feature will likely be required.
Bob Trim: “The biggest issue comes with exporting for AVID. As of version 10.1 of DaVinci, has had issues with the export of proper .mxf, .aaf, and .ale files needed for a feasible workflow within AVID. The issue is with both AVID and Resolve on the metadata side of things. You will need to find another tool for DIT work when as ked for AVI D deliverables. If the assets will be edited by you, or by a smaller posthouse where extensive VFX or audio work is not needed, DaVinci is a wonderful tool to use.”
Excellent software, slightly less intuitive than the alternatives, but it offers cataloging features as well.
This is an easy-to-use and very popular alternative.
Hedge is a bit of a unique solution in that it prioritizes speed and file verifications. Normally running a checksum can take you over 50% longer to run your copy. That makes a huge difference when you’re dealing with lots of data.
When your choice is between getting a copy 4-5 x faster, or even 1.5 – 2x faster and running checksums, you’re better of getting the copy faster and doing the visual inspection, right away. Most applications that run checksums, with the exception of programs like Hedge that trades off RAM caching the data for not adding separate read operations to the original media for checksum calculation, need additional time to prep a MHL or other set of hashes to act as checksums – there’s a lot that can happen in just an additional 10 minutes when it comes to copying footage, and if you want to reduce the actual risks of data loss, minimizing the time it takes to get that second copy is essential. This takes time, and if you cache to RAM and it’s not ECC RAM you’re actually introducing more risk that the copy will go wrong.
Hedge reads the data from the camera media, briefly caches it to RAM, and then writes it multiple output locations simultaneously. At the same time it generates hashes from the RAM cached data. RAM caching means that it’s capable of creating multiple copies and a media hash list faster than Finder or Windows Explorer can make a single copy. That’s a program with real value.
The caveat here is that by using a RAM cache, it actually increases the odds of a copy error, unless you’re running it on workstation class hardware with error corrected RAM (ECC RAM). Not by a small amount either – the probabilities of error go up by several orders of magnitude to the point where they can’t be ignored. Which means if you’re copy application is using a RAM cache, suddenly checksums add real value in verifying the copy went well!
Paul, one of the developers from Hedge:
“Other apps in the space all bypass the provisions the OS offers for copying and use a brute force method to determine if there’s an issue. We do it the other way around: we use the OS-built in issue reporting (that happens about once in 100m times), but crossverifiy the results. That results in about one in 1m files with an issue, which we then fix. Hedge is vastly more complicated under the hood than it seems. That we don’t have the interface tell you every bit and piece is deliberate: it’s what makes the UX good.”
Incidentally, before we talk of clearing the card, it’s good practice to not fill the card all the way. Don’t fill your media 100%. Try to swap for a new mag when the old one is around 80% full.
SD cards above 32GB will automatically be formatted with a file system call EXFAT which is both Mac and PC compatible. Formatting the cards in camera can also be a good idea. Develop a good habit now of managing your files from camera card to computer. I love an application called “ParaShoot” for this.
It’s good practice to format the camera card from the computer, generally as exFAT. This means that if you or someone else puts in a camera they’ll know it’s been through the offload process. If they get a card with media still on it, there’s always the question of whether or not it’s been offloaded. Formatting a card will set your heart racing on big shoots, but duplicate data can be dangerous. Again, it’s easier for a camera operator to receive the formatted mag, check that it’s clear, and then perform a camera re-format. The operator will likely contact you if there is still media on the card. This can be done from “Disk Utility” on a Mac, but my favorite way is via a simple, free app called ParaShoot.
This desktop application erases the card by flipping every bit of the first 2MB (2 * 1024 * 1024 bytes) of the card. Every 0 becomes a 1 and every 1 a 0. This destroys all file system information in a controlled way and can be easily reversed.
Making a backup is essential for critical work. Redundancy is a big deal, and keeping a copy offsite (via physical drive or in the cloud) is an even more secure option. In the normal DIT workflow the RAID array has some redundancy but wouldn’t be considered a backup. The shuttle drives that are going to their various destinations are a great backup.
All that being said, digital files are still subject to corruption. The most frequent culprits are the media itself or the camera that writes it. If your camera writes faulty data to a card, all a checksum does is verify that the corrupt data is corrupt on the source and destination. It’s always a good idea to perform as much of a visual inspection of copied data as time allows. The DIT is often the first person with the chance to really see what’s being shot on a good display and in a controlled environment. You can act as a sort of QC, advising the production if issues arise.
Once footage is safely offloaded, it’s not a bad idea to mark the camera media with a piece of green tape and send it back to the camera it came from.
The main array that you’ll put all your footage on needs to be sizable and speedy. Some people will even use a NAS with direct-attached-storage functionality to offload files on set and then plug it into the network when they arrive at the post facility. From there it will be transferred to shared storage for use by the editorial team. Back up to the main array first thing. You can copy from that to your shuttle drives simultaneously throughout the day. At day’s end if get a mag when everything is shutting down, you can copy from one card to all three locations (your shuttle drives and main array) all at once, but expect a performance hit of around 10%.
Pegasus and OWC Thunderbay are popular arrays. QNAP NAS is a valid DAS and NAS all-in-one. Opt for the QNAP as soon as you want multiple machines connected to the same storage. That’s why it’s “Network” Attached Storage. In the studio these other machines would be editorial workstations. In the field, another computer dedicated to generating proxies is the biggest reason to do this.
Again, dailies are simply small, easy-to-read conversions of the high quality original camera media. They’re light weight enough to be shared, often via cloud services, and they’ll usually contain burn-ins: text overlays with useful information. The are transcoded to something ubiquitous like h.264.
Burn-ins often include source timecode, file name and possibly even scene and take info. Workspace>Data Burn In is the panel where you configure this in Resolve. “Project” applies to the whole project and “Clip” to just that one clip.
You’ll get a confusing array of answers on what the “industry standard” folder structure is for moving camera media to a hard drive. Take anyone insisting that their way is standard with a grain of salt. I would first advocate that you find a way that makes sense for your workflow. Personally-tailored experience is far more valuable than other people’s opinions on the internet. If you’re working on a larger production, the best approach is to simply ask whoever you are delivering to what they want. If you’re the one creating dailies, backups and archival, find out from the post house how they want the files organized. The jobs of DIT, Data Wrangler and Dailies operator can be three separate positions, depending on the scale of the shoot. The placement of footage into daily (of half-day) folders and scene classifications helps clear up any misplaced scene or take labeling. It’s another form of QC and data verification provided by a DIT.
Camera reports, script supervisor’s notes, and sound notes head to post with the DIT’s drives containing original camera media and one-light transcodes. Pomfort has good material on their website detailing this.
There should be someone in the post house (post supervisor, editor or assistant editor) with an opinion on how things are organized. That said, here are a few ideas to consider:
Use CamelCase over the underscore _ to keep filenames shorter. It’s still quite readable and it makes the hyphens and underscores a more apparent organizational tool. E.g. 20190808-TheProject-v01. I used to work with underscores and hyphens but found it easier to solely use hyphens as separators.
Separating footage by day is generally a solid approach. It makes it easier to keep footage organized and ensure nothing goes missing.
Production Name (My Crazy-Cool Movie) 20190913 Date (by day) A_Cam A001 Proxies (if camera-generated, I’d put proxies here so they stay by their online counterpart) A002 B_Cam B001 Sound (dump of sound departments cards, possibly only at lunch or end of day.) Looks Reports (Dailies don’t necessarily need to go to editorial, but save them on local array & deliver to cloud or production team).
Similar example inside Finder:
The filename is Camera, Mag Number, Clip Number, Month and Day, Random 2-letter Hash. It tries to obviate file name repetition. Reel names are one way to avoid conforming issues. For example this was a shot I created from camera A, mag 54, clip 38, December 14th. ADD reel names if they aren’t there already in Resolve.
Canon’s XF-AVC update to C200 is similar:
I did a firmware update mostly because of customizable file-naming so not every first clip on a disk is named clip 0001.mp4. That’s a consumer-oriented compression scheme with a weak file naming protocol. Using MXF instead, I get the same compression, but in a different wrapper and with more robust support for things like useful filenames: A007C065_180712FR_NAME1.MXF
“If you’re working on a show like Vinyl that might have like, say a $200 million dollar budget for 9 episodes. That’s $25 million per episode and it’s 15 [filming] days. So that works out to roughly 2 or 3 million dollars a day and you’re the keeper of the footage. So if you screw up you might have cost production 2 to 3 million dollars in footage.”
“In pre-pro camera tests: That will be me, the DP, the colorist and maybe their dailies colorist is also there. We’ll sit there and tweak and tweak a deLOG to get a it to where everybody is happy and then the colorist will just export the deLOG and send it to me.”
“I have a 16 terabyte RAID 0 that I use for my on set storage. Everything is either Thunderbolt 2 or USB 3 for connectivity when offloading”
“The thing that you have to realise when doing large transfers is that whatever is the slowest part of your chain will dictate how fast things go. So if all your hard drives are plugged directly into your computer but then your card reader is like the third in a chain on the USB hub, then your transfer speed is going to be dictated by your card reader.”
A ‘headless’ server has no input devices attached to it. These can be essential as “render nodes” for transcode.
“If you use a Windows PC as a transcoding workstation you just have to hook it up via thunderbolt to the Mac and setup a homegroup with file access. Thunderbolt is able to emulate a 10GbE connection giving the Mac access to whatever storage is attached to the PC. Of course, that’s in theory (I haven’t tested it yet).”
“We are running the TVS-882ST3 as on-location storage for our DITs. Works great in the field with fast TB3 transfers and back in the office super fast transfers with the built in 10GbE to our main servers. Only thing that isn’t working as expected is the T2E (thunderbolt to Ethernet) functionality where you can use the nas as 10GbE interface for your TB3 equipped Mac.”
“MysteryBox: Regardless of what we’re putting the footage on for a client later, we always copy to a local RAID first: a RAID 5 Promise Pegasus2 R8 at the office, or a G-Technology G-SPEED Shuttle XL for on location transfers.”
Organization is important