blog.noetech.com

The Next Final Cut Pro

Here is what I want to see from Final Cut Pro 5.0.

They need project sharing desperately. Avid has a great solution with Unity. It may be over priced, but it allows seamless collaboration between editors. But it can be improved. I would like Apple to approach project sharing a little differently. I would like them to create a FCP server that hosts projects.

I imagine it working like this: The editor would open Final Cut and select Open Remote from the file menu. They would sign in with a username and password. A list of projects would appear listing all the ones that the user has access to. When they picked the project, all the Xsan volumes that were needed for the project would be mounted automatically.

An administrator could set up different permission sets for each user, limiting different settings and items the user could change. For example, you could set it so only admin users could changed the digitize directory and sequence settings, etc.

Multiple editors could work off the same projects, open the same bins etc. Admins could also link certain bins with certain projects. Say for example, you have a project for each episode and one for digitizing. Each episode project would have a link to the media folder in the dig project, so anytime that those bins are updated, they are automatically updated in the main edit projects.

Then to make the collaboration complete, there would be a viewer application that producers, APs, etc, could use to view the cuts over a normal ethernet connection. These users would sign in with just read only permissions and see a list of sequences in the project, or possibly specific ones set by editors. When the user went to play them, the server would automatically stream the sequence to the computer over ethernet, because these systems obviously would not be on the fibre network. With offlineRT, this should not be a problem.

Imagine a system were producers could be watching a cut as the editor is making it. No more making outputs at 6pm after the editor leaves and watching them in the morning after the dubs have been made. I'm sure editors wouldn't be big fans, but I know a few producers who would cream their pants.

There would also be a way for these users to leave notes on the cuts. Marking a transition that they want changed, a clip they want swapped out, etc. If they could also view all the digitized clips, they could even pull out the exact clips they wanted, instead of editors searching through the tapes to find a clip that the producer already found.

Tie it all in with Quicktime streaming server for network deliveries, and we would have complete collaboration, end to end.

That is how I picture the perfect post system.

Of course there are other things the FCP needs. It needs to handle large amounts of media much better. FCP 4.5 is a dog when you have 1,000 hours of footage. And don't tell me that too much, Avid handles it with flying colors. They also need a really solid I/O box. I've yet to find one that everyone is happy with.

If Apple makes something like this happen, I would switch our facility over in a minute and never look back.

Avid Unity and Apple's Xsan

We have an Xsan system setup at work. It's feeding 12 machines. So far we have been really happy with it.

We also have an Avid Unity Lanshare, which we've had for almost two years now, acquired before I worked there. The two products aim to server the same market, though Apple's target is a little less specific.

They approach the problem they aim to solve in two different ways. Apple's approach, which is more typical of SAN systems, is to assign storage pools to volumes. While Avid's is to assign Volumes to pools of storage.

This means that on the Xsan you can have multiple pools per volume, and on Avids, multiple volumes per pool.

If you work with Avid and FCP systems, you'll understand why they have these different approaches. Avid software is old school. It originated from a time where an Avid system was used solely as a cutting station. Where it required so much custom hardware, and huge costs, that it didn't make sense to use it for anything else.

Apple's FCP is new school, where a person may also be using Photoshop, After Effects or just surfing the web on the same machine they are cutting on.

Now of course, Avid systems don't require mass amounts of extra hardware and can be used just like FCP, sharing the computer with other applications, but the heritage is important to remember as you think about how Avid systems are designed and used.

A perfect example of how this heritage directly impacts modem Avid systems is with media storage. Avid assumes that any drive attached to the computer, besides the system drive, is for media storage and nothing else. Avid stores all it's media in a folder called OMFI Media Files on the root of the hard drive. No if, ands or buts. This made sense in the mid to late '90s when the machines were only used for editing. Now that's not true anymore. With 400 gig internal drives, and media compressed 15:1, it's understandable that some people might store more than just Avid media on a drive.

There's nothing that stops you from storing other things on the drive, but you have to keep the OMFI folder at the root. You can't break projects up into subfolders for better organization, etc.

This is a minor inconvenience when you are dealing with local storage, but in a SAN it's a huge problem. Imagine a post facility with 15 Avid systems and a SAN for media storage. In a perfect world you would have that cluster SAN setup to share one volume with subfolders for each project, or editor, or by whatever makes sense for that specific situation. Unfortunately with the way Avid handles and manages media that plan does not work. You have to store your media in that OMFI folder at the root of the volume, so you can't break it down into projects.

OK, so create a Volume for each project right? Well that works assuming that each project only has one editor working on it at the same time. But that's not what they mean by shared media systems. In our pretend post facility, multiple editors need to be able to access the same media at the same time. The Avid editing systems will conflict if you try to do that. The Avid media databases are constantly updating when new media is created or changed. So what happens is when editor A renders to the SAN, editor B's machine suddenly scans the database to add the media it thinks was just added. The problem here is that editor A's machine already added it to the db. This causes the databases to corrupt and requires them to be rebuilt fairly often.

You could work around that problem, it's not that big of a deal to have one machine scan, it's fast since it is over fibre. But there's a bigger problem. When the Avid digitizes it creates a temp file in a folder named Creating inside the OMFI folder. These temp files do not have unique names, the are named Creating01, Creating02 etc.

Now digitize in two edit bays to the same drive and see what happens. Not good.

At this point you might think that creating a volume for each project and each editor in that project would solve all these problems. Editor A reads and writes to Drive A, reads only from Drive B, etc. This does indeed work, with a few notable issues. If you mount the drives read only on the read only edit bays, they will get errors when trying to add to the media databases, which they will do whenever the other machine is digitizing or rendering. The errors are harmless, but annoying and an interference.

But the biggest problem with this situation is the lack of efficient use of your drive storage. On the Xsan, or most other SAN systems I would think, the smallest a volume can be is the size of one LUN. LUNs can not be split between volumes. So, assuming you are using a 3.5 terabyte xServe RAID, which at most can be split into 16 LUNs, your smallest volume will be about 140 gigs. Now let us assume that we have three editors working on one project, so that's three Volumes and three LUNS we need. That's a total of 420 gigs that have to be assigned to that project, not matter how many you actually need. Also, if you happen to need 450 gigs, you have to assign 560. You can see all the wasted space. At $0.43 a gig, roughly, it adds up. And depending on how many projects and editors you have, you may run out of LUNs way before you run out of storage space.

You are also missing out on one of the biggest advantages of using a clustered storage solution, the clustering. By using a LUN per volume, you can't split that volume, and folders, up over storage pools. You can't have, say, one folder on a RAID 5 set and another on a mirrored set. This would be really useful in situations where you have some media that is irreplaceable and not on tape, say VO or graphics. Then other media which you shot so you have on tape.

The way that Avid handles and manages media keeps you from using other SAN solutions, at least using them efficiently if you want to use shared media.

Avid gets around these limitations with their own shared media system, Unity, by making it volume based rather than pool bases. Specifically, you create the pools of storage first, then split them into volumes. Compare this to Apple, where you create the volumes and then added pools of storage.

Further, they get around multiple editors working the same volume by creating a subfolder in the OMFI media folder for each machine using the volume. I now I said you can't do this, and you can't. Avid can, and only on Unity workspaces. The Avid knows which folder you "own" and only writes to that one, reading from the others. (BTW, if anyone knows how to trick an Avid into thinking it's working with a Unity system, I would love to hear about it).

It's an effective solution, but it has it's limits. You don't have much flexibility when creating volumes and pools. Volumes can only be mirrored, protected in Avid terms, or stripped. There is no RAID 5, which is the best compromise between protection and speed. And when talking Avid drive prices, most small to medium shops simply can't afford to "protect" more than the project volume.

Of course there are advantages Avid's media management. Avid systems don't generally lose track of media because it only has one place to look for it. It's fast do to the database it keeps of known media. By having all the media in a predictable place allows them to have the databases track the media, no matter what project they were digitized for.

Anyone who's managed a post facility using FCP, not that many btw, can attest to how editors have a tendency to digitize all over the place. FCP also has a tendency to lose track of it. Re-linking in an Avid app is quick. It knows exactly where to look. In FCP it can take 10 times as long, if not more, because it has to scan every file on every drive.

It's unfortunate that Apple did not come up with a solution for working with Xsan and Avid. It's a big market that they could have easily entered, getting their foot in the door where they desperately want to be, post houses. The Xsan is half to a third the cost of a comparable Unity system. Because it uses xServe RAIDs, it has RAID 5. It's clustered and all fibre based. While the Unity system lacks any sort of true RAID system, is only fibre for their most expensive systems, not Express Pro, and, with the Lanshare, has many single points of failure.

I've heard of a SAN system that fakes the Unity protocol, and makes Avid software think it's working with true Unity workspaces, but I've yet to actually see it.

So what's are small company to do? Converting to FCP is an option, but in it's current form it's just not as robust as Avid solutions. Media management, project sharing and speed are all lacking in FCP 4.5. It does have mode-less editing, better graphics handling and you don't need separate online machines, but until it can handle 1,000 hours of media and 3 editors working on the same project at the same time, it can't really replace Avid. Hopefully 5.0 will include the features that production companies need for episodic reality shows, but until then it's not really an option. Not to mention the investment that companies have already made in Avid technology.

But I did say we have an Xsan system didn't I? So how are we using it despite all the issues laid out above? Simple, we use it in conjunction with our Lanshare.

When I was researching our options for storage late last year, I was reluctant to invest in Avid. Avid's storage has historically been much more expensive than the rest of the market. I don't like paying out $30,000 for 2.5 terabytes of storage, when you can get an xServe RAID with 5.6 terabytes for $13,000. This is for expanded storage, not the initial systems.

Not to mention that there is a good chance, but by no means guaranteed, that FCP will become a major player in the next few years. I didn't want to lock us into Avid and their exceptionally high priced systems.

The solution was to use the Xsan for media consolidation. Using the Lanshare for project sharing, digitizing and rendering. Then, everyday, moving the media down to the Xsan. This solution is by no means perfect, but it does give us a few things we were looking for. All our media is stored on fibre attached storage rather than ethernet. It's stored on RAID 5 drive sets. It has redundant metadata controllers and fibre switches. All things that the Lanshare does not give us, though Unity can have backup metadata controllers.

For the amount of storage that our Xsan system gives us, I estimate that we saved $70,000 over a Unity. It may have it's inconveniences, but none worth that much.

We were able to make the Xsan work for us because we already had a Lanshare. I suspect that there are other companies out there that have a Lanshare and have out gown it. This is a solution that works, and because it's not application specific, it leaves you open to use any NLE you desire in the future.