So I'm Raymond.

I'm not television's Raymond Blum,

that you may remember from "The Facts of Life."

I'm a different Raymond Blum.

Private joke.

So I work in site reliability at Google

in technical infrastructure storage.

And we're basically here to make sure that things, well,

we can't say that things don't break.

That we can recover from them.

Because, of course, we all know things break.

Specifically, we are in charge of making sure

that when you hit Send, it stays in your sent mail.

When you get an email, it doesn't go away

until you say it should go away.

When you save something in Drive,

it's really there, as we hope and expect forever.

I'm going to talk about some of the things

that we do to make that happen.

Because the universe and Murphy and entropy

all tell us that that's impossible.

So it's a constant, never-ending battle

to make sure things actually stick around.

I'll let you read the bio.

I'm not going to talk about that much.

Common backup strategies that work maybe outside of Google

really don't work here, because they typically scale effort

with capacity and demand.

So if you want twice as much data backed up,

you need twice as much stuff to do it.

Stuff being some product of time, energy, space, media, et

cetera.

So that maybe works great when you go from a terabyte

to two terabytes.

Not when you go from an exabyte to two exabytes.

And I'm going to talk about some of the things we've tried.

And some of them have failed.

You know, that's what the scientific method is for,

right?

And then eventually, we find the things that work,

when our experiments agree with our expectations.

We say, yes, this is what we will do.

And the other things we discard.

So I'll talk about some of the things we've discarded.

And, more importantly, some of things

we've learned and actually do.

Oh, and there's a slide.

Yes.

Solidifying the Cloud.

I worked very hard on that title, by the way.

Well, let me go over the outline first, I guess.

Really we consider, and I personally

obsess over this, is that it's much more important.

You need a much higher bar for availability of data

that you do for availability of access.

If a system is down for a minute, fine.

You hit Submit again on the browser.

And it's fine.

And you probably blame your ISP anyway.

Not a big deal.

On the other hand, if 1% of your data goes away,

that's a disaster.

It's not coming back.

So really durability and integrity of data

is our job one.

And Google takes this very, very seriously.

We have many engineers dedicated to this.

Really, every Google engineers understands this.

All of our frameworks, things like Big Table and formerly

GFS, now Colossus, all are geared towards insuring this.

And there's lots of systems in place

to check and correct any lapses in data availability

or integrity.

Another thing we'll talk about is redundancy,

which people think makes stuff recoverable,

but we'll see why it doesn't in a few slides from now.

Another thing is MapReduce.

Both a blessing and a curse.

A blessing that you can now run jobs

on 30,000 machines at once.

A curse that now you've got files

on 30,000 machines at once.

And you know something's going to fail.

So we'll talk about how we handle that.

I'll talk about some of the things

we've done to make the scaling of the backup resources

not a linear function of the demand.

So if you have 100 times the data,

it should not take 100 times the effort to back it up.

I'll talk about some of the things

we've done to avoid that horrible linear slope.

Restoring versus backing up.

That's a long discussion we'll have in a little bit.

And finally, we'll wrap up with a case study, where Google

dropped some data, but luckily, my team at the time

got it back.

And we'll talk about that as well.

So the first thing I want to talk about is what I said,

my personal obsession.

In that you really need to guarantee

the data is available 100% of the time.

People talk about how many nines of availability

they have for a front end.

You know, if I have 3 9s 99% of the time is good.

4 9s is great.

5 9s is fantastic.

7 9s is absurd.

It's femtoseconds year outage.

It's just ridiculous.

But with data, it really can't even be 100 minus epsilon.

Right?

It has to be there 100%.

And why?

This pretty much says it all.

If I lose 200 k of a 2 gigabyte file,

well, that sounds great statistically.

But if that's an executable, what's

200 k worth of instructions?

Right?

I'm sure that the processor will find some other instruction

to execute for that span of the executable.

Likewise, these are my tax returns

that the government's coming to look at tomorrow.

Eh, those numbers couldn't have been very important.

Some small slice of the file is gone.

But really, you need all of your data.

That's the lesson we've learned.

It's not the same as front end availability,

where you can get over it.

You really can't get over data loss.

A video garbled, that's the least of your problems.

But it's still not good to have.

Right?

So, yeah, we go for 100%.

Not even minus epsilon.

So a common thing that people think,

and that I thought, to be honest, when I first

got to Google, was, well, we'll just make lots of copies.

OK?

Great.

And that actually is really effective against certain kinds

of outages.

For example, if an asteroid hits a data center,

and you've got a copy in another data center far away.

Unless that was a really, really great asteroid, you're covered.

On the other hand, picture this.

You've got a bug in your storage stack.

OK?

But don't worry, your storage stack

guarantees that all rights are copied

to all other locations in milliseconds.

Great.

You now don't have one bad copy or your data.

You have five bad copies of your data.

So redundancy is far from being recoverable.

Right?

It handles certain things.

It gives you location isolation, but really, there

aren't as many asteroids as there

are bugs in code or user errors.

So it's not really what you want.

Redundancy is good for a lot of things.

It gives you locality of reference

for I/O. Like if your only copy is in Oregon,

but you have a front end server somewhere in Hong Kong.

You don't want to have to go across the world

to get the data every time.

So redundancy is great for that, right?

You can say, I want all references

to my data to be something fairly local.

Great.

That's way you make lots of copies.

But as I said, to say, I've got lots of copies of my data,

so I'm safe.

You've got lots of copies of your mistaken deletes

or corrupt rights of a bad buffer.

So this is not at all what redundancy protects you

against.

Let's see, what else can I say?

Yes.

When you have a massively parallel system,

you've got much more opportunity for loss.

MapReduce, I think-- rough show of hands,

who knows what I'm talking about when I say MapReduce?

All right.

That's above critical mass for me.

It's a distributed processing framework

that we use to run jobs on lots of machines at once.

It's brilliant.

It's my second favorite thing at Google,

after the omelette station in the mornings.

And it's really fantastic.

It lets you run a task, with trivial effort,

you can run a task on 30,000 machines at once.

That's great until you have a bug.

Right?

Because now you've got a bug that's

running on 30,000 machines at once.

So when you've got massively parallel systems like this,

the redundancy and the replication

really just makes your problems all the more horrendous.

And you've got the same bugs waiting to crop up everywhere

at once, and the effect is magnified incredibly.

Local copies don't protect against site outages.

So a common thing that people say is, well, I've got Raid.

I'm safe.

And another show of hands.

Raid, anybody?

Yeah.

Redundant array of inexpensive devices.

Right?

Why put it on one disc when you can put it on three disks?

I once worked for someone who, it was really sad.

He had a flood in his server room.

So his machines are underwater, and he still said,

but we're safe.

We've got Raid.

I thought, like-- I'm sorry.

My mom told me I can't talk to you.

Sorry.

Yeah.

It doesn't help against local problems.

Right?

Yes, local, if local means one cubic centimeter.

Once it's a whole machine or a whole rack, I'm sorry.

Raid was great for a lot of things.

It's not protecting you from that.

And we do things to avoid that.

GFS is a great example.

So the Google File System, which we use throughout all of Google

until almost a year ago now, was fantastic.

It takes the basic concept of Raid,

using coding to put it on n targets at once,

and you only need n minus one of them.

So you can lose something and get it back.

Instead of disks, you're talking about cities or data centers.

Don't write it to one data center.

Write it to three data centers.

And you only need two of them to reconstruct it.

That's the concept.

And we've taken Raid up a level or five levels, maybe.

So to speak.

So we've got that, but again, redundancy isn't everything.

One thing we found that works really, really well,

and people were surprised.

I'll jump ahead a little bit.

People were surprised in 2011 when

we revealed that we use tape to back things up.

People said, tape?

That's so 1997.

Tape is great, because it's not disk.

If we could, we'd use punch cards, like that guy in XKCD

just said.

Why?

Because, imagine this.

You've got a bug in the prevalent device drivers.

Used first SATA disks.

Right?

OK.