you've build one before, right?

[Sergey Brin - Co-Founder, Google] Yes. I think the most important thing

if you're going to build a search engine

is to have a really good corpus to start out with.

In our case we used the world wide web, which at time was certainly smaller than it is today.

But it was also very new and exciting.

There were all sorts of unexpected things there.

[David Evans] So the goal for the first three units for the course is to build that corpus.

And we want to build the corpus for our search engine

by crawling the web and that's what a web crawler does.

What a web crawler is, it's a program that collects content from the web.

If you think of a web page that you see in your browser, you have a page like this.

And we'll use the udacity site as an example web page.

It has lot's of content, it has some images, it has some text.

All of this comes into your browser when you request the page.

The important thing that it has is links.

And what a link is, is something that goes to another page.

So we have a link to the frequently asked questions,

we have a link to CS 101 page.

There's some other links on the page.

And that link may show in you browser with an underscore,

it may not, depending on how your browser is set.

But the important thing that it does,

is it's a pointer to some other web page.

And those other web pages may also have links

so we have another link on this page.

Maybe it's to my name, you can follow to my home page.

And all the pages that we can find with our web crawler

are found by following the links.

So it won't necessarily find every page on the web

If we start with a good seed page

we'll find lot's of pages, though.

And what the crawler's gonna do is start with one page,

find all the links on that page, follow them to find other pages

and then on those other pages it will follow the links on those pages

to find other pages and there will be lot's more links on those pages.

And eventually we'll have a collection of lot's of pages on the web.

So that's what we want to do to build a web crawler.

We want to find some way to start from one seed page,

extract the links on that page,

follow those links to other pages,

then collect the links on those other pages,

follow them, collect all that.

So that sounds like a lot to do.

We're not going to all that this first class.

What we're going to do this first unit, is just extract a link.

So we're going to start with a bunch of text.

It's going to have a link in it with a URL.

What we want to find is that URL,

so we can request the next page.

The goal for the second unit

is be able to keep going.

if there's many links on one page, you will want to be able to find them all.

So that's what we'll do in unit 2,

is to figure out how to keep going to extract all those links.

In unit three, well, we want to go beyond just one page.

So by the end of unit two we can print out all the links on one page.

For unit 3 we want to collect all those links, so we can keep going,

end up following our crawler to collect many, many pages.

So by the end of unit three we'll have built a web crawler.

We'll have a way of building our corpus.

Then the remaining three units will look at how to actually respond to queries.

So in unit four we'll figure out how to give a good response.

So if you search for a keyword, you want to get a response that's a list of the pages

where that keyword appears.

And we'll figure out in unit five a way to do that, that scales, if we have a large corpus.

And then in unit six what we want to do is, well, we don't just want to find a list,

we want to find the best one.

So we'll figure out how to rank all the pages where that keyword appears.

So we're getting a little ahead of ourselves now,

because all we're going to do for unit one,

is to figure out how to extract a link from the page.

And the search engine that we'll build at the end of this

will be a functional search engine.

It will have the main components that a search engine like Google has.

It certainly won't be as powerful as Google will be,

we want to keep things simple.

We want to have a small amount of code to write.

And we should remember that our real goal

is not as much to build a search engine,

but to use the goal of building a search engine as a vehicle

for learning about computer science

and learning about programming

so the things we learn by doing this

will allow us to solve lot's and lot's of other problems.