Edmond Lau: "The Effective Engineer" | Talks at Google - VoiceTube: Learn English through videos!

Subtitles section Play video

MALE SPEAKER: Thank you for coming to Edmond Lau's talk.
He will be talking about how to be a more effective engineer.
I met him about four months ago or so,
and it was also during one of his talks.
And I was just blown away by it.
And I thought it would be so useful to share his experience
and knowledge with the rest of Google.
He is actually also an ex-Google.
So, he worked at Google before.
He also worked at various startups.
And this talk will be based on his experience as well
as everyone he interviewed with, and what
makes for an effective engineer and for an effective team.
So without further ado, let me introduce to you Edmond Lau.
[APPLAUSE]
EDMOND LAU: Yeah, it's great to be back.
I think it's been maybe 10 years since I first joined
Google right out of school.
I joined the search quality team.
And since leaving Google, I've worked
at a bunch of different startups,
including Ooyala, Quora, and Qup.
And two of those were actually founded by former Googlers.
And so, even though I've been away
from Google for quite some time, Google culture
has actually been a large part of the various startups
that I worked at.
And earlier this year, I did publish a book, "The Effective
Engineer."
And what I like to do during this talk
is share with you some of the stories and lessons
that I've collected during these experiences.
And so my promise to you during this talk
is that you sort of walk away with a framework and a set
of actual strategies that you can apply to your day
to day job on how to become more effective as an engineer.
But, before we dive into the meat of the talk,
I want to do a quick poll and get a show of hands.
Who here has pulled a long night on a project for work before?
Who here has done that-- who's done work on the weekends
before?
A large number.
Who here has worked multiple months on a project, only
to see maybe it not actually launch?
What about working multiple months on a project and then
see it launch, but not actually sort
of seeing the impact or the effect
that you actually wanted?
And let's do one more.
Who here has to do pager duty?
And how many of you have been paged in the middle of night
or over the weekends before?
Yeah, so it's a good number of people.
And when this stuff happens, a lot of times we sort of wonder,
was all this time and energy, was it worth it?
Was this actually the best use of my time?
And this isn't a question that you just sort of ask at Google.
It's also questions that we ask at other places as well.
I remember when I left Google, the week
immediately afterwards, I jumped-- dove head first
into my first startup.
I didn't even take a week of break.
And this first startup was Ooyala.
It was an online video company focused on building
a platform for the enterprise.
It was founded by a former Googler.
I remember my very first week, the CTO told me,
you're responsible for building this feature that's
already been promised to a paying customer.
And the video player has written an actionscript,
because it was a Flash based video player.
And it was a language that I didn't know.
The servers were all written in Ruby
on Rails, which I had no familiarity with.
And there was not a single unit test
across the entire codebase.
And so, I was really scared of just accidentally breaking
something production.
And so, my first two weeks there,
I ended up pulling 70 to 80 hour weeks to try to get that done.
And when I want to talked to my CTO
about how intense this felt, his response was sink or swim.
There's no life preserver coming.
You're just going to have to work hard.
And that was a very intense and very stressful
time in my career.
And so, the next startup I joined
wasn't all that different.
When I join Quora, which was a question answer site,
the jobs page actually read, you should
be ready to make this startup the primary focus of your life.
That's sort of how intense the culture was.
And that time, all of these narratives
about how you need to work hard to succeed sort of really
lined up with how we were thinking.
My team and I, we really wanted to succeed.
But at the same time, we knew we were the underdogs.
While we were building an online video
platform for the enterprise, YouTube was free.
Our biggest competitor, Brightcove,
dominated most of the market.
And so we thought, one of our secret weapons
was that we're just going to outwork everyone else.
We're going to work so hard that we're
going to overcome all these obstacles and actually succeed.
And it took a while before I started wondering,
was this actually the best use of our time?
Because we worked on projects where
we'd spend multiple months building a future for a paying
customer, but then they would never use it.
Or at Quora, we'd build content tools.
And it just wouldn't get the adoption from users
that we actually wanted.
There were all these things where we spent so much time
on these projects, by the end of day,
it didn't really lead to any impact.
And so, we started sort of wondering,
can we actually work smarter?
Can we deliver significantly more value
while working fewer hours?
And it's sort of like after doing years of actually working
these crazy hours before I had this key insight that effort
doesn't really equal impact.
You may have good intentions, but those good intentions
don't actually lead to results.
Think about a staff engineer at Google.
They probably produce more than twice the impact
or twice the output of a more junior engineer,
but they're probably not working 2x the hours.
Or think about Jeff Dean.
So, I heard Google had to sort of invent some new engineering
level for him, like senior Google fellow.
He probably produces more than 10x the impact of most of us
in this room.
But he's not working 10x the hours
because that's physically impossible.
There aren't that many hours in the day.
And so, the notion that effort is
sort of what determines our impact
isn't really a correct one, and we need a better framework
for the really understanding and thinking and reasoning
about impact and effectiveness.
And so, that's sort of where I sort of came
upon this principle of leverage, where leverage
is defined as impact that you produce for the time
that you invest.
It's the rate of return on your investment of time and energy.
It's the ROI.
It's the rate of impact.
If you think about the Pareto Principle, the 80-20 Rule,
it's the 20% of work that actually
produces 80% of the results.
Those are the tasks, those are the activities that actually
are extremely high leverage.
Now, sort of a central thesis that I want you to take away
from this talk is that this central concept of leverage
is really the guiding metric that effective engineers should
use to determine how and where to spend their time.
Now, it might seem a little obvious and maybe
a little simple to some of us, but the same time, oftentimes
we're so engrossed in what we're doing,
we're so focused on the project that we're working on
that we don't really take the time to ask,
are we working the right thing?
We're attending meetings.
That might not be the best use of our time.
Or we're fixing the next urgent bug,
or we're just fighting the next fire.
Or we're working on projects that end up not shipping or not
going anywhere.
And we don't really take a step back to think about,
was our time on these projects actually well spent?
Were those activities actually high leverage?
And you can think about if you're walking along the road
and you see a boulder, that boulder is really hard to move.
But if you can find the right lever,
then you can apply just a little bit of force
and move that boulder out of the way,
because that lever amplifies your output.
In a similar way, in software engineering,
we're looking for those levers where
we can amplify our energy, our time and produce massive gains.
And so, the next question we might ask ourselves,
given that we have this framework,
is how do we actually apply it to engineering?
What are the highest leverage activities for engineers?
This is a question that I really personally wanted to answer.
I knew that working 60 to 80 hour weeks
simply wasn't sustainable.
It wasn't going to help us win in our different markets.
Also, I'd spend a lot of time on engineering hiring
and recruiting.
And I'd screen thousands of resumes or interview
maybe 500 candidates.
And I really want to know, how do we actually
identify the most effective engineers to hire for our team?
I'd also spent about maybe a year and a half building
the on boarding and training programs
for engineers at Quora.
This was basically what every new engineer at Quora
would go through.
And I wanted to know, how do we actually
train engineers and teach them to be more effective?
And so, I really wanted to know the answer to this question.
And so, that's what started me on a quest
where I quit my job at Quora and then basically took two years
to explore this question.
And I went around Silicon Valley.
I interviewed a bunch of engineering leaders
from different companies.
So, VPs, directors, managers, tech leads,
people from larger companies like Google, Facebook,
LinkedIn, Twitter.
A few were from more medium size companies like Dropbox, Square,
Airbnb, Box, Etsy.
And even smaller startups: Instagram, Reddit, Lyft
at the time was pretty small.
And I grilled them.
I asked them really hard questions.
Asked them, what separates the most effective engineers
you've worked with from everyone else?
What are the most valuable lessons
that you've learned in the past year?
What investments have you made for your team that
have paid the highest returns?
I really wanted to know what from their experiences proved
out to be the highest leverage activities that engineers
everywhere should be focusing on.
And now 22 months later, I basically
had a collection of stories, lessons, and actual strategies
on how to be more effective engineer.
Now, everyone's story was different,
but there were a lot of common themes.
And in this book, this is still a lot
of those common themes through actual strategies
that you and I can apply in our day to day jobs as engineers.
And so, what I'd like to do for the rest of this talk
is actually share with you five of those high leverage
activities for engineers that I think
would be really useful for us to incorporate in our day
to day jobs.
The first high leverage activity really
starts in our own mindset, how we
think about our own development as engineers.
And that activity is optimizing for learning
This Mantra is something that has guided all of my own career
decisions.
Every change I made from company to company or from company
to writing a book was because I saw an opportunity
to actually increase my learning rate.
And this is really important because learning
is something that actually compounds over time.
What you learn today sets you up for other opportunities
and other learning that you might have in the future.
And when I say that it compounds over time,
I actually mean a few things.
One, this curve of learning is actually exponential.
Just like if you were investing in your financial investments,
your investment portfolio, your financial investments
compound over time exponentially.
In a similar way, investments in yourself
also do the same thing.
So, the second implication is that the earlier
on in your career that you invest
in yourself, the more time that learning
has to compound the long run.
And thirdly even small deltas in your learning rate
can have tremendous impact on how much
knowledge and how much learning you have in the long term.
And so, it's really hard actually
to quantify how much we're learning, but suppose
you actually could.
Suppose you could actually improve yourself by 1% per day.
What would that mean?
Well, it would mean by the end of the year, after 365 days,
you'd actually be 37 times better than you
were at the being of the year.
That's a huge difference.
And the best engineers and the best engineering leaders
actually aggressively and relentlessly optimize
in their own learning.
One of the pieces of advice Tamar Bercovici, who
is a senior engineering manager at Box,
tells all of the engineers he manages,
is to own your own story.
What she means by that is that you need to take
control and take ownership of your own learning and growth,
rather than waiting for opportunities to come to you
and find opportunities to really invest in yourself.
Read books, take classes, work on side projects, attend talks.
These are all ways that you can invest in yourself
and then those investments will compound
in your career in the long run.
And so, when you think about growing your career,
ask yourself how might you improve yourself
every single day?
And commit to making a habit out of that.
Now learning is one thing that compounds.
But another thing that also compounds
and that also is very high leverage
is actually investing in your own iteration
speed, how quickly you can get things done.
The faster we can get things done,
the more impact we'll have in the long run.
And because we're all engineers, one
of the best ways that we can invest in our iteration speed
is actually by investing in tools.
When I was talking with Bobby Johnson, who's
the former engineering director at Facebook,
he made the observation to me that he
found that people who are successful, almost all of them
wrote a lot of tools.
He said the strongest engineers and his team
spend probably a third of their time working on tools.
Tools to do monitoring, tools to make debugging easier,
tools to just glue the system together.
But the surprising thing was that he
said this wasn't actually obvious on his team.
Because a lot of engineers, we like
working on the new shiny tool.
We want to be the author of some new system.
We want to build this new data storage system
and be the author of that.
When in reality, even more mundane tasks,
such as investing in tools and investing in your own iteration
speed, can be extremely high leverage and have huge payoffs.
This is sort of the reason why a lot of big companies, Google,
Facebook, Twitter, LinkedIn, Dropbox,
they all have teams the focus on development tools.
Because if you can decrease bill times by, say,
even one minute per day and engineers are building,
say, 10 times per day, and you have 1,000 engineers,
that's a one engineering month saved per day.
I remember when I joined Google back in 2006
and I was compiling in a Google web server on search quality
team.
It was something that you kicked off and then you went home
and it compiled itself overnight.
That's how long it took.
And by the time I left in 2008, sort
of when Blaze was getting introduced, bill times I think
dropped to around 20 minutes.
And that was huge.
I'm sure it's probably dropped down even further since then.
I'm not sure how fast it is now.
I see I see some shakes of the head.
But all of these investments in bill times
are extremely high leverage.
Because it means that you can, instead of doing things
in large batches, you're doing them
more iteratively over time.
One of the proudest accomplishments
that we did while at Quora was we
had a system where we could actually
deploy code production 40 to 50 times per day.
For every time an engineer pushed to commit to get,
it would automatically kick off a suite of a few thousand unit
tests.
And if all those tests passed, that code
would then ship to a canary machine, that would then
run another battery of tests.
And if that passed, the code would be automatically shipped
to all our webmachines production.
And that entire process only took six or seven minutes,
which meant that we could push code production 40, 50 times
a day.
And that changed deployments which,
from any team to this one off event
that they have to do maybe every week, maybe even every day,
to something that was a very normal part of a development.
And it meant if we had a question, how often is
this feature that's on our web page actually being used,
it means that an engineer could just add a simple log line,
push that code to production, and then start getting
some data back within minutes.
So, a lot of these questions that
are impossible with a slower workflow, we
were able to do because of a system
of continuous deployment.
And you might wonder, when does it make sense for us
to invest in tools?
And a good rule of thumb that I got from Raffi Krikorian,
who grew the infrastructure team at Twitter
from about 50 engineers to 450 engineers,
used to tell his team that if you have to do something
more than twice manually, you should build
a tool for the third time.
And that's a good rule of thumb, because we
tend to underestimate how often we're going to need
to manually do certain tasks.
Sometimes we get started, we do things manually
because it seems to be simpler.
And then requirements change, or we mess up
during our manual steps.
And then we end up spending a lot more time
than we actually could have.
Instead, if we had invested in tooling and automation sooner,
we could save ourselves a lot of time.
And so, when you're working on a project,
you should also always ask yourself, what are the events?
What are the bottlenecks that you face during development?
How can you speed those things up?
Because all of those improvements
will sort of compound in how fast
or how quickly you can get things done.
You can also ask same question when you're debugging, as well.
Say you're building an Android app
and you're debugging this photo sharing
flow that's a few clicks away from the home screen.
Could you wire up your flow so that when you start the app,
you land right on that screen?
Simple tweaks like to optimize your own debugging flow
to really optimize your iteration
speed can have a huge difference in the long run in how
productive and effective you are
I've talked a lot about how to get things done quickly.
But another important question to think about
is not only how to get things done quickly,
but how to get the right things done quickly.
And so, that's another key high leverage activity
is actually validating your ideas aggressively
and iteratively.
And a good example of how to do this right as well as how
to do this wrong actually comes from Etsy.
Etsy is a company that sells handmade goods online,
and last year they hit almost $2 billion in revenue.
And they had this one project where
they were trying to build infinite scrolling
on the results page.
So, when you type a query at Etsy,
you see a bunch of product products.
And the product page is paged.
And they were exploring, maybe we
should add infinite scrolling.
Similar to how on Google Images, if you scroll down on the image
search page, results just sort of keep loading onto that page.
And so they'd spent many months building this out,
ironing bugs, and just when they were about to ship,
they decide, we should probably test this.
And so, they ran an AB test.
And they found that click through rates actually dropped
by 10%.
Purchase rates actually dropped by nearly 25%.
So, there was no way they were going to actually launch
this product.
Then they actually spent some time
trying to figure out why this wasn't working.
And they realized that for this product change
to actually work, it sort of relies on two assumptions.
One is that if we show more results to users,
they'll be more likely to buy products.
And the second is that if we show results
faster to the users so that they don't actually have to page,
they would also choose to buy more.
But the key insight is that each of these assumptions
could have been independently validated with much less work.
If the assumption is that if we show more results to users,
they will buy more, they could have just increased
the page size.
And in fact, when they did that after the fact,
they found that that really had no impact on purchases.
An assumption that maybe a faster page
will lead to more purchases, that's something a little bit
harder to test.
But they got a little creative.
They artificially added some latency
to some fraction of users, made the page slower.
And when they tried that test, they
found out that also didn't have much of an impact.
Each of those tests were much easier to run.
Very little engineering effort required.
And if they had run those tests first,
then they would have realized their basic assumptions
behind infinite scrolling didn't even prove out in the wild.
And so, there was no real reason to actually invest all
that time and effort to build infinite scrolling.
They sort of took these lessons to heart
when they worked on a different project, which
was rebuilding the product page when you click through
on a search result.
They actually went through 14 different iterations
of this page when they were redesigning it.
And they broke down the redesign project
into a number of different testable assumptions.
Does showing more related products on the page actually
decrease bounce rate?
Does showing the price in your native currency,
does that increase purchases?
Does showing a default shipping option
and the price of a default shipping
option make a difference?
Does swapping sides of the page make a difference?
They broke down each of these assumptions, each
of these hypotheses, and tested each one.
And there were a bunch of ones that didn't work.
They failed.
But, all of the information that they collected from these 14
different tests helped them build confidence
in theories about which changes did matter,
which ones did make an impact.
And so with that, they were able to launch a redesign
that when I talked to Mark Hedlund, who
was the former VP of product engineering at Etsy,
he said that this was actually the single largest
experimental win in Etsy's history.
It was the most successful project
that they launched in terms of purchasing performance.
And it was only possible because they
learned that experiment driven product design
is a very powerful tool.
Now, this is something that I think,
at least when I was in search quality,
Google did pretty well.
They run a lot of AB tests.
They sort of validate a lot of changes
to prove these changes are actually improving searches
and improving the bottom line.
But another take away from Etsy is
that incrementally validating your assumptions
is a very high leverage technique.
If you can break down a large problem into smaller testable
hypotheses, and then evaluate each one,
you can really build confidence in what works
and what doesn't work.
Or if a test proves that your assumption is correct,
then you have more confidence that the path you're going down
is the right path.
If it proves your assumption is incorrect,
then that means you maybe need to revise your project plan.
At the very least, this could save you months of effort.
And in startups, there is this idea
of focusing on the minimal viable product.
What's the smallest version of the product that
takes the least amount of effort to build that you can
show to real users and get feedback
and validation that what you're building, what you're designing
is actually the right thing to do?
That's a very powerful idea.
Now something, while Google, especially search,
it does a good job with their AB testing,
I think the idea of building this minimal viable product
is an idea that they can definitely leverage more.
Because a lot of times at Google,
we sort of build products for speed.
We optimize for performance, we optimize for scale
rather than really focusing and asking a question.
Like, is this product actually the right
thing to build in the first place?
Because it doesn't matter if something is really performance
if it's not the right thing that users actually want.
One of the good rules of thumb that I got from Zach Brock,
who was an engineer manager at Square,
is that he would constantly ask his engineers, what's
the scariest part of the project that you're working on?
That's the part of the most unknowns, the most risks.
That's the part you should actually tackle first.
Because you want to reduce and tackle the risk
head on so that if it proves that these risky areas are
there aren't doable or don't impact your bottom line,
you can cut your losses short and move on.
And so, when you're working on projects,
you should really ask yourself, can you
break this down into smaller testable hypotheses?
Can you use an inexpensive test to validate
that what you're doing is the right thing?
How might you expend 10% of your effort upfront
to validate that the project you're working on
will actually work?
Because that 10% is a very high leverage use of your time.
It can save you months of wasted effort further on.
Validation definitely is very important
and can save you wasted effort.
But another powerful technique, another high leverage activity
that can help you reduce wasted effort,
is this idea of minimizing operational burden.
A lot of times, we spend so much time maintaining and operating
our software that we don't really have time
to actually build new things.
And if we can minimize the operational burden
that we actually have on a day to day basis,
we can spend a lot more time actually focusing on impact.
And a great story which really illustrate this point
comes from Instagram.
When they were acquired in 2012 by Facebook
for a billion dollars, they actually
only had a team of 13 employees.
And of those 13 employees, only five of them were engineers.
They had a really small team supporting
over 40 million users.
So by any metric, this was a pretty effective team.
And I was curious, what was it that allowed the Instagram
team to be that effective?
And so, I sat down with Mike Krieger, the co-founder
of Instagram, and I asked him, was there
any secret sauce to this?
What was your key technique?
And he said one of the most important mantras that they
had on Instagram was to do the simple thing first.
They actually had this on posters in the office.
During design meetings, when someone
was proposing a new feature, they would actually
challenge each other, is this the simplest thing
that we can do?
If not, why are we adding all this complexity?
The key insight here was that every feature they added,
every system they added, was another potential fire
that they might have to put out in the future.
And with such a small team, they couldn't
afford to spend all that time just putting out fires.
And they were very aggressive and very stringent
about cutting sources of complexity.
And this actually a very common theme.
And I mentioned in the beginning of how I went around and asked
a lot of leaders, what's the most valuable lesson you've
learned the past year?
And it turns out that a lot of engineering leaders,
including Mike Curtis, who is the head of engineering
at Airbnb, or Chris Lambert, who's the CTO of Lyft,
and a bunch of other leaders, all
said that they wished they had made things simpler.
They wish they hadn't added so much complexity
in the last year.
And the reason why this was such a common theme
is because we often ignore the hidden costs that
are associated with complexity.
When you introduce sources of complexity,
we're actually introducing an ongoing tax
that we have to pay as we develop software in the future.
And this tax actually comes in a variety of forms.
On the most basic level, there's code complexity.
That when we're dealing with a very complex code,
it's hard to ramp up on.
It's hard to understand.
It's hard to reason about.
And as engineers, sometimes we see complex code
and then we tiptoe around it.
We decide, we're not going to touch this code base
because it's a little hairy.
And then, we miss out on opportunities to actually build
things that might be impactful.
Or we do things in a roundabout way
just so that we can avoid the complex piece of code.
Besides code complexity, there's also system complexity.
So, how many moving pieces are there actually in your system?
I was teaching a workshop at Pinterest.
It was a five week workshop for engineers
on how to be more effective.
And one of the stories I gathered from them
was that back in 2011 when they were scaling their site,
they actually had six different data storage systems.
They were running Cassandra, MySQL, Membase, Memcached,
MongoDB, and Redis.
And they actually only had a team of three back end
engineers.
That's insane.
There were, like, two systems per engineer.
Each system on paper set claims that they
would solve some scalability problem that they were facing.
But in reality, each system just failed in its own special way.
It was just another fire that they had to set out.
And so, when you make choices that
really sort of fragment our infrastructure,
that has a lot of ongoing taxes that we have to pay.
It means that we're spending-- we're thus
able to pull together resources to really strengthen
libraries and abstractions for a particular system.
It means we have to take time to understand the failure
modes of each system that we introduce.
It means that every new engineer who joins a team
not to ramp up on one additional system.
And so, it took a while, but in the end,
Pinterest finally learned that it's much simpler
to have a system, a scheme, where you can just
add more machines to scale.
And so, they cut out a lot of the other systems
that it didn't really need and ended up
with a much simpler system that they could actually maintain.
So, we've talked about code complexity and system
complexity.
Another such complexity is product complexity.
Now, that comes when there isn't a clear vision for where
the product is going or there isn't enough product focus.
Product complexity leads a lot to code complexity
and simple complexity because it means
you have to write more code and build more systems in order
to maintain and support the features that you're building.
And when you have a wide surface area in your product,
every new feature that you want to add
becomes harder to add because you
think about how does this fit in to the existing context of all
the other features.
There's more code branches you have to look over.
There's more issues and bugs that are filed.
There's more context switching that you have to deal with
to go back and forth from feature to feature.
And all of the time that you spend doing this context
switching is time that you're not
investing in abstractions, [INAUDIBLE]
paying technical debt.
And so, those are all things that end up slowing you down.
[INAUDIBLE] last soft of complexity,
organizational complexity.
Because when you have all of this complexity in your code
and your systems and your product,
it means that you need a larger team to actually deal
and manage with them all.
And that means you have to spend more
time hiring and interviewing.
You spend more time training, onboarding, mentoring
new engineers.
And so, there's another tax on the organization as well.
Now alternatively, some teams decide
OK, let's just split our team to smaller teams
and each team sort of manage a separate component.
Or maybe even have a one person team for each component.
And there's cost for that as well.
It's less motivating to work on your own.
If you get stuck, it can become more demotivating.
The quality can go down because it's harder
to get feedback from your coworkers
if you don't the same shared context.
And your bus factor also goes down.
I remember when I was working at Ooyala,
there was one point in time where
I was the only person who was knowledgeable about this piece
of software called the logs processor.
So, the logs processor was responsible for processing
all of our customer's data and then produce the reports
that they could actually read and sort of understand
what their viewers were doing.
And I was taking a much needed vacation down in Hawaii.
I was hiking around one of the volcanoes there.
And then, I suddenly get a text, and it's from the CTO.
And it says, logs processor is down.
I said, well that sucks.
Unfortunately at that time, I was the only person
who knew how to fix it.
And also unfortunately, there's no Wifi in volcanoes.
And so, that wasn't a great situation.
It was bad for me because my vacation was interrupted.
It was bad for the team because they depended on me.
And it was bad for our customers because they couldn't
get the reports and the data that they needed
to operate their business.
And so, there are all these costs of complexity
on these different levels.
On the level of code, systems, product, organization.
And they all come from this core thing
where we added more complexity than we needed.
And so, that's why this theme of asking ourselves
what's the simplest solution to this problem is so important.
Because it allows us to stem off this complexity
explosion at its bud at the very beginning.
And so, minimizing operational burden is super important.
And the last high leverage activity I like to close on
is actually building a great engineering culture.
I mentioned earlier on how I've spent a lot of time
on engineering hiring and recruiting.
And one of the questions that I'll
ask a lot of the candidates I interview
is, what's one thing you like and one thing you dislike
about your previous company?
And over time, this really helped
paint a picture of what exactly attracts an engineer
to a company.
What are the things that turn them off?
And it should actually come as no surprise
that engineers, they like to work in environments where
they can focus on high leverage activities.
They want to work in environments
where there's a chance to focus on and optimize on learning.
They want to work at places where everyone
is very productive and getting a lot of things done,
and there's good tooling and good iteration speed.
They want to work at places where products aren't going
to waste because assumptions and hypotheses weren't
being tested earlier on.
They want to work at places where they can actually
build new features as opposed to just maintain and pay off
taxes for old existing ones.
Now, Google is a sort of great place
in that there's a very strong sense of engineering culture.
But this is also something that you can focus on your team,
as well.
On a team level, you can really ask,
what are the highest leverage activities
that you can start working on?
You can think about how can you invest in your own learning
and in your own growth to really sort
of further your own career?
You can think about what tools can you build
or what workflows can you improve for both yourself
and for your team to really improve your iteration speed?
Or think about a project that you're working
on that's a multimonth project.
Are there ways that you can break it down
into testable hypotheses that you can validate to make sure
that you're actually building the right thing for customers,
for users, or other members of the team?
Or think about if you're working on something
that seems a little bit too complex,
is there a way to actually make it simpler
and pay off these future taxes on complexity?
Or think about, how can you improve the engineering culture
in your own team so that new engineers who join the team
can ramp up much more quickly?
All of those high leverage activities
will make us more effective engineers.
I'll be happy to do some Q&A. Thank you all for coming.
[APPLAUSE]
I also a few limited copies of books
that we're going to hand out.
May be Cindy or [? Chang ?] can sort of hand them out as people
ask questions.
AUDIENCE: Unit tests.
I kind of read in your book and it kind of
presented unit tests as a high leverage kind of activity.
But at the same time, kind of like intuitively
when I think about writing unit tests,
I feel like it's kind of slow.
I got to write all these to cover all this code.
And if I want to refactor it, I got to change this unit test.
So, iteration speed kind of slows down.
Maybe you could talk a little bit
about exactly why unity tests are really
high leverage in spite of that kind of feeling that I get?
EDMOND LAU: Sure.
Great question, [INAUDIBLE].
So the question is, how do you reason about unit tests
with this framework of leverage?
And really, I think it's important to not be
religious about unit tests.
I think there are some teams that focus on, OK, we
need 100% coverage of our code.
And I don't think that's necessarily the best approach.
It's a lot better think about which types of tests
would be high leverage?
Which types of tests would be easy to write, and yet
pay off high rewards?
And the area of code bases that tend
to benefit from a lot more tests are
code paths that either see a lot of traffic,
code paths that a lot of engineers touch,
or code paths that are a lot riskier.
And so, if something were to go wrong, data would get corrupted
or the cost would be large.
In each of those areas, it can be
really beneficial to actually have a set of unit tests.
On the other hand, if you have a piece of code that's
sort of off to the side, not many people use it,
it's not touched that often, in those cases,
writing a unit test there might not have that strong a payoff.
And so, I would sort of break down unit tests
into different priority buckets and sort of focus on the ones
where there is more traffic, people
are changing it a lot more often, or it's a lot riskier.
Because those will have sort of a better rate of return
on your time.
AUDIENCE: So, it seems to me you have a lot of experience
with working with startups and small companies.
And I wonder actually how does your advice
translate to a bigger organization such as Google?
As you were talking through strategies,
I was thinking in my head about a particular example
that illustrates this strategy.
And for a big company, for example,
for the recent problems, organizational complexity kind
of becomes unavoidable.
So, it seems to me from your presentation,
you give advice for the more bottom
up approach, which Google is a lot like.
But for a big organization, actually
maybe some top down strategies are needed as well.
For example, for using product complexity and organization
complexity.
Do you have any thoughts on this?
EDMOND LAU: Yeah.
So the question, to rephrase a little bit,
is how do these ideas of focusing on complex--
or addressing organizational complexity apply to larger
companies like Google?
And I think it's a great question.
I think some of the clarity comes
from-- some of the strategies that an individual contributor,
an individual engineer where you can sort of influence
the complexity in large organizations
comes I think a lot from data.
So, I know Google is a very data driven company.
And a lot of times, if you can ask the right questions
and back that up with any data from users, data from activity,
you can sort shift organizations or products in a way
to sort of streamline it.
So, if you see a source of complexity from a feature
that you gather data and it seems like it's not actually
being used, I think that sort of forms
a more compelling case for decision makers
that, maybe this is an area of the product that isn't really
worth all the engineering effort that's being spent on it.
And so, I think data is was definitely
one area that can be used to amplify your voice when
you are making arguments of that nature.
I think another way that you can sort of reduce
complexity in these situations is to talk to the decision
makers and get clarity on what the focus
and what the mission is.
And make sure that the projects that you work on
are ones that sort of align with that mission and that focus.
Because if you're working on something that seems peripheral
to that mission or focus, then you're
sort of introducing additional sources of complexity
to that mission, that focus, that the organization has.
So, you want to at least make sure
that your own work is aligned with the goals of the rest
of the organization has.
AUDIENCE: So, I have two questions having
to do with the transition between from engineer
to engineer manager, so maybe they
are within the scope of the book.
One is about mentoring, the other is about technical data.
So, first would be, what are your thoughts on mentoring
in terms of leverage?
And the second is, how do you recommend
convincing an engineering manager
that the code complexity in your project
is worth taking a sprint break from delivering to actually
refactoring something thing from scratch with a much
simpler version?
EDMOND LAU: Good question.
So the two questions.
One is, is mentoring a leverage?
Two is how does refactoring and paying off technical debts
[INAUDIBLE] into this?
So with regards to the first one mentoring,
mentoring is definitely I think extremely high leverage.
I spent a year and a half when I was at Quora building out
the onboarding and mentoring programs
because I realize that even if an engineer would have spend
and one hour a day training a new hire,
that only represents like 1%-- it's like 20 hours.
That's like 1% of the new hire's total hours in that year.
And yet it can have a dramatic output, a dramatic impact
on that engineer's output.
And so, it's super valuable, I think,
to invest money in training your teammates and making sure
that you tell them and teach them
what the best practices are before they spend
a lot of time writing code that's
not following the right guidelines
or following bad designs.
And so, mentoring is super valuable.
And the second question is, how do you
convince engineering managers to actually focus
on or to budget time for technical debt?
And I think some of the best strategies I've seen for teams
is to rather than just to have that conversation
and spend all that energy convincing your engineering
manager after, like say, every project,
to instead sort of just set a sort
of consistent budget for that.
So some teams, for instance, every month
will have one week where it's code cleanup week.
Or after every sprint, they'll have some number of days
where they spend refactoring and cleaning code.
That way, it's-- you don't have to have this sort
of conversation every time after every project.
And in addition, you can use that time and prioritize
which areas of the code actually need to be refactored.
Which areas of code need to be cleaned up.
And so, I think it's more useful to have
a more general conversation where
to hit the goals for the sprints,
we're going to make some trade offs.
Those trade offs will lead to some debt.
And we want to have a consistent schedule where
every so often, we budget some number of days
to basically clean up that debt.
Yeah, it's a little tricky if you're a new engineer.
I would say that even as a new engineer,
the way that you should approach technical debt
is to focus on the areas where reducing that debt
allows you to be more productive.
So, if you're reducing debt on errors of the code
that you have to deal with yourself,
a lot of times doing that making,
that investment actually pays off on your own project
because you can get your project done sooner.
That's sort of, I think, a very easy way
to justify that time spent because you can provably
demonstrate that it actually accelerates your own project
speed.
And so, I would start there.
AUDIENCE: Some of the advice mentioned
tackling unknowns early in the project lifecycle and some
of the other advice was focusing on the simple things.
I wonder, do you have any advice on how
to identify unknowns that could be removed by focusing
on simplicity versus unknowns that
deserve their own investment and investigation?
EDMOND LAU: So, the question is, how do you
differentiate between unknowns that need to be investigated
and unknowns that you can potentially cut by focusing
on a simpler solution?
I don't have a simple answer for that but I
think some of the approaches that you can take
is to think about what are your goals for this certain project?
And if you didn't have to design a grand system, what
would be the simplest solution that you
could think of that could maybe hit most of those goals?
Because a lot of times, the complexity
comes from trying to address the last 10% or the last 5%.
And if you would take a more pragmatic approach,
maybe you can ask instead, what covers
80% percent of the use cases?
Can I come up with a solution for that?
And then maybe for the other cases,
can we do something different rather than making the core
solution more complicated?
And I think that's something that-- that's
a theme that I've definitely seen
other teams do with success.
Because then for the most part, they're
dealing with something that is simple to reason about.
AUDIENCE: So I have a question, a lot
of cases we have dependencies on other teams.
For example, my team, sometimes I go,
there's a problem in the [INAUDIBLE] Bluetooth stack.
Then so I have to make a talk with them or either
I'd have to look into the Bluetooth stack by myself.
Which sometimes can be harder because I don't really
have a lot of expertise on the low level code.
So, can you talk a little your experience
about how do you deal with team dependencies?
How to talk and actually cooperate with other teams?
EDMOND LAU: So the question is, how do you
deal with interteam dependencies where you're sort of blocked
on someone else?
So a lot of times when you're blocked on someone else
and they're not actually working on this thing
that you really care about, that happens because there's
a misalignment in incentives.
Somehow your team has a certain goal.
The other team has a certain goal.
And those goals, maybe our priorities don't really align.
And so, I think structurally, a good approach
is to try to make those goals align.
To see if you can agree that a certain focus is
a priority for both teams this quarter
and sort of agree from that perspective.
If they don't align, I think at least
having that conversation tells you then
where-- the part that you're blocked on,
where that fits on their priority list.
And so, at least having that conversation,
you know that maybe this is only like a medium to low priority
for the other team.
At that point, then you can make this decision of well,
do I wait for them to work on that?
Or do I then-- maybe I ramp up on that Bluetooth stack
to actually make progress.
But I think it really sort of starts
from that communication about goals and incentives.
If they align, then you can sort of rely more on that team.
If they don't align, you do have to make that trade off of,
maybe it's worth my time to learn this Bluetooth stack
so that I can get the thing out the door sooner rather
than waiting for them to get to it on the bottom
of their priority list.
AUDIENCE: So, you mentioned the 60 to 80
hour work weeks not being sustainable.
I was wondering what your typical work week is like?
And if you ever work weekends anymore?
And you also mention learning.
I was wondering how much time per week
do you set aside for learning.
Is it an hour on the weekend that you spend
or is it five minutes a day?
How do you incorporate that into your daily routine?
EDMOND LAU: So nowadays, I still work at a startup.
I work at a startup called Quip.
And we actually work very normal hours,
so we work maybe 40 hour weeks.
I do spend a lot of personal time
on just sort of investing in things like the book
or these talks or other ways I can teach
engineers to be more effective.
Mainly because that's something I'm excited to do.
In terms of how much time I invest in learning,
I personally love to read, so I end up
just reading a lot of books.
And I think books are a really large source and wealth
of information for me.
I don't know how much time I spend reading,
but I [INAUDIBLE] spend a lot of time reading.
MALE SPEAKER: Thank you very much for coming [INAUDIBLE].
[APPAUSE]