Subtitles section Play video Print subtitles MICHIEL BACCHIANI: My name is Michiel Bacchiani, I am part of the Google speech team. So we make that microphone button on your Android phone do something. So more specifically, I am a senior staff research scientist here at Google, and I lead the acoustic modeling group. That's part of the speech team. And so I've been doing speech recognition research for about 20 years. I've been at other places before here. I worked for about three years in Japan in the ATR-- the Advanced Telecommunications Research labs, which is sort of like a Bell Labs of Japan. And I worked at AT&T research for about five and 1/2 years, and I had a bit of a stint at IBM research in upstate New York for about a year and a half. And I've been at Google for almost exactly eight years. And speech recognition is sort of a typical hard applied science. So there's a lot of math a lot of statistical modeling. And as a result of it, we like to process large amounts of data, and so that brings about a large amount of computer science. And so Google looked like a very interesting place to work. So I was intrigued by this place when I was offered the opportunity to do something there. About eight years ago, I looked at it, and it certainly is a very different culture from what I was used to. So when I came from is the typical research lab where there are somewhat of the ivory tower syndrome where you sit together with all the other PhDs and, you know, you're doing exhaustive research, and then there's the development group that's somewhere else, and we don't really talk to each other. And there's a bit of finger pointing going back and forth where the researchers would say, they never take our new idea. And the development group would say that whatever code we get from research is such a mess, we have to all re-implement this. And both of those is true, but it doesn't really work well. And as a result of it, you don't really have an outlet that is built for your product. So you would live off publications, largely, and that's one way to get your stuff out and that would be exciting. But, so now I'm looking at this Google place that is clearly very different. Actually, one of the concerns I had for real is that, are these guys all sitting on colorful skippy balls high fiving each other and saying awesome all day long? Which is not really something I enjoy, because I did enjoy the hard science of what I was doing. Also in retrospect, what I think is very different from Google to do other places, I think, is sort of the three aspects, I think. There's a people component to it, which is we are extremely collaborative between engineering and research. In fact, we make hardly any distinction, where it's hard to distinguish whether somebody is a software engineer or a researcher. And it's just because, if you want to get your stuff out, it has to be engineered. And if you're an engineer and you engineer something, you might run into a problem and you have to research on how to solve that correctly. And so that's a nice environment where people are collaborative and there's no hang ups about what department you're in or what your title is exactly. and you really jointly get something out. I think the second aspect of it is a product. Your stuff really gets used by a lot of people. There's a real path to get whatever you code to get in the hands of people, and actually, people use it. And not only is it sort of a satisfaction that somebody will use your code. There's some interesting stuff that it brings up for research. I mean, it gives you signals to look at on how people are using things, whether it's working for them or not. And you can really focus on a user and sort of make their experience better, which is an interesting research setting in itself. Sort of, how do you make use of those signals to improve your product? And the third thing is sort of impact, because you know, you really can improve people's lives. We did some, you know, I work in the speech teams. Working with some deaf people and giving them opportunities they never had before is exciting. You really, I think, have a more profound impact on somebody's life. And I think a lot of Googlers have this mean altruistic streak where they really like the fact that their stuff can do something good for somebody. And that brings a lot of excitement about. So that might sound a little naive, and naive can be bad, because I have colleagues that look like they're 13-year olds. And I know they're not, but they look like that to me. And they think that the normal world is whatever our Google environment is, and even if you tell them that's not true, you can't really convince them. And we have this rich food culture, as you might have noticed. I literally heard a complaint to somebody that said, do they have to really put truffle oil and everything? There's somewhat of a bubble here, that's true. But in general, I think it's a pretty happy place to work. So more specifically to speech, I think that it is really this kind of company in this mindset that it's not surprising that it took speech, I think, from when I joined eight years ago from sort of user torture. You know, where it's like, say or press one for sales, or say or press two for, you know. To now, we're sitting at home and looking at an ad, and people are like, I like this cellphone because it has voice recognition, as a feature that they actually want. So again, sort of this user focus, I think, is a positive thing. And I think, in more general, it's not just the speech thing where it's been lucky that they had some success. Information retrieval, Google search has been applied and really is useful to people, I think machine translation, I think there are many, many more areas to come. So don't just take it from me. I think we have a very interesting panel with various other people in various other areas that you can quiz and we can discuss things. Some. Of the things I think that we will talk about a little before we go in our Q&A session is to discuss this whole collaboration, which I think is contrasting some of our relationships internally, as well as with the research community outside. So let's start with introductions, I guess, of the panel. So how about we go from left to right. So I'll let you guys do it. UMESH SHANKAR: My name's Umesh Shankar. I've been at Google for about seven years, previous to which I did my Ph.D. at UC Berkeley. I entered Berkeley as a systems student and fairly quickly ended up focusing on security and privacy, which is what I still do here at Google. I'm a software engineer. I lead the data protection engineering group, so we spend pretty much all our days figuring out how to make sure that all the private information that Google stores stays that way. And as far as I'm concerned, Google is really the most interesting place to work on security and privacy, because we don't really have that many theoretical attacks. Most of the stuff that most people think of as theoretical is real. And there's really no playbook for how to do data protection at the scale that we want to do it at with the performance and usability requirements that we have. I mean, it's just been an incredible learning experience for me here. And to a degree that I never really thought I would, I've used almost everything I learned in graduate school. EUGENE WEINSTEIN: Hi, I'm Eugene Weinstein, and I think I'm one of the 13-year-old looking people that Michiel was-- MICHIEL BACCHIANI: You wish. EUGENE WEINSTEIN: Well, I don't know, you might disagree, but in the sense that I've never really worked-- well, some small internships aside, I've never really worked at another company other than Google. So I was lucky enough to be able to do kind of the applied part of my Ph.D. Here as an intern while I was a Ph.D. Student at NYU, and I joined here full time about 3 and 1/2 years ago, and I work on speech with Michiel. So I work a lot on the internationalization side of speech. So at Google, internationalization is a big thing. We can't just get away with making our products work in English anymore. But for speech, this is like a whole new ball game. Because it's not just, like, translating some strings, right? We have to model new languages, right? So this is a difficult thing. So I kind of enjoyed being able to scale things up in that respect. Now, voice search supports 40 languages, we just launched four new ones a couple of weeks ago. So that's been so that's been really fun to work on. And the other scale issue that I've gotten to work on that I think has been really meaningful to me is the work that I've done with keeping speech models fresh. The things that people are talking about and the ways in which they say them are changing, right? As demographics change, things come into and out of popularity, and people change the way they search for things with their voice. And we have to take the petabytes of data that's coming in from users of our products and we have to make sure that we're servicing the user in a way that's still current and relevant. And so to me, it's been really fun to take the, let's say, theoretical things that I learned about in grad school and apply them at this ridiculous scale that I could never even conceive before I came here. So it's been really fun. OKSANA YAKHNENKO: Can you hear me? I can't hear myself. So, my name is Oksana Yakhnenko. I've been at Google for a little bit over two years. I joined in like December, 2010. I have my Ph.D. In computer science and I was mostly working on machine learning and applications to various things like text classification, computational biology, and computer vision. And then I've been [INAUDIBLE] in computer vision for about a year and a half. And one of the reasons, when I got on the job market, I was a little bit disillusioned with academia, in terms of, I publish a paper and you spend so much time writing this paper and you so much work in the paper. And then you have like four citations and that's kind of the end of it. I mean, sometimes you have more than that, but I actually felt like my work was affecting a very small niche of people. So basically, I started looking for a place where I could actually do some more impact. And now I have joined a group in Knowledge, so we work in search. And specifically, my group, we don't really do anything that you guys see directly, but we work sort of behind the scenes, and we work on information extraction. So there's this effort called Knowledge Graph, and this is basically the graph of related entities that Knowledge Panels uses to answer the questions and the queries that you guys type in the search engine. And basically, our project is to automatically populate this graph with a lot of unknown entities and unknown relationships. So things that we do, we do a lot of experimentations with like, hey, here I have this model. How good is this model? Not only on paper, that like, this model is going to give me 95% accuracy, which is like, OK, cool. But like, how many users is it going to impact, right? And it's like, even if my model says that it's 95% accurate, do I know that it's 95% accurate? Because not everything that's on the web is true, right? So that was basically some of the more challenging things that I've been working on, things that I enjoyed doing here. GIDEON MANN: Thank you. And thank you, Michiel. I don't think I've heard such an eloquent description of what it's like to work at Google. And I think I would echo a lot of the same things. Especially, Michiel talked about a mean altruistic streak. So I think that, if I have an affliction, or one of the afflictions I might have would be that, and so I've always been looking for opportunities at Google to use the work that I've done to have some kind of a broader impact. And one of the nice things about this place is that you have a platform to do that, and the opportunity to do that, and the support to do that. So in particular, I have worked on machine learning. My background's in natural language processing, but I'm moving a little bit more into machine learning. And in particular, I've been looking at how to make machine learning something that everybody can use, and something that is easy to use, and something that you can drop into your web service, and something that you can training easily. And so, it's been really exciting to pursue that here, because there's so many opportunities to get feedback on whether you're doing the right thing and going in the right direction. So I'll cut myself off there and turn it over, but I'd love to talk about that more with you guys. OMAR BENJELLOUN: So my name is Omar Benjelloun. Can you hear me? So yes, I've been at Google for about seven years. Before that, I did a Ph.D. In databases and I spent two years doing up [INAUDIBLE] also, working on databases and uncertain databases and reference reconciliation problems. And I also joined Google for similar reasons to what was discussed earlier, like having an impact and doing things that are really useful to more than the community of researchers that would maybe read my papers. So at Google, I've been also working on projects related to structured data. And so, I just want to mention two projects that I worked on. One is about, so I lead a project about public data and statistics. And the goal of that project was to and continues to be to get official data about, important statistics about, the world, countries. Your city gets that official data directly from the sources of the data, which are international or national organizations, or local organizations. There are lots of reconciliation data integration problems to solve in order to do that, and provide also means for any provider to give the data themselves and using an easy to understand format. And then reconsolidate all that data in a place where we can make it useful to users through visual tools to explore the data. Visualizations, ways to drill down, compare, and so on in a tool called Public Data Explore. And more recently, we've been surfacing that data directly in Google search. There's a statistics feature that we just launched about three months ago where you can, if you search for a population of India, you'll see graph that's interactive, and you can move your mouse and see the value over time. But we also try to give you context by figuring out what are the most relevant other countries to compare this data to in order to help you understand the content and help you explore. And currently, I'm working on another project in parallel, which is about, again, structured data, but figuring out what is important, how to rank the information that we show you. So if you search for, say, Brad Pitt, we show you a knowledge panel about this actor, and we'll select a few facts that are important about him. So how to select the right set of information and rank that information is this different problem than search ranking, because it's the structured data world. It's a very sort of both research-y and challenging problem, but it has direct applications and direct impact on our users. And so, that's what I find exciting about our work. MICHIEL BACCHIANI: All right, so you have some idea, I think, who is on our panel. So I would like to bring up some of the topics that I brought up, and you can chime in whether you think this is appropriate or not. One of the things is always the perpetual question about researcher versus engineering, and how it's organized, and why they can never get a straight answer out of any of us, whether they're a software engineer or a researcher. So anybody wants to chime in on that? GIDEON MANN: I'll take the bait. So if you can't get a straight answer, I think that's almost by design. I think that at the company, there are really a tremendous number PhDs, and the PhDs are in every part of the organization. I took a training today from someone who had a psychology PhD. And she was in the people [INAUDIBLE]. And I was surprised, though I probably shouldn't have been. Because that's really the nature of this organization. It's an amazing place in terms of the level of education and intelligent [INAUDIBLE] and everybody here. So as far as the distinction between these different roles, I don't think there is much, except it's more determined by your local contacts, who your local manager is, and your local group is. Alfred is one of the people that's been thinking about research a lot at Google. Likes to talk about that research being done opportunistically. [INAUDIBLE] I think that that's probably about right. MICHIEL BACCHIANI: I think you want to use the microphone so that we can hear you. when OKSANA YAKHNENKO: So we have to do annual reviews for yourself and each other in the company. So do the publications come into play for research scientists when you guys do your work? MICHIEL BACCHIANI: It's an important issue, I guess. OKSANA YAKHNENKO: Take the blue pill. OK. So when you do your performance evaluation, do the publications-- how much publications actually weight for research scientist? GIDEON MANN: So this is, again, my biased assessment, but I think Google as a company, really judges you on impact. And impact is a lot of different things. Publications, launches, standards, talks, contacts that you develop. So I think that it really, again, is you want to self define your impact and how much, according to the way that you've defined your impact, you've achieved it. Eugene, you feel like you want to add something? EUGENE WEINSTEIN: I definitely see it as, let's say you come to Google and you just do engineering projects, right? And don't write any papers and don't work on what people outside would view as quote unquote "research," right? That's OK. It's an engineering organization, right? But let's say you come here and you expect that you'll be able to just, like, work on papers and do theoretical stuff and not take part in contributing to any products, like some of these more traditional research organizations that Michiel alluded to, that's not really OK. So I think that it can definitely be a boost for researchers when it comes to performance, when it comes to evaluating your impact. But that can't be the totality of the contribution that you bring to the company. MICHIEL BACCHIANI: So can any of you sort of comment on projects that you're doing which you think are essential to have this integration between engineering and research? That you don't think could ever get done in the sort of traditional model, where it's completely distinct silos doing either research or engineering? Or where it would be a much longer process or much more involved to actually get that off the ground? You all work in different areas, so I guess there should be a wealth of examples of things you get out there that you think are good examples of mixing the two. UMESH SHANKAR: This those kind of works if you hold it really, really close to your face. OK. Cool. So maybe my group isn't the best example of this sort of inextricable link, but we're definitely forced to confront kind of the edges of standard practice very, very often. So in cryptography, one of sort of the first things that you learn is, don't make up our own, right? It's like very, very hard to get right and wherever possible, you should use a standard thing. But once in awhile, we come across situations where the standard thing doesn't quite have the properties that we need, and it's times like that I'm really, really glad that we have absolutely world class cryptographers who work at Google, that we talked to and say, OK, what are our options here? And sometimes, they'll actually come up with something different. In other cases they'll say, well, there's this new encryption mode that's sort of experimental. And we believe it has these very specific weaknesses and it either is or isn't appropriate to the particular thing that you're doing. Absent that expertise, I just don't see how we could possibly make a good decision of what to do. Conversely, because you're confronted with such big challenges, and because the culture of the company is one that really encourages people to take big bets, that already tends to blur the line between research and engineering. Because I think a lot of people's mind, engineering is something where, it's sort of really clear what you have to do and you just kind of do it on a quarter by quarter basis. And research is this thing where you really don't know what you are supposed to do, and you get a really long time to do it. In reality, I don't think either of those is really true. I certainly know as a graduate student, like, I had to publish stuff every year to kind of keep my name out there and do a good job, right? and I think for new faculty, it's probably much, much worse before they have tenure. Conversely, here, as I was saying before, there really isn't a playbook. It's very rare that we know in advance what is the right thing to do. And a big part of the challenge is figuring out that thing, trying really bold and innovative things, and finding out what the right thing is. Whether it's published at the end or not, I mean, that sounds like research to me, right? The difference as I was saying to some folks earlier is that it has to work. You don't get to make assumptions that aren't true. That's a really good forcing function, right? And then I certainly know that in my previous time in academia, like, yeah. A lot of people would publish stuff and at a conference, someone would ask a question like, but is that really true? And really, the person brushes it off, and that's kind of the end of it, right? And the difference is that here, you can't do that, because it has to work. And I think that constraint of it having to work is incredibly positive for innovation. It forces you to kind of stop doing things that don't work and try new avenues, and I think the end result is much, much better as a result of it. And I think you're saying also about how you want your impact to be real. And the net result is, that's what everyone wants here. We want to make a real difference, and that's part of that same function. I personally find it extremely satisfied to do that. OMAR BENJELLOUN: Just one last comment. I think one difference that I see between doing research or just innovative work at Google versus in academia is that the constraints that we have here are the real ones. They're the real, most challenging constraints that, if we solve them, then we have the potential to have an impact on-- to solve the problem for real. And I find that in academia, sometimes, at least in my experience, we tend to solve, sometimes, problems within constraints that sometimes can be artificial, or we try to define a set of constraints that make the problem tractable, without knowing if these are the real, most realistic constraints. While here, we have users and they use our services. And if we don't come up with something that works, then it doesn't work. So it has to work GIDEON MANN: Here, I'll give a concrete example of an area. So, distributed optimization. When you have a particular optimization problem with very many variables, larger than it's going to fit on one machine, you want to solve it very quickly. It's important to the business. How do you do that efficiently? How do you do that with low loss? People have been looking at this problem for decades. In the early '80s, there was work out of MIT for tickets to [INAUDIBLE] looked at this problem. But the networks have changed and the problems have changed. And so, there are a lot of people looking at this problem or variants. One of the key parts of the Google Brain project, if you read the paper carefully, is they had a very clever implementation of a large scale distributed optimizer. I mean, as Omar and Umesh have talked about, you're not forced to look at this problem, you're not forced to solve this problem in every context. Here, you're really forced to. You really need both a theoretical perspective and applied perspective to solve it. And then the end result is something pretty spectacular. EUGENE WEINSTEIN: So, we talked a bit about how the constraints are real, right, of what you have to-- oh sure we talked a lot about that. That's an area where we're constrained, but I think it's also worthwhile to bring up the areas where the constraints, the-- how shall I say it, the leeway that we have seems almost so real, which is in the computation infrastructure that we have access to hear Google. Sometimes, you have to solve a research problem just to get the thing to parallelize at the scale that you're allowed to run it at. The fact that we have to answer to Google's users introduces the constraint in that sense that we can't do stuff that's wrong, right? But the situation with the infrastructure that we have here at Google, it makes things that much more, I think, appealing to a computer scientist from just coolness standpoint and from the standpoint of how much you can get done. Sort of like the amount of leverage you can sort of bring to the effort that you are able to make as an individual by using this infrastructure. OKSANA YAKHNENKO: So I think this is to Omar's comment on constraints. So in my perspective, it's almost like your definition of research changes a little bit. So in academia, you have, like, either a set of known problems that you're working on, or you try to make up a problem, and you want to convince everybody else that this is important. Whereas research here is finding-- sometimes actually the problems jump you immediately. Because you have an algorithm that works amazingly well on paper and it works well on, like, a million examples, but then if you try to round it over the entire web, it just doesn't work. So how do you scale it up? So this is something that Eugene brought up. And then there's another way of looking at it is, like, how do I find a problem that I want to solve that's worth pursuing, right? And even answering a question like that is a research in itself. So it's like, what are users want, right? And if I want to get from A to B, what are the resources that I need? What are the kinds of people that I need to talk to? How do I evaluate the results that I have? How do I scale it up? How do I put it into production? Things like that. I think that's a lot cooler and has a lot more-- you learn a lot more, I think, by actually doing research that way than just, like, hey, here's a cool problem. Let me try to come up with another kernel for a support [INAUDIBLE] machine. MICHIEL BACCHIANI: So there's been a fair amount, I think, of academia bashing. But I think all of us have a warm and fuzzy feeling when it comes to academia, in the sense that we engage with the community quite a bit. And I think in various ways, we try to be supportive. So in contrast to all the telling them how they do everything wrong, maybe we should tell them a bit sort of, how we are in favor of them, and try to play nice with them. UMESH SHANKAR: We love you guys. No, seriously. OKSANA YAKHNENKO: Are there people who are in academia? Can you raise your hands please? OK, so we're not offending anybody. MICHIEL BACCHIANI: So I would like to bring up some other things we do, in terms of, you know, we do publish quite a bit. I think it's encouraged, which I think is a bit of a misnomer that we don't care. I think we do give a lot of monetary funds to people that are in academia. We have a giant program, in terms of visiting researchers. I think we try to be collaborative outside as well as inside, and maybe you could share some of you experience with that. GIDEON MANN: So I think, in addition to everything Michiel said, one of the interactions that I personally have enjoyed most is the intern program. Because it's a way not only of getting kind of fresh blood and fresh ideas, but it's also a way of strengthening connections to the academic community. So friends of mine that have gone into academia will send interns my way. And this is a way of kind of keeping those connections strong and also transmitting knowledge back and forth. So I'm thinking of a friend of mine, Noah Smith, who works at CMU/LTI, and he is like super, super into these advanced latent variable models and graphical models, and it's great, actually, to have his-- he has a student here, now. It's great to have his student here and get that learning here. And then I think also, it changes his student's perspective about how to run experiments and what experiments might be like. To say that, you know, can simply run 1,000 iterations or 1,000 different models at once, and what does that mean about the search space? What does that mean about the techniques that you might try and you might want to investigate? And so, I'm hoping that it really-- and I'm expecting that it's not going to be a one way street. It really is kind of like an exchange. I had a student from another friend of mine who was working on systems related problems. And I think when he came here, he saw that the depth of kinds of things was very different than the problems he had looked at. And so, it was also a lot of insight that he could bring back. And so that kind of relationship I find especially valuable. And that intern program is vast So among our panel, who has current interns in their team? So I think that gives you a sense. I don't think that's uncommon. UMESH SHANKAR: Just to kind of add to that, as I was saying before, and it sounds kind of funny but its actually also true. You know, as a proof that academic research really is valuable is that I really did end up using so much of what I learned in operating systems, databases, networking, of course security and privacy, all these areas. The concepts that were developed in academia, things that kind of maybe seem far fetched at the time, that only made sense to use an algorithm at an extremely large scale. Well, that time arrive. We're very grateful that kind of work had been done. And I think I wholeheartedly agree about the kind of interchange between a place like Google and academia. I mean, we go to conferences. That's a great example of that, where, not just Google, right, but many other companies in the industry have people who actively work on research topics, who publish. You go to the conferences, you learn from each other, right? And in some sense, the conferences, like, shows you how many similarities there are in the kinds of problems that people work. There is also something that, the flip side of my comment earlier that the things you do have to work, is that it takes time to keep them working. And there's a certain tax that you pay, often substantial, for that. And in academia, it is a little bit freer of those sorts of constraints, and there's something very nice about that too. It lets you go in perhaps more different directions than you would otherwise. If I were to draw another sort of, like, compare and contrast between academia and a place like Google is that I think Google actually really rewards taking long term big bets in maybe a smaller number of areas as an individual, which can have a really big payoff. Of course, the flip side is, maybe it doesn't work out and now you're like, man, I just a couple of years on this thing and it didn't end up really working out. Academia, I think, in general, it's quite common for people to work on more different projects at once, and kind of, OK, the idea captures your fancy, you can kind of start working on it, even more so than you could at a place like Google. And I think having both those models out there probably is a really good thing, in total, for when you look at what we, together, can produce. MICHIEL BACCHIANI: Gideon disagrees GIDEON MANN: You know, I'm not sure I disagree, but I don't think I've ever thought about that way. It's really interesting. I think that you're right. I mean, there's like a historical baggage you start to accumulate of, this is what the world, meaning Google, ecosystem looks like, and that's what you have to operate in. I don't think I've ever thought about it that way. Interesting MICHIEL BACCHIANI: Sometimes with publishing, this is something that I think I frequently hear, is that we are not allowed to published. There's, I think a rumor that I've-- not universally, but that's something I've heard frequently that-- or it's discouraged, or whatever. And I think it's very unfair, but again, this is one person saying it. But you guys want to comment on whether you think this is a fair statement, or where does this perception come from, if you had to guess? OKSANA YAKHNENKO: I think the perception comes from that a lot of people who actually go to Google sort of, like, disappear for a couple of years, and they don't publish. The reason they don't publish, I think, is actually twofold. Not enough time and not enough time to write a paper. Because you get so interested in what it is that you're doing and you become so involved in actually getting the thing to work and seeing the impact of this thing. Sometimes, you just don't really care to write it up, or sometimes you just don't really have enough time to write it up. With that said, when I joined I still had a couple of leftover projects for my grad-work, for my graduate work. And I took like two weeks or three weeks that I was allowed to use Google's resources, and like even like Google [INAUDIBLE] actually wrap it up and submit a publication that got accepted. They sent me to the conference. And this was, like, for the work that I've done that was still the ideas that where the academic ideas when I was in school. I'm an engineer, but I've done a couple of collaborations with people in research where it's sort of like a two way street, like where I tell them the types of problems that we're really interested in, and I simply just don't really have the time to look into it and where somebody from research actually was like, oh yeah, cool, yeah. This is like my area of expertise. I have an algorithm that could be well suitable for this. I can run some experiments and see what will come will come out of this. And then basically from something like this, a paper evolved, and we actually submitted a paper to conference. In terms of submitting a paper it's like super easy. This person worked for Corina Cortez, and it was basically like telling Corina about the paper, and she was like, yep, cool, you can submit. Like, this is the level of approval that it had to go through, pretty much. EUGENE WEINSTEIN: I think with respect to publishing, the things that we're saying here, you guys don't have to take our word for it. Just go to research.google.com, right, and you'll see that there are hundreds of papers coming out of Google. These aren't only people that are in the research organization. So I think, for most people here, I think it's pretty much a personal choice about whether you care, like Oksana said, whether you're motivated by getting your work out there in front of people outside of Google, or whether you want to focus more internally about presenting your work to your colleagues and getting buy-in from them and getting collaboration opportunities there. It's very much a personal choice, but if that's really your passion, I would say that the vast majority of the people are able to publish, and publishing is encouraged by managers, by VPs. MICHIEL BACCHIANI: We have a question. AUDIENCE: So I wonder, talk a little bit about the origin of problems. Obviously Google has lots of products that do things, and clearly problems that are directly related to those things are important and valuable to the company. Is it possible to have an idea that maybe doesn't have such an obvious connection to something for the company, and to what extent would you be able to actually run with that? A month? A year? Two years? Whatever. Does that make sense? EUGENE WEINSTEIN: So I don't know if you're familiar with the Chauffeur project, the self-driving car out in Mountain View, so that's one example of something that's totally not connected to the mission of the company, as most people would see it. So there are things like that. Like, would I be able to, right now, go to my manager and say, can I go and spend 50% of my time building a self driving car? Maybe not. That's not always the case, but there are these things that happen in that kind of organic way and that aren't always directly related to what your job role is. MICHIEL BACCHIANI: So it seems to be the perfect time to bring up the 20% project, which isn't quite 50%, but, you know. In contrast to what you said about the freedom to do off-the-wall things, I think there is some provision that provides a lot of people to do so. I think a lot of us, because we're so ingrained what we do and like what we do, we don't really make use of it. But it's a reality that exists, I think. Maybe can you guys comment on whether you have people doing 20% things? OMAR BENJELLOUN: So I considered as a manager, I've had many of the people on my team who are interested in doing 20% projects, and I've always encouraged them to do that, especially if it's something that's going to help them learn a new skill or advance in their career or establish new connections or find new, potentially interesting areas to develop. And it's generally been a very positive experience. Some people do it for a long time, some people do it short period of time, and then they stop doing it and then they come back to it. It varies. It varies a lot. One other comment I wanted to make is, also, in the projects that we work on, there's sort of a mix of projects. So I think that many product areas at Google have some established sort of short term projects that they know they want to get done and launch quickly, but they also have some sort of fraction of projects that are more sort of exploratory and are trying to do explore sort of longer term opportunities, and those can be more researchy. And it's always possible to go and propose an idea and say, hey, I think there's something really interesting here, and that's an interesting challenge, and if we solve this problem, this is going to have a big impact on what we're doing. And if you convince your director, your VP, then you might even get some headcount and engineers that can work on this project. GIDEON MANN: So Tom, to address your question, I think one thing that I didn't realize about this place, before I started is the autonomy and self direction that each individual engineer has. They have it in the choice of project, but they also have it in their interest of time. And so, you are so valuable as an engineer to the company that you are actually a prized resource, and the company's goal to some degree is to keep you as excited as possible. And so, to some degree, that means that you can kind of move and your place of interest. Now, I think what people usually find is that after a certain point there, they start thinking about aligning themselves to the company's larger interest. Now how can I support this larger area that the company is going in? It means that your projects will move faster, it means that you'll move faster, it means you'll get more support, more other people will be interested in joining your torch-bearing cause. But it really is kind of your own discretion. I don't think I understood how much that would be true about being here, but it certainly is. UMESH SHANKAR: Just to kind of follow up on that very briefly, I think like the usual process by which this kind of thing happens is, you start small, like you build a prototype. And the real process by which things get kind of sorted out and what ends up making it and what doesn't is, like, there's a marketplace of people's time because people's time is really valuable. And if your idea is really good, and you convince people to work on it, then your project will grow. And if you can convince people that your idea's good, maybe it's not that good. Because maybe it is and we're missing an opportunity, but I think, on the whole, the system works pretty well. And it works very organically, that means that you have to probably invest more time in learn skills of persuasion and things that you might otherwise to get it off the ground. That's not necessarily a bad thing, either. So yeah, small things can become big. If anything, I think maybe one mistake that people sometimes make in trying to start new projects is thinking too small. It's hard to get people to want to sign up to something which, even if it all works out great, still ends up being small. That's just not that exciting. So having a big vision for even something that starts out small really ends up being key. And that's really, I think, a lot of Google's all about. GIDEON MANN: I just want to piggyback on that I totally agree. And I think one feedback I get from-- I met with Jennifer Rexburg, who's a prof at Princeton, and she was like, it's amazing, I meet all these engineers at Google and they're all so able to communicate. And I think, partly, that's not the interview process. I think, partly, it's the work process. You're forced in kind of your daily process to communicate with other people. That is kind of what will propel you and communicate about your ideas. And so, just as Umesh was saying, it's like, that communication, persuasion, building the vision, that is kind of the part that will drive you on your project. So agenda-wise, I think we're getting close to starting the Q&A session. I've noticed that we have more and more questions, so I would like to encourage it even more. AUDIENCE: This is a segue [INAUDIBLE]. I understand that you are all PhDs in computer science [INAUDIBLE] as far as I can tell, seem to work on areas within Google that follow from graduate research context in a specific area for your dissertations. And so I was wondering, in part because I am a math PhD, not a computer science PhD, about the adjustment process that some people have [INAUDIBLE] Google, as friends of mine have PhDs in biophysics and then they start to work on video chats. One example that is true. Because I think it's kind of an adjustment process to be a database researcher in academia, and then a database researcher at Google, and it's another kind adjustment process to be a mathematician and then to work on [INAUDIBLE]. What kinds of [INAUDIBLE]? Just the kinds of resources and learning curve that people face in that? OKSANA YAKHNENKO: So, most likely, if you are, what was your example again? If you're a mathematician, well, if you're a mathematician, most likely, you will not be hired as a front end developer. Mostly because, as far as I know, and maybe actually, people who work in the HR maybe will be able to comment a little bit more on that, but that is as far as I know, they try to put you on a project where you really can contribute. Because, yeah. STU: May I try answering that one? I'm not on the panel, I apologize. [INAUDIBLE] We have, [INAUDIBLE] one of the earlier comments. Last time I asked, we had some 3000 people [INAUDIBLE], some of them are none of the above. [INAUDIBLE]. A whole bunch of psychology. [INAUDIBLE] a few medical doctors, Juris doctors, [INAUDIBLE]. But we have over 2500 in the computing cluster, which I will define as math, science, and engineering. And then we have another 300 or 400 in biophysics, [INAUDIBLE] which is as far out as you can get. And [INAUDIBLE] hiring outside research. Yes, we do take a look at skills. If you're going to come in as an engineer, you'd probably [INAUDIBLE] real programmer. And then we actually do a sorting hat with a serious effort to find a plausible place for what you know. We're assuming that you're a Ph.D. [INAUDIBLE]. And yes, of course there are courses on this, that, and the other. [INAUDIBLE]. And your colleagues will contribute [INAUDIBLE] but the goal is we can be doing something useful about a month after you show up [INAUDIBLE] in courses and learning [INAUDIBLE] how to make the elevators work. But after that, we expect you'll be doing something kind of useful in a month, and be valuably up to speed in [INAUDIBLE]. And by useful, I mean a team [INAUDIBLE] depending on you to actually do something. [INAUDIBLE]. MICHIEL BACCHIANI: More questions. AUDIENCE: The problem between academic research and corporate research is that every very smart idea you guys have [INAUDIBLE] is the competitive advantage. So How does that conflict with publishing? GIDEON MANN: I'll take that one. I had in mind to directly address this when Michiel was talking about publishing. I think that it is very clear that the fruits of all this labor is material advantage. And that it is valuable work, it's important work, and that it makes a difference at the company. And so I think that everything that gets published, people think about, OK, well, how much are we actually giving away? And I would say the things that are, in particular, I would say Urs is probably the gatekeeper. And if he feels like we're giving too much away, he'll say, let's maybe either delay it until the competitors have got in close enough, or let's hide some of the most key pieces, not so that it's useless, but just so that we don't give away the bank. And so, I think that, yes, clearly it has value, and clearly we don't want to give away. And so I think that. But in my experience, to give an example, we published a paper on distributed optimization. And they didn't want us to directly say the number of cycles, or the amount of time things had taken up on that racks or on the machines. So we converted it to some arbitrary scale, and that was enough to satisfy people that we weren't giving details away. So I think that there's always room for negotiation, even within that space. AUDIENCE: [INAUDIBLE]. GIDEON MANN: So, my impression is that Google's general position on patents is that they wish there weren't. They didn't exist. Google is involved in so much patent litigation that I think it would be beneficial to the company if the whole patent system was more evolved. Maybe Stu has a slightly different perspective. STU: True, but it ain't going away. So yes. Somebody will look to the publication [INAUDIBLE] might have been patented and submitted, in which case, some [INAUDIBLE]. We would be delighted to disarm, we just can't. GIDEON MANN: But I think to-- STU: Have that conversation. GIDEON MANN: But, to that point, I think that it's not-- STU: It rarely delays you more than a couple weeks to actually get the lawyers [INAUDIBLE] and [INAUDIBLE] probably one paper in 100 [INAUDIBLE]. GIDEON MANN: Oh, but I think part of the question was, how important to your career is patent filing, and I think it's-- STU: Very little. GIDEON MANN: Yes, MICHIEL BACCHIANI: There's a question over here. AUDIENCE: Obviously, [INAUDIBLE] products here [INAUDIBLE]. So, excuse me, so [INAUDIBLE]. Because people somehow appear to be [INAUDIBLE]. EUGENE WEINSTEIN: Yes definitely. So you mentioned 3,000, right? So Google is, what, 50,000 now? STU: Has 2500 PhD's, and nobody ever uses the title unless they're a psychologist or a lawyer or a medical doctor. So unless you actually ask, you probably won't know [INAUDIBLE] to get a PhD [INAUDIBLE] actually pretty damn good. AUDIENCE: [INAUDIBLE]. MICHIEL BACCHIANI: So I think it's not so much that it'll give you an edge, let's say, in title or in how much people respect your value, your opinion. It won't get you that. It's more the ability that you'll have to bring that research kind of flavor to the work that you do. And I think in turn, because you will be able to show that kind of skill, in turn, that will get you the respect and buy in from your colleagues. It won't be the fact that you have the Ph.D. There was also the adjustment aspect of the questions OKSANA YAKHNENKO: I think one thing that you'll learn as a Ph.D. Student, and it's mostly because you spend, like, five years working on one problem, is you really learn how to problem solve. So it's not necessarily the title that gives you the edge, but it's the fact that you, in the process of getting the PhD, you learn how to solve problems. So think about it. If you come to Google, you get thrown into an environment where most likely, you will be working on something that is or is not related to your Ph.D. Or to your research project directly, you still know how to approach a problem, how to solve it, how to evaluate your results. And I think that's like a great skill to have, for anybody. And I think most of people who actually work at Google have that skill already. So in terms of adjustment, I don't know. GIDEON MANN: I'll speak to adjustment. I was a post-doc, and I think my biggest adjustment coming here was that Google is not very hierarchical. And so, it's like when you're a graduate student or a post-doc, you have your adviser, and your adviser has a final say, to some degree. I mean, about what you do. It's really not like that here. It's a very big company, there are a lot of very different opinions, and as an individual engineer, you have a huge amount of autonomy and a huge of ability to persuade that is kind of separate from your manager and from your team. So one of the ways that the engineering groups work is that there's this interesting kind of a triad that decides how things happen. It's the manager of the team, it's the tech lead of the team, and the PM, potentially, of the team. And that triad is also subject to the whims of all the people on the team. And so it's like a very complicated process how things get done. And I think that understanding that model, which is different from the academic model, I think that is the biggest adjustment, and I think a healthy adjustment. I think academia is very rigid. But I think that's a big adjustment. OMAR BENJELLOUN: Just one more comment I want to make on that question. I think there's a very strong computer science culture at Google, and engineering culture, but also computer science. So if you come with a view of problems which is sort of academic or researchy, or wants to think about problems in a fairly rigorous way and model them property and solve them property and prove that you have the right solution, this is something that's very respected. And so, I think at Google more than anywhere else, doing things right and solving the problems sort of at a fundamental level is something that is valued and respected. AUDIENCE: I have a a comment first and then a question. [INAUDIBLE] question. The comment, well, you asked, or somebody [INAUDIBLE] the idea that people [INAUDIBLE] publishing. Well, the fact is, I'm in software engineering. I go to software engineering conferences. I publish in software engineering. And you see [INAUDIBLE]. I mean, it's not a value judgement, it's just a fact if I look at the program. So people come away saying, OK, there's lots of PhD's at Google, there are lots of smart people at Google. They're not [INAUDIBLE] in software engineering. What they do in software engineering [INAUDIBLE] Well, why is that? And so, I think that's where people get the idea. It's a very pragmatic [INAUDIBLE]. MICHIEL BACCHIANI: The comment I have, and this is my personal view of it, but I think the panel said it to some extent. But people join here, and it's a very different environment. So they have this engineering team mates that really want to get something going. And at some point, you have the system out there, and it serves people, and there's traffic going in, and you have the signals coming back from that which you want to optimize. So to some extent it's sort of like taking a break and doing the paper publications, and sort of formally write it up and doing the measurements in some kind of controlled way, which isn't usually frequent the way that an application is deployed, where it's just, you get more and more traffic. A lot of the things are not very well controlled in that sense. It makes it harder to do the paper and sort of becomes a distraction. I think they get so roped up into it. AUDIENCE: I was saying [INAUDIBLE] I was responding to the question that was asked, where do people get this idea from? I'm saying, that's where people get the idea from. MICHIEL BACCHIANI: I mean, to some extent, I bring it up because I think the misperception is that there's some kind of corporate shut down on any kind of publication because that's not what we value. I think that's very unfair to address us like that. AUDIENCE: OK. So, my question for you, though, which is completely [INAUDIBLE] is you spend a lot of time comparing Google to academia. Now, I'm somebody who's, my father's attended [INAUDIBLE] which, the minute I moved there, became AT&T Labs. And I spent a lot of time at Google Research [INAUDIBLE]. So as somebody who spent time in AT&T Labs, I ask you how you compare [INAUDIBLE] now, of course, you went there in a different life stage. [INAUDIBLE] According to this is a brand new PhD. [INAUDIBLE] How would you compare life at AT&T Research versus life at Google? MICHIEL BACCHIANI: So, when I joined AT&T Research, I gave you the context of getting hired at AT&T. I remember I was done with grad school and had lined up five or six interviews. And the first interview I did was AT&T because whoever was my contact there was very proactive and got that going, so it was becoming the first interview that I did. And I visited one day and had the interview process. And it was actually so much fun that I came back the next day, and this is '99, right? So the market is hot, I guess. And they're worried00 I didn't even factor in my mind, but they were eager to get people. So the next morning, I get a phone call saying, like, I still have to do the paperwork and the approval or whatever, but you're going to get an offer. At that point, I literally canceled all my other interviews because I liked it that much. So I loved AT&T Research When I joined, it was really a fun place to be. But I also felt that in my years there, I mean, there were nasty things because the company was going down the drain. But AT&T Research in itself was AUDIENCE: You have no idea how far down the drain-- MICHIEL BACCHIANI: But I mean, AT&T research in itself, I think, was the greatest place. Because all my colleagues there were famous, so there was no attitude. They was sort of like, do good work. There was a lot of interesting people to talk to. There was a collaboration within there, but the rest of the company was alien. In fact, I remember making fun of the rest of the company where you'd have these town hall meetings, and we didn't really understand what people are talking about, because it's like, the rest of the company, to some extent. So I think the contrast is, like, all of this is AT&T research. The entire company. It's very open, collegial, sort of collaborative, many aspects to it. So I think that's the best comparison. AUDIENCE: [INAUDIBLE]. GIDEON MANN: I'll take that. And so in 2009, I had this project I wanted to work on. I found a guy, [INAUDIBLE] Silverman, who was at NYU HD program. I convinced him that we should work on this, and as part of the project, [INAUDIBLE] in spreadsheets, machine learning, he checked out the spreadsheet's curve, he built it locally and he embedded our software into the spreadsheets. I thought it was the coolest crap I'd ever seen. And then he produced this demo which showed what it would be like to have this code in actually the spreadsheet. It was like a different group, a group that sat-- I think they may have at that point sat on the same floor as we did. But we didn't know them at all. And so we built the prototype, he and I [INAUDIBLE] mostly his work. And then he showed the demo to the group, and it was a very exciting moment, because we're able to show them their software doing something that they hadn't expected before. And so that particular group kind of laid dormant for a very long time. And I think recently, we went back and built a lot of those pieces in a different direction. But I think that pathway is a very common pathway. You see someone else's code, you check it out, you build it, you modify it, you talk to them, you try to get their feedback, you try to plan a way forwards, and it kind of moves this. And that's just saying it's organic. It's like, you can see just about any code in that company. You can send an email to just about anybody and they will respond. Actually, the more senior are, the quicker they respond. And I don't know why it works that way, but it seems to. As far as collaboration, internal collaboration, I think it is a wonderland. It's fantastic. people are very eager, they're very interested, and it's a nice place for that aspect of it. AUDIENCE: This one small anecdote. I worked in IBM research for quite a while, and IBM Research and IBM Software Group were two completely separate entities. And we had a prototype we wanted to build into the product, and it was a nightmare to get their hands on code. And there was envy, and protective, and politics. I mean, here, we have access to almost all of the code, you can talk to almost everybody, mean there is not the question of what forums are there for communication. The company is the forum for communication. OKSANA YAKHNENKO: And just to add to that, we have tons of resources to even like, you know, finding who's working on related projects. For example, everybody has like an internal page, and anybody in the company can actually access your internal page. All the projects have internal websites with, like, members, what they work on, what they're interested in, there's a mailing list for that. Then there are mailing lists on pretty much anything you can imagine, so there's like a mailing list for machine learning, and there's a mailing list for some internal specific tools, or like, mailing lists for some of the products that are being developed. It also happens from, well, in my experience anyway, is that, when I joined the company, I already knew a lot of people, not necessarily in New York, but in other offices. And just like, people I've met at conferences, or like people who were in the same field. And remote communication is like super easy. You literally go to a conference room, you press the button, and that person is on the screen in front of you. Magic. And then, as Gideon said, people are really, really willing, and really supportive. If you want to set up a meeting with somebody, you don't even have to, like, sometimes you can just put it on the calendar, and they will show up. It's like, it's magic. EUGENE WEINSTEIN: Also just to add one note about what venues there are for fostering collaborations. So we have a thing called beer and demos, which is, it's on Friday, right? Right, so you can show up to this room, It's very much like this, and you can have beer and watch people demoing their half working, or quarter working, or it might work, kind of demos. And if you have interest, if you're interested in the work, then you can come up to them and create a collaboration that way. We also have millions of tech talks, people, both internal and external speakers coming to present their work. And you know, there are visiting faculty, there are other people that are around and want to share their ideas with you and are interested in creating collaboration. So there's many venues that we haven't thought of. It Sorry? MICHIEL BACCHIANI: So you could be needing for 10 minutes already, so any questions that you think are really, really important to be answered by the panel rather than the person that will probably be sitting at your table? AUDIENCE: Ask a question? MICHIEL BACCHIANI: Yes absolutely AUDIENCE: [INAUDIBLE]. STU: I suspect that some of our offices in other countries [INAUDIBLE]. MICHIEL BACCHIANI: One of my favorite quotes is, some people in other places, probably for good reasons. Final question? AUDIENCE: So, [INAUDIBLE] it's most important, in response to the [INAUDIBLE] I was wondering what the extent of that was and how common [INAUDIBLE]. UMESH SHANKAR: Well, in some sense, it's part of the price of success. You build something and it's not being used, you don't pay the tax, but that's not really where people want to end up either. And part of it, too, is I think you sort of build in the cost of maintenance to your utility function of how good your thing is. Where that just becomes part of how you decide if a system is good. Right? It's not too good, even if it just works, or even if it works well. It's how easy is it to modify over time? How easy is it to maintain over time? How much of your resources or your team's resources is it taking? And so, you get better at it. Sort of naturally, things often evolves to a state where you spend your time either on doing new stuff or making the things that you already have more efficient and taking less maintenance burden so you get to a good state again. So yeah, it definitely various, I think. Working on infrastructure, your definition of success is making people that use your stuff successful in a sense, and that's what my team's work on is largely in the infrastructure area. So we probably pay a larger tax that most, but some of what-- I sort of alternate between calling it a tax and learning. Because a lot of the times, the things that we spend a lot of time with are basically customer service. So a new team is integrating with some technologies that we've built. We've built security and privacy infrastructure, and they'll have a novel scenario, right? They have different data access patterns from ones we've seen before, they're doing some mash up of kind of more internal facing systems with externally facing systems, and that's a new thing, and they're querying data in a different way, all this kind of stuff. OK, we haven't seen it before, right? And so it's like, ah man, we've got to put down the code that we're writing and help them figure out the right thing to do. Maybe we have to change our code, maybe they have to change their design, maybe we just come up with something clever. But the net effect of those things is that, when you design the next version of your system, you've already taken those into account. So I think maybe I was a little harsh in calling it always a tax, exactly. There is certainly an element of that, right? You end up doing some stuff kind of a lot, and it's like ugh. Got to update something again, right? Or some machine broke somewhere and now you've got to fix something. So that kind of stuff happens, right? But I said over time, the trend is towards, OK, well, if some machine broke for the 10th time and I'm having to spend the 10th time to fix it, maybe I need to engineer around that problem. And now you have a much more robust system too. And in terms of customer service, maybe you build your system in a way that people don't have to ask as many questions, and it's just more natural and easy for them to use. And that, too, is progress, so yeah, you try to find a good equilibrium.
A2 inaudible research people gideon academia mann PhDs at Google: The Embedded Approach to Research 60 8 Hhart Budha posted on 2014/06/12 More Share Save Report Video vocabulary