Placeholder Image

Subtitles section Play video

  • Welcome to the Analytics Engineering Podcast, featuring conversations with practitioners inventing the future of analytics engineering.

  • We are on site, Coalesce24.

  • I am sitting here with Scott from Brooklyn Data.

  • Scott, there's a lot of people who listen to this podcast who know who you are already.

  • Take a minute to just say hi and give a little bit of background.

  • Sure.

  • So I'm Scott.

  • I'm founder of Brooklyn Data.

  • We're a data consultancy that works with, let's say, companies all shapes and sizes to help them with their data strategy and then to implement that strategy.

  • And, you know, one of the tools we work with very often is dbt.

  • And I guess personally, I've been involved in the dbt community.

  • Oh, my God. 2016?

  • I mean, you are the dbt community.

  • I've had the pleasure of kind of, you know, knowing Tristan and the dbt gang for ages.

  • And it's just been a pleasure to watch it all grow.

  • And we're going to talk a lot more about your background.

  • Before I ask you my first question, I just wanted to do a little behind the scenes.

  • Both of us have been here all week.

  • This is Thursday, right?

  • I can't tell you.

  • I think it's Thursday.

  • We were both just talking about how our voices are like an octave lower than they normally are and because of the long week.

  • And it was really funny right there.

  • You and I have had a certain level of like end of conference energy.

  • And then I asked you this question, we're on air and you just you just did it.

  • And you like you brought the energy.

  • I'm like, OK, I will do it.

  • All right.

  • All right.

  • We're to rally.

  • That's good.

  • We haven't had a lot of sleep and I don't know what time zone or date is, but we'll rally.

  • OK, good.

  • I'm here with you.

  • So you have some some news to share that I think you've been talking with some some different folks about.

  • So I don't think we're going to break this live or anything, but but it is a little bit new information.

  • Do you want to get it out there?

  • I started Brooklyn Data in summer of twenty eighteen.

  • That is kind of crazy.

  • The number twenty two.

  • Yeah.

  • Something like that.

  • Yeah.

  • That seems all right.

  • Six years, six years, two months.

  • We grew rapidly.

  • I mean, I think big thanks to, you know, DBT and all the success the DBT community has had and build something really special.

  • And about a year and a half ago, we went to an acquisition we were acquired by Valir, which is an amazing digital agency based out of Boston.

  • And I couldn't have asked for a better home for Brooklyn Data.

  • And for the last year and a half, I've been working hard on integrating and making kind of any acquisition is like a marriage.

  • And I feel like I can do a whole podcast on just kind of integrations and acquisitions and everything I've learned.

  • But the story is the integration has gone great.

  • We're one company.

  • And as of kind of the end of this month, October, I'm going to be stepping back from the day to day.

  • So I'll still be a board member of Valir and Brooklyn Data.

  • I'm going to be involved in the strategy, big picture, all that kind of fun stuff, but kind of won't be doing any more hands on keyboard and not quite sure what's next.

  • No plans.

  • I mean, you know, my last day is, you know, end of October and the next day after that, I literally do not have a plan.

  • So I feel like I'm going to have to ask Tristan for some help on coming up with hobbies.

  • Yeah, I maybe I can help you just a tiny bit, but I'm not great at hobbies.

  • I don't remember the details of this, but when when you were at Casper, obviously, and I want to hear the whole history, but you were very involved in DBT and the DBT community, and then you started Brooklyn Data and and we kept in close touch.

  • At first, I was like, oh, like what's going to happen to DBT at Casper?

  • But that actually turned out to be totally fine.

  • But then I was like, OK, this is interesting.

  • At the time we were Fishtown Analytics, we were and you, a longtime friend and supporter, were now a competitor because, you know, you were also a professional services company.

  • But, you know, there's like this bond between the different people doing this very hard thing.

  • And so we stayed in touch.

  • And I seem to remember in the early days asking you, like how you were managing to be a husband, a new father and the start like the what did I say?

  • And well, and your answer was something like, I just don't sleep a lot.

  • And you said it a little bit with like crazed look in your eyes.

  • Can you talk about like, what did it take to get this thing off the ground?

  • Yeah, I think the answer is I didn't sleep a lot.

  • You hear the saying like it doesn't work if you love what you're doing.

  • And it's true.

  • I mean, the reason I have to find a hobby is because building Brooklyn Data has been a hobby.

  • It's been, you know, any free processor time I had on my mental CPU, I was thinking about new and innovative ways in Brooklyn Data, not because I was on the clock, not because I had to, but because I loved it.

  • And so the journey has been great.

  • We met when you were kind of, you know, working with us at Casper and helping us set up DBT.

  • And I felt kind of very privileged to be in the know on something that's like DBT.

  • It was special and cool.

  • And I told all my friends not because, you know, there was anything in it for me or I thought I'd start a business in it because DBT was awesome.

  • It was changing my life, changing my team's life and changing, you know, the data scene, especially in the New York data scene.

  • And so, you know, we had a lot of people asking us for advice because I think at that point we had probably one of the more sophisticated DBT setups back in, you know, 2017, 2018.

  • You'd already gone through some refactoring efforts then.

  • Yeah.

  • I remember the first time you had the DAG visualization.

  • And we were kind of you had to manually run it for us and it shook like it was like this.

  • Yeah.

  • Like this nest.

  • Before you could visualize the DAG, you just kind of built whatever.

  • You yellowed it.

  • And then all of a sudden you could see it.

  • And you're like, oh, yeah.

  • What did we do?

  • Man, it's like living in like a pigsty.

  • Then someone put some ice and you're like, oh, my God.

  • But yeah, then some people would ask me for questions and advice.

  • And I think there was a point where like, hey, I think there's a business here.

  • I think everyone, you know, very smart people are working on very cool pieces of software, DBT, Snowflake, you know, Looker at that time.

  • But, you know, there was a gap in professional services, people that could guide companies on this journey to get the most out of these tools.

  • Because the kind of classic consulting companies like had not really woken up.

  • Totally.

  • This.

  • And you couldn't just like show up to Accenture and say, like, help me build a modern data stack and move faster and all this stuff.

  • There's a whole crop of company.

  • You know, there's I think the original one was Dash 42.

  • That's right, too.

  • And super sharp.

  • For a little while, Das and Lucas and Nick were my like arch nemesis.

  • Really?

  • Well, just we were competing for all the very few deals that were out there.

  • And I can't believe that you at Casper ended up selecting us.

  • Anyway, it doesn't we don't have to get into all that.

  • But Das was out there.

  • Then we started and there were a couple like Montreal Data Co.

  • Yeah, Montreal Analytics is great.

  • Yeah, right.

  • And there's Serial and awesome.

  • There's Data Climber.

  • Do you remember anybody else from this?

  • I was catching up with Aaron yesterday.

  • Data Climber, Data Culture.

  • And there's a few different freelance folks.

  • But I think those were those were the usual suspects.

  • So it really was that these folks had gotten to see how to use this new set of tools ahead of where the market was.

  • I saw the light.

  • And everybody that I talked to at that time said, I think that there's another McKinsey to be built.

  • Yeah, I think you said that.

  • I think I said that multiple times to you.

  • And you're like, I don't know about that.

  • So now there's this kind of been this life cycle.

  • And I think every one of those companies has now joined a larger entity.

  • Yeah, I mean, Aaron Data Climber, you know, joined a larger company about, I know, six months ago and Montreal Analytics about a year, year and a half ago.

  • Yeah.

  • Do you think that the McKinsey of the modern data environment is just McKinsey?

  • Maybe.

  • Is there still like a great independent consultancy to be built out of this era of data?

  • I don't think there will be a pure play large scale data consultancy.

  • I think that's too narrow.

  • You need to be a broader.

  • You need to be a broader.

  • You need to go.

  • So there's a few things.

  • And the reason why I think you'll see that kind of Aaron and the rest of the gang ended up kind of joining forces with larger firms is you need scale.

  • And what's happened over the years is that the industry has just matured and you need scale for brand awareness to scale the investment systems and account management and sales teams and sponsoring conferences and all the things that, you know, when it was just me and my Brooklyn apartment coding, I couldn't have even imagined or afforded.

  • And so you need that scale.

  • And I'm a big believer that at least one of the reasons that Brooklyn Data and we chose to join Valeer, which is a digital agency, is you need multiple service lines, multiple complimentary service lines because data work by its very nature will ebb and flow.

  • You know, not every organization is going to refactor their data warehouse every single year.

  • And don't get me wrong.

  • We have these kind of long, durable, amazing relationships with our clients.

  • It typically starts with kind of a bigger project than it evolved to kind of other different projects.

  • And for many, many of our clients, we have a long, durable, you know, similar size relationship for many years.

  • And I think the best way to kind of keep those long, durable relationships is to take that trust that you have built with this client, with this kind of, you know, your stakeholder, with this organization and help them with other stuff, help them with other stuff.

  • Yeah.

  • And it's you've got their trust, you know, their organization, you're not doing a cold start.

  • In fact, you know, the beautiful thing about data is it plugs into everything.

  • And so we're already working with the marketing team.

  • So we're already working with the supply chain team.

  • So why not help them with other things?

  • Yeah, it is so interesting.

  • You know, I started my career at Deloitte and I've now gotten to see things on the other side, partnering with lots of the largest consulting organizations in the world.

  • And it really does seem like the biggest competitive mode that you can have is those like partner relationships with the business stakeholders.

  • You got that.

  • The trust and the context is just not replicable without like years and years of relationship building.

  • People buy from people.

  • It's with any product people buy from people, but with services, the people are the product.

  • So it's like doubly true.

  • And, you know, I've tried to kind of reinforce to my colleagues at Brooklyn Data that we're somewhat in the hospitality business, you know, we're in the surprise and delight business.

  • We're in the kind of really, you know, listen and read behind the lines, give people what they want and give them what they need.

  • You know, it isn't as I'm sure there are organizations that, hey, here's a spec, we'll build it.

  • But I think truly successful consultancies that we built at Brooklyn Data are empathetic, have these great relationships and are, like you said, partners.

  • Not it's, you know, listen, at the end of the day, there is going to be a transactional relationship to any third party consultancy, but it should feel as much like a partnership as possible.

  • And I think we've done that.

  • A lot of our clients call us like the consultancy that doesn't feel like a consultancy.

  • And I think that is the compliment.

  • Yeah.

  • Let's go back in time a little bit.

  • Your story and I don't I don't actually know that your full professional story and I want to get it out of you.

  • But I think that you are a little bit emblematic of like the data practitioner of a particular era.

  • I think that's very true.

  • You like started as an econ person, right?

  • Well, yes, I started.

  • I studied business.

  • I did management consulting, kind of like you, strategy consulting for four years, Excel, PowerPoint.

  • And it was very quantitative, very quantitative, but no sequel, no little VBA, record a macro, that kind of stuff.

  • But that's as technical as it got.

  • You built like models to forecast economic things and economic things, revenue, financials.

  • You know, I remember having, you know, waking up at two in the morning early in my career with Excel nightmares.

  • I feel like anybody that's done that and just now I look back and I see the testing and all the version control that you have in data.

  • I would have saved a lot of heartache in my early in my career.

  • And and your your Excel macros were strong.

  • I was really good.

  • I'm really I feel I'm very, extremely good at Excel shortcuts.

  • In 2009, I remember hanging out with some friends from business school and one of them was like a person who liked to start conversations with big questions.

  • And I remember he asked this question.

  • He said, what is the thing you think your top ten thousand in the world at?

  • And I said, I think I'm really good at Excel.

  • Everybody else had like better.

  • And I was just like, I'm really good at Excel.

  • There's like something.

  • And, you know, I went to pride.

  • I mean, it's still my happy spot.

  • Like, yeah, I love when I go to DVD.

  • I don't do much very often.

  • But when I do, it's like very tangible, very fulfilling.

  • I love when I do Excel.

  • I mean, and it's like no one's looking.

  • I'm still going to format it really nicely.

  • I mean, no one's going to see it.

  • This is just for me.

  • Yeah, it's going to look great.

  • I particularly find I really like it to always be the correct font size across the entire sheet.

  • And then the references from the other sheet have to be green and the constants have to be blue.

  • I mean, you got to.

  • A hundred percent, a hundred percent.

  • And we've never had this conversation before, but you got to know those conventions.

  • Green and blue, green, blue and black.

  • Exactly.

  • OK, this is where you kind of started.

  • You somehow got a job running data at Casper.

  • How did that happen?

  • Yeah, that's a head scratcher for me.

  • I'm not quite sure how I got that.

  • So I was living in London at the time in working consulting, moved to New York, had taken sabbatical for my consulting job.

  • That's kind of like a fallback if things didn't work out.

  • And I'm like, I'm going to go work in like tech.

  • And I didn't know what that meant.

  • And I just literally went to meetups and reached out to anybody I had the most tenuous connection with on LinkedIn, went to a meetup every night for like two months.

  • Oh, my God, that's a lot.

  • My goal was a meetup every night and at least two coffees every day.

  • And I just did it for months.

  • And I just got to know.

  • That was a good era for meetups, though.

  • Everyone was not like meetup out at that point.

  • Correct.

  • I mean, it was it was like the heyday, you know, people, their guard was down.

  • I mean, it was just like a good place to meet people.

  • And, you know, listen, it is in the heyday, especially post COVID.

  • But I do really, really recommend to everybody networking is key.

  • Build your network.

  • Go to meet people.

  • Meeting people in real life is that builds a much stronger relationship than digitally.

  • And so I kind of landed on tech and I was thinking about finance.

  • And then someone told me about this data thing that people are doing.

  • That was, you know, 2014.

  • So Redshift had come out like two years before.

  • It was really early days.

  • People talk about Vertica and all sorts of things like that.

  • And so a friend of a friend introduced me to the founders of Casper.

  • They brought me on as a, you know, a freelancer.

  • And I remember to the like, you know, if you could call it an interview, they asked, like, why should I hire you?

  • I said, you don't want to hire someone that knows the tool and only knows the tool.

  • You want to hire someone who knows how to think and can learn any tool.

  • Yeah, I didn't know where that came from, but it landed and it worked.

  • And I actually I do actually believe that very strong.

  • So you were the first quote unquote data.

  • Yeah, I was employee number like 16 at Casper.

  • Oh, and I was there for four years.

  • It grew to a hundred four hundred people.

  • The data team grew to 15 or 16 people.

  • Did it get acquired?

  • This isn't a nice company.

  • So, I mean, I think, you know, a lot of those DTC darlings had kind of you had a moment at a moment and then there was a little bit of a reckoning.

  • And so it went it went public.

  • I did not know that went public maybe to twenty nineteen, maybe.

  • And and it didn't do well, even though, you know, the first day, you know, you have this when you have shares, you have this kind of block up period and you can't sell it for six months.

  • And, you know, I think everybody in Casper was collectively just watching the stock slowly go down over six months, which is fine.

  • I mean, I I wouldn't trade anything.

  • Well, the Casper experience was amazing.

  • I learned so much.

  • It set my whole career up.

  • And it was like, you know, people talk about like the in the PayPal mafia.

  • I think there's like a Casper mafia, like everybody's doing very cool things.

  • And so so Casper, then then they went private again.

  • OK, so somebody took him private, took him private.

  • And it's I mean, I still buy Casper mattresses.

  • I still do.

  • I don't know anybody that works there, but I buy it out of brand affinity and like I bought the like top of the line mattress.

  • I'm still supposed to keep you cooler.

  • I was like, did it work?

  • It took.

  • I love it.

  • I it's actually worth your convincing me.

  • I always go to the standard.

  • But we had a podcast about data and now we're talking about this.

  • Here's the reason that I wanted to go back in time to this, because I think that historically data teams or it wasn't even a thing like data teams.

  • But mostly there were IT organizations that did data things because executives needed dashboards.

  • And so there was a data capability in that organization.

  • But when you hire people inside of IT, they are not they don't have the background that you just described.

  • And so all of a sudden there was this all the tools were changing.

  • There were startups.

  • There was no kind of existing data infrastructure.

  • And so you got the opportunity to take people who understood strategy and give them control of the infrastructure and like the keys to the castle.

  • And it produced like very interesting, neat things.

  • Yes.

  • I don't want us to go back to a world where like data is IT.

  • That's not to say that IT and data engineering and all like there's real systems to be built fine.

  • Agreed.

  • But there was a beauty in that the world that got created when the people that understood the business also like decided how the tech worked and how the word structure worked and all that stuff.

  • I agree.

  • It's interesting.

  • I think the evolution of the Casper data team kind of exemplified that.

  • And so when we started, everybody in the Casper data team looked a lot like me, former management consultants.

  • And actually, we looked a lot like our stakeholders.

  • We would start to specialize.

  • So we'd have people in the data team that would report with the marketing team and they would go to marketing meetings and be embedded.

  • But we got a point, I would say two years in where the scale of the data, of the challenge, of the complexity of the technical infrastructure we were managing.

  • And again, this wasn't rocket science, but, you know, we're trying to sort keys and disk keys and let's not be vacuuming anymore.

  • And, you know, Drew, my co-founder, went to a Halloween party one time as a redshift vacuum.

  • And I'm sure there's one person at the party that got it and found it hilarious.

  • Yep.

  • But I'm scarred from the sort keys and disk keys and vacuuming.

  • And so then we saw a little bit of a divergence and evolution of the team where we started seeing more STEM degrees, more people with kind of, you know, engineering or tech, not necessarily computer science backgrounds, but, you know, more technical backgrounds come in.

  • And we saw actually a divergence in the personas.

  • We still have the people that we had kind of the OG, you know, Casper data team.

  • They were a little bit more stakeholder facing.

  • I mean, you know, we were all one office.

  • We all talked to STEM stakeholders, but there were folks that were a little bit more, you know, focused on the technology, you know, writing good, clean pull requests.

  • And I think that evolution totally made sense.

  • They were all kind of and even I look back and those people that I'm describing, those are early analytics engineers.

  • Yep.

  • And, you know, we that that was kind of we organically saw the split of the analyst and the analytics engineer.

  • Yeah.

  • OK, so this is the thing that I think that the thread that I was trying to pull out, I have I had a conversation in 2015 with somebody who ran a, I think, modestly sized data team at Zillow, and they had a similar background to you.

  • They were like econometrics or something like that.

  • And, you know, Zillow is pretty quantitative place, good data systems, et cetera.

  • And I asked him to show me their workflow and their workflow was they had Redshift and they wrote 500 line SQL queries on Redshift.

  • Yeah.

  • Then they pushed like copy this data set.

  • They dumped it to Google Sheets.

  • Then they took the 500 line SQL query and like dumped it in a tab called SQL.

  • And it was just like this persona that like truly understood the quantitative part, but like knew nothing about how to build large scale systems.

  • And I think so much of the journey that I'm interested in in the past 10 years is like, how do you understand the the spectrum of personas that are needed to do great data work?

  • Because you really do need the systems thinkers and you really also do need the quantitative and more business facing people.

  • And they're often just not the same people.

  • And how do you get them to work together?

  • I agree.

  • I would also say, like, even if you get more technical, then you have more platform engineers or even further, like, you know, everybody's building on that.

  • That's the challenge.

  • But your vision and I think and what I've seen is get them all speaking in common language.

  • And that was SQL.

  • That was like one of the big ahas, you know, six, seven years ago when remember, I mean, there was a clear debate, Python versus SQL.

  • And we all kind of forget about that.

  • I'm not saying that Python lost, but like SQL has very much become the language of communication I've seen between kind of analysts and engineers.

  • It was like deeply uncool in 2016, 17.

  • And I don't know that it's now cool, but like it's not like a knock.

  • It's people don't not adopt DBT or tools like DBT because they're like, oh, that looks like a that's like a mark on my job, on my resume.

  • Yeah.

  • What do you think created the opportunity for change over the past decade?

  • Because things in data have changed a lot.

  • I'm a believer in the trends and forces theory of history and not the great man theory.

  • I don't think it's that Ali Goethe or whomever like invented some magical thing and like changed the world.

  • I think it's like there are these trends that happen.

  • It created space for us to do stuff.

  • Yeah, because you probably also see your story not as like, oh, my God, I'm so incredible.

  • I created this great, incredible company.

  • You probably see it as like it was a funny opportunity that I stepped into.

  • Yeah.

  • Holy shit.

  • This happened.

  • I mean, I think it's it's compute and storage just got cheaper, got cheaper.

  • And I think that's that is if you strip away all the things, that's the thing.

  • Yeah.

  • I mean, it's cloud driving compute and storage to be cheap for you to I mean, that's that's that's what kind of allowed us to to kind of E.L.T.

  • Let's you know, let's store, you know, not just kind of save some data.

  • What the heck?

  • Let's save it all.

  • You know, let's not just look at five days of data that look all the data.

  • I mean, it's just like storage, compute, columnar storage, Redshift.

  • I think Redshift was very much the big unlock.

  • It's not like because I remember early days I was chatting with.

  • I think that's how you know OG in the space that they have an emotional connection to Redshift.

  • But I remember chatting with people that had Vertica, Teradata.

  • It's not like people weren't doing it before.

  • What Redshift did was made it a click of a button.

  • And then, you know, you had these layer layers of Looker made BI and transformations like the first level transformations.

  • I remember we had our first Looker demo from Lloyd Tabb came into the Casper office, demoed Looker.

  • He's like, just put Looker on a read replica, do your transformations and PDTs.

  • And that would have been cool.

  • I never got a Lloyd demo of Looker.

  • I mean, he was very passionate, very convincing.

  • And we like sighed on the spot and it worked until it didn't until we outgrew the PDTs and nested PDTs.

  • And it wasn't performing and it was like, go get your coffee, you know, and then your dashboards will be ready.

  • But like all these things, Fivetran made it easier to bring data in, you know, DBT made it easier to transform and scale and use GitHub and version control.

  • So it's just all these kind of technologies that allow people to do something easier and easier and cheaper and cheaper.

  • People have always been making decisions since they were cavemen.

  • You know, do I hunt in the woods or the pasture?

  • I mean, like, is that a line that you use in sales meetings?

  • It does sound good.

  • I should use it.

  • It's a good line.

  • What the technology just has enabled us to use better and better decisions.

  • And I think at the end of the day, it's storage and compute getting cheaper.

  • Yep, yep.

  • And that brought us to there was like this fundamental thing that was going on, which I think you described very nicely.

  • And that brought us to this interesting point that mashed up against zero interest rate.

  • Yeah.

  • And the availability, like almost infinite availability of venture capital.

  • And, you know, we're at Coalesce right now.

  • And you could see in 21, 22, this show up by the like vendor booths at events. And all of a sudden it was like, there are 10 companies that do X thing.

  • And, and, and, you know, we're probably starting to back our way out of that a little bit, which is, you know, a messy process, but natural.

  • It feels like this is the year where, you know, the next 12 months is when things solidify.

  • Yeah.

  • That might be true.

  • But like every wave has its crest and there's a new wave.

  • There's a lot of interesting things going on in the space right now.

  • Iceberg, AI, there's just a lot.

  • If you created Brooklyn Datacode out of this one set of this one particular context, like what are the companies that are going to get created, like the professional services companies, what are they going to get created based on now?

  • What are the waves?

  • The modern data sec wave was gigantic.

  • And I think like, I think back and I feel very privileged to be at the right, exact right place at the right time when it happened.

  • And, and, and, and kind of just like the dbt community, like, I don't know if I will ever see something like that in my lifetime.

  • And that's the dbt, that's analytics engineering, that's modern data sec.

  • I think that was a, an experience that I'll maybe only see once in a lifetime, but I think there are other waves.

  • I think, you know, you see people talking about AI, of course.

  • I mean, I think, I think that's, that's a kind of opportunity.

  • Clean tech, energy, quantum.

  • I mean, have you seen, are there boutique AI consulting shops?

  • I haven't looked around, but I'm sure there has to be, right?

  • I'm, I'm not familiar with that, but I imagine it's, it's in a similar place to when maybe it was 2015, 2016, you know, when maybe there was a DOS 42 and a bunch of freelancers.

  • I think the only difference maybe with AI though, is that the modern data sec flew under the radar for many years.

  • Yeah.

  • AI went straight to quantum.

  • It was never like a New York times front page on the modern data stack.

  • And so it was actually easier to kind of.

  • I mean, we, we even today, I mean, we win against the big, big, big, you know, global consultants, as you know, because we know the tools better.

  • We've been using them.

  • I mean, I've been using dbt since 2016.

  • You know, it wasn't enterprise ready back then.

  • It wasn't on the radar.

  • You think?

  • It was a fun experience.

  • It's not even clear that it was Casper ready, but we made it work. Um, but I think that's the big difference is, is that, you know, you look at Accenture, you know, making, you know, one to $2 billion a year on, on AI work.

  • I mean, like that didn't exist.

  • I feel like I've heard people quote something like that before.

  • Um, that was definitely not the case in, in, in, um, in modern data stack.

  • So I think there's a, I definitely think there's a different dynamic.

  • I think there will be an opportunity there, but I think it's like burning hotter, faster and might blow up hotter, faster.

  • Um, that's interesting.

  • Even the whole AI industry, uh, it, it feels, um, we all universally agree that there's a there, there.

  • Um, I think we all universally agree that, uh, 90% of the startups we see will, will fail and these kind of VCs are paying really, really, really, really high, you know, valuations for investments.

  • I think a lot of people are going to get washed, but they'll also find that one next thing too.

  • But it's, it's going to be beautiful, creative destruction in the AI space. It's sometimes hard to disentangle the I'm bullish on the underlying technology versus I'm bullish on the equity returns currently.

  • A hundred percent.

  • I'm reading a book right now on exactly that.

  • Um, I think what moves markets.

  • Um, and it's, it's basically saying that, you know, not many people made money on canals, railroads, anything.

  • I mean, they like, they were hugely value creating for America, but a lot of people, a lot of people got wiped out.

  • A lot of people got wiped out.

  • And that's, I mean, you got to know, like anything, it's all about timing.

  • You got to ride the wave and know how to know when to hold them.

  • Know when to fold them.

  • Did they talk about the, um, the Panama canal in this book at all?

  • A little bit.

  • I, I have admittedly about 20% in it, but, um, get to the Panama canal stuff at what I read about this a little bit at one point and, um, the number of people who died in constructing the Panama canal is just astronomical.

  • It's like tens of thousands.

  • Yes.

  • It's shocking.

  • And the idea that we were just like, well, it's just, it's really important.

  • We're just going to keep going.

  • That is a different era.

  • And the wildness is it's that's, that's durable.

  • I mean, the, the Pam canal is still, I mean, they made it wider, the various six, the expansion project, but it's like, it's very similar to what it was.

  • When it started, that's, that's real value.

  • That's amazing.

  • Are you saying people are still going to use redshift on a hundred years?

  • I mean, well, I think redshift, redshift's coming to have a comeback. Yeah, totally.

  • There was a time where there was a meme in the dbt community that that was that, that the redshift was not where it needed to be.

  • But, but I think that that time has just multiple years behind us.

  • Totally.

  • It, it definitely feels even, you know, it coalesces here.

  • Like it, it, it was really exciting to, you haven't heard innovation and redshift in the same sentence in a long time.

  • And I feel like it coalesce recently, you know, when there's been, listen, there's been kind of serverless, there's been kind of various different notes, but like, even today, most people use dense compute still. I do think like redshift has a really cool ecosystem play.

  • I mean, zero copy ETL, like that's really cool stuff.

  • Yep.

  • He's like, you know, everybody's running a database on, on AWS.

  • Totally.

  • There was a lot of haymaid about the benefits that a company like a snowflake or Databricks could have because they could run on all the different hyperscalers, but there's also a very unique set of benefits that I think you can have if you're a cloud, like a one cloud native product, like you're talking about, like they can create integrations with their other services that like you just couldn't do otherwise.

  • So it's, it's neat to see that ecosystem evolve.

  • Do you have any thoughts on Iceberg?

  • It's still one of these destabilizers.

  • It's like, uh, we all, okay.

  • Maybe there's like very smart technologists that understood the 15 different components of a database, but I like never actually had to get to that level.

  • I was like, okay, I understand the optimizer.

  • I understand like some different things about redshifts.

  • And then Iceberg comes along and we're all thinking about file format.

  • Yeah.

  • And that was like, not a thing that we thought about for a long time.

  • And now all of a sudden we realize we like our concept of a data system has been a little bit shifted on its head.

  • Yep.

  • Are you finding customers engaged in this thought, like the people that you are working with care about this topic?

  • No.

  • Really?

  • I'm not yet.

  • I would say as a, as a conceptual concept, yes.

  • I don't think any of the, um, any of the use cases are perfectly ready for primetime.

  • I think everyone's excited about the direction of travel.

  • Um, it's really exciting to see Snowflake support Iceberg, um, both kind of as a native format and external tables.

  • And you play that forward.

  • Theoretically, you know, your abstracting storage made the best compute platform win.

  • Which both Ali and Sridhar said literally that line on stage at their conferences.

  • I mean, they said it.

  • No, I mean, they said it, but I have to think just like any software, like I think Snowflake will do enough, you know, and they'll all do enough to support external storage while also making it easier and better if it is, you know, they are the catalog.

  • And so it's like, it's, I mean, it's a balance.

  • Like you have a balance, open source versus paid.

  • Yeah.

  • And I think they're going to have a similar balance of, I mean, unless something dramatically happens, it's always going to be a better, more performant experience to have a, to query a Snowflake managed Iceberg table than an externally managed Iceberg table.

  • You don't think so?

  • Want Snowflake?

  • Wait, I'm giving, just got a look.

  • Maybe 5%, maybe 10%.

  • Data points that indicate that you can approach the same levels of performance.

  • I, I, I don't know.

  • Uh, I, I don't claim to be a super expert on the performance characteristics here, but I've seen some compelling data.

  • Listen, I think this is a very cool direction.

  • Um, I, I'm excited where it's going.

  • I don't think it'll be this, like you have this free-flowing data lake in Iceberg that kind of everybody goes, you know, you use Snowflake, use Databricks, you use this and I'll hit it.

  • Um, and it, with the kind of equal performance and ease, maybe the idea, I just feel like there's just going to be an interest of these platforms to kind of keep people in, in, in, in their ecosystem.

  • Totally.

  • That it's, it doesn't take a rocket scientist to understand how this like maps onto the incentives of the different platforms.

  • And I think that there will be an interesting question of like, hey, get it's some Fortune 500 data architect organizations, like they are really bought into this vision of the future.

  • And then you get to figure out how much those customers can turn the crank on.

  • Yeah, totally.

  • Listen, everybody I talk to, don't get me wrong, is excited about this.

  • Yep.

  • But they're, I think they're all in a holding pattern.

  • They're experimenting it.

  • They're reusing it for kind of ad hoc use cases.

  • I don't know.

  • I don't know of anybody who is like, uh, their, their entire data lake is on Iceberg and they use a eight different compute plan.

  • No, totally with you.

  • Yeah.

  • But I, I'm excited to experiment.

  • I'm excited to flip on Iceberg, um, as a kind of a materialization, materialization kind of, uh, method in, in dbt and, and play around with it.

  • I'm going to start, and the team is already starting to test into it.

  • And I think there might be a moment where we go to our clients.

  • It's like, Hey, it's got, it's hit the tipping point.

  • I think we should convert everything to Iceberg and start thinking about from an, like an Iceberg first, you know, mindset when doing architecture.

  • Yeah.

  • Fivetran has gone very at Iceberg.

  • Yep.

  • I'm not going to remember exactly what they're calling it, but they're making it very easy to take data and deliver it directly to Iceberg and then you can kind of use it from there, however you want.

  • I mean, that's a lot because we, you know, when we bring Fivetran into enterprise clients, a lot of them do want some sort of landing zone in some cloud storage before bringing it into their, um, data warehouse.

  • And I think that, I think that speaks to both kind of an Iceberg trend and like just a need in the enterprise to have a separate landing zone. One of the other interesting things that I've heard this week is that I think we first talked about the dbt semantic layer at Coalesce 21.

  • I remember that.

  • And there was a great video, the debut of the semantic layer.

  • I could, I don't even remember.

  • I think Drew did it, right?

  • I saw it, but yeah, I remember that way back, but yeah.

  • Yeah.

  • And it has been an interesting journey.

  • It took us two years and an acquisition to get the technology really working.

  • And the past year has seen, on our side, has seen usage numbers go up into the right, but from a small base.

  • And this year, the thing that I'm hearing over and over again is people say, I'm finally ready to make the investment to do this.

  • In the past, the semantic layer, so Coalesce 23, our semantic layer enablement sessions were oversubscribed.

  • We need to actually add more of them.

  • And then most of those folks went back to their offices and said like, I've got so much other shit to do.

  • But those same people are here this year and they're like, we're ready.

  • We're doing it.

  • Have you seen conversations in clients where there's, is there an interest in a semantic layer that spans BI tools?

  • Or is this another conversation like Iceberg where it's like that in theory sounds great, but we're just like not there yet.

  • A hundred percent.

  • There's interest.

  • I mean, the, the, you, you know, you, you've known even from the early days of Fisheye Analytics that, that, that the power of having some sort of semantic, semantic layer, I mean, like it was such an advantage for Looker.

  • Yeah.

  • It's really very powerful.

  • And everyone who was saying is like, I wish I had that elsewhere.

  • And I think Looker made some ways to do it, like a Tableau integration of this.

  • And I think nothing really ever landed.

  • But there's universally been a, been a desire.

  • I think people are really excited about doing this.

  • I think there's a few things that have happened.

  • I mean, it really only, it was only in the semantic layer, this kind of current metric flow version of it was only GA last summer, like summer of 23, right?

  • Yeah.

  • When it was integrated into dbt core.

  • And so, I mean, we're kind of a year, you know, 16 months in for being kind of GA.

  • There is the network effects of now all the integrations.

  • Like, I mean, I, we have a, I kept my eyes on one of our clients works at a large enterprise when you announced the Power BI integration.

  • People were really excited.

  • Eyes and a smile, ear to ear.

  • And that's game changing.

  • I mean, not only was kind of, you know, the, the metrics that are newer, but you had limited venues to where you could use it.

  • And I think that has all changed now.

  • And so people, you have more comfortable with it, more places to use it.

  • It does definitely feel like a moment.

  • I think that's like, it is, it feels much less, much more tangible and less theoretical than iceberg at the moment.

  • So there's a desire.

  • Okay.

  • I'll take that.

  • That's a compliment.

  • Okay.

  • So let's, let's close on the community.

  • You have seen a real journey from a couple of folks in a room at Casper to you hosted a meetup, you shared dbt with folks like Kickstarter and Venmo and it's like New York tech crew.

  • And then at some point it became kind of the, one of the darlings of Silicon Valley, BC for 12 to 24 months.

  • And now it feels like it's in a maturation phase.

  • Like there are still wonderful people that are here at coalesce that have been using dbt since the beginning.

  • But there's also people who this is the first time they've been a part of quote unquote, a community that is outside of the people that they work next to at their desk in a large enterprise and integrating these different sets of people and what they need.

  • It's been an interesting journey.

  • What do you think this group of people needs next, whether it's from their software or maybe more importantly, if like from each other, like how can, what's the next version of the dbt?

  • Yeah.

  • I mean, I first, maybe I'd step back and just say how grateful I am.

  • I mean, I, just to be kind of an early member, just watch, you know, you said the dbt community.

  • I just say my friends.

  • I mean, like, this is like the, you know, I go to coalesce for a few things.

  • I go to see the product announcements.

  • I go to see which vendors are cool.

  • I'm going to see people.

  • And I think for a lot of people, that's why, because it's the community.

  • I mean, we love the Slack version, but there's nothing beats in person.

  • And it's just been such a wonderful experience.

  • And what I've loved is how inclusive it is.

  • And I think that's the key to a success and what needs to continue.

  • And that's what the community needs because come as you are, you know, it also means, you know, wear a button down in a suit to coalesce, you know, come in a t-shirt, you know, be an executive at a, you know, fortune 500 or be an analyst at a startup.

  • Um, and so I think that's what we need to do.

  • And I think what the community has done a really good job of.

  • I saw some button downs.

  • I saw some blazers.

  • I don't know that I've seen anyone in a full suit yet.

  • You didn't see the tuxedo guy?

  • No, I'm just kidding.

  • If you want to wear a coalesce, a suit to coalesce, I'm here for it.

  • But I think that the thing is like what, what I think dbt and the dbt community has done is be very welcoming.

  • I think there were several moments and there will be moments in the future where it's just like, no, this is kind of more startup be like East coasty or this now dbt is, is a collection of people that have kind of, you know, shared interests, shared challenges, now a common language to express those challenges, the common community to like, to, to learn and interact.

  • And so it's been very cool how that's happened.

  • I think what it needs is that to continue because, um, the community is only going to get bigger.

  • The personas are only going to get more kind of, you know, varied.

  • Um, and that's, you know, I'd love this kind of, you know, I really love the one dbt aspect because it's community, but it's also the tool.

  • I mean, I felt, and I'm sure you felt for the longest time that, you know, I wanted to get everybody in dbt and our clients, but there were people that are never going to learn SQL.

  • There are people that.

  • No, might be able to, but it might be needed on ramp.

  • That's a little less intimidating than the command line.

  • And that's where, you know, the cloud ID came in.

  • Um, I mean, we're all used to it now, but like, I remember the first time I used command line, it was dbt and I had to pip install.

  • I literally had no idea how to do any of this stuff.

  • Um, and so, you know, we've made it easier.

  • I learned the command line because I was anti Microsoft in the 2000s.

  • And so I, I installed Fedora core six on my home computer, and then I was an expert at copying and pasting command line scripts from my browser into my stack overflow.

  • Yeah, right.

  • Exactly.

  • Yeah.

  • So, but you're, you're a hundred percent right.

  • And then I, so now the drag and drop is going to be really interesting and it's a, it's a good opportunity for make the tool more welcoming, but also it's an opportunity to, I mean, it's going to be very interesting to see what the talks look like next year, because there's going to be talks for people that literally know dbt only as a drag and drop interface.

  • It's a whole new persona. And listen, they, you know, they don't have, you know, they're not purple with five eyes.

  • I mean, they're like people and humans and analysts too, but, um, it's a different community.

  • And so we just need to keep welcoming and, and, and being open to all the different personas.

  • So I, and I think that was a long answer to, to what I think the community needs.

  • Yeah.

  • I, uh, I, I really agree with you.

  • The things that I think about a lot are what are the ways in which we connect with each other? I think that Slack does not scale at a certain level or like the human relationships on Slack don't scale at a certain level, like certainly continues to work as a platform, but it, uh, we could really build human relationships in a Slack channel with 200 people in it.

  • Yeah.

  • Whereas that's very challenging with a hundred thousand people.

  • A hundred some agree.

  • Also, it is just like, anyway, some of the digital stuff needs to change.

  • And then I think that in-person becomes more important than ever.

  • Um, so I, I'm, I'm spending a lot of time thinking about how do we just continue to facilitate the, that's the thing that I've always enjoyed.

  • And honestly, it's not something that I created.

  • It's something that I think Drew was very instrumental in creating is that there was always a helpfulness and a nonjudgmentalness to the vibe in the community.

  • And like the dbt community never, never had that.

  • Now that you say that, it does feel very Drew.

  • You see, you see like elements of Drew in the community.

  • Yeah.

  • How do we scale culture as things continue to get bigger and bigger?

  • And that's, that's hard, but I think.

  • I'd say it's worked so far and each year it's going to be different, but I mean, I, it feels like a, it's a priority for the organization.

  • So.

  • A hundred percent, a hundred percent.

  • What is the first hobby you're going to try on the first day after you?

  • Yeah.

  • It's, oh man, I haven't done much thinking, but I do have, I have this, I have this problem where people recommend a book and I just buy it.

  • And so I just literally have a stack of four years of four or five years of books in my house that I've never read.

  • It just gets larger and larger.

  • So, I mean, I feel like that's what I'm going to tackle.

  • Is there a chair that you're going to be sitting in?

  • Oh yeah.

  • There is a chair.

  • I identified the chair.

  • Okay.

  • I'm going to sit and read a book.

  • That's good.

  • And I think the other thing I'm really excited.

  • I mean, my, my kids are eight and five now, you know, my son who's five has only known me in Brooklyn data.

  • And my, my daughter's eight has only remembered me in Brooklyn data.

  • Yeah.

  • That's wild.

  • And so, um, I'm excited to, to sleep a little bit more going back to the first question and then spend a lot of time with my kids.

  • I think we're, we've got this really exciting window of time before they want to have nothing to do with me, um, when they're in their teens and I'm just going to enjoy the heck out of it.

  • Awesome.

  • Thanks for hanging out.

  • Thanks.

  • This is great.

  • The analytics engineering podcast is sponsored by dbt labs.

  • I'm your host, Tristan Handy.

  • Email us at podcast at dbt labs.com with comments and guest suggestions.

  • Our producers are Jeff Fox and Dan Poppy.

  • If you enjoyed the show, drop us a review or share with a friend.

  • Thanks for listening.

Welcome to the Analytics Engineering Podcast, featuring conversations with practitioners inventing the future of analytics engineering.

Subtitles and vocabulary

Click the word to look it up Click the word to find further inforamtion about it