Subtitles section Play video Print subtitles PETE FRISELLA: Hi, everybody. My name's Pete Frisella. I'm a developer advocate on the Google Analytics team. I'd like to introduce you to a new series we have called "Off the Charts." This series is about getting into the deep features of Google Analytics, understanding how it works, things you can do with it, and how to use the feature itself. So today, I'd like actually talk to you about getting started with cost data upload, and this is all about measuring non-Google paid campaigns. So let's talk about what is cost data upload. I'll kind of get into little bit more about what that means and what cost data upload works, how it works and the feature itself. Then we'll go into the architectural overview, which explains the deep details and also how you would actually go about uploading data to Google Analytics. And then finally, we'll talk a little bit about some opportunities that are available from a developer perspective and also from a business perspective. So with that, we'll get started. So what is cost data upload, and what does cost data itself mean? So this is all related to all the paid campaigns you may be running out there, all the information that's related to that paid campaign. Whether it be ad costs, clicks, impressions, the name of the campaign, all this itself relates to cost data. It's what we refer to as cost data in Google Analytics. So for example, if you are running a campaign, say with Google AdWords, for example, you'll know this is pretty familiar for you. You've got cost data, and if you link that to Google Analytics, which is what you should be doing, you actually get these reports within Google Analytics that highlight, for your paid campaigns, how they're performing. And they give you a lot more information around how to optimize certain key words, maybe landing page optimization. Now, these things are important things, but to really do this analysis, you need that cost data associated with the campaigns that you're running in Google AdWords. And this linking works really well today, and it takes care of all that for you. And it gives you some powerful tools to take care of analysis around campaigns and user interaction, and how campaigns are performing. The thing is, though, for a lot of people, why we encourage you to use Google AdWords, we love Google AdWords, obviously. There is other paid campaign services out there that people obviously use, and that includes things like other paid search services. You might be running in a campaign on a social network. You might be doing email marketing campaigns. So all of these campaigns are running out there, and you're getting users visiting your website or your property from these paid campaigns, and that's great. You're able to track what they're doing, the conversions possibly that are taking place, goal completions. And you're getting a good general sense of which paid campaigns are driving traffic to your site. This is great. So a report like this, that I'm showing here right now, is kind of common that you're looking at, for my particular campaigns, how are they performing and what kind of traffic is coming to my site because these campaigns? Now, you see that first row there, row number one, is a Google AdWords campaign. And you'll notice while you have user interaction data such as visits, and you have impressions and clicks cost data, you don't get the same kind of visual in cost data that you would have for AdWords. You don't get the same data for the other paid campaigns that you may have. So for example, we have an AdWords network campaign running, and we have an affiliate network campaign running. And you'll see, we know people are coming to the site, but we don't have the cost data within GA to do that analysis. There's kind of two things missing around this then. For the particular paid campaign itself, being able to analyze that in terms of the ROI and the metrics you might be getting. Click-through rate, cost per click, things like that, you kind of miss out on that. And then also, you don't really have the ability to compare how these services are performing against each other. So is the AdWords network driving more? Better conversions, or is Google AdWords driving better conversions? So you're missing that kind of complete view, that complete analysis, around your cost data across all your different paid campaigns that you might be having for all online advertising and digital marketing. But cost data kind of solves that problem. When we launched this feature about a month ago, we had some success stories right off the bat with a couple customers, working with partners of ours to get something in place and a solution for them. So, for example, we talked Next Analytics and Cardinal Path, and they came up with a solution that uses cost data upload. And they worked with Natural Wellbeing. And really, it was around fully automating this process for them, so being able to take their cost data automatically, export it, and upload it into Google Analytics using this new feature. And it worked really well for them in saving them a lot of time and doing this full automation. We also did the same thing with ShufflePoint and E-Nor. They came up with a solution using cost data upload again to help OEMPCWorld. And this one was really about, again, fully automation. And a lot of the analysis they were doing was kind of a manual process where you had your Google AdWords data, you had some other campaigns. You had to manually bring that stuff together and do analysis outside of GA. So this new feature really allowed them to bring all that into Google Analytics and to do that complete analysis within the product itself. So it saved them a lot of time, fully automated, a great solution for them. So I think it would be good just to start off with a quick demo to show you how this actually works. And then we're going to dive into details, and I can give you a little bit more background on how these actual steps that involved to do this. Sometimes, it's just easier to show you how it works, so I'll start off with a report. This is within Google Analytics. It's a custom report, and the example here is that I have a paid search ad running, and it's for an ad network. It's a CPC, and it's for the keyword "dogbone." And I can see-- so I've tagged my URLs, and I'll get into that a little bit later. So I know that someone's clicked on the URL, and they've visited my property, and I can see that there's been two visits. But just as how I described before, we don't have any cost data here. But the thing is, I actually do have the cost data. The service provider that I'm using allows me to export the cost data. I just don't have it in GA, so that's really what I want to do. I want to have that same data available in GA so I can do analysis and can carry out cost analysis. So just something to note here, the source is Ad Network, the medium is CPC, and the keyword is "dogbone." So as I mentioned, I actually do have this cost data. So you can imagine that I exported this from the paid search provider and have it in CSV file. And there's a header up here that has some dimensions that I'll explain later. But in row two, we can see the same values that I just showed you in report are actually here. We have Ad Network as a source, CPC is our medium, and we have this keyword here, "dogbone." And there's that cost data I was talking about. We have impressions, we have clicks, and we have ad cost, so I have this data. It's there. It's available, and there was a couple other keywords that I ran also. Now, let's actually upload it and see how this process works, and then we'll see-- once it's available in the report, you'll see how it's actually joined together. So I have a little script. It's a Python script, just to kind of test things and see how it's working, how the feature works. I'll start off with-- there's this new entity called custom data source. I can also explain what this means in a little bit, but basically, you upload the cost data to a custom data source, and you specify which date you'd like to upload the cost data for. So, in this example, this cost data actually is from yesterday, so I exported all of the cost data and metrics from yesterday into a CSV file, and it's available to upload. So I'm going to select the custom data source that I want to upload to. And I'll show you how this process works. So I have quite a few accounts here. I want to upload to the My Store account. I will select the web property, and then here's what I'm talking about in terms of custom data sources. I have three set up, and I'm just going to choose the first one, the Ad Network custom data source. So now I've selected it. If I go back up, you'll see now that it's actually selected, here, Ad Network, and then also the account and the web property. And then the upload data again is 2012, November 28. So I will do an upload of cost data, which is number three here. I know the file that I showed you, what the name of it is. Oops. Spelled that wrong. OK, so it's actually uploading right now to Google Analytics to that customer data source, and if all goes successful, we'll get a response back from the server saying it was a success. So it takes a little bit of time to process the data. It's not going to show up instantly. I will go back to the presentation. It could take anywhere from a few minutes to up to 12 hours, so you should see data within 12 hours. Depending on the kind of load and your profile and web property, it could take a little longer. I'm going to actually go back to the presentation. We've successfully uploaded cost data. I'll come back a little bit later, and we'll see if it's available in the report and if it's been joined to our actual campaign. In the meantime, let's talk about architectural overview, how this feature works, and how you would actually go about uploading cost data. So there's quite a few steps involved, but they're all pretty simple. The first four are really around how to prepare for a successful upload. And the fifth is really around reporting and how you would do analysis, the tools that are available for that within GA. We'll start with tagging campaign URLS. So obviously, the primary benefit to this feature is that you're able to join that user interaction data that's already in GA that you have, and you're able to join that with the cost data that might sit somewhere externally. So bringing those two together is really a huge benefit here. But in order to do that, it's important that you use campaign tracking parameters, so you should be doing this anyways. But the important part of this is that the campaign tracking parameters, the values you use as part of the campaign, should or have to match the values that you're uploading through cost data upload feature. So I'll explain a little bit more in detail what this means, but basically, they have to match, and if they match, then they'll actually get joined together in that case. So in this example here, I have an ad running. It's not a Google ad. It's another paid search provider. The link there actually has UTM parameters as part of it, these campaign tracking parameters, and I've specified an ad network, CPC as a medium, the campaign is summer sale, and again the term is "dogbone." So you've seen these values throughout, and we'll kind of explain how this all works now. OK, so you've tagged your URLs. All your destination URLs for all your paid campaigns are tagged. And again, this is not for Google AdWords. Google AdWords has a different mechanism for uploading cost data. You just link your profile to your Analytics account, and it'll take care of itself. This is for non-Google paid campaigns, just something to keep in mind. But once you've tagged those URLs, the next thing you want to do is actually get the data out of the external source and prepare it as a CSV file to upload, so let's take a look at what that means. The first thing is what can you actually upload. So there's quite a few different dimensions and three metrics that you can provide values for through cost data upload. There's actually nothing new here in terms of dimensions and metrics. So these use existing AdWords dimensions and metrics that you've seen and are probably familiar with, so it's nothing new there. So if you have any reports set up, like custom reports around performance of paid campaigns, those don't have to be changed once you start uploading cost data. Any zero values that you had before will just automatically be populated with this data that you've uploaded. For the dimensions, there are two that are required, which are source and medium. And then for metrics, there's three of them, and only one is required. One of the three is required. For any ones that are omitted, we'll just assume it's a value of zero basically for those. So this is the important part here. We talked about the tagging of the URLs, and this is why it's important. This is how the data actually gets joined and mapped together. So you have all your user interaction data sitting in GA already today, and you're doing that through normal measurement that you've already done all the time, so that's taken care of. Now, you're uploading cost data. And what's going to happen is the campaign tracking parameters, the values you use there and the ones you upload, if there's a match there, then the data will get joined together. This is why it's important that these things are synced across the two. Also, there is a specific case in terms of the case that is required for these things. The keyword is case insensitive. So it doesn't matter if the keyword is capitalized or not, they'll get joined regardless. But the other four dimensions are case sensitive, so you've got to be careful there and make sure these values are consistent in that regard. Also, it's up to you really to determine the naming conventions that are used in the external system. It's quite possible that the naming convention used by this external source might be a little bit different and not as matched up with the namings that we use for our dimensions within GA. So it's kind of up to you to look at what's available from the external provider and map and see how that maps these dimensions and the values you need to provide for these dimensions. It's kind of up to you to do that. And usually, it's a one-step kind of process once you've got it setup. Then you don't have to really worry about it anymore, but identifying those mappings at first is really important. OK, so we know what we can upload. Let's take a look at actually what you would upload. So I showed you a CSV file before. Here's another example. So this is for aggregates, right? You're uploading aggregate data on a daily basis. That's usually the typical use case. So best practice then is really to wait until the end of the day and then upload cost data. Or upload cost data for the previous day because you want to upload it in an aggregate, and you upload it for a specific date. You say I'm going to upload it for yesterday or the day before and so on. Again, you'll notice here that the values I have for the source and medium match the values that I use in my URLs for my campaigns as part of the campaign tracking parameters, and those should always be consistent. And then you have your actual metrics here, the cost data itself, the compressions, ads, and clicks. In this case, I provided values for all three because I had them available. And then, of course, you saw the list of all the dimensions that are available. Those can be included also. There is quite a few different constraints and requirements around what needs to be provided and the values you can provide. These are all available on our developer site, which I provide some links later for you guys. OK, so we've tagged our campaign URLs with campaign tracking parameters. We've got our CSV file that we exported out from the external source, and we've prepared it properly. And now we have to create a data source. Remember, we have to upload to a custom data source. We have to create that, and that's actually done through the web interface. OK, so I'll show you how it actually is done right now, and then I'll explain how exactly that this works. So it's at the web property level. And we're currently doing white listing, so some of you may have this, some of you may not, but it will be all rolled out to everybody in the next week or so I think. So if you go to the web property level in the admin panel, you'll see a tab called Custom Definitions. So I already have three custom data sources available in here. I'm going to create a new one, so I'll show you how it's done. So a new custom data source, You can imagine just, say, I want to have one for, let's say, email marketing, so I'll put email marketing campaigns, specify the type. We're uplinking cost data. That's the type. And then we link your profile. Linking profiles will enable this cost data to be available for those profiles. You can imagine you might have 100 profiles here. You can select which ones you want. There's a little search box to make it a little bit easier. And then you just select which profiles you would like the cost data to show up in. Click Create, and you'll see now I have a new custom data source for email marketing campaigns. And the important thing here is there's an ID, and you actually need that ID to do uploads, and I'll show you how that's done. I'm going to go back to the presentation. So let's look at what's a custom data source, and how does it relate to Google Analytics model? I think you're probably mostly familiar with this. This is the way GA is set up today. We have Web Properties. That's where all the user interaction data is collected. That's the level it's collected at. And then we have profiles, and profiles basically act as views of that data. So you can have one maybe for marketing. You might have one for sales, or certain filters apply that you want to have different views of your web property user interaction data. What's been introduced with cost data upload is custom data sources. So this resource kind of acts as representing external data sources. You could have a single or multiple external data sources per custom data source. It's up to you. You can have multiple custom data sources on your web property. I would say, as a best practice, I would create a custom data source per external data source. It's just a little bit easier to manage, and it allows you to kind of separate data from across the different sources that you might have. It's really important that you link a profile. So this, as I showed you before, when you link profiles, it enables that data to be available within those reports. If you don't have that linking, then the cost data will not show up in reports. So in this case, we have an example here where we've created a custom data source, and it's linked to two profiles, profile one and profile three. So those two will show data, and profile two will not show any cost data from that custom data source. In addition to that, we have what we call daily upload resources. These kind of represent the actual data you've uploaded to GA on a per-date basis. So you have one for each day, a resource, and these are important. They store information around which custom data source they belong to and also metadata around recent changes that have taken place with this particular resource. So if someone uploaded to it, deleted it, you get that kind of changelog history. It's part of that resource. You manage and create custom data sources within the UI, and you actually upload cost data using an API. You can actually also look at the history of a daily upload resource within the UI. I'll show you that, by going to the history. There's a little history link here. You can kind of get it. For that custom data source, you can see all the changes that have taking place. And you can actually get-- per day, you can see all the changes that have taken place. OK, also from a creation standpoint, in deletion, you need to be an admin. So admins, anything with modification to a resource, whether it's create or delete, is admin. And viewers, just regular users, can view data, but they can't do anything around creating or deleting or uploading. So that's important. OK, so we showed you how to create a data source, so let's recap. We've got our URLs tagged with campaign tracking parameters. We've got our CSV file that's prepared and which was exported out from the external source. We've created a custom data source using the web interface. Now, we actually want to upload the cost data, and we use this API to carry this out. So here's an example in Python. It's pretty simple, actually. You create a media file upload object, and you specify the CSV file. And then you actually just call the upload method for the daily uploads resources. And you specify a few parameters that are required, and you execute that. And it'll automatically upload the file for you and respond with a 200 if successful. If there was an error, typically-- I think when people first start out with this, they get errors around the file formatting for the CSV. The errors are really descriptive, so they really help you out in terms of figuring out what the problems are. There's also a few other operations you can carry out. You can list which daily uploads. This gives you your history and changes. You can look up for a year of history. You can look at what's taken place. And we also have a delete method. So if you want to get rid of data completely, you can actually delete it out of the daily resource. And this is important because the way the cost data is stored within GA, it's actually stored separately outside of your user interaction data. So this is great because you can kind of upload-- If you? Want to delete something, it doesn't actually touch any of your user interaction data. So feel free to upload and delete without any worrying about corrupting or having any effect on your user interaction data that's already there. All right, so once you've kind of done that, and you've actually successfully uploaded cost data, and it's been processed, you'll have the data available in your reports. So going back to the start of our presentation, now, the magic's happened. Before, we had zero values for the cost data. And you can see now when you've done it properly, this actual cost data will get joined. In reports that you would normally see zeroes now show you the campaign visits are now joined and associated with that uploaded cost data, which gives you all these nice other additional metrics, calculated metrics, around margin, ROI, and things like that. So now you can do your complete ROI analysis across all of your different paid campaigns. Reporting is available a few different ways. We have the API, of course. You can continue to use the core report in API to pull metrics out. Like I said before, the dimensions and measures haven't changed, so any reports you might have been running before will work just as they did before. There's no changes to be made. And then from the web interface, there's kind of two main options, which are you can build a custom report or there's a new cost analysis report under Traffic Sources. The cost analysis report, this is what it looks like. It's actually just for cost data upload. It was created for that. And then for custom reports, let's just see if our data that I had uploaded has been processed. So let's refresh this and, yes, you can see. That was the cost data we uploaded at the start of the presentation. It has been joined to my campaign visits for that particular keyword, and we can now see impressions, clicks, and cost. And of course, you could do-- there's some calculated metrics, like cost per click, and you can add those to your custom report. So great, it worked. So some of the opportunities are, for developers, you have an opportunity to build a connector. There's kind of two steps to this, I would say. There's the initial setup, which is kind of just creating custom data source, but then there's two cases that you really got to consider. One is historical upload. You'll notice when I did the upload, I uploaded to actually yesterday. It was data for the 28th, and you can actually upload data all the way back to 2005. So if you've been tagging your campaign URLs over the last seven years, you can actually go back and export the data and upload that. It's a little bit different use case, because typically when you export, you're going to export a large chunk of data from your system, and then you have to individually upload that for each day. So that initial historic upload is a little bit different use case, but the API really allows you to automate all that. And then going forward, there's an opportunity to build a connector that pulls the data down from the external source automatically and uploads it to GA. If you don't really have the development resources or you'd just like to get started, we do have four partners that have built integrations using this feature. Next Analytics, GA Data Uploader, Analytics Canvas, and ShufflePoint all have great integrations. And depending on your requirements, I think any one of them will be able to solve a lot of different use cases. So I'd definitely encourage you to check these out in our App Gallery. So to finish up here, we have just a few resources for you. We'll include these links at the bottom of the video for you, but all of this is available on our developer site, which is developers.googl e.com/analytics. And we also have the App Gallery, which is google.com/analytics/apps. All right, thank you very much, and you can start uploading. I think if we have some questions, we'll take some questions. MALE SPEAKER: All right, so taking a quick look at the Moderator. We have a few questions. One of them is from John, who says he's played around a little bit with uploading cost data, and his campaigns don't seem to be matching up. Are the UTM parameters case sensitive? PETE FRISELLA: Yeah, I think I touched on this earlier. So the keyword is normalized to lowercase no matter what. So that one, you don't even have to worry about case. But for the other ones, you do have to worry about case sensitivity. Those will affect what gets matched up. But again, because the data's stored separately, go ahead, upload things, and you'll see if you have a problem, you simply just delete the data. It doesn't affect any of your user interaction data, so there's no problem there. MALE SPEAKER: And Andrew asks, "Are we going to be able to apply advanced segments to uploaded cost data?" PETE FRISELLA: So because you're uploading aggregate totals for a day, it's not on a session basis. So you can't actually do the advanced segments like you would. It actually acts exactly the same way that AdWords does today, which is advanced segments would not apply for this particular case. All right, so thanks very much for taking the time and joining me today about learning about cost data upload. I'll see you guys next time. [MUSIC PLAYING]
B1 data upload cost data source custom adwords Off the Charts: Getting Cost Data into Google Analytics 137 9 Chris Lyu posted on 2015/12/14 More Share Save Report Video vocabulary