Today we are launching Dagster Plus, the next evolution of Dagster Cloud.

Dagster Plus is a major rethinking of what a data orchestrator is, where it sits in the stack, and how data platforms are constructed.

Since our launch in August of 2022, we've seen enormous growth in both the Dagster Core open source project and our commercial product.

We now have hundreds of commercial customers and the fastest growing open source community in the category.

With this growth comes broader and deeper relationships with our diverse set of customers.

As we engaged with these customers throughout 2022 and 2023, we heard a few key challenges.

They were spending too much money on infrastructure, especially on the data warehouse, ETL tools, and AI workloads, and struggled to understand, optimize, and control their spend.

Their stacks were sprawling and complex with hundreds of different integrations and point solutions and varying levels of maintenance, and no single pane of glass for understanding the data platform as a whole.

Inspired by the data mesh architecture, they wanted to empower individual teams to work autonomously end-to-end, but struggled to balance this with the centralized needs of the data organization.

We realized that we could solve many of these problems for customers by making a significant investment in tooling that sits on top of the core data orchestration engine.

So we spent the last 12 months spinning up a team dedicated to these problems, building furiously, and iterating with design partners.

I'm proud of what the team has accomplished, and I'm excited to present the fruits of our labor to you today.

So let's dig into what Dagster Plus is specifically and why Dagster Labs is uniquely positioned to deliver this type of product.

First and foremost, Dagster Plus is the next evolution of Dagster Cloud, our enterprise-class hosted product.

This means it includes everything that was part of Dagster Cloud, including enterprise access control, seamless scale up to tens of thousands of assets and hundreds of thousands of partitions, and serverless or hybrid on-prem managed infrastructure.

This also means that we're retiring the Dagster Cloud brand today.

Second, Dagster Plus is a suite of tools that sits on top of Dagster's data orchestration engine and benefits from this deep integration.

Today we'll be focused on four primary areas of the product.

We've built a brand new data catalog and data discovery capability in Dagster Plus, which enables all users of the data platform to discover and leverage data autonomously.

Dagster Plus builds on Dagster Core's asset checks feature, introducing additional capabilities around anomaly detection, freshness, and alerting.

We've made massive improvements to our branch deployments feature from Dagster Cloud and are launching a killer new capability, change tracking, with Dagster Plus.

And finally, Dagster Insights enables data teams to manage their spend on tools like Snowflake, Fivetran, and OpenAI without doing any additional integration work.

With Dagster Plus, Insights is now generally available and comes with many new features and integrations.

The upcoming presentations will dive into each of these in detail, but before we do, I wanted to touch on why we, rather than our competitors, are uniquely positioned to address these challenges.

Dagster has historically been considered a data orchestrator.

All of Dagster's main alternatives, such as Apache Airflow, proudly declare themselves as workflow-oriented tools.

They are primarily focused on the scheduling of black box tasks.

While this makes these tools quite flexible and suitable for a variety of non-data applications, they are not specialized to the data domain.

In fact, we believe that there is an impedance mismatch between the workflow-oriented tools like Airflow, which deals primarily with tasks, and the rest of the data platform, which deals primarily with data assets, like database tables, machine learning models, and files.

Dagster takes a different approach.

Dagster is the only widely adopted asset-oriented data orchestrator, built from the ground up with data assets and data awareness at its core.

This unique architecture enables Dagster Plus to deliver a category-redefining, integrated experience, bringing together unique capabilities into one tool in order to speed up your development work, reduce your costs, and simplify your data platform.

I'll get off my soapbox now.

Let's hear from Jared, who will tell us about the data catalog.

Thanks, Pete.

I'm Jared, the head of product at Dagster Labs.

Today we're announcing a brand new data catalog as part of Dagster Plus, but before we get into that, let's talk about the unfulfilled promise of data catalogs.

As your organization scales, the number of assets you have to manage also scales.

It doesn't take long before you end up with a large number of rarely used data models with poor documentation and questionable provenance.

The central premise of data catalogs is to resolve all this by helping teams avoid rework and duplication, and help data practitioners work more independently by enabling them to discover and use trusted data assets.

Unfortunately, because standalone data catalogs don't have a native understanding of data operations, they have to create observations about the state of data assets from the exhaust of the loosely integrated tools that comprise the data platform.

As a result, data engineering team ends up constantly troubleshooting problems of accuracy and timeliness in the syncing the state of their data assets to the data catalog.

All the while, they are trying to coax the rest of their stakeholders to follow a new set of development practices to ensure that it remains in sync.

Typically, the end result is a lack of adoption from the very downstream stakeholders that the data team was trying to help.

Unlike traditional workflow orchestrators, Dagster is asset oriented.

It places data assets at the core of the development framework.

The result is a powerful combination of context about data operations and the outputs of those operations.

This view of data assets and operations makes Dagster a natural tool for data discovery and documentation, creating a data catalog that serves the needs of all practitioners.

With the launch of Dagster+, teams who want to spend less time managing integrations for their tools and more time getting things done will find it particularly useful as a new system of record.

And while we're still early in our journey, we are already seeing that, for many teams, this eliminates the need for traditional point solution data catalogs.

The first major set of improvements focuses on the kinds of data that Dagster collects and displays at the asset definition level.

On the asset details page, you'll now be able to find a new variety of information about your data asset that helps you understand the status, technical state, definition, ownership, location, and more all at a glance.

You can see things like the latest status of your asset, including the data quality testing configured on the asset, its description, who owns the asset, what groups and tags are associated with it, as well as where the asset is located, its compute details, and the associated resources and compute kind.

And for structured assets, you'll now find a new rich set of details, including any raw SQL that defines the asset structure, metadata about column names, descriptions, and data types.

And an exciting new change that we're launching for enterprise customers is the ability to explore the lineage of individual columns on the page.

When automatically ingested from DBT or derived from user-provided metadata, Dagster can compute the relationship of individual columns and represent them on the page similar to our asset lineage graph.

Column lineage provides a clear understanding of how data moves and transforms within a system, allowing you to track things like the proliferation of PII or tracing a mistake through its various downstream tables.

This new feature will help you manage data quality, compliance, and increase the speed of your troubleshooting.

But our feature release doesn't stop with the asset data model and UI.

We're also releasing a new, more powerful search experience to help you quickly locate the most relevant data.

From this interface, you can search for data assets by metadata, including their name, their compute kind, the asset group, the asset owner, or by any associated tag that you add to your assets, and see information about all of your data assets by their associated metadata.

I want to call out a feature here that might get passed over, which is definition-level tags.

With definition-level tagging, you can more easily add cross-sectional data to organize and group your data assets.

So now you can tag all of your intermediary assets, or your assets with PII, or the assets that support your finance team, and make these assets easily discoverable by their relevant stakeholders.

Dagster's data catalog experience is powerful because of how it combines context about data pipelines and the data assets they produce.

But we know that not everyone needs to look under the hood inside Dagster.

Sometimes your data practitioners just need to know where to find the right asset.

And that is why we're also launching a new capability we're calling Catalog Mode, which allows you to hide most of the operational views of Dagster, and instead focus on the assets that Dagster has documented.

This new feature can be turned on by default for your viewers in your organization, or turned off when they need to go deeper.

So that's our high-level introduction to the brand new data catalog and search features in Dagster+.

We expect this addition will make it easier for all teams on the data platform to find information about their data assets and status, and to operate more autonomously, while reducing the operational burden on the platform team.

You might find it helps eliminate the need for standalone tools which will reduce context switching, help break down data silos, and reduce the total cost of ownership of your data platform.

We know that data cataloging is an important capability for teams, and we're excited to be rolling out this first version, but we're not stopping here.

Sooner Catalog will also consume data about assets that are not managed by Dagster, and represent them in the data catalog as external assets.

I hope this first overview has you interested in the new data cataloging capabilities in Dagster+, but check out the companion blog to find out more.

I'm Sandy, the lead engineer on the Dagster project.

Today I'm going to talk about the new data reliability features that we're launching as part of Dagster+, and how they help you deliver trustworthy data to stakeholders across your entire data platform.

The job of a data orchestrator is to help you produce and maintain the data assets that your organization depends on.

Whether those assets are tables, machine learning models, or reports, in order to be useful they need to be trustworthy and reliable.

That is, they need to consistently contain up-to-date and high-quality data.

But bugs are inevitable, and pipeline developers don't control the data that enters their pipelines.

So the only realistic way to achieve trustworthy data is with monitoring and issue detection, so that issues can be addressed before they affect downstream consumers.

Most data teams struggle with data reliability because they don't have robust monitoring.

Workflow-oriented orchestrators like Airflow will report if your data pipeline hits an error during execution, but don't offer visibility into the quality, completeness, or freshness of the data that they're updating.

Teams that adopt stand-alone data reliability tools often end up abandoning them, because they're difficult to fit into practices for operating data.

Data engineers end up needing to visit both their orchestrator and their reliability tool to understand the health of their data pipeline.

And if they want to trace data reliability alerts to the DAGs that generated the data, they need to try to integrate their data reliability tool with their orchestrator.

This usually means contending with query-tagging schemes that are fragile to communicate with or configure and maintain.

At Dagster Labs, we believe orchestration and data reliability go hand-in-hand.

Dagster's asset-oriented approach to data orchestration enables Dagster Plus to offer a full set of data reliability features.

These help you monitor the freshness of your data, the quality of your data, and changes to the schema of your data.

Dagster helps monitor both the source data that feeds your pipelines and the data produced by your pipelines.

Monitoring source data helps you find out early if your pipeline is going to run on stale, bad, or unexpected data.

And monitoring output data helps you ensure that the final product is on time and high-quality.

At the center of Dagster's data reliability stack is a feature called asset checks.

An asset check is responsible for determining whether a data asset meets some property, such as whether it contains the expected set of columns, contains no duplicate records, or is sufficiently fresh.

Asset checks can be executed in line with data pipelines or scheduled to execute independently.

Optionally, Dagster can halt your data pipeline when an asset check fails to avoid propagating bad data.

Asset checks were introduced as an experimental feature last year, and we've marked them as generally available in Dagster's recent 1.7 release.

Asset checks sit on top of Dagster's rich metadata system, which can be used to store any metadata, from row counts to timestamps to table column schema.

With Dagster+, asset checks can be used as a basis for alerting and reporting.

When an asset check fails, Dagster can notify the asset's owner.

And Dagster Plus Insights allows understanding the results of asset checks in aggregate.

For example, did we violate our asset freshness guarantees more times this week than last week?

Because Dagster makes it easy to see the checks that are defined for any asset, asset checks are also helpful for describing and enforcing data contracts.

For organizations following data mesh approaches, teams can use asset checks to communicate invariants about the data products that they expose to the rest of the organization.

Asset checks are already widely used in the Dagster community for enforcing data quality, often by wrapping DBT tests.

In our latest release, we've expanded the utility of asset checks beyond data quality to also cover data freshness and data schema changes.

So let's talk about those.

Data freshness means tracking when data assets are updated and identifying when one of them is overdue for an update.

Dagster's asset orientation and metadata system make it straightforward to track when data assets are updated.

And once you're tracking data updates, you can set up freshness checks to identify when your data is overdue.

Freshness checks can either be based on rules or based on anomaly detection.

For rules-based freshness checks, you set limits on how out of date it's acceptable for your asset to be, for example, requiring an update every six hours.

For anomaly detection freshness checks, Dagster Plus looks at the history of updates to your asset to determine whether recent behavior is consistent.

This is especially useful when you have many assets to monitor and don't want to figure out the specific rule that applies to each one.

As with all asset checks, you can get alerts when freshness checks fail, so you can be notified about freshness issues before they affect your stakeholders.

In addition to catching data freshness issues, Dagster can help catch changes in data schema.