Our Story

Oleander is the culmination of years of experience building data observability and metadata management tools. Marquez emerged out of WeWork where there was a solution needed to maintain the provenance of how datasets were consumed and produced within WeWork’s data platform, provide visibility into job runtimes, track the frequency of dataset access, centralize dataset lifecycle management, and define clear job dependencies. We wanted to answer questions like:

• What effects (if any) would job A have on job B if the consumption of dataset D by job A was delayed?
• What impact would updating the schema for dataset D have on job B?

We didn’t know it at the time, but these requirements would eventually influence and inspire the introduction of OpenLineage.

OpenLineage

At the beginning, there was some friction between ingesting lineage metadata and getting a useful operational dashboard since it required an understanding of the Marquez highly opinionated data model. We knew standardizing on an open format for lineage would require a community effort, as the problem was a shared experience and one that benefited from collaboration. Ask any data engineering team how they collect and store lineage metadata within their organization, you’ll likely get different answers.

So, we set out to simplify Data Observability backed by an open standard lead by Julien Le Dem. This open standard became OpenLineage, a framework-agnostic specification for collecting data lineage, with Marquez serving as the implementation of how to receive, process and query OpenLineage events.

Oleander

With the launch of oleander, we are committed to contributing towards fostering a healthy and vibrant community around OpenLineage. We will be creating educational resources and documentation around the technology to facilitate adoption. More importantly, we’ll be reimagining how users interact with lineage metadata. In addition, we offer consulting services to help organizations adopt OpenLineage in their data observability stack.

Please reach out to us on our contact form if you have any questions or want to leverage our expertise in building out your observability stack powered by OpenLineage.