Zed offers a new approach to data that makes it easier to manipulate and manage your data.
With Zed’s new super-structured data model, messy JSON data can easily be given the fully-typed precision of relational tables without giving up JSON’s uncanny ability to represent eclectic data.
Trying out Zed is easy: install the command-line tool zq.
zq is a lot like jq but is built from the ground up as a search and analytics engine based on the Zed data model. Since Zed data is a proper superset of JSON, zq also works natively with JSON.
While zq and the Zed data formats are production quality, the Zed project’s Zed data lake is a bit earlier in development.
For a non-technical user, Zed is as easy to use as web search. In contrast, Zed exposes its technical underpinnings in a gradual slope for a technical user, providing as much detail as desired, packaged up in the easy-to-understand ZSON data format and Zed language.
Why a Data lake implemented with Golang?
We think data is hard and it should be much, much more manageable.
While schemas are a great way to model and organize your data, they often get in the way when you are just trying to store or transmit your semi-structured data.
Also, why should you have to set up one system for search and another completely different system for historical analytics? And the same unified search/analytics system that works at cloud scale should run efficiently as a lightweight command-line tool on your laptop.
And rather than having to set up complex ETL pipelines with brittle transformation logic, managing your data lake should be as easy as git.
Finally, we believe a lightweight data store that provides easy search and analytics would be a great place to store data sets for data science and data engineering experiments running in Python and providing easy integration with your favorite Python libraries.
How does it work?
Zed solves all these problems with a new foundational data format called ZSON, a superset of JSON and relational models. ZSON is syntax-compatible with JSON but it has a comprehensive type system that you can use as little or as much as you like. Zed types can be used as schemas.
The Zed language offers a gentle learning curve, which spans the gamut from simple keyword search to powerful data-transformation operators like lateral sub-queries and shaping.
Zed also has a cloud-based object design that was modeled after the git design pattern. Commits to the lake are transactional and consistent. Search index updates are also transactionally consistent with any ingested data, and searches can run with or without indexes.
Quick Start
Check out the installation page for a quick and easy install.
Detailed documentation for the entire Zed system and language is available on the Zed docs site.
