A Golang based structured programming and distributed processing platform that enables highly concurrent, scalable, and maintainable workflows for Machine Learning and Data Processing. It is a fabric that connects disparate computation backends using a type-safe data dependency graph. It records all changes to a pipeline, making it possible to rewind time. It also stores a history of all executions and provides an intuitive UI, CLI, and REST/gRPC API to interact with the computation.

Flyte is more than a workflow engine — it uses workflow as a core concept, and task (a single unit of execution) as a top-level concept. Multiple tasks arranged in a data producer-consumer order create a workflow.

Workflows and Tasks can be written in any language, with out-of-the-box support for Golang, Python, Java, and Scala. Flyte was designed to manage the complexity of Data and ML teams and ensure they keep up their high velocity of delivering business-impacting features. One way it achieves this is by separating the control plane from the user plane. Thus, every organization can offer Flyte as a service to end-users where more infrastructure-focused folks manage the service. At the same time, the users use the intuitive interface of Flytekit.

Why using the Golang based Flyte

• Kubernetes-native workflow automation platform
• Data-aware and strongly-typed
• Ergonomic SDKs in Python, Java, and Scala
• Versioned, auditable, and reproducible pipelines
• Resource-aware and deployments at scale