Defined in a simple YAML format.

Go Project Highlights

• Schedule executions of DAGs with Cron expressions
• Install by placing just a single binary file
• Define dependencies between related jobs and represent them as a single DAG (unit of execution)

Motivation

Our ETL pipeline had many problems with the author’s projects. There were hundreds of cron jobs on the server’s crontab, and it is impossible to keep track of those dependencies between them. If one job failed, we were not sure which to rerun. They also have to SSH into the server to see the logs and run each shell script. So they needed a tool that could explicitly visualize and manage the dependencies of the pipeline. It would be nice to see each job’s job dependencies, execution status, and logs in a Web UI and to rerun or stop a series of jobs with just a mouse click!

Why not existing workflow schedulers, such as Airflow?

There are existing tools such as Airflow, Prefect, Temporal, etc, but in most cases, they require writing code in a programming language such as Python to define DAGs. In systems operating for a long time, complex jobs are written in hundreds of thousands of lines of code in other languages such as Perl or Shell Scripts. There is concern that adding another later Python code will further decrease maintainability. So we developed Dagu, which requires no coding and is easy to use and self-contained, making it ideal for smaller projects with fewer people.

How does it work?

Dagu is a single command that uses the local file system to store data. Therefore, no DBMS or cloud service is required. Dagu executes DAGs defined in declarative YAML format. Existing programs can be used without any modification.