Fast event-loop networking for #golang

Evio is an event loop networking framework that is fast and small. It makes direct epoll and kqueue syscalls rather than using the standard Go net package, and works in a similar manner as libuv and libevent.

https://github.com/tidwall/evio

The goal of this project is to create a server framework for Go that performs on par with Redis and Haproxy for packet handling. My hope is to use this as a foundation for Tile38 and a future L7 proxy for Go… and a bunch of other stuff.

Just to be perfectly clear

This project is not intended to be a general purpose replacement for the standard Go net package or goroutines. It’s for building specialized services such as key value stores, L7 proxies, static websites, etc.

You would not want to use this framework if you need to handle long-running requests (milliseconds or more). For example, a web api that needs to connect to a mongo database, authenticate, and respond; just use the Go net/http package instead.

There are many popular event loop based applications in the wild such as Nginx, Haproxy, Redis, and Memcached. All of these are single-threaded and very fast and written in C.

The reason I wrote this framework is so I can build certain network services that perform like the C apps above, but I also want to continue to work in Go.

Features

  • Fast single-threaded event loop
  • Simple API
  • Low memory usage
  • Supports tcp4, tcp6, and unix sockets
  • Allows multiple network binding on the same event loop
  • Flexible ticker event
  • Fallback for non-epoll/kqueue operating systems by simulating events with the net package
  • Ability to wake up connections from long running background operations

https://github.com/tidwall/evio

Share This:

An alternative server framework for #golang that uses I/O multiplexing

Shiny is an alternative server framework for Go that uses I/O multiplexing. It makes direct epoll and kqueue syscalls rather than the standard Go net package.

https://github.com/tidwall/shiny

It uses the Reactor pattern where the server waits for the OS to signal a readiness event. This is similar to the way that libuvlibeventhaproxynginxredis, and other high performance servers work.

The goal of this project is to create a simple server framework for Go that performs on par with Redis and Haproxy for packet handling, but without having to interop with Cgo. My hope is to use this as a foundation for Tile38 and other projects. Early benchmarks are exceeding my expectations.

This project is a work in progress. The API will likely change between now and Tile38 v2.0 release.

Features

  • Simple API. Only one entrypoint and four event functions
  • Low memory usage
  • Very fast single-threaded support
  • Support for non-epoll/kqueue operating systems by simulating events with the net package.

Share This:

#golang RESTful API Boilerplate with JWT Authentication backed by PostgreSQL

Go Restful API Boilerplate

Easily extendible RESTful API boilerplate aiming to follow idiomatic go and best practice.

https://github.com/dhax/go-base

Features

  • Configuration using viper
  • CLI features using cobra
  • dep for dependency management
  • PostgreSQL support including migrations using go-pg
  • Structured logging with Logrus
  • Routing with chi router and middleware
  • JWT Authentication using jwt-go in combination with passwordless email authentication (could be easily extended to use passwords instead)
  • Request data validation using ozzo-validation
  • HTML emails with gomail

Share This:

#golang tool to modify struct field tags

Golang tool to modify/update field tags in structs. gomodifytags makes it easy to update, add or delete the tags in a struct field. You can easily add new tags, update existing tags (such as appending a new key, i.e: dbxml, etc..) or remove existing tags. It also allows you to add and remove tag options. It’s intended to be used by an editor, but also has modes to run it from the terminal. Read the usage section below for more information.

https://github.com/fatih/gomodifytags

Share This:

#golang implementation of Data At Rest Encryption

Go implementation of the Data At Rest Encryption (DARE) format.

https://github.com/minio/sio

It is a common problem to store data securely – especially on untrusted remote storage. One solution to this problem is cryptography. Before data is stored it is encrypted to ensure that the data is confidential. Unfortunately encrypting data is not enough to prevent more sophisticated attacks. Anyone who has access to the stored data can try to manipulate the data – even if the data is encrypted.

To prevent these kinds of attacks the data must be encrypted in a tamper-resistant way. This means an attacker should not be able to:

  • Read the stored data – this is achieved by modern encryption algorithms.
  • Modify the data by changing parts of the encrypted data.
  • Rearrange or reorder parts of the encrypted data.

Authenticated encryption schemes (AE) – like AES-GCM or ChaCha20-Poly1305 – encrypt and authenticate data. Any modification to the encrypted data (ciphertext) is detected while decrypting the data. But even an AE scheme alone is not sufficiently enough to prevent all kinds of data manipulation.

All modern AE schemes produce an authentication tag which is verified after the ciphertext is decrypted. If a large amount of data is decrypted it is not always possible to buffer all decrypted data until the authentication tag is verified. Returning unauthenticated data has the same issues like encrypting data without authentication.

Splitting the data into small chunks fixes the problem of deferred authentication checks but introduces a new one. The chunks can be reordered – e.g. exchanging chunk 1 and 2 – because every chunk is encrypted separately. Therefore the order of the chunks must be encoded somehow into the chunks itself to be able to detect rearranging any number of chunks.

This project specifies a format for en/decrypting an arbitrary data stream and gives some recommendations about how to use and implement data at rest encryption (DARE). Additionall, this project provides a reference implementation in Go.

Share This:

A @golang object document mapper (ODM) for #MongoDB

The mongodm package is an object document mapper (ODM) for mongodb written in Go which uses the official mgo adapter.

https://github.com/zebresel-com/mongodm

Features

  • 1:1, 1:n struct relation mapping and embedding
  • call Save(),Update()Delete() and Populate() directly on document instances
  • call Select()Sort()Limit()Skip() and Populate() directly on querys
  • validation (default and custom with regular expressions) followed by translated error list (customizable)
  • population instruction possible before and after querys
  • Find()FindOne() and FindID()
  • default handling for IDCreatedAtUpdatedAt and Deleted attribute
  • extends *mgo.Collection
  • default localisation (fallback if none specified)
  • database authentication (user and password)
  • multiple database hosts on connection

 

Share This:

Jaeger is a Distributed Tracing System in #golang

Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system released as open source by Uber Technologies and written in Golang .

https://github.com/uber/jaeger

It can be used for monitoring microservice-based architectures:

  • Distributed context propagation
  • Distributed transaction monitoring
  • Root cause analysis
  • Service dependency analysis
  • Performance / latency optimization

Share This:

jwtauth – JWT authentication middleware for #golang HTTP services

https://github.com/go-chi/jwtauth

The jwtauth http middleware package provides a simple way to verify a JWT token from a http request and send the result down the request context (context.Context).

Please note, jwtauth works with any Go http router, but resides under the go-chi group for maintenance and organization – its only 3rd party dependency is the underlying jwt library “github.com/dgrijalva/jwt-go”.

This package uses the new context package in Go 1.7 stdlib and net/http#Request.Context to pass values between handler chains.

In a complete JWT-authentication flow, you’ll first capture the token from a http request, decode it, verify it and then validate that its correctly signed and hasn’t expired – the jwtauth.Verifier middleware handler takes care of all of that. Thejwtauth.Verifier will set the context values on keys jwtauth.TokenCtxKey and jwtauth.ErrorCtxKey.

Next, it’s up to an authentication handler to respond or continue processing after the jwtauth.Verifier. The jwtauth.Authenticator middleware responds with a 401 Unauthorized plain-text payload for all unverified tokens and passes the good ones through. You can also copy the Authenticator and customize it to handle invalid tokens to better fit your flow (ie. with a JSON error response body).

The Verifier will search for a JWT token in a http request, in the order:

  1. ‘jwt’ URI query parameter
  2. ‘Authorization: BEARER T’ request header
  3. Cookie ‘jwt’ value
  4. (optional), use jwtauth.Verify("state") for additional query/cookie parameter aliases

The first JWT string that is found as a query parameter, authorization header or cookie header is then decoded by the jwt-go library and a *jwt.Token object is set on the request context. In the case of a signature decoding error the Verifier will also set the error on the request context.

The Verifier always calls the next http handler in sequence, which can either be the generic jwtauth.Authenticatormiddleware or your own custom handler which checks the request context jwt token and error to prepare a custom http response.

Share This:

The Event Gateway combines both API Gateway and Pub/Sub functionality into a single event-driven experience. #golang

https://github.com/serverless/event-gateway

The Event Gateway combines both API Gateway and Pub/Sub functionality into a single event-driven experience. It’s dataflow for event-driven, serverless architectures. It routes Events (data) to Functions (serverless compute). Everything it cares about is an event! Even calling a function. It makes it easy to share events across different systems, teams and organizations!

Use the Event Gateway right now, by running the Event Gateway Example Application locally, with the Serverless Framework.

Features:

  • Platform agnostic – All your cloud services are now compatible with one another: share cross-cloud functions and events with AWS Lambda, Microsoft Azure, IBM OpenWhisk and Google Cloud Platform.
  • Send events from any cloud – Data streams in your application become events. Centralize events from any cloud provider to get a bird’s eye view of all the data flowing through your cloud.
  • React to cross-cloud events – You aren’t locked in to events and functions being on the same provider: Any event, on any cloud, can trigger any function. Set events and functions up like dominoes and watch them fall.
  • Expose events to your team – Share events and functions to other parts of the application. Your teammates can find them and utilize them in their own services.
  • Extendable through middleware – Perform data transforms, authorizations, serializations, and other custom computes straight from the Event Gateway.

The Event Gateway is a L7 proxy and realtime dataflow engine, intended for use with Functions-as-a-Service on AWS, Azure, Google & IBM.

Share This:

Smartcrop finds good image crops for arbitrary crop sizes #golang

Ssmartcrop finds good image crops for arbitrary sizes. It is a pure Go implementation, based on Jonas Wagner’s smartcrop.js

https://github.com/muesli/smartcrop

 

Share This: