Distributed Flow-Based Programming via Message Queues

MsgFlo is an implementation of the Flow-Based Programming using message queues as the communications layer between different processes. Currently supported message queue transports are AMQP and MQTT.

With MsgFlo you can build robust polyglot FBP systems spanning multiple computers/devices. A node can be implemented in any language, to reuse existing code, libraries and developer know-how.

In FBP each component is a black-box that processes and produces data, without knowledge about where the input data comes from, or where the output data goes. This ensures that a service is easy to change, and facilitates automated testing.

MsgFlo is designed to enable partial and gradual integration into existing systems; by using standard broker/transports, not placing restrictions on message payloads, allowing to use existing queue names, and integrating non-MsgFlo nodes seamlessly.

Horizontally scalable web services

A web service built using several groups of workers, each performing a set of tasks, and communicating with eachother using a messaging queue service. Some of the participants may provide HTTP REST interfaces or persistance to SQL/noSQL database, others just perform computation. Typical execution environments include Heroku, Amazon EC2, OpenStack, OpenShift. Typical messaging system used are AMQP, ZeroMQ, Amazon Simple Queue Service, Google Cloud Pubsub.

Embedded device networks — aka Internet of Things

A bigger embedded system is built using several embedded devices, each performing a set of tasks, and communicating with each other using a messing queue service (typically running on an IoT gateway). Some devices act as sensors, some as actuators and some provide computation.

Typical execution environments include Embedded Linux, microcontrollers. Typical messaging systems used are MQTT.