From 2ec55416ba95f6d67a34032bd61ed1d293b9d8b6 Mon Sep 17 00:00:00 2001 From: Alexey Rybalchenko Date: Mon, 30 Apr 2018 15:14:50 +0200 Subject: [PATCH] Update example READMEs --- examples/1-1/README.md | 10 +-- examples/1-n-1/README.md | 8 +- examples/README.md | 46 ++++++++++ examples/copypush/README.md | 6 +- examples/dds/README.md | 16 ++-- examples/multipart/README.md | 11 ++- examples/multiple-channels/README.md | 4 +- examples/multiple-transports/README.md | 8 +- .../ex-multiple-transports.json | 84 ------------------- examples/region/README.md | 5 ++ examples/req-rep/README.md | 6 +- 11 files changed, 85 insertions(+), 119 deletions(-) create mode 100644 examples/README.md delete mode 100644 examples/multiple-transports/ex-multiple-transports.json create mode 100644 examples/region/README.md diff --git a/examples/1-1/README.md b/examples/1-1/README.md index c5410e30..6dd28462 100644 --- a/examples/1-1/README.md +++ b/examples/1-1/README.md @@ -1,10 +1,10 @@ -Example 1: Sampler -> Sink -=============== +1-1: Sampler -> Sink +==================== A simple topology of two devices - **Sampler** and **Sink**. **Sampler** sends data to **Sink** via the **PUSH-PULL** pattern. -`runExample1Sampler.cxx` and `runExample1Sink.cxx` configure and run the devices in their main function. +`runSampler.cxx` and `runSink.cxx` configure and run the devices. -The executables take two required command line parameters: `--id` and `--mq-config`. The value of `--id` should be a unique identifier and the value for `--mq-config` a path to a config file. The config file for this example is `ex1-sampler-sink.json` and it contains configuration for the communication channels of the devices. The mapping between a specific device and the configuration (which can contain multiple devices) is done based on the **id**. +The executables take two command line parameters: `--id` and `--channel-config`. The value of `--id` should be a unique identifier and the value for `--channel-config` is the configuration of the communication channel. . -For this and the following example, all the commands needed to start the device are contained in the startFairMQExN.sh script (that can also be used for starting the example). +For this and the following example, all the commands needed to start the device are contained in the `fairmq-start-ex-*.sh` script (that can also be used for starting the example). diff --git a/examples/1-n-1/README.md b/examples/1-n-1/README.md index 06fc4640..860494ee 100644 --- a/examples/1-n-1/README.md +++ b/examples/1-n-1/README.md @@ -1,12 +1,12 @@ -Example 2: Sampler -> Processor -> Sink -=============== +1-n-1: Sampler -> Processor(s) -> Sink +====================================== A simple topology of three devices - **Sampler**, **Processor** and **Sink**. **Sampler** sends data to one or more **Processor**s, who modify the data and send it to one **Sink**. Transport with the **PUSH-PULL** pattern. -For this example both processor devices share same configuration, and can therefore use same setting from the JSON file. But since their ID still has to be unique, additional command line argument must be used to allow them to share configuration. This parameter is `--config-key`. +The communication channels for this example are configured via a JSON file. The matching between device and config entry is done either via id, or the config key. For this example both processor devices share same configuration, and can therefore use same setting from the JSON file. But since their ID still has to be unique, additional command line argument must be used to allow them to share configuration. This parameter is `--config-key`. In this example the Sampler is configured to **bind** its output and the Sink is configured to also **bind** its input. This allows us run any number of processors with the same configuration, because they all connect to same Sampler and Sink addresses. Furthermore, it allows adding of processors dynamically during run-time. The PUSH and PULL sockets will handle the data distribution to/from the new devices according to their distribution strategies ([Round-robin output for PUSH](http://api.zeromq.org/4-0:zmq-socket#toc14) and [Fair-queued input for PULL](http://api.zeromq.org/4-0:zmq-socket#toc15)). The Sampler sends out a simple text string (its content configurable with `--text` command line parameter, defaul is "Hello"). Each Processor modifies the string by appending its ID to it and send it to the Sink. -The provided configuration file contains two Processors. To add more Processors, you can either extend the configuration file, or create a separate file only for new processors. +The provided configuration file contains two Processors. To add more Processors, you can either extend the configuration file, or create a separate file only for new processors. Or combine JSON config with `--channel-config` from other examples. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 00000000..20c31be9 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,46 @@ +# FairMQ Examples + +Set of FairMQ examples. More examples that combine FairMQ with FairRoot can be found in the [FairRoot repository](https://github.com/FairRootGroup/FairRoot/tree/dev/examples/). + +## 1-1 + +A simple topology of two devices - **Sampler** and **Sink**. **Sampler** sends data to **Sink** with the **PUSH-PULL** pattern. + + +## 1-n-1 + +A simple topology of three device types - **Sampler**, **Processor** and **Sink**. **Sampler** sends data to one or more **Processor**s, who modify the data and send it to one **Sink**. Transport with the **PUSH-PULL** pattern. The example also shows the configuration via JSON files, as oposed to `--channel-config` that is used by other examples. + + +## DDS + +This example demonstrates usage of the Dynamic Deployment System ([DDS](http://dds.gsi.de/)) to dynamically deploy and configure a topology of devices. The topology is similar to those of Example 2, but now it can be easily distributed on different computing nodes without the need for manual reconfiguration of the devices. + + +## Copy & Push + +A topology consisting of one **Sampler** and two **Sink**s. The **Sampler** uses the `Copy` method to send the same data to both sinks with the **PUSH-PULL** pattern. In countrary to the **PUB-SUB** pattern, this ensures that all receivers are connected and no data is lost, but requires additional channels to be configured. + + +## Request & Reply + +This topology contains two devices that communicate with each other via the **REQ-REP** pettern. Bidirectional communication via a single socket. + + +## Multiple Channels + +This example demonstrates how to work with multiple channels and multiplex between them. + + +## Sending Multipart messages + +This example shows how to send a multipart message from one device to the other. (two parts message parts - header and body). + + +## Multiple Transports example + +This examples shows how to combine different channel transports (zeromq/nanomsg/shmem) inside of one device and/or topology. + +## Region example + +This example demonstrates the use of a more advanced feature - UnmanagedRegion, that can be used to create a buffer through one of FairMQ transports. The contents of this buffer are managed by the user, who can also create messages out of sub-buffers of the created buffer. Such feature can be interesting in environments that have special requirements by the hardware that writes the data, to keep the transfer efficient (e.g. shared memory). diff --git a/examples/copypush/README.md b/examples/copypush/README.md index 8ba7321d..55c0111f 100644 --- a/examples/copypush/README.md +++ b/examples/copypush/README.md @@ -1,4 +1,4 @@ -Example 4: Copy & Push -=============== +Copy & Push +=========== -A topology consisting of one **Sampler** and two **Sink**s. The **Sampler** uses the `Copy` method to send the same data to both sinks with the **PUSH-PULL** pattern. In contrary to the **PUB-PATTERN** pattern, this ensures that all receivers are connected and no data is lost, but requires additional sockets. +A topology consisting of one **Sampler** and two **Sink**s. The **Sampler** uses the `Copy` method to send the same data to both sinks with the **PUSH-PULL** pattern. In contrary to the **PUB-SUB** pattern, this ensures that all receivers are connected and no data is lost, but requires additional sockets. diff --git a/examples/dds/README.md b/examples/dds/README.md index 3ceaf99e..3ea9b7ca 100644 --- a/examples/dds/README.md +++ b/examples/dds/README.md @@ -1,12 +1,12 @@ -Example 3: DDS -=============== +DDS Example +=========== This example demonstrates usage of the Dynamic Deployment System ([DDS](http://dds.gsi.de/)) to dynamically deploy and configure a topology of devices. The topology is similar to those of Example 2, but now it can be easily distributed on different computing nodes without the need for manual socket reconfiguration of the devices. -To make use of DDS functionality the example executables need to find the DDS plugin libraries that are compiled with FairRoot when FairRoot find DDS installed. DDS location is given to CMake as follows: +To make use of DDS functionality the example executables need to find the DDS plugin libraries that are compiled with FairRoot when FairRoot find DDS installed. Custom DDS location can be given to CMake as follows: ```bash -cmake -DDDS_PATH="/path/to/dds/install/dir/" .. +cmake -DDDS_ROOT="/path/to/dds/install/dir/" .. ``` The description below outlines the minimal steps needed to run the example with DDS. For general DDS help please refer to DDS documentation on [DDS Website](http://dds.gsi.de/). @@ -40,13 +40,13 @@ If you want to deploy on a single host DDS 1.6+ provides a localhost rms plug-in ##### 3. Write DDS topology file that describes which tasks (processes) to run and their topology and configuration. -Take a look at `ex3-dds-topology.xml`. It consists of a definition part (properties, tasks, collections and more) and execution part (main). In our example Sampler, Processor and Sink tasks are defines, containing their executables and exchanged properties. The `
` of the topology uses the defined tasks. Besides one Sampler and one Sink task, a group containing Processor task is defined. The group has a multiplicity of 10, meaninig 10 Processors will be executed. Each of the Processors will receive the properties with Sampler and Sink addresses. +Take a look at `ex-dds-topology.xml`. It consists of a definition part (properties, tasks, collections and more) and execution part (main). In our example Sampler, Processor and Sink tasks are defines, containing their executables and exchanged properties. The `
` of the topology uses the defined tasks. Besides one Sampler and one Sink task, a group containing Processor task is defined. The group has a multiplicity of 10, meaninig 10 Processors will be executed. Each of the Processors will receive the properties with Sampler and Sink addresses. The configuration of the channel connection addresses is done by the DDS plugin via the channel names. The task property names must correspond to the channel names (data1, data2), with binding channels writing the properties and connecting channel reading the properties (see the example XML and JSON files). If `eth0` network interface (default for binding) is not available on your system, specify another one in the topology file for each task. For example: `--network-interface lo0`. -If you chose step 2b earlier, then modify the provided `ex3-dds-topology.xml` in the top that the following lines read as following: +If you chose step 2b earlier, then modify the provided `ex-dds-topology.xml` in the top that the following lines read as following: ```xml @@ -67,7 +67,7 @@ dds-server start -s Agents are submitted with: ```bash -dds-submit --rms ssh --config ex3-dds-hosts.cfg +dds-submit --rms ssh --config ex-dds-hosts.cfg ``` The `--rms` option defines a destination resource management system. The `--config` specifies an SSH plug-in resource definition file. @@ -80,7 +80,7 @@ dds-submit --rms localhost -n 12 ##### 6. Activate the topology. ```bash -dds-topology --activate ex3-dds-topology.xml +dds-topology --activate ex-dds-topology.xml ``` ##### 7. Run diff --git a/examples/multipart/README.md b/examples/multipart/README.md index 841cfcd9..fb458836 100644 --- a/examples/multipart/README.md +++ b/examples/multipart/README.md @@ -1,13 +1,12 @@ -Example 8: Sending Multipart messages -=============== +Sending Multipart messages +========================== A topology of two devices - Sampler and Sink, communicating with PUSH-PULL pattern. The Sampler sends a multipart message to the Sink, consisting of two message parts - header and body. -Each message part is a regular FairMQMessage. To combine them into a multi-part message, simply send all but the last part with `SendPart()` and the last part with `Send()` as shown in the example. +Each message part is a regular FairMQMessage. To combine them into a multi-part message use `FairMQParts`. Add messages to `FairMQParts` with `AddPart` method. -The ZeroMQ transport guarantees delivery of both parts together. Meaning that when the Receive call of the Sink receives the first part, following parts have arrived too. - -The header contains a simple data structure with one integer. The integer in this structure is used as a stop flag for the sink. As long as its value is 0, the Sink will keep processing the data. Once its value is 1, the Sink will exit its `Run()` method. +All parts are guaranteed to be delivered together. The Receive call in the sink will recive the entire parts structure. +The header contains a simple data structure with one integer. The integer in this structure is used as a stop flag for the sink. As long as its value is 0, the Sink will keep processing the data. Once its value is 1, the data handler method of the Sink will return false, which will exit the RUNNING state of the device. diff --git a/examples/multiple-channels/README.md b/examples/multiple-channels/README.md index ab3a8173..f9c5d162 100644 --- a/examples/multiple-channels/README.md +++ b/examples/multiple-channels/README.md @@ -1,5 +1,5 @@ -Example 6: Multiple Channels -=============== +Multiple Channels +================= This example demonstrates how to work with multiple channels and multiplex between them. diff --git a/examples/multiple-transports/README.md b/examples/multiple-transports/README.md index ff33996f..490a7ba9 100644 --- a/examples/multiple-transports/README.md +++ b/examples/multiple-transports/README.md @@ -3,10 +3,10 @@ Multiple Transports example This example demonstrates use of multiple transports (zeromq/nanomsg/shmem) within the same topology and/or device. It is a simple topology consisting of two samplers and a sink. The devices are connected via 3 channels: -![Multiple Transports example](../../../docs/images/fairmq-ex-multiple-transports.png?raw=true "Multiple Transports example") +![Multiple Transports example](../../docs/images/example_multiple_transports.png?raw=true "Multiple Transports example") -Each device has main transport that it uses. By default it is ZeroMQ, and can be overriden via the `--transport` cmd option. The device will initialize additional transports if any of the channels have them configured (e.g. via the JSON file, see `ex-multiple-transports.json`). +Each device has main transport that it uses. By default it is ZeroMQ, and can be overriden via the `--transport` cmd option. The device will initialize additional transports if any of the channels have them configured (e.g. as an option to `--channel-config`). -In this example sampler1 and sink are started with `--transport shmem`, making shared memory their main transport, sampler2 with `--transport nanomsg`. Additionally, the ack channel is configured to use zeromq as its transport via the JSON configuration. +In this example sampler1 and sink are started with `--transport shmem`, making shared memory their main transport, sampler2 with `--transport nanomsg`. Additionally, the ack channel is configured to use zeromq as its transport. -The main two things that a transport does is transfer of data and allocation of memory for the messages. By default, new messages are created via the main device transport. If a message has been created with one transport and is to be transferred with another, it has to be copied into a new message of the target transport. This happens automatically behind the scenes. To avoid this copy the device can create messages via `NewMessageFor(const string& channelName, int subChannelIndex, ...)` method, that creates the messages via the transport of the given channel (check sampler1 and sink for an example). +The main two things that a transport does is transfer of data and allocation of memory for the messages. By default, new messages are created via the main device transport. If a message has been created with one transport and is to be transferred with another, it has to be copied into a new message of the target transport. This happens automatically behind the scenes. To avoid this copy the device can create messages via `NewMessageFor(const string& channelName, int subChannelIndex, ...)` method, that creates the messages via the transport of the given channel (check sampler1 and sink for an example) or as the channel directly to create a message. diff --git a/examples/multiple-transports/ex-multiple-transports.json b/examples/multiple-transports/ex-multiple-transports.json deleted file mode 100644 index b030dec3..00000000 --- a/examples/multiple-transports/ex-multiple-transports.json +++ /dev/null @@ -1,84 +0,0 @@ -{ - "fairMQOptions": { - "devices": [ - { - "id": "sampler1", - "channels": [ - { - "name": "data1", - "sockets": [ - { - "type": "push", - "method": "bind", - "address": "tcp://127.0.0.1:5555" - } - ] - }, - { - "name": "ack", - "transport": "zeromq", - "sockets": [ - { - "type": "sub", - "method": "bind", - "address": "tcp://127.0.0.1:5557" - } - ] - } - ] - }, - { - "id": "sampler2", - "channels": [ - { - "name": "data2", - "sockets": [ - { - "type": "push", - "method": "bind", - "address": "tcp://127.0.0.1:5556" - } - ] - } - ] - }, - { - "id": "sink1", - "channels": [ - { - "name": "data1", - "sockets": [ - { - "type": "pull", - "method": "connect", - "address": "tcp://127.0.0.1:5555" - } - ] - }, - { - "name": "data2", - "transport": "nanomsg", - "sockets": [ - { - "type": "pull", - "method": "connect", - "address": "tcp://127.0.0.1:5556" - } - ] - }, - { - "name": "ack", - "transport": "zeromq", - "sockets": [ - { - "type": "pub", - "method": "connect", - "address": "tcp://127.0.0.1:5557" - } - ] - } - ] - } - ] - } -} diff --git a/examples/region/README.md b/examples/region/README.md new file mode 100644 index 00000000..7be16bf3 --- /dev/null +++ b/examples/region/README.md @@ -0,0 +1,5 @@ +Region example +============== + +This example demonstrates the use of a more advanced feature - UnmanagedRegion, that can be used to create a buffer through one of FairMQ transports. The contents of this buffer are managed by the user, who can also create messages out of sub-buffers of the created buffer. Such feature can be interesting in environments that have special requirements by the hardware that writes the data, to keep the transfer efficient (e.g. shared memory). + diff --git a/examples/req-rep/README.md b/examples/req-rep/README.md index 783e892c..5211aeb4 100644 --- a/examples/req-rep/README.md +++ b/examples/req-rep/README.md @@ -1,4 +1,4 @@ -Example 5: Request & Reply -=============== +Request & Reply example +======================= -This topology contains two devices that communicate with each other via the **REQUEST-REPLY** pettern. Bidirectional communication via a single socket. +This topology contains two devices that communicate with each other via the **REQ-REP** pettern. Bidirectional communication via a single socket.