Commit Graph

77 Commits

Author SHA1 Message Date
Alexey Rybalchenko
5aaf27bf02 Refactor the transport interface
- give transport Initialize() method with access to device config.
 - avoid using global context in the transport.
 - simplify shutdown procedure (no need for extra thread).
2017-04-20 11:07:49 +02:00
Dennis Klein
d7eb692951 remove no longer needed special treatment for logger directory 2017-04-20 11:07:49 +02:00
Dennis Klein
9b337ca075 remove redundant include path
The imported ZeroMQ target has a target property
INTERFACE_INCLUDE_DIRECTORIES which carries this
information.
2017-04-20 11:07:49 +02:00
Dennis Klein
d02ae9be2c remove no longer used variable 2017-04-20 11:07:48 +02:00
Giulio Eulisse
1ebbe80539 Add ZeroMQ include path 2017-04-20 11:07:48 +02:00
Dennis Klein
c41cf318bd export FairMQ targets and install headers hierarchically 2017-04-20 11:07:47 +02:00
Dennis Klein
35b3212f7e cmake cleanups and fixes
* fix overlinking of libFairMQ
* import dependencies in find modules
* provide FairMQinstall target
* add gtest helper module
* add cotire helper module
* add many comments
* cleanup fairmq cmake file
* add FindPStreams module
* link logger into libFairMQ
* drop root dictionary generation
* bump min required cmake version to 3.7.2
2017-04-20 11:07:47 +02:00
Matthias Richter
e184610c06 Adding parser implementation for key-value subopt format
A parser implementation for FairMQ channel properties.
The parser handles a comma separated key=value list format by using the
getsubopt function of the standard library.

The option key '--channel-config' can be used with the list of key/value
pairs like e.g.
--channel-config name=output,type=push,method=bind
2017-04-20 11:07:46 +02:00
Dennis Klein
b43d4c142d refactor and integrate with aliBuild/alienv
* accept ZEROMQ_ROOT as env or cmake variable to hint at install
    location (used by aliBuild/alienv)
  * added more docs
  * fixed mismatch between module name ZeroMQ and prefix of
    generated variables ZMQ -> ZeroMQ is now the preferred prefix
    (old variables are still available and marked deprecated)
  * the aliBuild zeromq recipe does not install zmq.hpp header,
    so the module searches now for the zmq.h header
    (which is anyways the only one we use)
  * some cosmetic changes
  * add option ZeroMQ_NO_DEPRECATION to suppress deprecation warning
2017-02-23 19:42:15 +01:00
Alexey Rybalchenko
c66fd6fe91 Adding multiple transports support & other fixes:
- Avoid polling when only one input channel is used.
 - Send only handles for shared memory transport.
 - Avoid waiting in the rate logger thread when nothing to log.
 - Hide warnings from generated files
 - Fix #483
2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
a332d9fc83 First version of the shared memory transport.
Use via `--transport shmem` cmd option. No pub/sub.
2016-12-18 14:50:58 +01:00
Alexey Rybalchenko
b166cedb63 Convert factory methods to return smart ptrs
- Convert factory methods to return smart ptrs.
 - Refactor state machine to use same thread for user states.
 - Remove unused includes and dependencies, use std.
2016-11-18 14:19:16 +01:00
Alexey Rybalchenko
da3010b20c Move config & control DDS functionality into plugins. 2016-10-28 14:14:13 +02:00
Alexey Rybalchenko
16fd63cd5b Enable new callback API
- OnData() channel data handler.
 - ConditionalRun() for devices without incoming data.
 - Header file with common main(), to be extended with getDevice/addCustomOptions.
 - Update examples (MQ/Tutorial3) to use the new API and config.
 - NewSimpleMessage() for simpler creation of small messages (additional copy).
 - Replace SetProperty/GetProperty with fConfig access.
 - Runtime configurable channel names for common devices.
 - Configurable logging interval per channel.
 - FairMQMultiplier for distributing same data to multiple outputs.
 - Cleanup state machine messages.
 - Cmd option to toggle signal handling.
 - Simpler API for send/receive timeouts.
 - Enable --log-to-file.
 - Fix coverity issues, warnings.
 - Various code cleanup and minor tweaks.
2016-09-30 14:36:35 +02:00
winckler
e0a03242ac Simplify logger filtering options, fix log error double printing issue 2016-09-19 11:20:22 +02:00
winckler
64f1fd2a3b - add test for serialization examples 1&2
- rename serialization files and scripts
- remove policy and host header files from fairmq installation
2016-07-01 13:05:37 +02:00
Florian Uhlig
92944fe322 Correct shell scripts to work correctly with dash.
Add more dependencies for fairmq.

Something does not work with rpath when using boost 1.61. The problem is only seen for opensuse.
2016-06-03 15:45:18 +02:00
Radoslaw Karabowicz
fd09321fd9 Modified runDDSCommandUI to accept command line argument.
If no arguments, it behaves like before.
Removed runDDSCommand and replaced calls to it in controlDDS.
2016-05-30 11:28:43 +02:00
Radoslaw Karabowicz
65ca151620 MQ/example9 Adapted to the new running scheme with DDS.
- removed the executables that are running only in DDS (ddsEx9Sampler.cxx,
ddsEx9TaskProcessor.cxx, ddsEx9FileSink.cxx, ddsParameterMQServer.cxx);
- adapted the executables to allow running within the DDS (setting channel names);
- changed the topology file to run the correct executables (ex9-dds-topology.xml);
- ex9-dds.json - changed the parameter channel name back to "data" (to be compatible with other examples);
- added a shell script to controlDDS.sh;

- added fairmq/fairmq-dds-command executable to send single commands to DDS;
- adapted parmq/parmq-server to run within DDS.
2016-05-30 11:28:42 +02:00
Alexey Rybalchenko
253604344f nanomsg: reset maximum receive msg size 2016-05-27 12:58:07 +02:00
Alexey Rybalchenko
a05dc80402 Refactor DDS example and tools to be able to run with/without DDS 2016-05-20 14:09:53 +02:00
Alexey Rybalchenko
b9883d3b13 Configuration and DDS example/tools updates
- Update DDS example command UI and extract it from example.
 - Unify address handling via DDS properties for dynamic deployment.
 - Update DDS docs with the new approach.
 - Allow `--config-key` to be used to access common config in JSON.
 - Allow common channel properties to be specified for all sockets.
 - Update MQ examples and Tuto3 with new config options.
 - Add start scripts to MQ examples for easier use.
2016-04-25 17:35:57 +02:00
Mohammad Al-Turany
9a340fb7f6 Use Variables for Boost libraries instead of explicit names
Use the vairables filled by FindBoost instead explicit names in the cmake files
2016-03-29 15:31:29 +02:00
Alexey Rybalchenko
732373faa2 Update multi-part features (nanomsg) and various fixes
- Implement nanomsg multipart with MessagePack.
 - Use the MessagePack from FairSoft and handle not found case.
 - Update splitter, merger and proxy devices to handle multi-part.
 - Let FairMQParts.At() return pointer reference (can be used for moving).
 - Add missing const specifier in the message interface.
 - Add transmit kernel size setting to channels (ZMQ_SNDBUF).
 - Remove FairMQBuffer device.
 - Remove old multi-part methods from Tutorial3 example (to be replaced with Parts API).
 - Make callback mandatory for newMsg(data, size, callback).
 - Add missing <vector> include in FairMQSocket.
2016-03-24 13:36:47 +01:00
Alexey Rybalchenko
e1fef82657 FairMQ: Extend Multipart and messaging API
- Extend the multipart API to allow sending vectors of messages or helper
   thin wrapper FairMQParts. See example in examples/MQ/8-multipart.
 - NewMessage() can be used in devices instead of
   fTransportFactory->CreateMessage().
   Possible arguments remain unchanged (no args, size or data+size).
 - Send()/Receive() methods can be used in devices instead of
   fChannels.at("chan").at(i).Send()/Receive():
   Send(msg, "chan", i = 0), Receive(msg, "chan", i = 0).
 - Use the new methods in MQ examples and tests.
 - No breaking changes, but FAIRMQ_INTERFACE_VERSION is incremented to 3
   to allow to check for new methods.
2016-02-29 16:25:39 +01:00
Alexey Rybalchenko
f1abb9ecdd Remove compile time transport interface switch
- Remove the compile time check of the transport implementation.
  The transport (zeromq/nanomsg) can be chosen at run time with:
  `device.SetTransport("zeromq"); // possible values are "zeromq" and "nanomsg"`.

  For devices that use FairMQProgOptions, the transport can be configured via cmd option:
  `--transport zeromq` or `--transport nanomsg`. Default values is "zeromq".
  The device receives the configured value with:
  `device.SetTransport(config.GetValue<std::string>("transport"));`

  Old method of setting transport still works. But the NANOMSG constant is not defined.

- Remove old `fairmq/prototest` directory. It was only used as a test for protobuf.
  The protobuf part of Tutorial3 does the same (with different values).

- Fix a bug in FairMQPollerNN, where the `revents` value was not initialized.
  This caused the `poller->CheckOutput()` to trigger when it should not.
2016-01-14 15:19:30 +01:00
NicolasWinckler
8f32579984 add a pointer data type serializer function in the boost serializer add the zmq and nanomsg directories in cmaklists that use the runSimpleMQStateMachine 2015-11-20 09:03:34 +01:00
Florian Uhlig
0894900f52 Add missing link dependencies. 2015-11-19 15:51:44 +01:00
Alexey Rybalchenko
383a220333 Update device configuration
- Move general config files out of example directory to fairmq/run.
 - Use FairMQProgOptions for MQ example 5.
 - Add SendPartAsync() for non-blocking send of a message part.
2015-11-11 11:09:42 +01:00
NicolasWinckler
837490cc38 add FairProgOptionsHelper.h in header list to be installed of fairmq/CMakeLists.txt 2015-11-09 14:17:44 +01:00
Alexey Rybalchenko
d0c20d3729 Rename /example to /examples and move MQ examples in it 2015-11-09 10:36:24 +01:00
Alexey Rybalchenko
307d698736 Add example for sending multipart messages 2015-11-06 14:46:15 +01:00
Alexey Rybalchenko
2e789e4439 Add ParameterMQServer and example of its use 2015-11-06 14:46:15 +01:00
NicolasWinckler
3a7175f8e8 add lmd sampler-unpacker-sink chain in example/MQ/lmdSampler 2015-11-04 11:03:11 +01:00
Alexey Rybalchenko
023d88d0ef Add FairMQ Example 6 - Working with multiple channels 2015-10-07 16:46:10 +02:00
Alexey Rybalchenko
19afacb504 FairMQ Examples cleanup
- Rename Tutorial3 MQ files for uniform naming.
  - Add search for dylib in FindDDS.cmake (OSX).
  - Add more detail to the DDS example readme.
  - MQ Example 3 (DDS): choose network interface via command line option.
  - Give FairMQ examples their own CMakeLists.txt for clarity.
  - Remove C++11 checks in Tutorial3 from the code (they are now in CMake).
  - Add Serializer for device properties (FairMQDevice::ListProperties()).
2015-10-05 18:06:55 +02:00
Alexey Rybalchenko
f13bb5995d Add FairMQ tests (PUB-SUB, PUSH-PULL, REQ-REP). 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
fbf7dbf2ba Add orthogonal OK/ERROR states.
Replace state check mutex with atomic.

Update DDS example documentation.
2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
105e734808 Add DDS and Copy+Push examples. 2015-09-28 12:17:24 +02:00
winckler
e0ca1f62b3 - Replace the fairmq logger with one based on Boost.Log
- Adapt FairProgOptions to the new logger
2015-09-28 12:17:23 +02:00
Alexey Rybalchenko
1302e77a16 Several FairMQ fixes and improvements:
- FairMQ: add possibility to poll on multiple channels.
- FairMQ: include command channel when polling on blocking calls (for unblocking without termination).
- FairMQ: move signal handler inside of FairMQDevice class (call FairMQDevice::CatchSignals() in the main function).
- FairMQ: add 'bool CheckCurrentState(statename)' (instead of 'GetCurrentState() == statename' that cannot be thread safe).
- FairMQDevice: add 'InteractiveStateLoop()' method that can be used to change states from the command line.
- FairMQDevice: add automatic transition to IDLE state if Run() exits without an external event.
- FairMQDevice: implement device reset.
- FairMQDevice: use unordered_map for device channels.
- FairMQChannel: improve address validation for channels.
- FairMQChannel: add ExpectsAnotherPart() method to check if another msg part is expected (old approach still works).
- FairMQ: remove invalid transition from the run files.
- FairMQFileSink: disable ROOT termination signal handler.
- Tutorial3: spawn xterm windows from start scripts without overlapping for better visibility.
- FairMQ Examples: update protobuf test and move its files to a common directory.
- FairMQStateMachine: improve feedback on invalid transitions (more readable).
2015-09-28 12:17:22 +02:00
winckler
d1bba61939 - FairMQ options:
a) move the XML parser into the FairMQ/options/FairMQParser.h
b) add a routine in FairMQProgOption to check whether the necessary XML or JSON input files are there, and send an error message if not there

- Policy based devices:
a) rename GenericSampler to base_GenericSampler and use an alias template named GenericSampler
b) in base_GenericSampler, rename template parameter to simple variables <T,U,… > and use typedef for clarity
c) introduce an anonymous function container in the base_GenericSampler host class with a register task template member function and an Executetasks()
d) add two new template parameters in base_GenericSampler for the anonymous function container map. parameter is K for the key type (default=int) and L for the value type (default=std::function<void()>)

- Tutorial7:
a) use FairMQProgOption to configure devices in tutorial7
b) introduce several template functions helper in tutorial7 to reduce code redundancy
c) show examples in tutorial7 of task registration with callback and lambda expression for the sampler devices
d) separate the executable build of the tutorial7 data generator to remove the Roofit banner when executing the MQdevices
2015-09-28 12:17:21 +02:00
Alexey Rybalchenko
2656d8098c Add first two simple FairMQ examples
fairmq/examples/1-sampler-sink
fairmq/examples/2-sampler-processor-sink
2015-06-22 16:35:00 +02:00
Alexey Rybalchenko
e5313d03fe Fix uncaught exceptions 2015-06-19 12:33:06 +02:00
Alexey Rybalchenko
d6a413534a Thread safe initialization of the channels 2015-06-19 12:32:56 +02:00
Alexey Rybalchenko
295d9bba57 add WaitForEndOfStateForMs(state, duration) 2015-06-17 16:01:11 +02:00
Alexey Rybalchenko
a3d919b763 Use FairMQProgOptions for Sink and BenchmarkSampler 2015-06-16 12:43:23 +02:00
Alexey Rybalchenko
7fda980710 Update FairMQStateMachine & introduce FairMQChannels
Organize sockets as a map of vectors of FairMQChannels.

Update FairMQStateMachine by removing SETTINGINPUT, SETTINGOUTPUT,
BIND and CONNECT states and by adding INITIALIZING_TASK, RESETTING_TASK
and RESETTING_DEVICE states. Run states functions in their own thread.
2015-06-16 09:29:53 +02:00
winckler
a2ebbbe450 move fairmq json parser to fairmq/options 2015-06-15 15:59:12 +02:00
winckler
f54fbb0f59 Add FairMQProgOptions for configuring MQDevices 2015-06-15 15:57:20 +02:00