Commit Graph

56 Commits

Author SHA1 Message Date
Alexey Rybalchenko
92cae6f9a4 FairMQProxy: remove outdated code 2017-06-23 11:45:46 +02:00
Alexey Rybalchenko
4bc54ad32b add --print-channels to print registered channels of the device 2017-06-23 11:45:46 +02:00
Alexey Rybalchenko
6b221d950c Add NewPoller() wrapper. 2017-05-23 08:32:13 +02:00
Alexey Rybalchenko
ac7017deb5 Re-enable rate limiting in benchmarkSampler 2017-04-20 11:07:49 +02:00
Dennis Klein
365a68010f various fixes
* fix include paths
* print on stderr
2017-04-20 11:07:47 +02:00
Alexey Rybalchenko
c66fd6fe91 Adding multiple transports support & other fixes:
- Avoid polling when only one input channel is used.
 - Send only handles for shared memory transport.
 - Avoid waiting in the rate logger thread when nothing to log.
 - Hide warnings from generated files
 - Fix #483
2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
e53ad151a7 Fix mismatch in FairMQSink parameter type. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
ac1da4db66 Require std::move when adding existing message to FairMQParts. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
a332d9fc83 First version of the shared memory transport.
Use via `--transport shmem` cmd option. No pub/sub.
2016-12-18 14:50:58 +01:00
Alexey Rybalchenko
b166cedb63 Convert factory methods to return smart ptrs
- Convert factory methods to return smart ptrs.
 - Refactor state machine to use same thread for user states.
 - Remove unused includes and dependencies, use std.
2016-11-18 14:19:16 +01:00
Alexey Rybalchenko
16fd63cd5b Enable new callback API
- OnData() channel data handler.
 - ConditionalRun() for devices without incoming data.
 - Header file with common main(), to be extended with getDevice/addCustomOptions.
 - Update examples (MQ/Tutorial3) to use the new API and config.
 - NewSimpleMessage() for simpler creation of small messages (additional copy).
 - Replace SetProperty/GetProperty with fConfig access.
 - Runtime configurable channel names for common devices.
 - Configurable logging interval per channel.
 - FairMQMultiplier for distributing same data to multiple outputs.
 - Cleanup state machine messages.
 - Cmd option to toggle signal handling.
 - Simpler API for send/receive timeouts.
 - Enable --log-to-file.
 - Fix coverity issues, warnings.
 - Various code cleanup and minor tweaks.
2016-09-30 14:36:35 +02:00
winckler
c57bbf58fa - Remove GenericSampler, GenericProcessor, and CRTP policy base classes
- Rename GenericFileSink to BaseMQFileSink, and move it to base/MQ/hosts directory
- Rename and clean files in the serialization examples
2016-06-29 12:24:13 +02:00
Alexey Rybalchenko
af971c6ab1 Allow rate limiting in the Shared Memory example 2016-05-18 16:01:21 +02:00
Alexey Rybalchenko
1f09317cd0 Handle both single and multi part payloads in splitter/merger/proxy 2016-05-17 12:27:33 +02:00
winckler
e977dfb609 update generic sampler with new serialization api 2016-04-06 12:05:50 +02:00
winckler
819a21f46a remove obsolete documentation 2016-03-31 19:13:51 +02:00
winckler
e215049db9 remove alternative serialization API 2016-03-31 19:12:48 +02:00
winckler
4c50409af5 add serialization API 2016-03-31 19:12:48 +02:00
Alexey Rybalchenko
732373faa2 Update multi-part features (nanomsg) and various fixes
- Implement nanomsg multipart with MessagePack.
 - Use the MessagePack from FairSoft and handle not found case.
 - Update splitter, merger and proxy devices to handle multi-part.
 - Let FairMQParts.At() return pointer reference (can be used for moving).
 - Add missing const specifier in the message interface.
 - Add transmit kernel size setting to channels (ZMQ_SNDBUF).
 - Remove FairMQBuffer device.
 - Remove old multi-part methods from Tutorial3 example (to be replaced with Parts API).
 - Make callback mandatory for newMsg(data, size, callback).
 - Add missing <vector> include in FairMQSocket.
2016-03-24 13:36:47 +01:00
Florian Uhlig
fa7040fe65 Fix compiler warnings.
Initialize all data members in initializer lists.
Reorder data members in initializer list to have the same order as in the class declaration.
Comment or remove unused parameters and unused variables.
Convert all old style casts to the correct and explicit c++ cast like const_cast, static_cast,
dynamic_cast or reinterpret_cast. In most cases static_cast is used.
2016-03-14 13:37:55 +01:00
Alexey Rybalchenko
bb36147099 Allocate dynamic size flatbuffers structure on the heap 2016-03-10 13:08:00 +01:00
Alexey Rybalchenko
82ab7670a9 Add FlatBuffers & MessagePack examples
- Add FlatBuffers serialization example to Tutorial 3
 - Add MessagePack serialization example to Tutorial 3
 - Performance improvements in Boost serialization example
 - Use `GetEntriesFast()` for FairTestDetectorRecoTask
 - Use `Clear()` instead of `Delete()` in MQ parts of Tutorial 3
 - Fix CMake variables from preventing compilation without nanomsg.
 - create macro/data directories in install directory
 - Get rid of data duplication in fill_parameters.C
 - Various cleanups and fixes
2016-02-22 13:06:37 +01:00
Alexey Rybalchenko
f1abb9ecdd Remove compile time transport interface switch
- Remove the compile time check of the transport implementation.
  The transport (zeromq/nanomsg) can be chosen at run time with:
  `device.SetTransport("zeromq"); // possible values are "zeromq" and "nanomsg"`.

  For devices that use FairMQProgOptions, the transport can be configured via cmd option:
  `--transport zeromq` or `--transport nanomsg`. Default values is "zeromq".
  The device receives the configured value with:
  `device.SetTransport(config.GetValue<std::string>("transport"));`

  Old method of setting transport still works. But the NANOMSG constant is not defined.

- Remove old `fairmq/prototest` directory. It was only used as a test for protobuf.
  The protobuf part of Tutorial3 does the same (with different values).

- Fix a bug in FairMQPollerNN, where the `revents` value was not initialized.
  This caused the `poller->CheckOutput()` to trigger when it should not.
2016-01-14 15:19:30 +01:00
Alexey Rybalchenko
0e1a1ad552 Allow to limit number of messages for the Benchmark sampler and sink 2016-01-14 14:19:03 +01:00
Alexey Rybalchenko
e4fed2fa1b Fix copy constructor & assignment operator warning in FairMQChannel
Fix Weffc++ warnings

  - Add missing copy constructors and assignment operators .
  - Hide the warning from FairMQStateMachine.h where it is produced by boost and/or is intended.
  - Some code cleanup.
2016-01-14 14:19:03 +01:00
Florian Uhlig
d254efb91f Fix compiler warnings. Mostly unused parameters have been removed. 2016-01-14 14:19:02 +01:00
NicolasWinckler
10d6482716 remove unfinished doc 2015-11-20 09:03:34 +01:00
NicolasWinckler
8f32579984 add a pointer data type serializer function in the boost serializer add the zmq and nanomsg directories in cmaklists that use the runSimpleMQStateMachine 2015-11-20 09:03:34 +01:00
NicolasWinckler
76626b4f4b add test dir to examples/MQ/GenericDevices + change policy typedef in generic device from private to protected 2015-11-12 11:24:07 +01:00
NicolasWinckler
1467360f75 access channel map with at 2015-11-12 11:24:07 +01:00
NicolasWinckler
13d3729fec enable/disable multipart functionnality for sending header if source policy has the proper signature
add comment to sendHeader
2015-11-09 14:08:27 +01:00
NicolasWinckler
6ce6887212 formatting, change and clean generic_sampler and SimpleTreeReader 2015-11-09 14:08:27 +01:00
NicolasWinckler
323033c9eb Add CRTP base class for the policies of the generic device 2015-11-04 11:02:32 +01:00
NicolasWinckler
559c7babca add doc for the generic devices 2015-11-04 11:02:11 +01:00
NicolasWinckler
fe91aa1634 add typedef for policies in host class, and make the sampler task container optional. 2015-11-04 11:02:10 +01:00
NicolasWinckler
946f1c9d00 add doc for the tutorial7 and the generic device 2015-11-04 11:01:40 +01:00
Alexey Rybalchenko
023d88d0ef Add FairMQ Example 6 - Working with multiple channels 2015-10-07 16:46:10 +02:00
Alexey Rybalchenko
19afacb504 FairMQ Examples cleanup
- Rename Tutorial3 MQ files for uniform naming.
  - Add search for dylib in FindDDS.cmake (OSX).
  - Add more detail to the DDS example readme.
  - MQ Example 3 (DDS): choose network interface via command line option.
  - Give FairMQ examples their own CMakeLists.txt for clarity.
  - Remove C++11 checks in Tutorial3 from the code (they are now in CMake).
  - Add Serializer for device properties (FairMQDevice::ListProperties()).
2015-10-05 18:06:55 +02:00
Alexey Rybalchenko
5136c88d3a Simplify Tutorial3 structure.
Use same executable for bin/boost/protobuf/root data format,
configured now via `--data-format <binary/boost/protobuf/tmessage>`
command line parameter.
2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
8b71e4d20b Use FairMQDevice::CatchSignals for Tutorial7 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
f13bb5995d Add FairMQ tests (PUB-SUB, PUSH-PULL, REQ-REP). 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
fbf7dbf2ba Add orthogonal OK/ERROR states.
Replace state check mutex with atomic.

Update DDS example documentation.
2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
a7ab33a10e Add new Send/Receive methods with smart pointers and no flag checks. 2015-09-28 12:17:24 +02:00
Alexey Rybalchenko
1302e77a16 Several FairMQ fixes and improvements:
- FairMQ: add possibility to poll on multiple channels.
- FairMQ: include command channel when polling on blocking calls (for unblocking without termination).
- FairMQ: move signal handler inside of FairMQDevice class (call FairMQDevice::CatchSignals() in the main function).
- FairMQ: add 'bool CheckCurrentState(statename)' (instead of 'GetCurrentState() == statename' that cannot be thread safe).
- FairMQDevice: add 'InteractiveStateLoop()' method that can be used to change states from the command line.
- FairMQDevice: add automatic transition to IDLE state if Run() exits without an external event.
- FairMQDevice: implement device reset.
- FairMQDevice: use unordered_map for device channels.
- FairMQChannel: improve address validation for channels.
- FairMQChannel: add ExpectsAnotherPart() method to check if another msg part is expected (old approach still works).
- FairMQ: remove invalid transition from the run files.
- FairMQFileSink: disable ROOT termination signal handler.
- Tutorial3: spawn xterm windows from start scripts without overlapping for better visibility.
- FairMQ Examples: update protobuf test and move its files to a common directory.
- FairMQStateMachine: improve feedback on invalid transitions (more readable).
2015-09-28 12:17:22 +02:00
winckler
d1bba61939 - FairMQ options:
a) move the XML parser into the FairMQ/options/FairMQParser.h
b) add a routine in FairMQProgOption to check whether the necessary XML or JSON input files are there, and send an error message if not there

- Policy based devices:
a) rename GenericSampler to base_GenericSampler and use an alias template named GenericSampler
b) in base_GenericSampler, rename template parameter to simple variables <T,U,… > and use typedef for clarity
c) introduce an anonymous function container in the base_GenericSampler host class with a register task template member function and an Executetasks()
d) add two new template parameters in base_GenericSampler for the anonymous function container map. parameter is K for the key type (default=int) and L for the value type (default=std::function<void()>)

- Tutorial7:
a) use FairMQProgOption to configure devices in tutorial7
b) introduce several template functions helper in tutorial7 to reduce code redundancy
c) show examples in tutorial7 of task registration with callback and lambda expression for the sampler devices
d) separate the executable build of the tutorial7 data generator to remove the Roofit banner when executing the MQdevices
2015-09-28 12:17:21 +02:00
Alexey Rybalchenko
a3d919b763 Use FairMQProgOptions for Sink and BenchmarkSampler 2015-06-16 12:43:23 +02:00
Alexey Rybalchenko
7fda980710 Update FairMQStateMachine & introduce FairMQChannels
Organize sockets as a map of vectors of FairMQChannels.

Update FairMQStateMachine by removing SETTINGINPUT, SETTINGOUTPUT,
BIND and CONNECT states and by adding INITIALIZING_TASK, RESETTING_TASK
and RESETTING_DEVICE states. Run states functions in their own thread.
2015-06-16 09:29:53 +02:00
winckler
445ef356d1 - Rename some function members of Generic MQ Devices and propagate the modifications accordingly.
- Add a generic merger.

add SYSTEM_INCLUDE_DIRECTORIES in project_template CMakeLists.txt so that boost include dirs are found.
2015-05-28 09:32:30 +02:00
Alexey Rybalchenko
26a1033e9d Rename device property to have a common format and some code cleanup.
FairMQDevice: Rename property for socket rate logging to have common format (old name is still available for compatibility).
FairMQ: Avoid using std namespace in class headers (may require adding std namespace to some child devices).
FairMQ: A bit of code cleanup
FairMQConfigurable: Stop with an error if a property assignment failed due to incorrect key.
2015-05-28 09:32:30 +02:00
winckler
6d65c4313a Split serializer policies into serializer and deserializer policies, and change serializer method names message() to SerializeMsg() and DeserializeMsg()
Improve boost serializer performance
2015-02-11 11:13:45 +01:00