Commit Graph

320 Commits

Author SHA1 Message Date
Alexey Rybalchenko
459b42add2 FairMQ: MakeTransport to create stand-alone transport for tests. 2017-04-28 13:30:51 +02:00
Alexey Rybalchenko
085de240c2 FairMQ: fix missing includes. 2017-04-28 13:30:51 +02:00
Alexey Rybalchenko
2a526b8625 shmem: introduce FairMQShmMonitor. 2017-04-28 13:30:51 +02:00
Alexey Rybalchenko
7b4a2ae932 shmem: configurable segment name. 2017-04-28 13:30:51 +02:00
Alexey Rybalchenko
14f9084a80 shmem: Heartbeats for the monitor. 2017-04-28 13:30:51 +02:00
Alexey Rybalchenko
a8d40abbd9 FairProgOptions: update ambiguous option name. 2017-04-28 13:30:51 +02:00
Dennis Klein
f39b41d68c tweak startup and linger times 2017-04-21 09:21:57 +02:00
Dennis Klein
bbc8da54b4 clean redundant doxygen comments 2017-04-21 09:21:57 +02:00
Dennis Klein
91831ff701 print debug output immediately
This helps in the cases when get killed by the ctest timeout.
2017-04-21 09:21:57 +02:00
Dennis Klein
a1fe478ce7 fix more -Winconsistent-missing-override warnings 2017-04-20 11:07:50 +02:00
Dennis Klein
132ae69953 add startup and linger times for pubsub 2017-04-20 11:07:50 +02:00
Alexey Rybalchenko
541538e3de only reset sockets if needed 2017-04-20 11:07:50 +02:00
Alexey Rybalchenko
c51575f88c fix clang++ -Winconsistend-missing-override warnings 2017-04-20 11:07:50 +02:00
Dennis Klein
f963a06e25 make connecting channel wait loop more flexible 2017-04-20 11:07:50 +02:00
Alexey Rybalchenko
ee0d2e0c55 add debug output to test devices 2017-04-20 11:07:50 +02:00
Dennis Klein
218af15bbb ensure that FairMQ.Protocols is not run in parallel
The testsuite uses hardcoded system resources (TCP ports)
and is consequently not thread-safe.
2017-04-20 11:07:50 +02:00
Dennis Klein
d866589be1 Two devices can't share device id 2017-04-20 11:07:50 +02:00
Dennis Klein
bf2b208749 update docs 2017-04-20 11:07:49 +02:00
Alexey Rybalchenko
7caf0a6aae Configurable shmem segment size 2017-04-20 11:07:49 +02:00
Alexey Rybalchenko
ac7017deb5 Re-enable rate limiting in benchmarkSampler 2017-04-20 11:07:49 +02:00
Alexey Rybalchenko
5aaf27bf02 Refactor the transport interface
- give transport Initialize() method with access to device config.
 - avoid using global context in the transport.
 - simplify shutdown procedure (no need for extra thread).
2017-04-20 11:07:49 +02:00
Dennis Klein
d7eb692951 remove no longer needed special treatment for logger directory 2017-04-20 11:07:49 +02:00
Dennis Klein
9b337ca075 remove redundant include path
The imported ZeroMQ target has a target property
INTERFACE_INCLUDE_DIRECTORIES which carries this
information.
2017-04-20 11:07:49 +02:00
Dennis Klein
d02ae9be2c remove no longer used variable 2017-04-20 11:07:48 +02:00
Giulio Eulisse
1ebbe80539 Add ZeroMQ include path 2017-04-20 11:07:48 +02:00
Dennis Klein
c41cf318bd export FairMQ targets and install headers hierarchically 2017-04-20 11:07:47 +02:00
Mohammad Al-Turany
608e34f26d Set CMake min. vers. to 3.6.0 and add conrtibutors file 2017-04-20 11:07:47 +02:00
Dennis Klein
365a68010f various fixes
* fix include paths
* print on stderr
2017-04-20 11:07:47 +02:00
Dennis Klein
13c9c09ab9 list fairmq authors centrally instead of in each file
listing authors in source files is inaccurate too. The version
control system documents this information most accurately, so per
file author lists are dropped for simplicity.
2017-04-20 11:07:47 +02:00
Dennis Klein
35b3212f7e cmake cleanups and fixes
* fix overlinking of libFairMQ
* import dependencies in find modules
* provide FairMQinstall target
* add gtest helper module
* add cotire helper module
* add many comments
* cleanup fairmq cmake file
* add FindPStreams module
* link logger into libFairMQ
* drop root dictionary generation
* bump min required cmake version to 3.7.2
2017-04-20 11:07:47 +02:00
Dennis Klein
7e34f7042f refactor tests
* move to gtest
* add shmem tests
* cleanup cmake file (organize tests in suites)
2017-04-20 11:07:47 +02:00
Matthias Richter
55a9d69908 Adding option '--channel-config' to MQProgOptions 2017-04-20 11:07:46 +02:00
Matthias Richter
9943be507e Adding helper function to print the raw boost property tree 2017-04-20 11:07:46 +02:00
Matthias Richter
e184610c06 Adding parser implementation for key-value subopt format
A parser implementation for FairMQ channel properties.
The parser handles a comma separated key=value list format by using the
getsubopt function of the standard library.

The option key '--channel-config' can be used with the list of key/value
pairs like e.g.
--channel-config name=output,type=push,method=bind
2017-04-20 11:07:46 +02:00
Alexey Rybalchenko
341464a793 Compatibility with nanomsg<=0.6 2017-03-21 14:53:07 +01:00
Alexey Rybalchenko
3c73c690f7 Transports: use transportType as static member. 2017-03-14 17:51:26 +01:00
Alexey Rybalchenko
b237b3f9d7 Remove unused code.
- Remove unused code in FairMQTools.
- Remove obsolete device options (now in device config).
2017-03-14 17:51:26 +01:00
Alexey Rybalchenko
2293c5e417 FairMQChannel: API to access socket metrics; make fSocket private. 2017-03-14 17:51:26 +01:00
Giulio Eulisse
2a72d58766 Remove unneeded if when deleting NULL / 0 / nullptr
C++ standard dictates that NULL / 0 / nullptr is a valid argument for
delete which then simply has no effect:

"The value of the first argument supplied to a deallocation function may
be a null pointer value; if so, and if the deallocation function is one
supplied in the standard library, the call has no effect."

I therefore think in these particular case it's safe to remove the ifs.
2017-03-13 15:46:56 +01:00
Florian Uhlig
d1cf852c98 Fox Coverity issues
Fix CID 131712. In case parameter string does not fit in variable return a failure.
Fix CID 162913. Initialize missing data member.
Fix CID 58901.
Fix CID 149137.
Fix CID 149143.
Fix CIDs 149156, 149150, 149147, 149145, 149144, 149137. Don't use command line parameter argv directly.
2017-03-03 15:45:23 +01:00
Alexey Rybalchenko
91b7a72ac5 get default network interface from the default route 2017-03-02 13:25:06 +01:00
Alexey Rybalchenko
eaa220e39e better tests output 2017-03-02 13:25:06 +01:00
Sandro Wenzel
9e477264c8 Better encapsulate Device config: Offer a getter method 2017-03-02 13:24:35 +01:00
Alexey Rybalchenko
c78b7e4cfa Fix nanomsg tests 2017-02-27 17:49:30 +01:00
Alexey Rybalchenko
07b760218f Fix coverity #157658 2017-02-27 13:09:16 +01:00
Alexey Rybalchenko
7c1f7aa3f2 Fix Tuto3 double delete, cleanup test output, undeclared var in NN socket. 2017-02-27 13:09:16 +01:00
Dennis Klein
b43d4c142d refactor and integrate with aliBuild/alienv
* accept ZEROMQ_ROOT as env or cmake variable to hint at install
    location (used by aliBuild/alienv)
  * added more docs
  * fixed mismatch between module name ZeroMQ and prefix of
    generated variables ZMQ -> ZeroMQ is now the preferred prefix
    (old variables are still available and marked deprecated)
  * the aliBuild zeromq recipe does not install zmq.hpp header,
    so the module searches now for the zmq.h header
    (which is anyways the only one we use)
  * some cosmetic changes
  * add option ZeroMQ_NO_DEPRECATION to suppress deprecation warning
2017-02-23 19:42:15 +01:00
Alexey Rybalchenko
c66fd6fe91 Adding multiple transports support & other fixes:
- Avoid polling when only one input channel is used.
 - Send only handles for shared memory transport.
 - Avoid waiting in the rate logger thread when nothing to log.
 - Hide warnings from generated files
 - Fix #483
2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
e53ad151a7 Fix mismatch in FairMQSink parameter type. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
3e7cb85816 Fix incorrect calculation of transfer rate for multipart. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
8cf1cbb930 Initialize shared memory only if it is used. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
ac1da4db66 Require std::move when adding existing message to FairMQParts. 2017-02-23 06:47:09 +01:00
Alexey Rybalchenko
65c19f07bf Fix compatibility with ZeroMQ 4.2.1 2017-02-09 12:19:23 +01:00
Florian Uhlig
617042b3c6 For tests on MacOSX use new MacOSX version 10.11 and FairSoft version may16p1.
Run fairmq tests sequentialy.
2017-01-27 06:16:11 -08:00
Alexey Rybalchenko
62df0b609b disable feature not available on GCC < 5 2016-12-19 12:06:27 +01:00
Alexey Rybalchenko
a332d9fc83 First version of the shared memory transport.
Use via `--transport shmem` cmd option. No pub/sub.
2016-12-18 14:50:58 +01:00
Alexey Rybalchenko
6c3b01f09c AttachChannel must update the config after it's done, use boost::split 2016-12-14 08:52:05 +01:00
Alexey Rybalchenko
0fb49a0986 State machine update
- Fix regression from last commit (preventing static run from proper shutdown).
 - Guard state changes (for the internal transitions) (msm::process_event is not thread safe).
 - Remove unused transition from RUNNING to EXITING.
2016-11-24 22:35:46 +01:00
Alexey Rybalchenko
b166cedb63 Convert factory methods to return smart ptrs
- Convert factory methods to return smart ptrs.
 - Refactor state machine to use same thread for user states.
 - Remove unused includes and dependencies, use std.
2016-11-18 14:19:16 +01:00
Alexey Rybalchenko
12f04c7237 Fix race condition in the state machine & channel. 2016-11-15 16:17:56 +01:00
mkrzewic
94a4d599eb Validate connection method per endpoint, not socket
this allows e.g. to ommit the socket method in config if all endpoints
specify the method modifier.
It still is fully backward compatible.
2016-11-13 14:22:51 +01:00
mkrzewic
892aa8008b Add missing #include <cstring> 2016-11-13 00:05:57 +01:00
mkrzewic
6bec8fc1db Simplify Tokenize(): nicer looking code = better code 2016-11-12 18:04:14 +01:00
mkrzewic
c2d7c49cf5 Support multiple endpoints per socket
Sent messages will be scheduled among the endpoints according to socket
type: PUB will send the same data to all endpoints simultaneously, PUSH
will do round robin transfer.
Incoming data is fair queued between endpoints.

This is a feature of at least zeromq and nanomsg.

_____________
To use: in the device configuration, instead of specifying just one address,
specify a comma separated list e.g.

tcp://localhost:123,ipc:///tmp/socket

the connection method (bind/connect) applies to all endpoints in this case.
______________
Mixing binding and connecting endpoints is supported:
prefix "@" means "bind", "+" (or ">") means connect, e.g.

+tcp://localhost:123,@ipc:///tmp/socket,ipc:///tmp/asd

(in case of missing prefix, the default channel method is used for that
endpoint).
2016-11-11 16:58:55 +01:00
Sebastien Binet
0a3f14c0e3 all: apply fer-json-fmt 2016-11-10 15:20:54 +01:00
Sebastien Binet
02f5e63784 all: use strict(er) and more regular JSON schema 2016-11-10 15:20:54 +01:00
Sebastien Binet
581e93968e fairmq/options: fix JSON example 2016-11-10 10:55:50 +01:00
Alexey Rybalchenko
79fba8ec4c Update JSON files & readme, use FairMQDevicePtr, cleanup. 2016-11-09 15:01:40 +01:00
Alexey Rybalchenko
da3010b20c Move config & control DDS functionality into plugins. 2016-10-28 14:14:13 +02:00
Alexey Rybalchenko
16fd63cd5b Enable new callback API
- OnData() channel data handler.
 - ConditionalRun() for devices without incoming data.
 - Header file with common main(), to be extended with getDevice/addCustomOptions.
 - Update examples (MQ/Tutorial3) to use the new API and config.
 - NewSimpleMessage() for simpler creation of small messages (additional copy).
 - Replace SetProperty/GetProperty with fConfig access.
 - Runtime configurable channel names for common devices.
 - Configurable logging interval per channel.
 - FairMQMultiplier for distributing same data to multiple outputs.
 - Cleanup state machine messages.
 - Cmd option to toggle signal handling.
 - Simpler API for send/receive timeouts.
 - Enable --log-to-file.
 - Fix coverity issues, warnings.
 - Various code cleanup and minor tweaks.
2016-09-30 14:36:35 +02:00
winckler
e0a03242ac Simplify logger filtering options, fix log error double printing issue 2016-09-19 11:20:22 +02:00
winckler
9ceab6099c add a signal/slot mechanism. 2 APIs : one generic version, and one string API (commented) 2016-09-19 11:20:22 +02:00
Alexey Rybalchenko
5e5ddd5b7b Fix further effc++ and reorder warnings. 2016-09-19 11:10:46 +02:00
Alexey Rybalchenko
7c99bd77cc Fix unused variables, init order and effc++ warnings. 2016-09-06 10:19:23 +02:00
Florian Uhlig
827c8dcd45 Remove warnings about unused parameters. 2016-08-12 13:25:07 +02:00
winckler
bed58891ea merge MQ parameters in the variable_map container of FairMQProgramOptions class 2016-08-05 08:34:15 +02:00
Alexey Rybalchenko
2ed0c52ae8 Test nanomsg transport only when nanomsg library is found. 2016-08-05 07:47:20 +02:00
Alexey Rybalchenko
13a5cea1d7 fix parallel running tests failure 2016-07-11 10:58:30 +02:00
winckler
64f1fd2a3b - add test for serialization examples 1&2
- rename serialization files and scripts
- remove policy and host header files from fairmq installation
2016-07-01 13:05:37 +02:00
winckler
c57bbf58fa - Remove GenericSampler, GenericProcessor, and CRTP policy base classes
- Rename GenericFileSink to BaseMQFileSink, and move it to base/MQ/hosts directory
- Rename and clean files in the serialization examples
2016-06-29 12:24:13 +02:00
Florian Uhlig
75aad5676d Use Boost_ variables instead of hardcoded names. 2016-06-08 15:38:14 +02:00
Florian Uhlig
92944fe322 Correct shell scripts to work correctly with dash.
Add more dependencies for fairmq.

Something does not work with rpath when using boost 1.61. The problem is only seen for opensuse.
2016-06-03 15:45:18 +02:00
Alexey Rybalchenko
f3bebe9321 Use different ports in tests to prevent parallel run fail 2016-06-01 11:49:33 +02:00
Alexey Rybalchenko
7062a3d280 Run FairMQ tests with both ZeroMQ and nanomsg 2016-05-31 10:42:06 +02:00
Alexey Rybalchenko
eb306a2c0d Remove color output from tests 2016-05-30 16:24:15 +02:00
Radoslaw Karabowicz
fd09321fd9 Modified runDDSCommandUI to accept command line argument.
If no arguments, it behaves like before.
Removed runDDSCommand and replaced calls to it in controlDDS.
2016-05-30 11:28:43 +02:00
Radoslaw Karabowicz
65ca151620 MQ/example9 Adapted to the new running scheme with DDS.
- removed the executables that are running only in DDS (ddsEx9Sampler.cxx,
ddsEx9TaskProcessor.cxx, ddsEx9FileSink.cxx, ddsParameterMQServer.cxx);
- adapted the executables to allow running within the DDS (setting channel names);
- changed the topology file to run the correct executables (ex9-dds-topology.xml);
- ex9-dds.json - changed the parameter channel name back to "data" (to be compatible with other examples);
- added a shell script to controlDDS.sh;

- added fairmq/fairmq-dds-command executable to send single commands to DDS;
- adapted parmq/parmq-server to run within DDS.
2016-05-30 11:28:42 +02:00
Alexey Rybalchenko
253604344f nanomsg: reset maximum receive msg size 2016-05-27 12:58:07 +02:00
Alexey Rybalchenko
cc88339791 Cleanup command line options
- Remove old cmd options --config-json-file and --config-xml-file.
   Now handled by --mq-config with file extension detection.
 - Fix --config-json-string and --config-xml-string
 - Move config-key to proper place (in --help)
2016-05-23 16:40:32 +02:00
Alexey Rybalchenko
4944d7ec41 Meaningful error on wrong channel name 2016-05-20 14:10:14 +02:00
Alexey Rybalchenko
a05dc80402 Refactor DDS example and tools to be able to run with/without DDS 2016-05-20 14:09:53 +02:00
Alexey Rybalchenko
af971c6ab1 Allow rate limiting in the Shared Memory example 2016-05-18 16:01:21 +02:00
Alexey Rybalchenko
1f09317cd0 Handle both single and multi part payloads in splitter/merger/proxy 2016-05-17 12:27:33 +02:00
Alexey Rybalchenko
e9f09143b2 Gracefully end the device also in the termination case (instead of abort). 2016-05-17 12:27:18 +02:00
Alexey Rybalchenko
ce42f36126 Remove redundant state change call and revert the termination handler. 2016-05-12 17:28:33 +02:00
Alexey Rybalchenko
28aa5e4caa Let ParseAll exit if unsuccessfull 2016-05-12 17:28:33 +02:00
Alexey Rybalchenko
31456e71ff Threads safe socket rate stats 2016-05-12 17:28:32 +02:00
Alexey Rybalchenko
3353e214a7 Add shared memory example
- Add shared memory example in examples/MQ/SharedMemory
 - Device/Task termination: try soft first, and abort if it fails
 - Interactive mode: prevent cin from blocking forever (poll)
2016-05-11 14:52:53 +02:00
Radoslaw Karabowicz
599d1b3e05 MQ/example9 Added testing of the static MQ execution.
- removed the obsolete startFairMQEx9.sh.in script
- added all *.root files in example9/macros/ to .gitignore;
- added run_digi.C to tests;
- added startFairMQEx9_Static.sh to tests;

- startFairMQEx9_Static.sh - script that will run the example non-interactively, statically;
- Pixel9MQConfig_Multipart.json - added acknowledgement channel to sampler and sink;
- runEx9Sampler.cxx and runEx9FileSink.cxx - added option to set the ack channel name;
- FairMQEx9FileSink - if ack channel name set, send acknowledgement of received message, save the tree and file at the end of Run;
- FairMQEx9Sampler - if ack channel name set, count acknowledgements and finish only if all events processed;
- FairMQEx9TaskProcessor - moved some printouts to debug.

General:
- fairmq/logger/logger.cxx - fixed colorless logger bug;
- parmq/runParameterMQServer.cxx - now using runStateMachine function (to enable static execution).
2016-05-06 14:46:17 +02:00
Alexey Rybalchenko
58754e03c7 Make sure logger is used before configuring it. 2016-04-28 10:28:18 +02:00