diff --git a/odin-data.md b/odin-data.md index 78f27e2..3f84153 100644 --- a/odin-data.md +++ b/odin-data.md @@ -1,5 +1,5 @@ # AEG ODIN data workshop -## 25 June 18 +## 25 June 18 (Updated 11/09/25) # Table of Contents @@ -89,21 +89,79 @@ number of received packets, missing packets, timestamp etc } FrameHeader; ``` -### External Software Dependencies +## External Software Dependencies -The following libraries and packages are required: +The following libraries and packages are required however on our dev machines they are pre installed: -* [CMake](http://www.cmake.org) : build management system (version >= 2.8) -* [Boost](http://www.boost.org) : portable C++ utility libraries. The following components are - used - program_options, unit_test_framework, date_time, interprocess, bimap (version >= 1.41) -* [ZeroMQ](http://zeromq.org) : high-performance asynchronous messaging library (version >= 3.2.4) -* [Log4CXX](http://logging.apache.org/log4cxx/): Configurable message logger (version >= 0.10.0) -* [HDF5](https://www.hdfgroup.org/HDF5): __Optional:__ if found, the frame processor application -will be built (version >= 1.8.14) -* [LibPCAP](https://github.com/the-tcpdump-group/libpcap): TCPDUMP packet capture libraries - usually - installed with distro install tools. e.g. yum -* [blosc](https://github.com/Blosc/c-blosc): An fast compression library - needed for the frame - processor compression plugin (__Beware__: API change with version 1.13 -> 1.14) +* [CMake](http://www.cmake.org) +* [Boost](http://www.boost.org) +* [ZeroMQ](http://zeromq.org) +* [Log4CXX](http://logging.apache.org/log4cxx/) +* [HDF5](https://www.hdfgroup.org/HDF5) +* [LibPCAP](https://github.com/the-tcpdump-group/libpcap) +* [blosc](https://github.com/Blosc/c-blosc) + +## Building odin-data and plugins + +### Create development environment + +* Create a project development dir (using AEG path convention if appropriate). This example uses the shared users folder. + +``` +$ mkdir -p /aeg_sw/work/users//develop/projects/ +$ cd /aeg_sw/work/users//develop/projects/ +``` + +### Clone, build and install odin-data + +* Clone `odin-data` repo from Github (ensuring the latest tagged release is used): +``` +$ git clone -b 1.11.0 https://github.com/odin-detector/odin-data.git +``` + +* Create an `install` directory to install `odin-data` and plugins into: +``` +mkdir install +``` + +* Create a build directory for CMake to use in `cpp` in the `odin-data` directory: +``` +$ cd odin-data/cpp +$ mkdir build && cd build +``` + +* Configure CMake to define use of correct packages and set up install directory: +``` +$ cmake -DCMAKE_INSTALL_PREFIX= /aeg_sw/work/users//develop/projects//install .. +``` + + +* Compile odin-data: +``` +$ make -j +``` + +* Install the compiled project into the install directory +``` +make install +``` + +* To test everything is working run either of the commands below in the install directory +``` +./bin/frameReceiverTest +# or +./bin/frameProcessorTest +``` +For example if you run `./bin/frameReceiverTest` should produce something like this: +```Running 34 test cases... +log4cxx: No appender could be found for logger (FR.DummyUDPFrameDecoder). +log4cxx: Please initialize the log4cxx system properly. +INFO - DummyTCPFrameDecoder version 1.11.0 loaded + +*** No errors detected +``` + +# The content below below is potentially outdated: ## Demo using EXCALIBUR as example @@ -168,293 +226,6 @@ I Sent 1 frames with a total of 132 packets and 1049648 bytes, dropping 0 packet ![EXCALIBUR image](images/excalibur.png) -## Building odin-data and plugins - -### Create development environment - -* Need to ensure the `aeg_sw` profile is loaded: -``` -$ source /aeg_sw/etc/profile.sh -``` -for `bash`-like shells, or -``` -$ source /aeg_sw/etc/profile.csh -``` -for `csh`-like shells. -* Allows you to use `environment-modules` to load various libraries, packages -etc into your environment. To see what's available: - -``` -$ module avail - -------------------------------------- /aeg_sw/etc/modulefiles -------------------------------------- -aeg_apps aeg_tools aeg_use_own - ----------------------------------- /usr/share/Modules/modulefiles ---------------------------------- -dot module-git module-info modules null use.own - ------------------------------------- /.privatemodules ------------------------------------ -odin-data - ---------------------------------- /aeg_sw/apps/Modules/modulefiles --------------------------------- -arduino/1-8-3 dawn/2-5-0 eclipse/472(default) pycharm/2017-1-4 -cmake/3-11-4 dawn/2-7-0 eclipse/472_20171218 - -------------------------- /aeg_sw/tools/CentOS7-x86_64/Modules/modulefiles ------------------------- -blosc/1-13-5 git/2-13-2 python/2 python/3-6 ruby/2-4 xilinx/arm/2015-2 -blosc/1-14-4 hdf5/1-10-0 python/2-7 python/3-6-2 ruby/2-4-1 zeromq/4-2-1 -boost/1-48-0 hdf5/1-10-4 python/2-7-13 python/3-6-5 spark/2-3-0 zeromq/4-2-3 -boost/1-66-0 log4cxx/0-10-0 python/2-7-15 qt/4-8-7 xilinx/2015-2 -boost/1-67-0 msgpack-c/2-1-5 python/3 ruby/2 xilinx/2017-2 -``` - -* Load the correct modules for odin-development: -``` -$ module load boost/1-67-0 hdf5/1-10-0 zeromq/4-2-3 log4cxx/0-10-0 blosc/1-14-4 -``` -* Sets up your shell environment with various paths etc: -``` -$ echo $BOOST_ROOT -/aeg_sw/tools/CentOS6-x86_64/boost/1-48-0/prefix -$ echo $HDF5_ROOT -/aeg_sw/tools/CentOS6-x86_64/hdf5/1-10-0/prefix -$ echo $ZEROMQ_ROOT -/aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix -$ echo $LOG4CXX_ROOT -/aeg_sw/tools/CentOS6-x86_64/log4cxx/0-10-0/prefix -$ echo $BLOSC_ROOT -/aeg_sw/tools/CentOS7-x86_64/c-blosc/1-14-4/prefix -``` - -* Create a project development dir (using AEG path convention if appropriate) - -``` -$ mkdir -p ~/develop/projects/ -$ cd ~/develop/projects/ -``` - -### Clone, build and install odin-data - -* Clone `odin-data` repo from Github (ensuring the latest tagged release is used): -``` -$ git clone -b 1.4.0 https://github.com/odin-detector/odin-data.git -Initialized empty Git repository in /develop/projects/odin-demo/odin-data/.git/ -remote: Counting objects: 5189, done. -remote: Compressing objects: 100% (69/69), done. -remote: Total 5189 (delta 57), reused 81 (delta 47), pack-reused 5073 -Receiving objects: 100% (5189/5189), 1.50 MiB | 1.23 MiB/s, done. -Resolving deltas: 100% (3440/3440), done. -``` - -* Create an `install` directory to install `odin-data` and plugins into: -``` -mkdir install -``` - -* Create a build directory for CMake to use N.B. ODIN uses CMake ***out-of-source*** build semantics: -``` -$ cd odin-data -$ mkdir build && cd build -``` - -* Configure CMake to define use of correct packages and set up install directory: -``` -$ cmake -DBoost_NO_BOOST_CMAKE=ON -DBOOST_ROOT=$BOOST_ROOT \ - -DZEROMQ_ROOTDIR=$ZEROMQ_ROOT -DLOG4CXX_ROOT_DIR=$LOG4CXX_ROOT \ - -DHDF5_ROOT=$HDF5_ROOT -DBLOSC_ROOT_DIR=$BLOSC_ROOT \ - -DCMAKE_INSTALL_PREFIX=~/develop/projects//install .. -``` -(This is verbose and error-prone but you only have to do it once per setup). - -* Check output from CMake for errors and correct paths: -``` --- The C compiler identification is GNU 4.4.7 --- The CXX compiler identification is GNU 4.4.7 --- Check for working C compiler: /usr/bin/cc --- Check for working C compiler: /usr/bin/cc -- works --- Detecting C compiler ABI info --- Detecting C compiler ABI info - done --- Check for working CXX compiler: /usr/bin/c++ --- Check for working CXX compiler: /usr/bin/c++ -- works --- Detecting CXX compiler ABI info --- Detecting CXX compiler ABI info - done --- Boost version: 1.48.0 --- Found the following Boost libraries: --- program_options --- system --- filesystem --- unit_test_framework --- date_time --- thread - -Looking for log4cxx headers and libraries --- Root dir: /aeg_sw/tools/CentOS6-x86_64/log4cxx/0-10-0/prefix --- Found PkgConfig: /usr/bin/pkg-config (found version "0.23") --- Found LOG4CXX: /usr/lib64/liblog4cxx.so --- Include directories: /usr/include/log4cxx --- Libraries: /usr/lib64/liblog4cxx.so - -Looking for ZeroMQ headers and libraries --- Root dir: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix --- checking for one of the modules 'libzmq' --- Found ZEROMQ: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/lib/libzmq.so --- Include directories: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/include --- Libraries: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/lib/libzmq.so - -Looking for pcap headers and libraries --- Found PCAP: /usr/lib64/libpcap.so (Required is at least version "1.4.0") --- Performing Test PCAP_LINKS --- Performing Test PCAP_LINKS - Success - -Looking for blosc headers and libraries --- Searching Blosc Root dir: /aeg_sw/tools/CentOS7-x86_64/c-blosc/1-14-4/prefix --- Found Blosc: /aeg_sw/tools/CentOS7-x86_64/c-blosc/1-14-4/prefix/lib/libblosc.so - -Searching for HDF5 --- HDF5_ROOT set: /aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix -Determining odin-data version --- Git describe version: 0.7.0 --- major:0 minor:7 patch:0 sha1:0.7.0 --- short version: 0.7.0 --- HDF5 include files: /aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix/include --- HDF5 libs: /aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix/lib/libhdf5.so/aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix/lib/libhdf5_hl.so --- HDF5 defs: --- Found Doxygen: /usr/bin/doxygen (found version "1.8.5") --- Configuring done --- Generating done --- Build files have been written to: /develop/projects/odin-demo/odin-data/build -``` -* This creates all the directories, makefiles etc needed to compile: -``` -$ tree -d -L 1 -. -├── bin -├── CMakeFiles -├── common -├── config -├── doc -├── frameProcessor -├── frameReceiver -├── lib -└── tools -``` -* Compile odin-data: -``` -$ make -j4 -``` -* Produces a lot of output first time through: -``` -Scanning dependencies of target CopyPythonToolModules -Scanning dependencies of target CopyTestConfigs -Scanning dependencies of target CopyClientMsgFiles -Scanning dependencies of target OdinData -[ 3%] [ 3%] Generating ../test_config/fp_log4cxx.xml -Generating ../../test_config/client_msgs/reconfig_endpoints.json -[ 4%] [ 6%] Generating ../test_config/fr_log4cxx.xml -Generating ../../test_config/client_msgs/reconfig_buffer_manager.json -[ 8%] [ 9%] Generating ../test_config/fr_test.config -Generating ../../test_config/client_msgs/config_ctrl_chan_port_5010.json -[ 11%] [ 13%] Generating ../test_config/fp_test.config -Generating ../../test_config/client_msgs/config_ctrl_chan_port_5000.json -[ 14%] [ 14%] Generating ../test_config/fr_test_osx.config -Built target CopyPythonToolModules -[ 16%] Generating ../../test_config/client_msgs/reconfig_rx_thread.json -[ 18%] Generating ../../test_config/client_msgs/reconfig_decoder.json -[ 19%] Generating ../test_config/fp_py_test.config -[ 21%] Generating ../test_config/fp_py_test_osx.config -[ 21%] Built target CopyClientMsgFiles -[ 22%] [ 24%] [ 26%] Generating ../test_config/fp_py_test_excalibur.config -Generating ../test_config/fr_excalibur1.config -Generating ../test_config/fr_excalibur2.config -[ 27%] [ 29%] Generating ../test_config/fp_excalibur1.config -Generating ../test_config/fp_excalibur2.config -[ 29%] Built target CopyTestConfigs -[ 31%] Building CXX object common/src/CMakeFiles/OdinData.dir/logging.cpp.o -[ 32%] [ 34%] Building CXX object common/src/CMakeFiles/OdinData.dir/SharedBufferManager.cpp.o -[ 36%] Building CXX object common/src/CMakeFiles/OdinData.dir/IpcReactor.cpp.o -Building CXX object common/src/CMakeFiles/OdinData.dir/IpcMessage.cpp.o -[ 37%] Building CXX object common/src/CMakeFiles/OdinData.dir/IpcChannel.cpp.o -Linking CXX shared library ../../lib/libOdinData.so -[ 37%] Built target OdinData - -<< snip >> - -Linking CXX executable ../../bin/frameReceiver -Linking CXX executable ../../bin/frameProcessor -[ 88%] Built target frameReceiver -[ 90%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameReceiverController.cpp.o -[ 91%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/DummyUDPFrameDecoderLib.cpp.o -[ 91%] Built target frameProcessor -[ 93%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameReceiverZMQRxThread.cpp.o -Linking CXX shared library ../../lib/libHdf5Plugin.so -[ 93%] Built target Hdf5Plugin -[ 95%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/DummyUDPFrameDecoder.cpp.o -[ 96%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameReceiverRxThread.cpp.o -[ 98%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameDecoder.cpp.o -Scanning dependencies of target frameProcessorTest -[100%] Building CXX object frameProcessor/test/CMakeFiles/frameProcessorTest.dir/FrameProcessorTest.cpp.o -Linking CXX executable ../../bin/frameReceiverTest -[100%] Built target frameReceiverTest -Linking CXX executable ../../bin/frameProcessorTest -[100%] Built target frameProcessorTest -``` - -* This compiles ***five*** applications into `build/bin`: -``` -$ tree bin -bin -├── frameProcessor -├── frameProcessorTest -├── frameReceiver -├── frameReceiverTest -└── frameSimulator - -0 directories, 5 files -``` - -* Run the unit test applications (optional): -``` -$ bin/frameReceiverTest -Running 32 test cases... - -*** No errors detected -$ bin/frameProcessorTest -bin/frameProcessorTest -Running 16 test cases... -TRACE - Frame constructed -TRACE - copy_data called with size: 24 bytes - -<< snip >> - -*** No errors detected -``` - -* Install `odin-data` into `/install` directory: -``` -$ make install -$ make install -[ 8%] Built target OdinData -[ 9%] Built target FrameReceiver -[ 13%] Built target DummyUDPFrameDecoder -[ 21%] Built target frameReceiver -[ 44%] Built target frameReceiverTest -[ 55%] Built target FrameProcessor -[ 57%] Built target DummyPlugin -[ 63%] Built target Hdf5Plugin -[ 68%] Built target frameProcessor -[ 70%] Built target frameProcessorTest -[ 70%] Built target CopyPythonToolModules -[ 90%] Built target CopyTestConfigs -[100%] Built target CopyClientMsgFiles -Install the project... --- Install configuration: "" --- Installing: /develop/projects/odin-demo/install/include/ClassLoader.h - -<< snip >> - -"/develop/projects/odin-demo/install/lib:/aeg_sw/tools/CentOS6-x86_64/boost/1-48-0/prefix/lib:/aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/lib:/aeg_sw/tools/CentOS6-x86_64/hdf5/1-10-0/prefix/lib" -``` - ### Clone, build and install project-specific plugins * Using EXCALIBUR as example - can be used as template and renamed