Skip to content

Updated odin-data.md #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ethan-Smith1 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Updated odin-data.md #11

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
373 changes: 72 additions & 301 deletions odin-data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# AEG ODIN data workshop
## 25 June 18
## 25 June 18 (Updated 11/09/25)

# Table of Contents

Expand Down Expand Up @@ -89,21 +89,79 @@ number of received packets, missing packets, timestamp etc
} FrameHeader;
```

### External Software Dependencies
## External Software Dependencies

The following libraries and packages are required:
The following libraries and packages are required however on our dev machines they are pre installed:

* [CMake](http://www.cmake.org) : build management system (version >= 2.8)
* [Boost](http://www.boost.org) : portable C++ utility libraries. The following components are
used - program_options, unit_test_framework, date_time, interprocess, bimap (version >= 1.41)
* [ZeroMQ](http://zeromq.org) : high-performance asynchronous messaging library (version >= 3.2.4)
* [Log4CXX](http://logging.apache.org/log4cxx/): Configurable message logger (version >= 0.10.0)
* [HDF5](https://www.hdfgroup.org/HDF5): __Optional:__ if found, the frame processor application
will be built (version >= 1.8.14)
* [LibPCAP](https://github.com/the-tcpdump-group/libpcap): TCPDUMP packet capture libraries - usually
installed with distro install tools. e.g. yum
* [blosc](https://github.com/Blosc/c-blosc): An fast compression library - needed for the frame
processor compression plugin (__Beware__: API change with version 1.13 -> 1.14)
* [CMake](http://www.cmake.org)
* [Boost](http://www.boost.org)
* [ZeroMQ](http://zeromq.org)
* [Log4CXX](http://logging.apache.org/log4cxx/)
* [HDF5](https://www.hdfgroup.org/HDF5)
* [LibPCAP](https://github.com/the-tcpdump-group/libpcap)
* [blosc](https://github.com/Blosc/c-blosc)

## Building odin-data and plugins

### Create development environment

* Create a project development dir (using AEG path convention if appropriate). This example uses the shared users folder.

```
$ mkdir -p /aeg_sw/work/users/<your_fedID>/develop/projects/<project-name>
$ cd /aeg_sw/work/users/<your_fedID>/develop/projects/<project-name>
```

### Clone, build and install odin-data

* Clone `odin-data` repo from Github (ensuring the latest tagged release is used):
```
$ git clone -b 1.11.0 https://github.com/odin-detector/odin-data.git
```

* Create an `install` directory to install `odin-data` and plugins into:
```
mkdir install
```

* Create a build directory for CMake to use in `cpp` in the `odin-data` directory:
```
$ cd odin-data/cpp
$ mkdir build && cd build
```

* Configure CMake to define use of correct packages and set up install directory:
```
$ cmake -DCMAKE_INSTALL_PREFIX= /aeg_sw/work/users/<your_fedID>/develop/projects/<project-name>/install ..
```


* Compile odin-data:
```
$ make -j
```

* Install the compiled project into the install directory
```
make install
```

* To test everything is working run either of the commands below in the install directory
```
./bin/frameReceiverTest
# or
./bin/frameProcessorTest
```
For example if you run `./bin/frameReceiverTest` should produce something like this:
```Running 34 test cases...
log4cxx: No appender could be found for logger (FR.DummyUDPFrameDecoder).
log4cxx: Please initialize the log4cxx system properly.
INFO - DummyTCPFrameDecoder version 1.11.0 loaded

*** No errors detected
```

# The content below below is potentially outdated:

## Demo using EXCALIBUR as example

Expand Down Expand Up @@ -168,293 +226,6 @@ I Sent 1 frames with a total of 132 packets and 1049648 bytes, dropping 0 packet

![EXCALIBUR image](images/excalibur.png)

## Building odin-data and plugins

### Create development environment

* Need to ensure the `aeg_sw` profile is loaded:
```
$ source /aeg_sw/etc/profile.sh
```
for `bash`-like shells, or
```
$ source /aeg_sw/etc/profile.csh
```
for `csh`-like shells.
* Allows you to use `environment-modules` to load various libraries, packages
etc into your environment. To see what's available:

```
$ module avail

------------------------------------- /aeg_sw/etc/modulefiles --------------------------------------
aeg_apps aeg_tools aeg_use_own

---------------------------------- /usr/share/Modules/modulefiles ----------------------------------
dot module-git module-info modules null use.own

------------------------------------ <home_dir>/.privatemodules ------------------------------------
odin-data

--------------------------------- /aeg_sw/apps/Modules/modulefiles ---------------------------------
arduino/1-8-3 dawn/2-5-0 eclipse/472(default) pycharm/2017-1-4
cmake/3-11-4 dawn/2-7-0 eclipse/472_20171218

------------------------- /aeg_sw/tools/CentOS7-x86_64/Modules/modulefiles -------------------------
blosc/1-13-5 git/2-13-2 python/2 python/3-6 ruby/2-4 xilinx/arm/2015-2
blosc/1-14-4 hdf5/1-10-0 python/2-7 python/3-6-2 ruby/2-4-1 zeromq/4-2-1
boost/1-48-0 hdf5/1-10-4 python/2-7-13 python/3-6-5 spark/2-3-0 zeromq/4-2-3
boost/1-66-0 log4cxx/0-10-0 python/2-7-15 qt/4-8-7 xilinx/2015-2
boost/1-67-0 msgpack-c/2-1-5 python/3 ruby/2 xilinx/2017-2
```

* Load the correct modules for odin-development:
```
$ module load boost/1-67-0 hdf5/1-10-0 zeromq/4-2-3 log4cxx/0-10-0 blosc/1-14-4
```
* Sets up your shell environment with various paths etc:
```
$ echo $BOOST_ROOT
/aeg_sw/tools/CentOS6-x86_64/boost/1-48-0/prefix
$ echo $HDF5_ROOT
/aeg_sw/tools/CentOS6-x86_64/hdf5/1-10-0/prefix
$ echo $ZEROMQ_ROOT
/aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix
$ echo $LOG4CXX_ROOT
/aeg_sw/tools/CentOS6-x86_64/log4cxx/0-10-0/prefix
$ echo $BLOSC_ROOT
/aeg_sw/tools/CentOS7-x86_64/c-blosc/1-14-4/prefix
```

* Create a project development dir (using AEG path convention if appropriate)

```
$ mkdir -p ~/develop/projects/<project-name>
$ cd ~/develop/projects/<project-name>
```

### Clone, build and install odin-data

* Clone `odin-data` repo from Github (ensuring the latest tagged release is used):
```
$ git clone -b 1.4.0 https://github.com/odin-detector/odin-data.git
Initialized empty Git repository in <home_dir>/develop/projects/odin-demo/odin-data/.git/
remote: Counting objects: 5189, done.
remote: Compressing objects: 100% (69/69), done.
remote: Total 5189 (delta 57), reused 81 (delta 47), pack-reused 5073
Receiving objects: 100% (5189/5189), 1.50 MiB | 1.23 MiB/s, done.
Resolving deltas: 100% (3440/3440), done.
```

* Create an `install` directory to install `odin-data` and plugins into:
```
mkdir install
```

* Create a build directory for CMake to use N.B. ODIN uses CMake ***out-of-source*** build semantics:
```
$ cd odin-data
$ mkdir build && cd build
```

* Configure CMake to define use of correct packages and set up install directory:
```
$ cmake -DBoost_NO_BOOST_CMAKE=ON -DBOOST_ROOT=$BOOST_ROOT \
-DZEROMQ_ROOTDIR=$ZEROMQ_ROOT -DLOG4CXX_ROOT_DIR=$LOG4CXX_ROOT \
-DHDF5_ROOT=$HDF5_ROOT -DBLOSC_ROOT_DIR=$BLOSC_ROOT \
-DCMAKE_INSTALL_PREFIX=~/develop/projects/<project-name>/install ..
```
(This is verbose and error-prone but you only have to do it once per setup).

* Check output from CMake for errors and correct paths:
```
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is GNU 4.4.7
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Boost version: 1.48.0
-- Found the following Boost libraries:
-- program_options
-- system
-- filesystem
-- unit_test_framework
-- date_time
-- thread

Looking for log4cxx headers and libraries
-- Root dir: /aeg_sw/tools/CentOS6-x86_64/log4cxx/0-10-0/prefix
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.23")
-- Found LOG4CXX: /usr/lib64/liblog4cxx.so
-- Include directories: /usr/include/log4cxx
-- Libraries: /usr/lib64/liblog4cxx.so

Looking for ZeroMQ headers and libraries
-- Root dir: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix
-- checking for one of the modules 'libzmq'
-- Found ZEROMQ: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/lib/libzmq.so
-- Include directories: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/include
-- Libraries: /aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/lib/libzmq.so

Looking for pcap headers and libraries
-- Found PCAP: /usr/lib64/libpcap.so (Required is at least version "1.4.0")
-- Performing Test PCAP_LINKS
-- Performing Test PCAP_LINKS - Success

Looking for blosc headers and libraries
-- Searching Blosc Root dir: /aeg_sw/tools/CentOS7-x86_64/c-blosc/1-14-4/prefix
-- Found Blosc: /aeg_sw/tools/CentOS7-x86_64/c-blosc/1-14-4/prefix/lib/libblosc.so

Searching for HDF5
-- HDF5_ROOT set: /aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix
Determining odin-data version
-- Git describe version: 0.7.0
-- major:0 minor:7 patch:0 sha1:0.7.0
-- short version: 0.7.0
-- HDF5 include files: /aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix/include
-- HDF5 libs: /aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix/lib/libhdf5.so/aeg_sw/tools/CentOS7-x86_64/hdf5/1-10-4/prefix/lib/libhdf5_hl.so
-- HDF5 defs:
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.5")
-- Configuring done
-- Generating done
-- Build files have been written to: <home_dir>/develop/projects/odin-demo/odin-data/build
```
* This creates all the directories, makefiles etc needed to compile:
```
$ tree -d -L 1
.
├── bin
├── CMakeFiles
├── common
├── config
├── doc
├── frameProcessor
├── frameReceiver
├── lib
└── tools
```
* Compile odin-data:
```
$ make -j4
```
* Produces a lot of output first time through:
```
Scanning dependencies of target CopyPythonToolModules
Scanning dependencies of target CopyTestConfigs
Scanning dependencies of target CopyClientMsgFiles
Scanning dependencies of target OdinData
[ 3%] [ 3%] Generating ../test_config/fp_log4cxx.xml
Generating ../../test_config/client_msgs/reconfig_endpoints.json
[ 4%] [ 6%] Generating ../test_config/fr_log4cxx.xml
Generating ../../test_config/client_msgs/reconfig_buffer_manager.json
[ 8%] [ 9%] Generating ../test_config/fr_test.config
Generating ../../test_config/client_msgs/config_ctrl_chan_port_5010.json
[ 11%] [ 13%] Generating ../test_config/fp_test.config
Generating ../../test_config/client_msgs/config_ctrl_chan_port_5000.json
[ 14%] [ 14%] Generating ../test_config/fr_test_osx.config
Built target CopyPythonToolModules
[ 16%] Generating ../../test_config/client_msgs/reconfig_rx_thread.json
[ 18%] Generating ../../test_config/client_msgs/reconfig_decoder.json
[ 19%] Generating ../test_config/fp_py_test.config
[ 21%] Generating ../test_config/fp_py_test_osx.config
[ 21%] Built target CopyClientMsgFiles
[ 22%] [ 24%] [ 26%] Generating ../test_config/fp_py_test_excalibur.config
Generating ../test_config/fr_excalibur1.config
Generating ../test_config/fr_excalibur2.config
[ 27%] [ 29%] Generating ../test_config/fp_excalibur1.config
Generating ../test_config/fp_excalibur2.config
[ 29%] Built target CopyTestConfigs
[ 31%] Building CXX object common/src/CMakeFiles/OdinData.dir/logging.cpp.o
[ 32%] [ 34%] Building CXX object common/src/CMakeFiles/OdinData.dir/SharedBufferManager.cpp.o
[ 36%] Building CXX object common/src/CMakeFiles/OdinData.dir/IpcReactor.cpp.o
Building CXX object common/src/CMakeFiles/OdinData.dir/IpcMessage.cpp.o
[ 37%] Building CXX object common/src/CMakeFiles/OdinData.dir/IpcChannel.cpp.o
Linking CXX shared library ../../lib/libOdinData.so
[ 37%] Built target OdinData

<< snip >>

Linking CXX executable ../../bin/frameReceiver
Linking CXX executable ../../bin/frameProcessor
[ 88%] Built target frameReceiver
[ 90%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameReceiverController.cpp.o
[ 91%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/DummyUDPFrameDecoderLib.cpp.o
[ 91%] Built target frameProcessor
[ 93%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameReceiverZMQRxThread.cpp.o
Linking CXX shared library ../../lib/libHdf5Plugin.so
[ 93%] Built target Hdf5Plugin
[ 95%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/DummyUDPFrameDecoder.cpp.o
[ 96%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameReceiverRxThread.cpp.o
[ 98%] Building CXX object frameReceiver/test/CMakeFiles/frameReceiverTest.dir/__/src/FrameDecoder.cpp.o
Scanning dependencies of target frameProcessorTest
[100%] Building CXX object frameProcessor/test/CMakeFiles/frameProcessorTest.dir/FrameProcessorTest.cpp.o
Linking CXX executable ../../bin/frameReceiverTest
[100%] Built target frameReceiverTest
Linking CXX executable ../../bin/frameProcessorTest
[100%] Built target frameProcessorTest
```

* This compiles ***five*** applications into `build/bin`:
```
$ tree bin
bin
├── frameProcessor
├── frameProcessorTest
├── frameReceiver
├── frameReceiverTest
└── frameSimulator

0 directories, 5 files
```

* Run the unit test applications (optional):
```
$ bin/frameReceiverTest
Running 32 test cases...

*** No errors detected
$ bin/frameProcessorTest
bin/frameProcessorTest
Running 16 test cases...
TRACE - Frame constructed
TRACE - copy_data called with size: 24 bytes

<< snip >>

*** No errors detected
```

* Install `odin-data` into `<project-name>/install` directory:
```
$ make install
$ make install
[ 8%] Built target OdinData
[ 9%] Built target FrameReceiver
[ 13%] Built target DummyUDPFrameDecoder
[ 21%] Built target frameReceiver
[ 44%] Built target frameReceiverTest
[ 55%] Built target FrameProcessor
[ 57%] Built target DummyPlugin
[ 63%] Built target Hdf5Plugin
[ 68%] Built target frameProcessor
[ 70%] Built target frameProcessorTest
[ 70%] Built target CopyPythonToolModules
[ 90%] Built target CopyTestConfigs
[100%] Built target CopyClientMsgFiles
Install the project...
-- Install configuration: ""
-- Installing: <home_dir>/develop/projects/odin-demo/install/include/ClassLoader.h

<< snip >>

"<home_dir>/develop/projects/odin-demo/install/lib:/aeg_sw/tools/CentOS6-x86_64/boost/1-48-0/prefix/lib:/aeg_sw/tools/CentOS6-x86_64/zeromq/4-2-1/prefix/lib:/aeg_sw/tools/CentOS6-x86_64/hdf5/1-10-0/prefix/lib"
```

### Clone, build and install project-specific plugins

* Using EXCALIBUR as example - can be used as template and renamed
Expand Down