Added support for metadata for job runs by maxses · Pull Request #208 · ohwgiles/laminar

maxses · 2024-03-27T21:42:49Z

With the laminarc tag command the build scripts can augment runs with metadata. This can be SCM-oriented information like the used branch or source-version.
The metadata is a list of key-value-pairs. The metadata is automatically shown in the run-view of a job in the web UI.

Each call of laminarc tag can assign one key/value pair.

Example:

laminarc tag $JOBNAME $NUMBER` \
		Branch "$(git -C $WORKSPACE branch --show-current)"

With the `laminarc tag` command the build scripts can augment runs with metadata. This can be SCM-oriented information like the used branch or source-version. The metadata is a list of key/value pairs. The metadata is automatically shown in the run-view of a job in the web UI. Each call of `laminarc tag` can assign one key/value pair. Example: ``` laminarc tag $JOBNAME $NUMBER` \ Branch "$(git -C $WORKSPACE branch --show-current)" ```

All keys from metainfo of all shown runs is concatenated. So only relevant colums for metadata is shown.

pkg-config is needed in docker image. "laminar.conf" has to be copied from "usr/etc/laminar.conf" to "etc/laminar.conf" in installation directory.

ohwgiles · 2024-03-28T22:25:53Z

 set(CMAKE_CXX_STANDARD 17)

 set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wextra -Wno-unused-parameter -Wno-sign-compare")
+set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=deprecated-declarations" )


What caused this? If it is not related to this change it should be dealt with separately. Also consider that only Debug builds get -Werror, so if this is really needed then it should be put in CMAKE_CXX_FLAGS_DEBUG

ohwgiles · 2024-03-28T22:26:37Z

    src/run.cpp
    src/server.cpp
    src/version.cpp
+    src/json.cpp


I don't think it is necessary to break this out into a separate file - see later comments for explanation

ohwgiles · 2024-03-28T22:29:01Z

+
+Example:
+```
+laminarc tag $JOBNAME $NUMBER Branch "$(git -C $WORKSPACE branch --show-current)"


Since we only intend this to be called from within the run, laminarc can handle reading $JOBNAME and $NUMBER, see for example what set does

ohwgiles · 2024-03-28T22:33:50Z

+       return run->tag(key, value);
+   else
+   {
+      LLOG(WARNING, "No active run with ", job, buildNum);


return false (propagates error to laminarc)

ohwgiles · 2024-03-28T22:39:01Z

+    // store metadata into run
+    bool tag(const std::string& key, const std::string& value);
+    // Return the meta as JSON text
+    std::string getMetaDataJsonString();


I don't think Run should know about JSON at all. Just do

const std::map<std::string, std::string>& metadata() const { return metadata; }

and do the conversion to JSON in the Laminar class with a local method there.

ohwgiles · 2024-03-28T22:42:50Z

    kj::PromiseFulfillerPair<RunState> finished;
    kj::ForkedPromise<RunState> finishedFork;
+
+    std::map<std::string, std::string> metaDataMap;


There is a minor consideration here: what order should the tags show up in the frontend? Using std::map means they will show up alphabetically, but I wonder if it would be preferable for them to show up in the order that they were created? Unfortunately I think that would require using vector<pair<string,string>> and manually checking for duplicates, or using boost::multi_index_container

Alphabetically is also fine

ohwgiles · 2024-03-28T22:43:46Z

+   LLOG(INFO, "Setting tag ", key, value );
+   metaDataMap[key]=value;
+   return true;
+}


make it void if it cannot fail

ohwgiles · 2024-03-28T22:48:20Z

+     <template v-show="Object.keys(job.metadata).length" v-for="(value, key) in job.metadata">
+       <dt>{{key}}</dt>
+       <dd>{{value}}</dd>
+     </template>


I think this would look better before / on top of the artifacts

ohwgiles · 2024-03-28T22:59:09Z

  <div style="grid-column: 1/-1">
+   <div v-show="false">{{metaDataKeys=["XBranch", "XVersion"]}}</div>
+   <div v-show="false">{{allJobs=jobsQueued.concat(jobsRunning).concat(jobsRecent)}}</div>
+   <div v-show="false">{{metaDataKeys=uniteMetadataKeys(allJobs)}}</div>


I think metadataKeys is best generated in the JS at the time the run list is received, and updated when a new run joins the list

ohwgiles · 2024-03-28T23:04:46Z

   <canvas id="chartBt"></canvas>
  </div>
  <div style="grid-column: 1/-1">
+   <div v-show="false">{{metaDataKeys=["XBranch", "XVersion"]}}</div>


ohwgiles · 2024-03-28T23:06:27Z

     <th class="text-center">Started <a class="sort" :class="(sort.field=='started'?sort.order:'')" v-on:click="do_sort('started')">&nbsp;</a></th>
     <th class="text-center">Duration <a class="sort" :class="(sort.field=='duration'?sort.order:'')" v-on:click="do_sort('duration')">&nbsp;</a></th>
     <th class="text-center vp-sm-hide">Reason <a class="sort" :class="(sort.field=='reason'?sort.order:'')" v-on:click="do_sort('reason')">&nbsp;</a></th>
+     <th class="text-center vp-sm-hide" v-for="(value, key) in metaDataKeys">{{key}}<a class="sort" :class="(sort.field=='reason'?sort.order:'')" v-on:click="do_sort('reason')">&nbsp;</a></th>


Do not sort by reason. It should be pretty simple to add sorting by metadata fields, using something like 'meta_' + key for sort.field and in the backend using ORDER BY metadata -> $key, but also happy just to not implement sorting for metadata

ohwgiles · 2024-03-28T23:10:03Z

 EOF
 echo /etc/laminar.conf > laminar/DEBIAN/conffiles
+mkdir -p laminar/etc
+mv laminar/usr/etc/laminar.conf laminar/etc/laminar.conf


I think this is a bug introduced elsewhere and this is not the right place to fix it

Fixed here 8c3d7f6

ohwgiles · 2024-03-28T23:10:13Z

 DOCKER_TAG=$(docker build -q - <<EOS
 FROM debian:11-slim
-RUN apt-get update && apt-get install -y wget cmake g++ capnproto libcapnp-dev rapidjson-dev libsqlite3-dev libboost-dev zlib1g-dev
+RUN apt-get update && apt-get install -y wget cmake g++ capnproto libcapnp-dev rapidjson-dev libsqlite3-dev libboost-dev zlib1g-dev pkg-config


djzort · 2025-11-08T13:59:50Z

This seems pretty useful. Hopefully it can be completed.

maxses force-pushed the feature/metadata branch from bba5a48 to 9c1cfad Compare March 27, 2024 22:43

maxses mentioned this pull request Mar 27, 2024

added support for build.env #207

Closed

maxses force-pushed the feature/metadata branch from 9c1cfad to accdff2 Compare March 27, 2024 23:20

maxses added 2 commits March 28, 2024 20:04

Show metainfo in the jobs run view

f6fe08b

All keys from metainfo of all shown runs is concatenated. So only relevant colums for metadata is shown.

Fixed building deb package for x86

4e01d14

pkg-config is needed in docker image. "laminar.conf" has to be copied from "usr/etc/laminar.conf" to "etc/laminar.conf" in installation directory.

ohwgiles reviewed Mar 28, 2024

View reviewed changes

Conversation

maxses commented Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

djzort commented Nov 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maxses commented Mar 27, 2024 •

edited

Loading