Edge intelligence packages, examples and unit tests by lucifercr07 · Pull Request #10 · KohliDev/liota

lucifercr07 · 2017-11-03T10:20:11Z

Added ModelPath to conf file for the edge_intelligence_models e.g windmill_iotcc_pfa.py etc.
Removed the training files from edge_intelligence_models i.e eval-build.py and build-model.py
Added readme information to the edge_intelligence_models.
Removed file_reader_package.py from edge_component
Added comments for actuator in edge_intelligence_models e.g windmill_iotcc_pfa.py etc.
Added loadModel for the tensor_flow_edge_component.py
Added files windmill_graphite_tf.py and windmill_iotcc_tf.py to the edge_intelligence_models.
Added rule_edge_component
Added windmill_graphite_rule.py
Added packages for edge_components (sklearn_edge_component, pfa_component,
tensorflow_edge_component, rule_edge_component.py).
Example packages for each edge_component (graphite_pfa_windmill_simulated.py,
graphite_rule_thermistor_simulated.py, graphite_rule_windmill_simulated.py,
graphite_sklearn_windmill_simulated.py, graphite_tensorflow_windmill_simulated.py).
Tests for edge_components under tests folder.

KohliDev · 2017-11-03T12:38:51Z

@@ -0,0 +1 @@
+{"@": "PrettyPFA document", "name": "Engine_1", "input": "double", "output": "double", "method": "map", "action": [{"@": "PrettyPFA line 14", "let": {"baseLL": {"@": "PrettyPFA line 14", "prob.dist.gaussianLL": ["input", {"@": "PrettyPFA line 14", "cell": "baseline", "path": [{"@": "PrettyPFA line 14", "string": "mu"}]}, {"@": "PrettyPFA line 14", "cell": "baseline", "path": [{"@": "PrettyPFA line 14", "string": "sig"}]}]}}}, {"@": "PrettyPFA line 15", "let": {"altLL": {"@": "PrettyPFA line 15", "prob.dist.gaussianLL": ["input", {"@": "PrettyPFA line 15", "cell": "alternate", "path": [{"@": "PrettyPFA line 15", "string": "mu"}]}, {"@": "PrettyPFA line 15", "cell": "alternate", "path": [{"@": "PrettyPFA line 15", "string": "sig"}]}]}}}, {"@": "PrettyPFA lines 16-17", "cell": "cusum", "to": {"params": [{"old": "double"}], "ret": "double", "do": [{"@": "PrettyPFA line 17", "stat.change.updateCUSUM": [{"@": "PrettyPFA line 17", "-": ["altLL", "baseLL"]}, "old", {"@": "PrettyPFA line 17", "cell": "reset"}]}]}}], "cells": {"cusum": {"@": "PrettyPFA line 8", "type": "double", "init": 0.0, "shared": true, "rollback": false}, "reset": {"@": "PrettyPFA line 9", "type": "double", "init": 0.0, "shared": false, "rollback": false}, "alternate": {"@": "PrettyPFA line 11", "type": {"fields": [{"type": "double", "name": "mu"}, {"type": "double", "name": "sig"}], "type": "record", "name": "Record_1"}, "init": {"mu": 24.23076923076923, "sig": 9.420276925706693}, "shared": false, "rollback": false}, "baseline": {"@": "PrettyPFA line 10", "type": "Record_1", "init": {"mu": 10.448598130841122, "sig": 2.0227690446443916}, "shared": false, "rollback": false}}}


analytics_model.pfa

KohliDev · 2017-11-03T12:41:06Z

+
+This folder contains [PFA](http://dmg.org/pfa/docs/motivation/), [scikit-learn](http://scikit-learn.org/) and [TensorFlow](https://www.tensorflow.org/) and RuleEdgeComponent examples.
+Trained [PFA](http://dmg.org/pfa/docs/motivation/), [scikit-learn](http://scikit-learn.org/) and [TensorFlow](https://www.tensorflow.org/) models are present under the trained_models directory.
+The generated data is using the [TI SensorTag](http://www.ti.com/ww/en/wireless_connectivity/sensortag/) which has been used for training the windmill model.


Remove extra TI SensorTag link.

KohliDev · 2017-11-03T12:43:34Z

@@ -0,0 +1,34 @@
+# Edge Intelligence Models


README in capitals everywhere

KohliDev · 2017-11-03T12:50:03Z

@@ -0,0 +1,77 @@
+# -*- coding: utf-8 -*-


KohliDev · 2017-11-03T12:50:47Z

@@ -0,0 +1,76 @@
+# -*- coding: utf-8 -*-


KohliDev · 2017-11-03T12:54:56Z

@@ -0,0 +1,31 @@
+30,1


windmill_tf_test
windmill_tf_train

KohliDev · 2017-11-03T12:56:06Z

+import os
+import numpy as np
+import tensorflow as tf
+


Independent of Liota.

Should we have this file as for sklearn and pfa model there is no such file which shows how the model was trained or to check it's accuracy etc.?

It's okay to keep one file to showcase how training is done.

KohliDev · 2017-11-03T14:51:29Z

+	def load_model(self):
+		pass
+
+	def set_properties(self, reg_entity, properties):


Remove set properties and remove abstract method

KohliDev · 2017-11-03T14:51:52Z

+
+    def __init__(self, model_path, actuator_udm):
+        super(SKLearnEdgeComponent, self).__init__(model_path, actuator_udm)
+        if not os.path.exists(model_path):


Add skv check

KohliDev · 2017-11-03T14:54:33Z

+import logging
+import Queue
+
+log = logging.getLogger(__name__)


Remove this

KohliDev · 2017-11-03T14:57:52Z

+			action_taken.put(value)
+
+		def get_action_taken():
+			return action_taken.get(block=True)


add pass and mention you can write your own login in comment

KohliDev · 2017-11-03T15:02:21Z

+        pass
+
+    def process(self, message):
+        self.actuator_udm(self.model.predict(message)[0])


Add a check for actuator_udm here

Added in edge_component.py

KohliDev · 2017-11-06T13:03:13Z

+import os
+import numpy as np
+import tensorflow as tf
+


It's okay to keep one file to showcase how training is done.

KohliDev · 2017-11-06T13:05:47Z

+			_rpm_list.append(int((abs(z_degree) * 0.16667)))
+	rpm_model_queue.put(rpm)
+	return rpm
+


Explain the queue approach for better understanding.

KohliDev · 2017-11-06T13:07:21Z

+
+# This example depicts how edge component can be used to take actions locally based on trained TensorFlow model
+# and actions can then be send to device using actuator_udm method defined here, which is currently just printing
+# actions on console, metrics are sent to Graphite DCC


The returned value as actions on console and metrics are being sent to Graphite DCC.

KohliDev · 2017-11-06T13:08:52Z

+# actuator_udm can be used to pass on the value to the actuator, as of now we are printing them
+def actuator_udm(value):
+	print value
+


Simulating windmill RPM which is a device connected to Edge.

KohliDev · 2017-11-06T13:48:08Z

+
+class PackageClass(LiotaPackage):
+
+    def run(self, registry):


The way package is written we might have to change and enforce actuator_udm and metrics to be passed in the package itself.

KohliDev · 2017-11-06T13:51:09Z

+		return np.array([message], dtype=np.float32)
+
+	def process(self, message):
+		prediction = list(self.model.predict_classes(input_fn=lambda:self.input_fn(message)))


Check for unregister

Done. unload package works now

KohliDev · 2017-11-06T13:52:36Z

+    __metaclass__ = ABCMeta
+
+    @abstractmethod
+    def __init__(self, model_path, actuator_udm):


Model_path check should be removed and put only in the packages needs it and ruleEdgeComponent will be implemented on basis of EdgeComponent.

Actutor_udm check will come here and the way packages are written will be changed.

KohliDev · 2017-11-06T13:53:13Z

+from liota.core.package_manager import LiotaPackage
+from liota.lib.utilities.utility import read_user_config
+
+dependencies = ["edge_systems/dell5k/edge_system"] 


Remove dependency

KohliDev · 2017-11-09T08:33:45Z

@@ -0,0 +1,139 @@
+# -*- coding: utf-8 -*-


You are publishing to graphite so rename the file to windmill_graphite_rule_edge.py

KohliDev · 2017-11-09T08:35:23Z

+
+# This example depicts how edge component can be used to take actions locally based on a rule defined
+# and actions can then be send to device using actuator_udm method defined here, which is currently just printing
+# the returned value as actions on console and metrics are being sent to Graphite DCC.


the returned value is from RuleEdgeComponent as actions. Only the gateway metrics are being published to Graphite DCC.

KohliDev · 2017-11-09T08:37:36Z

+
+# This example depicts how edge component can be used to take actions locally based on sklearn model
+# and actions can then be send to device using actuator_udm method defined here, which is currently just printing
+# the returned value as actions on console and metrics are being sent to Graphite DCC.


only the gateway metrics are being published to Graphite DCC.

KohliDev · 2017-11-09T08:38:29Z

@@ -0,0 +1,147 @@
+# -*- coding: utf-8 -*-


Add graphite in the name.

KohliDev · 2017-11-09T08:38:48Z

@@ -0,0 +1,145 @@
+# -*- coding: utf-8 -*-


Graphite in the name.

KohliDev · 2017-11-09T08:41:29Z

+		if met_cnt == 0:
+			return
+		for _ in range(met_cnt):	
+			m = reg_metric.values.get(block=True)


Rename "m", add comment

KohliDev · 2017-11-09T08:41:42Z

+			self.actuator_udm(0)
+
+	def _format_data(self, reg_metric):
+		met_cnt = reg_metric.values.qsize()


reg_metric_queue

As discussed since it's in abstract method in edge_component.py name not changed.

KohliDev · 2017-11-09T08:47:15Z

+action_taken = Queue.Queue()
+
+class PackageClass(LiotaPackage):
+	def create_udm(self, windmill_model):


We need to discuss this piece of code.

KohliDev · 2017-11-09T09:07:30Z

+	def build_model(self):
+		pass
+
+


Remove extra lines

KohliDev · 2017-11-09T09:12:13Z

+class PackageClass(LiotaPackage):
+	def create_udm(self, windmill_model):
+
+		def get_rpm():


get vibration

Rename the function

Edge intelligence packages, examples and unit tests

04377cf

KohliDev reviewed Nov 3, 2017

View reviewed changes

Correction after review

4d82df1

KohliDev reviewed Nov 3, 2017

View reviewed changes

KohliDev reviewed Nov 6, 2017

View reviewed changes

lucifercr07 added 5 commits November 6, 2017 19:35

Correction after review

ba9861c

Modifications to package files and examples

80933e5

Minor edits for examples

864ea96

Minor edits for examples

6eaf488

Added example package for pfa

44a6b28

KohliDev reviewed Nov 9, 2017

View reviewed changes

Modifications to packages and examples

bb4df52

		@@ -0,0 +1 @@
		{"@": "PrettyPFA document", "name": "Engine_1", "input": "double", "output": "double", "method": "map", "action": [{"@": "PrettyPFA line 14", "let": {"baseLL": {"@": "PrettyPFA line 14", "prob.dist.gaussianLL": ["input", {"@": "PrettyPFA line 14", "cell": "baseline", "path": [{"@": "PrettyPFA line 14", "string": "mu"}]}, {"@": "PrettyPFA line 14", "cell": "baseline", "path": [{"@": "PrettyPFA line 14", "string": "sig"}]}]}}}, {"@": "PrettyPFA line 15", "let": {"altLL": {"@": "PrettyPFA line 15", "prob.dist.gaussianLL": ["input", {"@": "PrettyPFA line 15", "cell": "alternate", "path": [{"@": "PrettyPFA line 15", "string": "mu"}]}, {"@": "PrettyPFA line 15", "cell": "alternate", "path": [{"@": "PrettyPFA line 15", "string": "sig"}]}]}}}, {"@": "PrettyPFA lines 16-17", "cell": "cusum", "to": {"params": [{"old": "double"}], "ret": "double", "do": [{"@": "PrettyPFA line 17", "stat.change.updateCUSUM": [{"@": "PrettyPFA line 17", "-": ["altLL", "baseLL"]}, "old", {"@": "PrettyPFA line 17", "cell": "reset"}]}]}}], "cells": {"cusum": {"@": "PrettyPFA line 8", "type": "double", "init": 0.0, "shared": true, "rollback": false}, "reset": {"@": "PrettyPFA line 9", "type": "double", "init": 0.0, "shared": false, "rollback": false}, "alternate": {"@": "PrettyPFA line 11", "type": {"fields": [{"type": "double", "name": "mu"}, {"type": "double", "name": "sig"}], "type": "record", "name": "Record_1"}, "init": {"mu": 24.23076923076923, "sig": 9.420276925706693}, "shared": false, "rollback": false}, "baseline": {"@": "PrettyPFA line 10", "type": "Record_1", "init": {"mu": 10.448598130841122, "sig": 2.0227690446443916}, "shared": false, "rollback": false}}} No newline at end of file

Conversation

lucifercr07 commented Nov 3, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment