-
Notifications
You must be signed in to change notification settings - Fork 16
Usage Walkthroughs
This document contains a series of walkthroughs, starting from the most simple case and moving into more complex scenarios.
Even if you have IoTEdge installed on a linux VM, you still need to run some configuration scripts to prepare for running our E2E tests.
Warning: These scripts will update your IoTEdge instance. In particular, if you have device identities that you want to preserve, you may want to back up the contents of your /etc/iotedge/config.yaml before you run through this process.
The setup process is already documented in the README.md document at the root of the edge_e2e project. The walkthrough here is the quick version.
IOTHUB_E2E_REPO_ADDRESSIOTHUB_E2E_REPO_USERIOTHUB_E2E_REPO_PASSWORD
If you want to run a private build of the edgeHub and edgeAgent containers, you need to set these variables
IOTHUB_E2E_EDGE_PRIVATE_REGISTRYIOTHUB_E2E_EDGE_PRIVATE_AGENTIMAGEIOTHUB_E2E_EDGE_PRIVATE_HUBIMAGE
Also, set IOTHUB_E2E_CONNECTION_STRING to the IotHub instance you want to use for your tests.
You need to run this script even if you're already running IoTEdge. This script was written so it can be called multiple times without causing damage.
This script:
- Sets up Python 3.6 and makes sure that it gets called with the
pythoncommand. - Sets up the Moby container runtime.
- Installs the IoTEdge daemon
- Starts up a private Docker container registry inside of a Docker container
- Pre-fetches some container images
Even if you've already created IoTEdge devices, you're better off using this script to create a new one. Every time you run his script, you create new registry identities.
Run iot-sdks-e2e-fx/scripts/create_new_edgehub_device.sh. This will create new registry identities for your IotEdge device along with a registry identity for a friend device that is used by the test scripts. It edits config.yaml to insert the device connection string for you, deploys a default set of modules, and restarts the iotedge daemon.
After this script is run, iotedge should be running with 4 modules: edgeAgent, edgeHub, nodeMod, and friendMod. nodeMod and friendMod both use a known good build of the node.js test wrappers. The intention is that nodeMod will be used as an SDK-under-test for your initial runs, but will soon be replaced by whatever SDK code you with to test. friendMod, on the other hand, will not be replaced. It will always use a known-good build of the node.js wrappers. Its purpose is to act as "the other actor" for a bunch of the tests. So, for instance, if a test needs to send an event to another module, friendMod will be that other module. Likewise, if your wrapper is missing an implementation (like deviceClient or eventHubClient), the test scripts will use friendMod to instantiate those objects. So, for example, if your wrapper implements deviceClient, the test case test_device_method_from_module_to_friend_device will use your SDK wrapper for the deviceClient object that receives the device method call. If your SDK does not implement deviceClient,the test scripts will use the node.js implementation of deviceClient for the object that receives the method call.
The Edge E2E tests is a suite of tests that exercise module functionality which can run against any language SDK under 6 possible scenarios:
- AMQP against IoTHub
- AMQP-WS against IoTHub
- MQTT against IoTHub
- MQTT-WS against IoTHub
- AMQP against EdgeHub
- MQTT against EdgeHub
When tests are run against IoTHub, the following things happen:
- Test cases are skipped if the functionality isn't supported when connecting directly to IoT hub (such as output events and method invoke)
- The
moduleClientis created using a connection string, and this connection string does not include aGatewayHostNamevalue. - Even though we're not connecting through
EdgeHub, we run using the wrapper that is running inside the container that is being managed by IoTEdge.
Point #3 may be confusing because we're using code that is inside an IoTEdge container, but we are not connecting that code to IoTEdge. A more sensible architecture would have us running this code in a container that is completely decoupled from IoTEdge, but that's not what we have today. It is safe to say, however, that the fact that IoTEdge is managing the lifetime of our test wrapper doesn't impact the tests if we're not connecting that container through edgeHub.
When tests are run against EdgeHub, the following happens:
- A full set of tests is executed. There may be one or two test cases skipped because of transient bugs, but this should be considered an exception rather than the rule.
- The
moduleClientis connected using the IoTEdge HSM environment (formerly known as "edgelet" and "edge daemon")
There is one other sceanrio that we can run, which is testing code through edgeHub, but outside of an IoTEdge container. This is used primarily for debugging. In this case, we connect with a connection string that includes the GatewayHostName value, and a CA certificate which was manually extracted from the edge environment. This functionality is enabled when we pass the --local option on the pytest command line.
The best way to run tests is to call pytest manually inside the iot-sdks-e2e-fx/test-runner folder.
- This assumes that your test container has already been built and deployed.
- It does not provide log files in any clean format and does not produce
JUnitresults - It only runs a single scenario
- It is able to run a subset of tests
- It is able to run using code inside a local debugger instead of running against a container (handy for debugging).
- It is able to test against IoTHub or
edgeHub, using connection string and extracted CA (if needed) - It can be run from a Linux bash prompt or a Windows command prompt on any machine. It does not need to be run on the machine that hosts your IoTEdge instance. (though, if it testing communication through edgeHub, it will need to be able to connect through your IoTEdge instance.)
The walkthroughs below cover some of the more common possible scenarios that you could run, but this is not the full list of possibilities.
To run this walkthrough, we will use the bash scripts in iot-sdks-e2e-fx/scripts to deploy our containers and we'll run the full set of tests by calling pytest manually.
- use iot-sdks-e2e-fx/scripts/deploy-test-container.sh to deploy your container by specifying the language that you want to deploy as a double-dash option (e.g.
--cfor c and--pythonfor python). Ex.:
iot-sdks-e2e-fx/scripts/deploy-test-container.sh --c
This script will also automatically deploy the friend module, which is always used for testing module-to-module communication. Note: if this is the first time you're using a particular LKG module, it may take a while for the edgeAgent to download and start the module (perhaps up to 10 minutes). You should not continue with the next step until you use the iotedge or docker CLI tools to verify that the module is running.
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/scripts$ ./deploy-test-containers.sh --c
Operating with device bertk-edge-ga-6_bertk_550 on on hub bertk-edge.azure-devices.net
Deploying the following containers:
friend: <REDACTED>/default-friend-module
c: <REDACTED>/c-e2e-v3
{
"$edgeAgent": {
<--snip-->
- Verify that your module is running. In this example, we can see that cMod is running, so we're good to continue:
iotedge list
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/scripts$ iotedge list
NAME STATUS DESCRIPTION CONFIG
cMod running Up 14 minutes <REDACTED>/c-e2e-v3
friendMod running Up 2 days <REDACTED>/default-friend-module
edgeHub running Up 2 days mcr.microsoft.com/azureiotedge-hub:1.0
edgeAgent running Up 2 days mcr.microsoft.com/azureiotedge-agent:1.0
- run pytest inside the iot-sdks-e2e-fx/test-runner folder. Here are some examples:
| command line | meaning |
|---|---|
pytest --csharp-wrapper --scenario=edgehub_module_client |
Use the csharp module to run against edgeHub using MQTT (the default). |
pytest --c-wrapper --scenario=iothub_module_client --transport=amqpws |
Use the C module to test against IoTHub using AMQP with websockets |
pytest --java-wrapper --scenario=edgehub_module_client --transport=amqp |
Use the Java module to test against edgeHub using AMQP |
At the end of the run, you will see your results.
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/test-runner$ pytest --node-wrapper --transport=amqp
========================== test session starts ==========================
platform linux -- Python 3.6.5, pytest-3.8.0, py-1.6.0, pluggy-0.7.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/bertk/repos/iot-sdks-e2e-fx/test-runner, inifile: pytest.ini
plugins: timeout-1.3.2
<--snip-->
test_module_twin_reported_properties.py::test_module_can_set_reported_properties_and_service_can_retrieve_them PASSED [100%]
========================== 29 passed in 173.71 seconds ==========================
Exception ignored in: <bound method ApiClient.__del__ of <swagger_client.api_client.ApiClient object at 0x7f50fb4b5c88>>
Traceback (most recent call last):
<--snip-->
An aside: If something goes wrong, you can see what is happening by looking at the docker logs (e.g. docker logs -f cMod )
The line "29 passed" is the important information. The Exception that happens after the run is complete can be ignored.
Running a single test inside of a container is just as easy as running an entire scenario and very similar to Walkthrough #1 above. Once your container is deployed (either LKG, like Walkthrough #1, or custom built, like Walkthrough #2), you just need to pass the filename for the test case and possibly the name of the test case along with the pytest command line.
All of the tests are inside the iot-sdks-e2e-fx/test-runner folder. All modules with tests inside are named with a test_ prefix, and all test cases are inside those files as functions that are named with test_ prefixes.
If you want to run the tests in a single .py module, you would call pytest passing the name of the .py file as follows
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/test-runner$ pytest --node-wrapper --transport=amqp test_module_send_telemetry.py
================================================= test session starts ==================================================
<--snip-->
test_module_send_telemetry.py::test_module_send_event_to_iothub PASSED [100%]
=============================================== 1 passed in 3.57 seconds ===============================================
If you want to run a single test, use the test_file_name.py::test_function_name syntax as follows:
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/test-runner$ pytest --node-wrapper --transport=amqp test_module_twin_desired_properties.py::test_service_can_set_multiple_desired_property_patches_and_module_can_retrieve_them_as_events
================================================= test session starts ==================================================
<--snip-->
test_module_twin_desired_properties.py::test_service_can_set_multiple_desired_property_patches_and_module_can_retrieve_them_as_events PASSED [100%]
=============================================== 1 passed in 5.65 seconds ===============================================
If you want to add additional debug spew from the test script, you can pass -s to pytest. Note: this will only give you output from the test runner. If you want output from the SDK code, you will need to inspect the docker logs for that container.
Running your SDK code inside of a debugger is almost as easy as running inside of a container, but you have to do some pre-configuration because you'll have the following configuration. Note, this configuration assumes that your debugger is running on the Windows machine that hosts your IoTEdge VM. This machine layout isn't technically required, but it's easy and it's the only configuration I've ever personally used. It's possible to run your debugger on a different physical machine, but you may need to do some extra network configuration to expose the correct ports from your IoTEdge machine, including getting them through your firewall.
Host Machine: Windows
- Hosts the Linux VM that's running IoTEdge under Hyper-V
- Runs debugger with SDK wrapper
- Runs test scripts (from
iot-sdks-e2e-fx/test-runner) viapytestfrom command line. - SDK wrapper connects to edgeHub instance on Linux VM via environment variables (connection string and CA certificate)
Client VM: Linux
- Runs IoTEdge daemon and IoTEdge containers
- Container for the SDK that you're debugging is technically running (it shows up in
iotedge info), but inert (not connected to edgeHub) - Containers for
friendMod,edgeHub, andedgeAgentare still running and active inside the IoTEdge container environment.
These steps assuming you've already called create-new-edgehub-device.sh to configure your IotEdge instance.
- Make sure you have Python 3.6 installed on your Windows machine and in the path along with the scripts folder. Since my machine doesn't have Python3.6 always in the path, I use
set path=%path%;c:\Python36;c:\Python36\scriptsIf you need to verify your Python install, you should verify both python and pip as follows. The version of python might be slightly different, but it should be OK as long as it's 3.6.x+. The version of pip is less important, but it should have the(python 3.6)info at the end.
F:\repos\iot-sdks-e2e-fx\test-runner>python --version
Python 3.6.3
F:\repos\iot-sdks-e2e-fx\test-runner>pip --version
pip 18.0 from c:\python36\lib\site-packages\pip (python 3.6)
- Clone the
iot-sdks-e2e-fxrepo on your Windows machine. - On your Windows machine, go to the
test-runnerfolder. Runpip install -r requirements.txt. This will grab the various python libraries that the test runner needs. - On your Linux machine, run
iot-sdks-e2e-fx/scripts/get-environment.sh windows. This gives you some handy text that you can copy/paste into your Windows environment to point the test runner at your Linux IoTEdge instance.
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/scripts$ ./get-environment.sh windows
@rem Set the following values in your environment:
@rem Environment for with device bertk-edge-ga-6_bertk_550 on on hub bertk-edge.azure-devices.net
@rem device_connection_string: "<REDACTED>"
set IOTHUB_E2E_CONNECTION_STRING=<REDACTED>
set IOTHUB_E2E_EDGEHUB_DNS_NAME=<REDACTED>
set IOTHUB_E2E_EDGEHUB_DEVICE_ID=<REDACTED>
set IOTHUB_E2E_EDGEHUB_CA_CERT=<REDACTED>
(The CA_CERT value will be a long Base64-encoded string)
-
On your Windows machine, make sure the environment variables from Step 4 are set in whatever environment you're going to run pytest from. It does not need to be set in your debugger environment. pytest will fail quickly if these variables are missing.
-
On you Linux machine, make sure the container for your SDK is deployed. Again, this container is inert, but it still needs to be present because the test runner needs it to exist as a module in the IoTHub registry. Again, it may take a while to download the Docker container image if this is the first time using it. Don't continue until your module (e.g.
javaMod) shows up asrunningwhen you runiotedge list.
bertk@bertk-edge-ga-6:~/repos/iot-sdks-e2e-fx/scripts$ ./deploy-test-containers.sh --java
Operating with device bertk-edge-ga-6_bertk_550 on on hub bertk-edge.azure-devices.net
Deploying the following containers:
friend: <REDACTED>/default-friend-module
java: <REDACTED>/java-e2e-v3
{
<--snip-->
- As a test of your test-runner environment, you can run tests using the deployed container. If you need to debug this, remember you're running with the test script on your Windows box, but the code that you're testing is still inside the deployed container. Use
docker logson the containers or callpytestwith-s
Note the values printed by pytest as Run Parameters. In particular, the test_module_uri below is http://bertk-edge-ga-6:8080. This tells us that your code under test is exposing a REST endpoint on this URI, which happens to be on my Linux VM
F:\repos\internals\iot-sdks-e2e-fx\test-runner>pytest --node-wrapper --local test_module_send_telemetry.py
============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.6.1, py-1.5.3, pluggy-0.6.0 -- c:\python36\python.exe
<--snip-->
Run Parameters:
module_id: nodeMod
friend_module_id: friendMod
friend_device_id: bertk-edge-ga-6_bertk_550_friend
test_module_uri: http://localhost:8080 (module under test)
friend_module_uri: http://bertk-edge-ga-6:8099 (friend container)
friend_device_uri: http://localhost:8080 (module under test)
registry_uri: http://localhost:8080 (module under test)
service_client_uri: http://localhost:8080 (module under test)
eventhub_uri: http://bertk-edge-ga-6:8099 (friend container)
using environment: True
test transport: mqtt
friend transport: mqtt
straight to iothub: False
collected 1 item
test_module_send_telemetry.py::test_module_send_event_to_iothub FAILED [100%]
================================== FAILURES ===================================
______________________ test_module_send_event_to_iothub _______________________
test_module_send_telemetry.py:34: in test_module_send_event_to_iothub
received_message = input_thread.get(test_utilities.default_eventhub_timeout)
c:\python36\lib\multiprocessing\pool.py:640: in get
raise TimeoutError
E multiprocessing.context.TimeoutError
---------------------------- Captured stdout setup ----------------------------
Of particular interest here is the fact that the test failed with a timeout error. There are two different ways this could have happened:
- The test runner could not communicate with the wrapper code exposed from the
test_module_urias shown above. This could be a stalled container, a badly configured firewall, a bad Windows environment, or any other of a number of things. - The test runner could be communicating with the wrapper, but some operation just happened to time out.
In the case of #1, there would be thread dumps for every thread inside Python.exe, like 90 or so. In the case of #2, the output is short and sweet and shows you two things:
- In the first failure text above, it shows that there was a failure on line 34 of test_module_send_telemetry.py. If you look at this code, it's in the middle of the test script, long after the connections have been established. (so there's no problem connecting from the test runner to the wrapper).
- Lower in the output, there's output from the pytest script stdout stream during the test. This output corresponds directly to print() calls inside test_module_send_telemetry.py.
---------------------------- Captured stdout call -----------------------------
connecting module client
connecting eventhub client
enabling telemetry on eventhub client
start waiting for events on eventhub
sending event: MWM4DIRY0FP3WZBIF3IQU6QN3OZO8RU61S8A23WVE1HR7RGCUYL3EJSEL1AVSYJU
wait for event to arrive at eventhub
-------------------------- Captured stdout teardown ---------------------------
- Now that you've verified your test runner is configured correctly, you can launch your wrapper inside of your debugger. This depends on which SDK you're using, and it's subject to change, but, in short:
- Node: after running
lerna run buildas usual, runazure-iot-sdks-node\edge-e2e\wrapper\nodejs-server-server\index.js - C: run
azure-iot-sdks-c\edge-e2e\build.cmd, then openazure-iot-sdks-c\edge-e2e\cmake\Project.slninside Visual Studio. The sub-project that you want to debug is namededge_e2e_rest_server. - Java: open
azure-iot-sdks-java\pm.xmlin your favorite debugger. Launchedge-e2e\src\main\java\Main.java - C#: TODO
- Python: TODO
When you launch the app, it will open a RESTAPI endpoint on some port. The test script will know which one. Some SDKs display the port to stdout, and some don't.
- Run pytest against the code in your debugger by passing the
--localflag along with whatever other flags you need. This flag does two things:
- It changes the
test_module_urito the appropriate port onhttp://localhost - It connects to edgeHub using connection string and CA instead of getting the trust bundle from the iotedge HSM.
You can verify that the test runner is configured correctly by looking at the
test_module_urithat pytest gives you.
F:\repos\internals\iot-sdks-e2e-fx\test-runner>pytest --node --local test_module_methods.py
<--snip-->
test_module_uri: http://localhost:8080 (module under test)
<--snip-->
========================== 4 passed in 18.14 seconds ==========================
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.