[Node 237] Tracking Issue

[jonakarl]

Report in comments when and what stability tests/errors have been perfomed/discovered (attach errorlogs). When node deemed stable this issue will be closed

[alfs]

Node was full grey in the inventory.
Green LED on APU and YepKit.
I pulled power plug + rebooted mifi's + replugged power after ~1 minute.
Checking back later to see the inventory status.

[alfs]

Node is grey in inventory, but APU and yepkit lights are green.
Did another reboot of node and modems, still grey in the inventory.

Next step is to check serial console output.

[alfs]

Tried management ethernet, no response.
Attached serial console, output:

Press F10 key now for boot menu:
drive 0x000f29f0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=31277232
Booting from Hard Disk...
WARNING - Timeout at ahci_command:155!
Boot failed: could not read the boot disk

Booting from ROM...
iPXE (PCI 00:00.0) starting execution...ok
iPXE initialising devices...ok

[alfs]

Tried reinstalling the node, but was unable to detect SSD, getting

WARNING - Timeout at ahci_command:155!

We tried exchanging the mSSD in the node, which solved the boot issue.
We therefore need a replacement ssd.

[kristrev]

One interesting thing is that we have not seen any reports about disk failures (that I can find), except from KaU. I am just curios if there is something for example with the electricity that kills the disks. We run the same in our routers, and those disks are abused in ways Monroe should have nightmares about :)

Also, did you ever find a reader for mSata-disks? It would be interesting to see what for example gparted has to say.

[alfs]

Didn't find an explicit m-sata reader, but we've got some mPCIe adapters for wifi-cards, maybe these work as well for storage devices? Or JonasK may have some other adapter perhaps.

[alfs]

#237 is now back online, but is not visible in the inventory and the ssh tunnel is down. It is however sending syslogs to the logging system. This is from the last ansible run, which seems successful:

Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git directory set to: /root/ansible
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git remote set to: [deleted/SA]
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git branch set to: monroe-production
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Ansible playbook set to: local.yml
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Ansible inventory set to: "localhost,"
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Ansible cache set to: /root/ansible-cache
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git remote is set to [deleted/SA]
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git remote is using the monroe-production checkout
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git branch SHA is b57ea0209989f630541fefff378b0cd7a8600fbd
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Initiating git repo setup in /root/ansible
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Git repo fetched in /root/ansible
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Git repo reset hard to monroe-production in /root/ansible
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: INFO: No change made to git repo, reusing b57ea0209989f630541fefff378b
0cd7a8600fbd
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Setup of git repo, attempt 1 was successful
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Executing ansible using b57ea0209989f630541fefff378b0cd7a8600fb
d
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Executing playbook local.yml with inventory "localhost,"
Oct 31 16:21:21 Monroe000db940de78 monroe-ansible: INFO: Playbook execution was successful
Oct 31 16:21:21 Monroe000db940de78 monroe-ansible: DEBUG: Cache already up to date
Oct 31 16:21:21 Monroe000db940de78 monroe-ansible: DEBUG: No temporary variables file found, skipping cache update of variables

Watchdog log indicates that the tunnel is OK as well:

...
Oct 31 16:40:46 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0
Oct 31 16:41:06 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0
Oct 31 16:41:26 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0
Oct 31 16:41:46 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0

But connection is refused at the tunnelpoint:

$ ./sshr-monroe 237
ssh: connect to host localhost port 10237: Connection refused

[jonakarl]

The only adapters I have is the ones you reefer to Stefan.

It is on the todo list to attach one of the faulty drives to one of those
adapters (but the todo list is long atm).

2016-10-31 16:43 GMT+01:00 Stefan A notifications@github.com:

#237 is now back online, but is not visible in the inventory and the ssh
tunnel is down. It is however sending syslogs to the logging system. This
is from the last ansible run, which seems successful:

Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git directory set to: /root/ansible
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git remote set to: [deleted/SA]
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git branch set to: monroe-production
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Ansible playbook set to: local.yml
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Ansible inventory set to: "localhost,"
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Ansible cache set to: /root/ansible-cache
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git remote is set to git@github.com:Celerway/celerway-ansible.g
it
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git remote is using the monroe-production checkout
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Git branch SHA is b57ea0209989f630541fefff378b0cd7a8600fbd
Oct 31 16:20:01 Monroe000db940de78 monroe-ansible: DEBUG: Initiating git repo setup in /root/ansible
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Git repo fetched in /root/ansible
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Git repo reset hard to monroe-production in /root/ansible
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: INFO: No change made to git repo, reusing b57ea0209989f630541fefff378b
0cd7a8600fbd
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Setup of git repo, attempt 1 was successful
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Executing ansible using b57ea0209989f630541fefff378b0cd7a8600fb
d
Oct 31 16:20:07 Monroe000db940de78 monroe-ansible: DEBUG: Executing playbook local.yml with inventory "localhost,"
Oct 31 16:21:21 Monroe000db940de78 monroe-ansible: INFO: Playbook execution was successful
Oct 31 16:21:21 Monroe000db940de78 monroe-ansible: DEBUG: Cache already up to date
Oct 31 16:21:21 Monroe000db940de78 monroe-ansible: DEBUG: No temporary variables file found, skipping cache update of variables

Watchdog log indicates that the tunnel is OK as well:

...
Oct 31 16:40:46 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0
Oct 31 16:41:06 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0
Oct 31 16:41:26 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0
Oct 31 16:41:46 Monroe000db940de78 watchdog: autotunnel.sh test exiting with value 0

But connection is refused at the tunnelpoint:

$ ./sshr-monroe 237
ssh: connect to host localhost port 10237: Connection refused

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#32 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AItalyM-V9DimVS2XNHcjc0bIanzKQweks5q5gyqgaJpZM4KP_i7
.

Jonas Karlsson
Senior Research Engineer

Karlstad University
SE-651 88 Karlstad, Sweden
Telephone: +46 54 700 15 64
Mobile: +46 70 672 06 20
Skype: karlsson.karl.jonas
Hangout: karlsson.karl.jonas
PGP/GPG : 0x88765911B284F3A6
https://keys.mailvelope.com/pks/lookup?op=get&search=0x88765911B284F3A6

twitter.com/kau
facebook.com/karlstadsuniversitet
KAU.SE

[alfs]

Kristian, I tried the disk in another node and booted finnix off PXE. It's not detected at boot time. Do you think a mPCIe-adapter would produce different results?

From dmesg:

root@1:~# dmesg |grep ata
[    0.000000] Memory: 3819100K/4162484K available (5543K kernel code, 1020K rwdata, 1904K rodata, 1256K init, 832K bss, 343384K reserved, 0K cma-reserved)
[   20.409993] Write protecting the kernel read-only data: 8192k
[   20.698648] libata version 3.00 loaded.
[   20.708354] ata1: SATA max UDMA/133 abar m1024@0xf7f08000 port 0xf7f08100 irq 19
... [ata2-ata6 removed/SA]
[   30.706830] ata1: softreset failed (1st FIS failed)
[   40.706821] ata1: softreset failed (1st FIS failed)
[   75.706786] ata1: softreset failed (1st FIS failed)
[   75.706804] ata1: limiting SATA link speed to 3.0 Gbps
[   80.894799] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
[   80.894815] ata1.00: link online but device misclassified
[   80.894822] ata1: link online but 1 devices misclassified, device detection might fail

So the non-functional drive fails "softreset", has a lower link speed, and fails identification.

To me this is a hardware error, not a partitioning/installation problem.

As a comparison, here's another node (#234) to show the detection of a functional storage device:

monroe@Monroe000db940de64:~$ dmesg |grep ata
[    0.000000] Memory: 4000044K/4162480K available (6028K kernel code, 1169K rwdata, 2804K rodata, 1368K init, 936K bss, 162436K reserved, 0K cma-reserved)
[    4.020401] Write protecting the kernel read-only data: 10240k
[    4.176070] libata version 3.00 loaded.
[    4.190097] ata1: SATA max UDMA/133 abar m1024@0xf7f04000 port 0xf7f04100 irq 19
... [ata2 - ata6 removed/SA]
[    4.658954] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[    4.659184] ata1.00: ATA-10: SATA SSD, S9FM02.5, max UDMA/133
[    4.659194] ata1.00: 31277232 sectors, multi 16: LBA48 NCQ (depth 31/32), AA
[    4.659411] ata1.00: configured for UDMA/133

[alfs]

The ssh tunnel problem is because it's using an old autotunnel configuration with the old monroe-inventory.celerway.no hostname, which is because ansible-wrapper is using the production branch. Can the autotunnel config be updated in all branches?

Trying to manually force the monroe-experiments branch fails:

root@Monroe000db940de78:~# ansible-wrapper -b monroe-experiments
2016-11-03 09:55:36 DEBUG: Git directory set to: /root/ansible
2016-11-03 09:55:36 DEBUG: Git remote set to: [the ansible url/SA]
2016-11-03 09:55:36 DEBUG: Git branch set to: monroe-experiments
2016-11-03 09:55:36 DEBUG: Ansible playbook set to: local.yml
2016-11-03 09:55:36 DEBUG: Ansible inventory set to: "localhost,"
2016-11-03 09:55:37 DEBUG: Ansible cache set to: /root/ansible-cache
2016-11-03 09:55:37 DEBUG: No repo found locally in /root/ansible
2016-11-03 09:55:37 DEBUG: Attempting to clone [the ansible url/SA] into /root/ansible and checkout monroe-experiments
2016-11-03 09:55:39 ERROR: Failed to clone [the ansible url/SA] into /root/ansible and checkout monroe-experiments
2016-11-03 09:55:39 ERROR: Setup of git repo, attempt 1 failed
2016-11-03 09:55:44 DEBUG: No repo found locally in /root/ansible
2016-11-03 09:55:44 DEBUG: Attempting to clone [the ansible url/SA] into /root/ansible and checkout monroe-experiments

A manual git clone of the ansible repository works however.

[relet]

The branch is called docker-experiments.

On Thu, 3 Nov 2016, 11:08 Stefan A, notifications@github.com wrote:

The ssh tunnel problem is because it's using an old autotunnel
configuration with the old monroe-inventory.celerway.no hostname, which
is because ansible-wrapper is using the production branch. Can the
autotunnel config be updated in all branches?

Trying to manually force the monroe-experiments branch fails:

root@Monroe000db940de78:~# ansible-wrapper -b monroe-experiments
2016-11-03 09:55:36 DEBUG: Git directory set to: /root/ansible
2016-11-03 09:55:36 DEBUG: Git remote set to: [the ansible url/SA]
2016-11-03 09:55:36 DEBUG: Git branch set to: monroe-experiments
2016-11-03 09:55:36 DEBUG: Ansible playbook set to: local.yml
2016-11-03 09:55:36 DEBUG: Ansible inventory set to: "localhost,"
2016-11-03 09:55:37 DEBUG: Ansible cache set to: /root/ansible-cache
2016-11-03 09:55:37 DEBUG: No repo found locally in /root/ansible
2016-11-03 09:55:37 DEBUG: Attempting to clone [the ansible url/SA] into /root/ansible and checkout monroe-experiments
2016-11-03 09:55:39 ERROR: Failed to clone [the ansible url/SA] into /root/ansible and checkout monroe-experiments
2016-11-03 09:55:39 ERROR: Setup of git repo, attempt 1 failed
2016-11-03 09:55:44 DEBUG: No repo found locally in /root/ansible
2016-11-03 09:55:44 DEBUG: Attempting to clone [the ansible url/SA] into /root/ansible and checkout monroe-experiments

A manual git clone of the ansible repository works however.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#32 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAISEHqTVEDukMYh2LRVa15qW6Zr9Pu9ks5q6aSNgaJpZM4KP_i7
.

[alfs]

Great, that branch worked better. On the interesting side, it seems that the biteback forced bootos reinstallation worked as the node now reinstalled. On the problematic side, it didn't roll over to docker-experiments automatically.

Anyway - now I'm getting this ansible error:

2016-11-03 10:21:33,454 p=22379 u=root |  changed: [localhost] => (item=python-requests,ansible)
2016-11-03 10:21:33,456 p=22379 u=root |  TASK: [common | ansible-wrapper present and updated] ************************** 
2016-11-03 10:21:33,463 p=22379 u=root |  fatal: [localhost] => Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 561, in _executor
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 666, in _executor_internal
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 884, in _executor_internal_inner
  File "/usr/lib/python2.7/dist-packages/ansible/runner/action_plugins/normal.py", line 57, in run
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 442, in _execute_module
  File "/usr/lib/python2.7/dist-packages/ansible/runner/__init__.py", line 1151, in _configure_module
  File "/usr/lib/python2.7/dist-packages/ansible/module_common.py", line 142, in modify_module
IOError: [Errno 2] No such file or directory: '/usr/share/ansible/files/copy'

root@Monroe-X:~# dpkg --status ansible|grep Version
Version: 2.1.1.0-1~bpo8+1