Skip to content

utilities tool #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Jo-stfc wants to merge 1 commit into
base: master
Choose a base branch
from
Open

utilities tool #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions utilities/aquilon/checkhost.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
hosts=`cat hostlist.txt`
#run aq show_hosts over multiple hosts and searches for passed string
#example: basheckhost.sh Sandbox checks if hosts are in a sandbox
#
for t in ${hosts}; do
echo "--------------------------------------------------------------------------------------------------------------"
echo $t
aq show_host --hostname ${t} | grep -i ${1}
done
9 changes: 9 additions & 0 deletions utilities/aquilon/compilehosts.sh
View file
Open in desktop

@Tom-Byrne Tom-Byrne Mar 31, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this functionality be replaced by aq compile --personality?

@Jo-stfc Jo-stfc Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've used this when hosts are across multiple personalities, if they're all the same personality you can use the aq compile --personality command instead

Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
hosts=`cat hostlist.txt`
#compiles all hosts in the hostlist file
#example: bash compilehosts.sh
#
for t in ${hosts}; do
echo "--------------------------------------------------------------------------------------------------------------"
echo $t
aq compile --hostname ${t}
done
22 changes: 22 additions & 0 deletions utilities/aquilon/hostlist.txt
View file
Open in desktop

@Tom-Byrne Tom-Byrne Mar 31, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lists of hosts are going to go out of date - do we really need these to be up on github?

@Jo-stfc Jo-stfc Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these can be updated edited as necessary - i've used reduced subsets for specific uses. We could keep the full list on github or do something else

@Tom-Byrne Tom-Byrne Mar 31, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make this more generic, it may be more appropriate to include instructions on how to generate an up to date list of hosts e.g. aq search_host --personality <personality> > hostlist.txt

@Jo-stfc Jo-stfc Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

open to suggestions on managing this

@Tom-Byrne Tom-Byrne Mar 31, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The usage of aquilon 'metadata' in our ceph clusters management has a few loosely defined principles:

  1. The source of truth for what a host is doing is its current personality. Anything else can and will go out of date.
  2. All production hosts with a particular personality should be doing the same thing, so any bulk operation should be done (at minimum) to all hosts with the same personality.

Basing tooling around these two things minimizes the amount of places where information can go stale, and reduces the chance of mistakes being made. It also makes our lives easier as we aren't having to understand what "service functionality" a list of hosts is representing, and then having to check that the list isn't out of date before continuing.

For this example, it's not obvious what the hosts on this list are doing now (e.g. are these all Echo gateways, or just a subset?), and it will be even less obvious in 6 months time when some of them are doing different things, and there are new hosts that should be on this list.

If you absolutely need a list of hosts spanning multiple personalities for some part of your workflow, this PR could contain the script to generate a temporary list from a set of personalities. There should be a piece of operational documentation that keeps track of what sets of personalities implement an overarching functionality that may need to be operated on in bulk (e.g. these sets of personalities provide external access to X), so someone can generate an up-to-date list if they need to reuse the tooling.

How does this sound?

@Jo-stfc Jo-stfc Mar 31, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, i'll get something that fetches a list based on personality instead

Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
ceph-gw1.gridpp.rl.ac.uk
ceph-gw2.gridpp.rl.ac.uk
ceph-gw3.gridpp.rl.ac.uk
ceph-gw4.gridpp.rl.ac.uk
ceph-gw5.gridpp.rl.ac.uk
ceph-gw6.gridpp.rl.ac.uk
ceph-gw7.gridpp.rl.ac.uk
ceph-gw8.gridpp.rl.ac.uk
ceph-gw10.gridpp.rl.ac.uk
ceph-gw11.gridpp.rl.ac.uk
ceph-gw12.gridpp.rl.ac.uk
ceph-gw13.gridpp.rl.ac.uk
ceph-gw14.gridpp.rl.ac.uk
ceph-gw15.gridpp.rl.ac.uk
ceph-gw16.gridpp.rl.ac.uk
ceph-svc01.gridpp.rl.ac.uk
ceph-svc02.gridpp.rl.ac.uk
ceph-svc03.gridpp.rl.ac.uk
ceph-svc05.gridpp.rl.ac.uk
ceph-svc97.gridpp.rl.ac.uk
ceph-svc98.gridpp.rl.ac.uk
ceph-svc99.gridpp.rl.ac.uk
22 changes: 22 additions & 0 deletions utilities/hostlist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
ceph-gw1.gridpp.rl.ac.uk
ceph-gw2.gridpp.rl.ac.uk
ceph-gw3.gridpp.rl.ac.uk
ceph-gw4.gridpp.rl.ac.uk
ceph-gw5.gridpp.rl.ac.uk
ceph-gw6.gridpp.rl.ac.uk
ceph-gw7.gridpp.rl.ac.uk
ceph-gw8.gridpp.rl.ac.uk
ceph-gw10.gridpp.rl.ac.uk
ceph-gw11.gridpp.rl.ac.uk
ceph-gw12.gridpp.rl.ac.uk
ceph-gw13.gridpp.rl.ac.uk
ceph-gw14.gridpp.rl.ac.uk
ceph-gw15.gridpp.rl.ac.uk
ceph-gw16.gridpp.rl.ac.uk
ceph-svc01.gridpp.rl.ac.uk
ceph-svc02.gridpp.rl.ac.uk
ceph-svc03.gridpp.rl.ac.uk
ceph-svc05.gridpp.rl.ac.uk
ceph-svc97.gridpp.rl.ac.uk
ceph-svc98.gridpp.rl.ac.uk
ceph-svc99.gridpp.rl.ac.uk
11 changes: 11 additions & 0 deletions utilities/logsearch.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
hosts=`cat hostlist.txt`
#searches xrootd and gridftp logs and ip address gw hosts listed above for parameter passed to script
#example: bash logsearch.sh 31.147.202.178 searches for that ip address in logs and ssh
#
for t in ${hosts[@]}; do
echo "--------------------------------------------------------------------------------------------------------------"
echo $t
( ssh root@${t} "cat /var/log/xrootd/unified/xrootd.log | grep -i \"${1}\"")
( ssh root@${t} "cat /var/log/gridftp/* | grep -i \"${1}\"")
( ssh root@${t} "ss -a | grep -i \"${1}\"")
done
30 changes: 30 additions & 0 deletions utilities/serviceop.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
hosts=`cat hostlist.txt`
# helper function to check if string is in list
function exists_in_list() {
LIST=$1
DELIMITER=$2
VALUE=$3
LIST_WHITESPACES=`echo $LIST | tr "$DELIMITER" " "`
for x in $LIST_WHITESPACES; do
if [ "$x" = "$VALUE" ]; then
return 0
fi
done
return 1
}

#run service operations on targetted services over multiple hosts
#example: bash serviceop.sh status xrootd@* reports the service status of all xrootd services on the hosts
#
if exists_in_list "start stop status restart" " " ${1}
then
echo "start"
else
echo "invalid op. valid ops are: start stop status restart"
exit
fi
for t in ${hosts}; do
echo "--------------------------------------------------------------------------------------------------------------"
echo $t
( ssh root@${t} "systemctl ${1} ${2}" )
done