Difference between revisions of "Resource Monitoring Tools"

From Steak Wiki
Jump to navigationJump to search
 
(42 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
There are programs available to watch various aspects of your system.
+
<small>There are programs available to watch various aspects of your system.
  
−
==iftop==
+
==Network==
 +
===iftop===
 
See active internet connections.  
 
See active internet connections.  
 
e.g.
 
e.g.
Line 7: Line 8:
 
Will show you websites that don't close a connection, when the tab is left open. A privacy and security nightmare. This is a reason why [[Javascript]] is bad.
 
Will show you websites that don't close a connection, when the tab is left open. A privacy and security nightmare. This is a reason why [[Javascript]] is bad.
  
−
==vmstat==
+
alt:
 +
netstat | head -n 20
 +
===Speed/Bandwidth===
 +
<pre>
 +
iperf3
 +
iperf
 +
ping with the -i flag to set interval to less than 0.1 seconds (unix (gnu?) only and not busybox).
 +
 
 +
speedtest-cli w/owrt default luci status, real-time graphs, traffic of the lan / wan interface.
 +
 
 +
ethtool will tell you if your nic supports 1000/m
 +
 
 +
mtr (see man page)
 +
mtr-packet (see man page)
 +
</pre>
 +
 
 +
==RAM==
 
See RAM usage. Can be watched, to monitor swapping.
 
See RAM usage. Can be watched, to monitor swapping.
 
e.g.
 
e.g.
Line 13: Line 30:
 
Leave it running. It will update every 3 seconds.
 
Leave it running. It will update every 3 seconds.
  
−
==iotop==
+
===htop===
 +
Take htop, and go in the menus. Change the update rate to
 +
0.1 seconds
 +
I think this view is superior to the default. Might slow down machine,
 +
so use with discretion, (i.e. don't leave it running).
 +
 
 +
==Filesystem==
 +
===iotop===
 
See HDD accesses.
 
See HDD accesses.
 
e.g.
 
e.g.
Line 21: Line 45:
 
  # iotop -d 0.01  or -d 0.1
 
  # iotop -d 0.01  or -d 0.1
 
delay flag can be set to be faster than 1 second. Some writes are missed otherwise.
 
delay flag can be set to be faster than 1 second. Some writes are missed otherwise.
 +
 +
 +
See also:
 +
https://hackaday.com/2020/11/05/linux-fu-monitor-disks/
 +
https://en.wikipedia.org/wiki/Bonnie%2B%2B
 +
 +
===nmon===
 +
There are a couple of hdd activity monitoring tools, but this one seems the best. Gives total percentage of
 +
filesystem activity usage, and lists all HDDs.
 +
 +
===Benchmarking===
 +
Bonnie++. You can adjust sector/cluster/block size and a filesystem and run some tests.
 +
 +
 +
===List Open Files===
 +
lsof
 +
Note: there are different types of lsof (e.g. busybox's)
 +
 +
===Filesystem metadata===
 +
 +
# dumpe2fs /dev/sda1 | less
 +
 +
==SQL Benchmarking==
 +
Mysql has its own collection of tools. See also the documentation on filesystems and mysql, e.g.
 +
 +
https://dev.mysql.com/doc/refman/8.4/en/optimizing-innodb-diskio.html
 +
innodb_page_size
 +
Consider using a page size that matches the internal sector size of the disk. Early-generation SSD
 +
devices often have a 4KB sector size. Some newer devices have a 16KB sector size. The default InnoDB
 +
page size is 16KB. Keeping the page size close to the storage device block size minimizes the amount of
 +
unchanged data that is rewritten to disk.
 +
 +
https://dev.mysql.com/doc/refman/8.0/en/innodb-file-defragmenting.html
 +
 +
https://fuzzyblog.io/blog/mysql/2014/08/28/optimizing-your-filesystem-for-mysql.html
 +
 +
You should also use a dedicated disk for the database, if you run into problems. Or a remote host.
  
 
==Monitor Library Reads from PID==
 
==Monitor Library Reads from PID==
Line 28: Line 89:
 
See what a program is doing. (Note: not available on ARM deb repos)
 
See what a program is doing. (Note: not available on ARM deb repos)
  
−
==Monitor IP address up/down via ping==
+
==cron monitoring scripts==
  
 +
===monitor ip address up/down via ping===
 
<pre>
 
<pre>
 
#!/bin/bash
 
#!/bin/bash
Line 43: Line 105:
 
#e.g. $ script.sh <ipaddress>
 
#e.g. $ script.sh <ipaddress>
 
# in /etc/crontab
 
# in /etc/crontab
−
#*/3 * * * *   root /root/email_alerts/test_up.sh 192.168.1.1 #tune this frequency based on your priority
+
#*/3 * * * * root /root/email_alerts/test_up.sh 192.168.1.1 #tune this frequency based on your priority
 
#0 */2 * * * root rm /var/log/networkalerts/*LOCKFILE
 
#0 */2 * * * root rm /var/log/networkalerts/*LOCKFILE
 
#0 0 * * *  root rm /var/log/networkalerts/*$(date +%A)*LOG
 
#0 0 * * *  root rm /var/log/networkalerts/*$(date +%A)*LOG
  
 
#keep track of time
 
#keep track of time
−
date >> /var/log/networkalerts/$LOGFILE
+
  date >> /var/log/networkalerts/$LOGFILE
−
ping -c 6 $SERVERIP >> /var/log/networkalerts/$LOGFILE
+
  ping -c 6 $SERVERIP >> /var/log/networkalerts/$LOGFILE
 
#nothing after ping, as we need return value
 
#nothing after ping, as we need return value
 
#if return val is error (see man on ping regarding count and deadline)
 
#if return val is error (see man on ping regarding count and deadline)
 
# == or -eq can be used. == is intuitive, therefore better
 
# == or -eq can be used. == is intuitive, therefore better
−
if test $? == 1
+
  if test $? == 1
−
then
+
  then
 
#if file empty
 
#if file empty
 
#[ -s FILE ] True if FILE exists and has a size greater than zero. Thus, you get "empty.txt" if "diff.txt" is not e>
 
#[ -s FILE ] True if FILE exists and has a size greater than zero. Thus, you get "empty.txt" if "diff.txt" is not e>
Line 62: Line 124:
 
# must be space between [ and -s and also last bracket. test brackets are unintuitive so don't use them.
 
# must be space between [ and -s and also last bracket. test brackets are unintuitive so don't use them.
 
#  if [ -s /var/log/networkalerts/$HISTORYFILE ]
 
#  if [ -s /var/log/networkalerts/$HISTORYFILE ]
−
if test -s /var/log/networkalerts/$HISTORYFILE
+
  if test -s /var/log/networkalerts/$HISTORYFILE
 
   then
 
   then
 
     exit 5
 
     exit 5
 
   else
 
   else
 
     # Use your favorite mailer here:
 
     # Use your favorite mailer here:
−
     # wiki.zoneminder.com/Email
+
     # wiki.zoneminder.com/Email explains how to configure email for devuan
 
     echo "alert" | mutt  -s "Network Down" -- $NOTIFYEMAIL
 
     echo "alert" | mutt  -s "Network Down" -- $NOTIFYEMAIL
 
     #lock file / history file
 
     #lock file / history file
 
     echo "alertsent" > /var/log/networkalerts/$HISTORYFILE
 
     echo "alertsent" > /var/log/networkalerts/$HISTORYFILE
 
   fi
 
   fi
 +
  fi
 +
 +
</pre>
 +
 +
===hdd full===
 +
<pre>
 +
#usage: feed $1 company name/subject
 +
df -h | grep 100%
 +
if    [ $? -eq 0 ]; then
 +
#send email
 +
  echo "hdd full" | mutt -s $1 alerts@emailaddress
 +
else
 +
  echo "do nothing"
 
fi
 
fi
  
 
</pre>
 
</pre>
 +
 +
===monitor hdd usage===
 +
<pre>
 +
#must run as root for access to dmesg
 +
 +
LOGFILE=/root/file.log
 +
SUBJECT="hdd details"
 +
 +
echo        ""  > $LOGFILE
 +
echo $COMPANY  >> $LOGFILE
 +
echo        "" >> $LOGFILE
 +
echo        "" >> $LOGFILE
 +
df -h          >> $LOGFILE
 +
echo        "" >> $LOGFILE
 +
echo        "" >> $LOGFILE
 +
lsblk          >> $LOGFILE
 +
echo        "" >> $LOGFILE
 +
echo        "" >> $LOGFILE
 +
dmesg  | grep -e sda -e sdb -e sdc -e sdd -e sde >> $LOGFILE
 +
echo "" | mutt -s $SUBJECT alerts@email -a $LOGFILE
 +
</pre>
 +
 +
===simple monitor===
 +
 +
The use of && or || for either a function that returns 1 or 0 (test via $? or just an echo) is the simplest way to script commands. Put it in cron.
 +
 +
ps x | grep myprogram | grep -vi grep || echo "program is not running..." | msmtp -a default email@address
 +
 +
===diff on folder contents===
 +
<pre>
 +
mkdir -p /var/log/folder_monitoring/folder_name
 +
tree    "directorytowatch" | head -n 500 > /var/log/folder_monitoring/diff1
 +
diff    /var/log/folder_monitoring/diff0  /var/log/folder_monitoring/diff1 > /var/log/folder_monitoring/diff_result
 +
cp      /var/log/folder_monitoring/diff1  /var/log/folder_monitoring/folder_name/diff1_$(date  +%Y_%m_%d)
 +
cp      /var/log/folder_monitoring/diff1  /var/log/folder_monitoring/diff0
 +
echo    "diff" | mutt -s "some_explanation" treediffe@email -a /var/log/folder_monitoring/diff_result
 +
 +
#use search / replace to adjust as needed (in nano that is ctrl-\)
 +
</pre>
 +
A few updates. You want to have a history so you can look at file changes over time or debug if something breaks (script has an error, or filesystem is unconnected). That is handled by copying the old tree with the date. Also, you want to limit how many results it returns, so you pipe it into a head.
 +
 +
 +
 +
 +
</small>

Latest revision as of 06:10, 25 October 2025

There are programs available to watch various aspects of your system.

Network

iftop

See active internet connections. e.g. 

# iftop -i eth1

Will show you websites that don't close a connection, when the tab is left open. A privacy and security nightmare. This is a reason why Javascript is bad.

alt: 

netstat | head -n 20

Speed/Bandwidth

 iperf3
 iperf
 ping with the -i flag to set interval to less than 0.1 seconds (unix (gnu?) only and not busybox).

 speedtest-cli w/owrt default luci status, real-time graphs, traffic of the lan / wan interface.

 ethtool will tell you if your nic supports 1000/m

 mtr (see man page)
 mtr-packet (see man page)

RAM

See RAM usage. Can be watched, to monitor swapping. e.g. 

$ vmstat 3

Leave it running. It will update every 3 seconds.

htop

Take htop, and go in the menus. Change the update rate to 

0.1 seconds

I think this view is superior to the default. Might slow down machine, so use with discretion, (i.e. don't leave it running).

Filesystem

iotop

See HDD accesses. e.g. 

# iotop --only
# iotop -o

only flag will show active processes only 

# iotop -d 0.01  or -d 0.1

delay flag can be set to be faster than 1 second. Some writes are missed otherwise.


See also: https://hackaday.com/2020/11/05/linux-fu-monitor-disks/ https://en.wikipedia.org/wiki/Bonnie%2B%2B

nmon

There are a couple of hdd activity monitoring tools, but this one seems the best. Gives total percentage of filesystem activity usage, and lists all HDDs.

Benchmarking

Bonnie++. You can adjust sector/cluster/block size and a filesystem and run some tests.


List Open Files

lsof

Note: there are different types of lsof (e.g. busybox's)

Filesystem metadata

# dumpe2fs /dev/sda1 | less

SQL Benchmarking

Mysql has its own collection of tools. See also the documentation on filesystems and mysql, e.g.

https://dev.mysql.com/doc/refman/8.4/en/optimizing-innodb-diskio.html 

innodb_page_size
Consider using a page size that matches the internal sector size of the disk. Early-generation SSD
devices often have a 4KB sector size. Some newer devices have a 16KB sector size. The default InnoDB
page size is 16KB. Keeping the page size close to the storage device block size minimizes the amount of
unchanged data that is rewritten to disk.

https://dev.mysql.com/doc/refman/8.0/en/innodb-file-defragmenting.html

https://fuzzyblog.io/blog/mysql/2014/08/28/optimizing-your-filesystem-for-mysql.html

You should also use a dedicated disk for the database, if you run into problems. Or a remote host.

Monitor Library Reads from PID

$ ltrace -p -pidhere-

See what a program is doing. (Note: not available on ARM deb repos)

cron monitoring scripts

monitor ip address up/down via ping

#!/bin/bash

SERVERIP=$1
LOGFILE=$1_$(date +%A)_LOG
HISTORYFILE=$1_$(date +%A)_LOCKFILE
NOTIFYEMAIL=myemail@address.com

#setup this script in cron each minute, and also
#crontab requires historyfile / lockfile to be blanked (echo "" > file) each day or each hour, whatever you prefer.
#mkdir /var/log/networkalerts
#e.g. $ script.sh <ipaddress>
# in /etc/crontab
#*/3 * * * * root /root/email_alerts/test_up.sh 192.168.1.1 #tune this frequency based on your priority
#0 */2 * * * root rm /var/log/networkalerts/*LOCKFILE
#0 0 * * *   root rm /var/log/networkalerts/*$(date +%A)*LOG

#keep track of time
  date >> /var/log/networkalerts/$LOGFILE
  ping -c 6 $SERVERIP >> /var/log/networkalerts/$LOGFILE
#nothing after ping, as we need return value
#if return val is error (see man on ping regarding count and deadline)
# == or -eq can be used. == is intuitive, therefore better
  if test $? == 1
  then
#if file empty
#[ -s FILE ] True if FILE exists and has a size greater than zero. Thus, you get "empty.txt" if "diff.txt" is not e>
#https://stackoverflow.com/questions/9964823/how-to-check-if-a-file-is-empty-in-bash
#https://mywiki.wooledge.org/BashGuide/TestsAndConditionals    for all the other tests like -s
#   [! -s file] to invert didn't work because of missing spaces (i think)
# must be space between [ and -s and also last bracket. test brackets are unintuitive so don't use them.
#  if [ -s /var/log/networkalerts/$HISTORYFILE ]
  if test -s /var/log/networkalerts/$HISTORYFILE
   then
    exit 5
   else
    # Use your favorite mailer here:
    # wiki.zoneminder.com/Email explains how to configure email for devuan
    echo "alert" | mutt  -s "Network Down" -- $NOTIFYEMAIL
    #lock file / history file
    echo "alertsent" > /var/log/networkalerts/$HISTORYFILE
   fi
  fi

hdd full

#usage: feed $1 company name/subject 
df -h | grep 100%
if    [ $? -eq 0 ]; then
#send email
   echo "hdd full" | mutt -s $1 alerts@emailaddress 
else
   echo "do nothing"
fi

monitor hdd usage

#must run as root for access to dmesg

LOGFILE=/root/file.log
SUBJECT="hdd details"

echo        ""  > $LOGFILE
echo $COMPANY  >> $LOGFILE
echo        "" >> $LOGFILE
echo        "" >> $LOGFILE
df -h          >> $LOGFILE
echo        "" >> $LOGFILE
echo        "" >> $LOGFILE
lsblk          >> $LOGFILE
echo        "" >> $LOGFILE
echo        "" >> $LOGFILE
dmesg   | grep -e sda -e sdb -e sdc -e sdd -e sde >> $LOGFILE
echo "" | mutt -s $SUBJECT alerts@email -a $LOGFILE

simple monitor

The use of && or || for either a function that returns 1 or 0 (test via $? or just an echo) is the simplest way to script commands. Put it in cron. 

ps x | grep myprogram | grep -vi grep || echo "program is not running..." | msmtp -a default email@address

diff on folder contents

mkdir -p /var/log/folder_monitoring/folder_name
tree     "directorytowatch" | head -n 500 > /var/log/folder_monitoring/diff1
diff     /var/log/folder_monitoring/diff0   /var/log/folder_monitoring/diff1 > /var/log/folder_monitoring/diff_result
cp       /var/log/folder_monitoring/diff1   /var/log/folder_monitoring/folder_name/diff1_$(date  +%Y_%m_%d)
cp       /var/log/folder_monitoring/diff1   /var/log/folder_monitoring/diff0
echo    "diff" | mutt -s "some_explanation" treediffe@email -a /var/log/folder_monitoring/diff_result

#use search / replace to adjust as needed (in nano that is ctrl-\)

A few updates. You want to have a history so you can look at file changes over time or debug if something breaks (script has an error, or filesystem is unconnected). That is handled by copying the old tree with the date. Also, you want to limit how many results it returns, so you pipe it into a head.



Retrieved from "http://steakwiki.com/index.php?title=Resource_Monitoring_Tools&oldid=2414"