Resource Monitoring Tools

There are programs available to watch various aspects of your system.

Network

iftop

See active internet connections. e.g.

# iftop -i eth1

Will show you websites that don't close a connection, when the tab is left open. A privacy and security nightmare. This is a reason why Javascript is bad.

alt:

netstat | head -n 20

Speed/Bandwidth

iperf3
iperf
ping with the -i flag to set interval to less than 0.1 seconds (unix only and not busybox).

speedtest-cli w/owrt default luci status, real-time graphs, traffic of the lan / wan interface.

ethtool will tell you if your nic supports 1000/m

RAM

See RAM usage. Can be watched, to monitor swapping. e.g.

$ vmstat 3

Leave it running. It will update every 3 seconds.

htop

Take htop, and go in the menus. Change the update rate to

0.1 seconds

I think this view is superior to the default. Might slow down machine, so use with discretion, (i.e. don't leave it running).

Filesystem

iotop

See HDD accesses. e.g.

# iotop --only
# iotop -o

only flag will show active processes only

# iotop -d 0.01  or -d 0.1

delay flag can be set to be faster than 1 second. Some writes are missed otherwise.

See also: https://hackaday.com/2020/11/05/linux-fu-monitor-disks/ https://en.wikipedia.org/wiki/Bonnie%2B%2B

nmon

There are a couple of hdd activity monitoring tools, but this one seems the best. Gives total percentage of filesystem activity usage, and lists all HDDs.

Benchmarking

Bonnie++. You can adjust sector/cluster/block size and a filesystem and run some tests.

List Open Files

lsof Note: there are different types of lsof (e.g. busybox's)

Filesystem metadata

# dumpe2fs /dev/sda1 | less

SQL Benchmarking

Mysql has its own collection of tools. See also the documentation on filesystems and mysql, e.g.

https://dev.mysql.com/doc/refman/8.4/en/optimizing-innodb-diskio.html

innodb_page_size
Consider using a page size that matches the internal sector size of the disk. Early-generation SSD
devices often have a 4KB sector size. Some newer devices have a 16KB sector size. The default InnoDB
page size is 16KB. Keeping the page size close to the storage device block size minimizes the amount of
unchanged data that is rewritten to disk.

https://dev.mysql.com/doc/refman/8.0/en/innodb-file-defragmenting.html

https://fuzzyblog.io/blog/mysql/2014/08/28/optimizing-your-filesystem-for-mysql.html

You should also use a dedicated disk for the database, if you run into problems. Or a remote host.

Monitor Library Reads from PID

$ ltrace -p -pidhere-

See what a program is doing. (Note: not available on ARM deb repos)

cron monitoring scripts

monitor ip address up/down via ping

#!/bin/bash

SERVERIP=$1
LOGFILE=$1_$(date +%A)_LOG
HISTORYFILE=$1_$(date +%A)_LOCKFILE
NOTIFYEMAIL=myemail@address.com

#setup this script in cron each minute, and also
#crontab requires historyfile / lockfile to be blanked (echo "" > file) each day or each hour, whatever you prefer.
#mkdir /var/log/networkalerts
#e.g. $ script.sh <ipaddress>
# in /etc/crontab
#*/3 * * * *   root /root/email_alerts/test_up.sh 192.168.1.1 #tune this frequency based on your priority
#0 */2 * * * root rm /var/log/networkalerts/*LOCKFILE
#0 0 * * *   root rm /var/log/networkalerts/*$(date +%A)*LOG

#keep track of time
  date >> /var/log/networkalerts/$LOGFILE
  ping -c 6 $SERVERIP >> /var/log/networkalerts/$LOGFILE
#nothing after ping, as we need return value
#if return val is error (see man on ping regarding count and deadline)
# == or -eq can be used. == is intuitive, therefore better
  if test $? == 1
  then
#if file empty
#[ -s FILE ] True if FILE exists and has a size greater than zero. Thus, you get "empty.txt" if "diff.txt" is not e>
#https://stackoverflow.com/questions/9964823/how-to-check-if-a-file-is-empty-in-bash
#https://mywiki.wooledge.org/BashGuide/TestsAndConditionals    for all the other tests like -s
#   [! -s file] to invert didn't work because of missing spaces (i think)
# must be space between [ and -s and also last bracket. test brackets are unintuitive so don't use them.
#  if [ -s /var/log/networkalerts/$HISTORYFILE ]
  if test -s /var/log/networkalerts/$HISTORYFILE
   then
    exit 5
   else
    # Use your favorite mailer here:
    # wiki.zoneminder.com/Email explains how to configure email for devuan
    echo "alert" | mutt  -s "Network Down" -- $NOTIFYEMAIL
    #lock file / history file
    echo "alertsent" > /var/log/networkalerts/$HISTORYFILE
   fi
  fi

hdd full

#usage: feed $1 company name/subject 
df -h | grep 100%
if [ $? -eq 0 ]; then
#send email
  echo "hdd full" | mutt -s $1 alerts@emailaddress 
else
  echo "do nothing"
fi

monitor hdd usage

#must run as root for access to dmesg

LOGFILE=/root/file.log
SUBJECT="hdd details"

echo "" > $LOGFILE
echo $COMPANY >> $LOGFILE
echo "" >> $LOGFILE
echo "" >> $LOGFILE
df -h   >> $LOGFILE
echo "" >> $LOGFILE
echo "" >> $LOGFILE
lsblk   >> $LOGFILE
echo "" >> $LOGFILE
echo "" >> $LOGFILE
dmesg | grep -e sda -e sdb -e sdc -e sdd -e sde >> $LOGFILE
echo "" | mutt -s $SUBJECT alerts@email -a $LOGFILE

simple monitor

The use of && or || for either a function that returns 1 or 0 (test via $? or just an echo) is the simplest way to script commands. Put it in cron.

ps x | grep myprogram | grep -vi grep || echo "program is not running..." | msmtp -a default email@address

diff on folder contents

mkdir -p /var/log/folder_monitoring/folder_name
tree "directorytowatch" | head -n 500 > /var/log/folder_monitoring/diff1
diff /var/log/folder_monitoring/diff0 /var/log/folder_monitoring/diff1 &> /var/log/folder_monitoring/diff_result
cp /var/log/folder_monitoring/diff0 /var/log/folder_monitoring/folder_name/diff0_$(date  +%Y_%m_%d)
cp /var/log/folder_monitoring/diff1 /var/log/folder_monitoring/diff0
echo "diff" | mutt -s "some_explanation" treediffe@email -a /var/log/folder_monitoring/diff_result

#use search / replace to adjust as needed (in nano that is ctrl-\)

A few updates. You want to have a history so you can look at file changes over time or debug if something breaks (script has an error, or filesystem is unconnected). That is handled by copying the old tree with the date. Also, you want to limit how many results it returns, so you pipe it into a head.