Infrequent IP address changes and No-IP

I use No-IP to provide dynamic DNS services so I can have remote access to my machine at home. However, the No-IP client doesn't send updates when my IP address doesn't change, and with my provider it tends to not change for quite some time. This causes No-IP to send me warning messages that my host is going to be deactivated from inactivity. In these notices, there is a link you can click to keep your host alive with its current IP. That got me thinking how I could force a periodic update.

I wrote the following script, pagecheck, to allow fetching an arbitrary web page and checking for some simple content in the page output:

#!/bin/bash
#Gets a web page and searches it for the specified text; if not found, or if a wget error results, returns
#an error code and prints error text.
#Arthur Penn - 16 Apr 2008

if [ $# -ne 2 ]; then
echo "pagecheck \"URL to page\" \"Success text to expect\""
exit 1
fi

PAGE=$(wget --no-verbose -O - "$1" 2>&1)
RC=$?
if [ 0 -eq $RC ]; then
SUCCESS=$(echo "$PAGE" | grep "$2")
if [ -n "$SUCCESS" ]; then
exit 0
else
echo "Did not find success message of \"$2\" in $1:"
echo "$PAGE"
exit 1
fi
else
echo "$PAGE"
exit 1
fi


This uses wget to fetch the web page and look for the content, and only prints output when it encounters problems. This makes it suitable for cron jobs. I added this script to /usr/local/bin (save it to a text file, and then: sudo cp pagecheck /usr/local/bin && sudo chmod +x /usr/local/bin/pagecheck). I then added the following script into the /etc/cron.monthly folder so it gets run once per month (don't do this more often to avoid excessive No-IP updates):

#!/bin/bash
# Touches the no-ip.com host dialog to confirm that the URL is still in use
/usr/local/bin/pagecheck "http://www.no-ip.com/hostactive.php?host=myhost&domain=noipdomain.net" "Update Successful"


T0 use this, you need to update the portions in red to match your No-IP domain (e.g. if your No-IP domain is fred.atx.net, the host would be "fred," and the domain would be "atx.net."

Since doing this, I haven't gotten any of the host deactivation messages from No-IP.

Make zenwin and zenwinmodeler ignore WMI errors

(This tip is also on the Zenoss wiki.)

At least in version 2.1.1, zenwin, zenwinmodeler, and zeneventlog have (IMO) a critical defect: if there are any /Status/WMI/Conn issues not in history for the device, they ignore the device. On our network, for some reason we end up with a lot of these events ('timegenerated' errors, various intermittent failures to connect, etc.). This causes the monitoring of our Windows servers to dramatically fall off as the system runs, and we miss critical issues.

I changed the behavior of these three systems to go ahead and attempt monitoring even if WMI issues are encountered. I learned that most of the time these WMI issues are spurious and successful monitoring CAN still be attempted. If you use this code, I recommend combining it with event commands to restart the zenoss daemons when it finds them dead.

Also, in zenwin, I added/improved the exception handling; a failure to create the watcher object occurs outside of a try block. Much of this code is an attempt to keep zenwin from crashing if it tries to monitor a Windows Server 2008 machine (Zenoss is not compatible with WS 2008 or Vista's WMI interface, and zenwin cannot monitor services on these devices). I ended up adding a hardcoded exclusion list so I can otherwise monitor the machine but have zenwin skip it. For some reason, zeneventlog seems to not crash, although it is not able to retrieve events from the WS 2008 machine either.

Please see the Zenoss wiki for the zenwin and zenwinmodeler diffs.

Find Zenoss event classes with transforms

(This tip is also on the Zenoss wiki.)

If you have entered transforms but can't remember where you entered them, type the following in zendmd (run zendmd from the command line on the Zenoss server as the zenoss user):

>>> for ec in dmd.Events.getSubOrganizers():
...      if ec.transform:
...      print ec.getOrganizerName()

Custom Zenoss graph based on multiple data points

(This tip is also on the Zenoss Wiki.)

If you want to make a custom graph in Zenoss based on more than one data point (such as a ratio or other calculation), you will need to enter a custom graph definition for RRDTool to use. I found some good guides on how to define graphs with RRDTool (such as this tutorial on CDEF and others at that site), but it took me a while to put this together with the available data points and variables in Zenoss so the graph would work.

Edit the performance template to which you wish to add the graph. Click the drop-down arrow next to Graph Definitions and choose "Add Graph..." and name it.

Click on the Graph Custom Definition tab and you are presented with a blank slate for your new graph's definition. It may be easiest to start with an example. I entered the following custom graph definition:

DEF:BusyThreads-raw=${here/fullRRDPath}/appThreads_BusyThreads.rrd:ds0:AVERAGE
DEF:RequestsPerSecond-raw=${here/fullRRDPath}/appThreads_RequestsPerSecond.rrd:ds0:AVERAGE
DEF:AppCurrentConnections-raw=${here/fullRRDPath}/currentConnections_appCurrentConnections.rrd:ds0:AVERAGE
CDEF:connectionsToThreads=AppCurrentConnections-raw,1,RequestsPerSecond-raw,BusyThreads-raw,+,+,/
LINE:connectionsToThreads#00cc00:"Connections to Threads/Thread Activity Ratio"
GPRINT:connectionsToThreads:LAST:cur\:%5.2lf%s
GPRINT:connectionsToThreads:AVERAGE:avg\:%5.2lf%s
GPRINT:connectionsToThreads:MAX:max\:%5.2lf%s\j

To break this apart, I have two data sources and three data points involved in my ratio that are part of the performance template with this graph:

Data source: appThreads
This data source has two data points, BusyThreads and RequestsPerSecond.

Data source: currentConnections
This data source has one data point, appCurrentConnections.

What I want to graph is a ratio based on these data points as follows:

appCurrentConnections / (BusyThreads + RequestsPerSecond + 1)

Basically, I want a measure of the amount of work in the queue (current connections) divided by the amount of work output my application is producing (a combination of the busy threads and requests per second it is handling, plus one to avoid the possibility of a divide-by-zero error).

With that established, we need to define the RRD DEFs (variables) used in the graph, one for each of the variables in the above calculation. Here's the one for the busy threads variable. I supplied BusyThreads-raw as the name that is used in the graph line:

DEF:BusyThreads-raw=${here/fullRRDPath}/appThreads_BusyThreads.rrd:ds0:AVERAGE


The key above is the TALES expression to get the variable from the Zenoss performance template into our RRDTool DEF variable: ${here/fullRRDPath}/dataSourceName_dataPointName

Regarding the :AVERAGE at the end: While there are many different RRD functions, the most common one I've seen used is the AVERAGE function, which takes a recent rolling average of the value in question. Please consult the RRDTool documentation for going deeper with this.

After providing DEF lines for each variable in my calculation, I need a CDEF line (calculated definition) for the actual calculation that puts the calculation together:

CDEF:connectionsToThreads=AppCurrentConnections-raw,1,RequestsPerSecond-raw,BusyThreads-raw,+,+,/

The calculation uses reverse Polish notation and the CDEF tutorial above has an excellent guide to understanding it, but basically you can think of it as a stack: the variables and constants are pushed onto the stack in order from left to right, and when the first operator (the leftmost plus sign) hits the stack, the top two items (in this case, the BusyThreads-raw and RequestsPerSecond-raw variables) are popped off the stack and added together (the operator is applied). The result is pushed back onto the stack. The next plus sign adds this sum with 1, and finally the division operator divides the AppCurrentConnections-raw variable by the topmost stack item (1 + RequestsPerSecond-raw + BusyThreads-raw).

Once we have our connectionsToThreads variable, we can graph it. The next line defines the one line on our graph:

LINE:connectionsToThreads#00cc00:"Connections to Threads/Thread Activity Ratio"

It refers to the connectionsToThreads variables, defines a color in hex notation, and defines a label. Finally, we can print some additional information on the graph:

GPRINT:connectionsToThreads:LAST:cur\:%5.2lf%s
GPRINT:connectionsToThreads:AVERAGE:avg\:%5.2lf%s
GPRINT:connectionsToThreads:MAX:max\:%5.2lf%s\j


Here we print the last, average, and maximum values of our graph line on the currently-viewed graph section.

Limitation: Thresholding
One thing I could not get working was to define a threshold based on my calculated value above. It seems that the thresholds are only valid on the values of the data points themselves, and I couldn't get a threshold working on my derived value above.

Moving a Zenoss event to history via the Transform expression

(This tip is also on the Zenoss Wiki.)

There are cases where certain events are just noise and you want them moved to history automatically, but perhaps without having ALL of the events in that event class moved to history. For example, you may wish to move certain events from one event class to another based on matching text and at the same time have these go straight to history.

To do this, enter the following in Transform of the event class mapping:

evt._action="history"

Move an event in Zenoss from one event class to another based on event text

(This tip is also on the Zenoss Wiki.)

Many events map to the /App/Failed event class, most notably the Windows Application Error_1000 error (http://<your Zenoss server>:8080/zport/dmd/Events/App/Failed/instances/Application%20Error_1000). I wanted to move some of these Application Error_1000 events to other event classes based on matching particular applications, but to leave the rest in /App/Failed. How does one do this?

To begin, confirm that you have an existing event class to receive the events. If not, create a new one by navigating through the "Events" tree from the left navigation to get to the desired parent class, and once there, click the drop-down arrow next to Subclasses and choose "Add New Organizer..." Enter the name for the new event class, e.g. "MyApplication."

Second, map an additional event class mapping to Application Error_1000. In /zport/dmd/Events/App/Failed, click the drop-down arrow to the left of EventClass Mappings and choose "Add Mapping..." For the ID of the mapping, type Application Error_1000_<name of the application to handle differently>, e.g. "Application Error_1000_MyApplication." (This event class mapping doesn't have to be named this way, but it helps to have the application name as the suffix, so that the mapping gets grouped with any other Application Error_1000 mappings in the list.)

Once you have done this, edit the properties of the new mapping. There are three key things you need to set:

• Event Class Key: Set this to: Application Error_1000
• Regex: I'm sure you can put in much more complicated regular expressions, but all that is necessary is to type some text from the event message, which will usually contain the application's executable name. If this is the case, all you need to enter is something like: MyApplication.exe
• Transform: Here, you need to key in the Python expression that will re-map the event to a different event class, e.g.: evt.eventClass="/App/Failed/MyApplication"
  

Save your changes to this new event class mapping. Now you need to sequence all the Application Error_1000* events so that this custom entry is matched first. Edit the new mapping and click on the Sequence tab. Make sure that your new mapping (Application Error_1000_MyApplication) has a lower sequence number than the generic Application Error_1000 entry. I'm not sure if the sequence numbers need to start at zero, but I've done it that way. So, make your new class sequence 0, and the generic Application Error_1000 class sequence 1. Don't forget to save your changes.

That's it--the events matching your custom event class mapping will be moved to the target event class, and all the others will be left in the original class.

Change Zenoss event severity based on message text

(This tip is also on the Zenoss Wiki.)

If you have events being mapped to a particular event class, generally one event severity gets applied to all of those events. If, however, you want to change the event severity of certain events based on the contents of the event message, do the following:

1. Navigate to the event class (under "Classes' in the left navigation, click Events, and then navigate to the event class containing the events you wish to conditionally map).
2. Using to drop-down arrow in the tab bar, choose More | Transform
3. In the Transform entry area, enter the following:

if evt.message.find("text to find") >= 0:
evt.severity = <desired severity>

For example:

if evt.message.find("timegenerated") >= 0:
evt.severity = 3

The above will change the severity of any events containing the text "timegenerated" from the default for the event class to 3 (warning). For your convenience, the event severity values are as follows:

Severity	Description
5	Critical
4	Error
3	Warning
2	Info
1	Debug
0	Clear