Find Zenoss event classes with transforms

(This tip is also on the Zenoss wiki.)

If you have entered transforms but can't remember where you entered them, type the following in zendmd (run zendmd from the command line on the Zenoss server as the zenoss user):

>>> for ec in dmd.Events.getSubOrganizers():
...      if ec.transform:
...      print ec.getOrganizerName()

Custom Zenoss graph based on multiple data points

(This tip is also on the Zenoss Wiki.)

If you want to make a custom graph in Zenoss based on more than one data point (such as a ratio or other calculation), you will need to enter a custom graph definition for RRDTool to use. I found some good guides on how to define graphs with RRDTool (such as this tutorial on CDEF and others at that site), but it took me a while to put this together with the available data points and variables in Zenoss so the graph would work.

Edit the performance template to which you wish to add the graph. Click the drop-down arrow next to Graph Definitions and choose "Add Graph..." and name it.

Click on the Graph Custom Definition tab and you are presented with a blank slate for your new graph's definition. It may be easiest to start with an example. I entered the following custom graph definition:

DEF:BusyThreads-raw=${here/fullRRDPath}/appThreads_BusyThreads.rrd:ds0:AVERAGE
DEF:RequestsPerSecond-raw=${here/fullRRDPath}/appThreads_RequestsPerSecond.rrd:ds0:AVERAGE
DEF:AppCurrentConnections-raw=${here/fullRRDPath}/currentConnections_appCurrentConnections.rrd:ds0:AVERAGE
CDEF:connectionsToThreads=AppCurrentConnections-raw,1,RequestsPerSecond-raw,BusyThreads-raw,+,+,/
LINE:connectionsToThreads#00cc00:"Connections to Threads/Thread Activity Ratio"
GPRINT:connectionsToThreads:LAST:cur\:%5.2lf%s
GPRINT:connectionsToThreads:AVERAGE:avg\:%5.2lf%s
GPRINT:connectionsToThreads:MAX:max\:%5.2lf%s\j

To break this apart, I have two data sources and three data points involved in my ratio that are part of the performance template with this graph:

Data source: appThreads
This data source has two data points, BusyThreads and RequestsPerSecond.

Data source: currentConnections
This data source has one data point, appCurrentConnections.

What I want to graph is a ratio based on these data points as follows:

appCurrentConnections / (BusyThreads + RequestsPerSecond + 1)

Basically, I want a measure of the amount of work in the queue (current connections) divided by the amount of work output my application is producing (a combination of the busy threads and requests per second it is handling, plus one to avoid the possibility of a divide-by-zero error).

With that established, we need to define the RRD DEFs (variables) used in the graph, one for each of the variables in the above calculation. Here's the one for the busy threads variable. I supplied BusyThreads-raw as the name that is used in the graph line:

DEF:BusyThreads-raw=${here/fullRRDPath}/appThreads_BusyThreads.rrd:ds0:AVERAGE


The key above is the TALES expression to get the variable from the Zenoss performance template into our RRDTool DEF variable: ${here/fullRRDPath}/dataSourceName_dataPointName

Regarding the :AVERAGE at the end: While there are many different RRD functions, the most common one I've seen used is the AVERAGE function, which takes a recent rolling average of the value in question. Please consult the RRDTool documentation for going deeper with this.

After providing DEF lines for each variable in my calculation, I need a CDEF line (calculated definition) for the actual calculation that puts the calculation together:

CDEF:connectionsToThreads=AppCurrentConnections-raw,1,RequestsPerSecond-raw,BusyThreads-raw,+,+,/

The calculation uses reverse Polish notation and the CDEF tutorial above has an excellent guide to understanding it, but basically you can think of it as a stack: the variables and constants are pushed onto the stack in order from left to right, and when the first operator (the leftmost plus sign) hits the stack, the top two items (in this case, the BusyThreads-raw and RequestsPerSecond-raw variables) are popped off the stack and added together (the operator is applied). The result is pushed back onto the stack. The next plus sign adds this sum with 1, and finally the division operator divides the AppCurrentConnections-raw variable by the topmost stack item (1 + RequestsPerSecond-raw + BusyThreads-raw).

Once we have our connectionsToThreads variable, we can graph it. The next line defines the one line on our graph:

LINE:connectionsToThreads#00cc00:"Connections to Threads/Thread Activity Ratio"

It refers to the connectionsToThreads variables, defines a color in hex notation, and defines a label. Finally, we can print some additional information on the graph:

GPRINT:connectionsToThreads:LAST:cur\:%5.2lf%s
GPRINT:connectionsToThreads:AVERAGE:avg\:%5.2lf%s
GPRINT:connectionsToThreads:MAX:max\:%5.2lf%s\j


Here we print the last, average, and maximum values of our graph line on the currently-viewed graph section.

Limitation: Thresholding
One thing I could not get working was to define a threshold based on my calculated value above. It seems that the thresholds are only valid on the values of the data points themselves, and I couldn't get a threshold working on my derived value above.

Moving a Zenoss event to history via the Transform expression

(This tip is also on the Zenoss Wiki.)

There are cases where certain events are just noise and you want them moved to history automatically, but perhaps without having ALL of the events in that event class moved to history. For example, you may wish to move certain events from one event class to another based on matching text and at the same time have these go straight to history.

To do this, enter the following in Transform of the event class mapping:

evt._action="history"

Move an event in Zenoss from one event class to another based on event text

(This tip is also on the Zenoss Wiki.)

Many events map to the /App/Failed event class, most notably the Windows Application Error_1000 error (http://<your Zenoss server>:8080/zport/dmd/Events/App/Failed/instances/Application%20Error_1000). I wanted to move some of these Application Error_1000 events to other event classes based on matching particular applications, but to leave the rest in /App/Failed. How does one do this?

To begin, confirm that you have an existing event class to receive the events. If not, create a new one by navigating through the "Events" tree from the left navigation to get to the desired parent class, and once there, click the drop-down arrow next to Subclasses and choose "Add New Organizer..." Enter the name for the new event class, e.g. "MyApplication."

Second, map an additional event class mapping to Application Error_1000. In /zport/dmd/Events/App/Failed, click the drop-down arrow to the left of EventClass Mappings and choose "Add Mapping..." For the ID of the mapping, type Application Error_1000_<name of the application to handle differently>, e.g. "Application Error_1000_MyApplication." (This event class mapping doesn't have to be named this way, but it helps to have the application name as the suffix, so that the mapping gets grouped with any other Application Error_1000 mappings in the list.)

Once you have done this, edit the properties of the new mapping. There are three key things you need to set:

• Event Class Key: Set this to: Application Error_1000
• Regex: I'm sure you can put in much more complicated regular expressions, but all that is necessary is to type some text from the event message, which will usually contain the application's executable name. If this is the case, all you need to enter is something like: MyApplication.exe
• Transform: Here, you need to key in the Python expression that will re-map the event to a different event class, e.g.: evt.eventClass="/App/Failed/MyApplication"
  

Save your changes to this new event class mapping. Now you need to sequence all the Application Error_1000* events so that this custom entry is matched first. Edit the new mapping and click on the Sequence tab. Make sure that your new mapping (Application Error_1000_MyApplication) has a lower sequence number than the generic Application Error_1000 entry. I'm not sure if the sequence numbers need to start at zero, but I've done it that way. So, make your new class sequence 0, and the generic Application Error_1000 class sequence 1. Don't forget to save your changes.

That's it--the events matching your custom event class mapping will be moved to the target event class, and all the others will be left in the original class.

Change Zenoss event severity based on message text

(This tip is also on the Zenoss Wiki.)

If you have events being mapped to a particular event class, generally one event severity gets applied to all of those events. If, however, you want to change the event severity of certain events based on the contents of the event message, do the following:

1. Navigate to the event class (under "Classes' in the left navigation, click Events, and then navigate to the event class containing the events you wish to conditionally map).
2. Using to drop-down arrow in the tab bar, choose More | Transform
3. In the Transform entry area, enter the following:

if evt.message.find("text to find") >= 0:
evt.severity = <desired severity>

For example:

if evt.message.find("timegenerated") >= 0:
evt.severity = 3

The above will change the severity of any events containing the text "timegenerated" from the default for the event class to 3 (warning). For your convenience, the event severity values are as follows:

Severity	Description
5	Critical
4	Error
3	Warning
2	Info
1	Debug
0	Clear

Zenoss Percentage Used for /Perf/Filesystem events

By default, Zenoss does not show the percentage of the filesystem used when showing disk space events (/Perf/Filesystem). The approach detailed below (also on the Zenoss Wiki) does the following:

• Shows both the percentage used in the event summary as well as the amount of free space remaining in GB
• Changes the event severity to critical (red) when the percentage free is 5% or less

Access http://:8080/zport/dmd/Events/Perf/Filesystem and enter the following for the transform:

fs_id = device.prepId(evt.component)
for f in device.os.filesystems():
    if f.id != fs_id: continue
    p = (float(f.usedBytes()) / f.totalBytes()) * 100
    freeAmtGB = (float(f.totalBytes() - f.usedBytes())) / 1024 / 1024 / 1024
    evt.summary = "Disk space low: %3.1f%% used (%3.2f GB free)" % (p, freeAmtGB)
    if p >= 95.0: evt.severity = 5
    break

Access Denied with Citrix WISP

I was attempting to install and activate the Citrix Web Interface for SharePoint on MOSS 2007 but was getting nowhere. The solutions appeared to add, deploy, and activate without error, but when I hit any of the Citrix administration links on the site collection root’s Site Settings page in the portal, all I received was 'Unknown Error.'

I had to first apply a registry hack to get the Citrix code to log anything that was going wrong. From its admin guide:

You can also find useful troubleshooting information in the log files stored in sharepoint\LOGS\LogFolder, where LogFolder is a location you can specify by setting a string value for LogFolder in the following registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\WISP.

Without creating this key, nothing is logged. After creating it, I could see a curious sequence of events when trying to activate the Citrix Access Core feature:

...
Adding List Item Event Receiver: Added
Adding List Item Event Receiver: Deleting
Adding List Item Event Receiver: Deleted
Elevating privilege, now running as DOMAIN\moss-pool-dev
Adding new configuration update job (ConfigUpdateJob-default-WIConfiguration(DMOSS01))
AddConfigurationUpdateJob() Error: Access denied.
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPJobDefinition.Update()
at Citrix.WISP.Configuration.Jobs.UpdateJob..ctor(String jobName, SPSite site, SPServer server, String configurationName, String configurationType, String filename, List`1 featureIds, SPJobLockType targetType)
at Citrix.WISP.Configuration.Jobs.Installer.<>c__DisplayClass2.b__0()
End of Elevating privilege, now running as DOMAIN\apenn
Adding new job CitrixAccessCoreDeployment(DMOSS01)(Citrix Access Core Service Provider Deployment) to DMOSS01(Application)
AddDeploymentJob() Error: Access denied.
at Microsoft.SharePoint.Administration.SPPersistedObject.Update()
at Microsoft.SharePoint.Administration.SPJobDefinition.Update()
at Citrix.WISP.AccessCore.Jobs.DeploymentJob..ctor(String jobName, SPSite site, SPServer server, SPJobLockType targetType, String title)
at Citrix.WISP.AccessCore.Jobs.Installer.<>c__DisplayClass2.b__0()
Updating Property Citrix.WISP.Site.Active in site https://devextranet.DOMAIN.local with value True
CitrixContentRedirectionModule: Add the web.config mod
- SPWebApp Name: Customer Extranet
- add/remove here: configuration/system.web/compilation/expressionBuilders
- the following:
CitrixAccessCore::Activate Error: Access to the path 'C:\Inetpub\wwwroot\wss\VirtualDirectories\devextranet.DOMAIN.local80\web.config' is denied.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.Xml.XmlDocument.Save(String filename)
at Microsoft.SharePoint.Administration.SPWebApplication.ApplyWebConfigModifications()
at Microsoft.SharePoint.Administration.SPWebService.ApplyWebConfigModifications()
at Citrix.WISP.AccessCore.FeatureReceiver.<>c__DisplayClass2.b__0()
at Microsoft.SharePoint.SPSecurity.CodeToRunElevatedWrapper(Object state)
at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass4.b__2()
at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param)
at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode)
at Citrix.WISP.AccessCore.FeatureReceiver.changeExpressionBuilderInWebConfig(SPWebApplication app, Boolean remove)
at Citrix.WISP.AccessCore.FeatureReceiver.FeatureActivated(SPFeatureReceiverProperties properties)
CitrixAccessCore: Ended at Monday, March 24, 2008 9:51:03 AM

It was apparent it was trying to take some actions under the application pool identity for the web application which failed with 'access denied,' and then was trying to use MY account (a domain administrator, MOSS farm administrator, and local MOSS server administrator) for other actions and STILL getting 'access denied!'

The administrator's guide for WISP does state:

To complete the installation you must have server farm administrator permissions, meaning you must be a member of the
administrators group on each of your Web and Application servers, and SQL security administrator with database creator rights on each of your SQL servers.

Those rights my account had; however, to correct the 'access denied' problem, I had to make my APPLICATION POOL IDENTITY:

• A member of the Farm Administrators group in MOSS; and
• A local machine administrator on the MOSS server.

C'mon Citrix, the application pool identity is supposed to be a low-privileged account.