Further adventures with Citrix WISP installation

Continuing with earlier efforts to get WISP working, I attempted to re-add/re-deploy the solutions after first undeploying/removing them. Prerequisite steps:

Turn on WISP logging. You can import the following registry key to do so (save as a .reg file and double-click it; don't forget to actually create the folder):

------COPY BELOW THIS LINE-----
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\WISP]
"LogFolder"="C:\\temp\\WISP_logs"
-----COPY ABOVE THIS LINE-----

Turn on verbose MOSS (ULS) logging:

1. Open Central Administration, click Operations.

2. Click Diagnostic Logging.

3. Under the Event Throttling section, set Verbose in "Least critical event
   to report to the trace log."

4. Under the Trace log section, type 20 in Number of log files.

5. Click OK.

To add the solutions:

stsadm -o addsolution -filename CitrixWssCore.wsp
stsadm -o addsolution -filename CitrixAppDeliveryWebPart.wsp
stsadm -o addsolution -filename CitrixContentRedirection.wsp
stsadm -o addsolution -filename CitrixMossCore.wsp

While adding seems to be pretty safe, you have to be very careful about the order of deployment of the various WSPs. The CitrixWssCore.wsp must be deployed first, and this deployment doesn't necessarily work. You must carefully review stsadm output to see what your results were before proceeding. After initiating the deployment, check the Timer Job Status page in CA to see if the job is complete and/or run stsadm -o enumsolutions and look for the solution in the list with the Deployed node set to true.

stsadm -o deploysolution -name CitrixWssCore.wsp -immediate -allowgacdeployment

On this attempt, I was deploying CitrixWssCore.wsp first as directed. Note the value of the LastOperationResult node:

<Solution Name="citrixwsscore.wsp">
  <Id>fe3deba9-9b5d-4105-9983-2af1db3c0e42</Id>
  <File>CitrixWssCore.wsp</File>
  <Deployed>FALSE</Deployed>
  <WebApplicationSpecific>FALSE</WebApplicationSpecific>
  <ContainsGlobalAssembly>TRUE</ContainsGlobalAssembly>
  <ContainsCodeAccessSecurityPolicy>FALSE</ContainsCodeAccessSecurityPolicy>
  <LastOperationResult>DeploymentFailedFileCopy</LastOperationResult>
  <LastOperationTime>4/21/2008 4:14 PM</LastOperationTime>
</Solution>

I didn't find any particular reason for this failure, but I repeated the operation, and the next time got success values in these nodes.

Once I deployed the key prerequisite package, it was on to the others. The CitrixAppDeliveryWebPart also balked:

<Solution Name="citrixappdeliverywebpart.wsp">
  <Id>8a0a1be2-7648-4703-9cca-8ea0fa625793</Id>
  <File>CitrixAppDeliveryWebPart.wsp</File>
  <Deployed>FALSE</Deployed>
  <WebApplicationSpecific>TRUE</WebApplicationSpecific>
  <ContainsGlobalAssembly>FALSE</ContainsGlobalAssembly>
  <ContainsCodeAccessSecurityPolicy>TRUE</ContainsCodeAccessSecurityPolicy>
  <LastOperationResult>DeploymentFailedFeatureInstall</LastOperationResult>
  <LastOperationTime>4/22/2008 10:34 AM</LastOperationTime>
</Solution>

This also occurred on my second attempt. So I started digging through the MOSS ULS logs (it's a good idea to turn on verbose logging before starting operations like this, as exceptions do not generally get logged to the Windows event log). I found the following key message:

Line 19554 : 04/22/2008 10:41:33.12 OWSTIMER.EXE (0x058C) 0x1404 Windows SharePoint Services Topology 8zpd High Solution Deployment : Error - Add Feature definition for citrixappdeliverywebpart.wsp Exception message - A feature with ID 94af8a34-19db-4114-876d-5a7a587a8405 has already been installed in this farm. Use the force attribute to explicitly re-install the feature.

Apparently, the uninstall procedure (I had made earlier attempts to install these solutions) did not properly remove the features. I was able to get this one to pass by adding the -force switch to the stsadm deploy solution command.

stsadm -o deploysolution -name CitrixAppDeliveryWebPart.wsp -immediate -allowgacdeployment -allowcaspolicies -url http://server -force

After this, I forced the remaining solution deployments as well (CitrixContentRedirection.wsp and CitrixMossCore.wsp) and the rest succeeded.

stsadm -o deploysolution -name CitrixContentRedirection.wsp -immediate -allowgacdeployment -force

stsadm -o deploysolution -name CitrixMossCore.wsp -immediate -allowgacdeployment -force


After deployment of these features comes activation. Unless your application pool identity has admin privileges in various areas (this should not be the case), you will need to use the stsadm commands to activate the features. You must activate the CitrixAccessCore feature, and you must activate it before the others. Fortunately, activation does not use a timer job, so you get immediate feedback from stsadm about the success or failure of your request.

stsadm -o activatefeature -name CitrixAccessCore -url http://server
stsadm -o activatefeature -name CitrixTopNavigation -url http://server
stsadm -o activatefeature -name CitrixQuickLaunchNavigation -url http://server
stsadm -o activatefeature -name CitrixAdministrationTool -url http://server
stsadm -o activatefeature -name CitrixAppDeliveryWebPart -url http://server
stsadm -o activatefeature -name CitrixContentRedirectionModule -url http://server
stsadm -o activatefeature -name CitrixContentRedirectionMenu -url http://server
stsadm -o activatefeature -name CitrixContentRedirectionNewMenu -url http://server


The key step after getting everything installed is to configure the connection to the Citrix farm. From the site collection root, go to Site Settings | Modify All Site Settings > Citrix Administration and specify the applicable settings. At first I tried going to the Advanced Administration and copying the contents of our Citrix farm's WebInterface.conf file into the large textbox, but that is not sufficient.

Now if I could just get past the "No Resources found" message presented by the Citrix Application Delivery web part...

*** UPDATE ***
I got the Application Delivery web part working. It turned out the CitrixWssCore feature did not deploy properly in spite of the success message returned by stsadm -o enumsolutions. The contents of the WISP logs were the only place I saw the exception:

AccessCore.DeploymentJob: Wednesday, April 23, 2008 1:36:01 PM
Service Provider Deployment run as Identity: DOMAIN\server-farm-account
CreateEventLog() Error: Requested registry access is not allowed.
at System.ThrowHelper.ThrowSecurityException(ExceptionResource resource)
at Microsoft.Win32.RegistryKey.OpenSubKey(String name, Boolean writable)
at System.Diagnostics.EventLog.CreateEventSource(EventSourceCreationData sourceData)
at System.Diagnostics.EventLog.CreateEventSource(String source, String logName)
at Citrix.WISP.Util.CoreLog.CreateEventLog(CultureInfo locale)
....
Info: Attempting to create Web Virtual Directory (CitrixAccessPlatform-f6ec3ff1-328a-4fdf-b78b-61f0f5b703d0):
ADS Path: IIS://localhost/W3SVC/1634379371/Root/f6ec3ff1-328a-4fdf-b78b-61f0f5b703d0
File Path: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\CitrixAccessPlatform\f6ec3ff1-328a-4fdf-b78b-61f0f5b703d0
IISObject: Exception has been thrown by the target of an invocation.

Our MOSS server farm account was not a local machine administrator on the SharePoint machine, and apparently this is required for the CitrixWssCore deployment to create the custom event log and its web site. I had to grant these rights and the redeploy the solution with the -force option to correct it. The key point is that you must review all possible logs, including the WISP logs, to check for installation/deployment problems.

Citrix, won't you please build a real installer for WISP that will take care of some of these details/checks?

Infrequent IP address changes and No-IP

I use No-IP to provide dynamic DNS services so I can have remote access to my machine at home. However, the No-IP client doesn't send updates when my IP address doesn't change, and with my provider it tends to not change for quite some time. This causes No-IP to send me warning messages that my host is going to be deactivated from inactivity. In these notices, there is a link you can click to keep your host alive with its current IP. That got me thinking how I could force a periodic update.

I wrote the following script, pagecheck, to allow fetching an arbitrary web page and checking for some simple content in the page output:

#!/bin/bash
#Gets a web page and searches it for the specified text; if not found, or if a wget error results, returns
#an error code and prints error text.
#Arthur Penn - 16 Apr 2008

if [ $# -ne 2 ]; then
echo "pagecheck \"URL to page\" \"Success text to expect\""
exit 1
fi

PAGE=$(wget --no-verbose -O - "$1" 2>&1)
RC=$?
if [ 0 -eq $RC ]; then
SUCCESS=$(echo "$PAGE" | grep "$2")
if [ -n "$SUCCESS" ]; then
exit 0
else
echo "Did not find success message of \"$2\" in $1:"
echo "$PAGE"
exit 1
fi
else
echo "$PAGE"
exit 1
fi


This uses wget to fetch the web page and look for the content, and only prints output when it encounters problems. This makes it suitable for cron jobs. I added this script to /usr/local/bin (save it to a text file, and then: sudo cp pagecheck /usr/local/bin && sudo chmod +x /usr/local/bin/pagecheck). I then added the following script into the /etc/cron.monthly folder so it gets run once per month (don't do this more often to avoid excessive No-IP updates):

#!/bin/bash
# Touches the no-ip.com host dialog to confirm that the URL is still in use
/usr/local/bin/pagecheck "http://www.no-ip.com/hostactive.php?host=myhost&domain=noipdomain.net" "Update Successful"


T0 use this, you need to update the portions in red to match your No-IP domain (e.g. if your No-IP domain is fred.atx.net, the host would be "fred," and the domain would be "atx.net."

Since doing this, I haven't gotten any of the host deactivation messages from No-IP.

Make zenwin and zenwinmodeler ignore WMI errors

(This tip is also on the Zenoss wiki.)

At least in version 2.1.1, zenwin, zenwinmodeler, and zeneventlog have (IMO) a critical defect: if there are any /Status/WMI/Conn issues not in history for the device, they ignore the device. On our network, for some reason we end up with a lot of these events ('timegenerated' errors, various intermittent failures to connect, etc.). This causes the monitoring of our Windows servers to dramatically fall off as the system runs, and we miss critical issues.

I changed the behavior of these three systems to go ahead and attempt monitoring even if WMI issues are encountered. I learned that most of the time these WMI issues are spurious and successful monitoring CAN still be attempted. If you use this code, I recommend combining it with event commands to restart the zenoss daemons when it finds them dead.

Also, in zenwin, I added/improved the exception handling; a failure to create the watcher object occurs outside of a try block. Much of this code is an attempt to keep zenwin from crashing if it tries to monitor a Windows Server 2008 machine (Zenoss is not compatible with WS 2008 or Vista's WMI interface, and zenwin cannot monitor services on these devices). I ended up adding a hardcoded exclusion list so I can otherwise monitor the machine but have zenwin skip it. For some reason, zeneventlog seems to not crash, although it is not able to retrieve events from the WS 2008 machine either.

Please see the Zenoss wiki for the zenwin and zenwinmodeler diffs.

Find Zenoss event classes with transforms

(This tip is also on the Zenoss wiki.)

If you have entered transforms but can't remember where you entered them, type the following in zendmd (run zendmd from the command line on the Zenoss server as the zenoss user):

>>> for ec in dmd.Events.getSubOrganizers():
...      if ec.transform:
...      print ec.getOrganizerName()

Custom Zenoss graph based on multiple data points

(This tip is also on the Zenoss Wiki.)

If you want to make a custom graph in Zenoss based on more than one data point (such as a ratio or other calculation), you will need to enter a custom graph definition for RRDTool to use. I found some good guides on how to define graphs with RRDTool (such as this tutorial on CDEF and others at that site), but it took me a while to put this together with the available data points and variables in Zenoss so the graph would work.

Edit the performance template to which you wish to add the graph. Click the drop-down arrow next to Graph Definitions and choose "Add Graph..." and name it.

Click on the Graph Custom Definition tab and you are presented with a blank slate for your new graph's definition. It may be easiest to start with an example. I entered the following custom graph definition:

DEF:BusyThreads-raw=${here/fullRRDPath}/appThreads_BusyThreads.rrd:ds0:AVERAGE
DEF:RequestsPerSecond-raw=${here/fullRRDPath}/appThreads_RequestsPerSecond.rrd:ds0:AVERAGE
DEF:AppCurrentConnections-raw=${here/fullRRDPath}/currentConnections_appCurrentConnections.rrd:ds0:AVERAGE
CDEF:connectionsToThreads=AppCurrentConnections-raw,1,RequestsPerSecond-raw,BusyThreads-raw,+,+,/
LINE:connectionsToThreads#00cc00:"Connections to Threads/Thread Activity Ratio"
GPRINT:connectionsToThreads:LAST:cur\:%5.2lf%s
GPRINT:connectionsToThreads:AVERAGE:avg\:%5.2lf%s
GPRINT:connectionsToThreads:MAX:max\:%5.2lf%s\j

To break this apart, I have two data sources and three data points involved in my ratio that are part of the performance template with this graph:

Data source: appThreads
This data source has two data points, BusyThreads and RequestsPerSecond.

Data source: currentConnections
This data source has one data point, appCurrentConnections.

What I want to graph is a ratio based on these data points as follows:

appCurrentConnections / (BusyThreads + RequestsPerSecond + 1)

Basically, I want a measure of the amount of work in the queue (current connections) divided by the amount of work output my application is producing (a combination of the busy threads and requests per second it is handling, plus one to avoid the possibility of a divide-by-zero error).

With that established, we need to define the RRD DEFs (variables) used in the graph, one for each of the variables in the above calculation. Here's the one for the busy threads variable. I supplied BusyThreads-raw as the name that is used in the graph line:

DEF:BusyThreads-raw=${here/fullRRDPath}/appThreads_BusyThreads.rrd:ds0:AVERAGE


The key above is the TALES expression to get the variable from the Zenoss performance template into our RRDTool DEF variable: ${here/fullRRDPath}/dataSourceName_dataPointName

Regarding the :AVERAGE at the end: While there are many different RRD functions, the most common one I've seen used is the AVERAGE function, which takes a recent rolling average of the value in question. Please consult the RRDTool documentation for going deeper with this.

After providing DEF lines for each variable in my calculation, I need a CDEF line (calculated definition) for the actual calculation that puts the calculation together:

CDEF:connectionsToThreads=AppCurrentConnections-raw,1,RequestsPerSecond-raw,BusyThreads-raw,+,+,/

The calculation uses reverse Polish notation and the CDEF tutorial above has an excellent guide to understanding it, but basically you can think of it as a stack: the variables and constants are pushed onto the stack in order from left to right, and when the first operator (the leftmost plus sign) hits the stack, the top two items (in this case, the BusyThreads-raw and RequestsPerSecond-raw variables) are popped off the stack and added together (the operator is applied). The result is pushed back onto the stack. The next plus sign adds this sum with 1, and finally the division operator divides the AppCurrentConnections-raw variable by the topmost stack item (1 + RequestsPerSecond-raw + BusyThreads-raw).

Once we have our connectionsToThreads variable, we can graph it. The next line defines the one line on our graph:

LINE:connectionsToThreads#00cc00:"Connections to Threads/Thread Activity Ratio"

It refers to the connectionsToThreads variables, defines a color in hex notation, and defines a label. Finally, we can print some additional information on the graph:

GPRINT:connectionsToThreads:LAST:cur\:%5.2lf%s
GPRINT:connectionsToThreads:AVERAGE:avg\:%5.2lf%s
GPRINT:connectionsToThreads:MAX:max\:%5.2lf%s\j


Here we print the last, average, and maximum values of our graph line on the currently-viewed graph section.

Limitation: Thresholding
One thing I could not get working was to define a threshold based on my calculated value above. It seems that the thresholds are only valid on the values of the data points themselves, and I couldn't get a threshold working on my derived value above.

Moving a Zenoss event to history via the Transform expression

(This tip is also on the Zenoss Wiki.)

There are cases where certain events are just noise and you want them moved to history automatically, but perhaps without having ALL of the events in that event class moved to history. For example, you may wish to move certain events from one event class to another based on matching text and at the same time have these go straight to history.

To do this, enter the following in Transform of the event class mapping:

evt._action="history"

Move an event in Zenoss from one event class to another based on event text

(This tip is also on the Zenoss Wiki.)

Many events map to the /App/Failed event class, most notably the Windows Application Error_1000 error (http://<your Zenoss server>:8080/zport/dmd/Events/App/Failed/instances/Application%20Error_1000). I wanted to move some of these Application Error_1000 events to other event classes based on matching particular applications, but to leave the rest in /App/Failed. How does one do this?

To begin, confirm that you have an existing event class to receive the events. If not, create a new one by navigating through the "Events" tree from the left navigation to get to the desired parent class, and once there, click the drop-down arrow next to Subclasses and choose "Add New Organizer..." Enter the name for the new event class, e.g. "MyApplication."

Second, map an additional event class mapping to Application Error_1000. In /zport/dmd/Events/App/Failed, click the drop-down arrow to the left of EventClass Mappings and choose "Add Mapping..." For the ID of the mapping, type Application Error_1000_<name of the application to handle differently>, e.g. "Application Error_1000_MyApplication." (This event class mapping doesn't have to be named this way, but it helps to have the application name as the suffix, so that the mapping gets grouped with any other Application Error_1000 mappings in the list.)

Once you have done this, edit the properties of the new mapping. There are three key things you need to set:

• Event Class Key: Set this to: Application Error_1000
• Regex: I'm sure you can put in much more complicated regular expressions, but all that is necessary is to type some text from the event message, which will usually contain the application's executable name. If this is the case, all you need to enter is something like: MyApplication.exe
• Transform: Here, you need to key in the Python expression that will re-map the event to a different event class, e.g.: evt.eventClass="/App/Failed/MyApplication"
  

Save your changes to this new event class mapping. Now you need to sequence all the Application Error_1000* events so that this custom entry is matched first. Edit the new mapping and click on the Sequence tab. Make sure that your new mapping (Application Error_1000_MyApplication) has a lower sequence number than the generic Application Error_1000 entry. I'm not sure if the sequence numbers need to start at zero, but I've done it that way. So, make your new class sequence 0, and the generic Application Error_1000 class sequence 1. Don't forget to save your changes.

That's it--the events matching your custom event class mapping will be moved to the target event class, and all the others will be left in the original class.