Taming the big, bad, nasty websites
Very easy:
#filter ReplaceNone
${myUnicodeString}
#end filter
in order to prevent escaping non-ASCII characters.
Now. That’s better.
Two free open-source tools are running the show for network and server-activity monitoring. The oldest and quite popular among network and system administrators is Nagios. Nagios does not only do monitoring, but also event traps, escalation and notification. The younger challenger is called Cacti. Unlike Nagios, it’s written in a scripting language [PHP] so no compiling is necessary – it just runs out of the box1. Cacti’s problem is that – at its current version – is missing lots of real-time features such as monitoring and notification. All these features are scheduled to be integrated in future versions of the product, but as with any open-source roadmap nothing is guaranteed, Anyway, this article is focusing on Cacti integration because it’s what I am currently using.
Cacti is built upon an open-source graphing tool called MRTG and a communication protocol SNMP. SNMP is not exactly a developer’s cup of tea, being more of a network administrator’s tool2. However, a monitoring server comes extremely handy in performance measurement and tuning, especially for complex performance behavior which can only be benchmarked long-term : such as large caches impact on a web application, or performance of long-running operations.
But is that specific variable you need to monitor, available with SNMP out of the box ? There is a strong chance it is. SNMP being an extensible protocol, lots of organization have recorded their own MIBs and respective implementations. Basically, a MIB is a group of unique identifiers called OIDs. An OID is a sequence of numbers separated by dots, for instance ‘.1.3.6.1.4.1.2021.11′; each number has a special meaning in a standard object tree – this example, the meaning of ‘.1.3.6.1.4.1.2021.11′ is ‘.iso.org.dod.internet.private.enterprises.ucdavis.systemStats’. Even you can have your own MIB in the ‘.iso.org.dod.internet.private.enterprises’ tree, by applying on this page at IANA.
Most probably you don’t really need your own MIB, no matter how ‘exotic’ your monitoring is, because:
a) it’s already there, in the huge list of existing MIBs and implementations
and
b) you are not bound to the existing official MIBs, in fact you can create your own MIB as long as you replicate it in the snmp configuration on all the servers that you want to monitor.
To take a look at existing MIBs, free tools are available on the net, IMHO the best one being MibBrowser. This multiplatform [Java] MIB browser has a free version which should be more than enough for our basic task. The screen capture shown here depicts a “Get Subtree” operation on the ‘.1.3.6.1.4.1.2021.11′ MIB; the result is a list of single value MIBs, such for instance ‘.1.3.6.1.4.1.2021.11.11.0′ which has the alias ‘ssCpuIdle.0′ and value 97 [meaning that the CPU is 97% idle]. You can see the alias by loading the corresponding MIB file [select File/Load MIB then choose 'UCD-SNMP-MIB.txt' from the list of predefined MIBs].
From command line, in order to display existing MIB values, you can use snmpwalk:
snmpwalk -Os -c [community_name] -v 1 [hostname] .1.3.6.1.4.1.111111.1
3 and the result is:
.1.3.6.1.4.1.2021.11 OID (.iso.org.dod.internet.private.enterprises.ucdavis.systemStats) snmpwalk -v 1 -c sncq localhost .1.3.6.1.4.1.2021.11 UCD-SNMP-MIB::ssIndex.0 = INTEGER: 1 UCD-SNMP-MIB::ssErrorName.0 = STRING: systemStats UCD-SNMP-MIB::ssSwapIn.0 = INTEGER: 0 UCD-SNMP-MIB::ssSwapOut.0 = INTEGER: 0 UCD-SNMP-MIB::ssIOSent.0 = INTEGER: 4 UCD-SNMP-MIB::ssIOReceive.0 = INTEGER: 2 UCD-SNMP-MIB::ssSysInterrupts.0 = INTEGER: 4 UCD-SNMP-MIB::ssSysContext.0 = INTEGER: 1 UCD-SNMP-MIB::ssCpuUser.0 = INTEGER: 2 UCD-SNMP-MIB::ssCpuSystem.0 = INTEGER: 1 UCD-SNMP-MIB::ssCpuIdle.0 = INTEGER: 96 UCD-SNMP-MIB::ssCpuRawUser.0 = Counter32: 17096084 UCD-SNMP-MIB::ssCpuRawNice.0 = Counter32: 24079 UCD-SNMP-MIB::ssCpuRawSystem.0 = Counter32: 6778580 UCD-SNMP-MIB::ssCpuRawIdle.0 = Counter32: 599169454 UCD-SNMP-MIB::ssCpuRawKernel.0 = Counter32: 6778580 UCD-SNMP-MIB::ssIORawSent.0 = Counter32: 998257634 UCD-SNMP-MIB::ssIORawReceived.0 = Counter32: 799700984 UCD-SNMP-MIB::ssRawInterrupts.0 = Counter32: 711143737 UCD-SNMP-MIB::ssRawContexts.0 = Counter32: 1163331309 UCD-SNMP-MIB::ssRawSwapIn.0 = Counter32: 23015 UCD-SNMP-MIB::ssRawSwapOut.0 = Counter32: 13730
Each of this values has its own significance, like for instance ‘ssCpuIdle.0′ which announces that the CPU is 96% idle.
In order to retrieve just a single value of the list, use its alias as a parameter to the snmpget command, for instance
snmpget -Os -c [community_name] -v 1 [hostname] UCD-SNMP-MIB::ssCpuIdle.0
Sometimes, you want to monitor something which you do not seem to find in the list of MIBs. Say, for instance, the performance of a MySQL database that your’re pounding pretty hard with your webapp4. The easiest way of doing this is to pass through a script – snmp implementations can take the result of any script and expose it through the protocol, line by line.
Supposing you want to keep track of the values obtained with the following script:
#!/bin/sh
/usr/bin/mysqladmin -uroot status | /usr/bin/awk '{printf("%f\n%d\n%d\n",$4/
10,$6/1000,$9)}'
The mysqladmin command and a bit of simple awk magic display the following three values, each on a separate line:
It is interesting to not that, while the first value is instantaneous gauge-like, the following two are incremental, growing and growing as long as new queries and new slow queries are recorded. Will keep this in mind for later, when we will track these values.
But for now, let’s see how these three values are exposed through snmp. The first step is to tell the SNMP daemon that the script has an associated MIB. This is done in the configuration file, usually located at /etc/snmp/snmp.d. The following line attaches the script [for example /home/user/myscript.sh] execution to a certain OID:
exec .1.3.6.1.4.1.111111.1 MySQLParameters /home/user/myscript.sh
the ‘.1.3.6.1.4.1.111111.1′ OID is a branch of ‘.1.3.6.1.4.1′ [meaning '.iso.org.dod.internet.private.enterprises']. We tried to make it look ‘legitimate’ but obviously you can use here any sequence you want to.
After restarting the daemon, let’s interrogate Mibbrowser for the freshly created OID, see the following image snmpwalk -Os -c [community_name] -v 1 [hostname] .1.3.6.1.4.1.111111.1 ; the result is:
enterprises.111111.1.1.1 = INTEGER: 1 enterprises.111111.1.2.1 = STRING: "MySQLParameters" enterprises.111111.1.3.1 = STRING: "/etc/snmp/mysql_params.sh" enterprises.111111.1.100.1 = INTEGER: 0 enterprises.111111.1.101.1 = STRING: "0.900000" enterprises.111111.1.101.2 = STRING: "18551" enterprises.111111.1.101.3 = STRING: "108" enterprises.111111.1.102.1 = INTEGER: 0 enterprises.111111.1.103.1 = ""
Great ! Now we have the proof that it really works and our specific values extracted with a custom script are visible through SNMP. Let’s go back to Cacti and see how we can make some nice charts out of them5.
Cacti has this nice feature of defining ‘templates’ that you can reuse afterwards. My strategy is to define a data template for each one of the 3 parameters I want to chart, using the ‘Duplicate’ function applied to the ‘SNMP – Generic OID Template’.
On the duplicate datasource template, you have to change the datasource title, name to display in charts, data source type [use DERIVE for incremental counters and GAUGE for instantaneous values], specific OID and the snmp community. Do it for the three values.
Using the three new datasource templates, create a chart template for ‘MySQL Activity’. That’s a bit more complicated, but it boils down to the following procedure, repeated for each of the 3 data sources:
All the rest is really fine tuning – deciding for better colors, wether to use autoscale or fixed scale and so on. By now, your graph template should be ready to use.
Note that for the incremental values ['DERIVE' type data sources] I’ve used titles such as ‘Thousands queries/5 min’ – the 5 minutes come from the Cacti poller which is set to query for data each 5 minutes. The end result is something like this one :
On this real production chart you’ll see a few interesting patterns. For instance, at 3 o’clock in the morning, there is a huge spike in all the charted parameters – indeed, a cron’ed script was provoking this spike. From time to time, a small burst of slow queries is recorded – still under investigation. What is interesting here is that these spikes were previously undetectable on the load average chart, which look clean and innocuous:
To conclude, SNMP is a valuable resource for server performance monitoring. Often, investigating specific parameters and displaying them in tools such as Cacti can bring interesting insights upon the behavior of servers.
Some SNMP implementations in different programming languages:
1 If you’re running Debian, Cacti comes with apt so it’s a breeze to install and run [apt-get install cacti]
2 a bit out of the scope of this article, SNMP also allows writing values on remote servers, not only retrieving monitored values.
3 Replace [hostname] with the server hostname and [community_name] with the SNMP community – default being ‘public’. The SNMP community is a way of authenticating a client to a SNMP server; although the system can be used for pretty sophisticated stuff, most of the time the servers have a read-only passwordless community, visible only in the internal network for monitoring purposes.
4 In fact, a commercial implementation of SNMP for MySQL does exist.
5 The procedure described here applies to Cacti v0.8.6.c
Until now, I've always had a certain respect for Sybase database.
Based on their history, I thought that the product is a sort of MS SQL
without the glitzy features – think Las Vegas without the lights, the cowboy boots and Eiffel Tower. Which gives: huge crowds of fat tourists in tall, dull buildings. But you always have Celine Dion, right ?*
Wrong ! Sybase ASE is a large, enormous, huge piece of steaming …
ummm … code. Ok, ok, I'm a bit over-reacting. In fact, Sybase is a
very good database – if you are still living in the 90s and the only
Linux flavour that you are able to manage is RedHat ASE. Otherwise,
it's a huge … you know.
Let's not talk about the extreme fragility of this … this product.
You never know when it crashes on you without any specific reason.
Sometimes, it's the bad weather. Crash. Restart. Sometimes, you
flushed twice. Crash. Restart. And maybe, yes maybe, you spent more
than 12 minutes in your lunch break. Crash.
Let's just talk about the JDBC drivers – latest version, downloaded
from the site in a temporary moment of insanity. Man, this is cutting
edge. The sharpest cutting edge you'll ever find – the more advanced
JDBC drivers bar none ! Excepting of course the fact that these
classes were compiled in that memorable day of 6 January 2002 [the day
when the last Sybase JDBC drivers were compiled]. But, don't let such
obscure details ruining your enthusiasm. Just download and use them
and you'll be amazed at their unique features – it's the only JDBC
driver which manages to put down DbVisualizer in different and
innovative ways. I'm restarting the poor thing (dbvis) at least 2 or 3
times per day, when working with Sybase.
Also, as a little quasi-undocumented tip, the letter d from
jconn2d.jar does not mean development [drivers] as some of you would be
inclined to think. In fact, it means debug which is the
abbreviation for put me in production poor lousy bastard and I'll
start spewing reams of useless messages through all the logging APIs
known by man and a few yet to be discovered, therefore instantly
slowing down to a crawl your puny little software.
Ermmm … well, all this nice introduction just to humbly confess that
we do have a couple of them Sybase licences floating around here and
some of our production databases are on Sybase ASE. While I can assure
you that this is going to change at least on the web backend, it's
also true that the beast must be alive in order to keep our company up
and running. And, that's part of my job.
Which is of course very strange for an IT management job. But then again:
when your sole DBA is overwhelmed by a horde of Sybase-specific tasks
(like for instance the log configured to truncate at a specific
threshold which naturally [for Sybase] does not truncate, suddenly
throwing all the users into log suspend in the middle of the peak
production time) – you have to enter into the damn kitchen and do some
cooking !
My endeavour was to perform simple data mining in order to migrate
some reports from a legacy system [no, you don't really want to know
what system]. Due to my limited time, starting a mildly complex Java
reporting project was out of the question, so a scripting language was
the natural choice. Python being the first option – unfortunately,
finding a working Sybase driver for Python is a challenge in itself.
But, thank God for Jython and zxJDBC ! In just a few minutes I was
wiring the tables for reporting. Here's a fake code snippet which lists the 'orders' with amounts < 1000, and you can't go any simpler:
from com.ziclix.python.sql import zxJDBC
conn = zxJDBC.connect("jdbc:sybase:Tds:myserver.mydomain.com:4110/mydb",\
"user","pass","com.sybase.jdbc2.jdbc.SybDriver")
cursor = conn.cursor()
cursor.execute("select count(orderid) from orders")
nb_orders=cursor.fetchone()[0]
print '%.0f total orders ...'%(nb_orders,)
cursor.execute("select orderid, amount from orders where amount <1000")
oids=cursor.fetchall()
for o in oids:
print 'Order %.0f for amount %.2f'%(o[0],o[1])
cursor.close()
conn.close()
Easy as pie. And once you got one down, you got'em all. For instance,
yesterday evening I wrote in about an hour a small Jython script which
exports some data. The same export process (running about an order of
magnitude slower) needed previously a couple days of development in
FoxPro. Ah, FoxPro - very juicy subject, but I'll keep it for my next
horror story. Until then, don't forget, when you have a monster to
tame and no time at all: try Jython !
*Let's suppose for a very brief instant that having Celine Dion at a certain location is a positive thing.
At first, this might seem a mind-boggling combination. What do
jython and PHP have in common (excepting the fact that I am a Python fan
and my current consulting task is in a PHP project) ?
Well, internationalizing a PHP app is pretty much a trivial task.
If you are a sensible PHP programmer insisting to use PEAR instead of randomly choosing a script from the tons of snippets
populating the “scripting websites”, I18N is probably the
safest choice.
Maybe – for you – application maintainability and performance are not exactly important concerns.
For me, they are. This is why I chose to store internationalized texts in files rather than database.
I'd rather keep the database for real data, which is created, modified, aggregated and such.
And I'd rather like to have an internationalized error message on the screen even if the database is down.
Now we know that we'll use I18N and text will be kept in some php files. However, I am no professional translator and
have no desire to translate or to manually maintain the correspondence between translators files and PHP files
(no, translators won't modify PHP code, stop this nonsense right away).
Code generation comes immediately in mind.
Basically, my first idea was to investigate wether the files used by the translators can be quickly transformed to PHP,
and if I am able to generate their formats from my own files (aka. “roundtrip internationalization process” ?).
Unfortunately, this is not an easy task – as the only clue was that the translators use Office tools such as Word or Excel, because they
rely upon some specialized translation software integrated with these products.
The easy choice is Excel, since it allows a better organization of data than having to search for tables in a Word document.
The hard choice is the tool that I'd use for automatically reading and even generating Excel files.
The difficulty comes from the fact I don't have Windows with Office installed on my desktop, just Gentoo Linux and OpenOffice.
Thus, I am unable to write a simple Python script which could perform my generation tasks via automation.
Fortunately, this is not the first time I am confronted with the issue.
I happen to know that there is a very nice Java tool that I wholeheartedly
recommend for your Excel processing needs :
JExcelApi.
Still, Java is a heavyweight programming language – it would be a really bad idea to fire up the
Automation scripts are already in cron and there's also a nice text document explaining translators where to get
their files and where to put them after modification. The resulting script is not exactly fast, but this is tooling
and not production so this should not be a problem after all.
Whatever your project contraints are, give Jython a try and you'll be amazed … As they put it on the
Useless Python site – If it were any simpler, it would be illegal.
Finally there's a trick not quite related with Jyhon, nevertheless interesting.
There is an easy way of solving the problem of translating phrases with real data inside them, with easy parameter swapping.
We'll use the good old sprintf but not directly. We'll pass through a not so popular but extremely useful function,
call_user_func_array. Suppose that our example needs the
user name and authorization profile description to display inside a nice message. All you have to do is to define placeholders
in I18N files which would fit as the first argument for sprintf. The following example should make it clearer:
| localization/en/login.php |
$messages = array( 'loggedin'=>'You are authenticated successfully as user %1$s with profile %2$s.' ); $this->set($messages); |
| localization/fr/login.php |
$messages = array( 'loggedin'=>'Vous avez le profile %2$s en tant qu'utilisateur %1$s.' ); $this->set($messages); |
| Simple passing of multiple parameters to I18N in PHP. Example function without error processing or data domain checking. |
#this is the multiple parameter function
function complexTranslation($i18n, $label, $params)
{
return call_user_func_array('sprintf',array_merge(array($i18n->_($label)),$params));
}
|
| Then, you have to initialize your I18N object. This can be done in a generic manner for all pages. |
#specific I18N initialization stuff require_once 'I18N/Messages/File.php'; $g_language_dir = dirname($_SERVER['PATH_TRANSLATED']).'/localization/'; $i18n =& new I18N_Messages_File($g_langCode,$script_name,$g_language_dir); |
| Finally, use the function. |
#translate the successfull login message $loginbox = Tools::complexTranslation($i18n,'loggedin',array($operator->name,$profile->description)); |