Who Is Minding Your Logs?
I have always said that you can tell who the Nonstop Users are in the audience by the questions they ask, such as:
Is this scalable?
What is the performance?
How do I monitor it?
The last question highlights the importance that NonStop system managers place on providing responsive support for their end users. That's why system performance monitoring tools like Prognosis, MOMI and others are very popular at Nonstop installations. On the other hand, monitoring logs on the NonStop doesn't get a lot of attention beyond the use of EMS or VHS.
Yet it is an important topic that is worth a closer look. How would you answer the questions below?
When do you look at your logs?
"When there is a problem" seems to be a common response. Surprised? Not really. Given how overloaded everyone is, there is very little time to monitor logs. Unfortunately, "When there is a problem" usually translates to "User calls up and reports a problem." Through routine monitoring of logs, problems can be detected and fixed before they affect the users.
Do you know where your logs are?
"Well... not all of them."
When I speak of logs, most users think of system logs like VHS and EMS logs. Think again. Realistically, on a typical system, there are many other logs, such as:
Application logs - entries written out by COBOL, TAL, C, C++ and other applications.
OSS logs - Are your running iTP Web server, Java, NonStop SOAP, etc? There are lots of good information in the OSS logs.
Subsystem logs - Some subsystems have their own logs.
3rd-party application logs - If you are running a 3rd-party application, e.g. Websphere MQ, chances are it has its own logs
So if you don't know where your logs are, trying to hunt them down to resolve a problem can be time-consuming and detrimental to your service level agreement (SLA) with your end-users.
Are you overloading your logs?
Some users try to alleviate this problem by routing EVERYTHING to EMS or VHS. Unfortunately, that creates another problem: log overload. When that happens, basically Operations staff stop looking at the hundred of log messages that fly off the screen. Yes, EMS has a very nice filtering capability that can be configured to filter out different types of messages. But the reality is that not everyone is up to the task of configuring filters, and certainly not for tens if not hundreds of different message types that flood the EMS log.
"What's in your wallet?"
I meant.. logs. If my guess is correct, chances are your logs contain more than just error messages. They probably have warnings, statistics and "stuff programmers write to log for their own use." That's another key reason a lot of people do not bother looking at logs: the content overwhelms them. See if this conversation sounds familiar:
Operator: "I just saw this message in the log. What should I do with it?"
Answer: "Oh, don't worry about it. It's a debugging statement."
Operator: "What about that one?"
Answer: "You can ignore that one, too."
The result is a misinterpretation: no need to look at log messages unless someone reports a problem.
Monitor your logs
It can help you improve the quality of your end-user service.
Know your logs
Know what is in them and where they are. There's usually a lot of useful information in those logs besides error messages.
Have a log monitoring strategy
Take control and plan how you want to use the log information.
Automate as much as possible
Have a procedure in place and automate it whenever possible.
As more users are starting to use OSS in one form or another, such as iTP Web Server, Java or SQL/MX, it becomes more important to pay attention to some of the logs that reside in the OSS space. In my next article, I will focus on these OSS logs.
Do you find this tutorial blog helpful? Let us know what you think, and how we can make it even better. Don't forget, you can subscribe to our blogs (top right-hand corner of this page) to get automatic email notification when a new blog is available.
||Phil Ly is the president and founder of TIC Software, a New York-based company specializing in software and services that integrate NonStop with the latest technologies, including Web Services, .NET and Java. Prior to founding TIC in 1983, Phil worked for Tandem Computer in technical support and software development.