What Is That Program Doing? Or, Why Is This System Using So Much Memory!?

First, let's get a quick concept out of the way. When dealing with computers, new sysadmins have to understand that when it comes to a system compromise (a computer gets hacked), the system cannot be trusted. It's like asking a person to objectively assess their mental state; when you ask a person's brain to evaluate itself, it's going to be biased and distorted. Similarly, when a person cracks a computer system, they can alter the programs and filesystem so that if you run utilities to show network connections, running programs, etc., the altered files will hide the unauthorized activity.

Pretty simple, yeah?

So how do you see unauthorized activity? If your computer was hacked and is now sending spam email...which is network activity...but if you run utilities everything is showing up as normal (because the altered files are hiding the unauthorized network activity), the computer can hide what's going on, but a second system on the network that is monitoring network activity will see all the activity. In other words, your hacked computer may have its fingers in its ears yelling, "LA LA LA LA LA" but a third party system will still see what that computer is doing.

You add a layer of abstraction to watch what's going on, and in the process you can learn a bit about what is actually going on with your computer.

Now...the topic of of the post.

I have been working with a technology called virtualization for some projects at work. It's really great stuff...it lets you create computers that exist only in software, letting you install and configure multiple computers that run on just one system. Schizophrenic, but very very handy for people like me that have to run print servers, web servers, and file sharing servers in an organization among other things because I can consolidate those systems onto one or two actual physical systems (with tons of RAM and disk space).

In the process of running tests to migrate some of our physical systems to virtual software systems we moved a printer server. This is a Windows computer whose entire job on our network is to spool print jobs and dole them out to our many many printers scattered around several buildings. Not uncommon in businesses to have printer servers configured in their network for easier management.

Normally we'd think that this isn't a huge task, either. Most of the time a server like this would sit largely unused; I mean, it just sits waiting for someone to send a print job, render it then send it to the appropriate printer. It's not crunching numbers to predict the weather.

"But Barry," you ask, "What does this have to do with hackers and second computers to watch what a system is doing?" Glad you asked. See, when you run a full-on virtual server system...in this case what's called a type 1 hypervisor...you get an abstracted view of a computer; memory use, disk use, network use...and so you can get a quick overview of what the computer is doing that ordinarily you wouldn't get to see with a physical computer.

What I saw was that out of (more than five, fewer than ten systems) on our virtualization server the printer server was taking up nearly 800 meg of memory and shooting between first and second place for CPU usage and network usage, depending on what the other systems virtualized on the testbed were doing at that point. What in blazes is it doing?

I logged into the machine and ran a wonderful tool from the sysinternals suite called Process Explorer (free download...I highly recommend the sysinternals suite of software to ANY system administrator or troubleshooter). From there I could monitor process (program) names, the command line from which they were running, memory and CPU usage, etc.

I found a couple things of note. First, the converter program I used to automagically convert the physical machine to a virtual machine runs a service that carries a large footprint of memory; now that the system was virtual, the converter program isn't needed. So I removed that program from add/remove programs and memory use for that virtual image dropped over 100 megabytes.

Second, the computer is running a database program as well as a Bash shell spawning Java. Huh?

In case you didn't know, Bash is a shell program that runs normally under a Unix system (like Linux), not Windows.

Process Explorer told me that both the database and the Bash shell (and in turn the Java system) was tied to a Dell utility tied to Openmanage. Many mental scars remind me that OpenManage is a set of utilities made by Dell for managing their servers...usually it has functions for things like monitoring fans and CPU temps, rebuilding RAID arrays, etc. etc...generally a headache to sort everything out and get working in the proper combination for your system. At least, that's my experience with it.

So I go into add/remove programs to remove the OpenManage software, since now that the system is virtual there really wasn't any Dell hardware for it to manage.

Surprise! I was wrong. Apparently at some point someone installed an OpenManage component for managing printers! It wasn't small either. In the add/remove programs there was a listing for "Dell Printer Software" and for "OpenManage Printer Manager", each of which was taking 1.6 gigabytes in storage space on the hard disk. Each. Not both together.

But since some departments wanted Dell multifunction printers on the network and I didn't install it I don't know if these software packages are something that are needed so I can't really just tear through and uninstall those programs without dealing with the possibility of Nasty Consequences(tm).

Apparently Dell tries to cut some corners to make their software more portable between Linux and Windows by using Java (I'm speculating since I'm not too thrilled with OpenManage software, so I don't install it on my Linux systems on Dell hardware). Part of their software workflow involves using a program called Cygwin to do something with logging or some management task; Cygwin is a port of Unix utilities to run on Windows (that's why I saw Bash running). You can run tools like Secure Shell or awk or sed or ls...many many many scripting and administration utilities...on Windows that normally you'd only see on Linux. Process Explorer popped up lines in the process list as some of these scheduled tasks were periodically popping up thn disappearing, no doubt adding to the memory and resource use of the virtual machine. To be clear, Cygwin has it's place and I think it's great when properly used. I've also seen it installed on systems that suddenly start spiking CPU usage because of one of the programs using Cygwin libraries (in that case, SSHD running on Windows as a service).

What lessons can we learn from this little educational field trip?

• Generic is good. Addon software is bad. Backed up by anecdotes on the StackOverflow podcast, that CD that came with your camera/printer/device should not be installed unless you have no other choice but to use it to get the device to work. Many modern operating systems include drivers to work various media devices, or for devices like HP printers, you often can go right to the website and download just the driver, without all the addon crap that will bog down your computer with extra programs that you don't need (or know what they're really doing).
• Virtualization can give you tools that will both teach you about using your system and open your eyes to some things your computer(s) may be doing in the background without your knowledge. There's no reason that a printer server should be sucking down resources like this one was except that it had some poorly optimized software installed that it probably doesn't need.
• Third party utilities like the Sysinternals Suite can help you track down oddball activity on your system (or insights on how things work) for free. There are tools that tell you what's connected to the system over the network, which program is writing and reading the hard disk, which program is hitting what part of the registry, and many other useful tools. Try it out if you run Windows.
• Audit your system once in awhile to see what's actually installed and what it's doing. If it's not needed, free up the drive space by uninstalling that program. Use Google to figure out what the programs are. Part of what contributes to your computer slowing down over time is having programs running in the background that take up space in memory and access the network and you probably don't need them. Become familiar with your system and it can help save you aggravation down the road...also it'll help you later on when you notice something that should not be in that process list running, so you can tell when something is out of the ordinary on your computer. Process Explorer even includes tools for you to Google process names from a menu, making it even easier to learn about what your system is doing!

Windows isn't the only platform to suffer from bloated, inefficient and/or poorly designed software, but since Windows has the majority of users who are non-technical in interest and nature it is the platform with developers who get away with creating shovelware much more often. Fight it by not using it. I'm not referring specifically to Windows, although I encourage not using that too...but rather don't use the bloated crap that comes as trinkets and addons that only serve to bog down your computer. Don't install software from CD's that came with your new tech toys unless you must; try plugging it into the computer first to see if Windows or Linux or the Mac recognizes the hardware and installs the necessary drivers for you first. Then you're using native tools and not someone else's idea of how you should use their tools (and sometimes screw up your system in the process). Check your computer and screen it for odd behavior and find out what those processes in the background of your system are doing so you can get rid of software that is slowing your computer without justified cause.

Anyone have any stories they'd like to share?