Based on my experience with various servers, file systems, 32-bit vs. 64-bit, and available RAM every server is going to have some upward limit on how many individual items it can reasonably index in a given amount of time. The result of having numerous mail user’s folders that have crept up to 10,000 or beyond means that the Kerio Connect system can fall behind simply because it hasn’t finished indexing. If it can’t index it can’t add new messages to that user’s folder and so on (somewhat similar to a cascade failure).

We are running Kerio Connect on a fast Intel Xserve with a RAID 5 with 7,200 RPM SATA drives. Now we could change the drives to a RAID 0 to speed things up a bit but that’s not an option right now (actually I am giving serious thought to putting in a flash-based boot drive for the OS and Kerio Connect and have the data on the RAID) or we could find some other method of speeding up disk access but I feel these mostly are workarounds. The server just isn’t that slow.

The one thing that must happen is that each user’s mail folders must be kept under 10,000 messages and so I established a policy with a 5,000 message limit (this gives us some cushion). After the break are the details of how I am using a shell script, Lingon, and Splunk to help us efficiently keep track of each user’s mail folder contents.

When I contacted Kerio about our issue they provided me with a find command that searches through the index file for each folder and outputs the totals to a text file. I modified this command so that it outputs the text file to someplace that Splunk can index. If you are not familiar with Splunk it is a self-contained log file index and search tool. I have found it to be a very powerful and useful tool. It is also very expensive, probably worth it, but I have been unable to sell the firm’s shareholders on the idea of paying for it. What I do is run the free version on each individual server, instead of centrally on a single server, and this gets me what I need. The downside that is instead of managing one Splunk installation I am managing 10.

What happens is that once a week my script runs and outputs the data to the text file. Then once a week I search through the data to find who needs help with their folders. To get this to work I had to first figure out how to get Splunk to identify which folders in the text file are over 5,000 messages.

Because the file with the data is just a text file and not actual log data Splunk doesn’t know what to do other than treat each line as a separate log entry. It turns out it isn’t that difficult to get Splunk to recognize how understand the data. Splunk has a built-in regex tool to make this step easy. Below is an example of the basic search to identify all folders that have more than 5,000 items in them.

Then way Splunk works you can simply click on a user’s name in the search results and then it will automatically refine the search to include just that user. Below is the final search string for the user “donk” (that’s me) and below that is the result of the refined search.

So Apple finally made it official a few weeks ago an now has a Mac mini server. It fits the bill for many business uses, creative uses, as well as even for a simple home network server.

In fact one business makes it their business to use Mac minis almost exclusively as their servers, providing web hosting and other services to their customers.

13.34 hours of which I can account for only 25 to 30 minutes.

Mostly I find it interesting, assuming this usage is typical, that Apple has the time, both in the factory and in the facility in the US where Xserve orders are put together, to run a machine for 12 hours before shipping it. Even a self-proclaimed IT Guru still learns something new everyday.

I found that the Mailman system which is modified by Apple seemed to not be generating any log files. Upon closer inspection I found that the Mailman system was creating logs they were just not where Apple’s server OS was expecting them.

At this point in time I don’t know if this is due to a glitch during the original installation of if this is just the way Apple set it up. Since I have found a few other people with this or a very similar issue I suspect it’s a glitch in Apple’s set up.

I found that the Apple default configs are expecting the logs to be written to

/var/logs/mailman/

but the Mailman* system is configured to write them to

/var/mailman/logs/

I was concerned that a future update/upgrade from Apple might break the logging if I went and changed the Mailman config so instead I set up a symlink for /var/logs/mailman pointing to /var/mailman/logs and this seems to work both from the GUI or the CL. This almost seems like a typo on Apple’s part.

Just thought this might be useful to anyone supporting Apple servers.

* This is not the default Mailman location but Apple’s modified Mailman location

Back in late November I tried to get iCal Server running on some older hardware and this is what happened…

I ran iCal Server + iCal in a test environment on a older Xserve–Dual G4 1GHz 2GB RAM Mac OS X Server 10.5.5–in this test environment with only a couple of test users and I noticed a significant increase in CPU activity due to iCal but it seemed that it wasn’t something the server couldn’t handle. A month later when I rolled out the service to 60+ users the server was brought to its knees. It ramped up to 100% CPU usage as more users logged in and by about 11AM that business day it ceased to be functional. Anyone trying to connect timed out before they received updates.

I have not been able to track down any specifics as far as something I can “fix” to prevent this problem as my iCal Server install is Apple default and other than running the CPU load at 100% there was not a single error reported/logged. I’m going to use brute force to work around the issue but I would like to find a better approach.

I am going to try to roll out this service again but hosted on a brand new Xserve 3.0GHz Quad-Core Intel Xeon 16GB RAM (I’m lucky in that I work for a bunch of lawyers).

Now two months later I think I have finally got it working…

Here are the details of what we have running in our offices:

Server Hardware:
Xserve 3.0GHz Quad-Core Intel Xeon 16GB RAM, 1.6TB RAID 5. (complete overkill?)

Software/OS:
Mac OS X Server 10.5.6 (completely stock, no open-source mods) which runs iCal, iChat, and VPN. On our network we run Open Directory, Kerberos, and we manage our own internal DNS (all Mac OS X server based). Everything more or less works (although I can’t even keep track of how many things I’ve kludged to “force” Apple server stuff to work the way it is supposed to. For example my DNS is a clean BIND install as I completely removed the Apple BIND setup yet it still plays nicely with the Server Admin GUI tool.

Users:
65 users with about 20 having pre-existing calendars from NUTD each with about 1400 events that were imported into iCal. All my users are using various Mac systems running 10.5 (with a few throwbacks on 10.4). I expect that nearly all my users will give about 30-40 users read permissions to their calendars and 5-8 users read/write permission.

So what happened when I rolled it out this time?
As before it was very slow for all users on the first day which I expected. There are 52,079 events listed on the server, 56 user calendars so far and 16 groups. There have been some brief CPU spikes around 91% but the average CPU load works out to 26%. I am hoping that after a week or so it will start to average 18% – 20% during business hours. Before I loaded all the imported calendars and users started connecting the server load averaged just 2% with the same three main services running (iCal, iChat, VPN).

Once each user has their own calendar fully loaded as well as any delegated calendars the iCal client remains very responsive since events are first being cached locally. The background sync with 56 users actively using the system has–so far–had little impact on our network. We have three satellite offices that connect over dedicated VPNs (sonicWALLs) so their connection speeds are not always very good but each local network is Gigabit ethernet and 802.11g wi-fi. After the initial day’s surge the average bandwidth usage on the server is under 40 KB/s in/out with some spikes of course (at this point most of the data is outbound which makes sense).

It is sad that such a powerful server was required to host this system but I expect this is largely due to the fact that I was importing many pre-existing calendars from an earlier system. If the iCal system had been rolled out with initially empty calendars I might have been able to get by with a slightly lesser server (but in any case the dual G4 Xserve would never had supported so many users).