Engage 2016 User Group Conference
I had the opportunity to present at the Engage 2016 user group conference last week in Eindhoven. I was extremely impressed on all levels. Thank you to Theo Heselmans for the opportunity to speak at this event! The session titles, abstracts and links to the presentations on SlideShare are below.
Adm02 Be a Domino Detective: Tackling Your Toughest Performance Problems
Abstract: Become a Domino performance investigator. This session will teach you industry best practices for Domino performance optimization. Learn how to take abstract symptoms like “Notes is slow” and break it down to a resolvable problem. See the methodology and tricks involved to find the true culprit using tools such as semaphore timeouts, memory dumps and server monitoring. Understand what impact running with obsolete tuning parameters can have on your environment. You will learn the best tips to implement along with do’s and don’ts for ensuring your Domino environment will perform optimally.
Adm07 The Health Check Extravaganza for Social and Collaboration Environments
Abstract: Are you concerned about your infrastructure being configured correctly? Do you have problems happen that you don’t know how to prevent? Do you think your servers might have room for improvement? Wonder no more. This session will show you what you need to be looking at to ensure your server environment is running as cleanly and efficiently as possible. You will learn what you need to be looking for in your server configuration, problems found at numerous customer environments and what steps should be taken to remedy the various situations covered in this session. Be preventative, not reactive! Performing a health check is one of the most economical ways to ensure your social and collaboration environments are running properly.
Keeping Your IBM Collaboration Environment Healthy
One of the critical things my company does for businesses is keeping their IBM Collaboration environments healthy. We do this by reviewing the current environment and infrastructure, detailing our findings and making specific recommendations for improvement. In doing this work, we have noticed patterns of findings.
At the Connect 2016 conference, myself and my colleague Luis Guirigay are presenting “The Health Check Extravaganza for Social and Collaboration Environments“. We will be sharing with you our collective knowledge of our experiences in working with numerous customers in evaluating their collaboration environments. We will share with you what you need to be looking at to ensure your server environments are running as cleanly and efficiently as possible. Our goal is to be preventive, not reactive!
Performing a health check is one of the most economical ways to ensure your social and collaboration environments are running properly. I hope to see you at our session today, Tuesday, February 2nd, at 1:15 PM in room Lake Highland.
Impact of Lack of Memory on CPU Usage
One of our customers asked me to analyze the performance of the IBM i LPAR that hosts their primary Domino and Sametime servers as they had some concerns. The LPAR hosts 5 Domino servers (Mail, Application, Administration, Sametime, and Dev/Test) along with the Sametime DB2, SSC, Proxy, and Meeting servers.
My analysis revealed quite high CPU utilization at times, in addition to faulting issues in the *BASE memory pool. The *BASE memory pool is pool 2, where all of the Domino and WAS-based Sametime servers run by default. Some of the Domino servers (Administration, Dev/Test, and Domino Sametime) had previously been moved to separate memory pools.
To determine which servers were causing the bulk of the faulting the *BASE pool, I created a query against the QAPMJOBL performance monitor database file. The two top faulters were the Mail and Application servers. I made a recommendation to move the Mail and Application servers to their own memory pool, allocating 36 GB of memory to each memory pool as a starting point. I also recommended moving the Administration, Dev/Test, and Domino Sametime server back to the *BASE memory pool. This quite dramatically changed the memory allocations on the server.
The table below shows the memory allocations as they were when I performed my analysis.
This next table shows memory allocations after implementing my recommendations.
My memory tuning recommendations were implemented on October 25th. The impact on CPU utilization was quite dramatic as shown in the graphic below.
Proper memory allocation is key for the best Domino performance!
Traveler server with Red status and long running push threads
We had a customer contact us recently about their Traveler server. The server was reporting a status of Red, CPU on the server was extremely high, and there were a number of these errors in the log file:
Traveler: User CN=User Name/O=Org on thread Push-1ca3 has been running for 8820 minutes.
One correlation for the users reporting this error, they were all Android users. No other device types were reporting this error.
In digging a bit deeper, we saw this error for each affected user in the log:
SEVERE Push-14ad9 User Name[Android_f5a7931af273cbdc] ConnectionNotificationSenderGCM$SendMessageRunnable.run#460 There was an issue sending the notification message to the Google Cloud Messaging (GCM) server. 3 attempts to re-establish the connection have failed, so the notification message cannot be sent. You can test network connectivity including firewall settings by trying https://android.googleapis.com/gcm/send in a web browser on this server. Exception Thrown: java.net.SocketException: Connection reset
Running a trace from the physical server the Traveler server is hosted on revealed an inability to connect to https://android.googleapis.com/gcm/send
Google Cloud Messaging (GCM) became available in Traveler 126.96.36.199. It is the default synchronization option for Android devices when they download the 188.8.131.52 or later IBM Verse client.
The fix to this issue is two-fold:
- Enable GCM push via notes.ini NTS_PUSH_ENABLE_GCM=True and restart the Traveler server.
- Ensure the Traveler server can connect to https://android.googleapis.com/gcm/send
I Have Moved My Blog
Welcome to my new blog!
I had previously blogged on BleedYellow.com, however that site is now defunct, so I have moved my blog. I chose DominoDiva.com for my new blog as my blog handle on BleedYellow was DominoDiva.
I am very much looking forward to getting back into blogging!!