ASO Can Quickly Boost
Performance for Free
ASO can quickly boost deliver performance for free
In 2011, IBM announced the Active System Optimizer (ASO) enhancement, available only on recent AIX levels for POWER7. The intent was to automatically apply best-practices performance tweaks to individual LPARs. Some of those tweaks include improvements to memory affinity—specifically associating targeted workloads to a specific core or set of cores and determining if memory can be relocated for higher affinity to cache and cores. An in-LPAR solution, ASO runs in the background of each LPAR it’s tuning. Keep in mind it doesn’t support active memory sharing or LPAR migration.
Getting Started
ASO is installed by default in a standard installation as part of the bos.aso fileset. Running lslpp –l | grep bos.aso shows the installed level set. The three ASO prerequisites are:
- AIX V6 TL08 or AIX V7
- The bos.aso fileset must be installed
- A POWER7/7+ server
To check the prerequisites:
# oslevel -s
7100-01-05-1228
This output shows the system is at: AIX V7 TL01 SP5.
# lslpp -l | grep aso bos.aso 7.1.1.15 COMMITTED Active System Optimizer bos.aso 7.1.1.15 COMMITTED Active System Optimizer # lsconf | grep ^Processor Processor Type: PowerPC_POWER7 Processor Implementation Mode: POWER 7 Processor Version: PV_7_Compat Processor Clock Speed: 3000 MHz
This indicates the system can run ASO without problems.
To use ASO, first start the subsystem, using startsrc –s aso. You can then monitor and control it using the asoo command. The following commands can be very useful:
# lssrc -s aso Subsystem Group PID Status aso 5373980 active
The ASO subsystem is started and is running as PID 5373980.
# ps -ef | grep 5373980
root 5373980 3604670 0 Jan 02 - 1:14 /usr/sbin/aso
The program being run is /usr/sbin/aso.
asoo –L List the current and reboot values # asoo -L NAME CUR DEF BOOT MIN MAX UNIT TYPE aso_active 0 0 0 0 1 boolean D #asoo -op aso_active=1 Permanently turn ASO on (0 is disabled and is the default)
To check what’s happening in the ASO logs; their location is specified in /etc/syslog.conf and defaults to:
aso.notice /var/log/aso/aso.log rotate size 128k time 7d aso.info /var/log/aso/aso_process.log rotate size 1024k
The aso.log records the on/off status and the reasons ASO goes into hibernation. Similarly, aso_process.log records the same information as well as details of any actions ASO takes and the processes it modifies.
My test LPAR is on a Power 720 and has an entitlement of 0.5, VPs set to 2 and 4GB of memory. The lssrad command shows I only have one socket, so there’s little to tune with respect to affinity.
# lssrad -av
REF1 SRAD MEM CPU
0
0 3699.81 0-11
Although the ASO subsystem is running, we still need to enable ASO as follows:
# asoo -o aso_active=1 Setting aso_active to 1
The log now shows:
Jun 16 17:32:46 testlpar aso:notice aso[5373980]: ASO enabled by tunable Jun 16 17:32:46 testlpar aso:notice aso[5373980]: [HIB] Current number of system virtual cpus too low (2 cpus) Jun 16 17:32:46 testlpar aso:notice aso[5373980]: [HIB] Increase system virtual cpus to at least 3 cpus to run ASO. Hibernating.
When planning where and when to run ASO, keep in mind it needs at least three virtual processors (VPs) to run. At this point, ASO was still enabled, so I disabled it again and used Dynamic LPAR to add another VP to the LPAR.# lsdev -C | grep proc proc0 Available 00-00 Processor proc4 Available 00-04 Processor proc8 Available 00-08 Processor # asoo -o aso_active=1 Setting aso_active to 1The log now shows:Jun 16 17:39:08 testlpar aso:notice aso[5373980]: ASO enabled by tunableUnfortunately, the test system wasn’t very busy so ASO decided to hibernate:LOG Jun 16 17:40:53 testlpar aso:notice aso[5373980]: [HIB] Used entitlement per unfolded vCPU is below threshold (1% of a core). Jun 16 17:40:53 testlpar aso:notice aso[5373980]: [HIB] ASO will hibernate until used entitlement is at least 30% of a core per unfolded vCPUInternal Analysis
On a busy system, ASO takes some time to profile and analyze the running workloads to dynamically tune the system for those workloads. Once it’s running, it needs no further interaction. ASO will examine AIX kernel data about processes and threads, and the POWER7 hardware performance counters. Based on that data, ASO makes tuning decisions. It will try to improve cache and memory affinity by moving workloads to cores that help them improve their affinity needs. It takes into account Workload Manager (WLM) and workload partition (WPAR) resource sets as well as constraints. It does this by looking at the workload over minutes, and setting scheduler resource allocation domain (SRAD) and resource set (RSET) rules in order to change the CPU a workload is running on. All changes are logged and tracked with ASO logging both the expected and actual gain seen.By default, ASO enables cache and memory affinity optimization, large page optimization, and memory prefetch. These restricted tunables should only be modified at the request of IBM Support.To improve the cache and memory performance of workloads, ASO uses one of three types of optimization: cache affinity, aggressive cache affinity or memory affinity. For cache affinity, it tries to reduce chip-to-chip cache movement. For a workload assigned cores across two sockets, ASO tries to place the workload so it fits within a socket using an RSET. For aggressive cache affinity, it tries to reduce the number of chips used and may compress the workload onto fewer cores to do so. For memory affinity, ASO tries to ensure that local memory is used. This often involves migrating pages for workloads to DIMMs closer to the core being used, thus reducing remote memory traffic. It uses CPU and memory RSETs to do this.Debugging
In the unlikely event you experience problems with ASO, the perfpmr.sh scripts don’t collect the ASO logs. So, you must set ASO into debug mode:asoo –o debug_level=3You’ll then need to recreate the problem and reset ASO to normal logging:asoo –o debug_level=0Finally, forward the aso_debug.out file to IBM Support.A Fine Tuning Tool
Free with AIX and easy to use, ASO offers great benefits for tuning. It monitors workloads on a busy system and dynamically moves them to gain the best affinity within an LPAR. With it, administrators can avoid making WLM and RSET changes to attain affinity.It can be turned off immediately if performance results are less than expected. It’s particularly useful for multi-threaded, long-running processes on larger systems, especially the Power 770 and above, which have multiple nodes. However, any of the multi-socket systems could benefit. Try ASO on test LPARs to gauge its potential benefits for performance.
