FreePBX – Use CPU Affinity to isolate Asterisk from MySQL + Apache (httpd)

A lot of people are using FreePBX as their distribution of choice. It’s a pretty awesome program (not without flaws, but what is?) that simplifies configuration of asterisk to make it easy for anyone to deploy their own phone system. By the end of this article, you will be able to hammer the MySQL database and httpd servers as much as you like without damaging voice quality. Perfect!

The problem with FreePBX, Trixbox, Elastix, Piaf (PBX in a flash), is that they are ‘All-in-one’ packages. What this means is that the database, webserver, and everything are all rolled into one. While this is great, and allows you to do everything from one server, as you grow you will quickly see why this becomes a problem.

If you use the standard out-of-the box install, the biggest competitor for CPU cycles will the database. Particularly when running CDR reports. It’s not uncommon for a complex query on a large CDR with millions of records to cause CPU usage to spike from MySql.

This is especially compounded if using 3rd party addons like Asternic, which use highly complex queries to generate all sorts of nifty data.

Even on a huge quad, eight, or sixteen core server, you are likely to run into some voice quality issues. Now if you do have a super beast server, you will probably only notice short interruptions in voice quality in rare intervals, but if you have mid range hardware this can quickly become a problem, and you may think your only solution is to get another server, migrate the database (and change all of the configuration..), or ditch FreePBX and go with a costly, pure, custom config. And you would be right, well almost…

CPU affinity has been around for a while now, but it’s still a very under-used and under-rated tool. So under appreciated that on Linux, there is no way (currently) to specify that a certain program should always use a certain affinity. Shame!

Here is the theory:
Let’s say you have 16 logical cores, (4x dual core(x2) with hyperthreading(x2))
Asterisk is heavily multi-threaded, so let’s say asterisk is using all of the CPU’s equally, and then you get a webrequest asking for a huge database query. Well httpd will likely go to 100% (on one thread only) and mysqld will spike as well. Now since asterisk is balanced across all CPU’s whatever is on the same thread as the 100% httpd will have to wait, and that causes the rest of asterisk to come out of sync. Very bad.. causes choppiness, or silence periods.

The goal here is clear: Keep asterisk running separately from other intensive CPU tasks.

You set asterisk to use about 60% of your CPU cores, and both mysql and httpd get the rest. You can adjust this depending on your usage of course, and it might not be a bad idea to give mysql a little more than httpd just in case.

If you had 16 (0-15) logical cores, you would want to give asterisk affinity on 0-9, mysql affinity on 10-15, and apache/httpd affinity on 12-15 (just an example, use what makes sense for you.)
Note: Most of the time, each core will report as 2 cores due to hyperthreading. These are not actual cores, so you should group them by twos. For instance, don’t give cpu affinity 0-8 to asterisk, and then assign httpd 9-15 because 8 and 9 are both on the same physical core.

The command to achieve this is taskset. (provided by schedutils package, often included by default like on CentOS).

taskset -p -c 0-9 `pidof asterisk`

This will set the affinity for asterisk on processors 0-9.
taskset -p -c 10-15 `pidof mysqld`
This gives mysqld affinity of 10-15.

Http is a bit more complicated since it spawns processes as needed.
ps -A | grep httpd
Will give you a list of the PID’s of currently running httpd threads, and you can set affinity on there.

If there is any interest, I can publish a rewritten version of the script that I run every 2 minutes via crontab to keep CPU affinity at the level I want it.

I know this was a complex post, but if you take the first few steps mentioned here and complete the journey, your install will go a lot further.

Update! You asked for it, here it is: Affinity Manager.

This script will keep httpd in check, as it constantly spawns new processes, so must be constantly watched.

Super Important! You MUST modify the script for your exact CPU architecture. This example has 16 logical “cores” [includes hyper threaded cores]. It is up to you to make sure that you are choosing “units” on the same CPU/core.

The array $arAffinity contains numbers of the logical CPU units for each task. Adjust this as necessary.
The script also “renices” [adjusts priority] of httpd to 10 which is lower priority. (19 is lowest, -20 is the highest priority). This will help especially if you run FreePBX and have many users accessing web services at the same time as the phones are running.

It will iterate over each instance of httpd, which is good since it can spawn dozens of threads depending on the configuration.

Enjoy! – Maybe if I have time I will update the script more, but don’t count on it.\

As always, there are no guarantees or warranties of any kind on this software. Don’t come to me if it breaks your system 🙂

If you found this useful, please consider sending a donation, I wouldn’t be asking if I didn’t think it was fair.
I give away these tools that are highly valuable for free, so if you can afford to do so, please donate towards web costs, and other endeavors.