Y U iowait?

You know how it is. The cat of legend knows how it is. The engineer with the non enterprise system knows it:

One day you are running the last version, there is no pressing reason to upgrade, but you just cannot help yourself….

Ubuntu 14.10 Uptopic moving to 15.04 Vivid for no good or justifiable reason other than new shinier versions of things, and then an element of surprise as the wheels come off.

In my defense, this was the last upgrade after three desktops – who had all made the jump without issue… . After managing to iron out the obvious gotcha’s I am left with one lasting wonder…. load creeping up.

Sitting at 2 for a bit, then 6 then 9 … odd.

Checking over htop, iotop, mytop, jnettop… there is nothing hogging resources. Nothing obviously incurring the load.

Then I saw it – in plain old top – whispering away without so much as a bold or colour hint 49wa. Not deviating all that much – sometimes dropping back to 25 – then back up to 49. THERE is the problem. They question is what on earth is causing it?

wa is short for iowait or in this case – specifically %iowait. Seeing the idle drop back in proportion to it – something is apparently keeping the disks busy despite the fact that iotop does not show this, and disk activity does not show this either. Curious huh?

root@example:~# ps aufx |grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6 0.0 0.0 0 0 ? D May08 0:00 \_ [kworker/u8:0]
root 74 0.0 0.0 0 0 ? D May08 0:00 \_ [kworker/u8:2]

Nothing appears out of the ordinary otherwise.

Three days of peering at the console. Trying to figure this out and no dice.

Hardware is nothing exotic either – the only thing I can put it down to is software RAID1 and possibly kernel support for the controller maybe?

I will continue to watch and wonder, reboot and break stuff in the name of fixing.

Curious huh?

UPDATE

Okay – so no USB installed now. We are alone. We can rule that out.

We are now really clutching at straws.

The issues were noticed after I upgraded from Utopic to Vivid.

However just before this I upgraded from MariaDB 5.5 to version MariaDB 10.

SO – listing Debian packages installed I notice there is a god-awful mess. Rather than the usual upgrade jolly of “Ooo – you are installing this – you will not be needing this any more” it is apparent there were hangovers from version 5.5 still kicking around.

Removing these didn’t go well. Removing everything did. Needless to say this also took any services that were tied to it such as mail, mail, web, logging – fabulous.

Everybody off, re install, and then bring back in the services that were using / reliant upon there being a DB installed.

Reboot.

iowait has gone when viewing in top. Load is pretty much nothing down from six or nine.

Amazing.

Great success.

… just so you know.

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

%d bloggers like this:
Skip to toolbar