Alignment issues

Introduction

The X-Scale CPU used on the NSLU2 expects all memory accesses to be aligned (that is, the memory address modulo the size of the accessed location in bytes must be zero). If this is not the case, the memory actually accessed can be different from the specified address, with whatever consequences it may have for the application (unexpected behaviour and crashes is likely).

[jbowler: this applies to accesses to 2 and 4 byte values, larger values such as 8 byte double precision floating point numbers, need only be aligned to a 4 byte boundary.]

The CPU can detect such accesses, however, and generate a trap whenever they occur. Installing a proper software trap handler can resolve the access on-the-fly, but the time it takes to correct the improper access has a disasterous impact on performance.

User level control

The Unslung Linux kernel has such a trap handler installed and its behaviour may be controlled through, and statistical information about its work obtained from, the file /proc/cpu/aligment. A brief look at this file uncovers a days work for this trap handler on my slug:

 bash-2.05b# cat /proc/cpu/alignment
 User:           0
 System:         44
 Skipped:        0
 Half:           44
 Word:           0
 Multi:          0
 User faults:    0 (ignored)

Here we see the default action for misaligned accesses in user space programs ("ignored" means none) and a total of 44 half word accesses have been corrected.

As above, this handler is by default set up to fix unaligned memory accesses in the kernel only. User programs will suffer the default undefined behaviour in connection with unalign memory access. This is not what we want. Neither do we want to make the trap handler fix all user accesses silently as it prevents buggy programs from being properly fixed (hide the symptom, remove the incentive to cure). What we do want (and you want to, even if you don't know it yet) is to record all such incorrect accesses and try to identify and fix the source of such traps. The possible bit settings for the trap handler are:

 1 - Generate warnings when user programs perform misaligned accesses
 2 - Enable fixup of misaligned access from user programs
 4 - Generate signal when user programs perform misaligned accesses

Setting the handler to, e.g. fix and log warning, is done by:

 echo 3 > /proc/cpu/alignment
 echo "Warning about and fixing misaligned acesses from user programs"

This value is reset upon reboot and can be set in a diversion script.

Development level impact

When developing, you not only want to get the depressing news that the program you're working on performs misaligned accesses, but you want to find out where in the program the offending access occur. The way to do this is to set the trap handler to generate a signal when the access occurs and run the program under GDB.

This may also be done post mortem. Run unstripped executables, enable signal generation on misalignment traps only and enabling core dumps (with ulimit -c unlimited) will let take the core dump into GDB and find the spot where the program crashed

When it comes to the parts of program we all share - the libraries, the standard C shared library is known to be good in this respect, while the statically linked version has flaws. So: only use dynamic libraries with your programs - both for memory reasons and for the sake of not introducing bugs through misaligned access.

[jbowler: so far as I am aware uClibc (as opposed to glibc) builds are alignment clean for both the static and dynamic versions of the library on OpenSlug.]

Thanks for all feedback on this issue on the list!

Bob(tm)