Function File: fesetprec (mode)

Change the precision of the floating point arithmetics. At present, fesetprec may work only on systems with x86 or x86_64 processors.

Possible values of mode are

‘1’

‘24’

Sets the x87 FPU precision mode to single precision (24-bit significand)

‘2’

‘53’

Sets the x87 FPU precision mode to double precision (53-bit significand)

‘3’

‘64’

Sets the x87 FPU precision mode to double extended precision (64-bit significand)

When successful, fesetprec returns 0. It is always recommended to verify the result experimentally by calling the function fetestenv, which tests the rounding mode and the precision of the floating point arithmetics.

Modern processors, including x86 and x86_64 architectures, support changing certain properties of their floating point arithmetics. Here, we focus exclusively on x86 and x86_64. These processors presently feature two types of floating point processing units: the older x87 FPU and the newer SSE unit. Only x87 FPU allows changing the precision, so fesetprec applies only to those instructions which are executed by x87 FPU. fesetprec changes the precision of the four basic arithmetic operations and of the square root on x87 FPU.

Since floating point operations on x86 or x86_64 may be executed either by x87 FPU or SSE units, fesetprec may affect all, some, or no Octave's operations. The rule of the thumb, however, is that 64-bit systems usually do their floating point on SSE by default, when 32-bit systems use x87 FPU. You can always check how your system reacts to changes made by fesetprec by verifying them with fetestenv.

It is important to know that all expressions and their intermediate results processed by Octave's interpreter are cast to temporary double variables and thus it is impossible to execute an interpreted expression in double extended precision. On the other hand, those libraries called by Octave which rely on x87 FPU will usually perform the computations using x87 FPU 80-bit registers, i.e. in double extended precision. This may lead to some subtle differences in how the precision mode affects the computations. Let's take a look at the simplest example:

          macheps = builtin('eps');
          fesetprec(3);
          x = [1, macheps/2, -1]*[1;1;1];
          y = 1 + macheps/2 - 1;

The built-in function eps returns the machine epsilon in double precision. Mathematically the expressions for x and y are equivalent, but in reality they may be not. The expression for x will call a BLAS subroutine and on a 32-bit system it will most likely be executed on x87 FPU using 64-bit double extended precision - so x will be greater than zero. On the other hand, the expression for y will be executed through the interpreter and these calculations will store all intermediate results in double precision, so that y will be equal to zero. In contrast to 32-bit systems, many 64-bit systems will do all floating point on SSE - and thus both x and y will then be equal to zero.

It is also known that some numerical libraries set their own precision modes or execute floating point operations on different units (say, they may use only the SSE unit when Octave uses exclusively the x87 FPU...). Therefore calls to such numerical libraries may also cancel changes made previously by fesetprec:

          fesetprec(1); fesetround(0); fetestenv
          -| Machine epsilon: 1.19209e-07
          -| Rounding mode:   0
          hilb(2)\ones(2,1); fetestenv
          -| Machine epsilon: 2.22045e-16
          -| Rounding mode:   0.5

(the above test has been run on an x86 32-bit system using Octave 3.0.2 package provided along with the Fedora 9 Linux distribution).

Changes issued by fesetprec are reflected by Octave's function eps: calling fesetprec replaces the built-in function eps with a dispatch fetestenv. In order to perform a more detailed check of actual parameters of the floating point arithmetics, use fetestenv directly.

A possible application of fesetprec, following the idea by W. Kahan, is an experimental (in)stability analysis. Running some code with various settings of the floating point arithmetics may reveal some instability of the numerical algorithm being used in the code or reveal possible ill conditioning of the very problem being solved.

Literature

1. W. Kahan, "How Futile are Mindless Assessments of Roundoff in Floating-Point Computation?", available on the Web: <http://www.cs.berkeley.edu/~wkahan/Mindless.pdf>.
2. W. Kahan, "Why I can Debug some Numerical Programs You can't", available on the Web: <http://www.cs.berkeley.edu/~wkahan/Stnfrd50.pdf>.
3. Intel 64 and IA-32 Architectures Software Developer's Manual, Volume 1: Basic Architecture, May 2007.

See also: fetestenv fesetround system_dependent eps fe_system_dependent