Converting a fisheye image into a panoramic, spherical or perspective projection

Written by Paul Bourke
November 2004

The source code implementing the projections
below is only availableon request for a small fee. It includes a demo
application and an invitation toconvert an image of your choice to
verify the code does what you seek. For more information
please contact the author.

The following documents various transformations from
fisheye into other projectiontypes, specifically standard perspective as
per a pinhole camera, panorama andspherical projections.Fisheye images
capture a wide field of view, traditionally
one thinks of 180degrees but the mathematical definition extends past
that and indeed there aremany physical fisheye lenses that extend past
180 degrees. The two main applicationsfor the following are: the
inspection of images/video from security cameras wherepanorama
or perspective views may be more natural to view, creating panorama
orspherical images that are blended together to form even wider field of
view images.

The general options for the software include the
dimensions of the output image as wellas the aperture of the output
panoramic or perspective frustum. Some otherrequirements arise from
imperfect fisheye capture such as the fisheye notbeing
centered on the input image, the fisheye not be aligned with
theintended axis, and the fisheye being of any angle.Another
characteristic of real fisheye images is their lack of linearitywith
radius on the image, while this is not addressed here as it requiresa
lens calibration, it is a straightforward correction to make.

The usual approach for such image transformations is
to perform the inverse mapping. Thatis, one needs to consider each pixel
in the output image and map backwards tofind the closest pixel in the
input image (fisheye). In this way every pixelin
the output image is found (compared to a forward mapping), it also
meansthat the performance is governed by the resolution of the output
image(and supersampling) irrespective of the size of the input image.A
key aspect of these mappings is also to perform
some sort of antialiasing, thesolutions here use a simple supersampling
approach.

The code here are all plain vanilla C tested on Unix
style gcc systems (specifically Mac and Linux),but the algorithms/code
can readily be modified for otheroperating systems and programming
languages.This is not meant to be a final application
but rather something you integrate into your code base. Having said
that it is wrapped up in a simple TGA image reader/writer for the
purposes of algorithm testing, the intent is that one would be
implementing the function into ones own code base. They all
operate on a RGB buffer (fisheye image) in memory.For each test utility
the usage message is provided. The source images for the
examplesprovided are provided along with the command line that generated
them.

Fisheye to perspective transformation
Software: fish2persp

Usage: fish2persp [options] fisheyeimage

Options

   -w n     perspective image width, default = 800

   -h n     perspective image height, default = 600

   -t n     aperture of perspective (degrees), default = 100

            maximum is 170 degrees

   -s n     aperture of fisheye (degrees), default = 180

   -c x y   offset of the center of the fisheye image,

            default is fisheye image center

   -r n     fisheye radius, default is half height of fisheye image

   -x n     tilt angle (degrees), default: 0

   -y n     roll angle (degrees), default: 0

   -z n     pan angle (degrees), default: 0

   -a n     antialiasing level, default = 1 (no antialising)

            sensible maximum 3

It should be noted at the outset that a fisheye projection is not a "distorted" image, and the process isn’t a “dewarping”. A fisheye like other projections is one of many ways of mapping a 3D world onto a 2D plane, it is no more or less "distorted" than other projections including a rectangular perspective projection ... it is what it is.

Example source fisheye image.

A critical consideration is antialiasing, required when sampling any discrete signal. The approach here is a simple supersampling antialiasing, that is,each pixel in the output image is subdivided into a 2x2, 3x3....grid andthe inverse mapping applied to the subsamples. The final value for the outputpixel is the weighted average of the inverse mapped subsamples.There is a sense in which the image plane is considered to be a continuousfunction. Since the number of samples that are inverse mapped is the principle determinant ofperformance, high levels of antialiasing can be very expensive, typically2x2 or 3x3 are sufficient especially for images captured from video in whichneighbouring pixels are not independent in the first place.For example a 3x3 antialiasing is 9 times slower than no antialiasing.In general the jagged edges are more noticeable in featureswith a sharp colour/intensity boundary.

Default perspective view looking forwards, 100 degrees horizontal field of view.

fish2persp -w 800 -a 3

The vertical aperture is automatically adjusted to
match the width and height.Controls are provided for any angle fisheye
as well as fisheyes that are notlevel or tilted, noting that the exact
order of the correction rotations mayneed to
be considered for particular cases.Note that a perspective projection
is not defined for greater than 180 degrees,indeed it gets increasingly
inefficient past around 140 degrees.The field of view can be adjusted as
well as the viewing direction.The following
example is a 120 degrees horizontal field of view and looking upwards
by 30 degrees.

fish2persp -w 800 -a 3 -x 30 -t 120

If "straight" lines are not straight that normally
means the fisheye center or radius are not specified correctly or the
angle is not defined correctly.Curvature in what should be straight
lines near the rim of the fisheye normally meansthe
fisheye lens has non-linearities near the rim (a deviation from the
mathematically purefisheye projection) and corrections need to be
applied.The following is looking right by 40 degrees and a narrower
field of view of 80 degrees.

fish2persp -w 800 -a 3 -z 40 -t 80

The center of the fisheye on the input image can be
found by projectinglines along vertical structure in the scene. Where
these lines intersectis a close approximation to the center of the
fisheye, assuming thecamera is mounted vertically.
Alternatively, and perhaps easier, is to identify the edges of
thefisheye and assume a perfect circular inscribed circle. Note that for
the example utilities providedhere the origin is assumed to be the
bottom left corner, unlike the more common top right
thatimage editing programs use.

To test the algorithm a fisheye rendering inside a
gridded cube is a good example,see image on left below. Any correct
perspective projection should result in straight lines.

Sample input image

Front pointing fisheye to panorama
Software: frontfish2pano

This case is developed mainly for "front
pointing" fisheyes although it does have applicationfor other
orientations. The projection is more correctly called a cylindrical
panorama.

Usage: frontfish2pano [options] fisheyeimage

Options

   -w n     panoramic image width, default = 800

   -h n     panoramic image height, default = -1

  -ap n     vertical aperture of panoramic, default = 100

  -af n     aperture of fisheye (degrees), default = 180

  -cf x y   center of the fisheye image, default is image center

   -r n     radius of the fisheye image, default is half the image width

  -fa n     angle for tilted fisheye, default = 0

  -fb n     angle for rotated fisheye, default = 0

   -a n     antialiasing level, default = 1 (no antialising)

Source fisheye image.

Transformation with the default settings is shown below.

frontfish2pano -a 3 -w 800

Correct for the fact that the camera is not quite
horizontal, this is thereason the vertical structure doesn't appear
vertical in the panoramic projection.Of course nothing is for free, one
looses a bit of the image in the bottomleft and
right corners.

frontfish2pano -a 3 -w 800 -fa -20

Set the vertical field of view of the panorama, in
the following cases narrowed fromthe default of 100 degrees to 80
degrees. As with perspective projections there is a limit, in this case,
to the vertical field of view, a hard limit at 180
degrees but increasingly inefficientpast 140 degrees.

frontfish2pano -a 3 -w 800 -fa -20 -ap 80

Fisheye to (partial) spherical projection
Software: fish2sphere

Usage: fish2sphere [options] imagefile

Options

   -w n     sets the output image size, default: 4 fisheye image width

   -a n     sets antialiasing level, default: 2

  -fa n     fisheye aperture (degrees), default: 180

   -c x y   fisheye center, default: center of image

   -r n     fisheye radius, default: half the fisheye image width

   -v n     rotate fisheye in latitude, default: 0

   -z n     roll fisheye, default: 0

   -d       debug mode

Source fisheye image.

Transformation using the default settings. Since a 180 degree (in this case) fisheyecaptures half the visible universe from a single position, so it makes sense that it occupieshalf of a spherical (equirectangular) projection, which captures the entire visibleuniverse from a single position.

In this case the camera is not perfectly horizontal, this and other adjustmentscan be made. In the example here the lens was pointing downwards slightly, thecorrection results in more of the south pole being visible and less of the northpole.

Note that the fisheye angle is not limited to 180 degrees, indeed one applicationfor this is in the pipeline to create 360 spherical panoramas from 2 cameras, eachwith a fisheye lens with a field of view greater than 180 to provide a blend zone.

This can be readily implemented in the OpenGL Shader Language, the followingexample was created in theQuartz Composer Core Image Filter.

The transformation can be performed in realtime using warp mesh files forsoftware such aswarpplayer or the VLC equivalentVLCwarp.
A sample mesh file is givenhere:fish2sph.data. Showing the result in actionis below.

Test cases for various fisheye apertures.
180 degree fisheye
120 degree fisheye
220 degree fisheye
220 degree fisheye and 90 degree latitude rotation
220 degree fisheye and 60 degree latitude rotation

Fisheye to (full) panorama
Software: fish2pano

The following is a slightly more general version of
conversion to a panoramic projection.It supports both spherical and
cylindrical projections, handles different fisheyeorientations but the
main difference is it is designed to image into
a full 360 projection obviously withparts not covered because of the
limited field of view of a fisheye compared to a fullspherical
projection.

Usage: fish2pano [options] fisheyeimage

Options

   -w n     panoramic image width, default = 1024

   -h n     panoramic image height, default = derived

  -af n     aperture of fisheye (degrees), default = 180

  -cf x y   center of the fisheye image, default is image center

   -x n     tilt fisheye about x (right) axis, default = 0

   -z n     rotate fisheye about lens axis, default = 0

   -r n     radius of the fisheye image, default is half the image width

   -a n     antialiasing level, default = 1 (no antialising)

   -c       cylindrical pano, default is spherical

   -v n     vertical aperture for cylindrical pano, default: 60

Example 1: Source fisheye image.

Black refers to the corners of the fisheye image and the grey to the unavailable data,that is, data outside the fisheye rectangle in which the circular fisheye is inscribed.Note that for a wider than 180 degree fisheye the black and grey regions will be correspondingly smaller. The "x" axis is to the right, the 90 degree rotation heregives the desired result for a forward pointing fisheye. A fisheye pointing straightup or straight down would more normally be transformed with "-x 0".

fish2pano -a 3 -w 800 -x 90

Cylindrical projection, vertical field 60 degrees and
100 degrees respectively.The vertical extent of the image is, normally,
determined correctly given the vertical field of view requested but
that can be overruled if desired.

fish2pano -a 3 -w 800 -x 90 -c
fish2pano -a 3 -w 800 -x 90 -c -v 100

Example 2: Source fisheye image

The default settings (no x axis rotation) provide
what one expects for an upper hemisphereas per a standard Earth map.
Note the apparent distortion towards the north pole, which againis not a
strictly distortion but a natural consequence
of the mathematics behind the projection.

fish2pano -a 3 -w 800

Cylindrical panorama with a 100 degree vertical field
of view, that is, from the equator (0 degreeslatitude) to 100 degrees
latitude.

fish2pano -a 3 -w 800 -c -v 100