org.openpnp.util

Class CameraWalker

java.lang.Object

org.openpnp.util.CameraWalker

public class CameraWalker
extends java.lang.Object

A CameraWalker iteratively moves the machine to drive some detectable feature in a camera image to a specific point in the image. This works for both top and bottom cameras and does not require the camera to be completely setup or calibrated, i.e., settings for rotation, flips, units per pixel, and lens distortion are all irrelevant to its operation. However, it does need a function to detect the feature in the image.

The feature detection function will typically process a CvPipeline to detect the feature. It is important that the stages used in the pipeline have no dependencies on any of the aforementioned uncalibrated settings. To aid in the detection, the function is passed a point in image coordinates where the feature is expected to be found and can be used to select the search area and/or mask-off regions of the image that may cause false detections. The function should return either the image coordinates of the feature if it is found or null if the feature could not be found.

The size of each step a CameraWalker takes is normally determined by the distance between the detected feature image coordinates and its desired image coordinates. That distance is scaled by the loopGain (defaults to 0.7 but can be changed using the setLoopGain method) to determine the step size. The maximum size of the steps a CameraWalker takes can be further limited by using the setMaxAllowedPixelStep and/or setMaxAllowedMachineStep methods.

The walk is considered done when the distance between the detected feature image coordinates and its desired image coordinates no longer decreases. The walk can also be completed if the detected feature is within a set distance (in pixels) of the desired image coordinates or if the machine step is below a set distance (in machine units). These can be set with the setMinAllowedPixelStep (defaults to 0) and setMinAllowedMachineStep (defaults to 0.0001 mm) methods respectively.

By default, a CameraWalker will only make safe Z moves; however, that behavior can be changed by using the setOnlySafeZMovesAllowed method.

Constructor Summary

Constructors

Constructor and Description
CameraWalker(HeadMountable movable, java.util.function.Function<Point,Point> findFeatureInImage) Constructs a new CameraWalker with the given parameters

Method Summary

Modifier and Type	Method and Description
void	cancelWalk()
RealMatrix	estimateScaling(Length testStepLength, Point expectedPoint) Estimates the machine units to pixel scaling by stepping the machine and observing how far the feature moves in the image.
Location	getDeltaLocation(Point deltaPoint) Gets an estimate of a change in machine location for a given change in pixels
double	getEstimatedMillimetersPerPixel() Gets the estimated millimeters per pixel
java.util.function.Function<Point,Point>	getFindFeatureInImage() Gets the feature detection function
java.util.List<double[]>	getImage2DCoordinates() Gets the list to which the CameraWalker adds image coordinates as the walk takes place.
Location	getLastFoundLocation() Gets the machine coordinates at the end of the last walk
Point	getLastFoundPoint() Gets the image coordinates of the detected feature at the end of the last walk
java.lang.Double	getLoopGain() Gets the loop gain to be used during the walk
java.util.List<double[]>	getMachine3DCoordinates() Gets the list to which the CameraWalker adds machine coordinates as the walk takes place.
Length	getMaxAllowedMachineStep() Gets the maximum allowable step in machine units that the walk can take
double	getMaxAllowedPixelStep() Gets the maximum allowable step in pixels that the walk can take
Length	getMinAllowedMachineStep() Gets the minimum allowable step in machine units that the walk can take.
double	getMinAllowedPixelStep() Gets the minimum allowable step in pixels that the walk can take.
double	getMirror()
HeadMountable	getMovable() Gets the Camera or Nozzle that will do the moving during the walk
RealMatrix	getScalingMat() Gets the scaling matrix
boolean	isOnlySafeZMovesAllowed() Checks to see if only safe Z moves are allowed
boolean	isReadyToWalk() Returns true if the CameraWalker is ready to walk.
boolean	isSingleStep()
boolean	isWalkComplete()
void	setFindFeatureInImage(java.util.function.Function<Point,Point> findFeatureInImage) Sets the feature detection function
void	setLoopGain(java.lang.Double loopGain) Sets the loop gain to be used during the walk, must be greater than 0 and less than or equal to 1.7 (use caution as very small loop gains may cause walks to take a very long time to complete and too large may cause unbounded machine oscillations).
void	setMaxAllowedMachineStep(Length maxAllowedMachineStep) Sets the maximum allowable step in machine units that the walk can take.
void	setMaxAllowedPixelStep(double maxAllowedPixelStep) Sets the maximum allowable step in pixels that the walk can take.
void	setMinAllowedMachineStep(Length minAllowedMachineStep) Sets the minimum allowable step in machine units that the walk can take.
void	setMinAllowedPixelStep(double minAllowedPixelStep) Sets the minimum allowable step in pixels that the walk can take.
void	setMirror(double mirror)
void	setMovable(HeadMountable movable) Sets the Camera or Nozzle that will do the moving during the walk
void	setOnlySafeZMovesAllowed(boolean onlySafeZMovesAllowed) Sets the only safe Z moves are allowed flag to the given state
void	setSaveCoordinates(java.util.List<double[]> machine3dCoordinates, java.util.List<double[]> image2dCoordinates) The CameraWalker will add (save) the coordinates visited during the walk to the lists provided here.
void	setScalingMat(RealMatrix scalingMat)
void	setSingleStep(boolean singleStep)
boolean	step()
Location	walkToPoint(Location startingMachineLocation, Point desiredImagePoint) Iteratively moves the machine in X and Y until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).
Location	walkToPoint(Location startingMachineLocation, Point startingImagePoint, Point desiredCameraPoint) Iteratively moves the machine in X and Y until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).
Location	walkToPoint(Point desiredImagePoint) Iteratively moves the machine in X and Y from its current location until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).
Location	walkToPoint(Point startingImagePoint, Point desiredImagePoint) Iteratively moves the machine in X and Y from its current location until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CameraWalker

public CameraWalker(HeadMountable movable,
                    java.util.function.Function<Point,Point> findFeatureInImage)

Constructs a new CameraWalker with the given parameters

Parameters:

movable - - the Camera or Nozzle that will be moved during the walk

signedUnitsPerPixelX - - the signed units per pixel in the X direction

signedUnitsPerPixelY - - the signed units per pixel in the Y direction

findFeatureInImage - - the feature detection function

Method Detail

estimateScaling

public RealMatrix estimateScaling(Length testStepLength,
                                  Point expectedPoint)
                           throws java.lang.Exception

Estimates the machine units to pixel scaling by stepping the machine and observing how far the feature moves in the image. The machine needs to be jogged such that the feature is visible in the image and close to the expectedPoint prior to calling this method. This method must be called prior to beginning a walk.

Parameters:

testStepLength - - the distance the machine will step from its current location. This should be kept relatively small.

expectedPoint - - the expected point where the feature is expected to be detected in the image

Returns:

the 2x2 scaling matrix that converts changes in image pixel locations to changes in machine coordinates

Throws:

java.lang.Exception - if the scaling can't be estimated

isReadyToWalk

public boolean isReadyToWalk()

Returns true if the CameraWalker is ready to walk.

walkToPoint

public Location walkToPoint(Location startingMachineLocation,
                            Point desiredImagePoint)
                     throws java.lang.Exception

Iteratively moves the machine in X and Y until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).

Parameters:

startingMachineLocation - - the machine is moved to this location before beginning the walk

desiredImagePoint - - the desired image coordinates of the feature when the walk is completed and is also the expected coordinates of the feature when the machine is at the starting location

Returns:

the location of the machine at the completion of the walk or null if the walk failed due to the feature not being found or the task being interrupted. The image coordinates of the feature corresponding to this location can be found with getLastFoundPoint

Throws:

java.lang.Exception

walkToPoint

public Location walkToPoint(Point desiredImagePoint)
                     throws java.lang.Exception

Iteratively moves the machine in X and Y from its current location until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).

Parameters:

desiredImagePoint - - the desired image coordinates of the feature when the walk is completed and is also the expected coordinates of the feature with the machine at its current location

Returns:

the location of the machine at the completion of the walk or null if the walk failed due to the feature not being found or the task being interrupted. The image coordinates of the feature corresponding to this location can be found with getLastFoundPoint

Throws:

java.lang.Exception

walkToPoint

public Location walkToPoint(Point startingImagePoint,
                            Point desiredImagePoint)
                     throws java.lang.Exception

Iteratively moves the machine in X and Y from its current location until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).

Parameters:

startingImagePoint - - the expected image coordinates of the feature with the machine at its current location

desiredImagePoint - - the desired image coordinates of the feature when the walk is completed

Returns:

the location of the machine at the completion of the walk or null if the walk failed due to the feature not being found or the task being interrupted. The image coordinates of the feature corresponding to this location can be found with getLastFoundPoint

Throws:

java.lang.Exception

walkToPoint

public Location walkToPoint(Location startingMachineLocation,
                            Point startingImagePoint,
                            Point desiredCameraPoint)
                     throws java.lang.Exception

Iteratively moves the machine in X and Y until the detected feature appears at the desired image coordinates (or until it is close enough depending on the settings of minAllowedPixelStep and minAllowedMachineStep).

Parameters:

startingMachineLocation - - the machine is moved to this location before beginning the walk

startingImagePoint - - the expected image coordinates of the feature when the machine is at the starting location

desiredCameraPoint - - the desired image coordinates of the feature when the walk is completed

Returns:

the location of the machine at the completion of the walk or null if the walk failed due to the feature not being found or the task being interrupted. The image coordinates of the feature corresponding to this location can be found with getLastFoundPoint

Throws:

java.lang.Exception - if the walk can't be started and/or completed

step

public boolean step()
             throws java.lang.Exception

Throws:

java.lang.Exception

cancelWalk

public void cancelWalk()

isWalkComplete

public boolean isWalkComplete()

getDeltaLocation

public Location getDeltaLocation(Point deltaPoint)

Gets an estimate of a change in machine location for a given change in pixels

Parameters:

deltaPoint - - the change in pixels

Returns:

the change in machine location

getMovable

public HeadMountable getMovable()

Gets the Camera or Nozzle that will do the moving during the walk

setMovable

public void setMovable(HeadMountable movable)

Sets the Camera or Nozzle that will do the moving during the walk

getScalingMat

public RealMatrix getScalingMat()

Gets the scaling matrix

setScalingMat

public void setScalingMat(RealMatrix scalingMat)

Parameters:

scalingMat - the scalingMat to set

getMirror

public double getMirror()

Returns:

the mirror

setMirror

public void setMirror(double mirror)

Parameters:

mirror - the mirror to set

getEstimatedMillimetersPerPixel

public double getEstimatedMillimetersPerPixel()

Gets the estimated millimeters per pixel

getFindFeatureInImage

public java.util.function.Function<Point,Point> getFindFeatureInImage()

Gets the feature detection function

setFindFeatureInImage

public void setFindFeatureInImage(java.util.function.Function<Point,Point> findFeatureInImage)

Sets the feature detection function

getLoopGain

public java.lang.Double getLoopGain()

Gets the loop gain to be used during the walk

setLoopGain

public void setLoopGain(java.lang.Double loopGain)

Sets the loop gain to be used during the walk, must be greater than 0 and less than or equal to 1.7 (use caution as very small loop gains may cause walks to take a very long time to complete and too large may cause unbounded machine oscillations).

getMaxAllowedPixelStep

public double getMaxAllowedPixelStep()

Gets the maximum allowable step in pixels that the walk can take

setMaxAllowedPixelStep

public void setMaxAllowedPixelStep(double maxAllowedPixelStep)

Sets the maximum allowable step in pixels that the walk can take. Must be greater than 0 (use caution as small values may result in walks that take a very long time to complete).

getMinAllowedPixelStep

public double getMinAllowedPixelStep()

Gets the minimum allowable step in pixels that the walk can take.

setMinAllowedPixelStep

public void setMinAllowedPixelStep(double minAllowedPixelStep)

Sets the minimum allowable step in pixels that the walk can take. Must be non-negative.

getMaxAllowedMachineStep

public Length getMaxAllowedMachineStep()

Gets the maximum allowable step in machine units that the walk can take

setMaxAllowedMachineStep

public void setMaxAllowedMachineStep(Length maxAllowedMachineStep)

Sets the maximum allowable step in machine units that the walk can take. Must be greater than 0 (use caution as small values may result in walks that take a very long time to complete).

getMinAllowedMachineStep

public Length getMinAllowedMachineStep()

Gets the minimum allowable step in machine units that the walk can take.

setMinAllowedMachineStep

public void setMinAllowedMachineStep(Length minAllowedMachineStep)

Sets the minimum allowable step in machine units that the walk can take. Must be greater than 0 (use caution as very small values can result in a long stream of very tiny moves being generated which can result in controller buffer overflows).

isOnlySafeZMovesAllowed

public boolean isOnlySafeZMovesAllowed()

Checks to see if only safe Z moves are allowed

Returns:

true if only safe Z moves are allowed, false otherwise

setOnlySafeZMovesAllowed

public void setOnlySafeZMovesAllowed(boolean onlySafeZMovesAllowed)

Sets the only safe Z moves are allowed flag to the given state

Parameters:

onlySafeZMovesAllowed - - the state to set

getLastFoundPoint

public Point getLastFoundPoint()

Gets the image coordinates of the detected feature at the end of the last walk

getLastFoundLocation

public Location getLastFoundLocation()

Gets the machine coordinates at the end of the last walk

getMachine3DCoordinates

public java.util.List<double[]> getMachine3DCoordinates()

Gets the list to which the CameraWalker adds machine coordinates as the walk takes place. Returns null if setSaveCoordinates was not used to set a list to save the coordinates.

getImage2DCoordinates

public java.util.List<double[]> getImage2DCoordinates()

Gets the list to which the CameraWalker adds image coordinates as the walk takes place. Returns null if setSaveCoordinates was not used to set a list to save the coordinates.

setSaveCoordinates

public void setSaveCoordinates(java.util.List<double[]> machine3dCoordinates,
                               java.util.List<double[]> image2dCoordinates)

The CameraWalker will add (save) the coordinates visited during the walk to the lists provided here. Note that both parameters must be non-null in order for the CameraWalker to save the coordinates.

Parameters:

machine3dCoordinates - - the list to which the machine coordinates are added

image2dCoordinates - - the list to which the image coordinates are added

isSingleStep

public boolean isSingleStep()

Returns:

the singleStep

setSingleStep

public void setSingleStep(boolean singleStep)

Parameters:

singleStep - the singleStep to set