NEURAL NETWORKS FOR 3D MOTION DETECTION FROM A SEQUENCE OF IMAGE FRAMES

  In video surveillance, video signals from multiple remote locations are displayed on several TV screens which are typically placed together in a control room. In the so-called third generation surveillance systems 3GSS, all the parts of the surveillance systems will be digital  and consequently, digital video will be transmitted and processed . Additionally in 3GSS some intelligence has to be introduced to detect relevant events in the video signals in an automatic way. This allows filtering of the irrelevant time segments of the video sequences and the displaying on the TV screen only those segments that require the attention of the surveillance operator. Motion detection is a basic operation in the selection of significant segments of the video signals.

Once motion has been detected, other features can be considered to decide whether a video signal has to be presented to the surveillance operator. If the motion detection is performed after the transmission of the video signals from the cameras to the control room then all the bit streams have to be previously decompressed this can be a very demanding operation especially .If there are many cameras in the surveillance system. For this reason, it is interesting to consider the use of motion detection algorithms operating in the compressed transform domain.

          In this thesis we present a motion detection algorithm in the compressed domain with a low computational cost. In the following Section, we assume that video is compressed by using motion JPEG MJPEG,  each frame is individually JPEG compressed.

Motion detection from a moving observer has been a very important technique for computer vision applications. Especially in recent years, for autonomous driving systems and driver supporting systems. Vision-based navigation method has received more and more attention world wide.

           One of its most important tasks is to detect the moving obstacles like cars, bicycles or even pedestrians while the vehicle itself is running in a high speed. Methods of image differencing with the clear background or between adjacent frames are well used for the motion detection. But when the observer is also moving, which leads to the result of continuously changing background scene in the perspective projection image, it becomes more difficult to detect the real moving objects by differencing methods. To deal with this problem, many approaches have been proposed in recent years.

Previous work in this area has been mainly in two categories: 1) Using the difference of optical flow vectors between background and the moving objects, 2) calibrating the background displacement by using camera’s 3D motion analysis result. Methods developed in calculate the optical flow and estimate the flow vector’s reliability between adjacent frames. The major flow vector, which represents the motion of background, can be used to classify and extract the flow vectors of the real moving objects. However, by reason of its huge calculation cost and its difficulty for determining the accurate flow vectors, it is still unavailable for real applications. To analysis the camera’s 3D motion and calibrate the background is another main method for moving objects detection.

For on-board camera’s motion analysis, many motion-detecting algorithms have been proposed which always expend on the previous recognition results like road lane-marks and horizon disappointing. These Methods show some good performance in accuracy and efficiency because of their detailed analysis of road structure and measured vehicle locomotion, which is, however, computationally expensive and over-depended upon road features like lane-marks, and therefore lead to unsatisfied result when lane mark is covered by other vehicles or not exist at all. Compare with these previous works, a new method of moving objects detection from an on-board camera is presented in this paper. To deal with the background-change problem, our method uses camera’s 3D motion analysis results to calibrate the background scene.

 With pure points matching and the introduction of camera’s Focus of Expansion FOE, our method is able to determine camera’s rotation and translation parameters theoretically by using only three pairs of matching points between adjacent frames, which make it faster and more efficient for real-time applications. In section 2, we will interpret the camera’s 3D-motion analysis with the introduction of FOE. Then, the detailed image processing methods for moving objects detection are proposed in section 3 and section 4 which include the corner extraction method and a fast matching algorithm. In section 5 experimental results on real outdoor road image sequence will show the effectiveness and precision of our approach.

1.2     OBJECTIVE

One goal has been to compile an introduction to the motion detection algorithms. There exist a number of studies but complete reference on real time motion detection is not as common .we have collected materials from journals, papers and conferences and proposed an approach that can be best to implement real time motion detection.

Another goal has been to search for algorithms that can be used to implement the most demanding components of an audio encoding and watermarking. A third goal is to evaluate their performance with regard to motion detected. These properties were chosen because they have the greatest impact on the implementation effort.

A final goal has been to design and implement an algorithm. This should be done in high level language or mat lab. The source code should be easy to understand so that it can serve as a reference on the standard for designers that need to implement real time motion detection.

CLICK HERE TO GET FULL MATLAB PROJECT CODE

Main Program Flow Chart

The main task of the software was to read the still images recorded from the camera and then process these images to detect motions and take necessary actions accordingly.  below shows the general flow chart of the main program.

It starts with general initialization of software parameters and objects setup. Then, once the program started the flag value which indicates whether the stop button was pressed or not is checked.

 If the stop button was not pressed it start reading the images then process them using one of the two algorithms as the operator was selected. If a motion is detected it starts a series of actions and then it go back to read the next images, otherwise it goes directly to read the next images. Whenever the stop button is pressed the flag value will be set to zero and the program is stopped, memory is cleared and necessary results are recorded. This terminates the program and returns the control for the operator to collect the results.

FIGURE 4.7.1 Main Program  Flow Diagram

4.7.2 Setup and Initializations

Figure 4.7.2 Setup and Initializations process

Figure 4.7.2 Show the flow chart for the setup and initialization process. This process includes the launch of the graphical user interface (GUI) where the type of motion detection algorithm is selected and threshold value (the amount of sensitivity of the detection) is being initialized. Also, during this stage a setup process for both the serial port and the video object is done.

This process takes approximately 15 seconds to be completed,(depending on the specifications of the PC used) for the serial port it starts by selecting a communication port and reserving the memory addresses for that port, then the PC connect to the device using the communication setting that was mentioned in the previous chapter. The video object is part of the image acquisition process but it should be setup at the start of the program.

4.7.3 Image acquisition

Figure 4.7.3Image acquisitions Process

After setup stage the image acquisition starts as shown in figure 4.7.3 above. This process reads images from the PC camera and save them in a format suitable for the motion detection algorithm. There were three possible options from which one is implemented. The first option was by using auto snapshots software that takes images automatically and save them on a hard disk as JPEG format, and then another program reads these images in the same sequence as they were saved.

 It was found that the maximum speed that can be attained by this software is one frame per second and this limits the speed of detection. Also, synchronization was required between both image processing and the auto snapshot software’s where next images need to be available on the hard disk before processing them .The second option was to display live video on the screen and then start capturing the images from the screen. This is a faster option from the previous approach but again it faced the problem of synchronization, when the computer monitor goes into a power saving mode where black images are produced all the time during the period of the black screen.

The third option was by using the image acquisition toolbox provided in MATLAB 7.5.1 or higher versions. The image acquisition toolbox is a collection of functions that extend the capability of MATLAB. The toolbox supports a wide range of image acquisition operations, including acquiring images through many types of image acquisition devices, such as frame grabbers and USB PC cameras, also viewing a preview of the live video displayed on monitor and reading the image data into the MATLAB workspace directly.

For this project video input function was used to initialize a video object that connects to the PC camera directly. Then preview function was used to display live video on the monitor. get snapshot function was used to read images from the camera and place them in MATLAB workspace.

         

The later approach was implemented because it has many advantages over the others. It achieved the fastest capturing speed at a rate of five frames per seconds depending on algorithm complexity and PC processor speed. Furthermore, the problem of synchronization was solved because both capturing and processing of images were done using the same software.

All read images were converted it into a two dimensional monochrome images. This is because equations in other algorithms in the system were designed with such image format.

4.7.4 Motion Detection Algorithm

A motion detection algorithm was applied on the previously read images. There were two approaches to implement motion detection algorithm. The first one was by using the two dimensional cross correlation while the second one was by using the sum of absolute difference algorithm. These are explained in details in the next two sub sections.

4.7.4.1 Motion Detection Using Cross Correlation

First the two images were sub divided into four equal parts each. This was done to increase the sensitivity of calculation where it is easier to notice the difference between part of image rather than a whole one. A two dimensional cross correlation was calculated between each sub image with its corresponding part in the other image.

 This process produces four values ranging from -1 to 1 depending on the difference of the two correlated images. Because the goal of this division was to achieve more sensitivity the minimum value of correlation will be used as reference to the threshold.  In normal cases, motion can easily be detected when the measured minimum cross correlation value is used to set the threshold. However, detection fails when images contain global variations such as illuminations changes or when camera moves.

Figure above shows a test case that contains consecutive illumination level changes by switching the light on and off. During the time where the lights are on (frames 1-50 and frames 100-145) the correlation value is around 0.998 and when the lights are switched off (frames 51-99 and frames 146-190) the correlation value is around 0.47. If the threshold for detection was fixed around the value of 0.95 it will continuously detect motion during the light off period.

 Figure 4.7.4.1 Variance Values as Reference for Threshold

To overcome this problem continuous re-estimation of threshold value was required. This can be done by using an adaptive filter but it is not easy to design. Another solution is to look at the variance of the set of data produced from the cross correlation process, and detect motion from it. This method solved the problem of changing illumination and camera movements.

                  The variance signal calculated from the same set of images of figure . It can be seen that the need for continuously re-estimate the threshold value is eliminated. Choosing a threshold of 1*10^-2 will detect the times when only the lights are switched on and off. This results into a robust motion detection algorithm with high sensitivity of detection.

4.7.4.2 Motion Detection Using Sum of Absolute Difference (SAD)

This algorithm is based on image differencing techniques.                                  

A large change in the scene being monitored by the camera this was done by moving the camera. During the time before the camera was moved the SAD value was around 1.87 and when the camera was moved the SAD value was around 2.2. If the threshold for detection was fixed around the value less than 2.2 it will continuously detect motion after the camera stop moving.

       To overcome this problem the same solution that was applied to the correlation algorithm will be used. The variance value was computed after collecting two SAD values and the result is shown for the same test case in figure 4.7.4.3below.

Figure 4.7.4.3 Variance Values as Reference for Threshold

This approach solve the need for continuously re-estimate the threshold value. Choosing a threshold of 1*10^-3 will detect the times when only the camera is moved. This results into a robust motion detection algorithm that can not be affected by illumination change and camera movements.

4.7.5 Actions on Motion Detection

          Before explaining series of actions happen when motion is detected it is worth to mention that the values of variance that was calculated .whether it was above or below the threshold will be stored in an array, where it will be used later to produce a plot of frame number Vs the variance value. This plot helps in comparing the variance values against the threshold to be able to choose the optimum threshold value.

Figure 4.7.5 Actions on Motion Detection

whenever the variance value is less than threshold the image will be dropped and only the variance value will be recorded. However when the variance value is greater than threshold sequence of actions is being started .

As the above flow chart show a number of activities happen when motion is detected.

 First the serial port is being triggered by a pulse from the PC; this

pulse is used to activate external circuits connected to the PC. Also a log file is being created and then appended with information about the time and date of motion also the frame number in which motion occur is being recorded in the log file. Another process is to display the image that was detected on the monitor. Finally the image that was detected in motion will be converted to a movie frame and will be added to the film structure.

4.7.6 Break and clear Process

After motion detection algorithm applied on the images the program checks if the stop button on GUI was pressed. If it was pressed the flag value will be changed from one to zero and the program will break and terminate the loop then it will return the control to the GUI. Next both serial port object and video object will be cleared. This process is considered as a cleaning stage where the devises connected to the PC through those objects will be released and the memory space will be freed. 

4.7.7 Data Record

Finally when the program is terminated a data collection process starts where variable and arrays that contain result of data on the memory will be stored on the hard disk.

 This approach was used to separate the real time image processing from results processing. This has the advantage of calling back these data whenever it is required. The variables that are being stored from memory to the hard disk are variance values and the movie structure that contain the entire frames with motion.

         

At this point the control will be returned to the GUI where the operator can callback the results that where archived while the system was turned on.

Next section will explain the design of the GUI highlighting each button results and callbacks.

4.8 GRAPHICAL USER INTERFACE DESIGN

The GUI was designed to facilitate interactive system operation. GUI can be used to setup the program, launch it, stop it and display results. During setup stage the operator is promoted to choose a motion detection algorithm and select degree of the detection sensitivity.

 Whenever the start/stop toggle button is pressed the system will be launched and the selected program will be called to perform the calculations until the start/stop button is pressed again which will terminate the calculation and return control to GUI. Results can be viewed as a log file, movie and plot of frame number vs. variance value.  illustrate a flow chart of the steps performed using the GUI.

Figure 4.8 GUI flow Chart

The complete GUI code is included in the appendix.

Figure 4.9 GUI Layout Design

The GUI as shown in figure 4.9 was designed using MATLAB GUI Builder. It consists of two radio buttons, two sliders, two static text box, four push buttons and a toggle button. Sliders and radio buttons were used to prevent entering a wrong value or selection.The radio buttons are being used to choose either SAD algorithm or 2D cross correlation algorithm.

 Both algorithms can not be selected at the same time.Sliders are used to select how sensitive the detection is, maximum sensitivity is achieved by moving the slider to the left.

The static text show the instantaneous value of the threshold as read from the sliders. When GUI first launch it will select the optimum value for both algorithms as default vale. Moving to the control panel Start / Stop button is used to launch the program when it is first pushed and to terminate the program when it is pushed again. Figure 4.10 show flow chart of commands executed when this button is pressed.

Figure 4.10 Start/Stop button flow chart

Whenever the Start/Stop button is first pushed it will set the value of the flag to 1 then it will check the value of the radio buttons to determine which main program to launch. A log file that contains information about time and date of motions with frame number is being opened when the log file icon is pushed.

The program is being closed and MATLAB is shutdown when the exit button is pushed. Whenever the show movie icon is pushed, MATLAB will load the film structure that was created before from the hard disk then it will convert the structure into a movie and display it on the screen at a rate of 5 frames per seconds. Finally the movie created will be compressed using the Indeo3 compression technique and will be saved on the hard disk under the name [film.avi]. Figure 4.11below show the commands executed when this icon is pushed.

Figure 4.11 Movie Button Callbacks

The last button which is the show plot icon loads the values of the variance to plot it against the frame number.

CHAPTER 7

CONCLUSION

We have presented an algorithm to detect motion in video sequences. It has a very low computational cost. After a period of calibration in which the threshold value is set, the algorithm is able to detect moving blocks in every frame by counting sign changes in the quadrants.

 The number of moving blocks per frame allows determination of the periods of time with motion. It seems to be simulation in stimulant processor provides accuracy rather than real time motion detection using m files. So here we concluded that real time motion detection can be best implemented by our propose approach based on correlation network and sum and absolute difference

CHAPTER-8

FEATURE ENHANCEMENT

         

     This paper describes 3D motion detection system.  Five special designed neural networks are introduced.  They are the correlation network, the rough motion detection network, the edge enhancement network, the background remover and the normalization network.

     This project provides the object image clearly when it is in motion.  The motion detection image is captured within 9-10 seconds.  The same function can be implemented in moving cameras as for an easy surveillance.  And the images captured within 2-4 seconds with the precise algorithm using a multiple input. (More crowded images). Since there were only two objects detected, it took only 8 seconds, for the processing of all three stages.

CLICK HERE TO GET FULL MATLAB PROJECT CODE

Introduction – Teaching in University

In the engineering sphere, courses are usually commenced off with introduction of formulae or equations that describe underlying principles of various theories. These equations are then transformed from printed symbols on the blackboard to actual observed occurrences in real world systems (through experiments conducted in laboratory classes). This is the teaching method that has been practised for centuries and is widely believed to be the best method by lecturers.

However, this current teaching approach can still be improved. In reality, most students find it difficult to relate the theories they learnt to the real world systems they actually see.

Figure 1 – Missing Link between Theories learnt in lectures and the Real World Applications.

This is because there are many limitations to what can be achieved in the laboratory (e.g. safety, cost). Besides, experiments are not always easy to grasp the first time doing it nor it is as straightforward as it may thought to be. An intermediate stage between these two phases is needed to ease and speed up the transition. This intermediate stage will be discussed in the next section but before that let’s see how teaching has evolved over the century and why this intermediate phase is important in this modern world.

The Evolution of Teaching

Everything evolves over time, and so is teaching. Looking back, chalkboard art (Figure on the right) was the popular method used to teach a couple of decades ago. However, there are disadvantages with this method.

• The teaching and learning process is slower.
  Lecturers have to write the notes on the blackboard and students have to copy them down. This ’2 times’ writing process impedes capability for speedier transfer of knowledge.
• Every diagram/graph/chart drawn is a sketch, not accurate, and static.
  Though lecturers may be able to draw diagrams accurately but students often misinterpret the sketches and then get baffled with their notes later on. In addition, diagrams are static and inhibit students’ capability to learn faster through visualisations.

• Explaining effects of a parameter change (in an equation) could be awful.
  Frequently, lecturers want to explain the significance of parameters in an equation to illustrate the effects they have on the real system. To achieve this, lecturers have to erase and redraw the diagram all over again, which is time wasting and painful.

As time goes by, technology advances and then comes the projector art. With the overhead projector, lecturers improved their teaching considerably. Below is the list of advantages (and disadvantages) of projector art.

• Cleaner
  Chalks are replaced with ink pens and transparencies. No more dry fingers, dusters and asthma.
• Eliminates inaccuracies
  Projector technology enables lecturers to display photocopied transparencies during lectures. This enhances visual learning through display of accurate drawings/graphs/diagrams and students will be less likely to copy down incorrect diagrams.
• Speedier learning
  
  
  With photocopied transparencies, lecturers now reduce time wastage by eliminating the need to write lots of notes or draw diagrams on the blackboard. The transparencies can also be photocopied and be distributed to the students as notes.
• (Disadvantage) Diagrams are still static
  
  
  Though accurate, the diagrams/graphs/charts are still static (i.e. no animation or simulation). This is like comparing between staring at an image and watching a movie. We tend to experience and learn more in a movie (or a series of moving images).
  

Next in the evolution timeline is through the usage of simulations. Numerous researches have been conducted and visual learning has been found to be the easiest and quickest way to learn new information. Simulations are a very good method to enhance visual learning.

Simulation in Modern Teaching

Simulation plays a critical role in modern teaching. In this present era, technology advances at an unimaginable rate (compared to that half a century ago) and is constantly keeping us on our toes (in anticipation of ‘what’s new’ down the road). The amount of information is rising, the number of professionals is increasing and new researches are published every day. However, the number of years of education in a typical University degree remains the same (4 years). This means there is higher density of information that modern students need to absorb. This highlights an important trend that educators need to take note in order to keep up with intellect’s ever-expanding sphere. Let’s see an example of a conveyor mechanism.

Conveyor Mechanism (Figure on the right)

With chalkboard art, lecturers are limited to drawing sketches and writing equations on the blackboard. Describing kinematics or the actual operation of this mechanism takes a great deal of effort.

Projectors enhance the teaching process with the ability to display an accurate representation of the mechanism immediately and how various parts are connected together. (For e.g., just by photocopying the above model onto a transparency)

However, simulation enhances the teaching process considerably. Let’s take a look at the figures below.

The figures above illustrate a virtual conveyor mechanism built in the Simulink environment. Here are some of the advantages of this model.

• With Simulink’s diverse features and capabilities, there is so much that can be achieved. A simple example shown here are the responses of the system (torque, linear and angular position) which are plotted as the simulation runs.
• Students are able to directly view how the conveyor mechanism operates when (for e.g.) a certain amount of torque is applied to the lower link. These allow them to visualize the system and relate it directly to the equations they learnt in lectures.

• Virtual model is safe, cheap and its parameters can be changed over and over again without any worries of unfortunate incidents. Students then can test and see for themselves (and learn) how the theories they learnt in lectures affect the behaviour of a system. Upon studying the system, then they can move on to the laboratory classes and relate what they discover in the simulation environment to the actual occurrences of real world systems.

• (Though no simulations here) Simulations catch attention and build interests to those who see it. This increases students’ passion for the subject they studied and hence be able to always relate their studies to what they see.
• Simulink allows lecturers to design models up to high level of complexity. Therefore, there’s almost no end to what can be designed in the Simulink environment and be shared during lectures. Simulink also enables students to link up with systems from other domains (e.g. electrical to mechanical to thermal) to test out the integration feasibility between systems.

Conclusion

In conclusion, as the phrase goes “Picture speaks a thousand words”, so is simulation in the teaching environment. With simulation, students are able to learn much more (through visual learning), as well as to understand the theories taught in lectures at a quicker rate. They’ll also be able to relate what was taught to the real engineering systems and hence increase their passion on the subject they’re learning.

End of the Article:

For more product information on Simulink, please refer to http://www.mathworks.com/products/simulink/?s_cid=global_nav

Article by James Ang, Application Engineer