DPRG mailing list archives: [DPRG] computer vision frame rate

Fri Apr 27 19:48:59 CDT 2007

Hi Ed,

The 290 milliseconds is for:

1. libjpeg decodes the 640x480 image
2. integral image transform
3. box feature convolution using transformed image

So there's more going on in addition to decoding the JPEG image.

Ideally, the camera would have uncompressed planar output at low
resolution. But I am using what I could easily find COTS. That
means a USB webcam with JPEG output ($30 from Fry's). I wanted
to go cheap to see what happens.

Pushing more into hardware is inevitable. I'm just not there at
this point. I figure that I need to get what I have working
before attempting a more advanced design. I am learning what
EmbedCV really needs to do in order to be useful.

I have been considering the VIA nano-ITX mainboards (pico-ITX
would be better but who knows when they will become available?).
A USB webcam with planar output using direct PCI DMA transfer
into the video framebuffer, and then GPU acceleration... This
is a completely different project, though. All of the algorithms
would change with the computer hardware.

Chris

-----Original Message-----
>From: Ed Okerson <ed at okerson.com>
>Sent: Apr 27, 2007 6:58 PM
>To: dprglist at dprg.org
>Subject: Re: [DPRG] computer vision frame rate
>
>Hi Chris,
>
>It appears to me that a significant amount of your time is spent decoding
>JPEG images.  Have you considered using a camera module that gives you raw
>image data instead of JPEG?  Some of the guys in the Home Brew Robotics
>club in San Jose were working on doing this with an FPGA.  I see they have
>now formed a company around this:
>
>http://www.roboticore.com/products.htm
>
>I know they were doing some image pre-processing with hardware blocks in
>the FPGA before passing it to the MicroBlaze processor, but surely you
>would gain a lot even by having one of these style camera modules
>connected directly to the bus of your ARM.
>
>Ed Okerson
>
>> Hello,
>>
>> The computer vision stuff I'm working on is starting to
>> work with a webcam, ARM9 SBC and RC toy car robot. Here
>> are some basic timings that give an idea of what to
>> expect.
>>
>> The webcam is:    Labtec Notebook, 640x480 CMOS, 15 fps
>> The computer is:  Technologic TS-7260 ARM9 200 MHz
>>
>> 1. Capture/sync frame 0 (usually corrupted)       80 ms
>> 2. Capture/sync frame 1 (use this one)            10 ms
>> 3. Decode JPEG, initial image processing         290 ms
>> 4. Threshold feature image for clear corridor    160 ms
>> 5. Goto step 1 and repeat
>>
>> I'm using EmbedCV for the computer vision stuff. With a
>> little optimization, I think that I can get a little
>> better than 2 fps in a look/drive loop. Beyond this, a
>> lower camera resolution is required.
>>
>> This has been very interesting so far. Several thoughts
>> come up.
>>
>> 1. Computer vision frame rate is determined by how long
>>    it takes to process each frame.
>>
>> Ok, I know this is obvious... If a camera has a frame
>> rate of 15 fps, then roughly 70 ms separate each image
>> in the video stream. If the computer vision processing
>> takes just 140 ms, then the frame rate of the system
>> drops down to 5 fps. More likely, the algorithms will
>> take longer than this. It is very difficult to maintain
>> a good frame rate without a fast or parallel computer.
>>
>> The alternative is a pipeline of slow computers to
>> maintain frame rate. But then latency increases. I
>> suspect this is what many network cameras do as high
>> latency is not a problem for that application.
>>
>> 2. Vision frame rate is determined by computation and
>>    perception rather than hardware
>>
>> The more the computer must think about each image, the
>> slower it will "see" regardless of camera frame rate.
>> So I think visual perception depends on what kind of
>> algorithms are required. If an environment requires
>> something cheap and fast like color image segmentation,
>> then the frame rate can be fairly high. But if the task
>> requires object recognition, then the frame rate drops
>> as it takes longer to process each image frame.
>>
>> The system must be able to accommodate this. It can't
>> have a fixed main control loop that is disturbed by
>> change in the vision processing. That leads to either
>> a task or multi-thread/process based design.
>>
>> Chris
>> _______________________________________________
>> DPRGlist mailing list
>> DPRGlist at dprg.org
>> http://list.dprg.org/mailman/listinfo/dprglist
>>
>
>
>_______________________________________________
>DPRGlist mailing list
>DPRGlist at dprg.org
>http://list.dprg.org/mailman/listinfo/dprglist