Spirit Machine Vision Discussion

The more technical aspects of Spirit, and carry-over discussion from Kickstarter updates.
Kevin
Posts: 133
Joined: Tue Jul 28, 2015 12:56 am

Spirit Machine Vision Discussion

Postby Kevin » Sat Dec 31, 2016 5:17 pm

Hello all!

In the recent Kickstarter updates, we've been discussing various options for running the machine vision system on Spirit. I built the prototype using SimpleCV, which works, but it's certainly not ideal. Much better solutions exist but as I'm not much of a software guy, I'm hoping for some help from the rest of you.

A few options I've had a quick look at:

OpenCV: This appears to be the go-to standard for this kind of work, though at first glance, it looks complicated.

OpenMV: This looks like a possibly great solution, but I haven't dug into it yet.
https://openmv.io/

Pixy from Charmed Labs: Not sure what software this device runs, but from the looks of it, they've got the vision aspect dialed in.
http://charmedlabs.com/default/pixy-cmucam5/


Requirements:

We would really like a solution that runs on Python 2.7. As far as I can tell, the latest OpenCV only runs on Python 3.0, but we have a few other things running on the robot requiring 2.7. I tried to move those to 3.0 a while back and it was a huge mess so if we can stick with 2.7 for the vision, so much the better.

The second big requirement is that it can be made easy. All of these bits of software appear quite complex, but if I can understand it well enough and I can write some simple functions that are "user friendly for the masses" then it'll work.

The production hardware will be a Raspberry Pi B3 with the original Rev1 Raspberry Pi camera (the 5MP design), though we will eventually move to the new Rev2 (8MP) Pi Camera, so if we could create a vision solution that is easy to swap between cameras, this would also be a huge plus. Eventually we will need to run it on the Rev2 8MP camera one way or the other as the Rev01 cams are harder to find these days.

So I'll open it up for discussion below. If you have a Pi and a camera module, and you have more advanced understanding of how all this stuff works, we would really appreciate your help. Share your thoughts on the different options, discuss between yourselves, play with some demo code, etc.

Thanks in advance for the help!!!

Cheers!
-Kevin

ihermit2
Posts: 1
Joined: Sat Dec 31, 2016 10:23 pm

Re: Spirit Machine Vision Discussion

Postby ihermit2 » Sat Dec 31, 2016 10:25 pm

I'd be happy to help if you want packages moved to 3.0 from 2.7 if they're open source. how long is the list that would need to be brought forward?

esba1ley
Posts: 2
Joined: Sun Jan 01, 2017 2:15 am

Re: Spirit Machine Vision Discussion

Postby esba1ley » Sun Jan 01, 2017 3:25 am

Kevin wrote:Hello all!

In the recent Kickstarter updates, we've been discussing various options for running the machine vision system on Spirit. I built the prototype using SimpleCV, which works, but it's certainly not ideal. Much better solutions exist but as I'm not much of a software guy, I'm hoping for some help from the rest of you.

A few options I've had a quick look at:

OpenCV: This appears to be the go-to standard for this kind of work, though at first glance, it looks complicated.

OpenMV: This looks like a possibly great solution, but I haven't dug into it yet.
https://openmv.io/

Pixy from Charmed Labs: Not sure what software this device runs, but from the looks of it, they've got the vision aspect dialed in.
http://charmedlabs.com/default/pixy-cmucam5/


Requirements:

We would really like a solution that runs on Python 2.7. As far as I can tell, the latest OpenCV only runs on Python 3.0, but we have a few other things running on the robot requiring 2.7. I tried to move those to 3.0 a while back and it was a huge mess so if we can stick with 2.7 for the vision, so much the better.



Kevin,

You can stick with Python 2.7 and push off a code overhaul (not sure how massive... would need to see what you're working with there) if you allow your product to use an earlier version of OpenCV.

Can you get away with running OpenCV 2.4? There are a bunch of openCV 2.4 components and python-opencv available via the Debian package manager under Raspbian...

Code: Select all

dpkg -l | grep opencv


returns

Code: Select all

ii  libopencv-calib3d-dev:armhf           2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-calib3d
ii  libopencv-calib3d2.4:armhf            2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Camera Calibration library
ii  libopencv-contrib-dev:armhf           2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-contrib
ii  libopencv-contrib2.4:armhf            2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision contrib library
ii  libopencv-core-dev:armhf              2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-core
ii  libopencv-core2.4:armhf               2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision core library
ii  libopencv-dev                         2.4.9.1+dfsg-1+deb8u1                     armhf        development files for opencv
ii  libopencv-features2d-dev:armhf        2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-features2d
ii  libopencv-features2d2.4:armhf         2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Feature Detection and Descriptor Extraction library
ii  libopencv-flann-dev:armhf             2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-flann
ii  libopencv-flann2.4:armhf              2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Clustering and Search in Multi-Dimensional spaces library
ii  libopencv-gpu-dev:armhf               2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-gpu2.4
ii  libopencv-gpu2.4:armhf                2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision GPU library
ii  libopencv-highgui-dev:armhf           2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-highgui
ii  libopencv-highgui2.4:armhf            2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision High-level GUI and Media I/O library
ii  libopencv-imgproc-dev:armhf           2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-imgproc
ii  libopencv-imgproc2.4:armhf            2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Image Processing library
ii  libopencv-legacy-dev:armhf            2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-legacy
ii  libopencv-legacy2.4:armhf             2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision legacy library
ii  libopencv-ml-dev:armhf                2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-ml
ii  libopencv-ml2.4:armhf                 2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Machine Learning library
ii  libopencv-objdetect-dev:armhf         2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-objdetect
ii  libopencv-objdetect2.4:armhf          2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Object Detection library
ii  libopencv-ocl-dev:armhf               2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-ocl2.4
ii  libopencv-ocl2.4:armhf                2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision OpenCL support library
ii  libopencv-photo-dev:armhf             2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-photo2.4
ii  libopencv-photo2.4:armhf              2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision computational photography library
ii  libopencv-stitching-dev:armhf         2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-stitching2.4
ii  libopencv-stitching2.4:armhf          2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision image stitching library
ii  libopencv-superres-dev:armhf          2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-superres2.4
ii  libopencv-superres2.4:armhf           2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Super Resolution library
ii  libopencv-ts-dev:armhf                2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-ts2.4
ii  libopencv-ts2.4:armhf                 2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision ts library
ii  libopencv-video-dev:armhf             2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-video
ii  libopencv-video2.4:armhf              2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision Video analysis library
ii  libopencv-videostab-dev:armhf         2.4.9.1+dfsg-1+deb8u1                     armhf        development files for libopencv-videostab2.4
ii  libopencv-videostab2.4:armhf          2.4.9.1+dfsg-1+deb8u1                     armhf        computer vision video stabilization library
ii  libopencv2.4-java                     2.4.9.1+dfsg-1+deb8u1                     all          Java bindings for the computer vision library
ii  libopencv2.4-jni                      2.4.9.1+dfsg-1+deb8u1                     armhf        Java jni library for the computer vision library
ii  opencv-data                           2.4.9.1+dfsg-1+deb8u1                     all          development data for opencv
ii  python-opencv                         2.4.9.1+dfsg-1+deb8u1                     armhf        Python bindings for the computer vision library


And for completeness, that's on a system that responds to

Code: Select all

uname -a


with

Code: Select all

Linux yoda 4.4.38-v7+ #938 SMP Thu Dec 15 15:22:21 GMT 2016 armv7l GNU/Linux


So it's the latest version of Raspbian Jessie (as of the writing of this post).

With all the above installed (and any required dependencies that the package manager determines), I can get open CV to work using the tutorials found here for both stills and video using the RasPi camera:

http://www.pyimagesearch.com/2015/03/30 ... nd-python/

I've also gotten OpenCV to work with finding interesting features to track and doing basic other Computer Vision tasks using the Open CV tutorials here:

https://media.readthedocs.org/pdf/openc ... troals.pdf

Is that enough to get you a basic set of stuff to install and get the functionality you want into your build for the project?

As you can tell, my personal vote is for opencv via the python interface and its inevitable crossover into C/C++ for higher performance... but that's just me. :D

-Erik
Last edited by esba1ley on Mon Jan 02, 2017 2:29 am, edited 1 time in total.

esba1ley
Posts: 2
Joined: Sun Jan 01, 2017 2:15 am

Re: Spirit Machine Vision Discussion

Postby esba1ley » Mon Jan 02, 2017 2:23 am

Kevin wrote:
The second big requirement is that it can be made easy. All of these bits of software appear quite complex, but if I can understand it well enough and I can write some simple functions that are "user friendly for the masses" then it'll work.



Computer vision isn't simple by its very nature. It requires some knowledge of vector and matrix mathematics (linear algebra) for it all to make sense. We should think about tutorials that get people thinking in terms of vectors/rays in order for the applications to make sense. Most of the OpenCV tutorials are pretty good, but they're a bit hodge-podge in terms of the path you want to take in terms of how to use the toolkit, because it's just that... a toolkit that has differnet tools that may or may not be used together.

Since your camera is on a pan/tilt head, a really simple application would be to track a blob of a specific color, and then command the pan/tilt head to track it. If you want the robot to follow it, have the robot close a control loop on "nulling" the pan channel error to zero by turning (relative to pointing "straight ahead" in the robot body space), regardless of the tilt angle. That would be a great exercise for the users to do, actually. That way you could have the robot "chase" someone holding a blue ball, or something like that. You could have it close a speed loop on tilt angle relative to horizontal such that looking horizontal would be full blast speed, and it would be throttled back to zero as the tilt angle increased away from the horizontal angle in the robot body space, all the while commanding the pan/tilt head to put the blob in the center of the camera field of view.

Hopefully that makes sense.

Some references that helped me are:

http://shop.oreilly.com/product/0636920022923.do

and

http://opencv.org/opencv-computer-visio ... ython.html

We'll need to distill those sources down into an education-environment-friendly presentation for the age range you're targeting. Adults can use those books directly.

An inspiration that took computer science principles and simplified them to conceptual activities that didn't require a computer that we could strive for can be found here:

http://csunplugged.org/

-Erik

Kevin
Posts: 133
Joined: Tue Jul 28, 2015 12:56 am

Re: Spirit Machine Vision Discussion

Postby Kevin » Fri Jan 06, 2017 6:05 pm

Thanks guys.

As far as making things simple, I'm not trying to teach the underlying theory of computer vision (at least not to start). I'm looking to create some simple python functions that do simple things, like find the biggest blue blob and return its center coordinates. That's what I'm doing with SimpleCV right now, though it's slow and difficult to tune.

I'll check into the links. The book looks like a good reference as well.

When writing code I always ask myself if I can make the functions simple enough that a 3rd grader can understand and use them. :) I think we'll loose a lot of people if we jump straight into vectors.

Does OpenCV 3.0 have significant benefits over the previous versions?

Have you guys looked at OpenMV? I'm all about using whatever software is most easy for a novice and some of the other options out there seem to advertise great performance and ease of use.

3dtech
Posts: 1
Joined: Thu May 11, 2017 7:16 am

Re: Spirit Machine Vision Discussion

Postby 3dtech » Thu May 11, 2017 8:42 am

thanks for this information.

odinshen
Posts: 2
Joined: Sat Oct 14, 2017 5:40 pm

Re: Spirit Machine Vision Discussion

Postby odinshen » Sun Oct 15, 2017 4:05 am

Hi,

Here is some stuff you guys may interesting which about apply AI/ML on Pi3, I will try it on my spirit rover and update here.
https://community.arm.com/graphics/b/blog/posts/cartoonifying-images-on-raspberry-pi-with-the-compute-library

Odin


Who is online

Users browsing this forum: No registered users and 1 guest