Leap Motion Controller Project: DIY Tony Stark Computer

 
From our STAR Maker: Geeve George

Leap Motion Controller for your Computer

Hi Guys, I’m Geeve George a 15 year old Maker. I love Computer Vision , Android Development and Algorithm Design. I am currently in 11th Grade and also a Research Collaboration Student with the MIT Media Lab India Initiative :)

Most of you might have watched the Iron-Man Movies. Tony Stark’s inventions are the main attraction in the iron-man series. My dream is to build the Iron-man suit some day. And the inventions in the Stark Tower are something which I always wish to make. You might have seen that Tony Stark controls his computer in the air. When I was younger, I used to always love the way tony stark simplified the interaction between the physical world and digital world. And so I would try to replicate them in my projects. Now I am happy to make an Instructable on a cool leap motion controller project to control your computer in the air :)

I would like to extend my special thanks to Intel, Instructables and Adafruit for awarding me the Adafruit Giftcards with which I was able to buy the Leap Motion Controller.

What are the stuff required to do this?

  1. Leap Motion Controller (Buy it here)
  2. Eclipse IDE for Java Developers (Download Here)
  3. Leap Motion Setup File (Download Here)
  4. Leap Motion SDK (Download Here)
  5. Java SE 7 (Download Here) ( Download JDK and JRE )

If you don’t know how to setup eclipse, I recommend this Introduction Tutorial to Java Series: link. Now let us move to the steps involved for creating the program for the leap motion controller.

How does it work?
Leap motion has two IR Cameras and three IR LEDs. The leap motion forms a 3D Collection of data using the three IR LEDs that put out an IR Dot pattern and the IR Cameras take pictures known as frames. The frames get sent from the leap motion device to the computer and we can program it. The leap motion analyses the pictures and the dot positioning using complex math and then compares the data of the two pictures taken by IR Cameras and then use the two 2D representations to get a 3D Representation. The leap motion device captures at 300 fps. We then program what happens to each of the frames sent to our computer from the leap motion controller.

Step 1 : Software Setup for the Leap Motion Controller

Leap motion supports a variety of Programming Languages, like C++, C#, Python, Java and Javascript.In this Tutorial we will be programming in Java. Please note that Leap Motion Library only works Java 6 or Java 7.

Initial Setup for the leap motion controller :

  1. Firstly, go here and download the Leap Motion Setup file and install the application. It installs everything to connect the leap motion controller to your computer.
  2. Next, go here (Developer Portal) and create an account. Then download the Leap Motion SDK.
  3. Then extract all the files of the zip file to any folder of your choice.

Configuring the Classpath :

In this tutorial we will be using the Eclipse IDE for Java Developers.

1. Open Eclipse and click on File > New > Java Project .

2. Give the project a name, and make sure you set execution runtime environment JRE to Java SE 1.7 as shown in the picture.

Leap Motion Controller Project Name

3. Then click on next, and open the Libraries Tab and select “Add External JARs” and navigate to the folder you previously extracted.

Leap motion controller project

4. Then open the LeapSDK folder and go to the libs folder and select the LeapJava.jar file.

5. Next, click on the triangular drop down button beside LeapJava.jar and then click on Native Library Location from the Drop Down Menu , then click on edit as shown in the picture.

Leap motion controller project

Leap motion controller project

6. Then click on External Folder and navigate to LeapSDK folder and select x64 or x86 folder based on your Operating System and click OK and then click on finish.

We now have the Leap Motion Controller set up with our project, now let’s get to some coding!

Step 2 : Leap Motion Listener

Now that we are done importing the libraries, let us start working on the project.

Initial Setup :

  1. First we need to create a new class in the source folder, and for that, click on the triangular drop-down arrow beside your project folder.
  2. Inside your project folder right click on the src folder go to new>and click on class to create a new Java Class.
  3. Give a name to the class and click finish.

Now lets start with the code. First we create a Leap Motion Listener. The listener detects when the leap motion controller is connected to the computer. (Note : Import the necessary libraries). The code for the leap motion listener is given below:

package starkmouse;
import java.io.IOException;
import java.awt.AWTException;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.awt.MouseInfo;
import java.awt.Point;
import java.awt.Robot;

import java.awt.event.InputEvent;
import java.awt.event.KeyEvent;

import com.leapmotion.leap.*;
import com.leapmotion.leap.Controller.PolicyFlag;

public class leapmoues {
	public static void main(String[] args) throws AWTException {
		Controller controller = new Controller();
		controller.setPolicyFlags(PolicyFlag.POLICY_BACKGROUND_FRAMES);

SampleListener listener = new SampleListener();
		controller.addListener(listener);
		// controller.enableGesture(Gesture.Type.TYPE_SCREEN_TAP);
		// controller.enableGesture(Gesture.Type.TYPE_SWIPE);
		controller.enableGesture(Gesture.Type.TYPE_CIRCLE);

System.out.println("Press Enter to quit...");
		try {
			System.in.read();
		} catch (IOException e) {
			e.printStackTrace();
		}

		controller.removeListener(listener);
	}
}class SampleListener extends Listener {

	boolean readyForControl = false;
	int screenWidth;
	int screenHeight;
	boolean iBoxGet = false;
	InteractionBox iBox = null;
	Robot robot;
	boolean isMoving = false;
	boolean unGrip = false;
	boolean wasFacingDown = true;
	boolean wasInTabState = false;
	boolean wasTabbing = false;
	boolean justCircleGestured = false;
	boolean isResizing = false;

	public void onConnect(Controller controller) {
		System.out.println("Connected");
		GraphicsDevice gd = GraphicsEnvironment.getLocalGraphicsEnvironment()
				.getDefaultScreenDevice();
		screenWidth = gd.getDisplayMode().getWidth();
		screenHeight = gd.getDisplayMode().getHeight();
		System.out.println("Screen Resolution: X: " + screenWidth + ", H: " + screenHeight);
		readyForControl = true;
		try {
			robot = new Robot();
			robot.setAutoDelay(5);
		} catch (AWTException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

Step 3 : Frame and Gesture Data from the Leap Motion Controller

Leap motion controller for computer

Leap Motion Frame Data which includes the Gestures (Circle, Pinch, Palm Face Down and all other gestures).

Leap motion has two IR Camera and three IR Led’s. The leap motion forms a 3D Collection of data using the three IR Led that put out an IR Dot pattern , the IR Camera’s takes pictures known as frames. The frames get sent from the leap motion controller to the computer and we can program it.

The leap motion analysis the pictures and the dot positioning using complex math and then compare the data of the two pictures taken by IR Camera and the use the two 2D representation to get a 3D Representation.

The leap motion device captures at 300 fps. We then program what happens to each of the frames sent to our computer from the leap motion controller.

public void onFrame(Controller controller) {

	Frame frame = controller.frame(); // The latest frame
		// Frame previous = controller.frame(1); //The previous frame
		// System.out.println("Frame available");
		if (!iBoxGet) {
			iBox = frame.interactionBox();
			iBoxGet = true;
			System.out.println("Interaction box set!");
		}
		// Pointable furthestFront = frame.pointables().frontmost();
		Hand rightHand = frame.hands().rightmost();
		Vector palmV = rightHand.palmVelocity();
		// System.out.println("Velocity: X: " + palmV.getX() + ", Y: " +
		// palmV.getY()
		// + ", Z: " + palmV.getZ());
		Vector palmN = rightHand.palmNormal();
		// System.out.println("Normal: X: " + palmN.getX() + ", Y: "
		// + palmN.getY() + ", Z: " + palmN.getZ());

	Point mouseLoc = MouseInfo.getPointerInfo().getLocation();
		int currentMouseX = mouseLoc.x;
		int currentMouseY = mouseLoc.y;

	if (readyForControl && rightHand.confidence() > .15) {
			if (!isMoving && !wasInTabState && frame.hands().count() > 1) {
				Hand leftHand = frame.hands().leftmost();
				if (leftHand.pinchStrength() > .8
						&& rightHand.pinchStrength() > .8) {
					if (!isResizing) {
						System.out.println("Resizing...");
						robot.keyPress(KeyEvent.VK_ALT);
						robot.keyPress(KeyEvent.VK_SPACE);
						robot.keyRelease(KeyEvent.VK_SPACE);
						robot.keyRelease(KeyEvent.VK_ALT);
						robot.keyPress(KeyEvent.VK_S);
						robot.keyRelease(KeyEvent.VK_S);
						robot.keyPress(KeyEvent.VK_DOWN);
						robot.keyPress(KeyEvent.VK_RIGHT);
						robot.keyRelease(KeyEvent.VK_DOWN);
						robot.keyRelease(KeyEvent.VK_RIGHT);
						isResizing = true;
					}
				}else{
					if(isResizing){
						System.out.println("Resizing complete!");
						robot.mousePress(InputEvent.BUTTON1_MASK);
						robot.mouseRelease(InputEvent.BUTTON1_MASK);
						isResizing = false;
					}
				}
			}

			// System.out.println("Confidence: " + rightHand.confidence());
			if (rightHand.grabStrength() > .99 && !wasInTabState && !isResizing) {
				if (!isMoving && palmN.getY() < .8) {
					robot.keyPress(KeyEvent.VK_ALT);
					robot.keyPress(KeyEvent.VK_SPACE);
					robot.keyRelease(KeyEvent.VK_SPACE);
					robot.keyRelease(KeyEvent.VK_ALT);
					robot.keyPress(KeyEvent.VK_R);
					robot.keyRelease(KeyEvent.VK_R);
					robot.keyPress(KeyEvent.VK_ALT);
                                        robot.keyPress(KeyEvent.VK_SPACE);
					robot.keyRelease(KeyEvent.VK_SPACE);
					robot.keyRelease(KeyEvent.VK_ALT);
					robot.keyPress(KeyEvent.VK_M);
					robot.keyRelease(KeyEvent.VK_M);
					robot.keyPress(KeyEvent.VK_DOWN);
					robot.keyRelease(KeyEvent.VK_DOWN);
					isMoving = true;
				}

				// System.out.println(rightHand.grabStrength());
			}

else {
//System.out.println("Not grabbing");
if (isMoving) {
robot.mousePress(InputEvent.BUTTON1_MASK);
robot.mouseRelease(InputEvent.BUTTON1_MASK);
isMoving = false;
if (palmN.getX() != 0 && palmN.getY() != 0 && palmN.getZ() != 0) {

if (palmN.getY() < -.1 && palmN.getZ() > -.8) {
if (currentMouseY <= 8) {

robot.keyPress(KeyEvent.VK_WINDOWS);
robot.keyPress(KeyEvent.VK_UP);
robot.keyRelease(KeyEvent.VK_WINDOWS);
robot.keyRelease(KeyEvent.VK_UP);
}

else {
if (screenWidth - currentMouseX <= 12) {

robot.keyPress(KeyEvent.VK_WINDOWS);
robot.keyPress(KeyEvent.VK_RIGHT);
robot.keyRelease(KeyEvent.VK_WINDOWS);
robot.keyRelease(KeyEvent.VK_RIGHT);
} else if (currentMouseX <= 12) {
									robot.keyPress(KeyEvent.VK_WINDOWS);
robot.keyPress(KeyEvent.VK_LEFT);
robot.keyRelease(KeyEvent.VK_WINDOWS);
robot.keyRelease(KeyEvent.VK_LEFT);
}
}
} else {

System.out.println("Normal: X: " + palmN.getX() + ", Y: " + palmN.getY() + ", Z: " + palmN.getZ());
robot.keyPress(KeyEvent.VK_ALT);
robot.keyPress(KeyEvent.VK_SPACE);
robot.keyRelease(KeyEvent.VK_SPACE);
robot.keyRelease(KeyEvent.VK_ALT);
robot.keyPress(KeyEvent.VK_N);
robot.keyRelease(KeyEvent.VK_N);
}
  }
	}
		}		if (!isMoving && !isResizing) {
				if (palmN.getY() < -.8 && palmN.getZ() > -.5) {
					wasFacingDown = true;
					wasTabbing = false;
					if (wasInTabState) {
						robot.keyPress(KeyEvent.VK_ENTER);
						robot.keyRelease(KeyEvent.VK_ENTER);
						wasInTabState = false;
					}
				} else if (palmN.getY() >= .8 && wasFacingDown
						&& !wasInTabState) {
					System.out.println("Alt tabbing");
					wasFacingDown = false;
					wasInTabState = true;
					wasTabbing = false;
                                        robot.keyPress(KeyEvent.VK_ALT);
					robot.keyPress(KeyEvent.VK_CONTROL);
					robot.keyPress(KeyEvent.VK_TAB);
					robot.delay(100);
					robot.keyRelease(KeyEvent.VK_TAB);
					robot.keyRelease(KeyEvent.VK_CONTROL);
					robot.keyRelease(KeyEvent.VK_ALT);
					try {
				Runtime.getRuntime().exec("cmd /c start " + C:\WindowSwitcher.lnk");
					} catch (IOException e) {
						// TODO Auto-generated catch block
						e.printStackTrace();
					}
					robot.delay(300);
				} else if (wasInTabState && !wasFacingDown && !wasTabbing
						&& palmN.getY() < .45) {                                         wasTabbing = true;				  				} else if (wasInTabState && !wasFacingDown && wasTabbing 						&& palmN.getY() > .75) {
					robot.keyPress(KeyEvent.VK_TAB);
					robot.keyRelease(KeyEvent.VK_TAB);
					wasTabbing = false;
				}
			}/*
			 * if (!isMoving && !wasInTabState) { /* if(palmN.getZ() <= -.7 &&
			 * rightHand.grabStrength() < .1){ 			 * System.out.println("Palm vertical velocity: " + 			 * rightHand.palmVelocity().getY()); //float resultVerticalV = 			 * Math.round(Math.abs(rightHand.palmVelocity().getY()) - 1); 			 * //if(resultVerticalV > 0){ robot.mouseWheel((int)
			 * Math.round(((rightHand.palmVelocity().getY()) / 500))); //}
			 * }else{
			 */
			if (!isMoving && !wasInTabState && frame.gestures().count() > 0
					&& frame.hands().count() == 1 && !isResizing) {
				CircleGesture circleGesture = new CircleGesture(frame
						.gestures().get(0));
				// System.out.println("Pinch strength: " +
				// rightHand.pinchStrength());
				if (circleGesture.durationSeconds() > .5 && !justCircleGestured
						&& rightHand.pinchStrength() > .8) {
					System.out.println("Closed a window!");
					robot.keyPress(KeyEvent.VK_ALT);
					robot.keyPress(KeyEvent.VK_F4);
					robot.keyRelease(KeyEvent.VK_F4);
					robot.keyRelease(KeyEvent.VK_ALT);
					justCircleGestured = true;
				}
			} else {
				justCircleGestured = false;
			}float xSpeed = (palmV.getX() / 6);
			float ySpeed = (palmV.getY() / 6);
			// System.out.println("xSpeed: " + xSpeed + ", ySpeed: " + ySpeed);
			robot.mouseMove((int) (currentMouseX + xSpeed),
					(int) (currentMouseY - ySpeed));		// }
		}
	}
}

 

 

The complete code for this project can be found here: Complete code or here: Github.

Running the Program and Results from the leap motion controller :

Now that we have completed programming the Leap Motion Controller we can run the program in eclipse. Now you can finally check out the working video of this project:

As you have seen in the video our program can recognise various Gestures. And now it’s your time to add more gestures using your own leap motion controller with the help from the Leap Motion Programming Guide here.

I’m also on YoutubeFacebook and Linkedin , feel free to ping me in case of any queries :)
My Personal Blog :http://www.geevegeorge.co.vu
This project can also be seen on Instructables: link.

Geeve George- STAR Maker at DIY Hacking

-Geeve George
I am a 15 year old Maker proudly from India!
I love making “moonshots”! I wish to help the society with my projects!
I work mainly with Arduino , Intel Edison and Galileo.I love Android Development!

 

 

 

 

 

Recent Posts
Showing 7 comments
  • Reply

    Ihr mobiler Schlüsseldienst in Nürnberg ohne versteckte Kosten zu
    einem Festpreis von 37,00 € inkl. Anfahrt. Nehmen Sie
    gleich Kontakt auf

  • av receiver vergleich
    Reply

    I have read some just right stuff here. Certainly price bookmarking for revisiting.
    I surprise how a lot attempt you put to make this type of fantastic informative web site.

  • meilleur aspirateur robot
    Reply

    each time i used to read smaller content that also clear their
    motive, and that is also happening with this article which I am reading at this place.

  • tilbud støvsuger
    Reply

    At this time I am ready to do my breakfast, afterward having my breakfast
    coming again to read other news.

  • suhail stark
    Reply

    It’s also mydream but iam a biginer to electronics and never worked with aurduino and other but iam a programmer

  • suhail stark
    Reply

    It’s also mydriasis but iam a biginer to electronics and never worked with aurduino and other but iam a programmer

Leave a Comment

− 5 = 1

Start typing and press Enter to search