Intro

In the course of developing an aframe component for a client, I’ve learned some tricks regarding threejs.

1. Detect cursor event from the 2D web into 3D scene.
2. Translate 2D behaviour into the 3D world.
3. Threejs screen/world conversion.

I will talk about each of them in the following segmens.

mouse cursor events in AFRAME

After aframe 0.9.0, things become much integrated when it comes to capture the cursor events into aframe component.

<a-scene cusor="rayOrigin: mouse"></a-scene>

After specifying “rayOrigin”, the scene will enable a raycaster component from the mouse position.

You can then listen to cursor events like you used to do with VR controllers. Since the interaction is calculated by a raycaster component, the event detail includes a set of interaction data which you can learn where the interaction occurs.

this.el.addEventListener("click", (e)=> {
  console.log(e.detail.intersection.distance)
})

cursor location in the 3D world

The original goal for this component, is to let user drag a 3d object around from the 2d screen statically, while displaying at the same position of the cursor on the screen.

For this to be true at every instance we move the cursor, it means that the object have to be relocated in the normal direction of the viewport plane where our cursor is on.

the threejs way to do this

For the original threejs way to do this, you can find many tutorials like this detailed one Three-js Drag and Drop.

The key here is this:

// capture mouse events here

// screen coordinates start at top-left,
// we reset it to the center of the screen, which is a 2*2 box.
var mouseX = (event.clientX / window.innerWidth) * 2 - 1;
var mouseY = -(event.clientY / window.innerHeight) * 2 + 1;

// z is unimportant, since the cursor is sticking on a plane parellel to the camera plane
var arbitraryZ = -1
var cursorIn3D = new THREE.Vector(mosueX, mouseY, arbitraryZ)

cursorIn3D.unproject(currentCam)

The last line is the most important. Since we can get 2d coordinates of a 3d point by projecting it into a viewing plane. We can reverse this process to get the 3d position of a 2d point sticking on the plane.

dragging

If we want to move objects where it always stay in front of the cursor:

// Lets say we start dragging by clicking.

var direction = cursorIn3D.clone().sub(currentCam.position).normalize()
var distance = currentCam.position.clone().sub(object.position).length()

// After capture mousemove event here...

var target = currentCam.position.clone().add(direction.multiplyScalar(distance))

object.position.copy(target)

easier way with AFRAME

As mentioned earlier, if we set “rayOrigin: mouse” in the cursor component, it actually setup a raycaster and calculate whether the current cursor position is pointing directly toward the object.

By the time we are notified by these events, the direction and distance of the instance is also provided by us in the event payload. With aframe things become much easier.

AFRAME.registerComponent("dragndrop", {
  init: function() {
    this.dist = null
    this.dir = new THREE.Vector3()

    this.scene = this.el.sceneEl
    this.camera = this.scene.camera.el

    this.el.addEventListener("mousedown", e =>{
      // update the base distance between the cursor to the object
      this.dist = this.el.object3D.position.clone()
        .sub(this.camera.object3D.position).length()

      this.dir.copy(this.scene.getAttribute("raycaster").direction)

      this.el.addState("being-dragged")
    })

    document.addEventListener("mousemove", () =>{
      // update the direction from raycaster coponent
      this.dir.copy(this.scene.getAttribute("raycaster").direction)
    })

    this.el.addEventListener("click", e=> {
      // complete the drag when click completed
      this.el.removeState("being-dragged")
    })

  },
  tick: function() {
    // only move it when certain state is meet
    if (this.el.is('being-dragged')) {
      var target = this.camera.object3D.position.clone()
        .add(this.dir.multiplyScalar(this.dist))

      this.el.object3D.copy(target)
    }
  }
})

additional gesture with mouse wheel

It’s natural for 3d applications to use wheel to send a object further/close, we can similarly implement this by:

var extra = 0
document.addEventListener("wheel", e=>{
  if (e.deltaY >0) {
    extra += 0.1
  } else {
    extra -= 0.1
  }
})

// change the distance to base + extra
var target = camera.object3D.position.clone().add(camDir.multiplyScalar(distance + extra))

this.el.object3D.position.copy(target)

You can similarly add secondary mouse action or other keyboard actions into the component.

thigns I’ve learned from mistakes

When developing AFRAME components there is a event slot in the component documentation. I was thinking it means for a place you can add event listener on the element it was initiated. I was wrong, it is per component only, meant to be use for global component cleanup.

conclusion

AFRAME is such a well developed framework that ideas are abstracted at a very high level where you can be very productive implementing E.C.S systems.

Here’s the component I’ve created:

npm module link live demo link

If you want to play around, you can check the live demo.