Robots and VR controls

You don’t write VR code, and you don’t build a headset app. Ballot supplies the headset experience and controls for each type of robot, and connects them to the controller your robot already has. Pilots never move joints directly: they send commands, such as where a gripper should go, which way to walk or where to fly, and your controller does the rest.

​

From pilot to robot

The headset draws the scene itself, so looking around never waits for the network. The robot’s actual motion comes back from the simulation, so what the pilot sees is what really happened.

​

Robot types and their commands

Commands always arrive at a fixed rate, in the robot’s own frame and units, and within the limits you declare, such as a top speed or a reach. The exact controls pilots use for each type are in Controls.

​

Three layers, and who owns each

This split is why you don’t write VR code. The interaction is the same for every robot of a type, your robot’s details come from your own files, and your controller stays yours.

​

What’s automatic, and what isn’t

When something is ambiguous, preparation stops and asks. It never guesses, and it never retunes your controller’s gains to make motion look smoother.

​

Legged robots, humanoids and drones

Robots that have to balance or fly bring their own controller: a trained walking policy, a model-predictive controller or a flight controller. Pilots say where to go; your controller keeps the robot upright or in the air. Preparation checks that it stands, walks, turns or hovers before anyone pilots it, and a robot that can’t stay up on its own isn’t made pilotable.

​

Several robots in one scenario

Each robot in a scenario is a seat with its own pilot and its own command stream. Commands from one seat never reach another seat’s robot. Robots interact only through your simulation: they can bump into each other, push the same crate or hand an object over, exactly as your physics says. See Create a scenario.

​

What pilots see

• The actual robots, as the simulation reports them, including teammates’ robots with name tags.
• What they’re asking for: a target marker at a gripper or hand, a route for a rover, a flight path for a drone, and a clear sign when a command has been limited.
• A preview of where an arm is heading, for robots where it’s been checked. It shows intent, not a promise: the real arm may take a slightly different posture.

​

Delays and dropouts

A remote robot always lags a little behind the pilot. Ballot keeps that safe and predictable:

• A command that arrives too late is dropped rather than applied late.
• If tracking, the connection or the display falls behind, control pauses: arms and hands hold, rovers and walkers stop, drones hover. The pilot takes control again to continue.
• After a reset or a reconnect, control starts fresh from the robot’s current state, never from an old command.

​

Swapping robots

If your framework can switch robots by name, as robosuite can, swapping an arm is a small change followed by ballot sim prepare. In a scenario, swapping means pointing a seat at a different robot package. Either way Ballot rechecks controls, reach and visuals, and new episodes record which robot was used. See robosuite.

​

Bring your own robot

To use a robot your framework doesn’t ship, package it with its model and controller. A model file on its own isn’t a robot yet: a URDF has no actuators, and neither format says how the robot is controlled. See Upload a robot.