Demonstration Learning on the Factory Floor
Demonstration learning, also known as learning from demonstration or imitation learning, has long been part of robotics research. Its central idea is that a robot observes a person performing a task and extracts a generalizable policy from those observations. Researchers have studied this for over two decades. The difference now is the ability to make it work reliably in uncontrolled, real-world environments.
A factory floor differs from a research lab. Lighting can vary, nearby machinery can create vibration, and surface finishes can change sensor readings. Operators also guide demonstrations at different speeds, with slight trajectory changes from one attempt to the next. Most practical deployment difficulties sit between "works in controlled conditions" and "works on a production floor."
This article follows the technical pipeline from captured demonstration to deployable policy, including the engineering difficulties we have addressed at every stage.
Step 1: Capture the Demonstration
The procedure starts with an operator physically guiding the robot arm along the target motion. In kinesthetic teaching mode, the robot's joint torque limits are lowered, allowing the arm to move by hand without resistance. The operator holds the end effector or arm body and guides it along the intended trajectory.
As the operator guides the arm, the system records joint encoder readings at high frequency, typically 100 to 500 Hz, based on task complexity and the precision required. These readings form a time-series of joint configurations throughout the demonstration. Wrist-mounted force-torque sensors, when available, add contact information. A camera in the cell contributes visual frames synchronized with the joint data stream.
The raw recording contains noise. Guidance can introduce micro-oscillations, the robot's compliance mode can cause slight position lag, and rapid movement can produce motion blur in camera frames. Noise filtering and temporal alignment across sensor streams come first in processing.
Step 2: Pose Estimation and State Representation
Raw joint angles help, but they do not provide enough information for a generalizable policy. To accommodate part variation, the system must know the end effector's position relative to the parts being manipulated, rather than only its absolute joint configuration.
Pose estimation converts raw sensor readings into a task-relative state representation. A central difficulty is detecting parts in industrial conditions. Parts in production bins can face different directions, have reflective or dark matte surfaces that defeat simple vision methods, or be partly hidden by neighboring parts.
We combine depth sensing with trained object detection models to determine each part's pose relative to the robot base frame. That result is combined with the recorded end-effector trajectory, expressing the demonstration in task-relative coordinates. This approach means small shifts in part position do not invalidate the learned policy. The policy is parameterized by the part's location, rather than absolute robot joint angles.
Reliable pose estimation on factory floors required extensive work with reflective metal surfaces, uneven overhead lighting, and partial occlusion patterns in full production bins. Better algorithms alone do not remove these issues. Sensor placement, lighting geometry, and fallback behavior are also needed when detection confidence falls below threshold.
Step 3: Extract Motion Primitives
A full demonstration trajectory usually has a recognizable structure: approach, grasp or contact, manipulation, and release or exit. Handling it as one monolithic sequence makes the policy brittle, since an error in one phase can carry into every later phase.
Motion primitive segmentation finds these phases and handles them as semi-independent components. It uses velocity profiles, contact force signatures, and learned phase transition models. The output is a collection of primitive motion segments, each defining its own state conditions for entry and exit.
This segmented form improves reliability. If the approach phase places the end effector slightly differently from the training demonstration, the grasp phase can adapt from its own starting condition instead of carrying forward the entire error. Adapting one phase at a time also lets the system add recovery actions when a phase fails or leaves an uncertain state.
Step 4: Train and Run the Policy
Policy training uses segmented, task-relative demonstration data to create a model that maps current state observations to control actions. At its simplest, the objective is behavioral cloning. Industrial reliability also calls for explicit treatment of out-of-distribution states, uncertainty estimates for action outputs, and a safe hold state when confidence drops below a threshold.
In a factory, a practical training set may consist of a single demonstration session covering twenty to forty repetitions. For that volume of data, model architectures that perform well in research with thousands of demonstrations need substantial adaptation. We have succeeded with structured models that encode strong inductive biases from the primitive decomposition instead of learning everything end to end from raw trajectories.
Inference runs on the Robai edge unit rather than in cloud. Control requires low latency: a policy taking 300 milliseconds to issue an action command cannot support fine manipulation tasks. On-device inference keeps the control loop tight and avoids the internet dependency that would be unacceptable in most production environments.
Where Lab Assumptions Fail
An honest account must include failure modes. In its current form, demonstration learning works well when a task falls within the distribution represented by the demonstrations. Fundamentally novel part configurations outside the training distribution lead to lower performance. A set of demonstrations for one task type does not make the system generalize to physically different tasks.
The quality of operator demonstrations matters as well. Repetitions with highly inconsistent speed or trajectories create noisier policies. We have worked to handle operator variation, but there is a lower limit where demonstration quality materially reduces policy quality. Fortunately, operators usually become more consistent after a few practice repetitions. The first few demonstrations can also be filtered out or given lower weight during training.
This is the technical pipeline we have built and tested with early-access partners in production environments. Its fundamentals fit the target task types: structured manipulation, pick-and-place variants, inspection, and rework motions. The boundary conditions are real, and we make no claim otherwise. For the specific goal of reducing robot task-change time in variable production, the pipeline works in the environments where we have deployed it.