Towards building an accurate proxy of human behavior from large motion data, a key problem is to model rich individual variation across a population of users. Using motion capture data from 20 keyboard typists, we learn and examine representations of distinct typing styles by training a generative model that replicates typing motions based on example clips from individual typists’ histories. Beyond correctly typing unseen target text, our model captures varied movement and finger use patterns, forming a latent space of typing styles that are separable and interpolable. Our findings suggest that individual examples could provide a basis for expressive behavior control in synthesizing individualized human motion.
The same prompt, “Thanks for your help this morning.”, typed in the style of three different users. Style weights (1, 0, 0), (0, 1, 0) and (0, 0, 1) select one user each.
None of these typists exist. Each video interpolates the style embeddings of the three users above with the convex weights shown, producing a typing style between them. Same prompt as above.
We treat typing as conditional trajectory generation. A latent diffusion model predicts the next window of hand motion and keypress states, conditioned on three things: the task (the target prompt and the current text buffer), the recent keypress and motion history, and a style embedding pooled from a handful of one-second reference clips of the target typist.
An autoencoder compresses motion-and-keypress windows into latents, and a Transformer denoiser learns to recover clean latents under these conditions. At inference the model runs autoregressively: decode a window, update the text buffer, roll the history forward, repeat. Because style enters only through examples, a new typist can be imitated from a few clips with no parameter updates, and interpolating between two typists’ embeddings blends their behavior.
Interpolating style embeddings blends finger use. Above, the model types held-out text while mixing typists A and B. As the interpolation weight moves from 0 to 1, the divergence from typist A’s finger-use distribution rises and the divergence from typist B’s falls, so the blend tracks both endpoints rather than collapsing onto one.
Example-based conditioning yields stylized, performant output. On 24 held-out English and Finnish sentences, one-second reference clips are enough to steer keystroke and movement patterns toward the target typist, and the effect strengthens with more clips (Nrefs). Typing demands sequential precision, so we add rejection sampling (RS): sample 64 candidate windows, simulate the resulting text, keep the one with the lowest error. RS cuts the error rate while leaving the style metrics essentially unchanged.
| Condition | UER ↓ | JSDF ↓ | JSDFK ↓ | SRA@1 ↑ |
|---|---|---|---|---|
| Nrefs = 0 | 0.126± 0.075 | 0.094± 0.067 | 0.292± 0.105 | 0.021± 0.143 |
| Nrefs = 1 | 0.131± 0.082 | 0.031± 0.029 | 0.091± 0.068 | 0.757± 0.429 |
| Nrefs = 5 | 0.155± 0.092 | 0.018± 0.015 | 0.047± 0.028 | 0.991± 0.094 |
| Nrefs = 5 + RS | 0.073± 0.055 | 0.019± 0.015 | 0.053± 0.032 | 0.988± 0.109 |
Mean ± standard deviation over 9,216 user–prompt pairs (UER, SRA@1) or over the 48 typists (JSD). UER: uncorrected error rate of the final text. JSDF / JSDFK: Jensen–Shannon divergence between generated and real finger-use / finger–key distributions. SRA@1: top-1 accuracy of recognizing the target typist from generated motion. 100 denoising steps throughout. Bold marks the best value per column.
