3D Dance Generation by Actor-Critic GPT with Choreographic Memory

CVPR 2022, Oral



Driving 3D characters to dance following a piece of music is highly challenging due to the spatial constraints applied to poses by choreography norms. In addition, the generated dance sequence also needs to maintain temporal coherency with different music genres. To tackle these challenges, we propose a novel music-to-dance framework, Bailando, with two powerful components: 1) a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequence to a quantized codebook, 2) an actorcritic Generative Pre-trained Transformer (GPT) that composes these units to a fluent dance coherent to the music. With the learned choreographic memory, dance generation is realized on the quantized units that meet high choreography standards, such that the generated dancing sequences are confined within the spatial constraints. To achieve synchronized alignment between diverse motion tempos and music beats, we introduce an actor-critic-based reinforcement learning scheme to the GPT with a newly-designed beat-align reward function. Extensive experiments on the standard benchmark demonstrate that our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively. Notably, the learned choreographic memory is shown to discover human-interpretable dancing-style poses in an unsupervised manner.

Bailando Framework

Two-Stage Choreography

Bailando applies a two-stage pipeline to choreograph. In the first stage, a pose VQVAE is deployed to encode and quantize meaningful dancing constituents from dancing motion data into a choreographic memory. In the second stage, a motion GPT is learnt to translate music to visually satisfactory dance. The GPT network is further improved by introducing an artifical "critic" to judge whether the generated dance is good and to guide the GPT to make change.



 title={Bailando: 3D dance generation via Actor-Critic GPT with Choreographic Memory,
 author={Siyao, Li and Yu, Weijiang and Gu, Tianpei and Lin, Chunze and Wang, Quan and Qian, Chen
 and Loy, Chen Change and Liu, Ziwei },
} }


Li Siyao
Email: siyao002 at