Skip to content
BranchesTags

Repository files navigation

cleave

Tests

Stem-separated music visualizer. Layer together drums, bass, vocal and other stems - each driving Milkdrop presets to create your own unique visual masterpieces.

Comprehensive visual editor allows you to browse and tune presets in real time - automate preset switching, layer in effects, post processing and a whole bunch more. Render the final output in high definition and high frame rates using ffmpeg.

Built on projectM and Demucs amongst others. Developed on WSL2; Linux with a display is the current checkout setup. A Windows zip is documented under Windows zip.

Requirements

  • Python 3.10+
  • FFmpeg
  • libprojectM 4.2+ (needs _opengl_render_frame_fbo and _set_frame_time)
  • Optional: NVIDIA GPU + CUDA for faster Demucs separation
  • WSL2: wsl-builds simplifies deps...
./wsl-stacker.sh spoddycoder dev-ai
./wsl-builder.sh media ffmpeg,libprojectm

WSL2 Audio

If you are experiencing audio glitches try disabling systemd-timesyncd. microsoft/wslg#1257.

WSL2 OpenGL + GPU

If the visual editor feels sluggish and CPU-bound, check that Mesa is using the GPU (not software llvmpipe)...

glxinfo | grep "OpenGL renderer"
# bad:  llvmpipe
# good: D3D12

# if you see llvmpipe, force Mesa's D3D12 driver...
export GALLIUM_DRIVER=d3d12

microsoft/wslg#1332

Setup

Create a virtual environment...

# using venv
python3 -m venv cleave
source cleave/bin/activate

# or using conda
conda create -n cleave python=3.10
conda activate cleave

Install dependencies...

# CUDA 13.0 (Linux + NVIDIA GPU)
pip install -r requirements-torch-cu130.txt
# or CPU-only
pip install -r requirements-torch-cpu.txt
# rest of deps
pip install -r requirements.txt
# for development and tests
pip install -r requirements-dev.txt

Releases

GitHub Releases are tagged source archives (vX.Y.Z) for Linux/WSL. Pick a tag, unpack it, and follow Setup above (plus Get Some Milkdrop Presets). System deps (Python 3.10+, FFmpeg, libprojectM 4.2+) are not bundled. cleave --version reports the running version.

The next tagged release attaches a Windows x64 zip (cleave-<version>-windows-x64.zip) and installer (cleave-<version>-windows-x64-setup.exe) as well. v0.1.0 is source-only. Until a tag is cut, testers can download both from Actions via workflow_dispatch on .github/workflows/windows-freeze.yml (5-day artifacts).

main is the integration trunk. User-visible work lands there and adds a bullet under [Unreleased] in CHANGELOG.md in the same change. cleave.__version__ stays at the last shipped tag until a milestone. Then Unreleased becomes a dated X.Y.Z section, the version is bumped, and vX.Y.Z is tagged from main. No calendar cadence. Do not collect work on a long-lived release or develop branch.

How to cut a tag: docs/structured-releases.md.

Windows zip and installer

The installer (cleave-<version>-windows-x64-setup.exe) defaults to Program Files\Cleave. Uninstall removes only the program folder; it does not delete Documents\cleave\ or %APPDATA%\cleave\. The zip (cleave-<version>-windows-x64.zip) is the same onedir tree without an installer: unpack and run from that folder.

Drop a wav onto cleave.exe, or run cleave.exe play <wav>. First run downloads Demucs and Beat This weights into Documents\cleave\models and shows named progress in the loading window. CPU separate is slow. The installer can download extra CUDA files from PyTorch (about 2 GB) for faster stem splitting when it detects an NVIDIA GPU (driver 580.88 or newer). Default answer is No. Skip or fail leaves CPU split. Silent setup does not download CUDA unless you pass /CUDA=1 (that switch still downloads if no NVIDIA GPU is present). The zip stays CPU-only.

cleave.exe play <wav>
cleave.exe play <project>
cleave.exe render <project>

The build is unsigned. If SmartScreen warns on the setup exe or the zip, choose Run anyway.

User data lives in Documents\cleave\ (same projects/, presets/, textures/ tree as Linux). Settings: %APPDATA%\cleave\config.yaml. Override the data root with CLEAVE_DATA.

Milkdrop presets still go under Documents\cleave\presets (see Get Some Milkdrop Presets; use that path instead of ~/.local/share/cleave/).

Maintainers: docs/windows-freeze.md.

Quick Start

Get Some Milkdrop Presets

Clone or place Milkdrop preset packs in the default location...

mkdir -p ~/.local/share/cleave/presets
cd ~/.local/share/cleave/presets
git clone https://github.com/projectM-visualizer/presets-cream-of-the-crop
git clone https://github.com/projectM-visualizer/presets-milkdrop-original

mkdir -p ~/.local/share/cleave/textures
cd ~/.local/share/cleave/textures
git clone https://github.com/projectM-visualizer/presets-milkdrop-texture-pack

Note: There are many thousands of Milkdrop presets available, these are just a few of the best.

cleave a track

./cleave.py play ~/music/mysong.wav

This will separate the track into its component stem tracks (bass, drums, vocals, other), perform some audio analysis, then open the editor.

Using cleave

CLI

./cleave.py --help

Available commands...

  • play play song in the editor, accepts a source audio file or project slug/path.
  • separate can be run on its own without opening the editor.
  • render accepts a project slug or path (not a source audio file).
  • backup archives a full project directory (mix, stems, configs, renders etc.) to a .cleave-tar.gz file.
  • restore unpacks a .cleave-tar.gz archive into projects/<slug>/ (slug from project.yaml).

Note: use --help on any command for options.

Editor

  • Press h to show context sensitive help and controls.
    • The help will change as you move around the interface with the arrow keys.
  • If you're using CPU to render, the editor may run at low frame rates with multiple layers
    • Change Editor Settings -> preview quality to help with this.
    • The final render will still be at full quality and full frame rate.

Preset curation

The editor has a preset curation mode that provides a simple way to categorise presets into folders for future use. Editor Settings -> editor mode -> preset curation. While focused on a preset file row:

  • f - copy the preset into favourites/ (original stays in the pack).
  • c - copy the preset into a cast role directory (see below for more info on cast roles)
  • b - move the preset into blacklist/ (permanently removed from pack).
  • All preset folders live under paths.preset_root (default ~/.local/share/cleave/presets/).
    • Browse them in the Editor like any other pack folder.
  • Subdirectories inside favourites/ or blacklist/ appear as destination choices in the confirm modal, which makes it easy to categorise them in any way you choose, eg:
favourites/a-tier/
favourites/b-tier/
favourites/lots-of-black/
favourites/full-colour-wash/

Timeline layer automation

The timeline provides a comprehensive way to automate layer visibility, preset choice, opacity and blend. When enabled, the standard layer visibility controls are disabled, the timeline takes over.

Song Markers
  • Provide a simple way to mark specific points in the song for preset cuts.
  • Press Ctrl + Enter to drop a song marker at any point in the song.
  • These can be used as snap points and are also used by the timeline preset as anchor points for generation.
Beat / Bar Grid
  • Powered by Beat This! an AI beat detection library.
  • By default it will use the full-mix stem for analysis.
    • Choose a different stem with the --beat-detection-stem switch.
  • You can snap cues to the grid either on record or after record.
Timeline Presets
  • This makes it easy to generate a complete layered visualisation of a song.
  • For best results you should curate presets into Roles.
  • If song markers are available, they will be used to drive the preset generation (do this for best results).
  • There are multiple song marker types:
    • - standard song marker, no special behaviour.
    • crescendo - used to denote where the visual intensity should build t, before crashing off to low intensity.
    • dimininuendo - used to denote where the visual intensity should reduce to, before returning to normal intensity.
    • begin - used to denote where crescendo or dimininuendo ramp should begin.
    • sustain - used to denote where crescendo or dimininuendo should hit maximum / minimum intensity.
CRESCENDO:   thin ↗↗↗ FULL ──── FULL ──── ► solo
             begin   sustain         crescendo

DIMINUENDO:  FULL ↘↘↘ thin ──── thin ──── ► restore
             begin   sustain         diminuendo

Project Directory

cleave creates a new directory under ~/.local/share/cleave/projects/ for each song. The project directory stores all files required in a self-contained bundle...

  • project.yaml - project metadata
  • cleave-viz.yaml - editor & final render configuration. Not everything in here is surfaced in the editor UI just yet
  • signals.json - audio analysis data (schema version 4) used by cleave effects and the opt-in timeline preset conductor; re-run separate on existing projects after a schema bump so envelopes stay current
  • mysong.wav - original source audio is copied into the project directory
  • stems/ - separated audio stems
  • renders/ - final output renders
  • presets/ - presets used by the project are copied into the project directory

Configuration

Cleave uses OS user-data directories. Default locations...

Linux (XDG):

  • User data: ~/.local/share/cleave/
    • projects/
    • presets/
    • presets/favourites/
    • presets/blacklist/
  • Configuration: ~/.config/cleave/config.yaml
    • Editor settings (preview quality, panel width, fade)

Windows (checkout or the Windows zip):

  • User data: Documents\cleave\ (same projects/, presets/, textures/ tree)
  • Configuration: %APPDATA%\cleave\config.yaml

On every OS:

  • Override the data root with CLEAVE_DATA (e.g. CLEAVE_DATA=. for a dev checkout).
  • When a project omits paths, preset browsing defaults to <data root>/presets.
    • paths.preset_root in cleave-viz.yaml overrides this default when set.

Windows freeze layout and FFmpeg/libprojectM sidecars: docs/windows-freeze.md.

Additional Details

Compositing

  • The editor supports up to eight libprojectM layers at tiered resolutions
  • Composited to 1280x720 content by default (editable cleave-viz.yaml)
  • Live preview upscales via editor.upscale and runs at display frame rate
  • Offline render output resolution is set under render.width / render.height (default 1280x720) and frame rate under render.fps
  • Each layer's libprojectM instance receives PCM from its assigned stem; stereo stems are fed as stereo, mono as mono.
  • Milkdrop draws on black, so cleave treats black as transparent and uses pixel brightness as blend weight (black-key default).

Cleave effects

Signal-driven compositor modifiers on top of each layer. Tune depths (0-100%).

Stem Effects
Drums pulse, flash, grit
Bass pulse (sub_bass, mid_bass), flash, grit
Vocals pulse, hue (pitch), flash, grit
Other pulse, flash, grit

Render overlay

  • TODO: Document

Post-processing

  • TODO: Document

Attribution

About

Stem-separated music visualizer

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages