TaleCraft

branching stories + voice + hand gestures, because tapping choices was apparently too normal.

An interactive storytelling experiment built around stories that can branch, speak, listen, and react to what the reader does.

Demo · Engineering notes · Gesture service

Flutter · GetX · Firebase · OpenAI · MediaPipe · Python

---

the idea

I wanted to see what happens when a story stops behaving like a page of text.

TaleCraft started with branching narratives, then kept collecting side quests: a visual storyboard editor, AI-generated story trees, text-to-speech, voice-controlled choices, saved progress, achievements, recommendations, and finally a camera mode where you can pick a branch with your hand.

The result is less “ebook reader” and more “small interaction playground disguised as a storytelling app.”

what you can do

🌲 Read stories that actually branch

Stories are represented as a tree of two node types:

story
├── choice 1 → story → ...
├── choice 2 → story → ...
└── choice 3 → story → ...

Each choice changes the route through the story, and different branches can end in different outcomes. Progress is stored so a reader can continue a route later instead of starting again.

🧩 Build stories visually

TaleCraft includes a storyboard-style creation flow for building story and choice blocks instead of writing one giant linear document.

Under the hood those blocks are serialized into nested JSON, then reconstructed into a graph when the reader opens the story.

✨ Generate a branching story from a prompt

The AI generation experiment asks a language model to return the same story → choice → story JSON structure the app already understands.

That means generated content does not need a completely separate reader. Once the JSON is parsed, it becomes the same graph as a manually authored story.

The public repo no longer contains an API key. For local experimentation the key is supplied with --dart-define. A real deployed app should put model calls behind a server rather than shipping provider credentials inside a mobile client.

🔊 Let the story read itself

flutter_tts turns narrative and choice text into an audio flow. When the narration reaches a decision, TaleCraft can wait for the reader instead of automatically continuing.

🎙️ Pick a choice with your voice

The listening mode uses speech recognition to match spoken choice numbers to the currently available branches.

So instead of pressing choice 2, you can just say it.

✋ Or pick it with your hand

This is the unnecessarily fun part.

The mobile app captures a camera frame, sends it to the separate Python gesture service, and uses the recognised numeric hand gesture as the selected branch.

camera frame
   ↓
base64 image
   ↓
Flask API
   ↓
OpenCV decode
   ↓
MediaPipe GestureRecognizer
   ↓
"1" / "2" / "3" ...
   ↓
confirm choice in Flutter
   ↓
continue story branch

The computer-vision service lives here: talecraft-gesture-recognition

🏆 Remember the route

TaleCraft stores reading progress and picked choices through Firebase. The reader can resume a previous path, complete a story, rate it, and unlock a special ending/achievement when a route reaches the configured achievement ending.

demo

how the pieces fit together

flowchart LR
    U[Reader] --> F[Flutter app]
    F --> G[Story graph]
    G --> C[Choice]
    C --> G

    F --> FB[Firebase]
    FB --> A[Auth]
    FB --> S[Stories / progress / ratings]
    FB --> ST[Storage]

    F --> O[Story generation API]
    O --> J[Branching JSON]
    J --> G

    F --> TTS[Text to speech]
    F --> STT[Speech to text]

    F --> CAM[Camera]
    CAM --> API[Gesture Flask API]
    API --> MP[MediaPipe]
    MP --> C

Loading

The detailed version is in docs/engineering.md.

project structure

lib/
├── controller/      # screen + interaction controllers
├── model/           # story, block, reader and saved-progress models
├── repository/      # Firebase-facing repositories
├── services/
│   ├── api/         # external API calls, including story generation
│   └── network/     # Dio client setup
├── view/            # Flutter screens
├── viewModel/       # API-facing view models
└── utils/           # styling, strings and reusable helpers

The architecture reflects what this project was: a fast-moving experiment where different interaction ideas were added one by one. It is not pretending to be a pristine modern starter template.

running it locally

Install Flutter dependencies:

flutter pub get

The app uses Firebase, so a valid Firebase configuration is required for auth, Firestore and Storage features.

For the AI story-generation experiment:

flutter run --dart-define=OPENAI_API_KEY=your_key

For gesture mode, run the companion service and point the app at a reachable /process_image endpoint. The original prototype used a LAN address because the Flask service was running on another machine/device during development.

Then:

flutter run

stack

Area	What I used
Mobile	Flutter / Dart
App state & navigation	GetX
Story graph	flow_graph + custom Block model
Auth / data / files	Firebase Auth, Cloud Firestore, Firebase Storage
AI story generation	OpenAI chat-completions experiment
Audio	flutter_tts
Voice input	speech_to_text
Camera	Flutter camera
Gesture recognition	Python, Flask, OpenCV, MediaPipe
Networking	Dio + HTTP

why this repo is still here

TaleCraft is an experiment, not a current production product.

I keep it public because it is a good snapshot of the kind of project I like building: start with one normal idea, then keep asking “what if the app could also do this?” until a story reader somehow ends up with AI, speech recognition and computer vision in it.

It also pairs nicely with the standalone gesture-recognition service, which shows the computer-vision side without making people dig through the Flutter app.

---

built by Senith Umesha · linkedin · portfolio