I Tried Dubbing a Video with OmniVoice (and Ended Up Fixing a Bug)

This time I tried OmniVoice, an open-source dubbing tool.

I ran it on my MacBook Air, and it installed way more easily than the other open-source tools I’ve tried. That might sound like a small thing, but it’s a big deal to me. Installing these tools is honestly pretty hard for me, so just getting one running is already half the battle.

Here’s the clip I wanted to dub, a short Korean piece, before OmniVoice touched it:

Original: the Korean clip I wanted to dub into English

Once I had it running, I started with English to Korean. That worked fine. So I flipped it around and tried Korean to English, and this time every single sentence threw an error. I ran it again and got the same thing. At first I figured it was because I had too much else open at once, so I closed everything down and tried again. Still failed.

That’s when I opened the log. Thing is, I’m not a developer, so even staring right at the log I couldn’t tell what was wrong. So I showed it to Claude and asked what was going on and how I could fix it.

Here’s what we worked out together. The tool has a step that listens to the audio and figures out what language it is, and for my video it got it right. The log even said it had detected Korean with 98% confidence. So it clearly knew. But by the time things got to the translation step, that info was gone. Since it couldn’t find the original language, the code fell back on what it does when it has no idea: just assume English. So it tried to translate English into English, and since there’s no such thing as an English to English translation pack, every sentence died with the same error. It knew the video was Korean, then forgot one step later.

Once we got that, Claude and I went into the code and found the exact spot where it falls back to English. We added a little backup plan there. If it doesn’t know the language, just look at the words in the subtitles and figure out which language it is. (Korean characters mean Korean, and it’s the same idea for Japanese, Chinese, and a few others.)

One thing did drive me a little nuts. Every time I edited the code and reopened the app, my fix was just gone. It took me a while to realize the app rewrites its own code from scratch every time it launches, so I’d been editing the wrong copy the whole time. Once I found the right place to put the fix, it finally stuck.

After that I ran it again, and Korean to English worked.

I figured other people probably hit the same trouble I did, so I posted it on GitHub as both an issue and a pull request. I only opened the pull request on the off chance it might actually get accepted. And OmniVoice took my fix in! I’m not a developer, just someone who likes AI dubbing, and getting to add something to someone else’s code made me really happy.

You can see it on GitHub: the issue I opened and the pull request that got merged.

Here’s how it turned out:

OmniVoice: dubbed Korean → English, with the original voice cloned

I’ll be honest, this still isn’t a result I’m happy with. If you listen closely, you can hear a bit of Korean leaking in at the start of some sentences. But the voice cloning is really good. I’ve tried a few open-source dubbing tools now, and I haven’t seen one match the original speaker’s voice this well. I’ll show you how I’m trying to get rid of that leak in the next post.