Now you can dub video captions & fix lip movement using AI into 28 languages! Here’s how

Captions, a groundbreaking AI-driven video editing startup, has unveiled its latest creation, the Lipdub app. This free app is set to revolutionise the way we experience videos by making them accessible to a global audience.

Video startup Captions launches new AI dubbing app Lipdub with Gen Z slang https://t.co/GD3dfkOtCY pic.twitter.com/z2vONgEByK

advertisement

— Oakbloom (@Oakbloom) October 10, 2023

With the power of AI, Lipdub can seamlessly translate video clips into 28 different languages, including widely spoken ones like French, Hindi, Spanish, Italian, and Portuguese. Moreover, it caters to niche audiences by providing translation options into Texas slang, Gen Z lingo, pirate-speak, and even baby talk.

One of the most intriguing aspects of Lipdub is its ability to adjust lip movements according to the selected target language. This innovation bridges the gap between languages, making video content more understandable and relatable to a diverse audience. While Lipdub offers an exciting array of possibilities, users may encounter minor synchronisation issues between audio and lip movements.

Easily share videos on social media

The functionality of Lipdub allows users to translate video clips featuring a single person talking for up to one minute. Once the translation is complete, users can effortlessly share their multilingual videos on various social media platforms.

Captions, the mastermind behind Lipdub, has garnered significant attention in the world of video editing. With over three million creators using their video editing app, and a whopping 100,000 daily users, Captions has established itself as a trailblazer in the field. Their app boasts several AI-powered features designed to elevate video quality.

These features include eliminating speech fillers like ‘ums’ and ‘ahs,’ reducing background noise, and enhancing speech clarity.

Additionally, the ‘AI Lipdub’ feature empowers users to fine-tune lip movements during post-production by simply modifying the transcript.

Remarkable journey and promising future

Captions, founded in 2021 by Gaurav Misra, who formerly served as the head of design engineering at Snap, has been on a remarkable journey. In June, the company secured a substantial $25 million in a Series B funding round led by Kleiner Perkins.

The round also saw participation from prestigious names like Sequoia Capital, Andreessen Horowitz (a16z), and SV Angel. So far, Captions has successfully raised a remarkable $40 million in funding, signaling the high demand and potential of AI-powered video editing.

The launch of Lipdub comes at a time when using technology for translation and AI-dubbing is gaining momentum. In June, YouTube unveiled plans to test an AI-powered tool allowing users to automatically dub their videos into multiple languages. YouTube is also working on refining lip-syncing for a seamless experience. Last month, the company revealed its integration of AI-powered dubbing directly into YouTube Studio, streamlining the process of converting videos into different languages.

Recently, ElevenLabs, an AI-powered voice-generating platform, launched its own dubbing tool, supporting up to 29 languages. These developments demonstrate the growing trend of using AI for dubbing and translation, with companies generating substantial revenue by catering to popular YouTubers and content creators.

The rise of AI-powered dubbing startups has piqued the interest of investors, with companies like U.K.-based Papercup and Israel-based Deepdub successfully raising millions of dollars. The launch of Lipdub by Captions adds another exciting chapter to the ever-evolving landscape of AI-driven video editing and translation technologies.