cross-posted from: https://lemmy.ca/post/37011397
The popular open-source VLC video player was demonstrated on the floor of CES 2025 with automatic AI subtitling and translation, generated locally and offline in real time. Parent organization VideoLAN shared a video on Tuesday in which president Jean-Baptiste Kempf shows off the new feature, which uses open-source AI models to generate subtitles for videos in several languages.
Finally, some good fucking AI
I was just thinking, this is exactly what AI should be used for. Pattern recognition, full stop.
Yup, and if it isn’t perfect that is ok as long as it is close enough.
Like getting name spellings wrong or mixing homophones is fine because it isn’t trying to be factually accurate.
I’d like to see this fix the most annoying part about subtitles, timing. find transcript/any subs on the Internet and have the AI align it with the audio properly.
YES! I can’t stand when subtitles are misaligned to the video. If this AI tool could help with that, it would be super useful.
What’s important is that this is running on your machine locally, offline, without any cloud services. It runs directly inside the executable
YES, thank you JB
This sounds like a great thing for deaf people and just in general, but I don’t think AI will ever replace anime fansub makers who have no problem throwing a wall of text on screen for a split second just to explain an obscure untranslatable pun.
Bless those subbers. I love those walls of text.
Translator’s note: keikaku means plan
They are like the * in any Terry Pratchett (GNU) novel, sometimes a funny joke can have a little more spice added to make it even funnier
It’s unlikely to even replace good subtitles, fan or not. It’s just a nice thing to have for a lot of content though.
Now I want some AR glasses that display subtitles above someone’s head when they talk à la Cyberpunk that also auto-translates. Of course, it has to be done entirely locally.
As long as the models are OpenSource I have no complains
And the data stays local.
As vlc is open source, can we expect this technology to also be available for, say, jellyfin, so that I can for once and for all have subtitles.done right?
Edit: I think it’s great that vlc has this, but this sounds like something many other apps could benefit from
It’s already available for anyone to use. https://github.com/openai/whisper
They’re using OpenAI’s Whisper model for this: https://code.videolan.org/videolan/vlc/-/merge_requests/5155
Has there been any estimated minimal system requirements for this yet, since it runs locally?
It’s actually using whisper.cpp
From the README:
Memory usage Model Disk Mem tiny 75 MiB ~273 MB base 142 MiB ~388 MB small 466 MiB ~852 MB medium 1.5 GiB ~2.1 GB large 2.9 GiB ~3.9 GiB
Those are the model sizes
Oh wow those pretty tiny memory requirements for a decent modern system! That’s actually very impressive! :D
Many people can probably even run this on older media servers or even just a plain NAS! That’s awesome! :D
deleted by creator
That explains why their subtitles have seemed worse to me lately. Every now and then I see something obviously wrong and wonder how it got by anyone who looked at it. Now I know why. No one looked at it.
deleted by creator
Malevolent Kitchen Intensifies
Ooooh I like this
This might be one of the few times I’ve seen AI being useful and not just slapped on something for marketing purposes.
And not to do evil shit
But the toppings contains potassium benzoate.
The nice thing is, now at least this can be used with live tv from other countries and languages.
Think you want to watch Japanese tv or Korean channels with out bothering about downloading, searching and syncing subtitles
The technology is nowhere near being good though. On synthetic tests, on the data it was trained and tweeked on, maybe, I don’t know.
I corun an event when we invite speakers from all over the world, and we tried every way to generate subtitles, all of them run on the level of YouTube autogenerated ones. It’s better than nothing, but you can’t rely on it really.No, but I think it would be super helpful to synchronize subtitles that are not aligned to the video.
This is already trivial. Bazarr has been doing it for all my subtitles for almost a decade.
And yet they turned down having thumbnails for seeking because it would be too resource intensive. 😐
I mean, it would. For example Jellyfin implements it, but it does so by extracting the pictures ahead of time and saving them. It takes days to do this for my library.
deleted by creator
I get what you are saying, but I don’t think there is any standardized format for these trickplay images. The same images from Plex would likely not be usable in Jellyfin without converting the metadata (e.g. to which time in the video an image belongs to). So VLC probably does not have a good way to understand trickplay images not made by VLC.
Video decoding is resource intensive. We’re used to it, we have hardware acceleration for some of it, but spewing something around 52 million pixels every second from a highly compressed data source is not cheap. I’m not sure how both compare, but small LLM models are not that costly to run if you don’t factor their creation in.
All they’d need to do is generate thumbnails for every period on video load. Make that period adjustable. Might take a few extra seconds to load a video. Make it off by default if they’re worried about the performance hit.
There are other desktop video players that make this work.
It is useful for internet streams though, not really for local or lan video.
Hes maaaa, hes what? Hes maaaaaa!
