Best AI Video Upscalers?

[Colon capital V]

Found a few old family videos, DVDs and miscellaneous youtube videos I'd like to upscale for me and my old folks, what's the current best upscalers I can use that's free and/or doesn't take 70 years to process?

 

[Soggy Floppa]

https://openmodeldb.info/

 

[Kaiba Cuqk63]

If you go the route of using an open source model with python or something, you might have a better time paying a few $ to run it with a better GPU on Google collab or runpod.

 

[AnsemSoD1]

Beep boop, I'm a robot.

 

[TheOtherSideofMakeBelieve]

I've been reasonably happy with winxvideo AI but it's only free for videos up to five minutes long so it probably won't do what you're wanting. $40 for a lifetime license (currently on sale) to do things as long as you want, but you can do a five minute sample to see if you like it.

Might be too slow for you, upscaling/interpolating to 4k/120 takes about 20 minutes/minute on a 1080ti/8core Zen 2 cpu/32 gigs of ram and my pc is unusable during that time. It also blows up file sizes but presumably handbrake could fix that if I ever need it

Actually, poking around with everything it does it actually has some really nice features. As a free user you can download videos from youtube and other sites (not sure of all the ones that are supported) with it, and it also has re-encoding that seems to do an excellent job, it took a clip from 400mb for 4k/120fps to 4k60fps at 29MB with an hevc encode, at the second highest quality setting it was reencoding at about 40 frames/second. Watching the clips side by side I thought it washed the colors out a bit but then I swapped which side of my monitor each one was on the original looked to be washed out so it was just my lighting/physical position in front of my curved monitor. The only real difference I could tell was the framerate

If you have a sample youtube video you want to see I could render out a minute of it to whatever settings you want. It just updated to 3.0 when I opened it to check all the features I haven't used so it's possible it's faster now, I haven't tried it aside from downloading a youtube video to see how it handled it, which seemed to be pretty well

E: Found a video I thought was worth experimenting with. The upscaling seems slower but my computer is usable for basic stuff, currently browsing the farms and watching today's MatI on Rumble. Not doing anything major, no interpolation, just a simple upscale from a 720p/60 download to 1080/60 on a video that already looked pretty bad, taking about an hour for a 1m40 video. When it finishes I'll re-encode it to H265 HEVC/30 fps, and then just do a basic re-encoded upscale from the original. Close to bedtime though so I'll probably have to post them tomorrow after work

E2: Render finished but the settings I used ended up as an unholy abomination lol. The upscaling model I used looked good in the preview stills but in motion it does not work. Might try again tomorrow but with how long it took I might not, maybe a 10-15 second sample or something

 

[Jeff Q. Anime]

I happen to be something of an expert on this subject, and the answer is: It's very complicated. There are oodles of free and open source upscaling architectures out there with a wide range of tradeoffs between quality and speed, but there's no guarantee that a pre-made model exists that will adequately deal with your particular degradations. The choice of architecture also depends on the hardware you have available and how deeply into the reeds you're willing to get with running more optimized inference software. Most decent open source work is in single image superresolution, not video superresolution. There can be issues with temporal stability depending on the degradation. The open source community in general is much more focused on single image superresolution than video. A lot of video superresolution is either proprietary or open source but difficult to implement for general use.

For IRL video I'd start by pirating a cracked version of Topaz video AI and see how well it performs for you. You can get it clean on rutracker. Otherwise it would probably be more expedient to join the enhance everything discord linked on openmodeldb and post some samples of the type of material you want to enhance.

 

[Post Reply]

I've pirated Topaz before, results were pretty good.

 

[cybertoaster]

I swapped which side of my monitor each one was on the original looked to be washed out so it was just my lighting/physical position in front of my curved monitor.

I thought the point of curved monitors is that it looks the same no matter the angle (if you're in the center that is).

https://openmodeldb.info/

Noice, got a page for that but for other AI models too?

I've pirated Topaz before, results were pretty good.

Guess it improved a lot, I used it like a year and a half ago, ran like shit and the video looked like a cartoon.

 

[Soggy Floppa]

Noice, got a page for that but for other AI models too?

Stable Diffusion (Tits warning): civitai.com
RVC voice models (Just an easier to browse huggingface mirror): https://voice-models.com/
Llama LLM models: https://ollama.com/library (just Ollama compatible ones, so not any of the custom fine tunes or any not converted for Ollama, for those you have to brave huggingface)

The rest are all buried on Huggingface, but good luck finding anything with their terrible UI

 

[Betonhaus]

is there one that can run on an old debian server with a Nvidia Quadro P400?

 

[Jeff Q. Anime]

is there one that can run on an old debian server with a Nvidia Quadro P400?

Probably your only viable options without waiting for the heat death of the universe are SPAN and Compact based models, which you can find on openmodeldb and the enhance everything discord. It depends on what exactly your goals are of course. Processing whole videos is a different beast from a small set of individual images.

 

[Cool Dog Returns]

Got tons of old VHS and .avis but what I seen doesnt look good

I think these niggas should train their models by taking an old VCR and recording a 4K stream to it, then running that video and the original 4K so the model can learn to rebuild it

Can do the same with old cameras, seen retro ytubers recording stuff with old handycams and such, everything looks like straight out of 1995 but people around have smartphones

 

[Betonhaus]

Probably your only viable options without waiting for the heat death of the universe are SPAN and Compact based models, which you can find on openmodeldb and the enhance everything discord. It depends on what exactly your goals are of course. Processing whole videos is a different beast from a small set of individual images.

Oh ok. What about a 36 core Xeon Ubuntu server with an Intel Arc GPU, and has a Windows Server VM allocated with 64gb ram, 12 cores, and a dedicated Nvidia k6000 gpu

 

[Jeff Q. Anime]

Oh ok. What about a 36 core Xeon Ubuntu server with an Intel Arc GPU, and has a Windows Server VM allocated with 64gb ram, 12 cores, and a dedicated Nvidia k6000 gpu

I haven't benchmarked CPU inference personally, but Xeon isn't enough to go on anyway. Like, what CPUs specifically? Do they even have avx512? Do they have VNNI instruction support? If not, CPU literally does not matter. It will never accomplish anything. Even then, if you have a fancy Cascade Lake system with all six channels of max speed 2933MHz memory, that's only 141GB/s per socket. An RTX 2070 has *at least* 448GB/ memory bandwidth, if not more depending on the specific model, and uses half the power, and costs half as much as any decent CL era Xeon CPU too. What you'll find is that you hit a wall with memory speed far before you actually saturate your potential CPU TDP. That's why newer platforms are racing for fuckin' 6000MHz+ DDR5 and getting 12+ memory channels and shit... Needs more GB/s. If you actually have a *new*, like, 2022 or newer platform, and it's ballsed out with the memory, you might be able to do something meaningful, but again I have no benchmark evidence.

I'm not sure about Arc performance, and that depends on the specific GPU as well, but I can tell you for certain that the K6000 is worthless. You need modern architectures with tensor cores or whatever equivalent marketing jargon AMD/Intel use to upscale entire videos at any decent quality model / speed. You don't need much vram to do upscaling. Even a 2060 would rape the k6000. I'm not telling you you need quad 4090s to do upscaling, but you just can't do it without wasting more money on electricity than you would on newer GPUs on anything older than Turing.

Of course, feel free to try it for yourself. This vapoursynth plugin is the most flexible solution available at the moment. https://github.com/AmusementClub/vs-mlrt

I think these niggas should train their models by taking an old VCR and recording a 4K stream to it, then running that video and the original 4K so the model can learn to rebuild it

Feel free to start providing datasets of matched native 4k and VCR translated footage at your earliest convenience. Personally I'm only interested in restoring DVD or bitcrunched stream-only anime.

 

[Cool Dog Returns]

Feel free to start providing datasets of matched native 4k and VCR translated footage at your earliest convenience.

Could get a handycam, rig it with a smartphone and record random shit on 4k and on VHSC at the same time, would be good to upscale home videos I guess?

Most of these upscalers seem to just guess shit up so it all looks weird and like painted or something, idk like I said I dont know shit about how upcalers works except ersagan always messes my AI pics

 

[General Emílio Médici]

Feel free to start providing datasets of matched native 4k and VCR translated footage at your earliest convenience. Personally I'm only interested in restoring DVD or bitcrunched stream-only anime.

It shouldn't be too hard to do that, but you would need some dedicated hardware which isn't common. I think it could be done by just having a screen which can connect to Coaxial and HDMI at the same time (doesn't need to be 4k but that would help) and stream the test footage in 4k to it, and have the VCR record the coaxial onto tape. Then you scan the tape and put the footage alongside the original so the LLM knows what to do.

 

[Cool Dog Returns]

It shouldn't be too hard to do that, but you would need some dedicated hardware which isn't common. I think it could be done by just having a screen which can connect to Coaxial and HDMI at the same time (doesn't need to be 4k but that would help) and stream the test footage in 4k to it, and have the VCR record the coaxial onto tape. Then you scan the tape and put the footage alongside the original so the LLM knows what to do.

Wouldnt be easier to just record the 4k to the vcr with an analog converter separately? its the same file

Another idea: record it again but from the vhs so its really fucked up, that way it can also fix old worn vhs tapes

 

[Jeff Q. Anime]

Most of these upscalers seem to just guess

Welcome to AI. Pick 2: Runs fast, is generally applicable, does a good job. You have to pick somewhere on that triangle and sacrifice something. Anything that runs decently quickly in terms of upscaling will need to be trained on, at the very least, a genre-applicable source material - digital recordings, film recordings, 3d animation, 2d animation, game textures... Whatever. There are architectures that can do everything decently well but they take forever to run. It's important to note that architectures and models are separate things. ESRGAN is just an architecture. There are dozens of good models for ESRGAN that can do all sorts of different things, but ESRGAN as an architecture is also pretty outdated in terms of performance / capability.

Then you scan the tape and put the footage alongside the original so the LLM knows what to do.

Easier said than done. The open source training frameworks don't just take video feeds, they require pairs of images in degraded/ground truth quality. They need to be the exact same frame, perfectly aligned, and all that. You also can't just shove in a whole video's worth of frames, you should be cherry-picking good ones with a high signal to noise ratio and then picking the best patches out of those frames. There are tools to help automate this of course but it's just not as simple as you're saying.

The problem with artificially inducing VHS degradation is that it's analogue. There's no determinism in how VHS tapes are artifacted, and creating a model to do it programmatically in a realistic way is difficult. Simple aging changes the signal integrity, and a whole host of potential problems can exist with syncing the chroma/luma planes and even timing the frames themselves correctly. I'm not an expert on VHS degradation. I can't expand on it much more than that. VHS tapes can be copied, recopied, aged, copied again, digitally copied, re-written to a tape with MPEG artifacts from that digital version, copied again... Etc.

Current open-source algorithms also cannot properly compensate for temporal artifacts, as they are single-image, period.

 

[General Emílio Médici]

The open source training frameworks don't just take video feeds, they require pairs of images in degraded/ground truth quality. They need to be the exact same frame, perfectly aligned, and all that.

Hence the whole streaming the source 4k video to a TV to be picked up by a VCR, that would let you perfectly match the frames.

The second point you mention is the real issue. My idea would work fine for preserved VHS tapes but any slight damages and it stops working. No idea how you would teach the LLM to deal with moldy or degraded signal.

 

[Jeff Q. Anime]

streaming the source 4k video to a TV to be picked up by a VCR, that would let you perfectly match the frames.

you would find that aligning analog video captures in ways that are useful for training is more difficult than you believe

also... they're not LLMs. they don't operate on language. they're convolutional neural networks designed to interpolate pixels.

Rather than creating training material yourself, you would be able to find movies or TV shows old enough to have degraded VHS releases, mastered on film, with modern UHD rescans. Of course the cuts and timing are never perfect and you run into the problem of aligning your data for training. But it's a more realistic option that captures more degradation.