A search can only find what's in its index. That sounds obvious, but it decides almost everything about whether SauceMatch finds your clip. The matching can be as careful as I like. If the video was never collected, the answer is nothing found.
this past week I worked on the unglamorous part: getting more into the index, faster.
tldr: on October 6th, the first full day with the new setup, the index took in 2.75 million new images from about 350,000 videos it had never seen before. An average day before that was about 1.3 million images from 145,000 videos.
What was holding it back
Until this week, one desktop PC did everything. It collected new frames, fingerprinted every one of them, rebuilt the index, uploaded the pictures you see in your results and answered searches.
It's also the computer I use for everything else. So whenever I needed it, collecting stopped. On a typical day lately, that was about half the day, and for the other half the index just sat there.
A spare laptop
I had a laptop that wasn't doing anything. Now it does most of the collecting.
It runs day and night, fingerprints everything it collects itself, and hands it all over to the PC every few hours. The PC keeps the parts that have to live in one place: the index, the search and the pictures in your results.
The laptop also works on more sites at the same time than the PC did. Before the switch, collecting brought in about 75,000 images an hour, for about half the day. Now it's about 115,000 an hour, every hour.
The gaps as you can see are gone, the benefit of moving it to a device i don't need to use daily.
Older videos too
Growing faster isn't only about new uploads. On several of the biggest sites, collecting now walks the whole catalogue, back to videos uploaded years ago, not just the newest or most popular pages. The clip you're trying to place might be from 2015. Or a random clip with very few views. That video needs to be in the index too.
Doing it on a small budget
SauceMatch is made by just me, a one person project and the budget is small. No team, no rented GPU servers. The collecting, the fingerprinting, the index and the search all run on one desktop PC and, now, one laptop. The pictures in your results are stored with a storage service, and on most computers your own browser does the fingerprinting for your search, which keeps the load on my side small.
A small budget means working around limits instead of paying them away. One example from this week:
- The index moved from memory onto the drive. It was taking up more of the PC's memory than it could spare, and adding more memory wasn't an option (Not with the crazy RAM prices nowadays). Now the index lives on the drive and the parts a search needs are read from there. When a search has to read from the drive, the lookup takes a second or two instead of a fraction of one.
what it means for you
- A better chance your clip gets found. More of what's out there is in the index, and it's growing more than twice as fast as before.
- Older and random videos, not just this month's uploads.
- More copies of the same video. When a video is on several sites, you get a link to each copy the index has, so one dead link isn't the end of it. More collecting means more of those copies. More copies means higher chances of finding the name you want, or the full clip, or the original. All wins.
- A reason to try again. As i write this, the index holds more than 19.6 million frames from about 2.3 million videos. My first post here, on October 4th, said 15 million. If a search found nothing last week, it's worth another go.
Where this is going
I want SauceMatch to be the go to place for finding where an NSFW picture or video came from, or full clips/names etc. Free, no blurred results, no unlock button, no BS, and good enough that you don't need to check anywhere else.
The index is the biggest part of getting there, and this was one step. Later, the PC gets new sites of its own to collect from, now that the laptop has taken most of the old ones off its hands.