archives

Idol Social Media Archiving Project

A couple weeks late again but archives have been updated. Here's a list of things that were done since the last update.

  • Instagram downloading has also been switched from ripme to gallery-dl.
  • The Tiktok downloader is broken again. Downloads will resume when and if that is fixed.
  • Removed duplicate images with jdupes

Downloads

Torrent Size: 323GB

Magnet: magnet:?xt=urn:btih:0fc5915d6b7a9124e76bb021ef2640972d4fc727&dn=Idol-Archive&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce

Todo

  1. Add old tiktok videos scraped with youtube-dl.
  2. Organize miscellaneous profiles and include those in the batch. This one will have to be worked on over time as I want to organize each profile in their respective group. I also need to consolidate these profiles saved with ripme with the files currently scraped with gallery-dl.
  3. Merge Tina Monoga & Tina Kudou (New profile name)

Next idols

  1. Oh! My Kitty
  2. Si4/Si Stella
  3. JumpingKiss

Cheerz.cz profiles are now added to the archive. A downloader has been made allowing for easy scraping directly on the seedbox.

This update comes a few weeks late but the project was becoming too manual so needed to make time to fix my workflow. The original script being used to scrape and move files on the seedbox was poorly written and inefficient. My goal is to do as little manual intervention as possible with these archives. Future archives should run automatically while scraping profiles, packing into 7z archives, transferring to the correct artist directories, and staging for the final torrent. Apart from some of the organization I've been putting off with old archives, the only manual work now is maintaining the profile lists for deleted or renamed accounts.

Now that most of this process is automatic I'll have more time to focus on the to-dos. I've made some progress with #2 on the todo list but still need to finish merging tiktok videos. A list of hundreds of profiles was posted at littleidols.net and a lot of those idols are unknown to me for #3 in the to-do list. They'll most likely stay unorganized due to lack of time but I do want to make sure I'm not archiving duplicates between my list and the one from littleidols.

Downloads

Torrent Size: 326GB

Magnet: magnet:?xt=urn:btih:04274fff45f26cf4eca748d0cc058ef269ee3814&dn=Idol-Archive&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce

Added

  1. Metropolis
  2. Crayons
  3. Whiterabbits

Todo

  1. Find out if gallery-dl can append posting date to files like ripme does.
  2. Add old tiktok videos scraped with youtube-dl.
  3. Organize miscellaneous profiles and include those in the batch. This one will have to be worked on over time as I want to organize each profile in their respective group. I also need to consolidate these profiles saved with ripme with the files currently scraped with gallery-dl.

Next idols

  1. Oh! My Kitty
  2. Si4/Si Stella
  3. JumpingKiss

The initial release of tiktok was added to this release. The tool being used is unable to scrape accounts with generic tiktok generated usernames so those will be added when the tool is fixed or users add a username. Many of the profiles were scraped with youtube-dl when it supported tiktok. Those videos still need to be added.

Downloads

Torrent Size: 310GB

Magnet: magnet:?xt=urn:btih:adac58b14c6e83cccc16322c44ee0b02ce423900&dn=Idol-Archive&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce

Added

  1. Devil Anthem
  2. TikTok Profiles

Todo

  1. Find out if gallery-dl can append posting date to files like ripme does.
  2. Add old tiktok videos scraped with youtube-dl.
  3. Organize miscellaneous profiles and include those in the batch. This one will have to be worked on over time as I want to organize each profile in their respective group. I also need to consolidate these profiles saved with ripme with the files currently scraped with gallery-dl.

Next idols

  1. Oh! My Kitty
  2. Si4/Si Stella
  3. Metropolis

The archiving script needs to be redone or fixed. After taking a closer look I noticed quite a few profiles were not being archived and transferred so this week required some manual work when organizing archives into the corresponding groups. TikTok profiles have been downloaded but still need to be added. That should be coming in the next release.

Downloads

Torrent Size: 305GB

Magnet: magnet:?xt=urn:btih:29cd28ff71d399c9f904dc6ad1b0b3bb2d035991&dn=Idol-Archive&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce

Added

  1. Niji no Fantasista
  2. TIP Photosessions

Todo

  1. Find out if gallery-dl can append posting date to files like ripme does.
  2. Add already downloaded TikTok profiles to the archiving script and merge with other files. TikTok files have been been transferred but files need to be consolidated from scrapes ran on other computers with the ones currently running. Youtube-dl is no longer supporting any fixes to the TikTok extractor so there will most likely be duplicates since switching tools to download TikTok videos.
  3. Organize miscellaneous profiles and include those in the batch. This one will have to be worked on over time as I want to organize each profile in their respective group. I also need to consolidate these profiles saved with ripme with the files currently scraped with gallery-dl.
  4. Remove archived profiles from script as profiles no longer exist. Some profiles have been archived but I still need to remove from the lists and archiving script.

Next idols

  1. Devil Anthem

There is still a lot of organization to be done cleaning up profile lists, profile scrapes on multiple computers, and removing deleted profiles from the script.

I'm still slowly switching over from ripme to gallery-dl. The experience has been more consistent and gallery-dl's skip option works compared to ripme's history.end config option, resulting in faster jobs with less 429 errors. I still need to take a closer look at gallery-dl's config for naming as I still like ripme's default with post date for instagram.

Downloads

Torrent Size: 277GB

Magnet: magnet:?xt=urn:btih:ea62ec3a35266c3699c7c0256f604d21eed4bcde&dn=Idol-Archive&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce

Added

  1. Gugu-LuLu
  2. Little Blossom & SisterS
  3. Niji no Conquistador

Todo

  1. Find out if gallery-dl can append posting date to files like ripme does.
  2. Add already downloaded TikTok profiles to the archiving script and merge with other files. TikTok files have been been transferred but files need to be consolidated from scrapes ran on other computers with the ones currently running. Youtube-dl is no longer supporting any fixes to the TikTok extractor so there will most likely be duplicates since switching tools to download TikTok videos.
  3. Organize miscellaneous profiles and include those in the batch. This one will have to be worked on over time as I want to organize each profile in their respective group. I also need to consolidate these profiles saved with ripme with the files currently scraped with gallery-dl.
  4. Remove archived profiles from script as profiles no longer exist. Some profiles have been archived but I still need to remove from the lists and archiving script.

Next idols

  1. Fantasista (Niji no Conquistador sub group)

This archive contains images from accounts I've personally followed as well as accounts scraped from littleidols.net. There are roughly 400,000+ photos and videos included totaling to around 250gb. This is the first public batch offered through torrent.

Downloads

Torrent

Magnet: magnet:?xt=urn:btih:f45bff599d68c27a7c326dfdd0fe6c7547f25611&dn=Idol-Archive&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce

Todo

  1. Add already downloaded TikTok profiles to the archiving script and merge with other files.
  2. Organize miscellaneous profiles and include those in batch.
  3. Remove archived profiles from script as profiles no longer exist.

Next idols

  1. Add Little Blossom and GuGu-LuLu.