1MB Club

Purging the 1MB Club

Mon, 24 Jul 2023

This project has been running for almost 3 years now, and I still enjoy adding more members to the club! The only issue is that I have never been great at ensuring the quality of the members remained consistent. This included removing dead links, sites that now forward to less-than-safe domains, etc.

That all changes today!

The First Purge

I wrote up a very crude script that checks the status of each existing club member’s URL and flags those that error-out. After running the check I found I could remove over 60 websites that were either dead or broken.

Note: If you happen to notice your website has been incorrectly caught in the “purge-crossfire”, then don’t hesitate to shoot me an email letting me know! I’m only human after all.

Checking URL Script

This ruby script is far from perfect but it works well for my own personal workflow. Feel free to steal this and tweak it for your own purposes as you see fit!

require 'httparty'
require 'nokogiri'

HTTParty::Basement.default_options.update(verify: false)

response = HTTParty.get('https://1mb.club')

document = Nokogiri::HTML(response.body)
website_urls = document.css("#container tr")

puts "Scanning website members URLs..."
website_urls.each do |single_site|
    begin
        url = single_site.css("a.site").first.attribute("href").value
        response = HTTParty.get(url, timeout: 4)
        puts "Checking: " + url
    rescue Exception
        puts "<!-------- ERROR: " + url
    end
end

1MB Club

Purging the 1MB Club

The First Purge

Checking URL Script

More Posts to Come

Special Thanks