r/canadaguns • u/KorporalKarnage • Jul 31 '26

Application / Regulation advice Guns.Rip and GunsOnline.ca

A where-to-shoot map and an aggregator.

Have fun!

Guns.Rip

and

GunsOnline.ca

I'm still entering gun ranges and no-shoot zones. I made an automated pin drop feature to edit/add crown sites/ranges and pics on the map.

The aggregator took a long time to train, but still might have the odd listing straggler. I'm making a dedicated CRON box to soft scrape all known sites. I made a built-in FRT checker (why not) so you can get more info on the gun you're looking at. The FRT is checked daily and I posted a searchable page for it. I might be adding pictures to it eventually.

I originally made these sites for myself so keep that in mind if you have "complaints". Suggestions and feedback are welcome!

39 Upvotes

92% Upvoted

31 comments sorted by

View all comments

Show parent comments

2

u/KorporalKarnage Jul 31 '26

Proxy scraper now online. Should show over 13k items. Changed my scraper script to fall back on softer methods if kicked. Still working on more proxies...

2

u/_Armpit AlmostProhibited.ca Aug 01 '26

With the exception of Shopify and Reliable, most retailers' host don't care if you are making a lot of requests from a single IP address/identity.

I crawl most of my retailers from a single US datacenter IP and it's fine, you just have to watch those select few that display pricing in USD if you do that.

I end up crawling roughly 130k products every single day and it takes 5 hours to finish. I can go faster, but that's not nice bot behaviour for retailers.

1

u/KorporalKarnage Aug 02 '26

I've been having a blast trying different ways getting around all these anti bot/scraper blocks these sites put up. I had no idea there were so many ways they put up roadblocks but given time, they all fall down.

So my setup as of now is dedicated RPi running cron jobs for all the different tasks (and there are many). I also abandoned the porto-proxy app, although it did work, it just isn't needed with the setup as of today.

Basically it goes PC (if on)->Rpi->Proxy server 1->if problems->fallback proxy server 2-> Rpi->deploy

I upped my limits to 5k per for now and let er run....

1

u/KorporalKarnage Aug 02 '26

Oh yeah, BassPro is NOT worth the effort! Holy....