r/canadaguns • u/KorporalKarnage • Jul 31 '26

Application / Regulation advice Guns.Rip and GunsOnline.ca

A where-to-shoot map and an aggregator.

Have fun!

Guns.Rip

and

GunsOnline.ca

I'm still entering gun ranges and no-shoot zones. I made an automated pin drop feature to edit/add crown sites/ranges and pics on the map.

The aggregator took a long time to train, but still might have the odd listing straggler. I'm making a dedicated CRON box to soft scrape all known sites. I made a built-in FRT checker (why not) so you can get more info on the gun you're looking at. The FRT is checked daily and I posted a searchable page for it. I might be adding pictures to it eventually.

I originally made these sites for myself so keep that in mind if you have "complaints". Suggestions and feedback are welcome!

42 Upvotes

96% Upvoted

31 comments sorted by

View all comments

9

u/LumberjackCDN Gold'n Loan Outfitters & Pawn Jul 31 '26

I disapprove, i dont see us on there lol.

2

u/KorporalKarnage Jul 31 '26 edited Jul 31 '26

Heh. Yeah, retailers were out of the scope for my map project. But if you sell online I do have a used/consignment section on the GunsOnline.ca site!

Edit: lol Nevermind! You're already on there! Just needed a better pagination. My bots run different on each retailer, they all scrape different.

2

u/KorporalKarnage Jul 31 '26

One more thing to add, my scraper caps out at 500 items/day. It may take more time to add the rest of your catalog (I think my log shows 5876 items in store).

This is how I programmed my scrapers, do a "light" touch every night so they don't get blocked (or my IP!).

3

u/_Armpit AlmostProhibited.ca Jul 31 '26

If it's 500 a day (per retailer or globally), you're going to be waiting weeks or months to index everything for some stores.

2

u/KorporalKarnage Jul 31 '26

Proxy scraper now online. Should show over 13k items. Changed my scraper script to fall back on softer methods if kicked. Still working on more proxies...

2

u/_Armpit AlmostProhibited.ca Aug 01 '26

With the exception of Shopify and Reliable, most retailers' host don't care if you are making a lot of requests from a single IP address/identity.

I crawl most of my retailers from a single US datacenter IP and it's fine, you just have to watch those select few that display pricing in USD if you do that.

I end up crawling roughly 130k products every single day and it takes 5 hours to finish. I can go faster, but that's not nice bot behaviour for retailers.

1

u/KorporalKarnage Aug 02 '26

I've been having a blast trying different ways getting around all these anti bot/scraper blocks these sites put up. I had no idea there were so many ways they put up roadblocks but given time, they all fall down.

So my setup as of now is dedicated RPi running cron jobs for all the different tasks (and there are many). I also abandoned the porto-proxy app, although it did work, it just isn't needed with the setup as of today.

Basically it goes PC (if on)->Rpi->Proxy server 1->if problems->fallback proxy server 2-> Rpi->deploy

I upped my limits to 5k per for now and let er run....

1

u/KorporalKarnage Aug 02 '26

Oh yeah, BassPro is NOT worth the effort! Holy....