Today i got a message on patreon from an angry webcomic artist complaining i was scraping their site.
To which i admitted it was totally true.
I spent about one hour composing the following message. Which is unedited (aside for a few added commas) and i wish to share. People have to understand why i scrape. I hope this explains something. Please don't kill me in the comments.
No, believe me, i was sincere. And i really want to give you, at least for the stolen bandwidth and for your hard times, a donation.
And yes, i was rude. Sorry about that.
Let me explain a bit better.
This will be long.
But i wish to paint the whole situation.
Because, yes at a glance i'm just a rude person who is stealing bandwidth.
I saw your art and saw the fact that you were... "forced to degrade it"? Maybe this is better put. This made me think that even if i was saving your comic for future reading i still was using your bandwidth and you were obviously having a bad time. So even if i don't check the "scrapes for future reading" folder very often i could not ignore that, and i became your pledger.
And i've a long list of webcomics i have subscribed here on patreon exactly because i've been scraping them. Each month my patreon bill is around 552$ before taxes (which are 22% here in italy). There's 12 Youtube channels, 2 indie video game companies, 3 "others"and 110 webcomics. (Also ... cough... about 10 "porn artists" can't deny that.)
I don't use RSS because in most cases the functionality is not implemented or is poorly implemented making me miss updates in cases where webcomics already update very rarely.
If this were an ideal world i would agree with you. I am just an idiot who steals and just tries to quell its conscience with some patreon change.
Also, as i said i "follow" about 400 webcomics. You just made me count them, which is something i don't do very often. It's "only" 378. Of these: 89 have updated in the last month. According to the "last modified" on my folder where i save the scrapes.
I scrape just the front page in the vast majorities of cases (285) because that is the "least damage" i can do, i go one link further in 32 cases because the site structure makes me do so.
And i have to physically see the page because it is incompatible with the scraper in 35 case.
Then there's those who i wish to follow, but haven't had the time yet. You're one of the 26 of those i started scraping daily.
This is a total of 378. 35 of which i have to "babysit".
But... why scrape?
If my bandwidth was what is normally found in most large cities i would agree with you. But here i have 1 whole megabit of bandwidth. Because i'm in the country.
Up to 3 years ago i was in dialup with two lines used just for internet. That is a whopping 14 kilobytes per second. Now i'm up to 120 and paying half what i was paying before.
But money really is the issue.
I already said i'm pouring 500$ per month on patreon.
But if i wanted a decent land line i would have to pay about 5000 euros per every 10 meters of landline from the nearest fiber optics pit (25 km away) to my house. Because those are the rates. I already checked. That is what internet costs if you don't have it. True, if i did that i would be bringing internet to my "huge" community of about 3000 people. The hero!
But, sarcasm aside, you can already see why the phone company isn't going to do it.
Each of us pays about 30 euros per month in internet fees. This is about 90 thousand euros per month.
Since we're about 3000 people even if all of our money was going to be "put aside" to move to fiber optics, there were no fees there was no people to pay for maintenance customer service and whatever. Even if pits did not have to follow geography and could be put in a straight line, even if no unforeseen problems would ever to be found in the road from the fiber optics pit to here (last time they found a roman villa, that had stopped an entire construction site for 6 months while the history preservation committee tried to decide how to build around that and the construction workers still had to be paid and the engineeers had to be re-paid to make a new project)...
...so even in optimal conditions we're currently paying each month about 180 meters of cable.
We should be done paying for the fiber optics switch in about... 11 years and one half.
Even if we account for the last 3 years... That's still 8 and one half years to go.
At which point the phone company will make us upgrade to fiber which will net the phone company.... 50 euros per month.... to upgrade to whatever will be the next new thing that will be around by then.
If we will still be paying what are the current rates.
But internet prices go down and we get less and less service because maintenance and customer service and everything still has to be paid by the landline company.
But if i want to follow artists the only way is to either babysit the computer by constantly reloading pages that fail to load or load improperly, due to... welllll... apparently pages weren't designed to be loaded in batches of 30 at 120 kilobytes per second.
So either i open one link.
Wait for those 10 seconds for the page to load.
(if the page is only 1 megabyte... which it is but only if i use adblocks, otherwise it's about 1.5 to 2 megabytes)
Then open a second link.
Or i use a scraper which does the same thing and still blocks ads, only it does all alone and overnight and auto-retries pages that have loaded improperly due to the packet loss or whatever other problem was there.
I agree that this is stealing. I just want to explain why i am forced to do it.
The problem with 120 kilobytes per second (1 whole megabit) is that you can do work with it (emails still are "useable" with an upload of 16 kilobytes per second even if you use webmails) but you cannot "enjoy" internet with it.
So i have to use a scraper to download webcomics and i have to use a program to download youtube videos. And if i want to use skype i have to stop everything that i'm doing.
Which is why i am grateful somebody invented patreon.
So either i read a few comics per day and then start actually working or i download those comics overnight and then read them quickly in the morning and then start working.
Yesterday... let's see... yesterday i downloaded 5 comics of the 26 that are in my future to read list. And 17 of the ones that i download daily. One from the 35 pages i had to open manually. If i had to open all those 378 pages manually, assuming using adblock thus just 1 megabyte per page in average that would amounth to 3780 seconds. Thus about a little over 1 hour just to open each page and read nothing, with infinite reflexes and a reaaaly focused mind. That is: in ideal conditions.
It already takes me an hour to scan quickly the webcomics i already downloaded to notice those 17 pages and thus when just checking which folders have updated. You can see where i'm going.
So i chose to steal and then pay pack via patreon whenever possible. The "second option".
You might say that i should stop following sooo many comics. But yes.
What else can i do? I'm a disabled person that has to stay in bed half of the day on a good day.
Watch TV? Sorry, i prefer webcomics, also i prefer working since, as i said i'm disabled and cannot wrok the full 8 hours straight in a row most of the times.
And while i still am an engineer and i do make decent money with my engineering work (luckily somebody else invented photovoltaics)...
...Because "all" i have to do is make projects....
...And i have my nice computer to do so...
...I just cannot enjoy internet "properly" like many people do.
True, i could move 25 km to the city where they already have internet.
Except here i have an home which was paid by my parents, over there i would have to pay 700 euros of monthly rent.
So either i spend 500 euros of patreon and then download comics with a scraper or 700 euros of rent and then download comics at 10 megabytes per second and get all the viruses i could get by removing the adblocker from the comics i follow.
Also... yeah, at least, since i'm a disabled person, here i can pay for a person to help me about 25 euros per hour for the 8 hours they have to help me clean the house and whatever i need at the moment.
In that city the rates are 35 euros per hour minimum.
So... there's also that.
I was rude. True. Sorry about that. I hope i've explained my situation better.
As i said. I still want to make my one time donation of at least 20$ and help you pay for your internet fees. So for now, until i've been able to read your comic properly i'm paying 1$ per month. And if you've got a paypal i'll send you the money there.
While writing the above i ran the scraper, so with a minimal internet connection usage (i was writing text on notepad++) i can already confirm that in 2 and one half hours since i started writing that message the scraper has just finished checking all the comic pages. The whole process takes about 2 hours and one half to complete, so yes... those numbers i wrote above? The "1 hour 3 minutes" ideal world condition to open all the comics at 10 pages per second with adblock... just is not real. A scraper is machine efficient and only reloads bad received files. And even it cannot do the work in less than 2 hours and one half, (just the front page in most cases) with my landline. Sooooo yeah. There's that.
Sorry to all people i scrape. Whenever possible i try to subscribe on patreon.