Shop Forum More Submit  Join Login

How to create your own bot from scratch

Journal Entry: Tue Apr 10, 2018, 4:28 AM


Greetings! 
– v1.2 (added some more info)

Someone inspired by dADroid-bot recently asked me to help them in the right direction of creating their own bot, and I wanted to write a complete guide on how a bot is made. Since this seemed to be something that other people might want to read in the future, I decided to write and share a comprehensive guide, listing all the required tools and skills to make a new bot from scratch without any past experience.

Now, I should mention that there are in fact third-party tools and services that can be used to create and/or replace any part of this guide, but considering that those services do or use more or less the same things explained here, I'd recommend to be at least familiar with the concepts explained here, and the basics of what is happening behind the scenes to run a bot.

I'd like to point out a few essential tools & skillsets required to start a new project like this, and I hope you'll make an amazing bot here!

With that said, let's get started!

Introduction – What is a bot?

a robot is simply a piece of software designed to work continuously behind the scenes to do what real humans do, automatically. Some bots are unfortunately made with malicious intent, but most bots are created to help people easily & quickly do what they'd otherwise regularly do by hand.

On DeviantArt, this generally means reading & writing comments, posts, journals and interacting with people.
(For more information about Good vs. Bad bots, have a look at here.)

There's also another category of completely different Deviant Art bots which happen to interact with the Deviant Art Messaging Network, which BotLab has had covered very well. Those bots are specific to chat system this site runs. This category is out of the scope of this tutorial.

Step 1 – Programming language

The first step to create a bot (or a software of any kind) is to be able to code.
Do you have any past experience with coding? If so, you might be able to focus writing your bot with that. If not, no worries, you'll need to start learning a new Programming Language. Which programming language to choose is not that important as long as it's capable of connecting to DeviantArt.

You'll just need to understand the basics of computer logic and should probably be comfortable to use the language on a regular basis. As an example for the programming language, dADroid-bot is coded in PHP, a standard web-development language. Other bots, such as dAbot (created by my friend Kishan-Bagaria) use Python, and some like dAhub I believe use Java (not to be confused with JavaScript).

I'll try to give a short description for each language, and I hope you find the right amd the most suitable one to use for you.

(Remember: memorizing it is not required, instead you should learn the logic behind it.)

  • Ruby: a beautiful, simple to understand and & easy to use but powerful tool to start programming.
    Try it here: tryruby.org

  • Python: a very popular scripting language, has a great, powerful and moderately easy to master syntax.
    Try it at: learnpython.org

  • Lua: a very flexible and easy scripting language. Here's an excellent YouTube video: www.youtube.com/watch?v=S4eNl1…

Once you've familiarized yourself with the concept, you can try some more advanced/complex languages in the future:

  • JavaScript (not to be confused with Java): the defacto standard scripting language. it's powerful yet easy, and originated from designing web pages and currently focused for modern web-apps.
    Try it at: www.javascript.com/try
    (I highly recommend learning this in the long run!)

  • PHP: the most-used web programming language, which only runs in the back-end (like servers, though it's possible to run on Desktop computers and your laptop too). It's powerful but some consider it to be ancient, and most modern developers try to avoid it. A very dangerous tool if handled uncarefully. A powerful, sharp tool if handled with care. :)
    dADroid is written mostly in PHP.

  • Java (not JavaScript): a powerful, moderately easy to understand and object-oriented language, which is somehow similar to JavaScript, yet completely different. Developed by Oracle Corporation and it's in wide use by many businesses. You don't need to bother yourself with it, at least for now. I believe dAhub was written in Java.

  • C#: developed by Microsoft, originally based on C and C++ syntax. Much better than ancient and unmaintained counterpart VB (visual basic) language, which was also designed by Microsoft. Until recently, it could only run on Windows operating systems (which meant not being able to run on Linux servers), though a minimalistic version for Linux has been recently released.
    It has a somehow similar syntax to JScript (Microsoft's fork of JavaScript.)
    Used widely in business environments, not exactly the most suitable for DIY projects like this, but has great potential, and as such is widespread, and hence has great support on websites like StackOverflow.com.
    Again, you do not bother yourself with it, just now. Might be a handy tool if you've picked up programming languages in academic classes (like college and university.)

Step 2 – API (connecting to DeviantArt)

Now that you've worked with some languages and picked up a general syntax (and hopefully chosen one of them to use), it's time to write an interface for your bot.

Using an API (application programming interface) your bot can talk to DeviantArt servers, and vice versa. If you imagine two pieces of code (one is your bot and the other one is DeviantArt server) are commuting with each other, the method they send and receive data is called an API. Kinda like what texting is for humans, if you will. 

Wikipedia has a great (and in-depth) article on APIs, if you want to read more. (Not bad reading material!)

DeviantArt uses two distinct set of APIs:
  1. a public one which DeviantArt provides here:
    www.deviantart.com/developers/

  2. an internal one called DiFi (DeviantArt Interactive Fragmented Interface) which they use themselves for calls between their site and their server.

    some kind souls have documented DiFi here:
    github.com/danopia/deviantart-…
a bot can use either (or even both) of these APIs to interface their bot with DeviantArt.

The public APIs are somehow limited, and you're required to register a key for your application to work with. However, it's free, documented regularly, and super-easy to use, and less prone to suddenly breaking.

The DiFi API is much more powerful, doesn't require signing up for a key, and is more popular with the DeviantArt modding community. However, DeviantArt site owners meant it to be only used for their own internal stuff, so it's a bit less reliable, can change at any given time without prior warning, and thus could potentially break your program. However, since DeviantArt hasn't really changed for several years, it's rare and unlikely that it completely remove this API.
(That's not to say instances of a bot completely breaking down in the past haven't occurred at all (like dAhub), but usually, developers are able to fix them and bring them back online completely, usually at a short time.)

Another concept which you should be familiar with is HTML scraping, which is the practice of fetching and loading the HTML code that is meant for the browsers from the DeviantArt servers into the bot and parsing it.

Professionals most often recommend using the official and public APIs, and you don't hear one recommending you to use the internal APIs. Against their warnings, however, dADroid, dAbot, (and dAhub to my knowledge) all use the DiFi internal API with the addition of some HTML scraping, because it's more powerful and portable.

Basically, anything you can do with the website can be done using DiFi, but they might try to block you (or your bot) – especially if your bot is made with malicious intent.

dADroid has a great HTML scraping and DiFi parsing utility built right into it.

The unofficial route gives you much more power but is harder to maintain. You may, however, choose to use the official way, which is a bit less powerful but easier to work with.

So what to use?

If you choose public APIs, everything is documented, but you need to register a key, and fewer functions are available to you.

If you prefer to have more functions available to use at the cost of unofficial support, you can go the DiFi way. If so, you can either write the code for DiFi yourself, use the already made samples, or politely ask someone to write it for you.
You can use these as samples to write your own. The one for dAhub is closed-source, meaning that you can not obtain the source code from him.

dADroid-bot (the one that I've created) is open-source, and actively maintained, but at the time of creating it I didn't have any intentions of releasing its source code to the public, so it's a bit messy.

Nonetheless, if you'd like to use PHP, you can use dADroid's source in your project. You can see a list of version history here: dadroid.ir/Changelog.txt


Step 3 – means of storage

So, at this point you've created a bot which can connect to DeviantArt, read its content, and post some results onto it.

Now, as the APIs are eyes and ears of your bot, you need a brain to process and store the content your bot receives.

What use would a bot have if it forgets everything whenever you close it?

You can use two primary methods of storing the content of your bot for later retrieval and use: a) as files on your disk and b) using a Database software.

For file-based storage, have a look at XML, YAML , and JSON. Don't get scared by lines of text! They're merely written to help your bot remember what's going on after each startup.

However, modern and complex bots can't rely on text files to accurately and efficiently store their data. So you need to use another tool to handle your data. 

Nowadays, the SQL (read SEQUEL) which stand for Structured Query Language is the standard way of writing and accessing data in Databases.

The most popular SQL servers (which you can use in your project) are MySQL and MariaDB at the moment. They are sister databases and you can use them interchangeably. The first one is currently developed by Oracle, the same company who maintains Java (not to be confused by JavaScript).

If you've ever heard of WordPress, it uses one of these Databases to hold its data, and I'm pretty sure that DeviantArt also uses something similar.

There's a lighter version of SQL which doesn't need a server and resides on disk, called SQLite. You can use it for your starting project. Fun Fact: Google uses SQLite in their Android OS all the time. So if you currently have a phone with Android as its OS, there's a pretty good chance that your phone uses SQLite!

Microsoft also uses their own propriety and non-free fork of SQL server, which is focused on businesses and you don't have to worry about.

All of these SQL variants (with the exception of Microsoft's) are free and open-source, you can download the software right away and use them in your projects.

Here are some great resources to learn SQL:
  1. www.w3schools.com/sql/
  2. www.codecademy.com/learn/learn…

Step 4 – logic, user settings & GUI

Now that your bot can connect to DeviantArt, get data, write to it, and remember what to do; you need to write the logic for what it actually needs to do.

The best way to do this is to write down your requirements for a bot, organize it into smaller block diagrams, and write the code for each block of your diagram. It's easier to follow (and debug) your bot's logic that way as it gets bigger, especially if you comment your work.

You also need to devise a means of controlling and "configuring" your bot. You can write an external web-page for its settings, or use DeviantArt comments. It's up to you! Once the command is received, the result will be saved in the Database.

At one point, you'll also probably need to implement timers in your logic, which on Linux servers are achieved by using cron jobs. Best idea to familiarize yourself with the concept, but no need to worry about it, for now.

When coding, it's best to always follow DRY practice: Don't Repeat Yourself. This means avoid copy-pasting code, which trust me, makes everything a hundred times harder. Instead, split your code into different meaningful and re-usable parts.

You can read more about on how to follow this pattern in a carefully designed fashion called MVC:
  1. Understanding Model-view-controller
  2. What is MVC?
  3. Wikipedia's on Model–view–controller
  4. MVC Pattern Tutorial
I'm not going to get deep into this, as there are already a lot of good articles written. Just remember, keep everything organized, and keep your code clean and well-commented, even if you're the only one working on the project, you wouldn't know if someone else would work on it tomorrow. Otherwise, you'll get into some unpredicted issues soon, like I have with the dADroid's code getting so messy that I need a full rewrite. :)

By the way, if you're going to version your bot (and you should), it's best to follow the Semantic Versioning practices. Remember: You probably won't be the only one working on the code, forever.


Step 5 – running the bot on a server

The last step is to run your bot somewhere which can it keep running in the background, as long as you wish.

For this task, generally, a server is used to keep your bot running. You can host your own server (which definitely isn't recommended, but I personally do anyways), or you can go the easier route, and buy a hosting plan (renting a server dedicated to your project.)

Generally, VPS hosting services are really cheap, and can run your code in the cloud. I recommend using them. Or, you can also choose to run your bot from your own personal computer, which means you need to keep your PC always on to keep your bot continuously running. If your computer (or a server for that matter) is turned off, it equals your bot not running.

You should choose your server plan very carefully according to your needs, especially if you intend to keep the bot running for a long time.

If you develop your bot in PHP (like I did) you can buy even cheaper plans meant for website hosting services, which could also host your bot program.

Final Notes – social side of the bot

Please try to keep your bot polite, and avoid spamming people. That's a very bad thing to do!

Don't do things that DeviantArt staff wouldn't like, they'd block you for no sensible reason, faster than what you'd think. I've learned this the hard way.

And it has become a standard convention on DeviantArt to mark your bot's generated messages with "I'm a bot" or similar notes. It's especially important if your bot generates replies to somehow identify it as a bot. (Like any instance of both dAhub and dADroid-bot. All of the popular bots on DeviantArt do this, to my knowledge.)

Conclusion

And there you go, a list of all essential points I could think of for a working and practical Deviant Art bot! I hope you learn new things along the way and have a great experience writing your own bot.

Be sure to check DeviousBots for some examples of useful bots around DeviantArt, and submit your own bot there if you made a great one!

You are encouraged to comment any questions you have or any problems that might occur in the future. I'll try my best to answer them when I can!

I tried to list everything that came to my mind, but I might update this journal in the future if something else also needs to be mentioned.

Have a great day.


Add a Comment:
 
:iconwhite-wolf-5:
White-Wolf-5 Featured By Owner Sep 18, 2018  Hobbyist General Artist
Great write-up!

Some of your statements stung a bit but in the end, I really don't care if my favorite languages are "ancient" as long as they still get the job done. lol

I agree completely that PHP should not be ignored or feared.  (Respected yes, but not feared.)  It's quite powerful and it's modular nature gives it so much versatility. 

Any way, thanks for sharing these great tips.
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Sep 21, 2018  Hobbyist General Artist
Thanks for your feedback!

I personally like PHP and am comfortable programming in PHP considering how versatile and powerful it is, as you put it.
As I said in my journal, even my own bot – dADroid – is mostly written in PHP.

But since it is one of the top popular programming languages, people often tend to misuse it.

This is a bit of generalization, but considering that Lerdorf initially made PHP as a simple templating cgi script,
many developers criticise PHP as an ancient language. Some doesn't give PHP the respect that it really deserves to have. 

It's true though, that with the hard work of the Zend developers and especially after the release of PHP 7.0,
many of the shortcomings of PHP has been addressed. However, many novice web developers use PHP inefficiently
or in a way that results in spaghetti code; while experienced programmers can use PHP to its true potential.

As an example, while PHP has introduced OOP in 2004 with the release of PHP 5, many still prefer to write procedural code
rather than using modern language features. They argument is that they "just want to get the job done", IMO they just don't care.

Personally, I love the syntax and non-strict semantics of PHP – but I dislike the non-consistency aspects of it, such as
$_VAR style superglobals, strrev(), usort() and etc global functions.

That's what I really meant by "handling it with care", to use PHP properly instead of carelessly.

The argument is that new languages could be better (since they are designed from scratch),
but that doesn't necessarily mean that PHP is (or should be) out of use.
Reply
:iconajcrwl:
ajcrwl Featured By Owner Aug 16, 2018
Hi. dAhub has closed, I'm thinking about developing my own bot. Pretty much everything you wrote here I've already figured out on my own, but I need a little help. Deviantart public APIs are easy to understand, however I can't figure out how to send points with the help of DiFi (or any other way?). You wrote that dADroid is written with PHP and that's exactly what I will be using. Right now I'm thinking through all the mechanics for the bot and the structure for data files, however without knowing how to give points the whole effort is.. pointless :D If you can give any insight into this, I'll be glad to learn.
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Edited Aug 17, 2018  Hobbyist General Artist
Hey ajcrwl

  • I'm not sure that the public APIs allow that at the moment, but if they did, that'd be the easiest way. Though I'm not entirely familiar with the public APIs as they're too limited for my taste.
  • As for the DiFi way, currently dADroid uses a combination of DiFi and HTML-scraping technique to achieve sending points with a function (or class method) called givePoints()
dADroid.givePoints() by DRSDavidSoft

I initially set my goal to make dADroid open-source, but since it's quite messy atm and needs a refactoring, I haven't uploaded it anywhere (like GitHub). But if you are interested, I can give you access to some parts of it, so you don't have to write them yourself.

If you are interested, you may use dADroid as a library in your bot, and we can work on the structure together.

Final note: as I have explained here, a site change is coming, which could render most of your work useless in the near future. I think it's better to halt development until the change is made, so the code can be written for the newer site UI.
Reply
:iconajcrwl:
ajcrwl Featured By Owner Aug 18, 2018
Thank you for this snippet, I realized I was sending points directly to user without checking for their donation widget. I figured out how to do it with some scraping, so all is well.

Sending points seems different from sending llamas, though. Anyway, you seem to have way better knowledge of PHP than I do, so maybe I'm doing it wrong, but it works, so idk :D To be honest, I simply wouldn't be able to use your code, because some parts of it are beyond my knowledge, but I'd be glad to get some help if I find myself stuck with any of DA's mechanics.

Regarding the upcoming update, I'd rather write everything now and do it quickly to have some grasp of things when everything changes AND be able to tell what changed. I just hope the change means something good, like expansion of APIs to work with points and llamas :D
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Aug 18, 2018  Hobbyist General Artist
You're quite welcome ;)

To be honest, I'm just scraping the "give form" as dA calls it,
as it happens to be that the form uses many of the same elements,
so I just figured to make it as a single function call.

👍 on doing all of it yourself, makes learning this stuff really practical,
that's also how I did it. You can still count on me if you got stuck anywhere :)

I think there's a chance they'll decide to replace the current framework, and
use something like Angular or React, which will make writing bots really hard.
Maybe hq or dt can answer that!

Word of advice: don't use `file_get_contents()` to request URLs, instead use `cURL()` calls.
Also, here's a wiki on the DiFi class, it may help you in some cases:
github.com/danopia/deviantart-…

In any case, I have written many scraping and form filling functions for this project,
so I'd still recommend working on one project instead of two, but maybe that's just
my opinion.

Wish you the best of luck! :D
Reply
:iconajcrwl:
ajcrwl Featured By Owner Aug 20, 2018
Damn, for some reason I thought I found a way to send llama badges with just one request, but I checked again and it doesn't work, so you're completely right, it's almost the same thing.

I figured from the start that file_get_contents will be useless because it throws errors for https requests. Using cURL without passing useragent resulted in 403 error, and without cookies I ran into captcha. So basically it's cURL + cookies + useragent for any request just to be safe.

So, I've been thinking through all the steps in bot logic and writing down charts to help myself later and ran into a bit of a problem with figuring out latest faves and watches. Let's just say that I maaaybe have figured out how to get timestamps for faves. But I don't see any way to know when a user added someone to friends. There is a difi call "GrusersUserpageinfoModule","getActivityDetails" which shows a short version of activity but it's not enough. Users can hide their activity page and then even the slightly longer list of 10 items is unavailable. Friends module on profile pages is also unreliable because core members can set it up to show people in alphabetical order or starting with oldest friends. API seems to return everyone in alphabetical order and sorted by groups(?? I'm not sure, maybe just alphabetical). Dates have to be stored somewhere somehow. CURSE YOU! 
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Aug 20, 2018  Hobbyist General Artist
I see you're running into the same problems I had! This would all be solved if the staff released an actual useful public API, but they won't do it, unfortunately.

Also, heads up if you want to scrape modal boxes, as there is a really annoying bug in dA that'll prevent reading more than 50,000 last records. 

Hope these sort of problems will get fixed in the next release of DeviantArt!
Reply
:iconxyuenna:
Xyuenna Featured By Owner Jul 16, 2018  Student Digital Artist
Are There's Some Risk When Creating A Bot? I Meant lagging Or So?
Reply
:icondailygiveget:
dailygiveget Featured By Owner Apr 22, 2018
Allot good information :)
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Apr 24, 2018  Hobbyist General Artist
Thanks, dailygiveget !
Reply
:iconiamthejam:
iamthejam Featured By Owner Apr 10, 2018
Here you are. Everything okay there? It's been a while.
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Apr 14, 2018  Hobbyist General Artist
Hey iamthejam, how have you been? It's been great here, just getting more busier everyday.
Reply
:iconiamthejam:
iamthejam Featured By Owner Apr 14, 2018
Same here. PM me or email me or YIM me when you can.
Reply
:iconlordcemonur:
lordcemonur Featured By Owner Apr 10, 2018  Professional Interface Designer
Great work!
Reply
:icondrsdavidsoft:
DRSDavidSoft Featured By Owner Apr 14, 2018  Hobbyist General Artist
Thank you, lordcemonur!
Reply
Add a Comment:
 
×

Featured in Collections

tutorials and helpful stuff by Showlover

Helpfulness by wintercool612

deviantART Related " ----------- by help-resources


More from DeviantArt



Details

Submitted on
April 10
Link
Thumb

Stats

Views
1,707 (4 today)
Favourites
24 (who?)
Comments
16