Shop Forum More Submit  Join Login

How to create your own bot from scratch

Journal Entry: Tue Apr 10, 2018, 4:28 AM


Greetings! 
– v1.2 (added some more info)

Someone inspired by dADroid-bot recently asked me to help them in the right direction of creating their own bot, and I wanted to write a complete guide on how a bot is made. Since this seemed to be something that other people might want to read in the future, I decided to write and share a comprehensive guide, listing all the required tools and skills to make a new bot from scratch without any past experience.

Now, I should mention that there are in fact third-party tools and services that can be used to create and/or replace any part of this guide, but considering that those services do or use more or less the same things explained here, I'd recommend to be at least familiar with the concepts explained here, and the basics of what is happening behind the scenes to run a bot.

I'd like to point out a few essential tools & skillsets required to start a new project like this, and I hope you'll make an amazing bot here!

With that said, let's get started!

Introduction – What is a bot?

a robot is simply a piece of software designed to work continuously behind the scenes to do what real humans do, automatically. Some bots are unfortunately made with malicious intent, but most bots are created to help people easily & quickly do what they'd otherwise regularly do by hand.

On DeviantArt, this generally means reading & writing comments, posts, journals and interacting with people.
(For more information about Good vs. Bad bots, have a look at here.)

There's also another category of completely different Deviant Art bots which happen to interact with the Deviant Art Messaging Network, which BotLab has had covered very well. Those bots are specific to chat system this site runs. This category is out of the scope of this tutorial.

Step 1 – Programming language

The first step to create a bot (or a software of any kind) is to be able to code.
Do you have any past experience with coding? If so, you might be able to focus writing your bot with that. If not, no worries, you'll need to start learning a new Programming Language. Which programming language to choose is not that important as long as it's capable of connecting to DeviantArt.

You'll just need to understand the basics of computer logic and should probably be comfortable to use the language on a regular basis. As an example for the programming language, dADroid-bot is coded in PHP, a standard web-development language. Other bots, such as dAbot (created by my friend Kishan-Bagaria) use Python, and some like dAhub I believe use Java (not to be confused with JavaScript).

I'll try to give a short description for each language, and I hope you find the right amd the most suitable one to use for you.

(Remember: memorizing it is not required, instead you should learn the logic behind it.)

  • Ruby: a beautiful, simple to understand and & easy to use but powerful tool to start programming.
    Try it here: tryruby.org

  • Python: a very popular scripting language, has a great, powerful and moderately easy to master syntax.
    Try it at: learnpython.org

  • Lua: a very flexible and easy scripting language. Here's an excellent YouTube video: www.youtube.com/watch?v=S4eNl1…

Once you've familiarized yourself with the concept, you can try some more advanced/complex languages in the future:

  • JavaScript (not to be confused with Java): the defacto standard scripting language. it's powerful yet easy, and originated from designing web pages and currently focused for modern web-apps.
    Try it at: www.javascript.com/try
    (I highly recommend learning this in the long run!)

  • PHP: the most-used web programming language, which only runs in the back-end (like servers, though it's possible to run on Desktop computers and your laptop too). It's powerful but some consider it to be ancient, and most modern developers try to avoid it. A very dangerous tool if handled uncarefully. A powerful and sharp tool if handled with care. :)
    dADroid is written mostly in PHP.

  • Java (not JavaScript): a powerful, moderately easy to understand and object-oriented language, which is somehow similar to JavaScript, yet completely different. Developed by Oracle Corporation and it's in wide use by many businesses.. You don't need to bother yourself with it, at least for now.

  • C#: developed by Microsoft, originally based on C and C++ syntax. Much better than ancient and unmaintained counterpart VB (visual basic) language, which was also designed by Microsoft. Until recently, it could only run on Windows operating systems (which meant not being able to run on Linux servers), though a minimalistic version for Linux has been recently released.
    It has a somehow similar syntax to JScript (Microsoft's fork of JavaScript.)
    Used widely in business environments, not exactly the most suitable for DIY projects like this, but has great potential, and as such is widespread, and hence has great support on websites like StackOverflow.com.
    Again, you do not bother yourself with it, just now. Might be a handy tool if you've picked up programming languages in academic classes (like college and university.)

Step 2 – API (connecting to DeviantArt)

Now that you've worked with some languages and picked up a general syntax (and hopefully chosen one of them to use), it's time to write an interface for your bot.

Using an API (application programming interface) your bot can talk to DeviantArt servers, and vice versa. If you imagine two pieces of code (one is your bot and the other one is DeviantArt server) are commuting with each other, the method they send and receive data is called an API. Kinda like what texting is for humans, if you will. 

Wikipedia has a great (and in-depth) article on APIs, if you want to read more. (Not bad reading material!)

DeviantArt uses two distinct set of APIs:
  1. a public one which DeviantArt provides here:
    www.deviantart.com/developers/

  2. an internal one called DiFi (DeviantArt Interactive Fragmented Interface) which they use themselves for calls between their site and their server.

    some kind souls have documented DiFi here:
    github.com/danopia/deviantart-…
a bot can use either (or even both) of these APIs to interface their bot with DeviantArt.

The public APIs are somehow limited, and you're required to register a key for your application to work with. However, it's free, documented regularly, and super-easy to use, and less prone to suddenly breaking.

The DiFi API is much more powerful, doesn't require signing up for a key, and is more popular with the DeviantArt modding community. However, DeviantArt site owners meant it to be only used for their own internal stuff, so it's a bit less reliable, can change at any given time without prior warning, and thus could potentially break your program. However, since DeviantArt hasn't really changed for several years, it's rare and unlikely that it completely remove this API.
(That's not to say instances of a bot completely breaking down in the past haven't occurred at all (like dAhub), but usually, developers are able to fix them and bring them back online completely, usually at a short time.)

Another concept which you should be familiar with is HTML scraping, which is the practice of fetching and loading the HTML code that is meant for the browsers from the DeviantArt servers into the bot and parsing it.

Professionals most often recommend using the official and public APIs, and you don't hear one recommending you to use the internal APIs. Against their warnings, however, dADroid, dAbot, (and dAhub to my knowledge) all use the DiFi internal API with the addition of some HTML scraping, because it's more powerful and portable.

Basically, anything you can do with the website can be done using DiFi, but they might try to block you (or your bot) – especially if your bot is made with malicious intent.

dADroid has a great HTML scraping and DiFi parsing utility built right into it.

The unofficial route gives you much more power but is harder to maintain. You may, however, choose to use the official way, which is a bit less powerful but easier to work with.

So what to use?

If you choose public APIs, everything is documented, but you need to register a key, and fewer functions are available to you.

If you prefer to have more functions available to use at the cost of unofficial support, you can go the DiFi way. If so, you can either write the code for DiFi yourself, use the already made samples, or politely ask someone to write it for you.
You can use these as samples to write your own. The one for dAhub is closed-source, meaning that you can not obtain the source code from him.

dADroid-bot (the one that I've created) is open-source, and actively maintained, but at the time of creating it I didn't have any intentions of releasing its source code to the public, so it's a bit messy.

Nonetheless, if you'd like to use PHP, you can use dADroid's source in your project. You can see a list of version history here: dadroid.ir/Changelog.txt


Step 3 – means of storage

So, at this point you've created a bot which can connect to DeviantArt, read its content, and post some results onto it.

Now, as the APIs are eyes and ears of your bot, you need a brain to process and store the content your bot receives.

What use would a bot have if it forgets everything whenever you close it?

You can use two primary methods of storing the content of your bot for later retrieval and use: a) as files on your disk and b) using a Database software.

For file-based storage, have a look at XML, YAML , and JSON. Don't get scared by lines of text! They're merely written to help your bot remember what's going on after each startup.

However, modern and complex bots can't rely on text files to accurately and efficiently store their data. So you need to use another tool to handle your data. 

Nowadays, the SQL (read SEQUEL) which stand for Structured Query Language is the standard way of writing and accessing data in Databases.

The most popular SQL servers (which you can use in your project) are MySQL and MariaDB at the moment. They are sister databases and you can use them interchangeably. The first one is currently developed by Oracle, the same company who maintains Java (not to be confused by JavaScript).

If you've ever heard of WordPress, it uses one of these Databases to hold its data, and I'm pretty sure that DeviantArt also uses something similar.

There's a lighter version of SQL which doesn't need a server and resides on disk, called SQLite. You can use it for your starting project. Fun Fact: Google uses SQLite in their Android OS all the time. So if you currently have a phone with Android as its OS, there's a pretty good chance that your phone uses SQLite!

Microsoft also uses their own propriety and non-free fork of SQL server, which is focused on businesses and you don't have to worry about.

All of these SQL variants (with the exception of Microsoft's) are free and open-source, you can download the software right away and use them in your projects.

Here are some great resources to learn SQL:
  1. www.w3schools.com/sql/
  2. www.codecademy.com/learn/learn…

Step 4 – logic, user settings & GUI

Now that your bot can connect to DeviantArt, get data, write to it, and remember what to do; you need to write the logic for what it actually needs to do.

The best way to do this is to write down your requirements for a bot, organize it into smaller block diagrams, and write the code for each block of your diagram. It's easier to follow (and debug) your bot's logic that way as it gets bigger, especially if you comment your work.

You also need to devise a means of controlling and "configuring" your bot. You can write an external web-page for its settings, or use DeviantArt comments. It's up to you! Once the command is received, the result will be saved in the Database.

At one point, you'll also probably need to implement timers in your logic, which on Linux servers are achieved by using cron jobs. Best idea to familiarize yourself with the concept, but no need to worry about it, for now.

When coding, it's best to always follow DRY practice: Don't Repeat Yourself. This means avoid copy-pasting code, which trust me, makes everything a hundred times harder. Instead, split your code into different meaningful and re-usable parts.

You can read more about on how to follow this pattern in a carefully designed fashion called MVC:
  1. Understanding Model-view-controller
  2. What is MVC?
  3. Wikipedia's on Model–view–controller
  4. MVC Pattern Tutorial
I'm not going to get deep into this, as there are already a lot of good articles written. Just remember, keep everything organized, and keep your code clean and well-commented, even if you're the only one working on the project, you wouldn't know if someone else would work on it tomorrow. Otherwise, you'll get into some unpredicted issues soon, like I have with the dADroid's code getting so messy that I need a full rewrite. :)

By the way, if you're going to version your bot (and you should), it's best to follow the Semantic Versioning practices. Remember: You probably won't be the only one working on the code, forever.


Step 5 – running the bot on a server

The last step is to run your bot somewhere which can it keep running in the background, as long as you wish.

For this task, generally, a server is used to keep your bot running. You can host your own server (which definitely isn't recommended, but I personally do anyways), or you can go the easier route, and buy a hosting plan (renting a server dedicated to your project.)

Generally, VPS hosting services are really cheap, and can run your code in the cloud. I recommend using them. Or, you can also choose to run your bot from your own personal computer, which means you need to keep your PC always on to keep your bot continuously running. If your computer (or a server for that matter) is turned off, it equals your bot not running.

You should choose your server plan very carefully according to your needs, especially if you intend to keep the bot running for a long time.

If you develop your bot in PHP (like I did) you can buy even cheaper plans meant for website hosting services, which could also host your bot program.

Final Notes – social side of the bot

Please try to keep your bot polite, and avoid spamming people. That's a very bad thing to do!

Don't do things that DeviantArt staff wouldn't like, they'd block you for no sensible reason, faster than what you'd think. I've learned this the hard way.

And it has become a standard convention on DeviantArt to mark your bot's generated messages with "I'm a bot" or similar notes. It's especially important if your bot generates replies to somehow identify it as a bot. (Like any instance of both dAhub and dADroid-bot. All of the popular bots on DeviantArt do this, to my knowledge.)

Conclusion

And there you go, a list of all essential points I could think of for a working and practical Deviant Art bot! I hope you learn new things along the way and have a great experience writing your own bot.

Be sure to check DeviousBots for some examples of useful bots around DeviantArt, and submit your own bot there if you made a great one!

You are encouraged to comment any questions you have or any problems that might occur in the future. I'll try my best to answer them when I can!

I tried to list everything that came to my mind, but I might update this journal in the future if something else also needs to be mentioned.

Have a great day.


dADroid: Automate your dA routine

Journal Entry: Tue Dec 30, 2014, 6:09 PM
:bulletgreen: dADroid (online - beta)
Developer: DRSDavidSoft

Please note: I would like to know your opinions, ideas, suggestions, feedbacks and comments on this project. - Also if you are a software developer like me, please let me know

3D Render: dADroid Crystals by DRSDavidSoft

:icondadroid-bot: dADroid-botchangelog 

Hi everyone!

For the past year I've been working on a DeviantArt-related project, called the dADroid (dADroid-bot) which is in design, a multi-purpose dA bot. Using dADroid, you can automate what you normally do easier, faster and more exact.

If you are a developer, you can use dADroid APIs to power your own bot, an example is the talkbot.

If you are a point account, you can use dADroid to automatically process activity, calculate points and give them to other users. Like what dAhub does and what dATrade formerly did.

If you are a regular user, then there is tons of features out there for you (and more coming everyday).

dADroid is currently a work in progress and in the beta stage, but feel free to test any of its available functions. Click here to begin: fav.me/d9gef09

I also want to collect suggestions, ideas and anything that you might have in mind regarding this project. Do you always wanted something that dA hadn't, like email notifications? Would you like to automate something that really annoys you doing manually everyday? What features that you need dA lack?

If you might have any cautions, critiques or anything in particular against this kind of things, I'd be more than glad to hear them. Any feedback is highly appreciated. :) (Smile)

Also, for the fellow developers out there, my collaboration position is open. If you'd like to integrate any part of dADroid in your own bot, or simply need help writing it, I'm there for you.

If you are interested in this project, be sure to read this:

:bulletblue: Do you miss messages left on your profile, or notes to you?
:bulletblue: Do you wish you could get an e-mail when a specified deviation shows up on your message center?
:bulletblue: Do you give llamas/watches/faves on a regular basis?
:bulletblue: Do you thank whoever gives you a llama randomly by returning another llama, or even leaving a message?
:bulletblue: Do you use points accounts like :icondahub: dAhub?
:bulletblue: Are you tired of checking the "newest" list every often, and giving llamas and/or watching deviants one by one?
:bulletblue: Or are you a points account owner?
:bulletblue: And you are would like to automate the points transfer process, but you don't have time or skill to write a bot yourself?

:bulletgreen: Then dADroid is here for you!
By joining the dADroid grid, you can automate your daily tasks, be relaxed and browse dA arts. dADroid will do the job for you. It will automatically:
- Thank people who have gave you watches/llamas/faves by sending a random comment from a specified list (plz accounts are also supported.)
- Send a welcome message when a deviant joins your group
- Return llamas to whoever have given you one
- Get you points :points: from points accounts, by giving watches/llamas/faves

o If you are a points account owner, dADroid can: o
- Find out if a specified deviant has watched/llama'd/faved a list of other deviants
- Manage how many deviants s/he has given, calculate and process the amount of points s/he earned
- Give the points to them on a regular basis or immediately

dADroid is currently helping in llama-list and Give-LlamaGetA-Llama!
The bot also automatically trades llamas every 5 hours in:
GiveALlamaGetALlama

Go watch dailygiveget, it is also powered by dADroid-bot.

:bulletred: How does it work?
After logging-in to dADroid, you'll see the configuration page.
You can set different rules for dADroid. For example, you can set a task for it to check the newest dAhub's watches/llamas/faves list, set a rule to automatically give llamas to users who have gave you watches/llamas/faves or enable the automatic welcome and thanks messages.
Once every couple of minutes, dADroid will go online and read your configurations. It checks what requests needs to be done, gathers the required information from :deviantart: DeviantArt, processes the data, and replies the data, including commands, back to the dA servers.

:bulletred: How to join?
The dADroid bot is at the beta stage at the moment.
Many parts of the source code is being written right now.
Please leave a comment on this journal and watch DRSDavidSoft for future notifications.
You'll be notified when the project is online.

:bulletred: I have an idea! / I have a request.
Please let me know if you have a suggestion or request. I am open to any good and useful idea you might have.
Just ask for something, and I will write the code for it! :D

:bulletred: I want to test the Project now!
:bulletred: I am a Programmer / Software Developer
You can join the dADroid test team by noting DRSDavidSoft.
Your feedback and comments are highly appreciated.
If you are a software developer, feel free to join coding the project. It will help debugging and finishing the project way more soon.

Thanks for taking time reading this!
Looking forward for your opinions. :) (Smile)



NOTES:
* dAhub modules would only work for verified accounts.
** Your account would still give watches/llamas/faves, only it would be automated by the bot, and not done manually by you (i.e. pressing the "Give" button).


Update:
Thanks for all the great and encouraging comments!
I have decided to finish this project and make it online for everyone.

P.S:
If you are one of these haters, no hard feelings, alright? Just ignore my project.


Fanfiction Updated:
Alone on the Island, Chapter 6.

If you haven't read this fan fiction, It's a fluffy J/C one. Read it over here.
If you've followed it, you can read the 6th chapter.

Thanks for reading! ^_^
This is my first journal, yay ^_^

I'm now 18 years old!

I've been waiting for this moment for a long time, and now I'm really excited now!

I want to thank my friends on dA, dAhub,
:iconiamthejam: for his comment on my page and :iconcindy-vortex4: for her beautiful (and delicious) birthday cake.

Thanks everyone! :D