[IoT] Simple IoT with free cloud solutions (tutorial)German GensetskiyBlockedUnblockFollowFollowingFeb 24Before we startThe resulting dashboard can be found here ;)IntroductionI love automation.
And I always wanted to automatize the watering of my plants (because I always forget about them).
And yes, there are plenty of different turnkey solutions, but that’s not a developer way.
As a first step, I decided to implement the monitoring of soil moisture.
But to monitor some value — I need to see that value somehow.
And also important to see the historical change of that value.
Therefore I need some dashboard and server to host it and some data storage.
And that’s cost money.
But I don’t need the full power and storage of even the smallest and cheapest server on Digital Ocean for example.
Ideally, I don’t want to spend money each month at all for that.
So, why not to just use a free plan of plenty of different cloud solutions for everything?And I was able to achieve that and now I’m ready to share it with you.
Weapon of choiceMQTT broker — CloudMQTTVery straight and really simple to set up cloud MQTT solution.
Has plenty of plans for different purposes and in case you will need to scale.
Also has a free plan named “Cute Cat” that allows you to have 5 devices and supports 10 Kbit/s.
Pretty enough for an experiment or PoC.
Backend and Database — Back4AppBackend as a Service with a pretty user-friendly interface.
Provides you a storage and REST API.
It will give you 10 requests per second and 10 000 requests per month.
It gives you around 13 requests per hour.
So be careful with testing ;)Dashboard — FreeboardSimple but powerful dashboard.
It provides you a cloud solution but there are no free plans.
But Freeboard is open-source so we can host it by ourselves.
Since it will be self-hosted — no limitations on what you can do.
Server — HerokuPopular Platform as a Service.
Has a free plan and gives you the possibility to scale up later with what you need.
Great way to host something simple like freeboard.
StructureProject structureSo, the device with a sensor (soil moisture sensor in our case) will publish its measurements using the MQTT protocol.
MQTT to HTTP server will listen to the topic where the measurements are published and will resend them through HTTP API to the Back4App.
Back4App will store all the history of measurements and provide us an endpoint to retrieve that data.
Freeboard will use that API to load the last 50 measurements and build sparkline based on it.
Since our device will work with MQTT and Back4App has only REST API, we need one more layer between them — a simple MQTT to HTTP translator that will listen our MQTT topic with sensors data and send it to the Back4App API.
In that particular case using MQTT is not required, the device simply can send an HTTP request directly to Back4App.
But it will be simpler to add more devices and implement communication between devices using MQTT broker.
First step: MQTT brokerSetting it up using CloudMQTT is as simple as possible.
Go to the web page, click on the button “Get a managed IoT broker today”.
Choose a free plan “Cute Cat” and Sign Up.
After that, you will need to choose a name and region for your project and then you will have all the credentials you need to work with it.
Here is a small example of how to connect and publish a message using MicroPython:import machinefrom umqtt.
simple import MQTTClientmqtt = MQTTClient( ubinascii.
hexlify(machine.
unique_id()), "m24.
cloudmqtt.
com", port=27730, user="[username]", password="[password]", ssl=True)mqtt.
publish('sensors/moisture', str(moisture_value).
encode())Second step: BackendBack4App gives you a way to rapidly set up your database and start working with it.
Just sign up, click on Build new app and choose a name.
After that, you will see a dashboard and a small tutorial will be started.
When you pass the tutorial you can create a new clear App or use that one if you want.
In the App, you’ll need to create a new class named moisture (actually you can name it whatever you want but in next steps, I will use that name).
By default, any class you create has the next columns: objectId, updatedAt, createdAt and ACL.
It’s useful for as to have a column so we don’t need to take care of the time of the measurement.
We need to add one more column named value with type Number to store the actual value of soil moisture.
And we’re mostly done after that!Security (optional)It’s not so important for some personal experiment to set up the security, so you can skip that part if you don’t want to make your dashboard public to show it to someone.
So, if you don’t want a stranger to be able using your API key from Freeboard (where you can’t hide it) to mess with your data (e.
g.
deleting everything just because he can).
You’ll need to restrict write rights for the moisture table and create a User that will have write privileges.
The class for the users already exists in App by default.
So we need to create a new record here.
Click on “Add a row” and fill out next fields: email, password, username, emailVerified.
Just like that:Example of user creationAfter filling out all of the required fields with valid values object must be created automatically and objectId will be assigned to it.
When the user created you need to set up security on the moisture table.
Click on the icon with shield and in the dialog window remove the checkmark from theWrite column, in the field below enter the username and press enter.
The checkmark on the Write field will be already set and the user’s ID will be shown instead of the user’s name.
Example of permissions settingsThe last step is to Log In with the created user to retrieve its session token.
Just send the GET request with username and password of the user specified in parameters.
From the response, you will need a sessionToken.
You can find an example here.
Third step: MQTT to HTTP bridgeFor that part, I decided to make a simple python script using Paho MQTT and Requests libraries.
It’s simply listening to thesensors/moisture topic and pushes each value that is published there.
Let’s begin with object creation with Back4App API.
Using requests it’s super simple:Example of object creation using Back4App APIAll of the keys can be found in the Back4App dashboard, in the App Settings, Security & Keys.
After we ready to create new objects the next step is to subscribe to our MQTT topic with measurements data.
Example of subscribing and listening for messages using Paho MQTTIn the example above when any new message will be published to the topic sensors/moisture the function on_message will be called.
You can try it out by publishing messages using WebSocket UI in your CloudMQTT console.
So, after we’ve figured out how to work with both parts all we need is to add an object creation to the on_message function:def on_message(client, obj, msg): print(f'Received {msg.
topic} (QoS: {msg.
qos}): {msg.
payload}') create_object(msg.
payload.
decode())DeploymentLast part of that step is to deploy our small app on Heroku.
First of all, you need to create an account on Heroku (if you don’t have one already).
After that simply create a new app, chose a name and region for it.
Also, we will need to specify a Procfile for Heroku so it will know how too run our application.
Simply create a file named Procfile (no extension needed) and put it there:server: python -u run.
pyAfter it’s done we’re ready to deploy our app:If you haven’t already, log in to your Heroku account and follow the prompts to create a new SSH public key.
$ heroku loginInitialize a git repository in the folder with your app$ git initAdd Heroku’s remote$ heroku git:remote -a app-nameDeploy your changes$ git add .
$ git commit -am “initial commit”$ git push heroku masterBasically, almost the same instructions can be found in the “Deploy” tab of your app dashboard.
After deployment is done you can test it using WebSocket UI in your CloudMQTT console.
Fourth step: DashboardWe will use Freeboard to simply create a dashboard for our small system.
First of all, clone it from Freeboard’s GitHub.
Then from the folder Git just created run npm install and then run grunt.
Unfortunately, for our setup, we can’t simply use Freeboard as is.
That’s because of the default sparkline widget in it designed to work with real-time data, but we need to draw a historical data from our database.
Therefore we will need to make a simple plugin with sparkline that able to show an array of historical data.
You can find my simple implementation of it in GitHub.
You can simply download a file, add it to the folder plugins in your Freeboard clone and then add the path to the plugin in the index.
html :<script type="text/javascript"> head.
js("js/freeboard_plugins.
min.
js", "plugins/historic.
sparkline.
js", // < Right there // *** Load more plugins here ***Set up your dashboardImportant: don’t refresh the page because all of the changes exist only in your browser until you save them.
Now you can simply open index.
html using your browser.
There you will find an empty and ready to go Freeboard interface.
You can play with it as you want, while I will describe to you how to prepare it for our system.
And the first step here is to add a data source.
In the “Datasources” panel click on ADD.
In the pop up select a type JSON and choose the name you like.
In the URL field specify Back4App endpoint to retrieve moisture.
https://parseapi.
back4app.
com/classes/moisture?limit=50&order=-createdAtI also added here parameters to retrieve only the last 50 records.
order=-createdAt — order rows by creation date in reversed order (the last one will go first).
limit=50 — retrieve only 50 rows.
Method must be GET and two headers need to be added:X-Parse-Application-Id: [Application ID key for Back4App]X-Parse-JavaScript-Key: [JavaScript key for Back4App]Example of data source set upAll of the keys can be found in the Back4App dashboard, in the App Settings, Security & Keys.
The next thing we need to add is out historical sparkline.
Click on ADD PANE to add a pane (wow).
Those panes basically a “shelves” for your widgets.
After pane is added — click on the plus icon in its corner.
In type select the Historical Sparkline.
Choose the title for it and add our data source datasources[“moisture”][“results”].
Example of historical sparkline setupIf everything goes well you must be able to see a small graph with your data (hope you have some random numbers in the database, right?).
I also wanted to add an average value for moisture.
Here comes another cool part of the Freeboard — you can use JavaScript function as a value for the widget.
To make it happen, create a new widget with type Text.
And near the value field — click on the .
JS EDITOR.
Put that simple code in there:var total = 0;for(var i=0; i < datasources["moisture"]["results"].
length; i++) { total += datasources["moisture"]["results"][i]["value"]; }return Math.
round( total / datasources["moisture"]["results"].
length);Example of the average value widget setupPersist your dashboardWe don’t want to lose all our hard work here just after we close the browser.
So we need to save our dashboard.
Click on the SAVE FREEBOARD in the heading panel and chose the way of JSON file formatting (I prefer PRETTY version).
Save your dashboard to the root directory of your clone of Freeboard, we will need it later to be accessible on the server.
Freeboard on HerokuAfter we are done with setting up the dashboard — we might be wanting to make it accessible from anywhere and not just locally.
Deploying Freeboard on the Heroku from scratch for the first time can be tricky.
But I already figured it out and will give you some simple instructions ;)You will need to create a new Heroku app for that part.
First of all, we need something to serve ours index.
html.
For that, we need to add some packages to the NPM: npm install connect serve-static –save.
After that, we need to create a simple server.
js file with the next content:var connect = require('connect');var serveStatic = require('serve-static');const PORT = process.
env.
PORT || 8000;connect().
use(serveStatic(__dirname)).
listen(PORT, function(){ console.
log('Server running.
');});The last step is to specify Procfile:web: node server.
jsAnd that’s all.
After it’s done commit all the changes and push it to the Heroku remote (the same process that was done in the MQTT to HTTP setup).
Load your board by defaultThat’s why we needed to add the dashboard JSON to the repository.
To load some JSON with dashboard you just need to add that small part to the end of the URL to your Freeboard #source=dashboard.
json.
E.
g.
https://cucumbers-freeboard.
herokuapp.
com/#source=cucumbers.
json.
Congratulations!If everything was smooth you now must have a working prototype of your cloud-based IoT infrastructure.
The one I ended up with you can find here:freeboardExample of the end result of the tutorialcucumbers-freeboard.
herokuapp.
comThanks for reading!.Hope you liked it.
German Gensetskiy under the support of Go Wombat Team.
P.
S.
In case of any problems with following the tutorial I will be happy to hear your feedback.
You can reach me out by the email: Ignis2497@gmail.
com.