Stream IoT Data into the REPODS Cloud Data WarehouseMarko PetzoldBlockedUnblockFollowFollowingJan 8A quick introduction on how to securely stream IoT data into the REPODS cloud data platform.
We assume you are already have an account and a pod in the Repods Cloud Data Warehouse.
If not get both for free here.
The IoT Messaging SetupPublish some Data to testConnect and Listen with Repods1.
The IoT Messaging SetupWe are going to setup a basic IoT Messaging environment here from scratch, but normally you would have some other setup running productively in the field.
It should be easy for you to skip the steps you don’t need.
In our example we are using the WAMP messaging protocolA basic IoT Messaging architecture consists of a message router node and a messaging client.
The RouterFor the router speaking WAMP we are going to use crossbar.
To install the router on your local computer you have two easy possibilities.
If you have dockerinstalled then open a terminal and simply dodocker run -p 8080:8080 crossbario/crossbarThis will download and run crossbar in a container.
If you have a python3 installation on your computer you can also dopip install crossbarcrossbar initcrossbar startOk, you have the router running, but it is running on your local machine and not reachable from the internet.
To temporarily give it a publicly reachable ip address the easiest way is to use tunneling with ngrok.
You do have to sign up but otherwise you can use it for free for our example here.
So download the tool unzip it in a folder then go to that folder with your terminal and do.
/ngrok http 8080You will get an output like thisPlease note the temporary ip-address you got which in the upper example isf79e3bc8.
ngrok.
iobut yours will be different !2.
Publish some Data to testTo receive some data into Repods we need some data that is being published.
Here you would have a sensor publishing temperatures or something similar in a real life example.
Here we are going to publish an incremental counter on the topicrepods.
example.
counterevery second.
You need node installed on your computer and then you can start a terminal and install the WAMP client “autobahn” withnpm install autobahnThen create a file publish_counter.
jswith the following content:Some data publisherNow you can start the publisher withnode publish_counter.
js… and should see the following outputConnected to Routerpublished on 'repods.
example.
counter' : 1published on 'repods.
example.
counter' : 2published on 'repods.
example.
counter' : 3.
3.
Connect and Listen with RepodsNow our fake “IoT Sensor” is publishing numbers to the router and we only need to connect to the router and start listening.
If you did not yet sign up on Repods than you can do this here for free.
You will also get a Free Data Pod to play around with.
Choose a Data PodWait until your free demo pod has been created.
Then choose a pod in the list you want to use for this demo and start it.
(If you start the pod for the first time it will take a minute to initializethe pod but after that, start and stop will be much faster)Start and Enter the PodOnce your pod is started you can enter the pod and we can start the setup of the IoT Import.
Open the IoT Import SetupGot to the Import Panel and click the IoT Import button.
Create a new ConnectionClick the small orange plus sign in the Connections list to create a new connection to an IoT router.
Here you need to fill out the form as shown in the next screenshot.
In the URL input cell please use the temporary URL that was provided to you by ngrok above.
In my case this was:f79e3bc8.
ngrok.
ioCreate a full URL by prefixing ws://and suffixing/ws like this:ws://f79e3bc8.
ngrok.
io/ws(The prefix refers to the websocket protocoll and the suffix to the standard route the router serves websocket connections on.
)Enter the Connection DetailsHit save and you should see your new connection MyLocalin the connection list.
Activate the Connection to your RouterNow click the little play button next to your connection.
After a moment the button should transform into a little stop button (red square) which indicates that the Connection was successfully established.
Good !Since one router can handle thousands of topics we need to setup a listener on our specific topic repods.
example.
counterto receive some data.
We do this in the Subscriptionslist which you can find below the Connections list.
Create a new SubscriptionIn the Subscriptions list click the orange plus sign to create a new subscription.
Fill in the Form as shown in the screenshot below.
Now a new Raw Table S_MY_COUNTER will be created to store the received messages of this subscription.
Activate the SubscriptionWhen you click the checkbox next to your new Subscription MyCounter then the Data Pod will subscribe to the topic repods.
example.
counteron the connection MyLocal.
Every 100 received messages or after 10 seconds (whichever comes first) a new data package will be inserted into the raw table S_MY_COUNTER.
You can observe this live by zooming in on the Import Panels graphical representation of the data packages.
Watch the packages arriving every 10 secondsClick on any of the grey package stripes and you can inspect the content of the package in the popup grid.
Inspect one received Data PackageIn the column payload_args you can find the received message content.
The purple color indicates that the datatype of this column is a postgres native JSON format.
If you want you can now create a Workbook in the Results panel and query your data with native postgres SQL:Query Your IoT Stream DataOutlookNow you are continuously receiving data into the Raw Table S_MY_COUNTER.
Here are a few suggestions on where to go from here.
Set sum reasonable package retention setting in the raw table settings of S_MY_COUNTER by clicking on the table name in the Import Panel.
You can add derived columns in the raw table settings and use this to extract some info from the payload_argscolumn.
(In our case a good transform would be payload_args[1]::text::numericcheck out here)Create a Pipe to process the data into a long term Evo Table.
.