Master Python through building real-world applications (Part 7)Data Collector Web App with PostgreSQL and FlaskDhrumil PatelBlockedUnblockFollowFollowingJan 29Working with database and queries can be pretty daunting to some, or maybe most of us.
Perhaps, I have lost 100 readers by now just because there’s PostgreSQL written in the subtitle.
But as you are here, I want you to know that it is an important thing to learn and I will make sure that it is like walking in the park for you.
To do that, before we go any further, I want you to read the quoted text.
Twice.
What sets highly actualized people apart from those who never live inspired lives is that they do those things that less developed people don’t like doing — even though they might not like doing them either.
— Robin Sharma in The Monk Who Sold His FerrariI, my self, don’t really enjoy it, but as Robin Sharma said, doing things that we don’t like doing is something that sets us apart from the herd.
In this part, we are going to create a website that collects data the user has entered (created using flask) and store it to our database (PostgreSQL).
Also, we will make it live so this can work in real time.
Important Note — I have already covered a detailed post (Part 4 of this series) where we create a web app using Flask and make it live using the Heroku app.
Therefore, I will primarily focus on the back end in this article.
If you haven’t read that post yet, going quickly through it would be a great idea as you’ll know how to set up the front end, virtual environment, and how to make the application live.
Go, I’ll wait for you here.
Alright, everybody on the same page?.Let’s begin.
Step 1 — Setting Up Front EndAs discussed earlier, we will focus on the back end mainly.
But that doesn’t mean we will abandon everything else.
I will briefly discuss everything so that it will serve as a refresher for learners following this series from the beginning.
Flask is a micro-framework for web development and is often used while working with the database.
We will create a webpage from which we can get the user input.
Creating a website with Flask is easy stuff.
Look at the code below.
I know you have gone through Part 4 so you have the idea of how this thing works.
The only thing new here is, methods = ['POST'] that’s because we are sending data to the server.
Also, make sure to set up the virtual environment for this application as well ( Step 4 in Part 4).
I have downloaded a template and modified to get data of height, weight, sex, and shoe size of the user.
You must be thinking why?.Taking height, weight is cool but why shoe size?.Right?.You will know at the end.
So the front end looks something like this.
It’s up to you how you design your front end.
Step 2— Setting Up Back End2.
1 Create a PostgreSQL DatabaseTo create and store data, we are going to need a database management system.
Hence, we are using PostgreSQL for it.
If you are on a Windows or Mac machine, there are installers available on the official website and if you are on a Linux machine, find the pgAdmin III app from the Ubuntu Software (16.
04 or later).
However, if you are using Ubuntu 15.
10 or earlier (which I am sure you are not), follow the first answer on this Stack Exchange page to install it manually (we have already created our virtual environment so do the needful apart from that).
Create UserOnce you have installed pgAdmin 3, create a user and set a password for that by typing following.
Where the first ‘postgres’ is the username and second ‘postgres’ is the database name.
Connect to the localhost serverOpen pgAdmin3 and from ‘file’ on the left top corner, select add server.
You will see the screen same as mine.
Fill it out with your username and password.
As you can see there we have postgres database we created while creating a user.
Right click on the ‘databases’ to create a new database to store our data.
I have created one which reads Data Collector.
That’s the coolest name I thought of.
Sorry.
So now we have our dataset, we have to create columns (table) in which the data will be stored.
2.
2 Connecting Database and Creating TableSQLAlchemy will help us connect to our database and psycopg is the PostgreSQL wrapper for Python.
We don’t have to dwell into that, let’s just install those two libraries and move on with our code.
pip3 install flask_sqlalchemypip3 install psycopg2We have to configure our app with username, password, and database name.
After postgresql:// the first is username then a colon and then password.
And at last, that’s our database.
Once done, we then have to create an instance of SQLAlchemy.
Then comes our class.
Object-Oriented Programming to the rescue.
If you are not aware of the OOP concepts, here is a good article for you.
With help of our instance, we create a model.
In which we define table name and all the column names for which we want user input.
We don’t want to run the whole code because it is incomplete, therefore we will run this model manually.
Go to your app directory, fire up python console (simply type python), and type the following.
from app import dbdb.
create_all()As you can see, we have our table and all our columns ready to store data.
All we have to do now is write 3 lines of actual code to get things done.
2.
3 Store Data to the DatabaseAs it will check the method is POST, if it is, it will save data user has entered to the variables.
Now we have to create a session, add data to the database and commit.
There we have it.
Done.
Step 3— Deploy the Web App to a Live ServerYou already know how to create 3 required files and make your web app live on Heroku App as discussed in Part 4.
To make this web app live, first, you just have to repeat the process.
Once the web app is live, we now have to create a database on Heroku server and provide that link in our main script in place of localhost.
To do that, log in to your Heroku CLI and create a database by typing following command.
Where ‘dataforml’ is the name of my app.
Note — Make your app live first and then perform these actions.
heroku addons:create heroku-postgresql:hobby-dev –app dataformlType heroku config –app dataforml to see the URI of your database and print it to your main script (See the code below, line 9).
Now, one final step.
Just like we created the table in our local host, we have to create it on Heroku.
So, open Python shell on Heroku by typing following,heroku run pythonfrom (name of your main file) import dbdb.
create_all()exit()And there it is.
Your web data collector app up and running.
This is how my final file looks like after configuring the Heroku database (line 9).
How to know the data is stored correctly?.To check the data, you can open the dashboard on Heroku.
com or type the following in your command line,heroku pg:psql –app dataformlIt will fire up a command line query writing, run select * from data and you will see all your recorded data here.
Cool huh?Final WordsBecause there are many small processes are involved in the back end, covering them all is a difficult task.
But I think I’ve covered all that is necessary.
And about the data, I am training a gender classifier which classifies the gender based on height, weight, and shoe size.
My web app is currently live at DataForML.
herokuapp.
com and it will take a minute to respond, I hope you will provide your input.
You can find all the codes (including front end) on my GitHub repository.
If you have any doubts regarding this or the previous post I’ve referred to, feel free to reach out via Email, Twitter or Linkedin.
If you are here for the first time, read out my other post from the links below.
__Part 1 Building an Interactive DictionaryPart 2 Creating Web Maps using FoliumPart 3 Building a Website BlockerPart 4 Build and Deploy a Website using Flask and Heroku AppPart 5 Twitter Sentiment Analysis in 3 MinutesPart 6 Scraping data from FIFA.
com using BeautifulSoupPart 7 Data Collector Web Application using PostgreSQL and Flask.. More details