ew_softwarefinanceScriptspython.
exe "C:/new_software/Web Scraping/Web-Scraping/Selenium Web Scraping/scraping-lazada.
py"pauseOnce you double click on this bat file, Windows will open your command prompt and run the web scraping tool.
To schedule this double click/execution, we will hook our task scheduler to the bat file.
Configure Task in Windows Task SchedulerWindows Task Scheduler is a default Windows Application to manage tasks in response of an event based or time based trigger.
For example, you could suggest a certain click and computer actions (such as reboot) or even suggest timing like every first day of financial quarter to execute the task.
In a larger picture, this task would contain the script and metadata to define what and how the action will be executed.
You can add certain security context in the argument and control where the scheduler will run the program in.
Windows will serialize all of these tasks as a .
job files in a special folder called Task Folder.
The process flow of Task Scheduler to automate application’s web scrapingIn this tutorial, we are going to set a time based event to run our application and dump the data into SQLite.
In total thereClick on Start Windows, search for Task Scheduler, and open it.
Click Create Basic Task at the right windowChoose your trigger time.
Pick the exact time for our previous selection.
Start a programInsert your program script where you saved your bat file earlier.
Click Finish.
Let’s get started!Click on Start Windows, search for Task Scheduler, and open it.
Task Scheduler Windows2.
Click Create Basic Task at the right window.
You should put your task name (e.
g: Web scraping) and description (e.
g: Web Scraping and SQLite Dump automatically every day at 6 pm)3.
Choose your trigger time.
You will have an option to pick the time trigger in daily weekly, and even monthly.
Logically, this choice depends largely on how often you want to refresh the values from your data source.
For example, if your task is to scrape MarketWatch Stocks balance sheet, you should run the scripts every financial quarter.
4.
Pick the exact time for our previous selection.
We will pick the month January, April, July, and September to indicate all early financial quarter.
5.
Start a programHere you will be able to start the Python Scripts, send an e-mail, and even display a message.
Feel free to choose ones which you are most comfortable with.
However, you should watch out as there are deprecated tasks which will be removed in the subsequent patches.
6.
Insert your program script where you saved your bat file earlier.
This will run Task Scheduler to your Python Script for automation.
Make sure you also include Start in to the location of your application folder to access all of the relevant elements (Selenium Browser executables / SQLite Disk)7.
Click Finish.
You can check your created task schedule in your front page of Task Scheduler.
Congratulations, you have set up your first automated scheduler in Windows.
ResultHere is the gif animation for your references.
Notice how the scheduler runs the Python Scripts by itself.
Once the scripts finish running, it will dump the extracted value inside SQLite database.
In the future, this application will run every time the trigger condition is met and append the updated values into SQLite.
Running Web Scraping Scripts with Task Scheduler then append it to SQLite DiskData appended into SQLite using Task SchedulerFinally…Boy laughing while reading a book, source: UnsplashI really hope this has been a great read and a source of inspiration for you to develop and innovate.
Please Comment out below to suggest and feedback.
If you really like it, please check out my profile.
There are more about Data Analytics and Python Projects articles which will suit your interest.
Happy coding :)About the AuthorVincent Tatan is a Data and Technology enthusiast with relevant working experiences from Visa Inc.
and Lazada to implement microservice architectures, business intelligence, and analytics pipeline projects.
Vincent is a native Indonesian with a record of accomplishments in problem solving with strengths in Full Stack Development, Data Analytics, and Strategic Planning.
He has been actively consulting SMU BI & Analytics Club, guiding aspiring data scientists and engineers from various backgrounds, and opening up his expertise for businesses to develop their products .
Please reach out to Vincent via LinkedIn , Medium or Youtube Channel.