Inside main(), add these lines to fetch the page’s full HTML:data_url = “https://finance.
yahoo.
com/quote/SPY/options"data_html = requests.
get(data_url).
contentprint(data_html)This fetches the page’s full HTML content, so we can find the data we want in it.
Feel free to give it a run and observe the output.
Feel free to comment out print statements as you go — these are just there to help you understand what the program is doing at any given step.
BeautifulSoup is the perfect tool for working with HTML data in Python.
Let’s narrow down the HTML to just the options pricing tables so we can better understand it: content = BeautifulSoup(data_html, “html.
parser”) # print(content) options_tables = content.
find_all(“table”) print(options_tables)That’s still quite a bit of HTML — we can’t get much out of that, and Yahoo’s code isn’t the most friendly to web scrapers.
Let’s break it down into two tables, for calls and puts: options_tables = [] tables = content.
find_all(“table”) for i in range(0, len(content.
find_all(“table”))): options_tables.
append(tables[i]) print(options_tables)Yahoo’s data contains options that are pretty deep in- and out-of-the-money, which might be great for certain purposes.
I’m only interested in near-the-money options, namely the two calls and two puts closest to the current price.
Let’s find these, using BeautifulSoup and Yahoo’s differential table entries for in-the-money and out-of-the-money options:expiration = datetime.
datetime.
fromtimestamp(int(datestamp)).
strftime(“%Y-%m-%d”)calls = options_tables[0].
find_all(“tr”)[1:] # first row is headeritm_calls = []otm_calls = []for call_option in calls: if “in-the-money” in str(call_option): itm_calls.
append(call_option) else: otm_calls.
append(call_option)itm_call = itm_calls[-1]otm_call = otm_calls[0]print(str(itm_call) + “!.“ + str(otm_call))Now, we have the table entries for the two options nearest to the money in HTML.
Let’s scrape the pricing data, volume, and implied volatility from the first call option: itm_call_data = [] for td in BeautifulSoup(str(itm_call), “html.
parser”).
find_all(“td”): itm_call_data.
append(td.
text)print(itm_call_data)itm_call_info = {‘contract’: itm_call_data[0], ‘strike’: itm_call_data[2], ‘last’: itm_call_data[3], ‘bid’: itm_call_data[4], ‘ask’: itm_call_data[5], ‘volume’: itm_call_data[8], ‘iv’: itm_call_data[10]}print(itm_call_info)Adapt this code for the next call option:# otm callotm_call_data = []for td in BeautifulSoup(str(otm_call), “html.
parser”).
find_all(“td”): otm_call_data.
append(td.
text)# print(otm_call_data)otm_call_info = {‘contract’: otm_call_data[0], ‘strike’: otm_call_data[2], ‘last’: otm_call_data[3], ‘bid’: otm_call_data[4], ‘ask’: otm_call_data[5], ‘volume’: otm_call_data[8], ‘iv’: otm_call_data[10]}print(otm_call_info)Give your program a run!You now have dictionaries of the two near-the-money call options.
It’s enough just to scrape the table of put options for this same data:puts = options_tables[1].
find_all("tr")[1:] # first row is headeritm_puts = [] otm_puts = []for put_option in puts: if "in-the-money" in str(put_option): itm_puts.
append(put_option) else: otm_puts.
append(put_option)itm_put = itm_puts[0] otm_put = otm_puts[-1]# print(str(itm_put) + "!." + str(otm_put) + "!.")itm_put_data = [] for td in BeautifulSoup(str(itm_put), "html.
parser").
find_all("td"): itm_put_data.
append(td.
text)# print(itm_put_data)itm_put_info = {'contract': itm_put_data[0], 'last_trade': itm_put_data[1][:10], 'strike': itm_put_data[2], 'last': itm_put_data[3], 'bid': itm_put_data[4], 'ask': itm_put_data[5], 'volume': itm_put_data[8], 'iv': itm_put_data[10]}# print(itm_put_info)# otm put otm_put_data = [] for td in BeautifulSoup(str(otm_put), "html.
parser").
find_all("td"): otm_put_data.
append(td.
text)# print(otm_put_data)otm_put_info = {'contract': otm_put_data[0], 'last_trade': otm_put_data[1][:10], 'strike': otm_put_data[2], 'last': otm_put_data[3], 'bid': otm_put_data[4], 'ask': otm_put_data[5], 'volume': otm_put_data[8], 'iv': otm_put_data[10]}Congratulations.You just scraped data for all near-the-money options of the S&P 500 ETF, and can view them like this: print(".") print(itm_call_info) print(otm_call_info) print(itm_put_info) print(otm_put_info)Give your program a run — you should get data like this printed to the console:{‘contract’: ‘SPY190417C00289000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘289.
00’, ‘last’: ‘1.
46’, ‘bid’: ‘1.
48’, ‘ask’: ‘1.
50’, ‘volume’: ‘4,646’, ‘iv’: ‘8.
94%’}{‘contract’: ‘SPY190417C00290000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘290.
00’, ‘last’: ‘0.
80’, ‘bid’: ‘0.
82’, ‘ask’: ‘0.
83’, ‘volume’: ‘38,491’, ‘iv’: ‘8.
06%’}{‘contract’: ‘SPY190417P00290000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘290.
00’, ‘last’: ‘0.
77’, ‘bid’: ‘0.
75’, ‘ask’: ‘0.
78’, ‘volume’: ‘11,310’, ‘iv’: ‘7.
30%’}{‘contract’: ‘SPY190417P00289000’, ‘last_trade’: ‘2019–04–15’, ‘strike’: ‘289.
00’, ‘last’: ‘0.
41’, ‘bid’: ‘0.
40’, ‘ask’: ‘0.
42’, ‘volume’: ‘44,319’, ‘iv’: ‘7.
79%’}Setting up recurring data collectionYahoo, by default, only returns the options for the date you specify.
It’s this part of the URL: https://finance.
yahoo.
com/quote/SPY/options?date=1555459200This is a Unix timestamp, so we’ll need to generate or scrape one, rather than hardcoding it in our program.
Add some dependencies:import datetime, timeLet’s write a quick script to generate and verify a Unix timestamp for our next set of options:def get_datestamp(): options_url = “https://finance.
yahoo.
com/quote/SPY/options?date=" today = int(time.
time()) # print(today) date = datetime.
datetime.
fromtimestamp(today) yy = date.
year mm = date.
month dd = date.
dayThe above code holds the base URL of the page we are scraping and generates a datetime.
date object for us to use in the future.
Let’s increment this date by one day, so we don’t get options that have already expired.
dd += 1Now, we need to convert it back into a Unix timestamp and make sure it’s a valid date for options contracts: options_day = datetime.
date(yy, mm, dd) datestamp = int(time.
mktime(options_day.
timetuple())) # print(datestamp) # print(datetime.
datetime.
fromtimestamp(options_stamp)) # vet timestamp, then return if valid for i in range(0, 7): test_req = requests.
get(options_url + str(datestamp)).
content content = BeautifulSoup(test_req, “html.
parser”) # print(content) tables = content.
find_all(“table”) if tables != []: # print(datestamp) return str(datestamp) else: # print(“Bad datestamp!”) dd += 1 options_day = datetime.
date(yy, mm, dd) datestamp = int(time.
mktime(options_day.
timetuple())) return str(-1)Let’s adapt our fetch_options method to use a dynamic timestamp to fetch options data, rather than whatever Yahoo wants to give us as the default.
Change this line:data_url = “https://finance.
yahoo.
com/quote/SPY/options"To this:datestamp = get_datestamp()data_url = “https://finance.
yahoo.
com/quote/SPY/options?date=" + datestampCongratulations.You just scraped real-world options data from the web.
Now we need to do some simple file I/O and set up a timer to record this data each day after market close.
Improving the programRename main() to fetch_options() and add these lines to the bottom:options_list = {‘calls’: {‘itm’: itm_call_info, ‘otm’: otm_call_info}, ‘puts’: {‘itm’: itm_put_info, ‘otm’: otm_put_info}, ‘date’: datetime.
date.
fromtimestamp(time.
time()).
strftime(“%Y-%m-%d”)}return options_listCreate a new method called schedule().
We’ll use this to control when we scrape for options, every twenty-four hours after market close.
Add this code to schedule our first job at the next market close:from apscheduler.
schedulers.
background import BackgroundSchedulerscheduler = BackgroundScheduler()def schedule(): scheduler.
add_job(func=run, trigger=”date”, run_date = datetime.
datetime.
now()) scheduler.
start()In your if __name__ == “__main__”: statement, delete main() and add a call to schedule() to set up your first scheduled job.
Create another method called run().
This is where we’ll handle the bulk of our operations, including actually saving the market data.
Add this to the body of run(): today = int(time.
time()) date = datetime.
datetime.
fromtimestamp(today) yy = date.
year mm = date.
month dd = date.
day # must use 12:30 for Unix time instead of 4:30 NY time next_close = datetime.
datetime(yy, mm, dd, 12, 30) # do operations here “”” This is where we’ll write our last bit of code.
“”” # schedule next job scheduler.
add_job(func=run, trigger=”date”, run_date = next_close) print(“Job scheduled.| “ + str(next_close))This lets our code call itself in the future, so we can just put it on a server and build up our options data each day.
Add this code to actually fetch data under “”” This is where we’ll write our last bit of code.
“””options = {} # ensures option data doesn’t break the program if internet is out try: if next_close > datetime.
datetime.
now(): print(“Market is still open! Waiting until after close…”) else: # ensures program was run after market hours if next_close < datetime.
datetime.
now(): dd += 1 next_close = datetime.
datetime(yy, mm, dd, 12, 30) options = fetch_options() print(options) # write to file write_to_csv(options)except: print(“Check your connection and try again.
”)Saving dataYou may have noticed that write_to_csv isn’t implemented yet.
No worries — let’s take care of that here:def write_to_csv(options_data): import csv with open(‘options.
csv’, ‘a’, newline=’.’) as csvfile: spamwriter = csv.
writer(csvfile, delimiter=’,’) spamwriter.
writerow([str(options_data)])Cleaning upAs options contracts are time-sensitive, we might want to add a field for their expiration date.
This capability is not included in the raw HTML we scraped.
Add this line of code to save and format the expiration date towards the top of fetch_options():expiration = datetime.
datetime.
fromtimestamp(int(get_datestamp())).
strftime("%Y-%m-%d")Add ‘expiration’: expiration to the end of each option_info dictionary like so:itm_call_info = {'contract': itm_call_data[0], 'strike': itm_call_data[2], 'last': itm_call_data[3], 'bid': itm_call_data[4], 'ask': itm_call_data[5], 'volume': itm_call_data[8], 'iv': itm_call_data[10], 'expiration': expiration}Give your new program a run — it’ll scrape the latest options data and write it to a .
csv file as a string representation of a dictionary.
The .
csv file will be ready to be parsed by a backtesting program or served to users through a webapp.
Congratulations!.