As part of my banking with ABN Amro I get a credit card managed by ICS Cards.
The single most annoying thing about ICS Cards is that they don’t provide any means of exporting my statements. The next most annoying thing is that they haven’t responded to any of my emails request that feature.
Time to help myself then, in true developer fashion, I’ll write my own.
I’ve used loads of languages over the years and different languages suit different tasks. Python turns out to be my go-to language for scraping web-pages and pulling out information from them. Mostly because it’s really easy to knock up a script and because there are loads of great libraries already available to do most of the grunt work.
So here it is. A quick and dirty script to log into my account, parse the monthly statement page, fix the formatting and write it out as CSV.
This is by no means a finished script (have you seen how messy it is!!) and it’s kinda hard-coded what it will download.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
__author__ = 'Martin Gill' from bs4 import BeautifulSoup import requests import csv import re url = "https://www.icscards.nl/abnamro/mijn/accountstatements?period=201310" loginUrl = "https://www.icscards.nl/pkmslogin.form" payload = {'username': 'Username', 'password': 'password', 'login-form-type': 'pwd'} s = requests.Session() r = s.post(loginUrl, payload, verify=False) data = r.content soup = BeautifulSoup(data) print(soup.prettify()) r = s.get(url) data = r.content soup = BeautifulSoup(data) table = soup.select("table.expander-table")[0] headers = table.select("th") outputRows = [] headerItems = [] for th in headers: headerItems.append(th.getText()) outputRows.append(headerItems) rows = table.select("tr") for row in rows: cols = row.select("td") rowItems = [] for col in cols: # Replace newlines with whitespace text = col.getText().replace("\n", " ").strip() rowItems.append(text) # skips the "header" row if len(rowItems) > 0: # Replace commas with dots (Dutch decimal to English) rowItems[5] = rowItems[5].replace(',', '.') # Fix comma and eliminate currency symbol and other crap rowItems[6] = re.sub(r"^.+\s(\d+),(\d+).*", r"\1.\2", rowItems[6]) # Credit/Debit handling if re.match(r"Debet", rowItems[4]): rowItems[6] = '-' + rowItems[6] outputRows.append(rowItems) with open('test.csv', 'w', newline='') as csvfile: writer = csv.writer(csvfile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) writer.writerows(outputRows) |
Here’s an example output once the script is done with it:
|
1 2 3 |
Datum,*,Omschrijving,Card-nummer,Debet / Credit,Valuta,Bedrag 03-09-2013,,GH *GITHUB.COM xxxx xxxxxx USA Land: Verenigde Staten Merchant code: Computers,xxxx,Debet,USD 7.00,-5.41 04-09-2013,,WORLD OF WARCRAFT SUB SCUNTHORPE NO GBR Land: Verenigd Koninkrijk Merchant code: Dir Market-Continuity,xxxx,Debet,GBP 8.99,-10.85 |
0
0
0
0
0
0
0
On the business side, ICS Card Services decided in all their wisdom to only offer downloadable PDF statements; the URLs you’re referring to don’t even exist there. I’ve called and emailed them and intend to follow up; I mean, it’s 2014, every other bank and non-bank (hello Paypal) offers export functions so I’m not sure what they were thinking, but they must’ve skipped coffee that day.
Does this still work for you on the consumer card site?
I agree with your sentiments, which is why I wrote this script.
It still works, as the site I log into hasn’t changed.
It’s not an overly reliable nor stable approach, as I had to intercept and reverse engineer the login process and then do the same for the URL scheme before even getting to parsing the page. I’m not sure it even works for statements with multiple pages, as I never had any to test.
Since ICS-cards appears to have custom sites for different banks (and business as well, it seems), those pages could well be different and break the script.
While this script is probably a very good starting place, you’d probably need to customise it for the site you have to log into.
I share your frustration. Thanks a lot for this script!
Unfortunately it doesn’t work any more, they must have changed the HTML. With some tweaking here and there, I got it to work again. Here’s the 2015 version 🙂
http://pastebin.com/6uab5g3S
Thanks for the update.
I’ve not used it since the update as only have 3-4 transactions a month.
An improved version of the script (with QIF output) is in github. Feel free to make a pull request, alternatively I’ll include your changes when I get a chance.
https://github.com/MartinSGill/ICS-Cards-Download-Statements