I did something similar 10 years ago during my move to SF. I needed to find an apartment within two weeks, so I built a crawler to do complex filtering and notify me immediately.
In my case, the tool was ultimately useless because I discovered the way SF housing worked for nice apartments was 1) Openings are posted on Friday 2) Saturday, open house collects applications 3) Deal is signed on Sunday and post taken down. Anything my crawler picked up was stuff that was overpriced or in bad condition. For all my sophistication, the dataset was just the bad stuff.
Apartment hunting is a great example of needing to know the underlying behaviors of your dataset before you use it to make decisions.
In my case, the tool was ultimately useless because I discovered the way SF housing worked for nice apartments was 1) Openings are posted on Friday 2) Saturday, open house collects applications 3) Deal is signed on Sunday and post taken down. Anything my crawler picked up was stuff that was overpriced or in bad condition. For all my sophistication, the dataset was just the bad stuff.
Apartment hunting is a great example of needing to know the underlying behaviors of your dataset before you use it to make decisions.