A lot of users have been reporting Craigslist feeds not getting refreshed over the last few days. We are very sorry for the frustration. Here is a quick update regarding this issue:
Lots of feedly users follow craigslist feeds. As the community grows the number of connections between feedly and craigslist has been growing. There are currently about 2.8 million craigslist feeds – lots of them are search feeds.
We currently consider a Craigslist feed like any other feed and try to update it every couple of hours to make sure that users can stay current with the latest posts. The growing number of requests must have triggered a switch somewhere in the Craigslist infrastructure.
We spent the afternoon discussing this. Here is our plan:
1) We have been reaching out Craig to see if there is a way for Craigslist to increase the limit we are running into – we ran into a similar situation in September and the Craigslist team was very nice and resolved the issue quickly. We will see if they can do something similar this time.
2) We are going to run a few map reduce jobs and see what proportion of the 2.5 million feeds have been actively read over the last 2 months. We will then create some special polling rules where active Craigslist feeds continue to be polled every couple of hours while non active feeds are only polled every 2-7 days. This should optimize the integration and get us bellow the limit despite the growing community. This requires about a week of development work on our side.
So you should expect to see your Craigslist feed to be updating again as soon as the optimization is in place. If you work at Craigslist and are part of feedly community and want to help us resolve this issue faster, you can reach me at firstname.lastname@example.org
Update – Feb 20th: We have carved time between Feb 27th and March 10th to review the way feedly connect to Craigslist and optimize the polling of the 2.6M feeds the community is reading. We will keep you posted on our progress as soon as that project starts.
Update – Feb 27th: Kireet is back from vacation and has started to thinck about how to optmize the feedly+craigslist integration. We are going to try to build something custom that scales to 2.6M connections. We will update this page with our dev plans by Monday.
Update – March 3rd: We are working on making the feedly poller smarter so that it knows which craiglist feeds are being actively read and which ones are not. With that knowledge we should be able to poll different feeds at different rates and reduce the overall number of requests feedly makes to Craigslist. There is a week or two of development and testing before we can push this into production. Once the change is in place, data should start to flow again. We will update this thread as soon as we have more information (I would like to thank Kireet for his hard work on what is not an simple snake to kill).
Update – March 8th: We finished re-implementing the integration. The new polling is about 10x more efficient than the previous version. It will go out next week. We will know at that point if this is enough to the data flowing between the two systems or not.
Update – March 11th: We just pushed out the optimized version. Data should start to flow again. We should know by the end of this week if this fix is going to be enough. I would like to thank Kireet for doing the engineering work necessary to optimize the integration.
Update – March 14th: Issue resolved. See http://blog.feedly.com/2014/03/14/fix-it-march-5-craigslist-pipe-repaired/