Update regarding the Craigslist feeds [RESOLVED]

A lot of users have been reporting Craigslist feeds not getting refreshed over the last few days. We are very sorry for the frustration. Here is a quick update regarding this issue:

Lots of feedly users follow craigslist feeds. As the community grows the number of connections between feedly and craigslist has been growing. There are currently about 2.8 million craigslist feeds – lots of them are search feeds.

We currently consider a Craigslist feed like any other feed and try to update it every couple of hours to make sure that users can stay current with the latest posts. The growing number of requests must have triggered a switch somewhere in the Craigslist infrastructure.

We spent the afternoon discussing this. Here is our plan:

1) We have been reaching out Craig to see if there is a way for Craigslist to increase the limit we are running into – we ran into a similar situation in September and the Craigslist team was very nice and resolved the issue quickly. We will see if they can do something similar this time.

2) We are going to run a few map reduce jobs and see what proportion of the 2.5 million feeds have been actively read over the last 2 months. We will then create some special polling rules where active Craigslist feeds continue to be polled every couple of hours while non active feeds are only polled every 2-7 days. This should optimize the integration and get us bellow the limit despite the growing community. This requires about a week of development work on our side.

So you should expect to see your Craigslist feed to be updating again as soon as the optimization is in place. If you work at Craigslist and are part of feedly community and want to help us resolve this issue faster, you can reach me at edwin@feedly.com

Update – Feb 20th: We have carved time between Feb 27th and March 10th to review the way feedly connect to Craigslist and optimize the polling of the 2.6M feeds the community is reading. We will keep you posted on our progress as soon as that project starts.

Update – Feb 27th: Kireet is back from vacation and has started to thinck about how to optmize the feedly+craigslist integration. We are going to try to build something custom that scales to 2.6M connections. We will update this page with our dev plans by Monday.

Update – March 3rd: We are working on making the feedly poller smarter so that it knows which craiglist feeds are being actively read and which ones are not. With that knowledge we should be able to poll different feeds at different rates and reduce the overall number of requests feedly makes to Craigslist. There is a week or two of development and testing before we can push this into production. Once the change is in place, data should start to flow again. We will update this thread as soon as we have more information (I would like to thank Kireet for his hard work on what is not an simple snake to kill).

Update – March 8th: We finished re-implementing the integration. The new polling is about 10x more efficient than the previous version. It will go out next week. We will know at that point if this is enough to the data flowing between the two systems or not.

Update – March 11th: We just pushed out the optimized version. Data should start to flow again. We should know by the end of this week if this fix is going to be enough. I would like to thank Kireet for doing the engineering work necessary to optimize the integration.

Update – March 14th: Issue resolved. See http://blog.feedly.com/2014/03/14/fix-it-march-5-craigslist-pipe-repaired/


Author: @feedly

Read more. Know more.

105 thoughts on “Update regarding the Craigslist feeds [RESOLVED]”

  1. This wasn’t an issue affecting me, but I appreciate the communication. Thanks for all the interaction with the community when things like this come up. :-)

  2. This was very much an issue affecting me – while I’m sure Feedly is wonderful for many reasons, it happens that Craigslist updates are all I use it for. In fact I updated to Pro solely because I thought it was only fair that I pay for something that was working SO well for me on a daily basis. I really hope this issue gets resolved soon, either by Craigslist or with Feedly carefully seeing which feeds are updated more frequently – mine will sure be in there.

    1. Jeremy, Thank you for the comment. Please expect to get an email from Kireet who is the head behind our back end later today. I have blanked your email to make sure that you are not spammed. We very much appreciate your help! -Edwin

  3. In addition to the reduced polling frequency, I would not object to a periodic nag asking whether I wanted to keep non-active feeds.

    I had accumulated several craigslist feeds that had become anachronisms. Tonight it was really easy to remove them with the “Remove” button. But before that it had been even easier to just stop reading them.

    Presumably the nag would provide a checkbox list of the feeds in question.

    Thanks to Feedly for letting us know what’s going on, and for providing such a wonderful service!

  4. Feedly, please tell us if this will be fixed? A blog post does not replace timely communication on an issue lasting over a month and affecting many. Are you guys just hoping that Craigslist users will quietly move away to another reader? If so, please do say that upfront. Your attitude on this issue has been lackluster.

  5. Craigslist has a track record of blocking sites that do a better job of presenting CL’s data to the users. It seems they would prefer to force people to use their shitshow of a UI. So I would expect this to be more of a Craigslist issue than a Feedly issue.

      1. I was wondering about that myself. I suppose “almighty Google” is plausible; they either had the clout or the money to keep it working. Or perhaps because Feedly offers a premium option, and the CL TOS forbids taking payment to access their data. Or the algorithms that Feedly is using to access CL are more resource-intensive than the ones that GR uses. I find that hard to believe, but until either Feedly or Craigslist pipes up, all I know is that it’s broke and I’m looking for an alternative.

  6. Feedly, Please give us an update to this situation. We only use the CL feeds. our account is the Pro ver. and at the moment, are being charged for a non functional service. I understand CL may be at fault here but if the problem wont be resolved, please let us know so we can cancel it. Any update is better than not recognizing the concerns of your users.

  7. What the hell is going on? Can anyone recommend a decent alternative to Feedly? The others I’ve used have shitty UI’s. Why is feedly being blacklisted by Craigslist?

  8. Feedly is so superior to others sites I have tried. I sure hope Feedly and craigslist’s get there problems worked out soon, as I sure depend on Feedly..

  9. I posted a reply about this back on January 30, saying how much I depend on Feedly (I pay for Pro services). Now it’s been most of a month with no fix or even an update on progress being made. Very disappointing. If it doesn’t get fixed, I’ll be one those canceling their Pro membership.

  10. Craigslist updates are my only reason for using Feedly. Without them, Feedly is worthless.

    Awful customer service to not update your users after this long. Shame.

  11. At least I can still click on the Craigslist feed name in Feedly and that serves as a bookmark to a Craigslist search page. Most of these searches I rarely read, but there are a few that I check every few days.

    1. That’s a great tip actually Duke. It never occurred to me to click over from the feed. A few extra mouse clicks but at least I can get to them. CL feeds going down has been my biggest nuisance using Feedly.

  12. It’s been well over a week, an update on this is far overdue. If the Craigslist folks are being jerks, then let us know. We’ll happily go pound on their door.

  13. Thanks for the work on this, I just came here to look for a new feed reader but after reading this I will give it a little time. It has been way too long but I won’t complain considering I am a freeloader. Thanks again for the service.

  14. So my only concern, and maybe you can answer this, is that I subscribe to Craigslist searches that have no results sometimes. Rare products, or specific price ranges, that don’t list any products but I watch that RSS to see if any pop up. Would this be considered a non-read feed? I don’t think it’s good to wait a week to see the RSS feed pop up on a product that is clearly rare or at a valuable price point.

    How do you suggest that issue be addressed?

    1. This could be an issue if they feeds you subscribe to post less than 1 post every 2 months. We are going to continue to crawl those feeds by once very 2 weeks.

  15. I am unable to re-add my CL feed. It’s giving me a 404. Is this a known issue (as in purposely not enabled yet)?

    1. you should be able to add them as you would any other feed. just copy the link from the RSS button on craigslist and paste it into the “Add Content” box in feedly.

  16. Thanks for getting CL feeds again Feedly, but now not only am I seeing my local items, it is pulling in all the posts below the “Few LOCAL results found. Here are some from NEARBY areas”. This is giving me hundreds of hits from hundreds of miles away but I just want to know if I can buy a patio table on my local CL. Thank

  17. i’m also having major issues with my recent feeds. 2 days ago i started getting feeds from weeks and months back, and i started getting duplicate feeds sometimes 3-4x. after marking the topic read the same rss will import non-stop. I normally get 1-2k feeds daily but now it’s more like 6k+. also the feeds aren’t in chronological order either.

  18. still not working. does anyone know of a reliable news feeder? Of course, if you did, you wouldn’t be reading this! I had such high hopes for feedly.

  19. Frustrating indeed. I really rely on being able to do good Craigslist searches as part of my income. That’s why I’m happy to be a paying subscriber. But I’m certainly looking for an alternative.

  20. Feedly has decided they’re done trying to work with craigslist and they wish you all the best.

  21. craigslist feeds down again 4 days and counting. I thought feedly was the alternative to google reader?

  22. Yeah, its been down and back up a few times over the past month and a half. I tried to remain patient because I love Feedly but its not over for me. I LOVE Digg Reader!!

  23. Almost certain I’m going to give up on feedly – Craigslist searches are the only thing I use it for, and that hasn’t worked for about 1/3 of 2014. Nobody, from paid-up Pro members like me to people trying it for the first time, is going to want to stick with a service that unreliable. It’s unfortunate, because when Craigslist searches on feedly do work, it’s PERFECT – the thoroughness, the speed, the leaving out results I’ve already seen, etc.

  24. This is broken again! Craigslist feeds are the most important ones to me. Guess I am looking for a new reader :(

  25. Looks like the issues are back again. Anyone else having problems? My CL feeds stopped working again

  26. It is ridiculous that Feedly started this discussion with a proud “RESOLVED”, then when the problem comes up again nobody from Feedly posts anything for weeks. I’ll double-check the posts from other users above for this, but can anyone suggest another service that does what Feedly used to do with Craigslist results?

    1. Was an avid user of Google Reader and they NEVER had this problem. I’m sure tons more Craigslist RSS traffic was generated back then, so it is puzzling why Feedly & Craigslist can’t seem to get it together. What I find most upsetting, however, is that Feedly reps are not being very forthcoming with what, if anything, they intend to do about the problem. Last response from them in this post goes back to March?? Just be upfront; if it ain’t getting fixed just tell us…

  27. This keeps happening! It’s not just happening to feedly. Digg reader and the old reader have no CL function either. I’ve been e-mailing CL daily and have heard nothing…. what’s the deal? This is weird, no one is reporting anything.

      1. Simply don’t care? Well, maybe you are going to care when enough off us stop using feedly. Good luck trying to get me to pay while this is still not working. To me this is a basic software architecture and design flaw. Feedly (as a system) requests CL feeds for all users in a centralized fashion (presumably to enable social features). That means that from CL perspective, feedly is making a lot of requests which could be labelled as spam or mining, etc’. A much better design would have been to have individual clients (be your browser or mobile app) check for updated CL feeds directly. In this case the request will be coming from the user and not feedly. The client can then reach back to feedly in support of social features. TBH, i would much rather have the CL feeds working and not “social features.” This have further impact on the feedly refresh rate (which at the moment is still not satisfactory). When you hit refresh in feedly, you are not actually checking for updated feeds directly, you are merely polling the feedly servers for updates. Under a more client side centric architecture, clients would poll directly for updates which would provide for a much faster refresh rate (i.e. hitting “refresh” would actually check the source of the feeds directly for updates). Please consider changing your approach…

  28. still not working for me ??? Im am not getting refreshed lists. I mite as well just go to Craigslist and check the new listings, there’s no point to use feedly any more for me.

Comments are closed.