Looks like a decent tool, but I personally would not use it in production. Took a few minutes to look through the code: They basically use the HTML pages instead of the APIs. What puts me off is that the code is missing tests altogether and has quite a few separation of concern issues. HTML extraction is everywhere, storage is embedded into the package. Cool proof of concept, not ready for production, I think.
Minor stuff: Printing instead of logging. Would prefer a package that only does the retrieval and nothing else. Hardcoded SQL(ite?) storage.
> They basically use the HTML pages instead of the APIs.
For Twitter in specific, isn't HTML scraping vastly preferable to using their official APIs? Otherwise you run into pretty arbitrary usage limits and missing features.
There's a small list of services where I think I prefer HTML scraping and browser piloting for a 3rd-party client: Twitter, Patreon, Facebook, LinkedIn, a few others. Services where the official APIs are underdeveloped or crippled to the point of almost uselessness.
I was teaching a hands on workshop (meetup) on how to use the Twitter API a few months ago, and the hardest part for everybody was getting those API keys, clicking through several pages, checking boxes, having it emailed to you.
Then it turned out Twitter refused half of the attendees the API key.. (maybe they thought it was spam coming from the same wifi, same time).
So then I just gave out my API key to the rest of the class, and in a few minutes it was blocked..
For a service that has a history of empowering users to protest and to spread news in crisis situations, it's a shame their API is so locked down.
Took me 20 seconds to install twint, and start scraping.
Took me 20 minutes to install twitter-dump, and despite a successful twitter-dump auth, I end up with 'Request exception Forbidden {"errors":[{"code":200,"message":"Forbidden."}]}'.
Not going to spend time to fix that, I'll use the dirty solution that works.
Minor stuff: Printing instead of logging. Would prefer a package that only does the retrieval and nothing else. Hardcoded SQL(ite?) storage.