Twitter data scraping: API vs Advabced Search

Question

Need to analyze tweets on a specific topic, my application (for using Twitter API) was not approved. Instead, I tried to do it manually using Twitter Advanced Search. However it's more burdensome than using easy-to-use Twitter API, I think advanced search doesn't retrieve all the relevant tweets containing a particular keyword, as I tested it for several case.

So, there are two questions. Firstly, am I right about incompleteness of advanced search in providing results? Secondly, is there another way (or turn around) to use API without need to approval?

Specifically, is there any limit for returning results by advanced search or it provides all the possible results, just like API?

Answer 1

I've found snscrape to work like a charm.

Github page: https://github.com/JustAnotherArchivist/snscrape

Make sure to double-check the rights and intended use, given that your application was not approved by Twitter. Wouldn't want to be doing anything illegal of course.

Answer 2

On question 1, "am I right about incompleteness of advanced search in providing results?" Yes, this is my recent experience, as of May 4, 2023. My still working API v2 returns over 1,000 tweets while the same search using advanced search after logging in returns less than 200.

On question 2, " is there another way (or turn around) to use API without need to approval?" Not that I know of. There used to be limited free API search and paid API search, as well as free academic API search. Going forward, only paid (very expensive) API search. I think they may have stated somewhere that the might consider academic access again, but who knows if and when.

Twitter has, at least temporarily, removed the ability to search without logging in, breaking scraping tools like snscrape. You can check on the snscrape GitHub page, under issues, to see if this changes.