missing favorites and error occurred - list index out of range [x] feed.mobile

[Arwa200]

Issue Template

Please use this template!

Initial Check

[QUESTION] What is the the way to get all favorites? Thanks.

checked the following:

• [] Python version is 3.6;
• [] Updated Twint with pip3 install --upgrade -e git+https://github.com/twintproject/twint.git@origin/master#egg=twint;
• [] I have searched the issues and there are no duplicates of this issue/question/request.

Command Ran

#!/usr/bin/python3
import twint
c = twint.Config()
c.Username = "USERNAME"
c.Since = "2018-01-01"
c.untill = "2018-12-31"
c.Limit = 20000
twint.run.Favorites(c)

Description of Issue

I have run the command above, to collect all favorite in CSV file,
I faced two problems:
first:
after a while of fetching some favorites tweets I got this error. Actually, yesterday it was bring tweets until (2018-06-07) but today bring until (2018-12-13)..
second:
it was not bring all favorites tweets, even though apply conditions like (--location, --lang, --year before 2014, --since 2015 and more, --until and --resume using last favorite's id fitch in csv) in a way to bring all favorites. it always start from beginning.

Environment Details

Using mac OS

Thank you In advance..

[pielco11]

Now as now Since and Until work only for twint.run.Search because those are the parameters that are passed to the Twitter Advanced Search.

Please note even if we add a "date-selector" for favorites, we will have to scrape every tweet and then filter out what we are not looking for. So there will not be speed improvements or something like that. Unfortunately that's how Twitter works and we can do almost nothing.

That said, it could be possible that Twitter stops returning new entities because you (as everyone else) in that case requested too many queries.

I tried and I got less tweets than expected

[Arwa200]

also another thing, why when I trying to get all favorites, it brings small amount of numbers than its recorded in twitter, then print this message

list index out of range [x] feed.mobile

(Lets say that i want them as whole bunch without any date searching or any other conditions)

[pielco11]

Got the time to try and I get that error, this needs to be addressed

[Arwa200]

also I don't really want to mix up all errors, but seems like retweet function does not work also, it gives me this message

list index out of range [x] run.Feed

[pielco11]

It was working a while ago, thanks for reporting!

[Arwa200]

you're welcome

[Arwa200]

i've tried with a new feature that you post
i got the following from log for favorites,
https://drive.google.com/file/d/1mpaos7opx9Y8W3GuI655Fd9TsZdZsBtJ/view?usp=sharing
for the last line says that no more tweet, i am sure that there more than this number, because i used an automatic scroll down to see if there is more tweet or not..

[pielco11]

Great! Thank you! That's really useful. I'm going to check it out in the next days, thank you for your patience -------- Messaggio originale -------- On 4 Feb 2019, 21:29, Arwa200 ha scritto:

…

i've tried with a new feature that you post i got the following from log for favorites, [https://drive.google.com/file/d/1mpaos7opx9Y8W3GuI655Fd9TsZdZsBtJ/view?usp=sharing](url) for the last line says that no more tweet, i am sure that there more than this number, because i used an automatic scroll down to see if there is more tweet or not.. — You are receiving this because you commented. Reply to this email directly, [view it on GitHub](#338 (comment)), or [mute the thread](https://github.com/notifications/unsubscribe-auth/AQjzJvvVyrY1c-YyGhyE08JDPP1zLM5Qks5vKJgagaJpZM4aWcde).

[pielco11]

@Arwa200 I have the solution, so Twitter is blocking the IP. Using a VPN or something solves the issue, I'm going to clean-up the code and push

Then you could try and let me know what you'll get

This might be related to #340

[Arwa200]

@pielco11 sure, so excited to try it with a new solution.

[pielco11]

Unfortunately it lasted just few times, I guess that I did not find the solution, yet. I'm still testing right now

[pielco11]

Firstly I tried with mobile.twitter.com and got the error, but the date of the last tweet is about a month before than the last scraped tweet which could still be ok since I do not get always the same amount of tweets.

Secondly I retried via twitter.com (Twint uses the mobile version, not desktop) and I can scroll down as much as I want.

Now let's go back to the first case, if I keep trying to scroll down, no more tweets are loaded. I have to wait a while and then click over "Try again". So I guess that Twitter is blocking mobile requests after a while and then asks for user interaction to keep scrolling

And everything ends here, to see the Likes of an user via desktop, you have to login

[pielco11]

@Arwa200 I hope that you will find this as an exhaustive debug operation, if you have other ideas or suggestions, just say

[Arwa200]

@pielco11 Thank you very much, appreciate your effort for building such great tool ..
well, I'm thinking if there is any way to scraping data from twitter Advanced search , while l know it is not supported favorite search just hashtags and tweets
also, have you ever try with a normal API with adding function to save last tweet's id that fetched then call for next time starting with that id?

[pielco11]

Twint automatically retries the query, even if I change the IP or wait a while (even 20 minutes) Twitter does not return more tweets

[pielco11]

#197