os.environ['HOWDOI_SEARCH_ENGINE'] FIX

[ShezaMunir]

Fixes the problem of not respecting HOWDOI_SEARCH_ENGINE environment variable, issue #377

1. --engine will be preferred over the env variable
2. In case google is one of the conflicting choices, the non-google search engine will be preferred, wherever it is specified, in the env variable or with --engine

Kindly review and let me know if any changes are needed.

[ShezaMunir]

There is some issue with duckduckgo, it is blocking the search due to which nose test is failing, so the checks are not passing

howdoi.howdoi.BlockError: Temporary block by search engine
-------------------- >> begin captured logging << --------------------
root: INFO: Fetching answers for query: make a log scale d3
root: INFO: Searching duckduckgo with URL: https://duckduckgo.com/html?q=site:stackoverflow.com%20make%20a%20log%20scale%20d3&t=hj&ia=web
urllib3.connectionpool: DEBUG: https://duckduckgo.com:443 "GET /html?q=site:stackoverflow.com%20make%20a%20log%20scale%20d3&t=hj&ia=web HTTP/1.1" 302 138
urllib3.connectionpool: DEBUG: https://html.duckduckgo.com:443 "GET /html?q=site:stackoverflow.com%20make%20a%20log%20scale%20d3&t=hj&ia=web HTTP/1.1" 403 None
root: ERROR: Unable to find an answer because the search engine temporarily blocked the request. Please wait a few minutes or select a different search engine.```

[gleitz]

Addresses #377

[gleitz]

I'm seeing the same issue with duckduckgo returning a 403. It happens on this line: https://github.com/gleitz/howdoi/blob/master/howdoi/howdoi.py#L276.

Would you please add a log message to this PR so that we get better logging output when using --explain?

    except requests.HTTPError:
        logging.info('Received HTTPError')
        result = None

I believe the issue here is that you fixed respecting HOWDOI_SEARCH_ENGINE and then a flurry of requests goes to DDG during the test and they block the request.

When you run the tests, the HTML of the search engine page is downloaded here https://github.com/gleitz/howdoi/blob/master/test_howdoi.py#L28-L32

So as long as we can get the tests to run once, the resulting HTML will be save and nosetests should pass in the future. I'm hoping if you wait a few hours the block will be lifted, and then you can put a small sleep (say, 15 seconds) after each assert in this test https://github.com/gleitz/howdoi/blob/master/test_howdoi.py#L114-L119. Then run nosetests and hopefully everything works. Then remove the sleep and check in the newly created cache files.

If that doesn't work, we may need to disable the DDG test case for now.

[gleitz]

You'll know the block has been lifted if you can successfully run

python -m howdoi make a pizza --engine duckduckgo --explain -C

[gleitz]

Hmm, it could be that DDG is just broken. I tried this on a remote machine I hadn't used howdoi on for a while and got this reponse:

> python3 -m howdoi make a pizza --engine duckduckgo --explain -C
INFO: Version: 2.0.16
Cache cleared successfully
INFO: Fetching answers for query: make a pizza
INFO: Searching duckduckgo with URL: https://duckduckgo.com/html?q=site:stackoverflow.com%20make%20a%20pizza&t=hj&ia=web
INFO: Received HTTPError
ERROR: Unable to find an answer because the search engine temporarily blocked the request. Please wait a few minutes or select a different search engine.

[gleitz]

So, let's just comment out the DDG test for now and we can fix that issue separately.

[ShezaMunir]

@gleitz I have commented out the test case for DDG in test_howdoi.py, so the checks are passing now. Will leave a reminder on issue #404 to enable the test once the issue is resolved.

[gleitz]

Very nice!