Python 3.4 chardet ImportError: No module named 'universaldetector'

[ghost]

Thanks for the awesome django app.

I may be missing something here:

(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ pip3 show django-csvimport

---
Name: django-csvimport
Version: 2.4
Location: /home/bob/django_mysql_test/django_env/lib/python3.4/site-packages
Requires: django, chardet
Location: /home/bob/django_mysql_test/django_env/lib/python3.4/site-packages
Requires: django, chardet
(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ pip3 show chardet

---
Name: chardet
Version: 2.1.1
Location: /home/bob/django_mysql_test/django_env/lib/python3.4/site-packages
Requires: 
(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ pip3 show django

---
Name: Django
Version: 1.7.1
Location: /home/bob/django_mysql_test/django_env/lib/python3.4/site-packages
Requires: 

However:

(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ python3 manage.py importcsv /home/bob/Documents/SGIS-wxpython/spreadsheets/MEE_Test/MEE_Test.csv > test_model.py
Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/base.py", line 503, in handle
    label_output = self.handle_label(label, **options)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/management/commands/importcsv.py", line 127, in handle_label
    warn = self.setup(mappings, modelname, charset, filename, defaults)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/management/commands/importcsv.py", line 156, in setup
    failed = self.check_filesystem(csvfile)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/parser.py", line 185, in check_filesystem
    self.csvfile = self.open_csvfile(csvfile)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/parser.py", line 44, in open_csvfile
    diagnose = chardet.detect(self.filehandle.read())
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/chardet/__init__.py", line 21, in detect
    import universaldetector
ImportError: No module named 'universaldetector'

I had to download chardet from: https://pypi.python.org/pypi/chardet2/2.0.3
Which is written to support python3?

It seems to run without issue after I extracted the files and moved them into the chardet directory in my virtualenv.

It also seems to work if I pip3 uninstall chardet and instead pip3 install chardet2

Thoughts? Am I doing it wrong? Or did something change?

Again, Thanks.

notes:

I had to do some digging and found that chardet had some syntax issues in python3.

Such as:

bob@bob-NE722:~/Documents/SGIS-wxpython/spreadsheets/MEE_Test$ python3 /usr/local/lib/python3.4/dist-packages/chardet/universaldetector.py
File "/usr/local/lib/python3.4/dist-packages/chardet/universaldetector.py", line 54
    self.done = constants.False

And then ran into this excellent article: http://docs.activestate.com/activepython/3.2/diveintopython3/html/case-study-porting-chardet-to-python-3.html

Which basically outlines everything I was having a problem with in regards to chardet

(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ pip3 show chardet2

---
Name: chardet2
Version: 2.0.3
Location: /home/bob/django_mysql_test/django_env/lib/python3.4/site-packages
Requires: 

[ghost]

I think this was an error on my part.

Apologies.

See: #50

[ghost]

Actually, what appears to have happened is that by pip3 install messytables it uninstalls chardet, and replaces it with an older version?

I opened an issue here: okfn/messytables#114

As a workaround I setup another virtualvenv for python2.7 to test out the functionality of inspectcsv.py to build the models.

And had to replace two lines in importcsv.py when using the python3.4 version, since the rows were not being split?

Traceback:

(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ python3 manage.py importcsv --model "importcsv_testapp.Csvimport" /home/bob/Documents/SGIS-wxpython/spreadsheets/MEE_Test/MEE_Test.csv
/home/bob/django_mysql_test/mysql_connector/mysql_connector/templates
Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/django/core/management/base.py", line 503, in handle
    label_output = self.handle_label(label, **options)
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/management/commands/importcsv.py", line 136, in handle_label
    errors = self.run()
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/management/commands/importcsv.py", line 204, in run
    mappingstr = self.parse_header(self.csvfile[0]) # this line breaks since it relies on messytables???
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/management/commands/importcsv.py", line 393, in parse_header
    headlist = [cleancol.sub('_', col) for col in headlist]
  File "/home/bob/django_mysql_test/django_env/lib/python3.4/site-packages/csvimport/management/commands/importcsv.py", line 393, in <listcomp>
    headlist = [cleancol.sub('_', col) for col in headlist]
TypeError: expected string or buffer

rows replaced like so since i was using tab delimited:

django-csvimport/csvimport/management/commands/importcsv.py

Line 215 in c878c73

for i, row in enumerate(self.csvfile[self.start:]):

row = row.decode().split('\t')

headlist:

django-csvimport/csvimport/management/commands/importcsv.py

Line 379 in c878c73

headlist = [cleancol.sub('_', col) for col in headlist]

headlist = headlist.decode().split('\t')

chardet:

(django_env) bob@bob-NE722:~/django_mysql_test/mysql_connector$ pip3 show chardet
---
Name: chardet
Version: 2.3.0
Location: /home/bob/django_mysql_test/django_env/lib/python3.4/site-packages
Requires: 

[edcrewe]

OK this is now fixed with messytables. Only other issue I found was libmagic can need a brew install for it to work on a mac.
Other part of using different delimiters is a duplicate of issue 31