How to OETL CSV

[ZachMurray]

Problem

OETL is not importing CSV.

Steps

1. Create a OrientDB Docker Container

docker run -d --name orientdb -p 2424:2424 -p 2480:2480 -e ORIENTDB_ROOT_PASSWORD=rootpwd orientdb

2. Get Graph-Database-Data in CSV-Format
3. Create OETL Configuration JSONs

{
    "config": {
        "log": "debug"
    },
    "begin": [],
    "source": {
        "file": {
            "path": "/home/dataverse_files/input/Anatomy.csv",
            "lock": true
        }
    },
    "extractor": {
        "row": {}
    },
    "transformers": [
        {
            "csv": {
                "separator": ",",
                "nullValue": "NULL",
                "skipFrom": 1,
                "skipTo": 1
            }
        },
        {
            "vertex": {
                "class": "V"
            }
        }
    ],
    "loader": {
        "orientdb": {
            "dbURL": "remote:localhost:2480",
            "dbUser": "root",
            "dbPassword": "rootpwd",
            "dbAutoCreate": true,
            "tx": false,
            "batchCommit": 1000,
            "dbType": "graph"
        }
    }
}

4. Copy or Mount Files onto OrientDB Docker Container

docker cp dataverse_files orientdb:/home/

5. Shell Interface into Orientdb-Container

winpty docker exec -it orientdb sh

6. Change directory to bin

cd bin 

7. Run OrientDB's ETL Tool "OETL.sh"

oetl.sh /home/dataverse_files/input/Orientdb/Anatomy-config.json

Results

Expected:

Options:

1. Data to be imported
2. Error telling me what to fix
3. Find similar questions-and-answers online

Actual:

OrientDB etl v.3.2.17 (build 211ebdf4e697bc32a41a4456f09b4c47f2a4eb4b, branch UNKNOWN) https://www.orientdb.com

XXXX-XX-XX XX:XX:XX:XXX INFO  Detected limit of amount of simultaneously open files is 1048576,  limit of open files for disk cache will be set to 523776 [ONative]create contextremote:localhost:2480
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:172)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParametersWithContext(OETLProcessorConfigurator.java:103)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:68)
        at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:121)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
        at java.lang.String.substring(String.java:1927)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.getDatabasePool(OETLOrientDBLoader.java:471)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.configure(OETLOrientDBLoader.java:409)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureComponent(OETLProcessorConfigurator.java:179)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureLoader(OETLProcessorConfigurator.java:218)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:143)
        ... 3 more

[tglman]

Hi,
you miss the database name in the db url this: "dbURL": "remote:localhost:2480" should be something like: "dbURL": "remote:localhost:2480/dbName"

[ZachMurray]

That caused to look for localhost:2480/Test/Test which isn't a remote-server/database.

Detected limit of amount of simultaneously open files is 1048576,  limit of open files for disk cache will be set to 523776 [ONative]create contextremote:localhost:2480/Test
dbName = Test
Exception in thread "main" com.orientechnologies.orient.core.exception.OConfigurationException: Error on creating ETL processor
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:172)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParametersWithContext(OETLProcessorConfigurator.java:103)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parseConfigAndParameters(OETLProcessorConfigurator.java:68)
        at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:121)
Caused by: com.orientechnologies.orient.core.exception.OStorageException: Cannot connect to the remote server/database 'localhost:2480/Test/Test'
        at com.orientechnologies.orient.core.db.OrientDBRemote.networkAdminOperation(OrientDBRemote.java:645)
        at com.orientechnologies.orient.core.db.OrientDBRemote.connectAndExecute(OrientDBRemote.java:675)
        at com.orientechnologies.orient.core.db.OrientDBRemote.connectAndSend(OrientDBRemote.java:315)
        at com.orientechnologies.orient.core.db.OrientDBRemote.exists(OrientDBRemote.java:260)
        at com.orientechnologies.orient.core.db.OrientDB.exists(OrientDB.java:293)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.getDatabasePool(OETLOrientDBLoader.java:475)
        at com.orientechnologies.orient.etl.loader.OETLOrientDBLoader.configure(OETLOrientDBLoader.java:409)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureComponent(OETLProcessorConfigurator.java:179)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.configureLoader(OETLProcessorConfigurator.java:218)
        at com.orientechnologies.orient.etl.OETLProcessorConfigurator.parse(OETLProcessorConfigurator.java:143)
        ... 3 more
Caused by: com.orientechnologies.common.io.OIOException: Error on connecting to localhost:2480
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNetworkConnection(ORemoteConnectionPool.java:48)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:55)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNewResource(ORemoteConnectionPool.java:13)
        at com.orientechnologies.common.concur.resource.OResourcePool.getResource(OResourcePool.java:107)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.acquire(ORemoteConnectionPool.java:81)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionManager.acquire(ORemoteConnectionManager.java:116)
        at com.orientechnologies.orient.client.remote.OStorageRemote.getNetwork(OStorageRemote.java:1974)
        at com.orientechnologies.orient.core.db.OrientDBRemote.networkAdminOperation(OrientDBRemote.java:632)
        ... 12 more
Caused by: com.orientechnologies.orient.enterprise.channel.binary.ONetworkProtocolException: Cannot read protocol version from remote server localhost/127.0.0.1:2480: java.io.EOFException
        at com.orientechnologies.orient.client.binary.OChannelBinaryAsynchClient.<init>(OChannelBinaryAsynchClient.java:105)
        at com.orientechnologies.orient.client.remote.ORemoteConnectionPool.createNetworkConnection(ORemoteConnectionPool.java:34)
        ... 19 more

[tglman]

My mistake as well, the port there is not the 2480(http protocol) but need to be 2424 (binary protocol), just remove it and will chose the default remote:localhost/dbName