Question

Please discuss issues and tips related to Cassandra

Answer 1

SSTABLE problem

We had a problem with DataStax Cassandra sstables.

Short description: Cassandra sstables broke randomly (mostly because great load). After broke we could not manage to repair it with the provided commands.

Environment:
- VMWare Workstation 10 / VMWare Player 6
- Guest: Ubuntu 12.04 64 bit
- CPU: 2 (i5-2500k, 3.3 GHz)
- Memory: 6 GB
- Disk: 10 GB (the underlying hardware was a Solid-State Drive; never reached more than 7GB usage)
- VMWare NAT
- No other additional (unnecessary) hardwares
- DataStax Cassandra 2.0.6 and 2.0.8 (from repository)
- JDK/JRE: (default) OpenJDK 1.7 64bit

Reproduction: write to Cassandra from the host on 128-256 threads (from a Java program using driver cassandra-driver-core), this results in full CPU and small (<10%) SSD load. With 16 threads the CPU load was 70-80%, but the problem also occurred using one thread or even when inserting a single row. If you cannot reproduce the problem, then try using 16 threads and kill the client script during import.

Behaviour: certain rows break and the driver returns with a consistency error (0 hosts replied but 1 was required). The problem only occurred when we were querying the broken records. The error log on the server contained a similar message (always with unable to seek to position):

ERROR 15:49:54,789 Error in ThreadPoolExecutor
java.lang.IllegalArgumentException: unable to seek to position 6774 in /root/import/exportdir/2/KEYSPACE/ColFam1/KEYSPACE-ColFam1-ic-3-Data.db (6523 bytes) in read-only mode
at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:306)
...

Resolution: the only resolution we found was to reinstall Cassandra (uninstall, remove pid and other files, reboot, install, config).

The problem also occurred when we sent the VM to sleep and resumed it (but it also occurred without sleeping the VM). When a record broke, we did not manage to repair or delete that.

Ágnes Barta, Lajos Cseppentő, Noémi Szilvásy

Answer 2

A bit loosely connected problem, but it took us an hour or two to solve the following issue:
We worked on a 64 bit Windows and we downloaded a 64 bit version Cygwin in order to use cqlsh conveniently to access the Cassandra database (we wanted to access the auto completion feature in the cassandra client. These auto completion features are unavailable if the shell is started from a Windows command prompt). The client was unable to start (no error message was given), so we decided to install a 32 bit Cygwin. After doing so, the cqlsh could run correctly.

Best regards,
Pineapple team

Answer 3

We've encountered several startup issues during the semester (using cassandra.bat). E.g. got the following exception recently:

INFO 12:17:50,458 Initializing system_traces.sessions
 INFO 12:17:50,465 Initializing system_traces.events
ERROR 12:17:50,757 Exception encountered during startup
FSWriteError in \var\lib\cassandra\data\system\local\system-local-tmp-jb-17-Statistics.db
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:120)
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:106)
        at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:145)
        at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:463)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:264)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)
Caused by: java.nio.file.FileSystemException: \var\lib\cassandra\data\system\local\system-local-tmp-jb-17-Statistics.db: The process cannot access the file because it is being used by another process.
        at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
        at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
        at java.nio.file.Files.delete(Files.java:1077)
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:116)
        ... 6 more
Exception encountered during startup: java.nio.file.FileSystemException: \var\lib\cassandra\data\system\local\system-local-tmp-jb-17-Statistics.db: The process cannot access the file because it is being used by another process.

Simmilar with running with Administrator privileges:

INFO 12:30:12,364 Initializing system_traces.sessions
 INFO 12:30:12,371 Initializing system_traces.events
ERROR 12:30:12,703 Exception encountered during startup   
java.lang.AssertionError: attempted to delete non-existing file system-local-tmp-jb-35-Filter.db
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:111)
        at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:106)
        at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:145)
        at org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:463)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:264)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)
Exception encountered during startup: attempted to delete non-existing file system-local-tmp-jb-35-Filter.db

In both cases the application terminated, though I'd expect Cassandra to recover from such exceptions.

Environment: Windows 8.1, Java: Oracle JDK 1.7.0_55

The easiest way to reproduce is to start Cassandra, then stop it right after
it's finished ('Listening for thrift clients ...' on the log). After about 3 to 5 starts, the exception is very likely to occur.

Answer 4

DataStax driver problem

If table in cassandra is too big, the queries will fail with NoHostAvailableException. In cqlsh the same query runs without any problem.

Exception in thread "main" com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:64)
at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:256)
at com.datastax.driver.core.ArrayBackedResultSet.fetchMoreResultsBlocking(ArrayBackedResultSet.java:197)
at com.datastax.driver.core.ArrayBackedResultSet.isExhausted(ArrayBackedResultSet.java:135)
at com.datastax.driver.core.ArrayBackedResultSet$1.hasNext(ArrayBackedResultSet.java:170)
at hu.bme.mit.namenotfound.cassandraconnector.CassandraConnector.getLaptopModelNumber(CassandraConnector.java:255)
at hu.bme.mit.namenotfound.cassandraconnector.App.main(App.java:57)
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
    at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:100)
    at com.datastax.driver.core.SessionManager.execute(SessionManager.java:417)
    at com.datastax.driver.core.ArrayBackedResultSet.queryNextPage(ArrayBackedResultSet.java:220)
    at com.datastax.driver.core.ArrayBackedResultSet.fetchMoreResults(ArrayBackedResultSet.java:212)
    at com.datastax.driver.core.ArrayBackedResultSet.fetchMoreResultsBlocking(ArrayBackedResultSet.java:195)
    ... 4 more

Answer 5

In Datastax QueryBuilder has a bug, if in where condition an Object is a String value then the getQueryString() function gives back wrong query text.

First workaround is use the toString() function to get query text.
Second workaround is build&use the querry as a Statement.

Example:

   String update = QueryBuilder.update(table).with(QueryBuilder.set("price", 25)).where(QueryBuilder.eq("manufacturer", "AMD")).getQueryString();
   System.out.println(update);

   update = QueryBuilder.update(table).with(QueryBuilder.set("price", 25)).where(QueryBuilder.eq("manufacturer", "AMD")).toString();
   System.out.println(update);

   Statement updateStatement = QueryBuilder.update(table).with(QueryBuilder.set("price", 25)).where(QueryBuilder.eq("manufacturer", "AMD"));
   System.out.println(updateStatement);

Output:

UPDATE cpu_benchmarks SET price=25 WHERE manufacturer=?;
UPDATE cpu_benchmarks SET price=25 WHERE manufacturer='AMD';
UPDATE cpu_benchmarks SET price=25 WHERE manufacturer='AMD';