Skip to content

java.lang.IllegalStateException: Batch statement cannot contain more than 65535 statements. #384

Closed

Description

@baibingzhi

复现步骤

  1. 创建属性,顶点,创建边,都有索引。
  2. 导入数据,自己做的数据,约200万顶点,100万边
  3. 删除schema,删除的顺序是 边索引->边->顶点索引->顶点->属性

现象:
数据量小时步骤都正常。后来导了一批数据,大约200万顶点100万边,在删除第一个边的索引时报错,请问如何解决,谢谢!

具体报错

2019-03-08 10:32:29 666937320 [task-worker-3] [WARN ] com.baidu.hugegraph.task.HugeTask [] - An exception occurred when running task: 331
java.lang.IllegalStateException: Batch statement cannot contain more than 65535 statements.
at com.datastax.driver.core.BatchStatement.add(BatchStatement.java:154) ~[cassandra-driver-core-3.2.0.jar:?]
at com.baidu.hugegraph.backend.store.cassandra.CassandraSessionPool$Session.add(CassandraSessionPool.java:156) ~[hugegraph-cassandra-0.8.0.jar:?]
at com.baidu.hugegraph.backend.store.cassandra.CassandraTables$SecondaryIndex.delete(CassandraTables.java:566) ~[hugegraph-cassandra-0.8.0.jar:?]
at com.baidu.hugegraph.backend.store.cassandra.CassandraStore.mutate(CassandraStore.java:201) ~[hugegraph-cassandra-0.8.0.jar:?]
at com.baidu.hugegraph.backend.store.cassandra.CassandraStore.mutate(CassandraStore.java:174) ~[hugegraph-cassandra-0.8.0.jar:?]
at com.baidu.hugegraph.backend.tx.AbstractTransaction.commitMutation2Backend(AbstractTransaction.java:243) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.backend.cache.CachedGraphTransaction.commitMutation2Backend(CachedGraphTransaction.java:135) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.backend.tx.IndexableTransaction.commit2Backend(IndexableTransaction.java:57) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.backend.tx.AbstractTransaction.commit(AbstractTransaction.java:166) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.backend.tx.GraphTransaction.commit(GraphTransaction.java:289) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.HugeGraph$TinkerpopTransaction.doCommit(HugeGraph.java:650) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:105) ~[gremlin-core-3.2.5.jar:3.2.5]
at com.baidu.hugegraph.HugeGraph$TinkerpopTransaction.commit(HugeGraph.java:612) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.job.schema.IndexLabelRemoveCallable.removeIndexLabel(IndexLabelRemoveCallable.java:64) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.job.schema.IndexLabelRemoveCallable.execute(IndexLabelRemoveCallable.java:39) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at com.baidu.hugegraph.job.Job.call(Job.java:36) ~[hugegraph-core-0.8.0.jar:0.8.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at com.baidu.hugegraph.task.HugeTask.run(HugeTask.java:203) [hugegraph-core-0.8.0.jar:0.8.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]

Activity

  1. added 2 commits that reference this issue on Mar 8, 2019
    c3e34b7
    69c53b7
  2. zhoney commented on Mar 8, 2019

    @zhoney
    Contributor

    @baibingzhi 感谢反馈!

    这是由于删除IndexLabel时会同时删除对应的所有索引信息,Cassandra后端删除时超过了65535的限制(Cassandra client本身的限制)。目前已经改进为分批处理,参见 #386 #387

    0.8版本的删除schema导致的删除顶点或者边,实现效率相对比较低,在大数据量的情况下,所需时间较长。该问题在0.9版本会优化。

    对于Cassandra这种标记删除的后端,不建议进行大数据量的反复删除,墓碑会降低后续查询的效率,请慎重使用大规模删除。实在需要的话,也要合理配置Cassandra的compaction,清理墓碑。

  3. added a commit that references this issue on Mar 8, 2019
    f4031f1
  4. added a commit that references this issue on Mar 11, 2019
    88e8285
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions