v1.0.0-alpha01

Major features

• Improve query performance
  • Utilize the bloom filter stored in the sst meta data to do filtering on the row groups according to the provided predicate
  • Refactor sst cache based on weighted LRU.
  • Add a new async parquet reader.
  • Support get_range api in aliyun oss to avoid OOM when sst is large.
  • Add memory based cache for object store.
• Improve CeresDB cluster mode
  • Implement distributed wal based on kafka.(unstable)
  • Implement expansion and transferLeader control with ceresmeta.(unstable)
  • Refactor cluster metadata management in ceresemta.
• Bug fix
  • Fix some bugs in WAL on OBKV.
  • Fix encoding bug in hybrid storage format.
  • Correct the order of the columns to be consistent with the order in which the user created the table. by @dust1
• SDK
  • Release ceresdb-golang-client
  • Replace grpc-io with tonic in ceresdb-rust-client

What's Changed

• chore: fix error message when table is not found by @ShiKaiWi in #348
• docs: Table operation doc by @MachaelLee in #318
• fix: adopt new parquet api by @jiacai2050 in #351
• refactor: utilize datafuison to support filter before merge procedure by @ygf11 in #326
• Refactor replace protobuf with prost by @ShiKaiWi in #352
• feat: region meta in wal manager on kafka by @Rachelint in #345
• docs: proposal of roadmap by @archerny in #356
• feat: add async parquet reader by @jiacai2050 in #355
• feat: encoding in wal's message queue impl. by @Rachelint in #358
• feat: add tools sst-convert by @jiacai2050 in #361
• fix: apply missing filtering on parquet reader by @ShiKaiWi in #360
• docs: update dynamic cluster mode doc by @ZuLiangWang in #357
• feat: replace ParquetExec with ParquetRecordBatchStream by @jiacai2050 in #368
• feat: region in wal's message queue impl. by @Rachelint in #362
• feat: support equal pruner by @ShiKaiWi in #372
• feat: add block list by @MachaelLee in #359
• feat: enhancement to mq's kafka implementation by @Rachelint in #375
• fix: add ignore to test on message queue's kafka implementation. by @Rachelint in #377
• fix:Disallow defining timestamp key column as Tag column by @QuintinTao in #374
• feat: add table id to LogEntry by @Rachelint in #379
• feat: add bloom filter when write sst by @jiacai2050 in #370
• feat: support like syntax in show tables statement by @QuintinTao in #331
• feat: support filter_row_groups by EqPruner by @ShiKaiWi in #380
• refactor: build row groups from record batch stream by @ShiKaiWi in #381
• refactor: show tables by @jiacai2050 in #382
• refactor: wal unit tests. by @Rachelint in #387
• feat: filter row groups by bloom filter by @ShiKaiWi in #383
• feat: implement region recovery by @Rachelint in #385
• feat: read all bytes out before do async bench by @jiacai2050 in #390
• feat: meta cache for both custom and parquet metadata by @ShiKaiWi in #392
• feat: sst-convert support convert to hybrid format by @jiacai2050 in #398
• feat: support get_range in aliyun oss by @ShiKaiWi in #395
• refactor: support multiple writing in a specific table by @Rachelint in #393
• fix: add meta client timeout by @ZuLiangWang in #399
• chore: update obkv-client to latest version by @chunshao90 in #401
• fix: wrong cargo lock by @ShiKaiWi in #405
• feat: correct column order by @dust1 in #340
• chore: update the ceresdbproto to latest by @ShiKaiWi in #406
• chore: remove manual installation of protoc by @ShiKaiWi in #407
• feat: refacor existing wal implementations to support scanning logs of shard by @Rachelint in #400
• chore: bump client in test harness by @jiacai2050 in #409
• docs: update http api by @jiacai2050 in #410
• test: add case sensitive demo by @jiacai2050 in #411
• feat: add memory based cache for object store by @jiacai2050 in #415
• feat: Impl wal on mq namespace by @Rachelint in #404
• feat: Impl wal on mq setup by @Rachelint in #416
• fix: cleaner bugs of wal on obkv by @Rachelint in #418
• feat: replay from flushed sequence by @Rachelint in #423
• feat: make manifest operations more configurable by @Rachelint in #424
• feat: setup MemCacheStore by @jiacai2050 in #421
• docs: fix broken conventional commit document link(#429) by @Huachao in #430
• fix: fix grpc service metrics by @chunshao90 in #431
• fix: purging buckets in initializing bug by @Rachelint in #432
• fix: load last sequence in obkv wal by @Rachelint in #434
• fix: drop bug of wal on mq, can't not drop runtime by @Rachelint in #433
• fix: hybrid storage encode bug in multi record batch by @chunshao90 in #426
• chore: add logs about replaying by @Rachelint in #438
• chore: return schema not found when schema not exists by @chunshao90 in #439
• refactor: manifest wal config by @Rachelint in #437
• fix: bug in distributed wal(obkv, kafka) by @Rachelint in #422
• chore: bump version to 1.0.0-alpha01 by @jiacai2050 in #440

New Contributors

• @MachaelLee made their first contribution in #318
• @ZuLiangWang made their first contribution in #357
• @Huachao made their first contribution in #430

Full Changelog: v0.4.0...v1.0.0-alhpa01

v0.4.0

Major features

• Improve CeresDB cluster mode
  • Refactor the Catalog and Cluster module for better interaction with CeresMeta
  • Refactor the create/drop table procedure
  • Support open/close shards on the ceresdb-server controlled by CeresMeta
• Support default value option of column by @ygf11
• Replace grpc-io with tonic
• Upgrade to datafusion 12 and arrrow 23
• Improve the CI
  • Support nightly benchmark with TSBS
  • Reduce the size of CeresDB binary compiled with release profile
  • Fix the random failure in the integration tests
• Bug fix
  • Avoid extra flushes when triggering one flush on table by @dust1

What's Changed

• refactor: remove SchemaIdAlloc and TableIdAlloc by @waynexia in #238
• chore: remove stale docs by @waynexia in #239
• chore: correct integration test's dir by @waynexia in #242
• fix:fix license check by @chunshao90 in #243
• fix: flush of one table might be triggered multiple times by @dust1 in #236
• chore: fix submodule init by @ShiKaiWi in #244
• feat: add scan batch size options by @jiacai2050 in #248
• feat: primary key definition support specify tsid column by @jiacai2050 in #254
• refactor: use TableManager to implement CatalogManager and SchemaManager by @waynexia in #260
• feat: collapsible columns supports variable length type by @chunshao90 in #241
• feat: base implementation of default value by @ygf11 in #246
• feat: add tsbs CI by @jiacai2050 in #264
• fix: null buffer assert by @jiacai2050 in #262
• feat: Implement some cluster commands by @waynexia in #265
• feat: persist collapsible info in parquet meta_data by @chunshao90 in #261
• chore: update parquet-testing by @chunshao90 in #266
• test: make harness test stable by @ShiKaiWi in #270
• chore: Upgrade to datafusion 12 & arrow 23 by @chunshao90 in #269
• feat: replace grpcio with tonic by @ShiKaiWi in #267
• chore: make use of workspace dependencies by @ShiKaiWi in #274
• build: fix test_panic_hook UT by @ShiKaiWi in #275
• docs: update example config for static deployment by @ShiKaiWi in #276
• test: add test case upgraded arrow&datafusion by @chunshao90 in #272
• docs: update complie and run chapter by @chunshao90 in #281
• feat: create schema in static routing by @chunshao90 in #285
• chore: adapt to latest ceresdbproto by @ShiKaiWi in #286
• refactor: remove event handling from cluster by @ShiKaiWi in #289
• feat: enhance sql handle, accept both json and text request by @jiacai2050 in #287
• feat: support flush tables periodically by @chunshao90 in #290
• chore: fix CI cache key, release mode should not mix with debug by @jiacai2050 in #292
• feat: support meta event service by @ShiKaiWi in #293
• feat: bump object store to 0.5.1, same with datafusion by @jiacai2050 in #296
• feat: support default value option of column like mysql by @ygf11 in #278
• feat: remove cluster catalog and refactor volatile catalog by @ShiKaiWi in #295
• refactor: modify exist wal by @Rachelint in #288
• chore: rename ShardTables to TablesOfShard by @ShiKaiWi in #300
• ci: support check disk quota by @ShiKaiWi in #301
• feat: add TabelCreator/TableDroper for executing create/drop table by @ShiKaiWi in #303
• fix: avoid running blocked-on task when building meta client by @ShiKaiWi in #305
• feat: support open&close shards by @ShiKaiWi in #307
• refactor: centralize the logic of choosing worker by @QuintinTao in #311
• chore: support set log file when running test harness by @jiacai2050 in #314
• test: set correct exit code for test harness by @jiacai2050 in #319
• feat: do create drop table by @ShiKaiWi in #310
• chore: minimize release binary size by @chunshao90 in #322
• feat: impl create drop table service by @ShiKaiWi in #323
• build: set correct version and let member packages inherit it by @archerny in #325
• chore: keep SQL original ident by @jiacai2050 in #328
• feat: add async inner to BatchLogIteratorAdapter by @Rachelint in #320
• chore: add quick release profile by @jiacai2050 in #330
• feat: separate status code between storage service and meta event service by @ShiKaiWi in #332
• fix: use right table manipulator when setup by @ShiKaiWi in #333
• feat: message queue and its kafka impl by @Rachelint in #306
• refactor: make mq interface simpler. by @Rachelint in #337
• fix: make open shard idempotent by @ShiKaiWi in #338
• fix: add missing version for meta request by @ShiKaiWi in #339

New Contributors

• @QuintinTao made their first contribution in #311

Full Changelog: v0.3.1...v0.4.0

v0.3.0

Major features

• Implement hybrid format. Check document below to see how to use
  • https://docs.ceresdb.io/analytic_engine/options.html
• Implement distributed cluster

What's Changed

• build(ci): 🔨 not run the build and push CI on forks by @zwpaper in #148
• build(ci): Test docker image build by @zwpaper in #150
• build(image): 🔨 replace entrypoint with bash, drop supervisor by @zwpaper in #152
• chore: Add build guidance for mac users by @tianlinzx in #169
• chore: add docker hub description workflow by @jiacai2050 in #130
• chore: add function tests by @Rachelint in #219
• chore: bump rust toolchain edition to 2021 by @waynexia in #180
• chore: bump sqlparser to 0.19 by @Rachelint in #172
• chore: change GH issue template to yaml format for better experience by @waynexia in #181
• chore: check Cargo.lock in CI by @jiacai2050 in #223
• chore: fix some grammar errors in docs by @ShiKaiWi in #200
• chore: remove components/tracing and components/tracing-example by @waynexia in #184
• chore: switch datafusion fork by @waynexia in #203
• chore: update slog-global dep to https://github.com/tikv/slog-global by @ShiKaiWi in #195
• chore: update the async_trait used by wal to 0.1.53 by @ShiKaiWi in #199
• docs: 🔨 fix slack dead link by @zwpaper in #141
• docs: Minor Grammar and Concision Changes in README.md by @jakemcf22 in #142
• docs: Update user guide summary by @jiacai2050 in #149
• docs: add data model in user guide by @chunshao90 in #132
• docs: add integration test section to CONTRIBUTING.md by @waynexia in #160
• docs: add quick start by @jiacai2050 in #133
• docs: add sql chapter for user guide by @waynexia in #136
• docs: add user guide workflow to generate html pages by @jiacai2050 in #137
• docs: fix build dir by @jiacai2050 in #197
• docs: improve options section by @jiacai2050 in #222
• docs: move architecture to user-guide by @jiacai2050 in #186
• docs: update Roadmap.md by @waynexia in #176
• docs: user guide about static routing by @ShiKaiWi in #134
• feat: Simplify SQL syntax that specifies timestamp key column by @dust1 in #140
• feat: RoleTable trait and LeaderTable implementation by @waynexia in #188
• feat: WalReplicator skeleton implementation by @waynexia in #179
• feat: add new table_option storage format by @jiacai2050 in #218
• feat: add wal write benchmark by @ygf11 in #178
• feat: meta client supports route_tables by @ShiKaiWi in #214
• feat: persist AlterSchema and AlterOptions to data WAL by @waynexia in #166
• feat: read hybrid format by @jiacai2050 in #207
• feat: read hybrid format by @jiacai2050 in #208
• feat: setup in different deployment mode by @ShiKaiWi in #190
• feat: support double quoted string. by @Rachelint in #196
• feat: support route in cluster mode by @ShiKaiWi in #215
• feat: write hybrid storage format by @jiacai2050 in #185
• feat: support to route for missing tables by @ShiKaiWi in #221
• fix: ci cache dependencies by @jiacai2050 in #193
• fix: deadlock when reconnect heartbeat channel by @ShiKaiWi in #217
• fix: insert negative number value by @ShiKaiWi in #212
• fix: select single quoted literal string. by @Rachelint in #173
• fix:select system.public.tables will return error by @dust1 in #174
• refactor: avoid validation for table constraints during parse by @ShiKaiWi in #164
• refactor: build Payload in encode_batch for LogBatchEncoder by @ygf11 in #209
• refactor: enhance cluster module based on meta_client_v2 by @ShiKaiWi in #100
• refactor: make read_batch object safe in WalManager by @ygf11 in #119
• refactor: make write method object safe in WalManager by @ygf11 in #159
• refactor: remove Meta and Wal type params in Instance by @ygf11 in #163
• refactor: remove old meta_client crate by @ShiKaiWi in #205
• refactor: replace ColumnOption::DialectSpecific with ColumnOption::Comment by @baojinri in #167
• refactor: store SpaceId and TableId in TableImpl by @waynexia in #202
• refactor: try to replace dynamic-dispatch with pre-code in wal write by @ygf11 in #189
• refactor: volatile implementation for catalog by @ShiKaiWi in #157

New Contributors

• @zwpaper made their first contribution in #141
• @jakemcf22 made their first contribution in #142
• @tianlinzx made their first contribution in #169
• @Rachelint made their first contribution in #172
• @baojinri made their first contribution in #167

Full Changelog: v0.2.0...v0.3.0

v0.2.0

Overview
In the v0.2.0 version, CeresDB implemented a static distributed deployment solution and completed some preparatory work for a cloud-native distributed cluster solution.
In addition, related documents have been improved to facilitate developers to use and understand CeresDB.

Features

• Supports Aliyun OSS #20
• WAL implementation based on OBKV #62
• Distributed version supports static topology(only supports routing in grpc protocol)
• Supports MySQL communication protocol #56
• Improve the integration testing framework #86

Thanks
@dust1 @ygf11 @messense @GoGim1 @li-jin-gou @infdahai @VirrageS