Unverified Commit c5f2551d authored by yiwu-arbug's avatar yiwu-arbug Committed by GitHub

Disable compactions on initial open of rocksdb (#29)

Summary:
On `TitanDBImpl::Open` we open rocksdb twice, and the first time is just to get list of CFs and construct table factory. Compaction can run at this point, generating SST files without custom table property. This can cause problem later on, since `OnCompactionCompleted` assume the property always exist and even use it to detect new blob file generated from the compaction. This patch disable auto compaction for the first rocksdb open to avoid such problem.

The test shown here demo the issue without disabling compaction: https://gist.github.com/yiwu-arbug/34c561d0daf4f9c293bb7b2825d77bf2

Test Plan:
Existing tests
Signed-off-by: 's avatarYi Wu <yiwu@pingcap.com>
parent b4eb0c50
...@@ -156,16 +156,32 @@ Status TitanDBImpl::Open(const std::vector<TitanCFDescriptor>& descs, ...@@ -156,16 +156,32 @@ Status TitanDBImpl::Open(const std::vector<TitanCFDescriptor>& descs,
s = env_->LockFile(LockFileName(dirname_), &lock_); s = env_->LockFile(LockFileName(dirname_), &lock_);
if (!s.ok()) return s; if (!s.ok()) return s;
// Descriptors for initial DB open to get CF ids.
std::vector<ColumnFamilyDescriptor> init_descs;
// Descriptors for actually open DB.
std::vector<ColumnFamilyDescriptor> base_descs; std::vector<ColumnFamilyDescriptor> base_descs;
for (auto& desc : descs) { for (auto& desc : descs) {
init_descs.emplace_back(desc.name, desc.options);
base_descs.emplace_back(desc.name, desc.options); base_descs.emplace_back(desc.name, desc.options);
} }
std::map<uint32_t, TitanCFOptions> column_families; std::map<uint32_t, TitanCFOptions> column_families;
// Opens the base DB first to collect the column families information. // Opens the base DB first to collect the column families information
// Avoid flush here because we haven't replaced the table factory yet. //
// Disable compaction at this point because we haven't add table properties
// collector. A compaction can generate a SST file without blob size table
// property. A later compaction after Titan DB open can cause crash because
// OnCompactionCompleted use table property to discover blob files generated
// by the compaction, and get confused by missing property.
//
// We also avoid flush here because we haven't replaced the table factory
// yet, but rocksdb may still flush if memtable is full. This is fine though,
// since values in memtable are raw values.
for (auto& desc : init_descs) {
desc.options.disable_auto_compactions = true;
}
db_options_.avoid_flush_during_recovery = true; db_options_.avoid_flush_during_recovery = true;
s = DB::Open(db_options_, dbname_, base_descs, handles, &db_); s = DB::Open(db_options_, dbname_, init_descs, handles, &db_);
if (s.ok()) { if (s.ok()) {
for (size_t i = 0; i < descs.size(); i++) { for (size_t i = 0; i < descs.size(); i++) {
auto handle = (*handles)[i]; auto handle = (*handles)[i];
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment