To support dictionary compression we added a v2 blob file header to store extra flags. However, the check in BlobFileReader , when reading a v1 blob file, falsely asserted the file header is shorter than expected:
https://github.com/tikv/titan/blob/ecb5cba016096309cbd9b566bdc4ce62307527a0/src/blob_file_reader.cc#L103
https://github.com/tikv/titan/blob/ecb5cba016096309cbd9b566bdc4ce62307527a0/src/blob_format.h#L394
Refactoring the code to bypass the check to fix the issue.

The issue is introduced in https://github.com/tikv/titan/pull/189. TiKV is affected when Titan is enabled and upgrade from pre-5.0 versions to >=5.0.0 versions. It will make TiKV fall in crash loop.

Also adding a titan_blob_file_dump tool to dump blob file content.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

In our CI environment we need to link pthread to enable sanitizer builds (ASAN, TSAN, UBSAN). Update cmake script for it.

It still needs #218 to get the CI pass.

Also disabling NoSpaceLeft test just like in Travis.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

This change is to fix ASAN test failure in CI.

The rocksdb logging.h file provides the ROCKS_LOG_XXX logging macro, and it contain a logic to shorten the source file being print. It use the length of its file path (e.g. "/path/to/rocksdb/logging/logging.h")  to truncate other source file path. Since Titan source could be in different directory, Titan source file path could be truncated incorrectly, or even cause out-of-range access to the string buffer. Copying the file into titan repo and make a Titan version of the macros to fix the problem.

Also doing clang format.
Signed-off-by: Yi Wu <yiwu@pingcap.com>
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* Add checkpoint support
Signed-off-by: Myth <caipengbo@outlook.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Specify IOPriority when creating blob files. Fix #209.
Signed-off-by: BaronStack <2689496754@qq.com>
Co-authored-by: Xinye Tao <xy.tao@outlook.com>

In our CI mac build, homebrew update often timeout. Mitigate this issue by upgrading Xcode image (spec was ineffective before).
Also optimize build time by adding format stage and fail_fast flag so that failed test can trigger immediate termination.
Signed-off-by: tabokie <xy.tao@outlook.com>

Adding a tool to dump Titan manifest file, for easy debugging.

Sample output:
```
next_file_number: 93015
column family: 1

column family: 2

column family: 3

column family: 0
file 92978, size 44644407, level 0
file 92997, size 44513780, level 0
file 93001, size 44639151, level 0
file 93005, size 44420372, level 0
file 93009, size 44595346, level 0
file 93010, size 23167116, level 0
file 93011, size 38966307, level 0
file 93012, size 21745288, level 0
file 93014, size 44181211, level 0
```
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Signed-off-by: Zhenhan Gong <zhenhan.gong@gmail.com>

Do the same cmake change in https://github.com/tikv/rocksdb/pull/219Signed-off-by: Yilin Chen <sticnarf@gmail.com>

This is a subtask of issue: https://github.com/tikv/tikv/issues/5743

> Read blob from file with dictionary compression

Mainly made following changes:

- Refactored `BlobDecoder`, add decompression related components in it.
- Refactored `BlobFileReader` and `BlobFileIterator`, initialize the decoder with corresponding data in blob files.
- add testcases

Issue Number: https://github.com/tikv/tikv/issues/8635

Fixes #118

Briefly I've done following things:
- add compression options in TitanCFOptions
- set flags in header when there is compression dictionary
- add some members(states, sample data, etc.) in BlobFileBuilder for dictionary compression logic
- write the dictionary in meta blocks, when dictionary compression is enabled
- refactor BlobEncoder to support dictionary update

currently for decoding
- update BlobFileReader and BlobFileIterator to return Status::NotSupported if a blob file come with dictionary compression.
- add a test case for BlobFileReader

Issue Number: https://github.com/tikv/tikv/issues/8635Signed-off-by: hexyoungs <chuxdesign@hotmail.com>

Update travis script to enable compression in tests. needed by #189

Also exclude NoSpaceLeft test from travis.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* separate deleteblobfiles from deletefiles (#185)
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Looks like it's not a undefined behavior to get a nullptr from compaction filter factory

Fixes #180 
Signed-off-by: DorianZheng <xingzhengde72@gmail.com>

* add more test cases
Signed-off-by: Connor <zbk602423539@gmail.com>

* fix cocurrent compaction while flush
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Closes #163 
Signed-off-by: Xu Qiaolun <Jamesxql@Gmail.com>

* fix wrong live data size when encounter rewrite failure
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* add deletefilesinrange during gc case
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* check obsolete before apply
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

After creating a new blob file meta, the live data size is 0. The stat is counted when adding the blob file but with live_data_size == 0. So we should update the stat when the live data size is initialized.
Signed-off-by: Connor <zbk602423539@gmail.com>

This is a new interface which is added to RocksDB recently and TiKV will call it if `enable_multi_thread_write` of `DBOptions` is true. So Titan must implement it by self.
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

exclude metadata size from file size, otherwise discardable ratio would be wrong when the blob file is small.
Signed-off-by: Connor <zbk602423539@gmail.com>

* titan level blob file count
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* store stats in rocksdb statistics
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Fix warn when titan is compiled with gcc 5.5.0
```
/home/pingcap/.cargo/git/checkouts/rust-rocksdb-e0eeb217423298aa/0e1a085/librocksdb_sys/libtitan_sys/titan/src/db_impl_gc.cc: In member function ‘rocksdb::Status rocksdb::titandb::TitanDBImpl::BackgroundGC(rocksdb::LogBuffer*, uint32_t)’:
/home/pingcap/.cargo/git/checkouts/rust-rocksdb-e0eeb217423298aa/0e1a085/librocksdb_sys/libtitan_sys/titan/src/db_impl_gc.cc:205:55: error: ‘gc_merge_rewrite’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
                           &shuting_down_, stats_.get());
```

Signed-off-by: liumh8 <571621369@qq.com>

Fixes #144 

* Green Blob GC by merge operator
Signed-off-by: tabokie <xy.tao@outlook.com>

Summary:
* On DB reopen, iterate all SST file's properties and sum up live data size of all blob files to use as GC stats.
* Remove previous logic to set GC mark on blob files on reopen.
* Refactor `OnFlushCompleted` and `OnCompactionCompleted`. Check file state is `kPendingLSM` before transit file state. 
* Refactor `BlobFileMeta`.

Test Plan:
Updated unit tests
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
When adding new blob file, titan directory need to be fsync, otherwise files can be lost after power outage.

Test Plan:
run `strace -yfe fsync ./titan_titan_db_test --gtest_filter=*.Snapshot` and see titan directory being fsync-ed in the output.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

1. use `#include "port/port.h"` instead of `#include "port/port_posix.h"`
2. added an "install" target for CMake to workaround alexcrichton/cmake-rs#18
3. MSVC reported 2 warnings in Titan (and tons of warnings in RocksDB), and these are fixed.

Summary:
Fix a typo in `TitanDBImpl::DeleteFilesInRange()` which makes GC stats not being updated after `DeleteFilesInRange()` call.

Test Plan:
See the new test
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Signed-off-by: tabokie <xy.tao@outlook.com>

* new thread pool for titan
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

fix compiling error: 
> error: declaration of ‘initialized’ shadows a member of 'this'
Signed-off-by: tabokie <xy.tao@outlook.com>

since level merge will read blob files sequentially, we should use prefetcher to accelerate merging.

experiment result:

Load 300GB (average value size 1KB) on Huawei NVME device, time to read blob file reduced about 50%.
Signed-off-by: wujy-cs <wujy.cs@gmail.com>