* add deletefilesinrange during gc case
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* check obsolete before apply
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

After creating a new blob file meta, the live data size is 0. The stat is counted when adding the blob file but with live_data_size == 0. So we should update the stat when the live data size is initialized.
Signed-off-by: Connor <zbk602423539@gmail.com>

This is a new interface which is added to RocksDB recently and TiKV will call it if `enable_multi_thread_write` of `DBOptions` is true. So Titan must implement it by self.
Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

exclude metadata size from file size, otherwise discardable ratio would be wrong when the blob file is small.
Signed-off-by: Connor <zbk602423539@gmail.com>

* titan level blob file count
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* store stats in rocksdb statistics
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Little-Wallace <bupt2013211450@gmail.com>

Fix warn when titan is compiled with gcc 5.5.0
```
/home/pingcap/.cargo/git/checkouts/rust-rocksdb-e0eeb217423298aa/0e1a085/librocksdb_sys/libtitan_sys/titan/src/db_impl_gc.cc: In member function ‘rocksdb::Status rocksdb::titandb::TitanDBImpl::BackgroundGC(rocksdb::LogBuffer*, uint32_t)’:
/home/pingcap/.cargo/git/checkouts/rust-rocksdb-e0eeb217423298aa/0e1a085/librocksdb_sys/libtitan_sys/titan/src/db_impl_gc.cc:205:55: error: ‘gc_merge_rewrite’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
                           &shuting_down_, stats_.get());
```

Signed-off-by: liumh8 <571621369@qq.com>

Fixes #144 

* Green Blob GC by merge operator
Signed-off-by: tabokie <xy.tao@outlook.com>

Summary:
* On DB reopen, iterate all SST file's properties and sum up live data size of all blob files to use as GC stats.
* Remove previous logic to set GC mark on blob files on reopen.
* Refactor `OnFlushCompleted` and `OnCompactionCompleted`. Check file state is `kPendingLSM` before transit file state. 
* Refactor `BlobFileMeta`.

Test Plan:
Updated unit tests
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
When adding new blob file, titan directory need to be fsync, otherwise files can be lost after power outage.

Test Plan:
run `strace -yfe fsync ./titan_titan_db_test --gtest_filter=*.Snapshot` and see titan directory being fsync-ed in the output.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

1. use `#include "port/port.h"` instead of `#include "port/port_posix.h"`
2. added an "install" target for CMake to workaround alexcrichton/cmake-rs#18
3. MSVC reported 2 warnings in Titan (and tons of warnings in RocksDB), and these are fixed.

Summary:
Fix a typo in `TitanDBImpl::DeleteFilesInRange()` which makes GC stats not being updated after `DeleteFilesInRange()` call.

Test Plan:
See the new test
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Signed-off-by: tabokie <xy.tao@outlook.com>

* new thread pool for titan
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

fix compiling error: 
> error: declaration of ‘initialized’ shadows a member of 'this'
Signed-off-by: tabokie <xy.tao@outlook.com>

since level merge will read blob files sequentially, we should use prefetcher to accelerate merging.

experiment result:

Load 300GB (average value size 1KB) on Huawei NVME device, time to read blob file reduced about 50%.
Signed-off-by: wujy-cs <wujy.cs@gmail.com>

Summary:
Refactor `TitanDBImpl::Open` to open base DB only once. Previously we open the DB twice - the first time only to obtain `cf_id` for each CF, and then initialize internals before open base DB for real. The patch change to open base DB only once, initialize internals after that. Before `TitanDBImpl` is initialized, `BaseDbListener` will be no-op and `TableBuilder` will not output blob files.

Test Plan:
Added new tests

Adding a new version to blob file format. v2 blob files come with 4 more bytes in the header to hold future flags. In particular if `kHasUncompressionDictionary` is set, the whole file is compressed using dictionary compression and the per-blob compression type should be ignored.

Also update `BlobEncoder` and `BlobDecoder` to allow passing in a dictionary and use it to  compress/decompress. Add unit test for it.

* fix status override
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* count io stats when finishing blob file
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: tabokie <xy.tao@outlook.com>

* update io stats for level-merge
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix algorithm for counting runs
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

With level merge enabled, we expect there are no more than 10 sorted runs of blob files in both of last two levels. But 

- Since we force using level_compaction_dynamic_level_bytes, last level blob files won't be merged again, sorted run number in last level will increase infinitely.
- Some hot range may contain more sorted runs.

This patch implement range merge:

- Check sorted run number of compaction range after last level compaction
- Mark blob files to merge in next compaction if sorted run number > max_sorted_runs

* add delete blobs in ranges
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Make titandb_stress print rocksdb and titan git hash at the beginning. This is useful in schordinger test to verify which version is being tested.

Test Plan:
run titandb_stress, which prints the following at the beginning:
```
2019/10/25-05:59:22  Initializing db_stress
RocksDB version           : 6.4
RocksDB hash              : rocksdb_build_git_sha:@9ab5a5f9e70d2ea5fc9bd22a455d52a5d082fd57@
Titan hash                : titan_build_git_sha:@b497731e@
```
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix titan_db_test typo
Signed-off-by: tabokie <xy.tao@outlook.com>

* test purging blob file in fallback mode
Signed-off-by: tabokie <xy.tao@outlook.com>

* fix test
Signed-off-by: tabokie <xy.tao@outlook.com>

* simplify
Signed-off-by: tabokie <xy.tao@outlook.com>

Summary:
Temporary fix for #93. Fixing it by waiting until no flush is ongoing before starting a GC job. The drawback is GC can starve if flush runs so frequent that there's no gap between flush jobs.

Test Plan:
See the new test. The test will fail with "Corruption: Missing blob file" error before the fix.

* merge TEST_StartGC with BackgroundGC
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

tiny fix
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Adjusting daily benchmark script:
* Default params are more close to what we use for TiKV
* change bulkload to receive `NUM_THREADS` param, also bulkload disable titan GC.

Test Plan:
local test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

This prevents TiKV building on MacOS. See also https://github.com/pingcap/rocksdb/pull/123Signed-off-by: Nick Cameron <nrc@ncameron.org>

This patch implemented live data based GC for level merge.

Delete blob files which have (discardable size == file_size - file header - file footer) in OnCompactionCompleted()
Mark last two level blob files to merge in next compaction if discardable size reached GC threshold
Using live data based GC instead of regular GC as long as level merge is enabled.
Signed-off-by: Yi Wu <yiwu@pingcap.com>