Signed-off-by: tabokie <xy.tao@outlook.com>

* new thread pool for titan
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

fix compiling error: 
> error: declaration of ‘initialized’ shadows a member of 'this'
Signed-off-by: tabokie <xy.tao@outlook.com>

since level merge will read blob files sequentially, we should use prefetcher to accelerate merging.

experiment result:

Load 300GB (average value size 1KB) on Huawei NVME device, time to read blob file reduced about 50%.
Signed-off-by: wujy-cs <wujy.cs@gmail.com>

Summary:
Refactor `TitanDBImpl::Open` to open base DB only once. Previously we open the DB twice - the first time only to obtain `cf_id` for each CF, and then initialize internals before open base DB for real. The patch change to open base DB only once, initialize internals after that. Before `TitanDBImpl` is initialized, `BaseDbListener` will be no-op and `TableBuilder` will not output blob files.

Test Plan:
Added new tests

Adding a new version to blob file format. v2 blob files come with 4 more bytes in the header to hold future flags. In particular if `kHasUncompressionDictionary` is set, the whole file is compressed using dictionary compression and the per-blob compression type should be ignored.

Also update `BlobEncoder` and `BlobDecoder` to allow passing in a dictionary and use it to  compress/decompress. Add unit test for it.

* fix status override
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* count io stats when finishing blob file
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: tabokie <xy.tao@outlook.com>

* update io stats for level-merge
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* fix algorithm for counting runs
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

With level merge enabled, we expect there are no more than 10 sorted runs of blob files in both of last two levels. But 

- Since we force using level_compaction_dynamic_level_bytes, last level blob files won't be merged again, sorted run number in last level will increase infinitely.
- Some hot range may contain more sorted runs.

This patch implement range merge:

- Check sorted run number of compaction range after last level compaction
- Mark blob files to merge in next compaction if sorted run number > max_sorted_runs

* add delete blobs in ranges
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Make titandb_stress print rocksdb and titan git hash at the beginning. This is useful in schordinger test to verify which version is being tested.

Test Plan:
run titandb_stress, which prints the following at the beginning:
```
2019/10/25-05:59:22  Initializing db_stress
RocksDB version           : 6.4
RocksDB hash              : rocksdb_build_git_sha:@9ab5a5f9e70d2ea5fc9bd22a455d52a5d082fd57@
Titan hash                : titan_build_git_sha:@b497731e@
```
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix titan_db_test typo
Signed-off-by: tabokie <xy.tao@outlook.com>

* test purging blob file in fallback mode
Signed-off-by: tabokie <xy.tao@outlook.com>

* fix test
Signed-off-by: tabokie <xy.tao@outlook.com>

* simplify
Signed-off-by: tabokie <xy.tao@outlook.com>

Summary:
Temporary fix for #93. Fixing it by waiting until no flush is ongoing before starting a GC job. The drawback is GC can starve if flush runs so frequent that there's no gap between flush jobs.

Test Plan:
See the new test. The test will fail with "Corruption: Missing blob file" error before the fix.

* merge TEST_StartGC with BackgroundGC
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

tiny fix
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Adjusting daily benchmark script:
* Default params are more close to what we use for TiKV
* change bulkload to receive `NUM_THREADS` param, also bulkload disable titan GC.

Test Plan:
local test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

This prevents TiKV building on MacOS. See also https://github.com/pingcap/rocksdb/pull/123Signed-off-by: Nick Cameron <nrc@ncameron.org>

This patch implemented live data based GC for level merge.

Delete blob files which have (discardable size == file_size - file header - file footer) in OnCompactionCompleted()
Mark last two level blob files to merge in next compaction if discardable size reached GC threshold
Using live data based GC instead of regular GC as long as level merge is enabled.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Fixing -Wmissing-braces compile warning, e.g. https://travis-ci.org/pingcap/rust-rocksdb/jobs/587241018

Test Plan:
`cmake .. -DCMAKE_CXX_FLAGS="-Wmissing-braces" && make src/titan_stats.o` without compile error.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Reset gc mark after sampling, Fix #87 

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Blob file does not come with meta blocks. Fix the inline description.

Test Plan:
N/A
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
On `CreateColumnFamilies` we should set `BlobFileSizeCollectorFactory`. Fixing it.

Test Plan:
added test
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* add more metrics
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
If `DropColumnFamilies` is called while GC is running, GC can fail because the CF is gone. The GC job will then set background error which halts Titan as a whole. To prevent it, we let `DropColumnFamilies` wait till there no running GC before proceed, and let GC job wait for pending drop CF requests before start running. Fixes #71.

Also fix `DropColumnFamilies` mark obsolete file as obsolete again, which will cause assert failure.

Test Plan:
Run titandb_stress and the above issues don't reproduce.

Summary:
#61 update default rocksdb to 6.4.x, but didn't update titandb_stress accordingly. Update titandb_stress to pick up recent changes.

Test Plan:
build
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* rename version_set to blob_set
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* In table builder, if target level is last two level, write values of lower level to new blob files.
Signed-off-by: wujy-cs <wujy.cs@gmail.com>

* add smallest_key and largest_key
* add number of file entires and file's level to blob file meta
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Add stats for read/write bytes and input/output file of internal operations and break down by operation type (being flush, compaction or GC). Later after we introduce vtable we can further breakdown compaction into top levels compaction, L_n-1 compaction and Ln compaction.

Test Plan:
Run db_bench and get sample output:
```
2019/08/29-23:49:29.593805 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593742) Titan internal stats for column family [default]:
2019/08/29-23:49:29.593813 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593753) OP          COUNT  READ(GB)  WRITE(GB) IO_READ(GB) IO_WRITE(GB)  FILE_IN FILE_OUT
2019/08/29-23:49:29.593821 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593760) ---------------------------------------------------------------------------------
2019/08/29-23:49:29.593830 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593768) Flush          3        0.0       34.7         0.0         18.0        0        3
2019/08/29-23:49:29.593839 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593784) Compaction     1        0.0        0.0         0.0          0.0        0        0
2019/08/29-23:49:29.593848 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593793) GC             1      360.9        1.3       354.3          2.8        8        0
```

Summary:
Update the code to base off RocksDB 6.4.x. There are a few internal interface changes and files moving around from 5.18 to 6.4. titandb_bench is also updated to base off the version in 6.4.x.

Test Plan:
Travis