This patch implemented live data based GC for level merge.

Delete blob files which have (discardable size == file_size - file header - file footer) in OnCompactionCompleted()
Mark last two level blob files to merge in next compaction if discardable size reached GC threshold
Using live data based GC instead of regular GC as long as level merge is enabled.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Fixing -Wmissing-braces compile warning, e.g. https://travis-ci.org/pingcap/rust-rocksdb/jobs/587241018

Test Plan:
`cmake .. -DCMAKE_CXX_FLAGS="-Wmissing-braces" && make src/titan_stats.o` without compile error.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Reset gc mark after sampling, Fix #87 

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Blob file does not come with meta blocks. Fix the inline description.

Test Plan:
N/A
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
On `CreateColumnFamilies` we should set `BlobFileSizeCollectorFactory`. Fixing it.

Test Plan:
added test
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* add more metrics
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
If `DropColumnFamilies` is called while GC is running, GC can fail because the CF is gone. The GC job will then set background error which halts Titan as a whole. To prevent it, we let `DropColumnFamilies` wait till there no running GC before proceed, and let GC job wait for pending drop CF requests before start running. Fixes #71.

Also fix `DropColumnFamilies` mark obsolete file as obsolete again, which will cause assert failure.

Test Plan:
Run titandb_stress and the above issues don't reproduce.

Summary:
#61 update default rocksdb to 6.4.x, but didn't update titandb_stress accordingly. Update titandb_stress to pick up recent changes.

Test Plan:
build
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* rename version_set to blob_set
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* In table builder, if target level is last two level, write values of lower level to new blob files.
Signed-off-by: wujy-cs <wujy.cs@gmail.com>

* add smallest_key and largest_key
* add number of file entires and file's level to blob file meta
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
Add stats for read/write bytes and input/output file of internal operations and break down by operation type (being flush, compaction or GC). Later after we introduce vtable we can further breakdown compaction into top levels compaction, L_n-1 compaction and Ln compaction.

Test Plan:
Run db_bench and get sample output:
```
2019/08/29-23:49:29.593805 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593742) Titan internal stats for column family [default]:
2019/08/29-23:49:29.593813 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593753) OP          COUNT  READ(GB)  WRITE(GB) IO_READ(GB) IO_WRITE(GB)  FILE_IN FILE_OUT
2019/08/29-23:49:29.593821 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593760) ---------------------------------------------------------------------------------
2019/08/29-23:49:29.593830 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593768) Flush          3        0.0       34.7         0.0         18.0        0        3
2019/08/29-23:49:29.593839 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593784) Compaction     1        0.0        0.0         0.0          0.0        0        0
2019/08/29-23:49:29.593848 7f6fd97fa700 (Original Log Time 2019/08/29-23:49:29.593793) GC             1      360.9        1.3       354.3          2.8        8        0
```

Summary:
Update the code to base off RocksDB 6.4.x. There are a few internal interface changes and files moving around from 5.18 to 6.4. titandb_bench is also updated to base off the version in 6.4.x.

Test Plan:
Travis

Summary:
We require passing `ROCKSDB_DIR` to cmake command, which is not easy to use. Update cmake script to make it clone rocksdb (currently pointing to pingcap/rocksdb tikv-3.0 branch) to make development use easier.

Test Plan:
local test + travis

* Add benchmark script

Limit Titan blob file read IO with rate_limiter, to avoid burst of IO starve online workloads.

* trigger next gc after gc if needed

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* check status of BatchFinishFile

* update discardable size before delete files in range
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Because db is locked when  add/sub  bg_gc_scheduled_, so this variable is no need to be atomic 

* check edits before written to manifest
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

- Set background error if encountered a critical error. Now it's set for two cases
    - BackgroundGC() error
    - LogAndApply() error after generate new blob file
- If background error is set, all following foreground write operations will be refused and return background error status

- There is no libatomic in MacOS clang(and maybe some other environment), so check if libatomic existed before link.
- Added MacOS build to travis. For TSAN build , a unused-command-line-argument is defined in RocksDB cmake file, witch will fail compilation on MacOS, skip it.

* fix iterator

* fix

* fmt

* schedule gc to bottom priority pool
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
* Add `GetTitanOptions()` and `GetTitanDBOptions()` API.
* Fix `SetOptions()` fail when `blob_run_mode` is not set.
* Update info log around printing options.

Test Plan:
Updated test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Adding some more info log logging to track blob file live cycle.

Test Plan:
Existing test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix version set log
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* introduce edit collector
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* Eliminate use of abort()
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
On `TitanDBImpl::Open` we open rocksdb twice, and the first time is just to get list of CFs and construct table factory. Compaction can run at this point, generating SST files without custom table property. This can cause problem later on, since `OnCompactionCompleted` assume the property always exist and even use it to detect new blob file generated from the compaction. This patch disable auto compaction for the first rocksdb open to avoid such problem.

The test shown here demo the issue without disabling compaction: https://gist.github.com/yiwu-arbug/34c561d0daf4f9c293bb7b2825d77bf2

Test Plan:
Existing tests
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
In fallback mode, table builder will try to read value from blob file and write the value back to LSM. If the corresponding blob file had been GC-ed and deleted, however, table builder will return error and fail compaction. Fixing it by writing the blob index to output if table builder get error when reading blob value.

Test Plan:
See the new test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>