* Fix GC may delete a already deleted blob file (#168)
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
When adding new blob file, titan directory need to be fsync, otherwise files can be lost after power outage.

Test Plan:
run `strace -yfe fsync ./titan_titan_db_test --gtest_filter=*.Snapshot` and see titan directory being fsync-ed in the output.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* Fix GC stat not being updated after DeleteFilesInRange (#135)

Summary:
Fix a typo in `TitanDBImpl::DeleteFilesInRange()` which makes GC stats not being updated after `DeleteFilesInRange()` call.

Test Plan:
See the new test
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* Use individual thread pool for background GC (#126)
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix status override
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Cherry-picking the following patches for fixing #93 
```
0a3f87a2 2019-10-15 yiwu@pingcap.com     Temp fix for data loss caused by concurrent flush (#96)
8ac5003c 2019-10-16 zbk602423539@gmail.. merge BackgroundGC with TEST_StartGC (#94)
b9915d9b 2019-10-07 zbk602423539@gmail.. check nullptr (#91)
```

Summary:
Fixing -Wmissing-braces compile warning, e.g. https://travis-ci.org/pingcap/rust-rocksdb/jobs/587241018

Test Plan:
`cmake .. -DCMAKE_CXX_FLAGS="-Wmissing-braces" && make src/titan_stats.o` without compile error.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

cherry-pick following patches
```
ff30897d 2019-09-23 zbk602423539@gmail.. reset gc mark after sampling (#88)
cb67cd56 2019-09-22 zbk602423539@gmail.. fix wrong property after restart (#82)
c6e408c2 2019-09-20 yiwu@pingcap.com     Fix blob file format description (#83)
fc4106d4 2019-09-18 yiwu@pingcap.com     Fix new CF missing BlobFileSizeCollector (#78)
10710bb1 2019-09-18 zbk602423539@gmail.. Add more metrics (#79)
8768067e 2019-09-17 yiwu@pingcap.com     Prevent CF being dropped while GC is running (#72)
07aa0655 2019-09-16 zbk602423539@gmail.. Rename version_set to blob_file_set (#69)
f963b880 2019-09-16 zbk602423539@gmail.. support histogram for titan stats (#74)
144e20f0 2019-09-12 wujy.cs@gmail.com    gc breakdown metrics (#73)
715dbd69 2019-09-11 zbk602423539@gmail.. Update BlobFileMeta format (#68)
468ddc97 2019-09-03 yiwu@pingcap.com     Add internal operation stats and dump to info log periodically (#62)
```

Summary:
We require passing `ROCKSDB_DIR` to cmake command, which is not easy to use. Update cmake script to make it clone rocksdb (currently pointing to pingcap/rocksdb tikv-3.0 branch) to make development use easier.

Test Plan:
local test + travis

* Add benchmark script

Limit Titan blob file read IO with rate_limiter, to avoid burst of IO starve online workloads.

* trigger next gc after gc if needed

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* check status of BatchFinishFile

* update discardable size before delete files in range
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Because db is locked when  add/sub  bg_gc_scheduled_, so this variable is no need to be atomic 

* check edits before written to manifest
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

- Set background error if encountered a critical error. Now it's set for two cases
    - BackgroundGC() error
    - LogAndApply() error after generate new blob file
- If background error is set, all following foreground write operations will be refused and return background error status

- There is no libatomic in MacOS clang(and maybe some other environment), so check if libatomic existed before link.
- Added MacOS build to travis. For TSAN build , a unused-command-line-argument is defined in RocksDB cmake file, witch will fail compilation on MacOS, skip it.

* fix iterator

* fix

* fmt

* schedule gc to bottom priority pool
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
* Add `GetTitanOptions()` and `GetTitanDBOptions()` API.
* Fix `SetOptions()` fail when `blob_run_mode` is not set.
* Update info log around printing options.

Test Plan:
Updated test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Adding some more info log logging to track blob file live cycle.

Test Plan:
Existing test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix version set log
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* introduce edit collector
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* Eliminate use of abort()
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
On `TitanDBImpl::Open` we open rocksdb twice, and the first time is just to get list of CFs and construct table factory. Compaction can run at this point, generating SST files without custom table property. This can cause problem later on, since `OnCompactionCompleted` assume the property always exist and even use it to detect new blob file generated from the compaction. This patch disable auto compaction for the first rocksdb open to avoid such problem.

The test shown here demo the issue without disabling compaction: https://gist.github.com/yiwu-arbug/34c561d0daf4f9c293bb7b2825d77bf2

Test Plan:
Existing tests
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
In fallback mode, table builder will try to read value from blob file and write the value back to LSM. If the corresponding blob file had been GC-ed and deleted, however, table builder will return error and fail compaction. Fixing it by writing the blob index to output if table builder get error when reading blob value.

Test Plan:
See the new test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Log Titan options on DB open. Sample: https://gist.github.com/yiwu-arbug/52720461651ad7019657d0dda3c50311

Also add logging around create CF/drop CF/destroy CF handle/set option.

Test Plan:
Run tests and look at the logs.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Override `Iterator* NewIterator(const ReadOptions& opts)`, otherwise, due to implicit casting, calling `db.NewIterator(TitanReadOptions::new())` will lead to a Rocksdb's iterator instead of Titan's.
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* add key only iter option
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: siddontang <siddontang@gmail.com>

Signed-off-by: siddontang siddontang@gmail.com

Copy from https://github.com/facebook/rocksdb/blob/master/CMakeLists.txt#L210