* trigger next gc after gc if needed

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* check status of BatchFinishFile

* update discardable size before delete files in range
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Because db is locked when  add/sub  bg_gc_scheduled_, so this variable is no need to be atomic 

* check edits before written to manifest
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

- Set background error if encountered a critical error. Now it's set for two cases
    - BackgroundGC() error
    - LogAndApply() error after generate new blob file
- If background error is set, all following foreground write operations will be refused and return background error status

- There is no libatomic in MacOS clang(and maybe some other environment), so check if libatomic existed before link.
- Added MacOS build to travis. For TSAN build , a unused-command-line-argument is defined in RocksDB cmake file, witch will fail compilation on MacOS, skip it.

* fix iterator

* fix

* fmt

* schedule gc to bottom priority pool
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Summary:
* Add `GetTitanOptions()` and `GetTitanDBOptions()` API.
* Fix `SetOptions()` fail when `blob_run_mode` is not set.
* Update info log around printing options.

Test Plan:
Updated test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Adding some more info log logging to track blob file live cycle.

Test Plan:
Existing test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix version set log
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* introduce edit collector
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* Eliminate use of abort()
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
On `TitanDBImpl::Open` we open rocksdb twice, and the first time is just to get list of CFs and construct table factory. Compaction can run at this point, generating SST files without custom table property. This can cause problem later on, since `OnCompactionCompleted` assume the property always exist and even use it to detect new blob file generated from the compaction. This patch disable auto compaction for the first rocksdb open to avoid such problem.

The test shown here demo the issue without disabling compaction: https://gist.github.com/yiwu-arbug/34c561d0daf4f9c293bb7b2825d77bf2

Test Plan:
Existing tests
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
In fallback mode, table builder will try to read value from blob file and write the value back to LSM. If the corresponding blob file had been GC-ed and deleted, however, table builder will return error and fail compaction. Fixing it by writing the blob index to output if table builder get error when reading blob value.

Test Plan:
See the new test.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Summary:
Log Titan options on DB open. Sample: https://gist.github.com/yiwu-arbug/52720461651ad7019657d0dda3c50311

Also add logging around create CF/drop CF/destroy CF handle/set option.

Test Plan:
Run tests and look at the logs.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Override `Iterator* NewIterator(const ReadOptions& opts)`, otherwise, due to implicit casting, calling `db.NewIterator(TitanReadOptions::new())` will lead to a Rocksdb's iterator instead of Titan's.
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* add key only iter option
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Signed-off-by: siddontang <siddontang@gmail.com>

Signed-off-by: siddontang siddontang@gmail.com

Copy from https://github.com/facebook/rocksdb/blob/master/CMakeLists.txt#L210

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* Add Titan internal stats

* add metrics for blob monitoring

* add override quantifier

* address comments

* fix minor issue

* address comments

* address comments

* address comments

* merge master

* turn on test output

* fix merge issue

* fix segfault in background gc

* fix loop

* remove loop

* Add blob-run-mode for downgrading Titan

* fix format issue

* address comment

* change string req to kabab-case

* address comments

* address comments

* fix bug from merge

* address comments

* address comments

* address comments

* address comment

* address comment

* pick comments from mutex-fix

* remove mutex fix

* fix mutex issue in blob storage

Add thread_safety_test to catch data race issues around CF add/drop and GC.

* Simplify CMake script
Signed-off-by: ice1000 <ice1000kotlin@foxmail.com>

* Exclude tests
Signed-off-by: ice1000 <ice1000kotlin@foxmail.com>

* add metrics
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* change codecov comment style
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Add an uint64 test flag `min-blob-size` for Titan in titandb_bench and titandb_stress tests.

Adding scripts/format-diff.sh to use clang-format to format code changes since diverge from master. Also adding travis CI job to check if the code needs to format. Also format all existing code.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

As Rocksdb described, `DropColumnFamilies()` only records the drop of the column family specified by ColumnFamilyHandle. The actual data is not deleted until the client calls `delete column_family`, namely `DestroyColumnFamilyHandle()`. We can still continue using the column family if we have outstanding ColumnFamilyHandle pointer.	

now there are some problems:

- after dropping column family, the blob files of it may be deleted before column family handle is destroyed.
- When opening DB without some existing column families(not supported yet), the blob file of these will be regarded as obsolete files and deleted. 
- After destroyed column handle(not drop the column, just not need to read/write it anymore), we need to remove inner blob storage and evict related cache.

use `lcov` to generate code coverage report together with GCC build, and send result to codecov. Also some minor refactor to travis config.
Signed-off-by: Yi Wu <yiwu@pingcap.com>

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

Update cmake script to add jemalloc and compression libraries' include directories. This is to fix missing headers in build, e.g: https://travis-ci.org/pingcap/rust-rocksdb/jobs/529604990

After the fix rust-rocksdb build will pass: https://travis-ci.org/yiwu-arbug/rust-rocksdb/jobs/529906506