Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1837047 is being indexed.

OAK-7569: Direct Binary Access

Blob APIs and S3/Azure implementation for direct binary upload/download. Modified patch from Matt Ryan

  1. … 23 more files in changeset.
OAK-7649: oak-run command to execute datastore garbage collection

- Enable 'datastore' command with operations --check-consistency & --collect-garbage

  1. … 6 more files in changeset.
OAK-7646: UploadStagingCacheTest#testConcurrentSameAddRequest fails intermittently

- Changed MeterStats#mark() to MeterStats#mark(1) which curiously fixes the issue.

- Added a slight delay in starting the 2nd thread

  1. … 1 more file in changeset.
OAK-7511: get rid of JSR 305 dependency - use jetbrains nullability annotations instead - oak-blob-plugins
  1. … 20 more files in changeset.
OAK-7638: Race condition when simultaneous request to stage file for async upload

- Fixed the entry condition to check for file existence and create if required

  1. … 2 more files in changeset.
OAK-7584: Add stats for DataStore GC performance

Added stats for candidates found, blobs deleted, size deleted

  1. … 2 more files in changeset.
OAK-7566: Rename metrics for DataStore garbage collection

- Renamed some metrics

- Removed a redundant counter FINISH_SUCCESS

  1. … 4 more files in changeset.
OAK-7522: Enable collection of simple operation stats for DataStore garbage collection

Added simple metrics for dsgc

  1. … 9 more files in changeset.
OAK-7209: Race condition can resurrect blobs during blob GC

Calling globalMerge() to retrieve and merge locally all blob ids files from DataStore

Credits to Csaba Varga for the patch

  1. … 1 more file in changeset.
OAK-7223: Files could be kept partially in case of disconnection from backends

- Created a utility method in FileIOUtils#copyInputStreamToFile which delegates to IOUtils but deletes file on exception thus removing any potential partial files created.

- Using the new method in FileCache load.

  1. … 4 more files in changeset.
OAK-7119: Restrict de-serialization mechanism for older serialized cache map in DataStoreCacheUtils to the classes required

- Restricted to whitelist of Map, HashMap, Long, Number, String

  1. … 1 more file in changeset.
OAK-6066: Migration of binaries relies on implementation details of the TarMK
  1. … 3 more files in changeset.
OAK-6827: Consistency check fails with active deletions

More test coverage

  1. … 1 more file in changeset.
OAK-6827: Consistency check fails with active deletions

Active deleted tracker to keep track of deleted files and provides methods to filter and reconcile

  1. … 2 more files in changeset.
OAK-6550: Make BlobTracker snapshot process smarter

- Enabled snapshot after removal

- And skipping regular snapshot if happened within the last interval

  1. … 1 more file in changeset.
OAK-5983: BlobGC should log the amount of space reclaimed after GC run is done

- Logging the total size that's cleaned up only for cases where the length is encoded in the ids.

  1. … 3 more files in changeset.
OAK-5546: [BlobGC] Adapt time to delete blobs based on lucene indexing activity

- Using the minimum of oldest checkpoint timestamp and the mark phase start time as the reference time to calculate max age of blobs to be considered for deletion

- Added test

- Updated NodeStoreService to pass whiteboard instance to the GC object

    • ?
    ./blob/BlobGCCheckpointRefTest.java
  1. … 4 more files in changeset.
OAK-6799 - OSGi service tests for Blob GC

- Tests for BlobGC Mbean initialization for OSGi *NodeStoreServices

    • ?
    ./blob/AbstractBlobGCRegistrationTest.java
  1. … 2 more files in changeset.
OAK-6798 - Basic Blob GC test for in-memory NodeStore and BlobStore

- Added a BlobGCTest with a in-memory test extension to MemoryNodeStore and an in-memory test BlobStore

OAK-6670 - Move FSBlobSerializer to oak-blob-plugins module
    • ?
    ./blob/serializer/FSBlobSerializerTest.java
  1. … 4 more files in changeset.
OAK-6669 - Blob serializer to serialize BlobStoreBlob
    • ?
    ./blob/serializer/BlobIdSerializerTest.java
  1. … 2 more files in changeset.
OAK-6630: Remove older cached FileDataStore relying on JR caching

- Remove older code relying on CachingFDS from JR2

  1. … 3 more files in changeset.
OAK-6621: Initialize a default sensible stats provider in UploadStagingCache

- Initializing DefaultStatsProvider in case none provided

  1. … 1 more file in changeset.
OAK-6413: FileCache getIfPresent doesn't update cache hit/miss counters

Fixed test expectations

OAK-6413: FileCache getIfPresent doesn't update cache hit/miss counters

Directly calling CacheLirs.getIfPresent() to properly update hit/miss counter

  1. … 1 more file in changeset.
OAK-6285: Test failure: UploadStagingCacheTest.testUpgrade

- Potential fix

OAK-6283: FileCache should ignore when file evicted with replacement

Ignore deletion from cache when evicted with REPLACE

  1. … 1 more file in changeset.
OAK-5935: AbstractSharedCachingDataStore#getRecordIfStored should use the underlying cache.get

- The call now returns a FileCachedDataRecord for cases where the cache did not have any entry but the backend has and lazily loads the input stream forcing load through the cache so that the cache is loaded

- Added test lazyLoadStream

- Fixed an issue with the test which would keep a stale reference to a file

  1. … 2 more files in changeset.
OAK-6095 : Get rid of dependency from oak-segment-tar to oak-core tests
    • ?
    ./blob/migration/AbstractMigratorTest.java
  1. … 20 more files in changeset.
OAK-6073 : Get rid of dependency from oak-blob-azure from oak-core
    • ?
    ./blob/datastore/BlobIdTrackerTest.java
    • ?
    ./blob/CompositeDataStoreCacheTest.java
    • ?
    ./blob/datastore/InMemoryDataRecordTest.java
    • ?
    ./blob/datastore/OakCachingFDSTest.java
    • ?
    ./blob/ConsolidatedDataStoreStatsTest.java
    • ?
    ./blob/SharedDataStoreUtilsTest.java
    • ?
    ./blob/datastore/OakFileDataStoreTest.java
    • ?
    ./blob/DataStoreCacheUpgradeUtilsTest.java
    • ?
    ./blob/datastore/DataStoreUtils.java
    • ?
    ./blob/datastore/CachingFileDataStoreTest.java
    • ?
    ./blob/datastore/DataStoreBlobStoreTest.java
  1. … 785 more files in changeset.