Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1865974 is being indexed.

OAK-8578:Reverting commit r1865962
  1. … 14 more files in changeset.
OAK-8552: Minimize network calls required when creating a direct download URI

Add a new method #isInlined to Blob class to check if the external blob is id inlined

  1. … 14 more files in changeset.
OAK-8575: Remove unneeded existence check in cloud data store getRecord() calls

  1. … 1 more file in changeset.
OAK-8520: Return existing DataRecord when completeUpload called for existing binary

  1. … 2 more files in changeset.
OAK-7998: Verify existence of binary before generating direct download URI

This change adds code and tests to ensure that when generating a direct

download URI we make sure the binary actually exists first. If caching

is being used it is possible that a client requests a direct download

URI for a blob that was already added the standard (i.e. non-direct)

way but is still in the cache and hasn't actually been uploaded to

cloud storage yet - and therefore cannot be downloaded by direct

download URI.

This change also adds some documentation changes to reflect this

possible situation.

  1. … 3 more files in changeset.
OAK-8463: Fix merge in consistency check to allow paths suffixed to blob ids

- Fix merge to ignore paths if present

  1. … 1 more file in changeset.
OAK-8447: Handle all available references in the DataStore when doing consistency check

- Now if available all reference files will be accounted for when doing consistency check

  1. … 1 more file in changeset.
OAK-8446: Enable datastore consistency check after completion of garbage collection

- Enables datastore consistency in a low cost way after GC

    • ?
    ./blob/SharedDataStoreMarkSweepGarbageCollectorTest.java
  1. … 4 more files in changeset.
OAK-8314: BlobIds created with direct uploads lack length suffix

Suffixing length to the ids being returned for for directly uploded binaries

  1. … 1 more file in changeset.
OAK-8115: Allow disabling of BlobTracker when not required

- BlobTracker can be disabled by setting config property snapshotIntervalSecs to 0 or below as this prop needs a sensible non-zero value anyways to be effective

  1. … 5 more files in changeset.
OAK-8013: Disable filename* part of Content-Disposition specification for direct binary access.

  1. … 1 more file in changeset.
OAK-7951: Datastore GC stats not updated with failure when "Not all repositories have marked references available"

- Missing License header

    • ?
    ./blob/SharedDataStoreMarkSweepGarbageCollectorTest.java
OAK-7951: Datastore GC stats not updated with failure when "Not all repositories have marked references available"

Based on patch from Wim Symons

- Bubble up exception in case of not able to run sweep because of not all repositories not having references available

- Mark failure in stats which can then be queried

    • ?
    ./blob/SharedDataStoreMarkSweepGarbageCollectorTest.java
  1. … 1 more file in changeset.
OAK-7858: S3#getAllIdentifiers may trim listing when filtering out metadata objects

- Continue iteration till s3 indicates no more entries

  1. … 3 more files in changeset.
OAK-7798: Return stream for downloaded files rather than directly from backend

- assert temp file presence before reading

OAK-7798: Return stream for downloaded files rather than directly from backend

- Slightly shorter path for ds in test

- Using the FileIOUtils.copyInputStreamToFile in test

- Some rearrangement to use existing folders

OAK-7798: Return stream for downloaded files rather than directly from backend

- Returning stream from a downloaded file rather than directly from backend

  1. … 1 more file in changeset.
OAK-7753: Enable collection of stats for DataStore checkConsistency operation

Added statistics for checkConsistency operation

  1. … 3 more files in changeset.
OAK-7727: Aggregate references from repositories with same repositoryId

- A different references file is created and then aggregated to account for all references available

  1. … 3 more files in changeset.
OAK-7738: Add method to check for existence of metadata in SharedDataStore

- Added a new method to the ShareDataStore and implementations

  1. … 11 more files in changeset.
OAK-7737: Add edge cases checks and tests for metadata related methods S3/FileDataStore

- Superseded by SharedDataStoreTest which covers all SharedDataStore tests

OAK-7737: Add edge cases checks and tests for metadata related methods S3/FileDataStore

- Added tests for metadata operations

    • ?
    ./blob/datastore/SharedDataStoreTest.java
  1. … 4 more files in changeset.
OAK-7692: [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

Patch from Alexander Klimetschek

  1. … 1 more file in changeset.
OAK-7569: Direct Binary Access

Blob APIs and S3/Azure implementation for direct binary upload/download. Modified patch from Matt Ryan

  1. … 23 more files in changeset.
OAK-7649: oak-run command to execute datastore garbage collection

- Enable 'datastore' command with operations --check-consistency & --collect-garbage

  1. … 6 more files in changeset.
OAK-7646: UploadStagingCacheTest#testConcurrentSameAddRequest fails intermittently

- Changed MeterStats#mark() to MeterStats#mark(1) which curiously fixes the issue.

- Added a slight delay in starting the 2nd thread

  1. … 1 more file in changeset.
OAK-7511: get rid of JSR 305 dependency - use jetbrains nullability annotations instead - oak-blob-plugins
  1. … 20 more files in changeset.
OAK-7638: Race condition when simultaneous request to stage file for async upload

- Fixed the entry condition to check for file existence and create if required

  1. … 2 more files in changeset.
OAK-7584: Add stats for DataStore GC performance

Added stats for candidates found, blobs deleted, size deleted

  1. … 2 more files in changeset.
OAK-7566: Rename metrics for DataStore garbage collection

- Renamed some metrics

- Removed a redundant counter FINISH_SUCCESS

  1. … 4 more files in changeset.