Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates

Changeset 1869379 is being indexed.

OAK-8576: Extend instrumentation for data stores.

This commit adds instrumentation for other methods in data stores, including

support for metrics for SharedDataStore implementations and for direct

binary access functions, as well as improving existing instrumentation for

uploads and downloads. It also adds error counts to aid in identifying

and diagnosing failure conditions in production environments.

    • ?
    ./blob/datastore/BlobStoreStatsTestableFileDataStore.java
  1. … 8 more files in changeset.
OAK-8599: Fix direct binary upload init to return null if disabled

  1. … 3 more files in changeset.
OAK-8552: Add data store config to skip exists check on download URI generation

This change adds a new optional configuration parameter for AzureDataStore and S3DataStore. The new parameter, named "presignedHttpDownloadURIVerifyExists", has a default value of true which means the createHttpDownloadURI() method will ensure the requested blob exists in blob storage before creating the signed download URI. If this parameter is set to false, the exists check will be skipped, resulting in higher signed download URI generation performance but potentially returning a URI to a nonexistent blob in some cases (see OAK-7998).

  1. … 7 more files in changeset.
OAK-8578: Introduce API to check whether blob inlined in Id

OAK-8552: Minimize network calls required when creating a direct download URI

Add a new method #isInlined to Blob class to check if the external blob is id inlined

  1. … 9 more files in changeset.
OAK-8578:Reverting commit r1865962
  1. … 14 more files in changeset.
OAK-8552: Minimize network calls required when creating a direct download URI

Add a new method #isInlined to Blob class to check if the external blob is id inlined

  1. … 14 more files in changeset.
OAK-8575: Remove unneeded existence check in cloud data store getRecord() calls

  1. … 1 more file in changeset.
OAK-8520: Return existing DataRecord when completeUpload called for existing binary

  1. … 2 more files in changeset.
OAK-7998: Verify existence of binary before generating direct download URI

This change adds code and tests to ensure that when generating a direct

download URI we make sure the binary actually exists first. If caching

is being used it is possible that a client requests a direct download

URI for a blob that was already added the standard (i.e. non-direct)

way but is still in the cache and hasn't actually been uploaded to

cloud storage yet - and therefore cannot be downloaded by direct

download URI.

This change also adds some documentation changes to reflect this

possible situation.

  1. … 3 more files in changeset.
OAK-8463: Fix merge in consistency check to allow paths suffixed to blob ids

- Fix merge to ignore paths if present

  1. … 1 more file in changeset.
OAK-8447: Handle all available references in the DataStore when doing consistency check

- Now if available all reference files will be accounted for when doing consistency check

  1. … 1 more file in changeset.
OAK-8446: Enable datastore consistency check after completion of garbage collection

- Enables datastore consistency in a low cost way after GC

    • ?
    ./blob/SharedDataStoreMarkSweepGarbageCollectorTest.java
  1. … 4 more files in changeset.
OAK-8314: BlobIds created with direct uploads lack length suffix

Suffixing length to the ids being returned for for directly uploded binaries

  1. … 1 more file in changeset.
OAK-8115: Allow disabling of BlobTracker when not required

- BlobTracker can be disabled by setting config property snapshotIntervalSecs to 0 or below as this prop needs a sensible non-zero value anyways to be effective

  1. … 5 more files in changeset.
OAK-8013: Disable filename* part of Content-Disposition specification for direct binary access.

  1. … 1 more file in changeset.
OAK-7951: Datastore GC stats not updated with failure when "Not all repositories have marked references available"

- Missing License header

    • ?
    ./blob/SharedDataStoreMarkSweepGarbageCollectorTest.java
OAK-7951: Datastore GC stats not updated with failure when "Not all repositories have marked references available"

Based on patch from Wim Symons

- Bubble up exception in case of not able to run sweep because of not all repositories not having references available

- Mark failure in stats which can then be queried

    • ?
    ./blob/SharedDataStoreMarkSweepGarbageCollectorTest.java
  1. … 1 more file in changeset.
OAK-7858: S3#getAllIdentifiers may trim listing when filtering out metadata objects

- Continue iteration till s3 indicates no more entries

  1. … 3 more files in changeset.
OAK-7798: Return stream for downloaded files rather than directly from backend

- assert temp file presence before reading

OAK-7798: Return stream for downloaded files rather than directly from backend

- Slightly shorter path for ds in test

- Using the FileIOUtils.copyInputStreamToFile in test

- Some rearrangement to use existing folders

OAK-7798: Return stream for downloaded files rather than directly from backend

- Returning stream from a downloaded file rather than directly from backend

  1. … 1 more file in changeset.
OAK-7753: Enable collection of stats for DataStore checkConsistency operation

Added statistics for checkConsistency operation

  1. … 3 more files in changeset.
OAK-7727: Aggregate references from repositories with same repositoryId

- A different references file is created and then aggregated to account for all references available

  1. … 3 more files in changeset.
OAK-7738: Add method to check for existence of metadata in SharedDataStore

- Added a new method to the ShareDataStore and implementations

  1. … 11 more files in changeset.
OAK-7737: Add edge cases checks and tests for metadata related methods S3/FileDataStore

- Superseded by SharedDataStoreTest which covers all SharedDataStore tests

OAK-7737: Add edge cases checks and tests for metadata related methods S3/FileDataStore

- Added tests for metadata operations

    • ?
    ./blob/datastore/SharedDataStoreTest.java
  1. … 4 more files in changeset.
OAK-7692: [DirectBinaryAccess] Upload token HMAC signature must be base64 encoded

Patch from Alexander Klimetschek

  1. … 1 more file in changeset.
OAK-7569: Direct Binary Access

Blob APIs and S3/Azure implementation for direct binary upload/download. Modified patch from Matt Ryan

  1. … 23 more files in changeset.
OAK-7649: oak-run command to execute datastore garbage collection

- Enable 'datastore' command with operations --check-consistency & --collect-garbage

  1. … 6 more files in changeset.
OAK-7646: UploadStagingCacheTest#testConcurrentSameAddRequest fails intermittently

- Changed MeterStats#mark() to MeterStats#mark(1) which curiously fixes the issue.

- Added a slight delay in starting the 2nd thread

  1. … 1 more file in changeset.