Checkout Tools
  • last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
OAK-8903: Support for removing repository markers for transient repositories

- System flag oak.datastore.sharedTransient which if set to true will force deletion of the repository marker from the datastore on close()

- Added 2 new default setter/getter methods for repository ids in the SharedDataStore interface with the only implementation needed in the DataStoreBlobStore (which is the glue between BlobStore & DataStore)

- The construction code (*NodeStoreService & tests) don't need to add the metadata file to the DataStore directly and can simply call the setRepositoryId() on the DataStoreBlobStore.

- Additional tests and changes to existing test to change the call above in 3.

  1. … 9 more files in changeset.
OAK-8878: Expose downloaded (and uploaded) blob bytes as metrics, not only as time series

Apply contribution by Axel Hanikel

  1. … 1 more file in changeset.
OAK-8780 - Remove the synchronized block on the FSBackend

Fix the failing tests

OAK-8780 - Remove the synchronized block on the FSBackend

Slightly modified patch from Ankita Agarwal

Moved the copy of file to a temp file outside the synchronized block

  1. … 1 more file in changeset.
fix svn:eol-style
  1. … 5 more files in changeset.
OAK-8280: Add support to request ignoring domain override for signed URIs.

This change introduces BlobUploadOptions so options can be requested for

both upload and download URIs, and adds the option for both URI requests

to ignore the domain override if one is set.

  1. … 23 more files in changeset.
OAK-8104: Update to Azure SDK v8.6.0 to fix signed URI C-D headers.

The purpose of this commit is to fix Content-Disposition header support for signed download URIs. The filename* portion of the header specification was disabled in OAK-8013 due to a bug in the Azure SDK pre-8.5.0. This commit adds filename* portion support back into Oak, which addresses the following issues:

- OAK-8104 - Fix the Content-Disposition header correctly (required upgrading the Azure SDK version to 8.6.0)

- OAK-8607 - Undo the workarounds implemented in OAK-8013.

  1. … 4 more files in changeset.
OAK-8807: Update FileCache to use non-cache segment blocking method

- Building the cache without the CacheLoader and using the get method with a valueLoader

  1. … 1 more file in changeset.
OAK-8786: Remove temporary file on upload immediately after a sync upload

- Delete temporary file after sync upload

  1. … 1 more file in changeset.
OAK-8576: Extend instrumentation for data stores - fix svn:eol-style
  1. … 1 more file in changeset.
OAK-8576: Extend instrumentation for data stores.

This commit adds instrumentation for other methods in data stores, including

support for metrics for SharedDataStore implementations and for direct

binary access functions, as well as improving existing instrumentation for

uploads and downloads. It also adds error counts to aid in identifying

and diagnosing failure conditions in production environments.

  1. … 8 more files in changeset.
OAK-8599: Fix direct binary upload init to return null if disabled

  1. … 3 more files in changeset.
OAK-8552: Add data store config to skip exists check on download URI generation

This change adds a new optional configuration parameter for AzureDataStore and S3DataStore. The new parameter, named "presignedHttpDownloadURIVerifyExists", has a default value of true which means the createHttpDownloadURI() method will ensure the requested blob exists in blob storage before creating the signed download URI. If this parameter is set to false, the exists check will be skipped, resulting in higher signed download URI generation performance but potentially returning a URI to a nonexistent blob in some cases (see OAK-7998).

  1. … 7 more files in changeset.
OAK-8578: Introduce API to check whether blob inlined in Id

OAK-8552: Minimize network calls required when creating a direct download URI

Add a new method #isInlined to Blob class to check if the external blob is id inlined

  1. … 9 more files in changeset.
OAK-8578:Reverting commit r1865962
  1. … 14 more files in changeset.
OAK-8552: Minimize network calls required when creating a direct download URI

Add a new method #isInlined to Blob class to check if the external blob is id inlined

  1. … 14 more files in changeset.
OAK-8575: Remove unneeded existence check in cloud data store getRecord() calls

  1. … 1 more file in changeset.
OAK-8520: Return existing DataRecord when completeUpload called for existing binary

  1. … 2 more files in changeset.
OAK-7998: Verify existence of binary before generating direct download URI

This change adds code and tests to ensure that when generating a direct

download URI we make sure the binary actually exists first. If caching

is being used it is possible that a client requests a direct download

URI for a blob that was already added the standard (i.e. non-direct)

way but is still in the cache and hasn't actually been uploaded to

cloud storage yet - and therefore cannot be downloaded by direct

download URI.

This change also adds some documentation changes to reflect this

possible situation.

  1. … 3 more files in changeset.
OAK-8463: Fix merge in consistency check to allow paths suffixed to blob ids

- Fix merge to ignore paths if present

  1. … 1 more file in changeset.
OAK-8447: Handle all available references in the DataStore when doing consistency check

- Now if available all reference files will be accounted for when doing consistency check

  1. … 1 more file in changeset.
OAK-8446: Enable datastore consistency check after completion of garbage collection

- Enables datastore consistency in a low cost way after GC

  1. … 4 more files in changeset.
OAK-8314: BlobIds created with direct uploads lack length suffix

Suffixing length to the ids being returned for for directly uploded binaries

  1. … 1 more file in changeset.
OAK-8115: Allow disabling of BlobTracker when not required

- BlobTracker can be disabled by setting config property snapshotIntervalSecs to 0 or below as this prop needs a sensible non-zero value anyways to be effective

  1. … 5 more files in changeset.
OAK-8013: Disable filename* part of Content-Disposition specification for direct binary access.

  1. … 1 more file in changeset.
OAK-7951: Datastore GC stats not updated with failure when "Not all repositories have marked references available"

- Missing License header

OAK-7951: Datastore GC stats not updated with failure when "Not all repositories have marked references available"

Based on patch from Wim Symons

- Bubble up exception in case of not able to run sweep because of not all repositories not having references available

- Mark failure in stats which can then be queried

  1. … 1 more file in changeset.
OAK-7858: S3#getAllIdentifiers may trim listing when filtering out metadata objects

- Continue iteration till s3 indicates no more entries

  1. … 3 more files in changeset.
OAK-7798: Return stream for downloaded files rather than directly from backend

- assert temp file presence before reading

OAK-7798: Return stream for downloaded files rather than directly from backend

- Slightly shorter path for ds in test

- Using the FileIOUtils.copyInputStreamToFile in test

- Some rearrangement to use existing folders