Checkout Tools
  • last updated 8 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
OAK-8280: Add support to request ignoring domain override for signed URIs.

This change introduces BlobUploadOptions so options can be requested for

both upload and download URIs, and adds the option for both URI requests

to ignore the domain override if one is set.

  1. … 26 more files in changeset.
OAK-8825: Add check for bucket existence to minimize test failures due to missing buckets

  1. … 2 more files in changeset.
OAK-8599: Fix direct binary upload init to return null if disabled

  1. … 3 more files in changeset.
OAK-8600: Change log level to DEBUG in getRecord() when blob does not exist

With recent optimizations to getRecord() in the cloud data stores, it is

not an unexpected thing for the corresponding blob to not exist. We

want to avoid filling logs with INFO level log messages and stack

traces for missing records if the calling code has a reasonable

expectation that the record may not exist. A DataStoreException will

still be thrown that must be caught by the caller, in case this

situation is not expected. Just the log level is changed.

  1. … 1 more file in changeset.
OAK-8580: Add distinct cloud data store loggers for stream activity

This change allows for specific monitoring of specific named loggers

that indicate where cloud data stores are streaming bytes to or

from cloud storage. Useful for instances that are trying to

identify where upload or download streaming through the repository

is taking place for those that are attempting to replace such

activity with direct binary access in clients.

  1. … 1 more file in changeset.
OAK-8552: Add data store config to skip exists check on download URI generation

This change adds a new optional configuration parameter for AzureDataStore and S3DataStore. The new parameter, named "presignedHttpDownloadURIVerifyExists", has a default value of true which means the createHttpDownloadURI() method will ensure the requested blob exists in blob storage before creating the signed download URI. If this parameter is set to false, the exists check will be skipped, resulting in higher signed download URI generation performance but potentially returning a URI to a nonexistent blob in some cases (see OAK-7998).

  1. … 6 more files in changeset.
OAK-8574: Reduce cloud service API calls in completeHttpUpload()

  1. … 1 more file in changeset.
OAK-8520: Return existing DataRecord when completeUpload called for existing binary

  1. … 2 more files in changeset.
OAK-7998: Verify existence of binary before generating direct download URI

This change adds code and tests to ensure that when generating a direct

download URI we make sure the binary actually exists first. If caching

is being used it is possible that a client requests a direct download

URI for a blob that was already added the standard (i.e. non-direct)

way but is still in the cache and hasn't actually been uploaded to

cloud storage yet - and therefore cannot be downloaded by direct

download URI.

This change also adds some documentation changes to reflect this

possible situation.

  1. … 3 more files in changeset.
OAK-7858: S3#getAllIdentifiers may trim listing when filtering out metadata objects

- Continue iteration till s3 indicates no more entries

  1. … 2 more files in changeset.
OAK-7738: Add method to check for existence of metadata in SharedDataStore

- Added a new method to the ShareDataStore and implementations

  1. … 13 more files in changeset.
OAK-7737: Add edge cases checks and tests for metadata related methods S3/FileDataStore

- Added tests for metadata operations

  1. … 5 more files in changeset.
OAK-7712: Configuration property names to manage direct binary access should be consistent

Patch provided by Matt Ryan

OAK-7569: Direct Binary Access

Blob APIs and S3/Azure implementation for direct binary upload/download. Modified patch from Matt Ryan

  1. … 25 more files in changeset.
OAK-7536: Update aws java sdk version to 1.11.330

- Updated to sdk version 1.11.330

- Changed to have endpoint generation to put a 's3.' prefix which is common to all regions.

  1. … 2 more files in changeset.
OAK-6841: Revert Changes made in OAK-6575 before 1.7.10 is released

reverting r1811179

  1. … 3 more files in changeset.
OAK-6841: Revert Changes made in OAK-6575 before 1.7.10 is released

reverting r1811183

    • ?
    ./blob/cloud/aws/s3/CloudFrontS3SignedUrlProvider.java
  1. … 2 more files in changeset.
OAK-6802: Manage 'secret' property internally in S3/AzureDataStore

- S3 - Closing stream and rereading the stored value such that the only key associated with the actual stored value used

OAK-6802: Manage 'secret' property internally in S3/AzureDataStore

S3 implementation to manage the secret property internally and remove the need to be configured

  1. … 1 more file in changeset.
OAK-6575: Provide a secure external URL to a DataStore binary

- Moved classes from org.apache.jackrabbit.oak.blob.cloud.aws.s3 to org.apache.jackrabbit.oak.blob.cloud.s3

    • ?
    ./blob/cloud/s3/CloudFrontS3SignedUrlProvider.java
  1. … 2 more files in changeset.
OAK-6575: Provide a secure external URL to a DataStore binary

Patch from Ian Boston

    • ?
    ./blob/cloud/aws/s3/CloudFrontS3SignedUrlProvider.java
  1. … 5 more files in changeset.
OAK-6603: [oak-blob-cloud] Remove the older S3 connector relying on JR caching

- Removing unused code and removing references from tests/utils/services etc

  1. … 11 more files in changeset.
OAK-6374: S3Backend masks actual thrown error when problem in filtering properties

- Fixed by not using Maps.fromProperties

- Logging error before filtering also instead of debug

  1. … 2 more files in changeset.
OAK-6242: Add 'container' as an acceptable property name for s3 bucket

- Now s3 bucket can also be specified using the container property

  1. … 1 more file in changeset.
OAK-6047: incorrect metatype annotations

Remove metatype=true

OAK-5146: S3 getOrCreateReferenceKey() should provide a default implementation

* Delegating to super.getOrCreateReferenceKey() to use a transient key when secret not configured

* Tests

  1. … 2 more files in changeset.
OAK-5077: Class "AbstractS3DataStoreService" should be declared abstract
OAK-5005: S3 async upload fails to catch and log exceptions other than DataStoreException

Log warning with details and add to the retry mechanism to upload again

OAK-4870: Implement caching for S3DataStore

* Use fix for getReference() for S3DataStore

* Added test for reference when secret not available.

* Throwing exception when secret not available.

  1. … 1 more file in changeset.
OAK-4870: Implement caching for S3DataStore

S3DataStore implementation using AbstractCachingDataStore

    • ?
    ./blob/cloud/s3/AbstractS3DataStoreService.java
    • ?
    ./blob/cloud/s3/S3DataStoreService.java
    • ?
    ./blob/cloud/s3/SharedS3DataStoreService.java
  1. … 5 more files in changeset.