In AEM as Cloud Service, Asset Ingestion & Processing has been changed. It offers “AEM Cloud Assets Service” a program powered by “Asset Microservices” which provides a scalable & resilient processing of assets using cloud service & handling of Dynamic Media.
It’s a well thought architecture which resolves load & performance related issues which are normally observed on Author & Publish Instances during asset ingestion & processing. AEM as Cloud Service no longer supports famously known “DAM Update Asset Workflow”.
The key benefits of asset microservices are
- Scalable architecture for seamless processing of resource-intensive operations.
- Efficient indexing & text extractions that does not impact AEM environments.
- Minimize the need for workflows to handle asset processing in AEM environments. This frees up the resources, minimize the load on AEM & provides scalability.
- Improved resilience of asset processing, Potential issues while handling atypical files, such as corrupted files or extremely large files do not impact deployments performance.
- Simplified configuration of asset processing for the admins.
- Asset processing is managed & maintained by Adobe to provide best known configuration for handling renditions, metadata & text extraction for various file types.
- Ability to configure post-processing workflow to add user-specific actions & integrations.
AEM as Cloud Service Assets Service (Courtesy: Adobe)
Let’s divide the architecture into topics to understand in detail
Direct Binary Access
- This feature enables AEM to upload/download binaries directly to/from a supported Oak blob storage without streaming the binaries through application and Oak [AEM is based on Apache Jackrabbit Oak].
- To use this feature underlying storage provider must support the capability & the corresponding Oak Blob Store must implement feature support.
- This feature frees AEM applications from the network, memory & disk resource utilization involved in transferring binary files thus by eliminating potential bottleneck of network capacity on AEM application server & it’s deployment environment.
- It provides application with scalability, high-bandwidth access directly to the cloud storage.
- This API is independent of cloud provider & relies on standard HTTP binary upload & download.
- The following dialog shows basic architecture for binary storage & access. It involves three parties A remote client, Server Application (AEM) & Cloud Binary Storage.
Basic Architecture Diagram of Binary Cloud Storage & Access (Courtesy: Apache)
Binary Cloud Storage
- In AEM On-Premises “crx-quickstart/repository/datastore” folder is used to store binary data with three storage options as below
- File Data Store
- Amazon S3 Cloud Data Store
- Azure Cloud Data Store.
- In AEM Assets Service, “Direct Binary Access” feature is implemented in below storage options.
- Amazon S3 Cloud (S3DataStore)
- Azure Blob Storage (AzureDataStore)
- Clients, such as web browsers or Adobe Asset Link, send an upload request to Experience Manager & start uploading the binary directly to the binary storage.
- When the direct binary upload completes, the client notifies Experience Manager.
- Experience Manager sends a processing request to asset microservices. The request content depends on the processing profiles configuration in the Experience Manager that specify which renditions to generate.
- Asset microservices back-end receives the request, dispatches it to one or more microservices based on the request. Each microservice accesses the original binary directly from the binary cloud storage.
- Result of the processing, such as renditions are stored in the binary cloud storage.
- Experience Manager is notified that the processing is complete along with direct pointers to the generated binaries (renditions). The generated renditions are available in Experience Manager for the uploaded asset.
- When Experience Manager calls “Dynamic Media”, it serves the requested asset by pulling the asset binaries from “Binary Cloud Storage”.
Custom Asset Post Processing
- There are use cases where assets ingested/uploaded requires additional processing, for such cases “Custom Post-Processing Workflows” can be used.
- Post-processing workflows are regular Adobe Experience Manager workflow models, created & managed in AEM Workflow Editor.
- AEM can be configured to automatically trigger the post-processing workflows after asset processing completes.
In AEM Assets Service, assets can be directly uploaded to the cloud storage by using HTTP API as stated above. The steps to upload binary files are as below.
- Submit an HTTP request to upload a new binary.
- Put the content of the binary to one or more URIs provided by the initiate request.
- Submit an HTTP request to inform the server the status of the uploaded binary.
AEM as Cloud Service Asset Upload (Courtesy: Adobe)
- Submit an HTTP Post request to the desired folder where asset to be create/updated.
- Include “.initiateUpload.json” to indicate that the request is to initiate upload of a binary file.
- For Example: if the path for asset to be uploaded in “/assets/folder” then POST request is
- The content type of the request body should be application/x-www-form-urlencodedform data, containing the following fields:
- (string) fileName: The name of the asset as it appears in Experience Manager.
- (number) fileSize: The file size, in bytes, of the asset being uploaded.
- The output of initiating an upload includes one or more upload URI values.
- If more than one URI is provided, the client splits the binary into parts and make PUT requests of each part to each URI, in order to use all URIs.
- Ensure that the size of each part is within the minimum and maximum sizes as specified in the initiate response
- CDN edge nodes help accelerate the requested upload of binaries.
- After all the parts of a binary file are uploaded, submit an HTTP POST request to the complete URI provided by the initiation data.
- The content type of the request body should be application/x-www-form-urlencodedform data, containing the following fields.
Open-source Upload Library
Adobe provides two open source libraries & tools for upload of assets.