Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions ChangeLog.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
Note: This is an Azure Storage only package. The all up Azure node sdk still has the old storage bits in there. In a future release, those storage bits will be removed and an npm dependency to this storage node sdk will
be taken. This is a GA release and the changes described below indicate the changes from the Azure node SDK 0.9.8 available here - https://github.com/Azure/azure-sdk-for-node.

2018.08 Version 2.10.1

ALL
* Added a parameter `enableGlobalHttpAgent` to all services. To enable global HTTP(s) agent, please set `{blob|queue|table|file}Service.enableGlobalHttpAgent` to true.
* Fixed a bug that content type value is incorrect for json.

2018.06 Version 2.10.0

ALL
Expand Down
16 changes: 16 additions & 0 deletions ISSUE_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
### Which service(blob, file, queue, table) does this issue concern?


### Which version of the SDK was used?


### What's the Node.js/Browser version?


### What problem was encountered?


### Steps to reproduce the issue?


### Have you found a mitigation/solution?
7 changes: 7 additions & 0 deletions browser/ChangeLog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
Note: This is the change log file for Azure Storage JavaScript Client Library.

2018.08 Version 2.10.101

ALL
* Generated browser compatible JavaScript files based on Microsoft Azure Storage SDK for Node.js 2.10.1.
* Fixed a bug that content type value is incorrect for json.
* Fixed an issue that user agent is set in browser environment.

2018.06 Version 2.10.100

ALL
Expand Down
97 changes: 52 additions & 45 deletions lib/common/services/storageserviceclient.js
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
//
//
// Copyright (c) Microsoft and contributors. All rights reserved.
//
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// http://www.apache.org/licenses/LICENSE-2.0
//
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
//
//
// See the License for the specific language governing permissions and
// limitations under the License.
//
//

// Module dependencies.
var request = require('../request-wrapper');
Expand Down Expand Up @@ -254,12 +254,14 @@ StorageServiceClient.prototype._performRequest = function (webResource, body, op
if(!options.clientRequestId) {
options.clientRequestId = guid.v1();
}

webResource.withHeader(HeaderConstants.CLIENT_REQUEST_ID, options.clientRequestId);

// Sets the user-agent string
var userAgentComment = util.format('(NODE-VERSION %s; %s %s)', process.version, os.type(), os.release());
webResource.withHeader(HeaderConstants.USER_AGENT, Constants.USER_AGENT_PRODUCT_NAME + '/' + Constants.USER_AGENT_PRODUCT_VERSION + ' ' + userAgentComment);
// Sets the user-agent string if the process is not started by the browser
if(!process.browser) {
var userAgentComment = util.format('(NODE-VERSION %s; %s %s)', process.version, os.type(), os.release());
webResource.withHeader(HeaderConstants.USER_AGENT, Constants.USER_AGENT_PRODUCT_NAME + '/' + Constants.USER_AGENT_PRODUCT_VERSION + ' ' + userAgentComment);
}

// Initialize the location that the request is going to be sent to.
if(azureutil.objectIsNull(options.locationMode)) {
Expand All @@ -277,18 +279,18 @@ StorageServiceClient.prototype._performRequest = function (webResource, body, op
}

this._initializeLocation(options);

// Initialize the operationExpiryTime
this._setOperationExpiryTime(options);

// If the output stream already got sent to server and got error back,
// If the output stream already got sent to server and got error back,
// we should NOT retry within the SDK as the stream data is not valid anymore if we retry directly.
// And it's very hard for SDK to re-wind the stream.
//
// If users want to retry on this kind of error, they can implement their own logic to parse the response and
// If users want to retry on this kind of error, they can implement their own logic to parse the response and
// determine if they need to re-prepare a stream and call our SDK API to retry.
//
// Currently for blobs/files with size greater than 32MB (DEFAULT_SINGLE_BLOB_PUT_THRESHOLD_IN_BYTES),
// Currently for blobs/files with size greater than 32MB (DEFAULT_SINGLE_BLOB_PUT_THRESHOLD_IN_BYTES),
// we'll send the steam by chunk buffers which doesn't have this issue.
var outputStreamSent = false;

Expand Down Expand Up @@ -333,7 +335,7 @@ StorageServiceClient.prototype._performRequest = function (webResource, body, op
var requestStream;

var requestWithDefaults;

if(self.proxy) {
if(requestWithDefaults === undefined) {
requestWithDefaults = request.defaults({'proxy':self.proxy});
Expand All @@ -356,7 +358,7 @@ StorageServiceClient.prototype._performRequest = function (webResource, body, op
if (contentLength !== undefined) {
errorMessageBuffer = new Buffer(contentLength);
}

requestStream.on('data', function (data) {
if (contentLength !== undefined) {
data.copy(errorMessageBuffer, index);
Expand Down Expand Up @@ -402,7 +404,7 @@ StorageServiceClient.prototype._performRequest = function (webResource, body, op
if(azureutil.objectIsNull(options.disableContentMD5Validation) || options.disableContentMD5Validation === false) {
response.contentMD5 = internalHash.digest('base64');
}

response.length = responseLength;
endResponse = response;
});
Expand Down Expand Up @@ -443,7 +445,7 @@ StorageServiceClient.prototype._performRequest = function (webResource, body, op
if (!azureutil.isBrowser() && Buffer.isBuffer(body.outputData)) {
// Request module will take 200MB additional memory when we pass a 100MB buffer as body
// Transfer buffer to stream will highly reduce the memory used by request module
finalRequestOptions.body = new BufferStream(body.outputData);
finalRequestOptions.body = new BufferStream(body.outputData);
} else {
finalRequestOptions.body = body.outputData;
}
Expand Down Expand Up @@ -578,10 +580,10 @@ StorageServiceClient.prototype._buildRequestOptions = function (webResource, bod
}
webResource.withHeader(HeaderConstants.ACCEPT_CHARSET, 'UTF-8');

// Browsers cache GET/HEAD requests by adding conditional headers such as 'IF_MODIFIED_SINCE' after Azure Storage 'Authorization header' calculation,
// which may result in a 403 authorization error. So add timestamp to GET/HEAD request URLs thus avoid the browser cache.
// Browsers cache GET/HEAD requests by adding conditional headers such as 'IF_MODIFIED_SINCE' after Azure Storage 'Authorization header' calculation,
// which may result in a 403 authorization error. So add timestamp to GET/HEAD request URLs thus avoid the browser cache.
if (azureutil.isBrowser() && (
webResource.method === Constants.HttpConstants.HttpVerbs.GET ||
webResource.method === Constants.HttpConstants.HttpVerbs.GET ||
webResource.method === Constants.HttpConstants.HttpVerbs.HEAD)) {
webResource.withQueryOption(HeaderConstants.FORCE_NO_CACHE_IN_BROWSER, new Date().getTime());
}
Expand All @@ -597,7 +599,7 @@ StorageServiceClient.prototype._buildRequestOptions = function (webResource, bod
if(!azureutil.objectIsNull(options.timeoutIntervalInMs) && options.timeoutIntervalInMs > 0) {
webResource.withQueryOption(QueryStringConstants.TIMEOUT, Math.ceil(options.timeoutIntervalInMs / 1000));
}

if(options.accessConditions) {
webResource.withHeader(HeaderConstants.IF_MATCH, options.accessConditions.EtagMatch);
webResource.withHeader(HeaderConstants.IF_MODIFIED_SINCE, options.accessConditions.DateModifedSince);
Expand All @@ -616,7 +618,7 @@ StorageServiceClient.prototype._buildRequestOptions = function (webResource, bod
webResource.withHeader(HeaderConstants.SOURCE_IF_NONE_MATCH, options.sourceAccessConditions.EtagNonMatch);
webResource.withHeader(HeaderConstants.SOURCE_IF_UNMODIFIED_SINCE, options.sourceAccessConditions.DateUnModifiedSince);
}

if (!webResource.headers || webResource.headers[HeaderConstants.CONTENT_TYPE] === undefined) {
// work around to add an empty content type header to prevent the request module from magically adding a content type.
webResource.headers[HeaderConstants.CONTENT_TYPE] = '';
Expand All @@ -634,6 +636,8 @@ StorageServiceClient.prototype._buildRequestOptions = function (webResource, bod
delete webResource.headers[HeaderConstants.CONTENT_LENGTH];
}

var enableGlobalHttpAgent = this.enableGlobalHttpAgent;

// Sets the request url in the web resource.
this._setRequestUrl(webResource, options);

Expand All @@ -657,14 +661,17 @@ StorageServiceClient.prototype._buildRequestOptions = function (webResource, bod
//set encoding of response data. If set to null, the body is returned as a Buffer
requestOptions.encoding = options.responseEncoding;
}

if (options && options.clientRequestTimeoutInMs) {
requestOptions.timeout = options.clientRequestTimeoutInMs;
} else {
requestOptions.timeout = Constants.DEFAULT_CLIENT_REQUEST_TIMEOUT_IN_MS; // 2 minutes
}

requestOptions.forever = true;
// If global HTTP agent is not enabled, use forever agent.
if (enableGlobalHttpAgent !== true) {
requestOptions.forever = true;
}
}

callback(error, requestOptions);
Expand Down Expand Up @@ -946,7 +953,7 @@ StorageServiceClient.prototype._setRequestUrl = function (webResource, options)
if(host && host.lastIndexOf('/') !== (host.length - 1)){
host = host + '/';
}

var fullPath = url.format({pathname: webResource.path, query: webResource.queryString});
webResource.uri = url.resolve(host, fullPath);
webResource.path = url.parse(webResource.uri).pathname;
Expand Down Expand Up @@ -1033,7 +1040,7 @@ StorageServiceClient.prototype.parseMetadataHeaders = function (headers) {
metadata[key] = headers[header];
}
}

return metadata;
};

Expand All @@ -1043,7 +1050,7 @@ StorageServiceClient.prototype.parseMetadataHeaders = function (headers) {
*
* @this {StorageServiceClient}
* @param {object} [options] The request options.
* @param {LocationMode} [options.locationMode] Specifies the location mode used to decide which location the request should be sent to.
* @param {LocationMode} [options.locationMode] Specifies the location mode used to decide which location the request should be sent to.
* Please see StorageUtilities.LocationMode for the possible values.
* @param {int} [options.timeoutIntervalInMs] The server timeout interval, in milliseconds, to use for the request.
* @param {int} [options.clientRequestTimeoutInMs] The timeout of client requests, in milliseconds, to use for the request.
Expand All @@ -1052,40 +1059,40 @@ StorageServiceClient.prototype.parseMetadataHeaders = function (headers) {
* execution time is checked intermittently while performing requests, and before executing retries.
* @param {bool} [options.useNagleAlgorithm] Determines whether the Nagle algorithm is used; true to use the Nagle algorithm; otherwise, false.
* The default value is false.
* @param {errorOrResult} callback `error` will contain information if an error occurs; otherwise, `result` will contain the properties
* @param {errorOrResult} callback `error` will contain information if an error occurs; otherwise, `result` will contain the properties
* and `response` will contain information related to this operation.
*/
StorageServiceClient.prototype.getAccountServiceProperties = function (optionsOrCallback, callback) {
var userOptions;
azureutil.normalizeArgs(optionsOrCallback, callback, function (o, c) { userOptions = o; callback = c; });

validate.validateArgs('getServiceProperties', function (v) {
v.callback(callback);
});

var options = extend(true, {}, userOptions);

var webResource = WebResource.get()
.withQueryOption(QueryStringConstants.COMP, 'properties')
.withQueryOption(QueryStringConstants.RESTYPE, 'service');

options.requestLocationMode = RequestLocationMode.PRIMARY_OR_SECONDARY;

var processResponseCallback = function (responseObject, next) {
responseObject.servicePropertiesResult = null;
if (!responseObject.error) {
responseObject.servicePropertiesResult = ServicePropertiesResult.parse(responseObject.response.body.StorageServiceProperties);
}

// function to be called after all filters
var finalCallback = function (returnObject) {
callback(returnObject.error, returnObject.servicePropertiesResult, returnObject.response);
};

// call the first filter
next(responseObject, finalCallback);
};

this.performRequest(webResource, null, options, processResponseCallback);
};

Expand All @@ -1096,7 +1103,7 @@ StorageServiceClient.prototype.getAccountServiceProperties = function (optionsOr
* @this {StorageServiceClient}
* @param {object} serviceProperties The service properties.
* @param {object} [options] The request options.
* @param {LocationMode} [options.locationMode] Specifies the location mode used to decide which location the request should be sent to.
* @param {LocationMode} [options.locationMode] Specifies the location mode used to decide which location the request should be sent to.
* Please see StorageUtilities.LocationMode for the possible values.
* @param {int} [options.timeoutIntervalInMs] The server timeout interval, in milliseconds, to use for the request.
* @param {int} [options.clientRequestTimeoutInMs] The timeout of client requests, in milliseconds, to use for the request.
Expand All @@ -1112,30 +1119,30 @@ StorageServiceClient.prototype.getAccountServiceProperties = function (optionsOr
StorageServiceClient.prototype.setAccountServiceProperties = function (serviceProperties, optionsOrCallback, callback) {
var userOptions;
azureutil.normalizeArgs(optionsOrCallback, callback, function (o, c) { userOptions = o; callback = c; });

validate.validateArgs('setServiceProperties', function (v) {
v.object(serviceProperties, 'serviceProperties');
v.callback(callback);
});

var options = extend(true, {}, userOptions);
var servicePropertiesXml = ServicePropertiesResult.serialize(serviceProperties);

var webResource = WebResource.put()
.withQueryOption(QueryStringConstants.COMP, 'properties')
.withQueryOption(QueryStringConstants.RESTYPE, 'service')
.withHeader(HeaderConstants.CONTENT_TYPE, 'application/xml;charset="utf-8"')
.withHeader(HeaderConstants.CONTENT_LENGTH, Buffer.byteLength(servicePropertiesXml))
.withBody(servicePropertiesXml);

var processResponseCallback = function (responseObject, next) {
var finalCallback = function (returnObject) {
callback(returnObject.error, returnObject.response);
};

next(responseObject, finalCallback);
};

this.performRequest(webResource, webResource.body, options, processResponseCallback);
};

Expand Down Expand Up @@ -1164,14 +1171,14 @@ StorageServiceClient._normalizeError = function (error, response) {
// blob/queue errors should have error.Error, table errors should have error['odata.error']
var errorProperties = error.Error || error.error || error['odata.error'] || error['m:error'] || error;
normalizedError.code = errorProperties.message; // The message exists when there is error.Error.

for (var property in errorProperties) {
if (errorProperties.hasOwnProperty(property)) {
var key = property.toLowerCase();
if(key.indexOf('m:') === 0) {
key = key.substring(2);
}

normalizedError[key] = errorProperties[property];

// if this is a table error, message is an object - flatten it to normalize with blob/queue errors
Expand Down
4 changes: 2 additions & 2 deletions lib/common/util/constants.js
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ var Constants = {
* @const
* @type {string}
*/
USER_AGENT_PRODUCT_VERSION: '2.10.0',
USER_AGENT_PRODUCT_VERSION: '2.10.1',

/**
* The number of default concurrent requests for parallel operation.
Expand Down Expand Up @@ -1448,7 +1448,7 @@ var Constants = {
* @const
* @type {string}
*/
JSON_CONTENT_TYPE_VALUE: 'application/json;',
JSON_CONTENT_TYPE_VALUE: 'application/json',

/**
* The header that specifies storage SKU, also known as account type.
Expand Down
2 changes: 2 additions & 0 deletions lib/services/blob/blobservice.core.js
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ var StorageError = errors.StorageError;
* the value specified by the singleBlobPutThresholdInBytes property in size.
* useNagleAlgorithm Determines whether the Nagle algorithm is used for requests made via the Blob service; true to use the
* Nagle algorithm; otherwise, false. The default value is false.
* enableGlobalHttpAgent Determines whether global HTTP(s) agent is enabled; true to use Global HTTP(s) agent; otherwise, false to use
* http(s).Agent({keepAlive:true}).
* @constructor
* @extends {StorageServiceClient}
*
Expand Down
2 changes: 2 additions & 0 deletions lib/services/file/fileservice.core.js
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,8 @@ var ArgumentError = errors.ArgumentError;
* parallelOperationThreadCount The number of parallel operations that may be performed when uploading a file.
* useNagleAlgorithm Determines whether the Nagle algorithm is used for requests made via the file service; true to use the
* Nagle algorithm; otherwise, false. The default value is false.
* enableGlobalHttpAgent Determines whether global HTTP(s) agent is enabled; true to use Global HTTP(s) agent; otherwise, false to use
* http(s).Agent({keepAlive:true}).
* @constructor
* @extends {StorageServiceClient}
*
Expand Down
2 changes: 2 additions & 0 deletions lib/services/queue/queueservice.js
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ var ServiceStatsParser = azureCommon.ServiceStatsParser;
* defaultLocationMode The default location mode for requests made via the Queue service.
* useNagleAlgorithm Determines whether the Nagle algorithm is used for requests made via the Queue service; true to use the
* Nagle algorithm; otherwise, false. The default value is false.
* enableGlobalHttpAgent Determines whether global HTTP(s) agent is enabled; true to use Global HTTP(s) agent; otherwise, false to use
* http(s).Agent({keepAlive:true}).
* @constructor
* @augments {StorageServiceClient}
*
Expand Down
Loading