When using SMB share with Active Directory authentication on File Gateway, sometimes you may get System Error 5 – Access is Denied. There could be a few reasons why you may get this error. In this article, we will go through couple of them.
One of the general reasons you get this error is mismatch between the client SMB version, and the File Gateway expecting SMB version. In AWS File Gateway, you can set and allow the SMB versions that clients can use with SMB security strategy.
SMB Security Strategy :
By default, SMB security strategy is set to Enforce Encryption. With Mandatory Encryption, clients that support SMBv3 are only allowed to establish connection with File Gateway. You can get the current SMB security strategy on your File Gateway running the following command. Replace generic values with your account ID and Gateway ID.$ aws storagegateway describe-smb-settings --gateway-arn arn:aws:storagegateway:us-west-2:11122223333:gateway/sgw-1234ABCD
In below output, you can see the SMB security strategy is set to Mandatory Encryption which allows only SMBv3 connections.
{
“GatewayARN”: “arn:aws:storagegateway:us-west-2:11122223333:gateway/sgw-1234ABCD
“,
“DomainName”: “example.com,
“ActiveDirectoryStatus”: “JOINED”,
“SMBGuestPasswordSet”: true,
“SMBSecurityStrategy”: “MandatoryEncryption”,
“FileSharesVisible”: true
}
Now if I try to mount the SMB file share on a client like Windows 2008 or older, which doesn’t support SMBv3, I would see error System Error 5 – Access is Denied. You can fix this by updating the SMB security strategy to either MandatorySigning or ClientSpecified running the following command. Using ClientSpecified you can let the client to negotiate SMBv1, v2 or V3 to connect and map File Gateway SMB share.
$ aws storagegateway update-smb-security-strategy --gateway-arn arn:aws:storagegateway:us-west-2:1122223333:gateway/sgw-1234ABCD--smb-security-strategy ClientSpecified
Cross Domain hostname with Active Directory Forest Trust:
This is another scenario where you can receive this error. You have a File Gateway that is joined to domain example.com which has an Active Directory forest trust relationship with domain contoso.net. You joined the File Gateway to example.com Active Directory domain. But, you have set the hostname in the File Gateway configuration to the trusted domain as fgw.contoso.net as show in below image. You can connect to Gateway VM console and verify below configuration from 2. Network Configuration 4. View Hostname Configuration

This is because of the updates that Microsoft released to the Domain Controllers after Jan 2022. You can find more details here in this link. You can see the following notes that say the requests over a domain trust must use the same domain name as trusting domain.
- The requests over a domain trust must use the same domain name as the trusting domain.
- The requests over a forest trust must use a domain name that is a member of the trusting forest, and doesn’t have a name collision from other forests.
- The requests forwarded by an RODC must use a client name that the RODC has been previously authorized to cache secrets for
If you enable NTLM logging as below and verify in netlogon file under C:Windows\debug you could see messages like LsaIFilterInboundNamespace.

To fix this, you have to change the hostname on the File Gateway configuration and set it to hostname to which the gateway is joined to. In this case, as the gateway is joined to example.com, you have to set the hostname like fgw.example.com and retry mapping the share. After correcting the hostname on the gateway configuration to the domain that is gateway joined to(trusted domain), you can map the share without this error.

Feel free to leave comments if you have any questions.