HomeAIDemystifying Azure Storage Account Community Entry | by René Bremer | Oct,...

Demystifying Azure Storage Account Community Entry | by René Bremer | Oct, 2024


Service endpoints and personal endpoints hands-on: together with Azure Spine, storage account firewall, DNS, VNET and NSGs

Related Community — picture by Nastya Dulhiier on Unsplash

Storage accounts play a significant function in a medallion structure for establishing an enterprise knowledge lake. They act as a centralized repository, enabling seamless knowledge trade between producers and customers. This setup empowers customers to carry out knowledge science duties and construct machine studying (ML) fashions. Moreover, customers can use the information for Retrieval Augmented Technology (RAG), facilitating interplay with firm knowledge by way of Giant Language Fashions (LLMs) like ChatGPT.

Extremely delicate knowledge is often saved within the storage account. Protection in depth measures should be in place earlier than knowledge scientists and ML pipelines can entry the information. To do protection in depth, a number of measurement shall be in place comparable to 1) superior menace safety to detect malware, 2) authentication utilizing Microsoft Entra, 3) authorization to do wonderful grained entry management, 4) audit path to watch entry, 5) knowledge exfiltration prevention, 6) encryption, and final however not least 7) community entry management utilizing service endpoint or personal endpoints.

This text focuses on community entry management of the storage account. Within the subsequent chapter, the totally different ideas are defined (demystified) on storage account community entry. Following that, a hands-on comparability is finished between service endpoint and personal endpoints. Lastly, a conclusion is drawn.

A typical situation is {that a} digital machine must have community entry to a storage account. This digital machine typically acts as a Spark cluster to investigate knowledge from the storage account. The picture under offers an outline of the obtainable community entry controls.

2.1 Overview of networking between digital machine and storage account — picture by creator

The parts within the picture may be described as follows:

Azure international community — spine: Site visitors all the time goes over Azure spine between two areas (until buyer forces to not do it), see additionally Microsoft international community — Azure | Microsoft Be taught. That is no matter what firewall rule is used within the storage account and regardless whether or not service endpoints or personal endpoints are used.

Azure storage firewalls: Firewall guidelines can prohibit or disable public entry. Widespread guidelines embrace whitelisting VNET/subnet, public IP addresses, system-assigned managed identities as useful resource cases, or permitting trusted providers. When a VNET/subnet is whitelisted, the Azure Storage account identifies the visitors’s origin and its personal IP deal with. Nonetheless, the storage account itself shouldn’t be built-in into the VNET/subnet — personal endpoints are wanted for that goal.

Public DNS storage account: Storage accounts will all the time have a public DNS that may be entry by way of community tooling, see additionally Azure Storage Account — Public Entry Disabled — however nonetheless some stage of connectivity — Microsoft Q&A. That’s, even when public entry is disabled within the storage account firewall, the general public DNS will stay.

Digital Community (VNET): Community by which digital machines are deployed. Whereas a storage account isn’t deployed inside a VNET, the VNET may be whitelisted within the Azure storage firewall. Alternatively, the VNET can create a personal endpoint for safe, personal connectivity.

Service endpoints: When whitelisting a VNET/subnet within the Storage account firewall, the service endpoint should be turned on for the VNET/subnet. The service endpoint needs to be Microsoft.Storage when the VNET and storage account are in the identical area or Microsoft.Storage.World when the VNET and storage are in numerous areas. Be aware that service endpoints can be used as an overarching time period, encompassing each the whitelisting of a VNET/subnet on the Azure Storage Firewall and the enabling of the service endpoint on the VNET/subnet.

Non-public endpoints: Integrating a Community Interface Card (NIC) of a Storage Account throughout the VNET the place the digital machine operates. This integration assigns the storage account a personal IP deal with, making it a part of the VNET.

Non-public DNS storage account: Inside a VNET, a personal DNS zone may be created by which the storage account DNS resolves to the personal endpoint. That is to guarantee that digital machine can nonetheless hook up with the URL of the storage account and the URL of the storage account resolves to a personal IP deal with moderately than a public deal with.

Community Safety Group (NSG): Deploy an NSG to restrict inbound and outbound entry of the VNET the place the digital machine runs. This may forestall knowledge exfiltration. Nonetheless, an NSG works solely with IP addresses or tags, not with URLs. For extra superior knowledge exfiltration safety, use an Azure Firewall. For simplicity, the article omits this and makes use of NSG to dam outbound visitors.

Within the subsequent chapter, service endpoints and personal endpoints are mentioned.

The chapter begins by exploring the situation of unrestricted community entry. Then the small print of service endpoints and personal endpoints are mentioned with sensible examples.

3.1 Not limiting community entry — public entry enabled

Suppose the next situation by which a digital machine and a storage account is created. The firewall of the storage account has public entry enabled, see picture under.

3.1.1 digital machine and storage account with public entry created

Utilizing this configuration, a the digital machine can entry the storage account over the community. For the reason that digital machine can be deployed in Azure, visitors will go over Azure Spine and can be accepted, see picture under.

3.1.2 Site visitors not blocked — public community entry enabled

Enterprises usually set up firewall guidelines to restrict community entry. This entails disabling public entry or permitting solely chosen networks and whitelisting particular ones. The picture under illustrates public entry being disabled and visitors being blocked by the firewall.

3.1.3 Site visitors blocked — blocking visitors in storage account firewall

Within the subsequent paragraph, service endpoints and chosen community firewall guidelines are used to grant community entry to storage account once more.

3.2 Limiting community entry by way of Service endpoints

To allow digital machine VNET entry to the storage account, activate the service endpoint on the VNET. Use Microsoft.Storage for throughout the areas or Microsoft.Storage.World for cross area. Subsequent, whitelist the VNET/subnet within the storage account firewall. Site visitors is then blocked once more, see additionally picture under.

3.2.1 Site visitors not blocked — service endpoint enabled and added to in storage account firewall

Site visitors is now accepted. When VNET/subnet is faraway from Azure storage account firewall or public entry is disabled, then visitors is blocked once more.

In case an NSG is used to dam public outbound IPs within the VNET of the digital machine, then visitors can be blocked once more. It is because the general public DNS of the storage account is used, see additionally picture under.

3.2.2 Site visitors blocked — NSG of digital machine blocking public outbound visitors

In that case, personal endpoints shall be used to guarantee that visitors doesn’t depart VNET. That is mentioned within the subsequent chapter.

3.3 Limiting entry by way of Non-public endpoints

To reestablish community entry for the digital machine to the storage account, use a personal endpoint. This motion creates a community interface card (NIC) for the storage account throughout the VNET of the digital machine, making certain that visitors stays throughout the VNET. The picture under offers additional illustration.

3.3.1 Site visitors not blocked — Non-public endpoint created to Storage account, public entry disabled

Once more, an NSG can be utilized once more to dam all visitors, see picture under.

3.3.2 Site visitors blocked — NSG of digital machine blocking all outbound visitors

That is nevertheless counterintuitive, since first a personal endpoint is created within the VNET after which visitors is blocked by NSG in the identical VNET.

Enterprise all the time requires community guidelines in place to restrict community entry to their storage account. On this weblog publish, each service endpoints and personal endpoint are thought of to restrict entry.

Each is true for service endpoints and personal endpoints:

For service endpoints, the next maintain:

  • Requires to allow service endpoints on VNET/subnet and whitelisting of VNET/subnet in Azure storage account firewall.
  • Requires that visitors leaves the VNET of the digital machine that’s connecting to the storage account. See above, the visitors stays on the Azure spine.

For personal endpoints, the next maintain:

  • Public entry may be disabled within the Azure Storage firewall. See above, public DNS entry of storage account will stay.
  • Site visitors doesn’t depart the VNET by which the digital machine additionally runs.

There are numerous different issues to think about whether or not to make use of service endpoints or personal endpoints (prices, migration effort since service endpoints have been on the market longer than personal endpoints, networking complexity when utilizing personal endpoints, restricted service endpoint help of newer Azure providers, laborious restrict of quantity personal endpoints in storage account of 200).

Nonetheless, in case it’s required (“will need to have”) that 1) visitors shall by no means depart VNET/subnet of digital machine or 2) it’s not allowed to create firewall guidelines in Azure storage firewall and should be locked down, then service endpoint shouldn’t be possible.

In different eventualities, it’s attainable to think about each options, and the perfect match needs to be decided primarily based on the particular necessities of every situation.



Supply hyperlink

latest articles

Lightinthebox WW
Earn Broker Many GEOs

explore more