{"id":1668,"date":"2021-06-02T19:13:43","date_gmt":"2021-06-03T02:13:43","guid":{"rendered":"http:\/\/blog.nillsf.com\/?p=1668"},"modified":"2021-06-02T19:13:52","modified_gmt":"2021-06-03T02:13:52","slug":"automatically-turning-on-diagnostic-settings-using-azure-policy","status":"publish","type":"post","link":"https:\/\/nillsf.com\/index.php\/2021\/06\/02\/automatically-turning-on-diagnostic-settings-using-azure-policy\/","title":{"rendered":"Automatically turning on diagnostic settings using Azure Policy"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Earlier today, Camila Martins joined the latest episode Unsung Heroes of the Cloud. She did an amazing job explaining how to manage Azure diagnostics settings at scale:<\/p>\n\n\n\n<iframe loading=\"lazy\" width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/uXhE93tJ1Ls\" title=\"YouTube video player\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n\n\n\n<p class=\"wp-block-paragraph\">She did such a great job explaining things, that I actually want to try out what she showed. The goal of this blog post is to explore how to automatically turn on diagnostics settings using Azure Policy. Specifically, I want to turn this on automatically for network security group (NSG) flow logs. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For an introduction to Azure Policy, <a href=\"https:\/\/blog.nillsf.com\/index.php\/2019\/11\/02\/using-azure-policy-to-deny-public-ips-on-specific-vnets\/\">please refer to this earlier post<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s have a look. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What are diagnostic settings?<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The embedded video explains what diagnostic settings are. In summary, they are resource-specific logs in Azure that are not stored by default. As a customer, you have to enable them per service. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">By capturing diagnostic logs, you get more resource-specific information. In the case of for example Azure storage, that is an overview of the transactions happening in your storage account. By default, Azure monitor will only capture high-level information such as total transaction count, without for example item level details.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the case of NSG flow logs, the flow logs generate detailed information about source IP and port, destination IP and port, the protocol, whether the traffic was allowed\/denied, and also a counter of packets. This can be very valuable information for security analysis as well as for troubleshooting (I&#8217;ve used NSG flow logs more than once to verify if traffic is reaching its intended location).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let&#8217;s have a look at how to automatically enable them.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How to do it automatically<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To enable automatically enable diagnostic settings, you can use Azure Policy. Azure Policy has the option to &#8220;deployIfNotExists&#8221; when a new resource is created that doesn&#8217;t have the flow logs enabled. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There is a default policy definition that you can use to enable this called &#8220;Deploy a flow log resource with target network security group&#8221;. Let&#8217;s have a look at what setting this up looks like.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Setting it up<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To start, <a href=\"https:\/\/portal.azure.com\/#blade\/Microsoft_Azure_Policy\/PolicyMenuBlade\/Definitions\">open the policy definitions blade<\/a> in the Azure portal. There you can look for the policies containing the &#8220;flow log&#8221; string. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"493\" src=\"\/wp-content\/uploads\/2021\/06\/image-1024x493.png\" alt=\"\" class=\"wp-image-1670\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-1024x493.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-300x145.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-768x370.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-1536x740.png 1536w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image.png 1789w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Looking for the flow log policies and selecting &#8220;Deploy a flow log resource with target network security group&#8221;<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Select the  &#8220;Deploy a flow log resource with target network security group&#8221; policy and click the assign button. Next, you&#8217;ll have to configure the policy. First, provide it a scope (which in my case will be my full subscription).<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"950\" height=\"1024\" src=\"\/wp-content\/uploads\/2021\/06\/image-1-950x1024.png\" alt=\"\" class=\"wp-image-1671\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-1-950x1024.png 950w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-1-278x300.png 278w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-1-768x828.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-1.png 1040w\" sizes=\"auto, (max-width: 950px) 100vw, 950px\" \/><figcaption>Assigning a scope for the policy<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Next, you&#8217;ll need to provide the region, a storage account resource ID in that region, and the network watcher in that region. To get the storage account ID, open your storage account and go to endpoints to find your resource ID. For network watcher, open the network watcher service in the portal, and copy-paste the name of the resource in the right region. The resource group name should be NetworkWatcherRG. Now provide all those inputs for the policy creation:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"638\" src=\"\/wp-content\/uploads\/2021\/06\/image-2-1024x638.png\" alt=\"\" class=\"wp-image-1672\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-2-1024x638.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-2-300x187.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-2-768x479.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-2.png 1056w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Provide the details of the region, storage account ID, and network watcher.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Next, you can optionally create a remediation task. A remediation task would give you the option to &#8220;fix&#8221; all non-compliant resources directly. This is optional, but I&#8217;m enabling it nonetheless. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1005\" height=\"1024\" src=\"\/wp-content\/uploads\/2021\/06\/image-3-1005x1024.png\" alt=\"\" class=\"wp-image-1673\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-3-1005x1024.png 1005w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-3-294x300.png 294w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-3-768x783.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-3-60x60.png 60w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-3.png 1054w\" sizes=\"auto, (max-width: 1005px) 100vw, 1005px\" \/><figcaption>Optionally setting up a remediation task<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Finally, you could specify a non-compliance message in the final step and then create the policy. I didn&#8217;t specify a non-compliance message and immediately created the policy.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"932\" height=\"1024\" src=\"\/wp-content\/uploads\/2021\/06\/image-4-932x1024.png\" alt=\"\" class=\"wp-image-1674\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-4-932x1024.png 932w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-4-273x300.png 273w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-4-768x843.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-4.png 1048w\" sizes=\"auto, (max-width: 932px) 100vw, 932px\" \/><figcaption>Creating the policy<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now that the policy got created, let&#8217;s try it out!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Trying it out<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">To try it out, I created a new NSG in West US 2 in the portal:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"545\" src=\"\/wp-content\/uploads\/2021\/06\/image-5-1024x545.png\" alt=\"\" class=\"wp-image-1675\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-5-1024x545.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-5-300x160.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-5-768x409.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-5.png 1125w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Creating a new NSG<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">It then took a couple of seconds for the NSG to be created. When I checked it immediately after creation, the flow log wasn&#8217;t created:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"697\" src=\"\/wp-content\/uploads\/2021\/06\/image-6-1024x697.png\" alt=\"\" class=\"wp-image-1676\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-6-1024x697.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-6-300x204.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-6-768x523.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-6.png 1527w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">It then took a good 10 minutes for the flow log to appear. <\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"307\" src=\"\/wp-content\/uploads\/2021\/06\/image-7-1024x307.png\" alt=\"\" class=\"wp-image-1677\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-7-1024x307.png 1024w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-7-300x90.png 300w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-7-768x230.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-7.png 1492w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>After a while the NSG flow log appeared<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">I found the timing to be a bit weird, but checking the activity log showed that it went through fine. The deployIfNotExists action indeed started the moment the NSG was created, it just took a good 10 minutes to deploy.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1009\" height=\"1024\" src=\"\/wp-content\/uploads\/2021\/06\/image-8-1009x1024.png\" alt=\"\" class=\"wp-image-1678\" srcset=\"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-8-1009x1024.png 1009w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-8-296x300.png 296w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-8-768x779.png 768w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-8-60x60.png 60w, https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/image-8.png 1109w\" sizes=\"auto, (max-width: 1009px) 100vw, 1009px\" \/><figcaption>Checking the activity log to confirm how long it took to create the NSG flow log<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">But anyway: that&#8217;s how you can automatically create NSG flow logs for newly created NSGs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Summary<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In this post, we explored how to automatically set up NSG flow logs using Azure Policy. We used the portal to do this for a single region, but as you can guess, this can easily be automated for multiple regions (and multiple services) using automation tools. If you&#8217;re interested in this, I would recommend checking out this video on Enterprise-Scale<a href=\"https:\/\/www.youtube.com\/watch?v=wWLxxj-uMsY\"> Landing Zones Devops<\/a> that covers this pretty well using GitHub Actions.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Earlier today, Camila Martins joined the latest episode Unsung Heroes of the Cloud. She did an amazing job explaining how to manage Azure diagnostics settings at scale: She did such a great job explaining things, that I actually want to try out what she showed. The goal of this blog post is to explore how [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":1680,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[2,4,36,85],"tags":[102,38,49,50],"class_list":["post-1668","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure","category-management","category-networking","category-security","tag-azure-policy","tag-networking","tag-policy","tag-security"],"jetpack_featured_media_url":"https:\/\/nillsfblog.blob.core.windows.net\/media\/2021\/06\/2021-06-02-19_11_47-PowerPoint-Slide-Show-Customize-core-dumps-in-Azure-Kubernetes.pptx.png","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1668","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/comments?post=1668"}],"version-history":[{"count":3,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1668\/revisions"}],"predecessor-version":[{"id":1681,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/posts\/1668\/revisions\/1681"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media\/1680"}],"wp:attachment":[{"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/media?parent=1668"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/categories?post=1668"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nillsf.com\/index.php\/wp-json\/wp\/v2\/tags?post=1668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}