Monitoring Conditional Access Policies in Entra

Hi folks! It's about time for a new blog post, and this one will be important from a security perspective. We are hopefully all monitoring and auditing several accounts, but we may be forgetting about other important things. What if our Conditional Access policies were modified? What if someone created a new policy, or in the worst case scenario, what if someone deleted all policies?

Well, this is something I have had in my mind for a long time now. Late last year, Microsoft finally released the deleted policies feature in Conditional Access. Now, if a policy is deleted, it's possible to restore it within 30 days before it gets hard deleted.

But how do we actually get alerts if someone soft deletes one, several, or all policies? How do we know if someone makes changes or creates new policies? Well, that's what I will show today.

We can do it in an easy way with detection/alert rules and Azure Monitor, by ingesting audit logs into one of our Log Analytics Workspaces in Azure.

What to start with?

First of all, we need to check if audit logs are already being ingested into the Log Analytics Workspace or not. If you don't have a Log Analytics Workspace, you would first need to create one in Azure, of course, which is a pretty straightforward process.

1. Go to entra.microsoft.com 2. Go to Users -> Sign-in logs -> Export Data Settings or to Monitoring and health -> Diagnostic settings 3. In my case I'm already ingesting both Audit and Sign In logs to my Log Analytic Workspace. We only need AuditLogs for this scenario

4. Now Go to portal.azure.com, find Log Analytic workspace and press Alerts -> Create -> Alert rule

5. On Condition choose "Custom Log Search"

6. Than press on View result and edit query in Logs 7. When new window opens, choose KQL mode in upper right corner 8. Check if anything has arrived yet from Audit log in Entra (adjust time range if needed) Use this query AuditLogs

| count

and hit Run

9. Look at number on results, if 0, than Logs are not ingested to workspace. Double check your Diagnostic settings in Entra (go to start) or give it more time.

10. Now, use one of the queries from github and add it to "Search query" section, feel free to test it, but remember, it will only look 5 min back. Of course, feel free to change it and tweak query your way. I'll use delete detection query in this tutorial Create 3 different rules with provided detection queries. I'll use delete detection query in this tutorial I called my policies and detection queries on github for CA-Policy-Deleted Detects when a Conditional Access policy is soft deleted. Adds a `RiskFlag` that automatically highlights if the deletion was performed by an app or service principal (no UPN) rather than a human. CA-Policy-Modified Detects when an existing Conditional Access policy is changed. Uses `mv-expand` to unpack every individual field that changed, giving you a proper before/after diff with `OldValue` and `NewValue` per row. Also auto-flags 🔴 High-Impact Field Changed when critical fields like `state`, `conditions`, or `grantControls` are touched. CA-Policy-Created Detects when a Conditional Access policy is soft deleted. Adds a `RiskFlag` that automatically highlights if the deletion was performed by an app or service principal (no UPN) rather than a human. You can find all 3 queries on my github repo https://github.com/vladjoh/CA-Monitoring

9. Everything else can stay as is for testing, change it how you prefer to have it when time is there.
   

   

   12. Press Actions button when ready

   13. Select your action group or create new. Microsoft also now allows to add Email subject to our email alerts. I already have 1 group called AG-CA-Policy-Alerts

   

   14. Add email's and phone numbers to specific people who should recieve all these alerts. Remember, it's only allowed to add one email and phone number per notification.
   

   

   15. Select severity, I reccomend 0-Critical for deletion alerts and add a name and description to you alert rule. Here is my example
   

   

   16. Now when you have done everything you can review and save. 17. Now you need to do all these steps again for 2 other policies. 18. Test the results by creating, deleting or modifying any CA policy on your tenant. PS: Always do it on test tenant first or with dummy policy without any impacts on users! 19. Here is an example on fired email alert to my email
   

   

   And here is example on SMS alert
   

   

   If I scroll down on email I can press on "View filtered search results" . This one will redirect you straight to Log Analytic Workspace Logs page (as long as this specific account have access of course ;)

   

   As you can see, I get both delete operations up and I can easily see which policies are deleted, who deleted them etc.
   

   

   
   

   So, to summarize, monitoring our critical services is very important. One little gap or misconfiguration in our conditional access policies can cause big security issues. And we can't trust every admin out there who is eligible to make changes in Conditional Access. So, Zero Trust guys -say no more.
   

   NB: This method and the queries may not be the best ones, so please don't judge it too much. I know it can also be done with Logic Apps as well for better reporting and automated response, but that's the next thing I want to test. I'm just showing what's possible with Azure Monitor only. And like I said, feel free to tweak existing detection queries! Thanks!