[P 3.1] Build Sheet: Redundant CRL and AIA Web Servers

By the end of this you have two web servers behind one name serving your root CA certificate and revocation list over HTTP, with the issuing CA able to publish to both directly and no replication technology between them. This is the article that makes the URLs the root recorded real.


By the end of this article the name your root certificate authority wrote into its configuration resolves, two web servers serve the files behind it, and a machine that has never been near your network can fetch your revocation list and validate a certificate against it.

This is ordinary web hosting with an unusual consequence of getting it wrong, so the steps are short and the reasoning matters more than the mechanics. Everything is performed twice, once on each web server, except the DNS record and the final test.

Prerequisites. Two Windows Server 2022 or 2025 machines named PKI-WEB01 and PKI-WEB02, domain joined, each with a D: volume. The machine that will become your issuing authority already exists, is domain joined and is named SUBCA, because its computer account is granted write permission here even though the certificate services role is not installed on it until the next article. Permission to create a record in a DNS zone your team is authoritative for. The root certificate and revocation list exported at the end of the previous article, available on removable media or a file share. If you are using a load balancer, an unused virtual address on it.


Step 1. Confirm you own the zone, then create the name

Before creating anything, confirm the name you are about to bake into every certificate lives in a DNS zone your team controls and can change quickly. The name used throughout this series is pki.catsnackjack.com, in a zone whose authoritative servers are the organisation’s own domain controllers.

If your directory runs as a child of a publicly registered domain, do not put the PKI name in the parent. You need a record resolving to internal addresses, changeable at short notice, for the next decade. Use the child zone you are authoritative for and accept the longer name. Whatever you choose is used identically in both policy files, in every distribution point command on both authorities, and in the virtual directory below.

Then choose how the name resolves to two servers.

SettingValueWhy
Resolution methodLoad balancer virtual address (preferred)A single A record to a virtual address, with the balancer probing the actual file path on both nodes. A failed server leaves rotation in seconds and clients never see it.
Resolution methodDNS round robin (acceptable, document it)Two A records for the same name. Simpler, and not high availability: DNS health-checks nothing, so a failed server keeps receiving roughly half of all requests until somebody removes the record by hand.
Record TTL, round robin only300 secondsCaps how long clients keep using a stale answer after you pull a failed address. A default TTL measured in hours turns a five-minute repair into an afternoon.
Probe target, load balancer onlyThe /pki/ path, not the portA port check confirms IIS is listening. It does not confirm the virtual directory resolves or the files are present, which are the failures that actually occur.

With a load balancer, create one A record for pki pointing at the virtual address. Without one, create two A records with the same name, one per web server.

# Load balancer: one record to the virtual address
Add-DnsServerResourceRecordA -ZoneName "catsnackjack.com" -Name "pki" `
  -IPv4Address "10.10.10.50" -TimeToLive 01:00:00

# Round robin: one record per web server, short TTL
Add-DnsServerResourceRecordA -ZoneName "catsnackjack.com" -Name "pki" `
  -IPv4Address "10.10.10.51" -TimeToLive 00:05:00
Add-DnsServerResourceRecordA -ZoneName "catsnackjack.com" -Name "pki" `
  -IPv4Address "10.10.10.52" -TimeToLive 00:05:00

Resolve-DnsName pki.catsnackjack.com

Expected result: the name resolves to the virtual address, or to both server addresses. Failure mode: if resolution succeeds from a domain controller but fails from a client, you are almost certainly looking at a zone the internal servers are not authoritative for, which is the situation this step exists to prevent.


Step 2. Create the folder and share on both servers

Perform on PKI-WEB01 and PKI-WEB02. The folder is both a network share, so the issuing authority can publish into it, and the source of a web virtual directory, so clients can read it. Those two access paths need different permissions and both must be right.

SettingValueWhy
Share permission, Domain AdminsFull ControlAdministrative management of published files, including the annual manual root CRL copy.
Share permission, SUBCA$Full ControlThe issuing authority publishes as its computer account, not as a user. The dollar suffix denotes a computer account; in an account picker you must change the object type to Computers to find it.
NTFS, SUBCA$Modify, applied to this folder, subfolders and filesThe minimum that allows creating, overwriting and deleting the revocation list on each publication cycle. Read alone produces a publication that fails silently on the authority.
NTFS, IIS AppPool\DefaultAppPoolGranted in step 3, not hereThe worker process identity that serves the files does not exist until the web server role creates the application pool. Granting it at this point cannot succeed, so it belongs in the step that installs IIS.
Share permission, EveryoneRemovedThe default entry. Nothing anonymous needs share access; anonymous read happens over HTTP, not SMB.

The folder permission is set with icacls rather than with the .NET access control object model that most PowerShell examples reach for, and that is a decision about where this server is going to run rather than a matter of taste. A machine carrying application control policy, whether App Control for Business or AppLocker, puts PowerShell into constrained language mode automatically, and in that mode New-Object System.Security.AccessControl.FileSystemAccessRule cannot run at all, because the type is not on the short list the mode permits. What makes that worth an article paragraph is the shape of the failure rather than the failure itself. The rule objects are never constructed, so nothing is added to the access list, but Set-Acl still runs against the unmodified list and reports success. The permission looks applied. The consequence surfaces four steps later as a 403 on a file that is plainly present on disk, and it reads as a web server problem. A certificate services estate is exactly the kind of environment where application control belongs, so the .NET pattern is the wrong default here and native tooling is the right one. icacls is a binary rather than a language construct, the mode does not apply to it, and when it fails it says so.

New-Item -ItemType Directory -Path "D:\PKI" -Force

New-SmbShare -Name "PKI" -Path "D:\PKI" `
  -Description "PKI CRL and Certificate Distribution" `
  -FullAccess "catsnackjack\Domain Admins","catsnackjack\SUBCA$"

icacls "D:\PKI" /grant 'catsnackjack\SUBCA$:(OI)(CI)(M)'

Get-SmbShareAccess -Name "PKI"
icacls "D:\PKI"

Read the permission string as three parts. (OI) is object inherit and (CI) is container inherit, which together are the exact equivalent of the container and object inherit flags with no propagation flag, and are what the security dialog calls this folder, subfolders and files. (M) is modify, and (RX) in the next step is read and execute. If you are ever granting to a principal by SID rather than by name, the SID takes a leading asterisk.

Two parsing details, because between them they account for most of the time people lose on this line. The inheritance flags are parenthesised and parentheses are PowerShell syntax, so the permission argument has to be quoted; single quotes are the safer choice because the computer account name ends in a dollar sign, which PowerShell would otherwise read as the start of a variable. If you would rather not quote at all, the stop-parsing token passes the rest of the line to the binary untouched: icacls "D:\PKI" --% /grant catsnackjack\SUBCA$:(OI)(CI)(M), effective until the next newline. And send one complete statement at a time. The console host holds a pasted multi-line block at a >> continuation prompt until it sees an empty line, which is the paste waiting rather than the command failing, and pressing Enter on an empty line releases it.

Expected result: the share lists Domain Admins and SUBCA$ with Full Control, icacls reports that it successfully processed one file, and the folder listing shows SUBCA$ with (OI)(CI)(M). Failure mode: an error naming a mapped account or a failed name resolution means the computer account was typed without its dollar suffix or with the wrong domain, and unlike the .NET form, which accepted an unresolvable name and failed later, icacls refuses it at the point you make the mistake. That is the behaviour you want in a build sheet.


Step 3. Install IIS, grant the worker process, publish the virtual directory

Perform on both servers. This is a static file server and needs almost nothing. Install the web server role with the default feature set, grant the application pool identity read access to the folder, then add a virtual directory named pki under the default web site, pointing at D:\PKI.

SettingValueWhy
Web server role featuresDefault set, management tools includedStatic file serving is all this host does. Every additional module is surface area on a server whose availability the entire certificate estate depends on.
NTFS, IIS AppPool\DefaultAppPoolRead and Execute, applied to this folder, subfolders and filesThe worker process identity that serves the files, once step 4 points anonymous requests at it. Without this entry every request returns 403 while the files sit plainly on disk.
Virtual directorypki under Default Web Site, physical path D:\PKIMakes the path segment in the URLs the root already recorded resolve. The name is not adjustable now; it was fixed when the root was built.
Directory browsingEnabled on the virtual directory onlyMakes the folder self-describing during a support call. Enabled at server level it would apply to anything else this host ever serves.
Install-WindowsFeature Web-Server -IncludeManagementTools

# the application pool identity does not exist until the role above creates it
icacls "D:\PKI" /grant 'IIS AppPool\DefaultAppPool:(OI)(CI)(RX)'

Import-Module WebAdministration
New-WebVirtualDirectory -Site "Default Web Site" -Name "pki" -PhysicalPath "D:\PKI"

Set-WebConfigurationProperty -Filter /system.webServer/directoryBrowse `
  -Name enabled -Value $true -PSPath "IIS:\Sites\Default Web Site\pki"

Get-WebVirtualDirectory -Site "Default Web Site" -Name "pki"
icacls "D:\PKI"

The ordering inside that block is load bearing, and it is the one thing in this article I would expect an experienced administrator to get wrong by habit, because folder permissions are conventionally set at the same time as the folder. IIS AppPool\DefaultAppPool is a virtual account, created by the Windows Process Activation Service at the moment the application pool is created, which is to say when the web server role is installed. Before that the name does not resolve on this machine or on any other, and the grant cannot succeed. It is also local rather than directory-based, so in the account picker you have to change the location from the domain to the local server; the usual reaction to not finding it there is to conclude it does not exist and grant something broader instead, which is how a distribution point ends up with Users granted read on a folder that did not need it.

Directory browsing on the virtual directory is a deliberate choice rather than laziness. It makes the folder self-describing during a support call, so anyone can open the URL in a browser and see which files are actually being served and when they were last written. The folder contains a public certificate and a signed list of revoked certificates, both of which are meant to be world-readable, so there is nothing here to protect.

Expected result: the role installs, icacls reports one file processed, the virtual directory is returned by the read back, and the folder listing now shows both SUBCA$ with (OI)(CI)(M) and the application pool identity with (OI)(CI)(RX). Failure mode: if the grant reports that no mapping between account names and security IDs was done, you ran it before the role finished installing, which is the ordering this step exists to enforce. Run the one line again once the install has completed.


Step 4. Point anonymous access at the application pool

Perform on both servers. Anonymous access is what makes a distribution point work, and by default IIS serves anonymous requests as IUSR rather than as the application pool. That matters here because the permission you granted in step 3 was to the application pool, so left at the default the two do not meet and every request returns 403 with the files sitting plainly on disk. Point anonymous authentication at the application pool identity instead of adding a second principal to the access control list. One identity holds the permission and makes the request, which is one thing to audit rather than two.

Import-Module WebAdministration

# an empty user name means "use the application pool identity"
Set-WebConfigurationProperty `
  -Filter "/system.webServer/security/authentication/anonymousAuthentication" `
  -Name userName -Value "" `
  -PSPath "IIS:\" -Location "Default Web Site/pki"

Get-WebConfigurationProperty `
  -Filter "/system.webServer/security/authentication/anonymousAuthentication" `
  -Name userName `
  -PSPath "IIS:\" -Location "Default Web Site/pki"

Expected result: the read back returns an empty value, which is how IIS expresses the application pool identity. Failure mode to watch for: a 401.3 or a 403 on a file you can see on disk and can open locally is this exact mismatch and not a file permission problem, so adding permissions for the wrong principal will not fix it.


Step 5. Allow double escaping

Perform on both servers, at the server level rather than on the virtual directory. This is the single setting most likely to be missed, and its symptom is specific enough to be worth memorising.

Set-WebConfigurationProperty -Filter /system.webServer/security/requestFiltering `
  -Name allowDoubleEscaping -Value $true -PSPath "IIS:\"

# equivalent
# appcmd set config /section:requestfiltering /allowdoubleescaping:true

Get-WebConfigurationProperty -Filter /system.webServer/security/requestFiltering `
  -Name allowDoubleEscaping -PSPath "IIS:\"

Delta revocation list filenames contain a plus character. That plus is not decoration: it is the %9 token you configured on the root, the marker the authority substitutes to distinguish a delta list from a base list, and it will appear on the issuing authority too. IIS request filtering rejects it by default. The failure is an HTTP 404.11, described as a request denied because the URL contains a double escape sequence, and it appears only for delta lists while base lists download perfectly. That is why it survives testing: everything works, until months later you enable delta lists on the issuing authority and a subset of validation starts failing for reasons that appear to have nothing to do with a web server nobody has touched. Delta lists are disabled on the root and are optional on the issuing authority, so you may never rely on this. Set it anyway, now, while you are here.

In the console this is the same setting: select the server node itself, not the site, open Request Filtering, choose Edit Feature Settings, and enable Allow double escaping.


Step 6. Place the root files on both servers

Copy the root certificate and the root’s revocation list, both exported at the end of the previous article, into D:\PKI on each web server. Both files are required and for different reasons: the certificate is what the AIA URL serves, so a validating client can obtain the root’s own certificate and complete the chain, and the revocation list is what the CDP URL serves, so that client can then confirm the root has not revoked the certificate below it. The root wrote both URLs into itself before it signed anything. Copying only the certificate produces a chain that builds and then fails revocation checking, which is a harder failure to read than a chain that does not build at all. These arrive by hand and will do so once a year for the rest of the hierarchy’s life, because the machine that produces them has no network.

Copy-Item "E:\RootExport\*.crt" "\\PKI-WEB01\PKI\"
Copy-Item "E:\RootExport\*.crl" "\\PKI-WEB01\PKI\"
Copy-Item "E:\RootExport\*.crt" "\\PKI-WEB02\PKI\"
Copy-Item "E:\RootExport\*.crl" "\\PKI-WEB02\PKI\"

Get-ChildItem "\\PKI-WEB01\PKI\", "\\PKI-WEB02\PKI\" | Select-Object PSComputerName, Name, Length, LastWriteTime

Expected result: identical filenames, sizes and timestamps on both servers, and both a .crt and a .crl present on each. Compare them rather than assuming, because the failure this whole design exists to avoid is one server quietly serving an older file than the other.

Note what is not happening here. There is no replicated folder, no scheduled synchronisation, and no third component between the two servers. The root’s files are placed by hand once a year, and the issuing authority’s revocation list, from the next article onward, is written to both shares by the authority itself on every publication cycle. Nothing in this architecture can silently stop converging, because nothing is converging.


Step 7. Prove each server individually, then the name

Test the servers separately before testing the name, so that a failure tells you which layer it is in.

Invoke-WebRequest -Uri "http://PKI-WEB01/pki/" -UseBasicParsing | Select-Object StatusCode
Invoke-WebRequest -Uri "http://PKI-WEB02/pki/" -UseBasicParsing | Select-Object StatusCode

Invoke-WebRequest -Uri "http://pki.catsnackjack.com/pki/" -UseBasicParsing | Select-Object StatusCode

Expected result: 200 from each server individually, showing the directory listing, then 200 through the name. A 403 at this point has two possible causes and they are different problems: either the application pool identity does not have read access to the folder, which is step 3, or anonymous requests are still arriving as IUSR rather than as the application pool, which is step 4. Confirm the first with icacls "D:\PKI" and read the output rather than trusting that the grant ran, because a permission that was never applied and a permission that was applied to the wrong principal look identical from the browser. A 404 means the virtual directory name or physical path is wrong, which is also step 3.

Then fetch the actual files by their full names, because a directory listing proves the path resolves and proves nothing about the files themselves.

$base = "http://pki.catsnackjack.com/pki"
$crt  = "$base/ROOTCA_CatSnackJack%20Root%20CA.crt"
$crl  = "$base/CatSnackJack%20Root%20CA.crl"

Invoke-WebRequest -Uri $crt -UseBasicParsing | Select-Object StatusCode, @{n='Bytes';e={$_.RawContentLength}}
Invoke-WebRequest -Uri $crl -UseBasicParsing | Select-Object StatusCode, @{n='Bytes';e={$_.RawContentLength}}

Spaces in the authority’s common name become %20 in the URL, which is normal and works. If you would rather not deal with encoded names for the next decade, that is an argument for a common name without spaces, and it is an argument you can only act on before the root is built.


The end-to-end test

Everything above proves the plumbing from inside. The test that matters is whether a machine with no relationship to your organisation can validate a certificate from this hierarchy, because that is the population the design exists for.

From a machine that is not domain joined and is on a different network from the web servers, with the root certificate copied to it locally, run a verification that is explicitly told to fetch what it needs.

certutil -URL "http://pki.catsnackjack.com/pki/CatSnackJack%20Root%20CA.crl"

certutil -verify -urlfetch RootCA.crt

The first command opens a retrieval tool; choose to retrieve the CRL and confirm it reports a verified status rather than a failure. The second walks the certificate and reports on every location it was told to fetch. What you are looking for is that the revocation list is retrieved successfully, is inside its validity window, and that no location times out.

Then do the part everybody skips. Stop the web service on PKI-WEB01 and run the same test again, then start it, stop the service on PKI-WEB02, and run it a third time. With a load balancer both should pass without a perceptible delay. With DNS round robin you should expect roughly half of attempts to stall and then fail, which is exactly the behaviour described in the previous article, and seeing it once is worth more than reading about it. If you are running round robin, that observed behaviour is what you write into the handover documentation.


A worked example: the directory that is a child of a public domain

Take an organisation whose Active Directory domain is csj.catsnackjack.com, where the internal DNS servers are authoritative for csj.catsnackjack.com and the parent catsnackjack.com is registered publicly and managed by a marketing agency with a two-week change process.

The tempting name is pki.catsnackjack.com, because it is shorter and looks more permanent. It is the wrong answer for two reasons that both bite later. The record would live in a zone the PKI team cannot edit, so removing a failed web server’s address during an incident becomes somebody else’s ticket. And it would need to resolve to internal private addresses, which an externally hosted public zone will generally not do, so internal clients cannot reach it at all.

The correct choice is pki.csj.catsnackjack.com, inside the zone the internal servers own. Every appearance of the name changes with it: both policy files, both distribution point configurations, the virtual directory tests, and the load balancer probe. The URL is longer and nobody will ever comment on it. The alternative is a distribution point you cannot change, in a zone you do not control, embedded in every certificate for the next ten years.


Rollback

This is the one part of the hierarchy that is genuinely reversible, because nothing here is embedded in a certificate. The name is, but the servers behind it are not, and that is the whole point of putting a name in front of them.

Remove-WebVirtualDirectory -Site "Default Web Site" -Name "pki"
Remove-SmbShare -Name "PKI" -Force
Uninstall-WindowsFeature Web-Server

Rebuild, re-point the DNS record or the load balancer, and clients notice nothing. Do not remove or repoint the name itself, ever, for as long as any certificate carrying it remains unexpired.


Completion checklist

The name resolves from a domain-joined client and from a machine outside the network. Both web servers return a directory listing for the /pki/ path individually. The name returns the same listing. The root certificate and the root’s revocation list are both present on both servers, download by full URL from each, and are byte-identical between them. Double escaping is enabled at server level on both. The share grants Full Control to Domain Admins and to the issuing authority’s computer account, and icacls on the folder shows Modify for that computer account and Read and Execute for the application pool identity, read from the actual output rather than assumed. A verification from an unmanaged machine outside the network retrieves the revocation list and reports it valid. If you are using round robin, the observed failure behaviour with one server stopped is written down and handed over.

The distribution layer now exists, and the URLs the root recorded before it signed anything are real. The next article builds the authority that will actually issue certificates, and the first thing it does after installation is publish a revocation list to both of these servers by itself.


On-Prem PKI
‹ Previous: [P 3] Revocation Is a Distribution Problem
Next: [P 4] The Issuing CA: What Enterprise Integration Buys and What It Costs