Make sure your Certbot installation is up to date

As of late 2020 Certbot is no longer maintained as an apt package and you should make sure it is running through Snap on Ubuntu. This procedure is totally safe and easy to do.

Please read our full article on this topic in order to get up to speed.

Checking if your Let's Encrypt Certificate is working

Typically Certbot runs fine in Webdock the first time you run it and problems crop up over time or when you do changes to your configuration. Often the first indication you get of a problem is when you get an email from Let's Encrypt with a title similar to "Let's Encrypt certificate expiration notice for domain xxx"

When you generate certificates with Webdock we automatically add a Cron job which keeps your certificate up to date. If you get this email then this means your certificate may be about to expire and you need to check if renewals are working.

In Webdock you can simply run the "Test Certbot renewal" script on your server on the Server Scripts page. 

What this script essentially does is runs the following command on your server and returns the output so you can inspect what is happening

certbot renew --dry-run

In the following sections we detail various error messages which may show up here and what to do about them.

Invalid response / The client lacks sufficient authorization

If you get a message similar to the following

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for mydomain.com
http-01 challenge for www.mydomain.com
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. mydomain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mydomain.com/.well-known/acme-challenge/pYpAC6kT25C0itcTNKd8hwb_0VaoPxJVIkVg5_xn-N4 [77.111.240.95]: 403
IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   unauthorized
   Detail: Invalid response from
   http://mydomain.com/.well-known/acme-challenge/pYpAC6kT25C0itcTNKd8hwb_0VaoPxJVIkVg5_xn-N4
   [77.111.240.95]: 403

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Try the suggested fix: Check if your DNS records are OK

First thing to check are your DNS records and whether your A and AAAA records are OK and pointing to your server. However, if your certificate has worked before and you reach your website just fine, then it is unlikely this is the culprit.

However, you should  keep an eye on whether there are any web forwards configured (some DNS providers allow this) e.g. if you forward www to non-www or vice-versa, this may trip up Certbot. In which case remove the domain you are forwarding using DNS from your certificate. This should resolve the issue.

To remove a domain in Webdock you simply run Server Identity with the domains you want your webserver to actually respond to and then run the SSL Certificates tool once more.

On Apache: Make sure you exclude Certbot from any .htaccess rewrites

If you are on Apache, check if you have a .htaccess file with rewrites in your web root. If you have such a file you should place a rule to exclude Let's Encrypt Certbot from any rewrites by placing the exception before any RewriteCond statements. For example: 

RewriteEngine on
RewriteRule ^\.well-known\/acme-challenge\/ - [L]
RewriteCond %{SERVER_NAME} =mydomain.demo.com [OR]
RewriteCond %{SERVER_NAME} =www.mydomain.com [OR]
RewriteCond %{SERVER_NAME} =livesite.mydomain.com [OR]
RewriteCond %{SERVER_NAME} =mydomain.com [OR]
RewriteCond %{SERVER_NAME} =mydomain.dk [OR]
RewriteCond %{SERVER_NAME} =www.mydomain.dk
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

Once you've placed the acme-challenge RewriteRule in there, try running Certbot again.

On Apache: Try rolling back completely and nuking any Certbot config

If your DNS records and rewrites are ok and Certbot renew still fails, you should try and issue the certbot rollback command:

certbot rollback

If this gives you errors, try removing the Let's Encrypt SSL configuration file located at (in default Webdock stacks):

/etc/apache2/sites-enabled/webdock-le-ssl.conf

Next run Certbot rollback - and if it succeeds keep running it untill it says it hasn't touched your configuration. You should now be back to a stock configuration and you should be able to run the SSL Certificates tool once more to generate a new certificate.

If you are not using Webdock and following this guide, you should make sure you back up your configuration before doing this and make sure your config is as you want it and working for vanilla http before running Certbot.

On Nginx: Make sure you exclude Certbot from any deny rules

Similar to Apache, if you have custom configuration which excludes access to hidden directories, such as the following:

# Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
# Keep logging the requests to parse later (or to pass to firewall utilities such as fail2ban)
location ~ /\. {
    deny all;
}

You need to add the following rule beofre this in order to allow Certbot to validate using the web root method:

# Necessary for Let's Encrypt Domain Name ownership validation
location ~ /.well-known {
  allow all;
}

Error while running apache2ctl configtest

This error is related to the last fix detailed above and can be solved in the same manner, if you encounter a message like the following:

Error while running apache2ctl configtest.
Action 'configtest' failed.
The Apache error log may have more information.

AH00526: Syntax error on line 30 of /etc/apache2/sites-enabled/000-default-le-ssl.conf:
SSLCertificateFile: file '/etc/letsencrypt/live/mydomain.com/fullchain.pem' does not exist or is empty

Simply delete the old config and try again

During the Webdock Let's Encrypt SSL installation, a Certbot rollback command hasn’t completely cleared an old default config file. You can resolve this by removing the file /etc/apache2/sites-enabled/000-default-le-ssl.conf and run Certbot once more.

No such file or directory error from Nginx

Sometimes during a Certbot rollback operation or when Certbot tries to renew/install a certificate, Nginx cannot start the webserver as there is till old Certbot configuration hanging around in your Nginx vhost config file. You may see an error like the following:

Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: [emerg] BIO_new_file("/etc/letsencrypt/live/mydomain.com-0001/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/mydomain.com-0001/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

To solve this issue, remove any references to SSL certificates from your Nginx configuration file. The lines typically look like this:

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

Once you have removed these lines run Certbot rollback untill it says nothing was modified. If you are using Webdock you would then run the Server Identity tool and finally the SSL Certificates tool and you will now have a shiny new (and working) certificate installed on your server.

If you are not using Webdock, you would likely issue another certbot rollback command and see if it succeeds, next review your config and make sure it is as you want it and that it works for vanilla http, finally you would run Certbot again to generate your certifcate.

Timeout Errors

If you get an error similar to the following:

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   connection
   Detail: Fetching
   https://mydomain.com/.well-known/acme-challenge/m5JIZwyOioenm7XraPru2FV-8VPtCXF1YpoHmctfVDc:
   Timeout during connect (likely firewall problem)

You need to check whether your DNS records are OK and whether your webserver is listening on your server on port 80 and 443. Make sure you check whether you can ping and ping6 your domain. Typically we see this error happening is if AAAA records for a domain have not been updated, or there are old records pointing to a wrong IP hanging around - but a timeout might also indicate a networking issue (for example a firewall) as stated in the message, or that your server is actively refusing connections for some reason.

Another cause for this issue is that Nginx and Certbot are not aligned as to where your web root is located and Certbot is placing its verification files for web root authentication in the wrong place. A quick way to fix this in Webdock is to simply set the Web Root once more (the small pencil icon next to the web root in the Webdock dashboard) and then run renewal again.

The error may also look something like this:

Waiting for verification...
Cleaning up challenges
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)

In our experience, this specific message is due to Let's Encrypt not being able to connect via. IPv6 - so check that IPv6 works if you have AAAA records for your domain, otherwise all the advice above is also applicable, i.e. check your DNS and connection issues in general. On some older Ubuntu systems (Xenial especially) a general package upgrade with apt update and apt upgrade has been seen by us as an easy resolution to some ipv6 connection issues.

Connection Refused Error

If you get a Connection Refused error when doing renewals, this may be due to old-style Certbot configuration not having written the appropriate configuration to your Nginx config in order to allow Nginx to listen on your IPv6 address. You would typically see an error like the following:

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mydomain.com
   Type:   connection
   Detail: Fetching
   https://mydomain.com/.well-known/acme-challenge/ewpBCX7N0nzDyBZZILYP-y9sKHI4seFGac4Se7TpwfA:
   Connection refused

What this error really is saying is: Certbot could not connect to your domain on port 443. Typically what we see is that if you now check what is listening on which ports by running netstat you will get something like the following:

root@myserver:~# netstat -tapn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      441/mongod
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      772/mysqld
tcp        0      0 127.0.0.1:11211         0.0.0.0:*               LISTEN      440/memcached
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      716522/nginx -g dae
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      614876/pure-ftpd (S
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      449/sshd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      716522/nginx -g dae
tcp6       0      0 :::80                   :::*                    LISTEN      716522/nginx -g dae
tcp6       0      0 :::21                   :::*                    LISTEN      614876/pure-ftpd (S
tcp6       0      0 :::22                   :::*                    LISTEN      449/sshd

Notice how nginx is listening on tcp (ipv4) port 443 but not tcp6 (ipv6) ? This is the problem as the Let's Encrypt certificate servers prefer to connect through ipv6 if there are AAAA records set up for your domain. 

In order to solve this, check that you have the following line in your Nginx config (on Webdock servers the vhostconfig file is typically found at /etc/nginx/sites-enabled/webdock) , and if not, add it:

listen [::]:443 ssl ipv6only=on; # managed by Certbot

Next, restart Nginx:

systemctl restart nginx

Check that nginx is listening on tcp6 port 443 and try your renewal again. It should work this time - but if not, then make sure nginx is listening on the appropriate interfaces and on both ports 80 and 443 and is reachable from the outside, because if it is not you will get this error message.

Account creation on ACMEv1 is disabled

If you get an error message similar to the following

Certbot failed in schonherr Certbot message: Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
The client lacks sufficient authorization :: Account creation on ACMEv1 is disabled. Please upgrade your ACME client to a version that supports ACMEv2 / RFC 8555. See https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430 for details.
Please see the logfiles in /var/log/letsencrypt for more details.

This simply means that the installed version of Certbot is too old and doesn't support newer authentication methods. Simply upgrade your system with:

apt-get update -y
apt-get upgrade -y

If it prompts you what to do about existing config files just choose the defaults which is to keep the existing configuration. Once the system is upgraded it is a good idea to reboot the system, or at the very least your webserver and php-fpm before trying again.

Please note: During the system upgrade your webserver will likely go down temporarily. Plan accordingly.

The request message was malformed

Similarily an indication your Certbot is too old may be seen from the following Certbot output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer apache
Renewing an existing certificate
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: urn:ietf:params:acme:error:malformed :: The request message was malformed :: Method not allowed. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

The solution here is, as with the error above, to upgrade your system. The same warning applies: upgrading may bring down your webserver temporarily.

Test Certbot Renewal script hangs for a long time, prompts for a new webroot in output

Sometimes Webdock users experience that the Test Certbot renewal scripts hangs for a long time and then outputs something similar to the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Non-interactive renewal: random delay of 362 seconds
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain.com
Cleaning up challenges
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: Missing command line flag or config entry for this setting:
Select the webroot for mydomain.com:
Choices: [Enter a new webroot, /var/www/html]

(You can set this with the --webroot-path flag). Skipping.
All renewal attempts failed. The following certs could not be renewed:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
** DRY RUN: simulating certbot renew close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  /etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

This is due to Certbot having "forgotten" what the appropriate web root is for your domain. We use web root authentication in order to avoid certain other issues with Cloudflare and Cloudflare caching. In any case, the solution here is to set the web root once more on your server, after which you should run the SSL Certificates tool once more. If you still get errors, try running the Server Identity tool first.