If you receive an Invalid OpenStack Identity
Credentials
message when you talk to an
OpenStack service, it might be caused by the changeover
from UUID tokens to PKI tokens in the Grizzly release.
Learn how to troubleshoot this error.
The PKI-based token validation scheme relies on
certificates from the Identity Service that are fetched
through HTTP and stored in a local directory. The location
for this directory is specified by the
signing_dir
configuration option.
In your services configuration file, look for a section
like this:
[keystone_authtoken] signing_dir = /var/cache/glance/api auth_uri = http://127.0.0.1:5000/ auth_host = 127.0.0.1 auth_port = 35357 auth_protocol = http admin_tenant_name = service admin_user = glance
If your service lacks this stanza, the keystoneclient/middleware/auth_token.py file specifies the defaults. If no value is specified for this directory, it defaults to a secure temporary directory. Initialization code for the service checks that the directory exists and is writable. If it does not exist, the code tries to create it. If this fails, the service fails to start. However, it often succeeds but problems occur later.
The first thing to check is that the
signing_dir
does, in fact, exist.
If it does, check for the presence of the certificate
files inside there:
$ ls -la /var/cache/glance/api/
total 24 drwx------. 2 ayoung root 4096 Jul 22 10:58 . drwxr-xr-x. 4 root root 4096 Nov 7 2012 .. -rw-r-----. 1 ayoung ayoung 1424 Jul 22 10:58 cacert.pem -rw-r-----. 1 ayoung ayoung 15 Jul 22 10:58 revoked.pem -rw-r-----. 1 ayoung ayoung 4518 Jul 22 10:58 signing_cert.pem
This directory contains two certificates and the token revocation list. If these files are not present, your service cannot fetch them from the Identity Service. To troubleshoot, try to talk to the Identity Service to make sure it is serving out the files, as follows:
$ curl http://localhost:35357/v2.0/certificates/signing
This command fetches the signing certificate:
Certificate: Data: Version: 3 (0x2) Serial Number: 1 (0x1) Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, ST=Unset, L=Unset, O=Unset, CN=www.example.com Validity Not Before: Jul 22 14:57:31 2013 GMT Not After : Jul 20 14:57:31 2023 GMT Subject: C=US, ST=Unset, O=Unset, CN=www.example.com
Note the expiration dates of the certificate:
Not Before: Jul 22 14:57:31 2013 GMT Not After : Jul 20 14:57:31 2023 GMT
The token revocation list is updated once a minute, but the certificates are not. One possible problem is that the certificates are the wrong files or garbage. You can remove these files and run another command against your server: They are fetched on demand.
The Identity Service log should show the access of the
certificate files. You might have to turn up your logging
levels. Set debug = True
and
verbose = True
in your Identity
Service configuration file and restart the Identity
Service server.
(keystone.common.wsgi): 2013-07-24 12:18:11,461 DEBUG wsgi __call__ arg_dict: {} (access): 2013-07-24 12:18:11,462 INFO core __call__ 127.0.0.1 - - [24/Jul/2013:16:18:11 +0000] "GET http://localhost:35357/v2.0/certificates/signing HTTP/1.0" 200 4518
If the files do not appear in your directory after this, it is likely one of the following issues:
Your service is configured incorrectly and cannot talk to the Identity Service. Check the
auth_port
andauth_host
values and make sure that you can talk to that service through cURL, as shown previously.Your signing directory is not writable. Use the chmod command to change its permissions so that the service (POSIX) user can write to it. Verify the change through su and touch commands.
The SELinux policy is denying access to the directory.
SELinux troubles often occur when you use Fedora/RHEL-based
packages and you choose configuration options that do not
match the standard policy. Run the setenforce
permissive command. If that makes a
difference, you should relabel the directory. If you are
using a sub-directory of the
/var/cache/
directory, run the
following command:
# restorecon /var/cache/
If you are not using a /var/cache
sub-directory, you should. Modify the
signing_dir
configuration option
for your service and restart.
Set back to setenforce enforcing
to
confirm that your changes solve the problem.
If your certificates are fetched on demand, the PKI validation is working properly. Most likely, the token from the Identity Service is not valid for the operation you are attempting to perform, and your user needs a different role for the operation.