5.10.1. Web Interface errors
/var/log/tomcat5/catalina.out — Check this logfile first if you get errors in the RHN Satellite web UI when viewing, scheduling, or working with kickstarts.
/var/log/httpd/error_log — Check this logfile second for possible sources of web UI errors
5.10.2. Anaconda Startup errors
If you get errors during the initiation of Anaconda where it can't find the kickstart file:
+-------------+ Error downloading kickstart file +-------------+
| |
| Unable to download the kickstart file. Please modify the |
| kickstart parameter below or press Cancel to proceed as an |
| interactive installation. |
| |
| dhat.com/cblr/svc/op/ks/profile/rhel5-i386-u3:1:Example-Org_ |
| |
| +----+ +--------+ |
| | OK | | Cancel | |
| +----+ +--------+ |
| |
| |
+--------------------------------------------------------------+
You can check the following items:
Verify httpd is running on your RHN Satellite
Verify cobblerd is running
Verify you can fetch the above file using wget from a different host. For example:
wget http://satellite.example.com/cblr/svc/op/ks/profile/rhel5-i386-u3:1:Example-Org
run cobbler check from the CLI. You should see only this output:
# cobbler check
The following potential problems were detected:
#0: reposync is not installed, need for cobbler reposync, install/upgrade yum-utils?
#1: yumdownloader is not installed, needed for cobbler repo add with --rpm-list parameter, install/upgrade yum-utils?
#2: The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed
#3: fencing tools were not found, and are required to use the (optional) power management features. install cman to use them
If you see complaints about problems with httpd, cobblerd, or others, you must resolve those issues.
Note
In the case of a system reprovision, check the following URL:
http://sat_FQDN/cblr/svc/op/ks/system/$system_name:$org_id
In the case of a guest reprovision you can optionally check the following URL:
http://sat_FQDN/cblr/svc/op/ks/system/$system_name:$org_id:$guest_name
5.10.3. Anaconda content errors
+------------------+ Package Installation +------------------+
| |
+-------------------------+ Error +--------------------------+
| |
| The file chkconfig-1.3.30.1-2.i386.rpm cannot be opened. |
| This is due to a missing file, a corrupt package or |
| corrupt media. Please verify your installation source. |
| |
| If you exit, your system will be left in an inconsistent |
| state that will likely require reinstallation. |
| |
| |
| +--------+ +-------+ |
| | Reboot | | Retry | |
| +--------+ +-------+ |
| |
| |
+------------------------------------------------------------+
Clients will fetch content from RHN Satellite based on the --url parameter contained within the kickstart. For example:
url --url http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3
If you receive errors from Anaconda stating it can't find images or packages you should first check that the above URL will generate a 200 response:
wget http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3
--2009-08-19 15:06:55-- http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3
Resolving satellite.example.com... 10.10.77.131
Connecting to satellite.example.com|10.10.77.131|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [text/plain]
Saving to: `ks-rhel-i386-server-5-u3.1'
2009-08-19 15:06:55 (0.00 B/s) - `ks-rhel-i386-server-5-u3.1' saved [0/0]
If you don't get a 200 response check the error logs. After checking the base URL you can check the actual file Anaconda tried to download:
# grep chkconfig /var/log/httpd/access_log
10.10.77.131 - - [19/Aug/2009:15:12:36 -0400] "GET /rhn/common/DownloadFile.do?url=/ks/dist/ks-rhel-i386-server-
5-u3/Server /chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1" 206 24744 "-" "urlgrabber/3.1.0 yum/3.2.19"
10.10.76.143 - - [19/Aug/2009:15:12:36 -0400] "GET /ks/dist/ks-rhel-i386-server-5-u3/Server/chkconfig-
1.3.30.1-2.i386.rpm HTTP/1.1" 206 24744 "-" "urlgrabber/3.1.0 yum/3.2.19"
10.10.76.143 - - [19/Aug/2009:15:14:20 -0400] "GET /ks/dist/ks-rhel-i386-server-5-u3/Server/chkconfig-
1.3.30.1-2.i386.rpm HTTP/1.1" 200 162580 "-" "urlgrabber/3.1.0 yum/3.2.19"
10.10.77.131 - - [19/Aug/2009:15:14:20 -0400] "GET /rhn/common/DownloadFile.do?url=/ks/dist/ks-rhel-i386-server-
5-u3/Server/chkconfig-1.3.30.1-2.i386.rpm HTTP/1.1" 200 162580 "-" "urlgrabber/3.1.0 yum/3.2.19"
If those requests are not appearing in the access_log file, the system may be having trouble with the networking setup.
If those requests are appearing but are generating errors, see the previously mentioned log files for errors.
You can also try manually downloading the files:
wget http://satellite.example.com/ks/dist/ks-rhel-i386-server-5-u3/Server/chkconfig-1.3.30.1-2.i386.rpm
Then you can see if the package is available.
5.10.4. Cobbler log files
In addition to Satellite logs, cobbler also keeps some data in /var/log/cobbler/. When troubleshooting failed virtual installs, koan also saves the libvirt guest creation XML in /var/log/koan, which can occasionally be useful.
5.10.5. Tracebacks from Taskomatic
If you receive emails such as:
Subject: WEB TRACEBACK from satellite.example.com
Date: Wed, 19 Aug 2009 20:28:01 -0400
From: RHN Satellite <[email protected]>
To: [email protected]
java.lang.RuntimeException: XmlRpcException calling cobbler.
at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:72)
at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:76)
at com.redhat.rhn.taskomatic.task.SingleThreadedTestableTask.execute(SingleThreadedTestableTask.java:54)
at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Caused by: redstone.xmlrpc.XmlRpcException: The response could not be parsed.
at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:434)
at redstone.xmlrpc.XmlRpcClient.endCall(XmlRpcClient.java:376)
at redstone.xmlrpc.XmlRpcClient.invoke(XmlRpcClient.java:165)
at com.redhat.rhn.manager.kickstart.cobbler.CobblerXMLRPCHelper.invokeMethod(CobblerXMLRPCHelper.java:69)
... 4 more
Caused by: java.io.IOException: Server returned HTTP response code: 503 for URL: http://someserver.example.com:80/cobbler_api
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1236)
at redstone.xmlrpc.XmlRpcClient.handleResponse(XmlRpcClient.java:420)
... 7 more
This indicates there is a problem found between the 'taskomatic' service and 'cobblerd' communicating. Check:
Verify httpd is running on your RHN Satellite
Verify cobblerd is running
Verify no firewall rules that would prevent localhost connections from one process to the above path
5.10.6. Registration Issues
At the end of the kickstart there is a %post section that will register your kickstarted machine to the RHN Satellite:
# begin Red Hat management server registration
mkdir -p /usr/share/rhn/
wget http://satellite.example.com/pub/RHN-ORG-TRUSTED-SSL-CERT -O /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
perl -npe 's/RHNS-CA-CERT/RHN-ORG-TRUSTED-SSL-CERT/g' -i /etc/sysconfig/rhn/*
rhnreg_ks --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
--activationkey=1-c8d01e2f23c6bbaedd0f6507e9ac079d
# end Red Hat management server registration
Breaking this down into the 4 steps you have:
1) mkdir -p /usr/share/rhn/
Creating a directory to house the custom SSL cert used by the RHN Satellite
2) wget http://satellite.example.com/pub/RHN-ORG-TRUSTED-SSL-CERT -O /usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT
Fetch the SSL certificate to use during registration
3) perl -npe 's/RHNS-CA-CERT/RHN-ORG-TRUSTED-SSL-CERT/g' -i /etc/sysconfig/rhn/*
Search/replace the SSL certificate strings from the rhn-register configuration files.
4) rhnreg_ks --serverUrl=https://satellite.example.com/XMLRPC --sslCACert=/usr/share/rhn/RHN-ORG-TRUSTED-SSL-CERT --activationkey=1-c8d01e2f23c6bbaedd0f6507e9ac079d
Register to the RHN Satellite with the SSL certificate and an activation key. Every Kickstart Profile includes an Activation Key that assures that the system is assigned the correct base and child channels, gets the proper System Entitlements, and is associated with the previous System Profile if you are re-provisioning an existing system.
If the rhnreg_ks command fails you may see errors in the ks-post.log indicating:
ERROR: unable to read system id.
And calls to rhn_check return the error above you know the system failed to register to the RHN Satellite.
The best way to troubleshoot this is to view the kickstart file and copy-paste the four steps from above into the shell prompt and run them after the system comes back from kickstarting. Generally rhnreg_ks will produce usable error messages which should help you figure out what is failing during registration.
5.10.7. Directory structure for Kickstarts and Snippets
Kickstarts — The base path where the kickstart files are stored is /var/lib/rhn/kickstarts/. Within this directory, raw (non-wizard generated) kickstarts reside in the subdirectory upload while wizard generated ones are in the wizard subdirectory, thus:
Raw Kickstarts: /var/lib/rhn/kickstarts/upload/$profile_name--$org_id.cfg
Wizard Kickstarts: /var/lib/rhn/kickstarts/wizard/$profile_name--$org_id.cfg
Snippets — Cobbler Snippets are stored in /var/lib/rhn/kickstarts/snippets. Cobbler accesses snippets in this structure via a symbolic link in /var/lib/cobbler/snippets callled spacewalk — thus /var/lib/cobbler/snippets/spacewalk. Satellite's RPMs expect Cobbler's kickstart and snippet directories to be in their default locations — it is not recommended to alter them.
Snippets: /var/lib/rhn/kickstarts/snippets/$org_id/$snippet_name
