TWiki> Sandbox Web>Sandbox Web? >JohnWeigandSandbox>WeigandGlideinTroubleshooting (2011/01/04, JohnWeigand)EditAttach

glideinWMS Troubleshooting

Description

This document is intended to assist in troubleshooting problems / communication issues between the various glideinWMS services.

VOFrontend / Factory Communication

This section documents several cases where the VOFrontend cannot authenticate/authorize itself with the factory. No glideins will be started for the frontend until these conditions are resolved.

VOFrontend logs

The errors shown below are from the frontend's log files
There is no indication of any problem in the Factory logs or WMS Collector Condor log files.

Found an untrusted factory

info log:
[2010-09-29T09:07:24-05:00 26824] WARNING: Found an untrusted factory ress_ITB_GRATIA_TEST_2@v2_4_3@factory_service at cms-xen21.fnal.gov; ignoring.

debug log:
[2010-09-29T09:07:24-05:00 26824] Found an untrusted factory ress_ITB_GRATIA_TEST_2@v2_4_3@factory_service at cms-xen21.fnal.gov; identity mismatch ' weigand@cms-xen21.fnal.gov'!='factory@cms-xen21.fnal.gov '

Show details Close

frontend config
<frontend frontend_name="frontend_service-v2_4_3"
<collector DN="/DC=org/DC=doegrids/OU=Services/CN=cms-xen21.fnal.gov" factory_identity="factory@cms-xen21.fnal.gov"

Reason:
The frontend config's security element security_name attribute does not match
the factory config's frontend element name attribute.

condor_status -collector <WMSCollector_node:port> -long |grep -i AuthenticatedIdentity? |sort -u

Factory logs

The errors shown below are from the factory's log files
There is no indication of any problem in the frontend logs.

Not in whitelist

[2010-09-27T15:37:22-05:00 5094] WARNING: Client frontend_service-v2_4_3.main (secid: frontend_identity) not in white list. Skipping request

Show details Close

frontend config
<frontend frontend_name="frontend_service-v2_4_3"
<collector my_identity="frontend_identity@cms-xen21.fnal.gov
<security security_name="frontend_identity"
<proxy security_class="frontend"

factory config
<frontend name="frontend_service" identity="frontend_service@cms-xen21.fnal.gov"
<security_class name="frontend" username="vo_cms"

Reason:
The frontend config's security element security_name attribute does not match
the factory config's frontend element name attribute.

Not coming from a trusted source

[2010-09-28T09:40:45-05:00 12265] WARNING: Client frontend_service-v2_4_3.main (secid: frontend_identity) is not coming from a trusted source; AuthenticatedIdentity? frontend_identity@cms-xen21.fnal.gov!=frontend_service@cms-xen21.fnal.gov. Skipping for security reasons.

Show details Close

frontend config
<frontend frontend_name="frontend_service-v2_4_3"
<collector my_identity="frontend_identity@cms-xen21.fnal.gov"
<security security_name="frontend_identity"
<proxy security_class="frontend"

factory config
<frontend name="frontend_identity" identity="frontend_service@cms-xen21.fnal.gov"
<security_class name="frontend" username="vo_cms"

Reason:
The frontend config's collector element my_identity attribute does not match
the factory config's frontend element identity attribute

No mapping for security class frontend of x509_proxy_0

[2010-09-28T09:59:21-05:00 12822] WARNING: No mapping for security class frontend of x509_proxy_0 for frontend_service-v2_4_3.main (secid: frontend_identity), skipping and trying the others
[2010-09-28T09:59:21-05:00 12822] WARNING: No good proxies for frontend_service-v2_4_3.main, skipping request

Show details Close

frontend config
<frontend frontend_name="frontend_service-v2_4_3
<collector my_identity="frontend_identity@cms-xen21.fnal.gov
<security security_name="frontend_identity"
<proxy security_class="frontend"

factory config
<frontend name="frontend_identity" identity="frontend_identity@cms-xen21.fnal.gov"
<security_class name="frontend-2" username="vo_cms"

Reason:
The frontend config's proxy element security_class attribute does not match
the factory config's security_class element name attribute.

Client provided invalid ReqEncIdentity

[2010-10-04T13:28:40-05:00 8176] Client ress_ITB_GRATIA_TEST_3@v2_4_3@factory_service@frontend_service-v2_4_3.main provided invalid ReqEncIdentity? ( frontend_service @cms-xen21.fnal.gov!= frontend_identity @cms-xen21.fnal.gov). Skipping for security reasons.

Show details Close

frontend config

<security classad_proxy="/home/cms/grid-security/x509_glidein_xen22_proxy" proxy_DN=" /DC=org/DC=doegrids/OU=Services/CN=glidein/cms-xen22.fnal.gov "

factory config
<frontend name="frontend_service" identity="frontend_service@cms-xen21.fnal.gov"
<security_class name="frontend" username="vo_cms"

CONDOR_LOCATION/certs/condor_mapfile

GSI " ^\/DC\=org\/DC\=doegrids\/OU\=Services\/CN\=glidein\/cms\-xen22\.fnal\.gov$ " frontend_identity

Reason:

When the VOFrontend contacts the WMS Collector using the frontend configuration file's security element proxy_DN/classad_proxy attribute, the WMS Collector Condor uses the certs/condor_mapfile to map the VOFrontend to a name. This name identifies how the factory knows the VOFrontend on the Factory node (frontend_identity@cms-xen21.fnal.gov). This must match with the factory configuration file's frontend element identity attribute (frontend_service@cms-xen21.fnal)

VOFrontend-Factory Mappings

frontend config

<frontend frontend_name="frontend_service-v2_4_3"
<collector my_identity="frontend_identity@cms-xen21.fnal.gov"
<security security_name="frontend_identity" proxy_DN=" /DC=org/DC=doegrids/OU=Services/CN=glidein/cms-xen22.fnal.gov "
<proxy security_class="frontend"

factory config
<frontend name="frontend_identity" identity="frontend_identity@cms-xen21.fnal.gov"
<security_class name="frontend" username="vo_cms"

CONDOR_LOCATION/certs/condor_mapfile
GSI " ^ \/DC\=org\/DC\=doegrids\/OU\=Services\/CN\=glidein\/cms\-xen22\.fnal\.gov$ " frontend_identity

Factory reconfig errors

These are errors that can occur when executing factory_startup reconfig.

Failed to create base clientlog dir

ERROR: Failed to create base clientlog dir (user xxx_cms):
Error running '/usr/local/glideins/v2_4_3_alpha_1/condor-wms/bin/../sbin/condor_root_switchboard mkdir 0 2'
code 256:["option 'user-uid' has an invalid uid in file: <stdin>:1\n"]
Reconfiguring the factory [FAILED]

NOTE: This may only occur when privilege separation is in effect... maybe???

Show details Close

frontend config
<frontend frontend_name="frontend_service-v2_4_3
<collector my_identity="frontend_identity@cms-xen21.fnal.gov
<security security_name="frontend_identity"
<proxy security_class="frontend"

factory config
<frontend name="frontend_identity" identity="frontend_identity@cms-xen21.fnal.gov"
<security_class name="frontend" username="xxx_cms"

Reason:
The factory config's security_class element username attribute is not defined
in the /etc/condor/privsep_config file as a valid-target-uids value:

valid-caller-gids = us_cms
valid-target-uids = vo_cms
valid-target-gids = vo_cms
valid-dirs = /usr/local/glideins-clients/v2_4_3_alpha_1/clients/logs
valid-dirs = /usr/local/glideins-clients/v2_4_3_alpha_1/clients/proxies
procd-executable = /usr/local/glideins/v2_4_3_alpha_1/condor-wms/sbin/condor_procd valid-caller-uids = weigand

-- JohnWeigand - 2010/09/28

Edit | Attach | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r8 - 2011/01/04 - 20:10:48 - JohnWeigand
 
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback