Tips: resource limits in OpenDJ

Photo by Scallop Holden http://www.flickr.com/photos/scallop_holden/

OpenDJ, the open source LDAP directory services in Java, defines a few global resource limits to prevent client connections or operations from abusing the server’s resources. These limits are

• the maximum number of entries returned to a search request (size-limit, default is 1000),
• the maximum amount of time to spend returning results to a client (time-limit, default is 60 seconds),
• the maximum number of entries to look through while processing a search request (lookthrough-limit, default is 5000),
• the maximum amount of time a connection can sit idle before the server disconnect it (idle-time-limit, default is unlimited).

There are default values for all of these limits in the Global configuration, but they can also be set on a per user basis. The global limits are read or set using dsconfig :

$ bin/dsconfig get-global-configuration-prop -p 4444 -X -n -h localhost \
 -D cn=directory\ manager -w secret12
Property : Value(s)
--------------------------------------:------------------------
bind-with-dn-requires-password : true
default-password-policy : Default Password Policy
disabled-privilege : -
entry-cache-preload : false
etime-resolution : milliseconds
idle-time-limit : 0
lookthrough-limit : 5000
max-allowed-client-connections : 0
max-psearches : unlimited
proxied-authorization-identity-mapper : Exact Match
reject-unauthenticated-requests : false
return-bind-error-messages : false
save-config-on-successful-startup : true
size-limit : 1000
smtp-server : -
time-limit : 60 s
writability-mode : enabled

The per user limits have a different LDAP attribute name and can be found or set directly in users’ entry, or through Collective Attributes. The Directory Manager entry has such specific limits set, so that everything is unlimited.

$ bin/ldapsearch -D "cn=directory manager" -w secret12 -p 1389 -X -b "cn=config" \
  '(objectClass=inetOrgPerson)' ds-rlim-time-limit ds-rlim-size-limit \
  ds-rlim-lookthrough-limit ds-rlim-idle-time-limit
dn: cn=Directory Manager,cn=Root DNs,cn=config
ds-rlim-lookthrough-limit: 0
ds-rlim-time-limit: 0
ds-rlim-idle-time-limit: 0
ds-rlim-size-limit: 0

If you decide to change the default global settings, for example the idle-time-limit, to force idle connections to be closed by the server after some time (often a smaller time than the settings of the load-balancer in between your applications and the OpenDJ servers), please remember that you might also want to change the limit for “cn=Directory Manager”, especially if your client applications are connecting with Directory Manager credentials.

Cache strategy for OpenDJ LDAP directory server

System administrators that are familiar with legacy LDAP directory servers know that one of the key for the best performance is caching the data. With Sun Directory Server or OpenLDAP, there are 3 levels of caching that could be done : the filesystem level, the database level and the entries level. The filesystem level cache is managed by the OS and cannot be controlled by the application. Using the filesystem cache is good when the directory server is the only process on the machine, and/or for initial performance. The database level cache allows faster read or write operations, and also includes the indexes. The later cache is the higher level cache, and usually the one that provides the best performances as it requires the least processing from the server to return entries to the applications, and it has the least contention.

OpenDJ has a different design for its database and core server, and thus the caching strategy needs to be different.

By default, OpenDJ does have a database cache enabled, and 3 different kind of entry caches, all disabled. The reason for the 3 entry caches is that they are implementing for different needs and access patterns. But all have in common a specific filter to select which entries to cache, and some settings as to how much memory to use. During our stress and performance tests, we noticed that using an entry cache for all accessed entries added a lot of pressure on the garbage collector, and also caused more garbage collection from the old generation, often leading to either fragmentation of the memory, or more frequent full GC (also known as “Stop the world GC”). This resulted in an overall lower consistent average response time and throughput.

So, we recommend that you favor the database cache, and do not setup an entry cache, except for specific needs (and do not try to activate all 3 entry caches, this may lead to some really strange behavior).

The default settings with OpenDJ 2.4 is that 10 % of the JVM heap space will be used for the database cache. With OpenDJ 2.5 (soon to be released), we have bumped the default to 50% of the heap space. If you’re tuning the heap size and make it larger than 2GB, we recommend that you keep that 50% ratio or even increase it if the heap size exceeds the 3GB.

If you do have a few very specific entries that are very often accessed, like large static groups that are constantly used for ACI or group membership by application, then the entry cache becomes handy, and then you want to set a filter so only these specific entries are cached.

For example, if you want to cache at most 5 entries, that are groupOfNames, you can use the following dsconfig command:

bin/dsconfig set-entry-cache-prop --cache-name FIFO
 --set include-filter:\(objectclass=GroupOfNames\)
 --set max-entries:5 --set max-memory-percent:90 --set enabled:true
 -h localhost -p 4444 -D "cn=Directory Manager" -w secret12 -X -n

Otherwise, you’d better of running with no entry cache. OpenDJ read performance are such that the directory server can respond to tens of thousands if not hundred of thousands searches per second with average response time in the order of a milli-second. This should be good enough for most applications !

OpenDJ: Extensible indexes for Internationalization.

While taming the subject of indexes, we recently had some discussion with one of our users who complained about long response times with some language specific search filter such as (cn:fr.6:=*John*).

These extended filters rely on I18N Collation matching rules and indexes that I’ve described in an old post for OpenDS.

It turned out that he had defined the matching rule for the index, and rebuilt it, but had missed an important part: the index-type did not include “extensible”.

The proper command to create an extensible index is the following :

dsconfig set-local-db-index-prop --backend-name userRoot --index-name cn \
 --set index-extensible-matching-rule:fr.6 \
 --add index-type:extensible \
 --hostname localhost --port 4444 \
 --bindDN cn=Directory\ Manager --bindPassword ****** \
 -X -n

fr.6 is the shortcut for the French substring collation matching rule which full OID is 1.3.6.1.4.1.42.2.27.9.4.76.1. 

Note that if you don’t specify the extensible index-type, the server will not build the index for the extensible matching rule. The use of the index-type is consistent with the other types of index, equality or else, and allows you to disable and re-enable extensible indexes without having to re-enter all  OIDs.

OpenDJ Tips: More on troubleshooting indexes and search performances

In a previous post I talked about analyzing search filters and indexes. Matt added in a comment that OpenDJ has another mean of understanding how indexes are used in a search. Here’s a detailed post.

The OpenDJ LDAP directory server supports a “magic” operational attribute that allows an administrator to get from the server information about the processing of indexes for a specific search query: debugsearchindex.

If the attribute is set in the requested attributes in a search operation, the server will not return all entries as expected, but a single result entry with a fixed distinguished name and a single valued attribute debugsearchindex that contains the information related to the index processing, including the number of candidate entries per filter component, the overall number of candidate, and whether any or all of the search is indexed.

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(mail=user.*)(cn=*Denice*))" debugsearchindex
Password for user 'cn=Directory Manager': *******
dn: cn=debugsearch
debugsearchindex: filter=(&(mail=user.*)[INDEX:mail.substring][COUNT:2000](cn=*Denice*)[INDEX:cn.substring][COUNT:1])[COUNT:1] final=[COUNT:1]

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "objectclass=*" debugsearchindex
Password for user 'cn=Directory Manager': *********
dn: cn=debugsearchdebugsearchindex: filter=(objectClass=*)[NOT-INDEXED] scope=wholeSubtree[COUNT:2007] final=[COUNT:2007]

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "mail=user.1*" debugsearchindex
Password for user 'cn=Directory Manager': *********
dn: cn=debugsearch
debugsearchindex: filter=(mail=user.1*)[INDEX:mail.substring][COUNT:1111] scope=wholeSubtree[COUNT:2007] final=[COUNT:1111]

Note that sometimes, OpenDJ tries to optimize the query and use some other index than the regular one for the query. For example, it might use the equality index for an initial substring filter. The index used during the search does appear in the debugsearchindex attribute. Also, once the result set has been narrowed down to very few entries, it will stop using index and evaluate directly the entry set, as for the example below:

$ bin/ldapsearch -h localhost -p 1389 -D "cn=Directory Manager" -b "dc=example,dc=com" "(&(cn=Denice*)(mail=user.9*))" debugsearchindex
Password for user 'cn=Directory Manager':
dn: cn=debugsearch
debugsearchindex: filter=(&(cn=Denice*)[INDEX:cn.equality][COUNT:1])[COUNT:1] final=[COUNT:1]

Disabling Replication in OpenDJ 2.4.

Enabling replication between multiple instances of the OpenDJ LDAP directory server is pretty simple and straightforward. You can check for yourself in the Replication chapter of the Administration Guide.

But fully disabling replication can be tricky with OpenDJ 2.4, mostly because of a known issue with the dsreplication disable –disableAll command : OPENDJ-249 : Doing dsreplication disable –disableAll is throwing a javax.naming.CommunicationException when removing contents of “cn=admin data”.

We are fixing this issue in OpenDJ 2.5, but for those who have deployed OpenDJ 2.4 and want to know how to fully remove all references to a replica in the topology, here are the steps to manually disable replication :

Note, all these steps should be done using ldapmodify, or an LDAP browser such as OpenDJ Control-Panel’s Manage Entry or Apache Directory Studio.

1. For each replica to be disabled connect to it on the admin port (4444) and:
   1. MANDATORY: set the “ds-cfg-enabled” property to “false” in “cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config”
   2. OPTIONAL: recursively remove the entries beneath “cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config” using individual delete operations. Note that the configuration backend does not support the sub-tree delete control, so this has to be done iteratively. This step is also not mandatory, since replication was fully disabled in the previous step
   3. MANDATORY: remove each entry beneath “cn=Servers,cn=admin data” except the entry itself. I find the easiest way to do this is to perform a sub-tree delete and then add back the base entry
   4. OPTIONAL: remove (purge) unused instance keys from beneath “cn=instance keys,cn=admin data” *except* own key. This step is really independent of replication: administrators should periodically purge unused instance keys anyway when they are sure that they are no longer needed (e.g. used for signing backups, etc)
   5. MANDATORY: delete “uniqueMember” in “cn=all-servers,cn=Server Groups,cn=admin data”
2. On one of the remaining enabled replicas, connect to it via the admin port and:
   1. MANDATORY: remove each disabled server beneath “cn=Servers,cn=admin data”
   2. OPTIONAL: remove (purge) each disabled instance key beneath “cn=Servers,cn=admin data” (see 1.4)
   3. MANDATORY: remove each disabled server from uniqueMember in “cn=all-servers,cn=Server Groups,cn=admin data”
   4. MANDATORY: get list of all remaining servers from “cn=all-servers,cn=Server Groups,cn=admin data”
3. For each of the remaining enabled replicas obtained in step 2.4 connect to it via the admin port and:
   1. MANDATORY: remove each disabled server(rsPort) from ds-cfg-replication-server in “cn=replication server,cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config”
   2. MANDATORY: remove each disabled server(rsPort) from ds-cfg-replication-server in “cn=*,cn=domains,cn=Multimaster Synchronization,cn=Synchronization Providers,cn=config”

An important tuning flag for OpenDJ with 64bit JVM…

If you’re running OpenDJ with a 64bit JVM with less than 32GB of heap size, be aware of the need to explicitly set the -XX:+UseCompressedOops option (unless you want to disable it).

Compressed oops is supported and enabled by default in Java SE 6u23 and later, when running a 64bit JBM with a value of -Xmx lower than 32GB. You can find more information about Compressed Oops in Java technical notes here: http://download.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html

However, OpenDJ internal database, in order to estimate properly the occupation of the DB cache and tune the cache eviction threads, needs to take into account the compressed oops option. For this is relies on the JVM option to be set explicitly. If the option is not explicitly set, the database may consider the cache full when it’s not, and run cache eviction too early, resulting in less optimized performances.

So, with 64bit JVM, make sure you add the -XX:+UseCompressedOops option to the start-ds line in the config/java.properties file. Then run bin/dsjavaproperties and restart OpenDJ to benefit from the new settings.

LDAP: Matching against the current time in OpenDJ

In LDAP, attributes have different syntaxes. The one used to indicate date and time is the GeneralizedTime, a string representation of the date and time, typically expressed in GMT time. For example, when an entry is modified, the server maintains the modifytimestamp attribute and sets a value like 20110825120001Z (for 2011, Aug 25, 12:00:01 GMT).

LDAP client applications often have to search for entries based on these date and time attributes, whether it is to find the entries that have been modified , or had the password changed recently… The way it is typically done, is the following: get from the system the current date, add or substract some fixed time (for example if you want to know the entries modified in the last 10 minutes), transform  to a GeneralizedTime, use that string in a search filter: (modifyTimestamp >= 20110825130000Z). If the application repeats that search a minute later, it has to recompute the value again, and again…

Ideally what application writers would like is to express the filter as an expression like (modifyTimestamp>=${CurrentTime} – 10 mn). However this is not compliant with LDAP. The proper way to solve this is to use extensible matching rules, and for that purpose, we’ve added 2 “relative time” matching rules in OpenDJ, the Open source LDAP Directory services for Java: one for “lower than” and one for “greater than”.

matchingrules: ( 1.3.6.1.4.1.26027.1.4.6 NAME ( 'relativeTimeLTOrderingMatch' 'relativeTimeOrderingMatch.lt' )
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 )
matchingrules: ( 1.3.6.1.4.1.26027.1.4.5 NAME ( 'relativeTimeGTOrderingMatch' 'relativeTimeOrderingMatch.gt' )
  SYNTAX 1.3.6.1.4.1.1466.115.121.1.24 )

The way the matching rules work is pretty simple : (attribute:MatchingRule:=Offset), where the offset is a signed integer follow by its unit, either s for seconds, m for minutes, h for hours, d for days or w for weeks.

You can translate a statement to “is Attribute greater than (or lower than) CurrentTime +/- Offset”

(lastLoginTime:1.3.6.1.4.1.26027.1.4.6:=-4w) will match all entries who have a lastLoginTime value smaller than the Current Time minus 4 weeks, i.e. all entries who have a lastLoginTime older than 4 weeks.

(pwdExpirationTime:1.3.6.1.4.1.26027.1.4.5:=5d) will match all entries that have pwdExpirationTime greater than the Current Time plus 5 days, i.e. all entries that will expire in more than 5 days.

The true benefit of those matching rules, is actually when expressing policies in the OpenDJ server, for example for granting or denying access based on some attribute with a generalizedTime syntax, such as last login time, pwdChangedTime, modifyTimeStamp …

For example, imagine an auxiliary objectClass representing a service, with some specific attributes including an expiration date : validUntil. Now, you want to allow these attributes to be read only if the expiration date is not passed.

aci: (targetattr="serviceAttr1 || serverAttr2")(targetfilter="(validUntil:1.3.6.1.4.1.26027.1.4.5:=0s)")
  (version 3.0; acl "Read Valid service attributes"; allow (read, search, compare)
  userdn="ldap:///all";)

As you can see, this is a good way to hide (deny access to) stale data in a directory server, and to simplify client applications that need to search for entries based on some generalizedTime attributes. For example, consider using these “relative time” matching rules for all your audit queries for expired or unused accounts.

Finally, remember that the OpenDJ directory server doesn’t allow unindexed searches by default. So you might also want to create an index for the “relative time” matching rules. That’s a 2 steps process :

Define the index

$ bin/dsconfig create-local-db-index --backend-name userRoot --set index-type:extensible \
 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.5 \
 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.6 \
 --index-name createTimestamp -h localhost -p 4444 \
 -D cn=Directory\ Manager -w secret12 -n -X

Rebuild the index

$ bin/rebuild-index -b dc=example,dc=com -i createTimestamp \
 -h localhost -p 4444 -D cn=directory\ manager -w secret12 -X

OpenDJ: Analyzing Search Filters and Indexes

LDAP directory services greatly rely on indexes to provide fast and accurate search results.

OpenDJ, the open source LDAP directory services for the Java platform, provides a number of tools to ensure indexes are efficiently used or to optimize them for even better performances.

To start with, OpenDJ rejects by default all unindexed searches, unless the authenticated user has the privilege to perform them. Unindexed searches are rejected because they result in scanning the whole database, which consumes lots of resources and time. There are legitimate uses of unindexed search though, and OpenDJ offers a way to control who can perform them through a privilege. To learn more about privileges, how to grant them, please check the Administration Guide or some of my previous posts.

When unindexed searches are completed, OpenDJ (starting with revision 7148 of the OpenDJ trunk, and therefore OpenDJ 2.5) does logs the “Unindexed” keyword as part of the Search Response access log message. But the access log file can also be used to identify search operations that are not making an optimal use of indexes. Simply check for those search responses that have been returned with an etime (execution time) greater than the average.

The access log example below contains both an unusually high etime (expressed in ms) and the Unindexed tag.

[27/Jul/2011:20:27:27 +0200] SEARCH RES conn=0 op=1 msgID=2 result=0 nentries=10001 Unindexed etime=1846

The verify-index command let you check that no index is corrupted (i.e. no data is missing from indexes).

The rebuild-index command let you build or rebuild an index that would be corrupted or had its configuration changed.

One of the tuning parameter of indexes is the index-entry-limit (which was known in Sun DSEE as the AllIDsThreshold), the maximum size of entries kept in an index record, before the server stop maintaining that record and consider it’s more efficient to scan the whole database. For more information on the index entry limit, check the Section 7.2.4 Changing Index Entry Limits of the Indexing chapter of the Administration Guide.

OpenDJ provides a static analyzer of indexes which can help to understand how well the attributes are indexed, as well as help to tune the index entry limit. This tool is a function of the dbtest utility and is simply used as follow:

$ bin/dbtest list-index-status -n userRoot -b "dc=example,dc=com"

Index Name Index Type JE Database Name Index Valid Record Count Undefined 95% 90% 85%

---------------------------------------------------------------------------------------------------------------------------------------
id2children                Index       dc_example_dc_com_id2children                true         2             0          0    0    0
id2subtree                 Index       dc_example_dc_com_id2subtree                 true         2             0          0    0    0
uid.equality               Index       dc_example_dc_com_uid.equality               true         2000          0          0    0    0
aci.presence               Index       dc_example_dc_com_aci.presence               true         0             0          0    0    0
ds-sync-conflict.equality  Index       dc_example_dc_com_ds-sync-conflict.equality  true         0             0          0    0    0
givenName.equality         Index       dc_example_dc_com_givenName.equality         true         2000          0          0    0    0
givenName.substring        Index       dc_example_dc_com_givenName.substring        true         5777          0          0    0    0
objectClass.equality       Index       dc_example_dc_com_objectClass.equality       true         6             0          0    0    0
member.equality            Index       dc_example_dc_com_member.equality            true         0             0          0    0    0
uniqueMember.equality      Index       dc_example_dc_com_uniqueMember.equality      true         0             0          0    0    0
cn.equality                Index       dc_example_dc_com_cn.equality                true         2000          0          0    0    0
cn.substring               Index       dc_example_dc_com_cn.substring               true         19407         0          0    0    0
sn.equality                Index       dc_example_dc_com_sn.equality                true         2000          0          0    0    0
sn.substring               Index       dc_example_dc_com_sn.substring               true         8147          0          0    0    0
telephoneNumber.equality   Index       dc_example_dc_com_telephoneNumber.equality   true         2000          0          0    0    0
telephoneNumber.substring  Index       dc_example_dc_com_telephoneNumber.substring  true         16506         0          0    0    0
ds-sync-hist.ordering      Index       dc_example_dc_com_ds-sync-hist.ordering      true         1             0          0    0    0
mail.equality              Index       dc_example_dc_com_mail.equality              true         2000          0          0    0    0
mail.substring             Index       dc_example_dc_com_mail.substring             true         7235          0          0    0    0
entryUUID.equality         Index       dc_example_dc_com_entryUUID.equality         true         2002          0          0    0    0

Total: 20

If an index contains a non zero value (N) in the undefined column, it means N index keys have reached the index entry limit and are no longer maintained. This can be normal, for example with the ObjectClass equality index, where the vast majority of entries will have the same objectclasses (top, Person, organizationalPerson, inetOrgPerson). But, for other attributes, such as cn, it may indicate that the index entry limit is too low.

Finally, OpenDJ has an option to do a live analysis of search filters and how they use indexes. To enable live index analysis, simply enable it for the database backend that contains the data :

dsconfig set-backend-prop --backend-name userRoot  --set index-filter-analyzer-enabled:true \
 --set max-entries:50 -h localhost -p 4444 -D cn=Directory\ Manager -w ****** -n -X

The max-entries parameter specifies how many filter items are being analyzed and kept in memory. Only the last max-entries will be kept. If there is a huge variety of requests against the directory service, you might want to increase the number. However, keep in mind that the analysis is kept in memory, and the higher the number the largest the impact on the overall performances of the server.

We do not recommend that you leave the index analysis enabled all the time, especially in production. The index analyzer should be used to gather statistics over a flow of requests for a short period of time, and should be disabled afterwards to free the resources.

The result of the index analyzer can be retrieved under the cn=monitor suffix, more specifically as part of the database environment of the backend.

$ bin/ldapsearch -p 1389 -D cn=directory\ manager -w secret12  \
-b "cn=userRoot Database Environment,cn=monitor" '(objectclass=*)' filter-use

dn: cn=userRoot Database Environment,cn=monitor
filter-use: (uid=user.*) hits:1 maxmatches:20 message:
filter-use: (tel=*) hits:1 maxmatches:-1 message:presence index type is disabled
  for the tel attribute
filter-use: (objectClass=groupOfURLs) hits:1 maxmatches:0 message:
filter-use: (objectClass=groupOfEntries) hits:1 maxmatches:0 message:
filter-use: (objectClass=person) hits:1 maxmatches:20 message:
filter-use: (objectClass=ds-virtual-static-group) hits:1 maxmatches:0 message:
filter-use: (aci=*) hits:1 maxmatches:0 message:
filter-use: (objectClass=groupOfNames) hits:1 maxmatches:0 message:
filter-use: (objectClass=groupOfUniqueNames) hits:1 maxmatches:0 message:
filter-use: (objectClass=ldapSubentry) hits:1 maxmatches:0 message:
filter-use: (objectClass=subentry) hits:1 maxmatches:0 message:

hits represents the number of time this filter was used. the maxmatches represents the maximum number of entries that were returned for that filter.

Index analysis and tuning is not a simple task, and I recommend to play with these tools  a lot on a test environment to understand how to get the best out of them. But, as you can see, OpenDJ provides you with all the tools you need to get the best performances out of your LDAP directory.

Newbie help : How to reset the Directory Manager’s password ?

We get this question quite often on IRC or mailling lists, from newbies who’ve installed OpenDJ (or OpenDS) for evaluation and forgot the Directory Manager’s password.

So here are the steps :

Make sure OpenDJ is stopped.

bin/stop-ds

Generate an encoded password for Directory Manager :

bin/encode-password -s SSHA512 -c AS3cur3PassW0rd
Encoded Password:  "{SSHA512}G/knE0xkyW2Af3+1MFy+yPYxchGgLuqog71R4njPJcs9t5NDAadqLxU7pxZjZkrDquQeb5aq7tum1ZFC3uE+r4Nmuil4S46A"

Copy the string within quotes (without the quotes), and edit the config/config.ldif file.

Go down to the following entry

dn: cn=Directory Manager,cn=Root DNs,cn=config

Replace the value of userPassword with the newly generated one.

dn: cn=Directory Manager,cn=Root DNs,cn=config
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: top
objectClass: ds-cfg-root-dn-user
userpassword: {SSHA512}G/knE0xkyW2Af3+1MFy+yPYxchGgLuqog71R4njPJcs9t5NDAadqLxU7pxZjZkrDquQeb5aq7tum1ZFC3uE+r4Nmuil4S46A
...

You can now restart the server and administer it.

OpenDJ: Troubleshooting LDAP SSL connections

Troubleshooting Secure Socket Layer (SSL, also now standardized as TLS) issues is not trivial and there is no secret sauce specific to OpenDJ.

Should an LDAP SSL connection fails due to the server, you should find a descriptive error message in the server’s errors log (in logs/errors). But sometime the connection is aborted by the client with some obscure message. Often we see a message ending with “ javax.net.ssl.SSLHandshakeException: no cipher suites in common“.

Java has some debugging capabilities embedded and they are pretty easy to use with the OpenDJ LDAP directory server, which just need to be restarted with some additional arguments: -Djavax.net.debug=all or -Djavax.net.debug=ssl.

There are two ways to add extra arguments to the OpenDJ server startup command, using an environment variable, or using the java.properties file.

Using env variable

- you define the OPENDS_JAVA_ARGS environment variable. And you restart the server. If you do so, make sure you include all previous arguments.

OPENDS_JAVA_ARGS='-server -Xms1G -Xmx1G -Djavax.net.debug=ssl,handshake,trustmanager' bin/start-ds

Using the java.properties file

Edit the java.properties file in the config directory.
Since you probably only want to track the OpenDS directory server SSL access, you should append the -Djavax.net.debug=ssl,handshake,trustmanager args to the start-ds line (rather than applying it to all commands).

start-ds.java-args=-Xms1G -Xmx1G -server -Djavax.net.debug=ssl,handshake,trustmanager

Save the file and run the dsjavaproperties command:

bin/dsjavaproperties

Now restart the server, using the start-ds command

Where is the output ?

All SSL related logs are output in the logs/server.out file.
To test, you can use ldapsearch :

bin/ldapsearch -Z -X -p 1636 -b "" -s base '(objectclass=*)'

And if you look into the logs/server.out file, you will see something similar to this:

Using SSLEngineImpl.
 Allow unsafe renegotiation: false
 Allow legacy hello messages: true
 Is initial handshake: true
 Is secure renegotiation: false
 LDAP Request Handler 0 for connection handler LDAP Connection Handler 0.0.0.0 port 1636, READ: SSL v2, contentType = Handshake, translated length = 81
 *** ClientHello, TLSv1
 RandomCookie: GMT: 1287771875 bytes = { 68, 231, 5, 253, 105, 26, 137, 36, 38, 238, 12, 141, 110, 12, 59, 10, 192, 135, 113, 119, 108, 153, 10, 31, 127, 120, 110, 61 }
 Session ID: {}
 Cipher Suites: [SSL_RSA_WITH_RC4_128_MD5, SSL_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_RC4_40_MD5, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
 ...

This will help you to identify what part of the secure connection is failing and fix it.

Note that -Djavax.net.debug=ssl enables debug of the SSL connections, while -Djavax.net.debug=all enables full debugging including use of certificates, and more. You can also find more debug options by using -Djavax.net.debug=help.

 

OpenDJ Tip: Auto-completion of dsconfig command

With OpenDJ LDAP directory services, a single command-line tool, dsconfig, is used to manage every configuration parameter of the server.

The dsconfig command has several modes and useful options, some of them are not well known. So let me repeat them.

• interactive mode: if started with no parameter or no command, the tool goes in interactive mode. This is a nice way to discover the various parts of OpenDJ configuration.
• advanced mode: There are really a lot of configuration parameters in OpenDJ, so not all of them can be read or set by default. An advanced mode allows to view and edit the hidden ones.
• scripted mode: dsconfig can be used with a file containing several commands, and will call them one after the other.
• teaching mode: if the option –displayCommand is used in interactive mode, it will display the complete command to use for non interactive mode, or for scripted mode.

But regardless of the different modes, there are really too many commands and options to remember. So shell completion can come to the rescue.

Just insert the following lines in your .bashrc or .cshrc file.

Bash:

complete -W "`bin/dsconfig --help-all|grep '^[a-z].*'`" dsconfig

csh:

set DSC = `<OpenDJPath>/bin/dsconfig --help-all|grep '^[a-z].*'`
complete dsconfig "p/1/($DSC)/"

And now type bin/dsconfig set[TAB] and the list of commands appears magically…

$ dsconfig set-password-[TAB][TAB]
set-password-generator-prop       set-password-storage-scheme-prop
set-password-policy-prop          set-password-validator-prop
$ dsconfig set-password-

OpenDJ: Enabling the External Change Log on a single server

Yesterday, I described how easy it is to enable Multi-Master Replication between 2 instances of OpenDJ. One of the nice thing with OpenDJ replication, is that all changes are also publicly available (subject to access control) through LDAP under the cn=changelog suffix, also nick-named the External Change Log.

But the command to enable replication and thus the External Change Log requires 2 servers. So what if you want to enable the External Change Log on a single server ?

Well you cannot use the dsreplication command, but you can configure the OpenDJ instance with the dsconfig utility.

$ bin/dsconfig create-replication-server -h ldap1.example.com -p 4444 \
  -D "cn=directory manager" -w secret12 -X -n \
  --provider-name "Multimaster Synchronization" --set replication-port:8989 \
  --set replication-server-id:2 --type generic
  
$ bin/dsconfig create-replication-domain -h ldap1.example.com -p 4444 \
  -D "cn=directory manager" -w secret12 -X -n \
  --provider-name "Multimaster Synchronization" --set base-dn:dc=example,dc=com \
  --set replication-server:ldap1.example.com:8989 --set server-id:3 \
  --type generic --domain-name example_com

If you want to be able to join this server in a replication topology, you should also create the global administrator’s entry. If you do so, then you will be able to use the dsreplication enable command as illustrated here.

$ bin/dsframework create-admin-user -X -h ldap1.example.com -p 4444 \
  -D "cn=Directory Manager" -w secret12 --userID admin --set password:password

Once enabled, you can read or search the changes with ldapsearch or other LDAP clients :

$ bin/ldapsearch -D cn=directory\ manager -w secret12 -h ldap1.example.com -p 1389 \
  -J "1.3.6.1.4.1.26027.1.5.4:false:;" -b "cn=changelog" '(objectclass=*)'
dn: cn=changelog
cn: changelog
objectClass: top
objectClass: container

# Public changelog exchange control(1.3.6.1.4.1.26027.1.5.4): dc=example,dc=com:0000012fd9bdf863000300000001;
dn: replicationCSN=0000012fd9bdf863000300000001,dc=example,dc=com,cn=changelog
targetDN: cn=a,ou=people,dc=example,dc=com
changeNumber: 0
changes:: b2JqZWN0Q2xhc3M6IHBlcnNvbgpvYmplY3RDbGFzczogdG9wCmNuOiBhCnNuOiBhCmVudH
 J5VVVJRDogNWQzMTNlY2UtYjY4Mi00MDFiLTg2NmYtM2NiZWNlMWNjNTJjCmNyZWF0ZVRpbWVzdGFtc
 DogMjAxMTA1MTAxMTQ5NTZaCmNyZWF0b3JzTmFtZTogY249RGlyZWN0b3J5IE1hbmFnZXIsY249Um9v
 dCBETnMsY249Y29uZmlnCg==
changeType: add
changeTime: 20110510114956Z
objectClass: top
objectClass: changeLogEntry

# Public changelog exchange control(1.3.6.1.4.1.26027.1.5.4): dc=example,dc=com:0000012fd9be46af000300000002;
dn: replicationCSN=0000012fd9be46af000300000002,dc=example,dc=com,cn=changelog
targetDN: cn=a,ou=people,dc=example,dc=com
changeNumber: 0
changes:: YWRkOiBkZXNjcmlwdGlvbgpkZXNjcmlwdGlvbjogTmV3IG9uZQotCnJlcGxhY2U6IG1vZG
 lmaWVyc05hbWUKbW9kaWZpZXJzTmFtZTogY249RGlyZWN0b3J5IE1hbmFnZXIsY249Um9vdCBETnMsY
 249Y29uZmlnCi0KcmVwbGFjZTogbW9kaWZ5VGltZXN0YW1wCm1vZGlmeVRpbWVzdGFtcDogMjAxMTA1
 MTAxMTUwMTZaCi0K
changeType: modify
changeTime: 20110510115016Z
objectClass: top
objectClass: changeLogEntry

Note: the search above uses the “Cookie Control” which is the optimized way to search the External Change Log. The value “;” means that the “cookie” is unknown, and therefore the search starts from the first change. If you want to continue from the last change received, provide the string value that is either in the ChangeLogCookie operational attribute (returned if asked for) or the comment before the change itself.

More OpenDJ Tips…

I’ve already mentioned that Mark Craig has joined ForgeRock and started to blog about OpenDJ.

Here’s a few tips he’s recently posted about OpenDJ :

• Start OpenDJ at boot time on Linux
• JSPWiki/Tomcat using LDAP auth
• LDAP Account Manager and OpenDJ
• XWiki LDAP authentication with OpenDJ
• …

I’m sure there are more to come.

If you’re not following Mark’s blog feed yet, you should !

Directory Administrative Accounts – cont.

In a previous post, I’ve explained how to create multiple administrative accounts in the OpenDJ directory service. Today we’re going to look at restricting what applications can do with these administrative accounts.

In the OpenDJ directory service, there are 2 types of authorization systems :

• Privileges control who can perform which administrative tasks : backup, restore, stop and restart of the server, managing acl…
• Access Controls Lists govern the access to the data through LDAP operations.

Most operations involving sensitive or administrative data require that the user has both the privilege and authorization. This allows finer-grain authorization for specific data related action such as managing acl or reseting passwords.

The Privilege SubSystem

Privileges are assigned to users and apply globally to the directory service. Any user can be granted or denied any privilege and by default only the RootDN users are assigned a default set of privileges.

That set of privileges assigned to RootDN users is defined by the “default-root-privilege-name” property, which can be listed or modified using the dsconfig command.

To list the current default privileges assigned to all RootDN users :

bin/dsconfig -h localhost -p 4444 -X -D “cn=directory manager” -j /var/tmp/dmpassfile -n get-root-dn-prop

Property                    : Value(s)
----------------------------:--------------------------------------------------
default-root-privilege-name : backend-backup, backend-restore, bypass-acl,
                            : bypass-lockdown, cancel-request, config-read,
                            : config-write, disconnect-client, ldif-export,
                            : ldif-import, modify-acl, password-reset,
                            : privilege-change, server-lockdown,
                            : server-restart, server-shutdown, subentry-write,
                            : unindexed-search, update-schema

To make sure that all searches are done with proper indexes, you may want to remove the privilege to all Administrative Accounts to perform unindexed searches :

bin/dsconfig -h localhost -p 4444 -X -D “cn=directory manager” -j /var/tmp/dmpassfile -n set-root-dn-prop –remove default-root-privilege-name:unindexed-search

Note: Removing this privilege to all Administrative accounts including the default “cn=Directory Manager” may have side effects for certain internal operations such as group membership, referential integrity…

Whenever adding a new Administrative Account under the “cn=Root DNs,cn=config” container, it automatically inherits from those privileges. But each administrative account can then be denied or added specific privileges by adding values of the “ds-privilege-name” attribute directly in the user entry, in the form of the privilege name or a Minus sign followed by the privilege name.

For example, I can make sure that my newly added Administrative Account is subject to access controls and cannot modify access controls lists, I modify the entry as followed :

Create a temporary file modAdminPrivileges.ldif with the following LDIF modification statement:

dn: cn=Second Admin,cn=Root DNs,cn=config
changetype: modify
add: ds-privilege-name
ds-privilege-name: -bypass-acl
ds-privilege-name: -modify-acl
-

And then apply it to the directory service with the following command :

bin/ldapmodify -h localhost -p 4444 -X -Z -D “cn=directory manager” -j /var/tmp/dmpassfile -f modAdminPrivileges.ldif

Using similar commands and different privileges, one could completely separate the administrative tasks and restrict each Administrative Account to its specific tasks.

But the great thing about privileges is that they can also be granted to regular users part of the Directory Information Tree, allowing them to become administrators for very specific tasks.

The complete list of privileges supported in OpenDJ 2.4 is below:

• backend-restore: Ability to perform backend restore operations.
• bypass-acl: Ability to bypass access control evaluation.
• bypass-lockdown:Ability to bypass server lockdown mode.
• cancel-request: Ability to cancel arbitrary client requests.
• config-read: Ability to read the server configuration.
• config-write: Ability to update the server configuration.
• data-sync: Ability to participate in a data synchronization environment.
• disconnect-client: Ability to terminate arbitrary client connections.
• jmx-notify: Ability to subscribe to JMX notifications.
• jmx-read: Ability to perform read operations via JMX.
• jmx-write: Ability to perform write operations via JMX.
• ldif-export: Ability to perform LDIF export operations.
• ldif-import: Ability to perform LDIF import operations.
• modify-acl: Ability to modify access control rules.
• password-reset: Ability to reset user passwords.
• privilege-change: Ability to change the set of privileges for a user, or to change the set of privileges automatically assigned to a root user.
• proxied-auth: Ability to perform proxied authorization or request an alternate authorization identity.
• server-lockdown: Ability to lockdown a server.
• server-restart: Ability to request a server restart.
• server-shutdown: Ability to request a server shutdown.
• subentry-write: Ability to perform write operations on LDAP subentries.
• unindexed-search: Ability to perform an unindexed search
• update-schema: Ability to update the server schema.

Multiple Directory Administrative Users

Most of LDAP directory servers configure a single well known directory administrative account (cn=Directory Manager [,dc=example,dc=com]) which has full access to everything. While there is a need to have one special user to bootstrap the server, we are too often seeing that special account being used by all applications that have specific administrative needs : the provisioning application, the email management application, …

OpenDJ has different mechanisms to define multiple administrative accounts, but today, I’m going to focus on the “Root DNs” i.e. defining multiple Directory Managers.

The default administrative account is “cn=Directory Manager”, and is stored in the configuration under the “cn=Root DNs,cn=config” container entry.

Adding another administrative account is as simple as adding another entry under that container, with one specific objectClass : ds-cfg-root-dn-user.

Create a file newAdmin.ldif

dn: cn=Second Admin,cn=Root DNs,cn=config
cn: Second Admin
objectclass: top
objectclass: person
objectclass: organizationalPerson
objectclass: inetOrgPerson
objectclass: ds-cfg-root-dn-user
sn: Second Admin
ds-cfg-alternate-bind-dn: cn=Admin2,dc=example,dc=com
ds-pwp-password-policy-dn: cn=Root Password Policy,cn=Password Policies,cn=config
userPassword: password42

ldapmodify -a -D cn=Directory Manager -j /var/tmp/dmpassfile -f  newAdmin.ldif

Processing ADD request for cn=Second Admin,cn=Root DNs,cn=config
ADD operation successful for DN cn=Second Admin,cn=Root DNs,cn=config

If you prefer, you can choose not to set the password in the LDIF file, but set it in a secure way afterwards :

$ bin/ldappasswordmodify -p 1389 -D “cn=directory manager” -j /var/tmp/dmpassfile -a “”cn=Admin2,dc=example,dc=com” -N /var/tmp/newpw
The LDAP password modify operation was successful

Where /var/tmp/dmpassfile contains the password for “cn=directory manager” and /var/tmp/newpw the new password for Admin2.

Did you notice the “ds-cfg-alternate-bind-dn” attribute in the definition of the new administrative account ? This enables to authenticate to the directory server with the DN value specified in this attribute, while the entry still has a DN and is located under the “cn=config” suffix.

So now, don’t hesitate to create different administrative accounts for the various applications that need special access to the directory.

In a follow-up post, I will explain how to restrict what those administrative accounts can do in the OpenDJ directory service.