Archive for July, 2009
I’m taking some time off the network with my family. I expect to be back online end by August 24.
Enjoy the silence…
Technorati Tags: blog
The agenda for the 2nd International LDAP Conference has been published, and the conference really looks very attractive to me (sure I’ll be presenting and thus I’m a little bit biased ). I can’t wait to be in Portland OR, meet with all the LDAP experts from the various open source projects or directory services vendors and have those lively discussions.
I hope to see you there.
Back in March, in the series of OpenDS tips, I had written an article and a script to help converting schema files for OpenLDAP to a format suitable for OpenDS.
I received multiple suggestions for the scripts as well as sample schema files that would not load in OpenDS after being converted. Most of the time, it was due to the lack of respect of the RFC syntaxes.
Another year has passed and we already end of July. Today is the anniversary day for the OpenDS project which is turning 3 this year.
As usual, this is also time to look back in the mirror and consider what we’ve achieved.
A little more than 10 days ago, we announced the availability of OpenDS 2.0, the new and stable release of our LDAPv3 directory server. OpenDS 2.0 came just about one year after OpenDS 1.0 and 6 months after OpenDS 1.2.
You can read about OpenDS 2.0 features in the Release Notes, but also in the various articles that have relayed our own announcement such as:
- Ben Rockwood: OpenDS 2.0 Released, also on TuxWire: the Linux blog.
- Java.net Editor’s Daily Blog: OpenDS 2.0.0 Provides High-Performance Directory Services
- Marina Sum: OpenDS 2.0 ships, New Features in OpenDS 2.0
Yesterday, Sun publicly announced the general availability of Sun OpenDS Standard Edition 2.0, a Sun supported version of the OpenDS project, as well as the release of OpenSSO Express Build 8 (due in a couple of weeks).
Sun OpenDS Standard Edition 2.0 has the same features as OpenDS 2.0. Differences are in the branding, the license, the documentation that is available from docs.sun.com in HTML and PDF and of course the support services offered by Sun.
Mark Craig has already posted an illustrated article describing how easy it was to install Sun OpenDS Standard Edition 2.0 on Windows XP.
OpenSSO Express builds are supported snapshots of OpenSSO development. As Pat Patterson, Community Manager for OpenSSO and covering all Identity Products at Sun, detailed on his blog, OpenSSO Express Build 8 includes a new Mobile One Time Password Feature, the Fedlet for .Net and a new task flow enabling single sign-on to Salesforce.com.
As OpenDS is getting mature, we’re seeing public endorsement and use of it. In the last couple of weeks, we had 2 success stories including the use of OpenDS :
- Gluu, a San Antonio TX based startup, offers Cloub base Identity Services based on OpenDS server and DSML gateway.
- Symeos, a French company, offers an Online Identity Management platform based on Sun open source solutions including Sun OpenDS Standard Edition.
Finaly within a year, the OpenDS Community has more than doubled, in term of members in the community, but as well in the number of active contributors and participants in the #opends IRC channel, and in term of unique visitors on the www.OpenDS.org.
I’m proud of what we’ve accomplished in 3 years and even more of the past year. We still have a lot of ideas and customers requirements to build in the OpenDS project. Overall we know where we want to go and we hope our new executives will agree that it’s a nice and viable path to follow…
There is an issue in the start and stop scripts that is preventing OpenDS 2.0 to be installed via Java Web Start on Mac OS X 10.5 with the latest version of the JVM (Update 4 a.k.a 1.6.0_13). I’ve discovered the problem at the same time we were releasing OpenDS 2.0.0 release candidate 4 which was planned to be the last release candidate. So the fix is not the release but has been committed to the trunk.
The issue is that the new JVM does use a larger default minimal heap size and reject any calls with -Xmx if the maximum heap size is smaller than its internal default (around 30MB).
Still OpenDS 2.0 can be installed on Mac OS X and used with the latest JVM, by downloading the Zip file, unzipping it and doing minor edition in the start-ds and stop-ds scripts.
$ unzip ~/Desktop/OpenDS-2.0.0.zip
$ cd OpenDS-2.0.0/bin
In the start-ds and the stop-ds scripts, replace all occurences of the string "-Xms8M -Xmx8M" with "-client"
$ cp start-ds start-ds.orig
$ sed -e "s/-Xms8M -Xmx8M/-client/g" < start-ds.orig > start-ds
$ cp stop-ds stop-ds.orig
$ sed -e "s/-Xms8M -Xmx8M/-client/g" < stop-ds.orig > stop-ds
You can now run the setup program (or launch the QuickSetup application) to install and configure the OpenDS directory server.
I’m mostly done with the series of post concerning the new features of the latest release of OpenDS, the opensource LDAPv3 directory service. Yesterday, Mathieu, the developer behind Assured Replication, reminded me of a small but important new feature of OpenDS, in the area of replication: the ability to configure Replication Groups.
A replication group is a simple way to relate replicated OpenDS directory servers together. It’s useful when there are more than 2 replicated servers, when the replicated servers are within different data-centers or to distinguished primary servers from secondary servers.
Replications groups are identified by a group ID which is a unique number assigned to a replication domain on a directory server and to replication servers. Group IDs determine how a directory server domain connects to an available replication server. From the list of configured replication servers, a directory server first tries to connect to a replication server that has the same group ID as that of the directory server. If no replication server with a compatible group ID is available, the directory server connects to a replication server with a different group ID.
In practice, it allows to prioritize how the replication traffic is done between the servers. In the case of multiple data-centers, it’s preferable that all directory servers in a data-center are connected to replication servers in the same data-center. Only in the case of absence of a local replication server, a directory server will connect to a remote replication server.
Note that when configuring replication with OpenDS 2.0 and the dsreplication utility, both the replication server and the directory server are configured in the same process and thus the same host. It will be very rare if the replication server is not working for its directory server.
The figure below is an illustration of 2 Replication Groups, one for each data center.
Now to configure a replication group ?
A replication group is configured on each directory server and replication server that should be part of the same group.
On the directory server, the replication group is configured per replication domain (i.e. per replicated suffix).
First identify the replication domain
$ bin/dsconfig -D “cn=directory manager” -j /tmp/passwdfile -n -s list-replication-domains –provider-name “Multimaster Synchronization”
cn=admin data (domain 29167)
cn=schema (domain 9674)
dc=example,dc=com (domain 14741)
Then set the group ID for the domain
$ bin/dsconfig -D “cn=directory manager” -j /tmp/passwdfile -n set-replication-domain-prop –provider-name “Multimaster Synchronization” –domain-name “dc=example,dc=com (domain 14741)” –advanced –set group-id:5
For the replication server
$ bin/dsconfig -D “cn=directory manager” -j /tmp/passwdfile -n set-replication-server-prop –provider-name “Multimaster Synchronization” –advanced –set group-id:5
Repeat this to the other directory servers and replication servers that should be part of the same group.
Note that there is a group by default with the group ID 1.
Configuring replication groups have some impact when using Assured Replication, since Assured Replication only works within a single group. So groups can be used to limit the impact of network latency when using Assured Replication, or to constrain the changes to be more consistent in a single data-center.
To me, the biggest innovation in this release is "Assured Replication", an extension to the loose consistency multi-master replication feature that brings tighter consistency of data between replica. "Assured Replication" is not to be taken for a full synchronous and transactional replication mechanism. A change is not transactionally applied to a set of or all replicas of a topology. With "Assured Replication", the response to an LDAP modification is delayed until the change has been received or applied by other servers, in a best effort mode. It provides a greater assurance that a change is not lost even if the server receiving it crashes.
Assured Replication can function in 2 modes :
- Safe Data Mode: an update must be propagated to a defined number of Replication Servers before returning a response to the client. So if the server or the replication server is stopped, the data is still available to all other replicas.
- Safe Read Mode: an update must be propagated to all directory servers in the domain before the client is returned a response for the update.
Of course, for both modes, it’s possible to configure a timeout interval to prevent LDAP clients to be waiting indefinitely if some servers are not available.
Configuring Assured Replication is pretty straightforward but cannot be done when setting up replication itself. So the first step is to configure Multi-Master Replication for a domain with dsreplication.
$ bin/dsreplication enable –host1 localhost –port1 5444 –bindDN1 ‘cn=directory manager’ –bindPassword1 secret12 –replicationPort1 8989 –host2 localhost –port2 6444 –bindDN2 ‘cn=directory manager’ –bindPassword2 secret12 –replicationPort2 8990 –adminUID admin –adminPassword secret12 –baseDN "dc=example,dc=com" -X -n
Establishing connections ….. Done.
Checking Registration information ….. Done.
Configuring Replication port on server localhost:5444 ….. Done.
Configuring Replication port on server localhost:6444 ….. Done.
Updating replication configuration for baseDN dc=example,dc=com on server localhost:5444 ….. Done.
Updating replication configuration for baseDN dc=example,dc=com on server localhost:6444 ….. Done.
Updating Registration configuration on server localhost:5444 ….. Done.
Updating Registration configuration on server localhost:6444 ….. Done.
Updating replication configuration for baseDN cn=schema on server localhost:5444 ….. Done.
Updating replication configuration for baseDN cn=schema on server localhost:6444 ….. Done.
Initializing Registration information on server localhost:6444 with the contents of server localhost:5444 ….. Done.
Initializing schema on server localhost:6444 with the contents of server localhost:5444 ….. Done.
Replication has been successfully enabled. Note that for replication to work you must initialize the contents of the base DN’s that are being replicated (use dsreplication initialize to do so).
$ bin/dsreplication initialize –baseDN "dc=example,dc=com" –adminUID admin –adminPassword secret12 –hostSource localhost –portSource 5444 –hostDestination localhost –portDestination 6444 -X -n
Initializing base DN dc=example,dc=com with the contents from localhost:5444:
23 entries processed (100 % complete).
Base DN initialized successfully.
for a detailed log of this operation.
$ bin/dsreplication status -h localhost -p 5444 –adminUID admin –adminPassword secret12 -X
dc=example,dc=com – Replication Enabled
Server : Entries : M.C. (1) : A.O.M.C. (2) : Port (3) : Security (4)
localhost:5444 : 23 : 0 : N/A : 8989 : Disabled
localhost:6444 : 23 : 0 : N/A : 8990 : Disabled
Now that replication is setup, we can enable the Assured Replication mode, using the dsconfig utility. For this, on each of the OpenDS direcotry servers, we first need to retrieve the full name of the replication domain.
$ bin/dsconfig -D cn=directory\ manager -w secret12 -n -s list-replication-domains –provider-name "Multimaster Synchronization"
cn=admin data (domain 29167)
cn=schema (domain 9674)
dc=example,dc=com (domain 14741)
$ bin/dsconfig -D cn=directory\ manager -w secret12 -n set-replication-domain-prop –provider-name "Multimaster Synchronization" –domain-name "dc=example,dc=com (domain 14741)" –advanced –set assured-type:safe-data –set assured-sd-level:2
Note that the Replication Domain has a different value on each server, so you have to repeat these 2 commands on each instance.
Setting the assured level for Safe Data to 2 means that the server will make sure the data has been received by at least 2 replication services before returning to the LDAP client the response to the update request.
From a client point of view, there should be no difference, except that the server might take a little longer to return the response to an update request. In our measures, we found that the response time increased by 25% for Safe Data Level 2, which seems a lot, but honestly, when the response time is in the order of 2ms, it’s hard to notice !
You can find more information about Assured Replication on OpenDS 2.0 documentation wiki, both in the overview of OpenDS Replication Architecture and the Replication Administration Guide, and more specifically Assured Replication Administration Guide
OpenDS 2.0 has just been released and there are several new and exciting features in it.
Today we will be taking a closer look at the I18N Collation Matching Rules.
In LDAP, most of the data is made of DirectoryStrings which are UTF-8 encoded strings. LDAPv3 specifications and more precisely RFC 4518, defines the way to prepare UTF-8 strings to be compared in LDAP and OpenDS being fully compliant with LDAPv3 implements this RFC.
This means that the server will properly case-fold non-ascii characters and be able to compare properly and in a case insensitive way, none ascii characters like the French é or Japanese characters.
Let’s work with an example, an entry with the givenName “Hélène” illustrated on the right (click on the image for a larger view).
If I search the directory for that givenname, I can retrieve the entry:
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname=hélène)’
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname=HÉLNE)’ givenName
Note : DN, CN and GivenName are base64 encoded in the result as expected per LDIF Specifications.
Note : To be able to type in correctly the string “hélène” in a terminal (like to the filters above), make sure the LANG is set to use UTF-8 encoded characters (LANG=en_US.UTF-8).
Let’s see what happens if I’m searching for the same user without the accentuated letters.
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname=helene)’
Nothing returned. This is because in Unicode the letter e and é do not normalize the same. Now this is a big problem, especially in Europe because we do not like when our name is not written how it should be and also because the person that searches may not remember exactly how to spell the name or may not know how to type the composed character on his machine. Also in French (but with other locales as well), the letters e and é and É are considered comparing equal.
That’s where the I18N Collation Matching Rules come to the rescue.
OpenDS 2.0, like his far ancestor Sun Directory Server, supports by default a set of extensible matching rules that are locale specific.
This means I can now search for the GivenName according to the Collation Rules associated with French or German or Norvegian or Japanese.
Each locale has been assigned an OID and then there are 6 different matching rules per locale : LowerOrEqual, LowerThan, Equality,GreaterOrEqual, GreaterThan , Substring.
So if one would like to match givenname for equality according to the French collation rules, the filter would be the following: (givenname:126.96.36.199.188.8.131.52.184.108.40.206.1.3:=Helene)
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname:220.127.116.11.18.104.22.168.22.214.171.124.1.3:=helene)’ givenName
Or for a substring match, still according to the French collation rules:
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname:126.96.36.199.188.8.131.52.184.108.40.206.1.6:=hel*)’ givenName
But remembering OID for each locale and type of matching is not easy. So we’ve also provided some shortcuts in the form of the locale name and a short string representing the different matching; lte, lt, eq, gte, gt, sub
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname:fr.eq:=helene)’ givenName
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname:fr.sub:=hel*)’ givenName
$ bin/ldapsearch -p 2389 -b “dc=example,dc=com” ‘(givenname:de.eq:=helene)’ givenName
So not only those I18N Collation Matching Rules can be used in Search filters to search, but they can be used for indexing as well, and also for server side sorting.
Unfortunately, setting extensible matching rules for indexes is not possible from the ControlPanel. So it has to be done with dsconfig.
$ dsconfig set-local-db-index-prop \
–backend-name userRoot \
–index-name givenName \
–add index-extensible-matching-rule:fr.eq \
–hostname ludovic-poitous-computer-2.local \
–port 5444 \
–trustStorePath /Users/ludo/dev/Tests/OpenDS2rc4/config/admin-truststore \
–bindDN cn=Directory\ Manager \
–bindPassword ****** \
Don’t forget to rebuild the index for the givenName attribute (bin/rebuild-index -b dc=example,dc=com -i givenname).
You can find more information about the I18N Collation Matching Rules on the OpenDS 2.0 Documentation Wiki.
Today we will look at the new feature in the Control Panel: the monitoring aspect.
At first look when we compare the Control Panel for OpenDS 1.2 (on the left below) and the one for OpenDS 2.0 (on the right below), not much has changed. Except for the left bar menu that contains a new section: Monitoring and a status bar at the bottom of the panel.
Let’s look a little more in details what kind of monitoring information for the OpenDS server is now available:
In the General Information Panel, you will find overall statistics such Connections, Up Time, but also information related to the system like the name, OS, number of CPUs, available memory in the JVM. You will also have a screen dedicated to the Java VM specific information (see below) such as the version, the vendor, the running arguments, a lot of details about the various segments of memory of the VM. A third screen highlights the work load. The fourth one gives details about the entry cache and the fifth one gives details about the local backend database environment, i.e. low level statistics on memory, transactions, locks in the database. There are more than 50 parameters from the database that you can retrieve and look at with the “Show Operations…” button.
The Connection Handler.
The Monitoring Panel for the Connection Handler gives greater details about the LDAP operations that are processed by the server, sorted by kind of Connection: what goes on the LDAP port, what happens on the LDAPS port and the Administrative operations.
There are many more statistics information available from the OpenDS server, either through LDAP, JMX or SNMP if you have enabled the later two. Not everything can be displayed in the GUI, and in future version we will provide external tools or widget to graph some of the monitoring information. Meanwhile, play with the Monitoring screens of the OpenDS 2.0 Control Panel, and let us know what you think, what you like, what you feel is missing.
And for a supported version of OpenDS, please check Sun OpenDS Standard Edition 2.0
Yes, the release of OpenDS 2.0 was a big news for all of us in the development team. But we’re really happy that the news is also big on Java.net.
Many thanks to the editors, and particularly to Marina Sum who keeps posting about the OpenDS to her blog, by passion and kindness.
This is a general LDAP tip and it applies to OpenDS the open source LDAP directory service in Java, as well as Sun Directory Server (all versions) and other LDAP servers:
How can I know the number of entries under a specific node of the Directory Information Tree ?
Well it’s simple. Every entry contains an operational attribute that specify the number of immediate subordinates entry : numSubordinates.
So to retrieve the number of entries under a specific node of the DIT, for example in the ou=people,dc=example,dc=com, a simple read is required.
$ bin/ldapsearch -p 3389 -D "cn=directory manager" -w - -b "ou=people,dc=example,dc=com" -s base ‘(objectclass=*)’ numsubordinates
Password for user ‘cn=directory manager’:
This attribute is defined in an expired Internet-Draft but has been well implemented in many servers. There are often some limitations, like the value only counts entries on the same server, but overall it’s a very useful attribute especially when browsing through the DIT.
OpenDS and Sun Directory Server also implements another attribute : hasSubordinates, defined in X.501. hasSubordinates is a boolean and returns "true" or "false" depending on whether the entry is a branch or a leaf in the Directory Information Tree.
OpenDS 2.0 has just been released and there are several new and exciting features on it.
Today we will focus on one simple feature that greatly reduce cost of administration: scheduled tasks.
Being a Directory Server administrator often implies that you have to perform some administrative tasks on a regular basis. One of those tasks for example that an administrator has to do is a backup of the database. With most Directory Servers, the administrator would write a script to be run on a specific time of the day (or rather the night) that would proceed with the backup.
With OpenDS and the Recurrent Tasks, we’ve simplified this to the extreme: Just instruct OpenDS to do a backup on a weekly or daily basis, and as long as the server is running, it will execute the backup procedure at the desired time.
Here’s how to schedule an hourly, compressed backup for the main back-end :
$ bin/backup -p 5444 -D cn=directory\ manager -w secret12 -n userRoot \
-d ./backups -c –recurringTask ’0 * * * *’
Recurring Backup task BackupTask-dc89d98e-4ade-410e-ad19-325279af8f67
Now, just wait for the hour to pass, and check if the backup has been taken
The string passed as a parameter following the –recurringTask option has the same format as for the crontab(5) time/date: a 5 integer pattern field, separated by blank spaces: Minute (0-59), Hour (0-23), Day Of Month (1-31) Month Of Year (1-12) Day Of The Week (0-6 with 0 being Sunday).
The recurrent tasks are not limited to backups. They can be applied to all tasks, although some may not be that useful to everyone. Although I do see some use of a daily import of an LDIF file from a well know location, as a way to synchronize with external sources.
And of course, you can list the scheduled and recurrent tasks with dsconfig and cancel them if needed.
In the next release of OpenDS, you will be able to configure the recurrent tasks with the Control Panel. If you can’t wait, you can try with the latest daily build.
You can find more information on recurrent tasks on the OpenDS Documentation Wiki.
OpenDS is an LDAPv3 compliant Directory Service written entirely in Java. The 2.0 release has many new features since OpenDS 1.0 that was released a year ago:
• A graphical control panel that enables basic server and data administration is available and replaces the OpenDS 1.0 status-panel
• An administration connector manages all administration related traffic to the server. By separating user operations and administration operations, the administration connector ensure a better quality of service and simplify logging and monitotring
• Connections can be secured and encrypted with SASL mechanisms
• Access Control mechanism has been enhanced to control access based on the level of security of the connection
• The ;binary transfert option is now supported
• Standard schema files related to Solaris and OpenSolaris LDAP naming services are provided by default
• Setup and tools provide an enhanced support for the JCEKS keystore and alternate security providers
• A new mode for Multi-Master Replication providing greater consistency and availability of data: Assured Replication
• Recurring tasks allow an administrator to schedule repeated tasks such as backups
• New extensible matching rules and indexing allowing comparing, ordering of data according to specific locales and languages
• Better monitoring information for the server and for Replication
• Full compliance with RFC 4518 and matching of UTF-8 in attributes with a DirectoryString syntax
• VLV indexes are now built during the Import
• Works with IBM JVM (Java 6 SR4 required)
• Works by default with JConsole and VisualVM when JMX Connection Handler is enabled
• Default settings and ergonomics have been improved reducing the need for tuning parts of the server
• Greatly improved performances and stability over time of those performances
• Resolved a possible security issue when Pre-ReadEntry, Post-ReadEntry and Assertion Controls were enabled
OpenDS 2.0.0 is a promotion of OpenDS 2.0.0 Release Candidate 4, built with revision 5492, to the stable and finalized version.
It can be installed with the Java WebStart QuickSetup or downloaded as a Zip file.
A DSML-to-LDAP Gateway is available as a War file.
Like for previous OpenDS releases, a snapshot of the documentation wiki has been setup. The documentation is still being verified and a few links might not be functional yet. We expect it to be finalized by the end of next week.
You can find more information about OpenDS 2.0 in the release notes.
For a supported version of OpenDS, please check the Sun OpenDS Standard Edition 2.0 home page or get it directly from Sun Download Center.
I’d like to address a special thank to our external contributors who have helped making this release a better release, especially Christian Brennsteiner for the German translation of messages, Tosiki Iga for the Japanese translation, D.J Hagberg for the performance enhancements, Andy Wang for the IBM JVM Support.
Thanks also to all users who have raised issues during the development phase, helping us with testing the server in ways we can’t.
This is a major milestone for the OpenDS project, but there is more to come… Make sure you check the Roadmap and you participate to it.
Today is my 14th anniversary at Sun. This is way more than I had anticipated but I can’t say that I’ve been really counting the days or years. I’ve been very fortunate to meet and work with very bright people, to participate in the birth and development of a very successful product line. I’ve done and I’m still doing things that get me excited, passionate, with a really good balance of the technical and the business aspects. Looking back from where we’ve started with Sun Directory Services, I’m amazed and I’m proud by the amount of business we’ve driven for Sun. And still our customers are pushing us to innovate and we’re taking LDAP directory server to levels that a few years ago we would not even consider as possible. So there is still some interesting work ahead and that keeps me going to the office every day with motivation and a steaming brain.
However, I know there will not be a 15th anniversary at Sun, just simply because Sun will no longer be Sun very soon. The page is turning… Let see what the next chapter will be about !
I’ve just received a “certificate of attendance as a Speaker” from the FISL organizers. This is a very nice gesture and adds to the amazing experience that is FISL. Big thanks to the organizers and more specifically Fernanda Weiden who had to cope with the egos of over 320 speakers.