Monthly Archives: November 2011

LDAP Fiasco

Last week I logged into the ticket management system to look at updates to cases overnight and saw a pop-up for a “widespread issue”, basically two school, involving LDAP. So I looked up the case. The two schools were on the same cluster. Most likely the problem was on my end, which sucks.

Security people like to change firewall settings to block us. At least that is the most likely fault for these LDAP stopped working cases. From one and then all the nodes for their cluster I could connect to both school’s LDAP servers. So not the firewall.

Next most likely fault was the IP changed or failure in load balancing the LDAP servers. Would two completely unrelated institutions change their LDAP server IPs at the same time? Or have load balancing issues? Both of these seemed unlikely.

So I checked the webct.log on a node and found an error I had never previously seen that there were 50 open LDAP connections so it could not open a new one. Cases were there for both schools. Having never seen the error before, my best guess was the something happened to cause connections to open but not close. A good client would close after less than 300 seconds and ideally less than 30.

So I restarted the node. Users started logging in fine. So I kicked off a rolling script to restart all the nodes one-by-one in the background for no end user impact. It took about an hour to restart all ten. Everyone was happy that resolved the issue.

In retrospect a strategy I might have used could have been to narrow the available nodes to users to those which had already restarted. It took about 6 minutes for each to restart and the restarts happened one by one. The first four also run chat. The next six where regular nodes. Blocking access to this latter set would force users to the first ones and additional nodes as they were fixed. Time to users definitely getting a good node could have dropped to 18 minutes rather an hour. Though… With enough users, this could have overwhelmed the cluster. Before 9:30am, there was not enough traffic to overwhelm three nodes. Maybe better safe though.

from Rants, Raves, and Rhetoric v4

Congressional Character

I was looking for a quote on why public education is important to democracy when I ran across this in Quotes on the Importance of Voting.

Now more than ever the people are responsible for the character of their Congress. If that body be ignorant, reckless, and corrupt, it is because the people tolerate ignorance, recklessness, and corruption. — James Garfield, “A Century of Congress” published in Atlantic, July 1877.

Some others:

The man who reads nothing at all is better educated than the man who reads nothing but newspapers. — Thomas Jefferson

Nearly all men can stand adversity, but if you want to test a man’s character, give him power. — Abraham Lincoln

The one I sought:

Whenever the people are well-informed, they can be trusted with their own government.  — Thomas Jefferson

from Rants, Raves, and Rhetoric v4

TED Talks: Trust, morality – and oxytocin

I heard about “eight hugs a day” months ago. I have brought it up in conversation a dozen times since. Glad the video is finally out.

Where does morality come from — physically, in the brain? In this talk neuroeconomist Paul Zak shows why he believes oxytocin (he calls it “the moral molecule”) is responsible for trust, empathy, and other feelings that help build a stable society.

Some interesting points:

    • Countries with high numbers of trustworthy people are more prosperous.
    • In an experiment, the more money a person received (trust), the more money they would voluntarily return (trustworthiness). Oxytocin increased with trust.
    • Massage, dancing, praying increase oxytocin.
    • A con works by schemer demonstrating he or she trusts the victim which produces trustworthiness.
    • Trust key to society and species survival.
    • Using social media produces increases in oxytocin.
    • Give eight hugs a day to make yourself happier.

If the below video does not work, then click Trust, morality – and oxytocin.

http://video.ted.com/assets/player/swf/EmbedPlayer.swf

from Rants, Raves, and Rhetoric v4