Hadoop Troubleshooting: Inconsitent storage detected, name and edits storage do not match

Encountered a error that prevented hadoop from starting. The namenode log contained the following:

/opt/hadoop-0.20.2/logs/hadoop-root-namenode-*.log

2010-05-12 13:42:16,406 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ...
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
...
2010-05-12 13:42:16,690 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean
2010-05-12 13:42:16,737 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.IOException: Inconsitent storage detected, name and edits storage do not match
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:794)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:292)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
2010-05-12 13:42:16,739 INFO org.apache.hadoop.ipc.Server: Stopping server on 9000
2010-05-12 13:42:16,741 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: Inconsitent storage detected, name and edits storage do not match
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:794)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:364)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:292)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:201)
at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
2010-05-12 13:42:16,742 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:

I noticed the timestamps were different between the following name and edits areas:

# ls -l /hadoop/dfs/edits/current/
total 12
-rw-r--r-- 1 root root 4 May 11 17:17 edits
-rw-r--r-- 1 root root 8 May 11 16:17 fstime
-rw-r--r-- 1 root root 101 May 11 16:17 VERSION
# ls -l /hadoop/dfs/name/current/
total 332
-rw-r--r-- 1 root root 326458 May 11 17:17 fsimage
-rw-r--r-- 1 root root 8 May 11 17:17 fstime
-rw-r--r-- 1 root root 101 May 11 17:17 VERSION
#

So I copied the latest version to the other and then hadoop started fine.

# cp /hadoop/dfs/name/current/fstime /hadoop/dfs/edits/current/
cp: overwrite `/hadoop/dfs/edits/current/fstime'? y
# cp /hadoop/dfs/name/current/VERSION /hadoop/dfs/edits/current/
cp: overwrite `/hadoop/dfs/edits/current/VERSION'? y
#

I wish I would have seen what the actual difference was but I was likely some timestamp of some sort possibly due to a unclean shutdown of the node itself.

Love Sonnet LVI

Sweet love, renew thy force; be it not said
Thy edge should blunter be than appetite,
Which but to-day by feeding is allay'd,
To-morrow sharpened in his former might:
So, love, be thou, although to-day thou fill
Thy hungry eyes, even till they wink with fulness,
To-morrow see again, and do not kill
The spirit of love, with a perpetual dulness.
Let this sad interim like the ocean be
Which parts the shore, where two contracted new
Come daily to the banks, that when they see
Return of love, more blest may be the view;
    Or call it winter, which being full of care,
    Makes summer's welcome, thrice more wished, more rare.

(Shakespeare, William. Sonnet 56)

Love Sonnet II

When forty winters shall besiege thy brow,
And dig deep trenches in thy beauty's field,
Thy youth's proud livery so gazed on now,
Will be a totter'd weed of small worth held:
Then being asked, where all thy beauty lies,
Where all the treasure of thy lusty days;
To say, within thine own deep sunken eyes,
Were an all-eating shame, and thriftless praise.
How much more praise deserv'd thy beauty's use,
If thou couldst answer 'This fair child of mine
Shall sum my count, and make my old excuse,'
Proving his beauty by succession thine!
    This were to be new made when thou art old,
    And see thy blood warm when thou feel'st it cold.

(Shakespeare, William. Sonnet 2)

A New Baby

My wife, Ellie, and I just had our first baby: a beautiful baby girl we named Lara. (See our pics). Lara is about 2 weeks old, and she changed my life.

Having a child changes you. For myself, I first noticed the change when my wife became pregnent. I have many character flaws and insensitivites for sure, but starting from when I knew my wife was pregnant and as she continue to grow, I felt a growing gentleness for this women caring my child. I no longer tolerated in myself the act of winning an arguement for the sake of winning or of having the last word. Her peace was paramount. And her requests and honey-do's were no longer interruptions. Yes, I am lazy at heart, and yes I do sometimes grudgingly perform some task especially when it takes away from some other self-indulging activity, but I reveal my irritation much less (I hope) and really do have a growing pleasure in pleasing and meeting her needs, knowing that what I am doing helps lighten the heavy load she daily carries. She underwent so much discomfort during the pregnancy, and even now she takes more than her share of the burden with the baby. I wish I could do more, but I'm just not equipped to breast feed Lara. Thank God! :-)

My wife is an amazing person. When Lara came home from the hospital, my wife voluntarily moved into the guest room to be with the new baby, leaving me to sleep in peace in our bedroom. Can you believe that! What selfless devotion to her husband. That act alone has save me countless sleepless hours. If that wasn't enough, she happily performs tasks that are low but necessary, like changing the diapers. I had to make a request one day so I wouldn't feel guilty about not having to change Lara's diaper yet.

These and many more things she does without complaint, and most wonderful of all, she gave life to our healthy baby girl. That in itself is an act of incredible beauty and sacrifice on her behalf: to go through those tough 9 months of constant nausea, to have her body grow and become more uncooperative, and to finally have a 24+ hour long delivery with was extremely painful for her. Amazing this women is! Grateful am I.

You know I started this post, wanting to talk about Lara. But I turned out to be a devotion to my wife. This is good. I was proud when Mother's day come. Lara was born on May 3, and so Mother's day had fresh significance to me. I was glad to celebrate my wife - the mother of my child. She is an amazing mother to Lara and wife to me. And I hope I will celebrate it along with all special occasions with them both for years to come.

But I would like finish the thought I started on at first and say that fatherhood (parenthood) not only changed me, but made me more complete complete. It has made my human experience more complete. Without it, my life would definitely be less. Ellie and I were tempted to place our careers first with the hopes of starting a family second. That was a mistake we fortunately avoided. If I was a noble man, I would have adopted, for there are so many children without a family. But I am thankful for what God gave us. Life's too short not to love another - especially a baby.

Grad School and the Java Hurdle

Resuming classes now at Johns Hopkins University (JHU) Part-Time Engineering Graduate Program for Computer Science. I've been off for more than a year, so it will be hard to get back into the swing of things for this upcoming summer semester.

To compound the stressful challenge, many of the courses are now requiring Java for their software programming assignments projects, and I'd rather not take the Java course JHU offers because it doesn't count towards my degree. It would just be intolerable to spend my summer taking a class that didn't count. Then again, it would be worse if I failed my required class because I lacked sufficient Java knowledge. And though I have obsessive perfectionistic tendencies, I am also very lazy, so the odds of me self-learning Java are not high. But you know, writing this little factoid pisses me off and is actually highly motivating! (I guess my self-help blogging strategy is paying off :-)

Fortunately, I was not without some motivation earlier and I've been going through the Java textbook the summer Java instructor is using in preparation for the other class I intend to take. Out of the 1000+ page textbook I'm on page 140. Ugh. Still got a long way to go, since the first few chapters were easy stuff general to most programming languages. But the pressuer is on. My other class starts in 3 weeks! Since I'm no longer going to take the Java class, I wish now I would have bought the Java 1.5.0 version of the textbook instead of the older 1.4.2 API edition. Oh well. I'm using the 1.5 java compiler and hitting Sun's API documentation so I guess that will be sufficient supplement if the assignments require 1.5.0 techniques.

I started to use the Eclipse IDE, which offers cool autocompletion features, but it was slowing me down way to much, so I ended at the command line. I'm usually working in some interpreted language like PERL/PHP/BASH so I had to make myself a simple one button 'javarun' routine to retain sanity while I beat my head against the Java syntax/class wall. I need to go place ice on my head and go to bed, but before I go, here are some Java links I found useful.

• Java 1.5.0 API
• Environment Varabiles
• Java 1.5.0 Tutorial

Fifty Major Philosophers - part 1

Mediations on the "Fifty Major Philosophers" by Diane Collinson.

What is the world in which we live? This question was raised by the earliest of historically great thinkers on many levels. For Thales, Anaximander, and Anaximenes, Heraclitius (circa 650-500 BC) what the physical substance of the material world was of interest and each proposed it to be water, aperion, air, and fire, respectively. (Aperion being a indefinite, spatially infinite, all-encompasing substance producing balanced motion between all cosmic opposites like light and dark, hot and cold.)

Like a baby first discovering the world, life is first experienced through the physical world and so naturally the philosphers are remembered for humanities first baby step or question, asking "What is the world physically made of?". This is a question one arrives at first when searching for the answer to "What or 'who' am I?". Answer that, and questions like, "Why am I here? What is my purpose or destiny in life? Where did I come from and where am I going?" are more accurately answered.

Pythagoras, best known for his geometrical theorums, believed the principle behind all things was not a elemental substance but rather numbers, where both the cosmological matter and meaning were expressed. The essence of things being numeric is highly debatable, but understanding everything could be expressed in mathematical terms, I personally think, was one of the greatest monumental ideas human history. Today, so much transformation of our world is achieved through mathematics, and at hyper speeds through the use of computers, it useful to know how to define both physical or mental objects in a formal system for further learning.

With newly aquired knowledge, one risks misinterpretation or worse misapplication. In comes Zeno to capture these errors in a set of paradoxes that often appear between the passage of one's physical experiences and human reasoning. With the fact that the measurement of time is infinitely divisible and yet a segment of time is finite, Zeno wonders how motion is possible and he illustrates this paradox in the story of Achilles and the tortoise. Here, Zeno grapples between time, space and human perception. How can Achilles, known for speed, ever surpass the tortoise whose been given a head start if at all times during the race, the lapse of time between him and the tortoise can be divided infinitely. When the problem is only expressed in time, it appears at every point in time, the tortoise will always be ahead since there are an infinite number of time segements dividing them from each other. Of course you don't need differential calculus to understand the passage of distance during time, but with thoughts concerning more unfamiliar, transient, realms of knowledge about the external world in which we live, especially when related to eternal questions, our minds can easily leave us perplexed or worse misguided.

Keeping in Touch

I've always wanted to keep better in touch with friends and family.