- 浏览: 213735 次
- 性别:
- 来自: 北京
文章分类
- 全部博客 (114)
- hbase (3)
- akka (7)
- hdfs (6)
- mapreduce (1)
- hive (0)
- zookeeper (8)
- storm (0)
- geese (0)
- leaf (0)
- stormbase (0)
- scala (2)
- oozie (11)
- zeromq (1)
- netty (3)
- mongodb (0)
- sqoop (2)
- flume (3)
- mahout (1)
- redis (0)
- lucene (1)
- solr (1)
- ganglia (3)
- 分布式理论 (2)
- hadoop (42)
- others (14)
- mq (1)
- clojure (3)
- flume ng (1)
- linux (1)
- esper (0)
最新评论
-
javalogo:
[b][i][u]引用[list]
[*][*][flash= ...
什么是Flume -
leibnitz:
what are they meanings
Hadoop Ganglia Metric Item -
di1984HIT:
没用过啊。
akka 介绍-Actor 基础 -
di1984HIT:
写的不错。
Hadoop管理-集群维护 -
developerinit:
很好,基本上介绍了
什么是Flume
根据日志:
StandBy NN启动过程
1.获得Active NN Checkpoints信息
2.在内存中,注册Live Nodes
3.SB NN 进入Safe Mode
4.从Datanod获取包信息
5.离开Safe Mode
Checkpointing active NN at bigdata-4:50070
Serving checkpoints at bigdata-3/172.16.206.206:50070
2012-08-02 11:07:24,761 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1124007630-172.16.206.209-50010-1343796008920
2012-08-02 11:07:24,763 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.209:50010
2012-08-02 11:07:24,763 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1236477555-172.16.206.206-50010-1343796211132
2012-08-02 11:07:24,764 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.206:50010
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode extension entered.
The reported blocks 8854 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 29 seconds.
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.209:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:24,996 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 84 msecs
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.206:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 45 msecs
2012-08-02 11:07:44,998 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 8863 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 9 seconds.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 32 secs.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is OFF.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 2 datanodes
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
StandBy NN启动过程
1.获得Active NN Checkpoints信息
2.在内存中,注册Live Nodes
3.SB NN 进入Safe Mode
4.从Datanod获取包信息
5.离开Safe Mode
Checkpointing active NN at bigdata-4:50070
Serving checkpoints at bigdata-3/172.16.206.206:50070
2012-08-02 11:07:24,761 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1124007630-172.16.206.209-50010-1343796008920
2012-08-02 11:07:24,763 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.209:50010
2012-08-02 11:07:24,763 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.registerDatanode: node registration from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0) storage DS-1236477555-172.16.206.206-50010-1343796211132
2012-08-02 11:07:24,764 INFO org.apache.hadoop.net.NetworkTopology: Adding a new node: /default-rack/172.16.206.206:50010
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode extension entered.
The reported blocks 8854 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 29 seconds.
2012-08-02 11:07:24,995 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.209:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:24,996 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.209, storageID=DS-1124007630-172.16.206.209-50010-1343796008920, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 84 msecs
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: BLOCK* processReport: Received first block report from 172.16.206.206:50010 after becoming active. Its block contents are no longer considered stale.
2012-08-02 11:07:25,041 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* processReport: from DatanodeRegistration(172.16.206.206, storageID=DS-1236477555-172.16.206.206-50010-1343796211132, infoPort=50075, ipcPort=50020, storageInfo=lv=-40;cid=CID-d1f20578-a690-4538-a945-c166fd41cda4;nsid=1959114293;c=0), blocks: 12908, processing time: 45 msecs
2012-08-02 11:07:44,998 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON.
The reported blocks 8863 has reached the threshold 0.9990 of total blocks 8863. Safe mode will be turned off automatically in 9 seconds.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 32 secs.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is OFF.
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 2 datanodes
2012-08-02 11:07:54,999 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
发表评论
-
Hadoop TestDFSIO
2013-04-21 21:02 2387@VM [bigdata@bigdata hadoo ... -
Hadoop NNBENCH
2013-04-21 20:46 1592@VM [bigdata@bigdata hadoop]$ ... -
Hadoop 安装手册
2013-04-08 15:47 1158Hadoop 安装手册 软件准备 ... -
What do real life hadoop workloads look like
2012-09-10 15:52 782http://www.cloudera.com/blog/20 ... -
CDH4 HA 切换时间
2012-09-05 15:15 4292blocksize:35M filesize 96M zk-s ... -
CDH4 HA 切换
2012-09-05 10:51 1329HA 切换问题 切换时间太长。。。 copy 0 ... ... -
CDH4 HA test
2012-08-01 14:55 2568场景: NN HA 设置成功,HA切换客户端出现异 ... -
Hadoop TextOutput
2012-07-29 21:08 867TextOutputFormat 分隔符参数: mapredu ... -
Hadoop SteamXMLRecordReader
2012-07-28 23:59 667StreamXmlRecordReader 设置属性 str ... -
Hadoop NLineInputFormat
2012-07-28 23:52 1595NLineInputFormat 重写了splits 设置 ... -
KeyValueTextInputFormat
2012-07-28 23:40 916key/value 分割符 mapreduce.input. ... -
Hadoop 控制split尺寸
2012-07-28 23:08 1284三个参数决定Map的Split尺寸 1.mapred.min ... -
Setting up Disks for Hadoop
2012-07-22 12:13 838Setting up Disks for Hadoop He ... -
Upgrade hadoop need think about it
2012-07-21 17:17 835Compatibility When movin ... -
Hadoop 0.23 config differ from 0.20.205
2012-07-21 17:14 892http://hadoop.apache.org/common ... -
Hadoop hdfs block 状态
2012-07-15 13:37 6861.In Service -
Hadoop 配置不当引起集群不稳
2012-07-05 15:35 980配置不当内容 资源配置不当:内存、文件句柄数量、磁盘空间 ... -
Hadoop管理-集群维护
2012-07-03 15:27 49471.检查HDFS状态 fsck命令 1)f ... -
Hadoop Ganglia Metric Item
2012-06-27 11:13 1985dfs.FSDirectory.files_delete ... -
Hadoop 参数
2012-06-27 10:05 981转发自:http://www.cnblogs.com/g ...
相关推荐
hadoop cdh5安装hadoop cdh5安装hadoop cdh5安装hadoop cdh5安装hadoop cdh5安装
hadoop cdh5 centos 64位系统本地库编译文件 不会替换可以找我帮忙替换
presto-hadoop-cdh4.zip,CDH4 Hadoop for Presto的阴影版本CDH4 Hadoop for Presto的阴影版本
CDH5安装手册:Apache Hadoop的服务的部署比较繁琐,需要手工编辑配置文件、下载依赖包等。Cloudera Manager以GUI的方式的管理CDH集群,提供向导式的安装步骤。由于需要对Hive,HBase,Impala,Spark进行功能测试,就...
HadoopCDH发行版架构
hadoop cdh版本伪分布式集群搭建详细图文教程
之前一直在网上找社区版hadoop的配置,发现CDH安装直接界面搞定,自动生成配置。供检查学习。
Hadoop CDH5.7.0离线安装与暴力卸载(坑集锦)与暴力卸载(坑集锦)
hadoop cdh3u6的版本在eclipse3.6.2下的插件,将jar包放入eclipse安装路径下的plugins文件夹,重启即可。
hadoop cdh 编译所需下载的所有软件,很全的
HadoopHA集群 批量启动脚本HadoopHA集群 批量启动脚本HadoopHA集群 批量启动脚本HadoopHA集群 批量启动脚本
hadoopcdh4较为详细的安装过程,亲自测试可行
简单的基于clouderea的HA部署文档
hadoop-cdh-4.5.0 eclipse plugin hadoop eclipse 插件 hadoop-eclipse-plugin-2.0.0-cdh4.5.0.jar
最近学习hadoop,综述了大家的博客,终于弄出来一篇自己认为比较完成的文档。 centos6.2 mini hadoop-2.0.0-cdh4.3.0.tar.gz zookeeper-3.4.5-cdh4.3.0.tar.gz hbase-0.94.6-cdh4.3.0.tar.gz
Centos 7安装配置Hadoop生态圈CDH5版本 其中包括hadoop hbase hive spark 等组件安装
hadoop 高可用启动脚本,运行启动zookeeper集群和hadoop ha集群
Hadoop2.x HA环境搭建Hadoop2.x HA环境搭建Hadoop2.x HA环境搭建
hadoop-2.6.0-cdh5.14.0 源码 。
hadoop-2.6.0-cdh5.7.0版本.zip