[cdh-user] Re: RuntimeException running the hdfs command (core-site.xml not found)

I had the same problem.  When I try 'hadoop fs -ls /', the java runtime error occurs 'cannot find core-site.xml'.  Yet I could access the hdfs via text browser elinks.

Weirdly enough a restart of the machine the next day solved the issue. I was playing around with hadoop and spark classpaths mainly for slf4j purposes and I'm guessing some link change messed up the hadoop configuration.  It returned back to normal now, everything running 'smoothly'.

On Thursday, 29 May 2014 08:45:58 UTC+1, Manish wrote:

Yes, I checked /etc/hadoop/conf/ configuration. 

Directory contains container-executor.cfg, core-site.xml, hadoop-env.sh, hdfs-site.xml, log4j.properties, mapred-site.xml, and taskcontroller.cfg configuration files.

In core-site.xml contains correct path for HDFS:

<property>

    <name>fs.defaultFS</name>

    <value>hdfs://------:8020</value>

  </property>

I have deployed the client configuration also from CM UI but still same error.

On Thursday, May 29, 2014 12:53:06 PM UTC+5:30, Harsh J wrote:

Can you please check if your /etc/hadoop/conf/ carries all cluster
configs, with proper contents? If not, can you perform a Deploy Client
Configuration from the CM UI?

On Thu, May 29, 2014 at 12:49 PM, Manish Verma <manis...@gmail.com> wrote:
> I have installed via automated installation.
>
> All component works fine, if i will run HDFS command in terminal throws an
> error regarding core-site.xml not found.
>
>
> On Thursday, May 29, 2014 11:43:25 AM UTC+5:30, Rinku Garg wrote:
>>
>> How did you installed via automated installation or manual installation?
>> Please share complte logs.
>>
>> Thanks
>> Rinku Garg
>>
>>
>> On Thu, May 29, 2014 at 11:03 AM, Manish Verma <manis...@gmail.com> wrote:
>>>
>>> I am also facing same problem.
>>>
>>> After running the command, following is the error message:
>>>
>>> Exception in thread "main" java.lang.RuntimeException: core-site.xml not
>>> found
>>> at
>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2257)
>>> at
>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2185)
>>> at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2102)
>>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:979)
>>> at org.apache.hadoop.conf.Configuration.set(Configuration.java:953)
>>> at
>>> org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:1289)
>>> at
>>> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:320)
>>> at
>>> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:478)
>>> at
>>> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:171)
>>> at
>>> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:154)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at org.apache.hadoop.fs.FsShell.main(FsShell.java:308)
>>>
>>>
>>> On Wednesday, April 23, 2014 2:19:30 AM UTC+5:30,jlwi...@gmail.com
>>> wrote:
>>>>
>>>> Just installed CDH 5.  Install went fine.  Console looks good.  But,
>>>> when I run the hdfs command I get the following:
>>>>
>>>> ...RuntimeException: core-site.xml not found.
>>>>
>>>>
>>>> Any ideas?
>>>>
>>>>
>>>> Thanks,
>>>> Jim
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "CDH Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to cdh-user+u...@cloudera.org.
>>>
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout.
>> 

[hue-user] Re: HUE building error

Hi,
 Please let me know where i should update the above change? 
Thanks in advance
 Shreyas M

On Monday, 15 September 2014 19:36:25 UTC+5:30, Romain Rigaux wrote:

You probably need to update
https://github.com/cloudera/hue/blob/master/maven/pom.xml#L40

with one of this value:
http://repository.cloudera.com/cloudera/libs-snapshot-local/org/apache/hadoop/hadoop-test/maven-metadata.xml


<metadata><groupId>org.apache.hadoop</groupId><artifactId>hadoop-test</artifactId><versioning><latest>2.5.0-mr1-cdh5.2.0-SNAPSHOT</latest><versions><version>0.20.2-cdh3u7-SNAPSHOT</version><version>2.0.0-mr1-cdh4.3.0-SNAPSHOT</version><version>2.0.0-mr1-cdh4.5.0-SNAPSHOT</version><version>2.0.0-mr1-cdh4.5.0.1-SNAPSHOT</version><version>2.0.0-mr1-cdh4.5.0.2-SNAPSHOT</version><version>2.0.0-mr1-cdh4.6.0-SNAPSHOT</version><version>2.0.0-mr1-cdh4.7.0-SNAPSHOT</version><version>2.0.0-mr1-cdh4.8.0-SNAPSHOT</version><version>2.2.0-mr1-cdh5.0.0-SNAPSHOT</version><version>2.2.0-mr1-cdh5.0.0-beta-2-SNAPSHOT</version><version>2.3.0-mr1-cdh5.0.0-SNAPSHOT</version><version>2.3.0-mr1-cdh5.0.1-SNAPSHOT</version><version>2.3.0-mr1-cdh5.0.2-SNAPSHOT</version><version>2.3.0-mr1-cdh5.0.3-SNAPSHOT</version><version>2.3.0-mr1-cdh5.0.4-SNAPSHOT</version><version>2.3.0-mr1-cdh5.1.0-SNAPSHOT</version><version>2.3.0-mr1-cdh5.1.1-SNAPSHOT</version><version>2.3.0-mr1-cdh5.1.3-SNAPSHOT</version><version>2.3.0-mr1-cdh5.1.4-SNAPSHOT</version><version>2.3.0-mr1-cdh5.2.0-SNAPSHOT</version><version>2.5.0-mr1-cdh5.2.0-SNAPSHOT</version></versions><lastUpdated>20140915091417</lastUpdated></versioning></metadata>

On Fri, Sep 12, 2014 at 5:20 AM, <amithsha...@gmail.com> wrote:

hi, i am also facing the same problem.have tried in hadoop 2.4.1 with hue 3.6.0 and also in hadoop 2.3.0 with hue 3.5.0

INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.024 s
[INFO] Finished at: 2014-09-12T17:42:52+05:30
[INFO] Final Memory: 10M/109M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project hue-plugins: Could not resolve dependencies for project com.cloudera.hue:hue-plugins:jar:3.5.0: Could not transfer artifact org.apache.hadoop:hadoop-test:jar:2.3.0 from/to cloudera.snapshots.repo (https://repository.cloudera.com/content/repositories/snapshots): Failed to transfer file: https://repository.cloudera.com/content/repositories/snapshots/org/apache/hadoop/hadoop-test/2.3.0/hadoop-test-2.3.0.jar. Return code is: 409 , ReasonPhrase:Conflict. -> [Help 1]

you should try to update, as Romain said, the <hadoop.version> of YOUR_HUE_FOLDER/maven/pom.xml with one of the versions provided on http://repository.cloudera.com/cloudera/libs-snapshot-local/org/apache/hadoop/hadoop-test/maven-metadata.xml

Putting it back on the list:

======================

From your command line can you do a

curl http://repository.cloudera.com/cloudera/libs-snapshot-local/org/apache/hadoop/hadoop-test/2.0.0-mr1-cdh4.3.0-SNAPSHOT/maven-metadata.xml

? Do you see any output? Do you have Maven configured in any special way in your build system? 

On Tue, Jan 6, 2015 at 11:55 AM, Enrico Berti<enrico.berti@cloudera.com> wrote:

From your command line can you do a

curl http://repository.cloudera.com/cloudera/libs-snapshot-local/org/apache/hadoop/hadoop-test/2.0.0-mr1-cdh4.3.0-SNAPSHOT/maven-metadata.xml

? Do you see any output? Do you have Maven configured in any special way in your build system? 

e.

On Tue, Jan 6, 2015 at 6:24 AM, shreyas m<shreyas080@gmail.com> wrote:

Hi,

     Thank you very much, but it has not yet solved the error. I'm trying to do it on a plain apache hadoop 2.4

Error is
[WARNING] Could not transfer metadata org.apache.hadoop:hadoop-test:2.0.0-mr1-cdh4.3.0-SNAPSHOT/maven-metadata.xml from/to cloudera.snapshots.repo (https://repository.cloudera.com/content/repositories/snapshots): peer not authenticated
[WARNING] Failure to transfer org.apache.hadoop:hadoop-test:2.0.0-mr1-cdh4.3.0-SNAPSHOT/maven-metadata.xml fromhttps://repository.cloudera.com/content/repositories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of cloudera.snapshots.repo has elapsed or updates are forced. Original error: Could not transfer metadata org.apache.hadoop:hadoop-test:2.0.0-mr1-cdh4.3.0-SNAPSHOT/maven-metadata.xml from/to cloudera.snapshots.repo (https://repository.cloudera.com/content/repositories/snapshots): peer not authenticated
Downloading: https://repository.cloudera.com/content/repositories/snapshots/org/apache/hadoop/hadoop-test/2.0.0-mr1-cdh4.3.0-SNAPSHOT/hadoop-test-2.0.0-mr1-cdh4.3.0-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------[INFO] Total time: 2.203s
[INFO] Finished at: Tue Jan 06 10:52:02 IST 2015
[INFO] Final Memory: 10M/236M

[cdh-user] Sqoop2 MAPREDUCE Exception

When iam trying to start a job in sqoop2, each time the job is failing with these exceptions

Errors:

1. org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0017:Error occurs during extractor run

2. org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0013:Cannot write to the data write

3. org.apache.sqoop.common.SqoopException: MAPRED_EXEC_0018:Error occurs during loader run

Iam trying to import Table with NULL values in it to HDFS,

Please let me know , if any solution,

[cdh-user] How to use Avro timestamp in CDH 5.3?

CDH 5.3 release note says, Apache Avro has new date, time, timestamp, and duration binary types.

What I need is a timestamp in the impala table created from a parquet file, how to achieve this with the new Avro feature introduced by AVRO-739

I write a schema with

{"name": "ServerTime", "type": "long", "logicalType": "timestamp-millis", "default": 0}

but when I use mvn generate-sources, ServerTime is still a java long, and if I write the record to parquet using AvroParquetWriter,

it’s int64, and then I load the file to impala, ServerTime is now bigint, not timestamp .

Note that I've updated avro, parquet-avro, avro-maven-plugin to the latest cdh5.3.0 version

[hue-user] Hue mysql Database setup fails

I have been trying to set up hue to an external mysql database & it fails with below error when I try to do a         

    sudo -u hue hue/build/env/bin/hue dumpdata > tmp.json   

        

mysql host is 127.0.1.1

I granted privileges to hue user using command:

grant all on hue.* to 'hue'@'%' identified by '123';

in pseudo-distributed.ini i made the following changes in [database] section:

 engine=mysql

    host=127.0.1.1

    port=3306

    user=hue

    password=123

    name=hue

    ## options={}

& yet it fails with below error:        

    

    File "hue/build/env/bin/hue", line 8, in <module>

        load_entry_point('desktop==3.7.0', 'console_scripts', 'hue')()

      File "/home/vamsi/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry

        execute_manager(settings)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/management/__init__.py", line 459, in execute_manager

        utility.execute()

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/management/__init__.py", line 382, in execute

        self.fetch_command(subcommand).run_from_argv(self.argv)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/management/base.py", line 196, in run_from_argv

        self.execute(*args, **options.__dict__)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/management/base.py", line 231, in execute

        self.validate()

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/management/base.py", line 266, in validate

        num_errors = get_validation_errors(s, app)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/management/validation.py", line 103, in get_validation_errors

        connection.validation.validate_field(e, opts, f)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/backends/mysql/validation.py", line 14, in validate_field

        db_version = self.connection.get_server_version()

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/backends/mysql/base.py", line 415, in get_server_version

        self.cursor().close()

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/backends/__init__.py", line 317, in cursor

        cursor = self.make_debug_cursor(self._cursor())

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/backends/mysql/base.py", line 387, in _cursor

        self.connection = Database.connect(**kwargs)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.3c1-py2.7-linux-i686.egg/MySQLdb/__init__.py", line 81, in Connect

        return Connection(*args, **kwargs)

      File "/home/vamsi/hue/build/env/lib/python2.7/site-packages/MySQL_python-1.2.3c1-py2.7-linux-i686.egg/MySQLdb/connections.py", line 188, in __init__

        super(Connection, self).__init__(*args, **kwargs2)

    _mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on '127.0.1.1' (111)")    

Any help would be really appreciated. I am ready for a team viewer interaction too, if someone could solve this issue for me.

Perhaps there's a typo in the config -- to use the IPv4 loopback address, I believe it should be "127.0.0.1"

Thank U so much Andrew. It solved my issue. Thx for such quick response. I owe U.

Now I am experiencing this issue when I try to run a pigscript. what could have been the problem?.Please help!

[scm-user] Re: Hive Metastore service health check

    I have found same issue in my cluster,we have 2 cluster , one is CM 5.0 and other is 5.1 version, This issue existing in 5.1 version CM. 

   can you provide the bug link?

在 2014年9月9日星期二UTC+8上午12时44分20秒，Gilad Wolff写道：

Hi Guy,

There is a known bug with the hive canary that may cause it to fail constantly - the client-configs we are using to connect to the hive metastore are partial. This may be the root cause of this failure. Do you have security enabled on the cluster? Did you change the hadoop.rpc.protection configuration option? A fix is going to be available very soon with the next release of cloudera manager (5.1.3) and hopefully it will solve the problem. In the meantime you can disable the hive metastore canary.

G.

On Mon, Sep 8, 2014 at 7:32 AM, Guy Doulberg <guy...@gmail.com>wrote:

Hi guys,

In our CM there is a health check called Hive Metastore Canary Health Test

This check is constantly failing,

A little investigation  shows that the check creates a DB (cloudera_manager_metastore_canary_test_db_hive2_hivemetastoreXXX) and a table in it (cm_test_table). 

The check then try to delete the table and destroy the DB, the check fails while trying to do that. 

As a result we end up with many databases in HIVE that contain one table. 

Furthermore, we are not able to delete the table becasue the hive complains the serde is null... we looked in hive metastore SDS table, and it creates the table with null SERDE, null outputformant and null inputformat

Did any of you encountered such a thing?

Thanks 

To unsubscribe from this group and stop receiving emails from it, send an email to scm-users+...@cloudera.org.

在 2014年9月9日星期二UTC+8上午12时44分20秒，Gilad Wolff写道：

Hi Guy,

There is a known bug with the hive canary that may cause it to fail constantly - the client-configs we are using to connect to the hive metastore are partial. This may be the root cause of this failure. Do you have security enabled on the cluster? Did you change the hadoop.rpc.protection configuration option? A fix is going to be available very soon with the next release of cloudera manager (5.1.3) and hopefully it will solve the problem. In the meantime you can disable the hive metastore canary.

G.

On Mon, Sep 8, 2014 at 7:32 AM, Guy Doulberg <guy...@gmail.com>wrote:

Hi guys,

In our CM there is a health check called Hive Metastore Canary Health Test

This check is constantly failing,

A little investigation  shows that the check creates a DB (cloudera_manager_metastore_canary_test_db_hive2_hivemetastoreXXX) and a table in it (cm_test_table). 

The check then try to delete the table and destroy the DB, the check fails while trying to do that. 

As a result we end up with many databases in HIVE that contain one table. 

Furthermore, we are not able to delete the table becasue the hive complains the serde is null... we looked in hive metastore SDS table, and it creates the table with null SERDE, null outputformant and null inputformat

Did any of you encountered such a thing?

Thanks 

The fix I mentioned above is in cm 5.1.2. What version of cm 5.1 are you running?

[phoenix-hbase-user] Existing hbase table mapped as view - field cast as double

I don't know if this is a bug with sqlline, phoenix or unique to my installation
but based on the datatype I get a different answer for the same query.  

I have a table in hbase 

 get 'beta_accounts_created', '20130425020340|000000564259|00000'
COLUMN                               CELL
 d:balance                           timestamp=1420266742812, value=100

I go to phoenix and create the following view
CREATE VIEW "beta_accounts_created" ( 
   pk VARCHAR PRIMARY KEY  
 , "d"."balance"       DOUBLE
);

The query is straight foward

select "d"."balance" from  "beta_accounts_created" where pk = '20130425020340|000000564259|00000';

 -4.3912361131335697E71

then I recreate the view with a different number datatype
CREATE VIEW "beta_accounts_created" ( 
   pk VARCHAR PRIMARY KEY  
 , "d"."balance"       BIGINT
);

select "d"."balance" from  "beta_accounts_created" where pk = '20130425020340|000000564259|00000';

-5678986353556037888

then I recreate the view with as a VARCHAR datatype and then I get the right answer
CREATE VIEW "beta_accounts_created" ( 
   pk VARCHAR PRIMARY KEY  
 , "d"."balance"       VARCHAR
);
 
100

Is this known behavior?
hbase 0.98.6
phoenix 4.2.2
sqlline 1.2.2