2014년 12월 31일 수요일

[hue-user] [cdh-user] "No available logs" in Pig, Hue version 3.0.0

I am attempting to run Pig jobs in Hue, and keep getting "No available logs". My Pig scripts probably have errors in them, but I'd like to try to figure out what I'm doing wrong myself before posting the errors here, since they're probably really simple issues. But, without logs, its quite impossible to fix. 

I've followed several discussions on both this forum and others regarding this issue, but the answers are not very detailed, such as "Configure oozie service" and "Copy the sharelibs*.tar.gz into HDFS and point th hue.ini at it." I checked my entire node configured with the HUE service and found no "sharelibs*.tar.gz" file. I would have assumed that all of this should have been set up via the CDH 5.0 beta install, which I installed all services for.

I can post any logs anyone wants, but I didn't want to post all the logs in case I was posting irrelevant information and spamming. 

I tried to check the status of the Oozie service, via http://172.16.18.10:11000/oozie/, but I get:

type Exception report
message java.lang.IllegalStateException: No output folder
description The server encountered an internal error that prevented it from fulfilling this request.
exception
org.apache.jasper.JasperException: java.lang.IllegalStateException: No output folder
 org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:538)
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:364)
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313)
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 org.apache.oozie.servlet.AuthFilter$2.doFilter(AuthFilter.java:126)
 org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:384)
 org.apache.oozie.servlet.AuthFilter.doFilter(AuthFilter.java:131)
 org.apache.oozie.servlet.HostnameFilter.doFilter(HostnameFilter.java:84)


Would anyone have any suggestions? As I mentioned, I followed the CDH 5.0 installation with no errors.

















It seems like Oozie itself has some problems. How did you install CDH? Also, can you check the log file for oozie (/var/log/oozie/oozie.log usually)?















I used the Cloudera Enterprise 5 beta 2 installer on the primary node, and it installed all the required parts on the other nodes. Pretty much a textbook install across the board. All the standard options were picked, and I installed all the parts.





I checked /var/log/oozie on the oozie node in question, and there is no single "oozie.log". I have an oozie-audit.log, multiple logs named oozie-cmf-oozie-OOZIE_SERVER-<nodename>.log.out.<date> and an oozie-instrumentation.log (with old rotated copies). Which one would you like to see?

















Actually, if you have CM, we can check the status and logs of Oozie from CM. Can you go to the Oozie service in CM (Services > Oozie), provide the list of instances, and if they're all green? If they aren't, can you provide the oozie logs through CM? If you go to Diagnose > logs... you should be able to see the oozie logs. You'll need to choose the sources from that window and only select Oozie.















Everything is good under the particular instance for the Oozie service... I have it on one node only, and the only alert I saw was clock offset by about 4.9 seconds. I've reconciled that, but it was giving the same error even when under a "Good" status before. I just re-tested and verified the same results: I get a progress bar, but no log output.





I found logs, but not under the same place as "Diagnose > Logs". I don't see any errors under that log file, just WARNs and INFOs. The WARNs also seem like they should be INFO actions, since they are just "job done" statuses, and database update complete statuses.















The logs are dependent on the Oozie service to be in a full functional state. The error you've provided previously is indicative of some kind of permissions issue I think.





Could you check the tomcat logs? It might have some more info.







I've include cdh-user@ to see if any one else can help with this.





















(Sorry, I forgot to forward my previous response to the group....)

----- Start response -----


I checked the catalina.out log files from the 15th (sorry I haven't gotten back sooner). I found the following SEVERE errors in it:

Feb 15, 2014 10:42:25 AM org.apache.jasper.EmbeddedServletOptions <init> SEVERE: The scratchDir you specified: /opt/cloudera/parcels/CDH-5.0.0-0.cdh5b1.p0.57/lib/oozie/oozie-server/work/Catalina/localhost/oozie is unusable.
Feb 15, 2014 10:42:25 AM org.apache.catalina.startup.HostConfig deployDirectory INFO: Deploying web application directory ROOT
Feb 15, 2014 10:42:26 AM org.apache.jasper.EmbeddedServletOptions <init> SEVERE: The scratchDir you specified: /opt/cloudera/parcels/CDH-5.0.0-0.cdh5b1.p0.57/lib/oozie/oozie-server/work/Catalina/localhost/_ is unusable.

I've corrected this issue, and will re-test as soon as I clean up some space issues on the nodes.

My question is, if this works, why didn't the Oozie monitoring from CM catch this?

------ End response ------

I tested the above, and it didn't appear to help. The log files for Oozie's catalina.out don't appear to be updating, either, unless it requires a restart to realize the directories were created. I'll check the other catalina log files to see if they caught anything as well.















I'd give it all a shot :). As far as CM goes, I don't really know. It might have been difficult for CM to see this, but definitely good to note. Thanks for reporting this and let us know if restarting helps!

















Unfourtunately, none of that worked. It still hangs when reporting the log file, and the UNIX log files for catalina don't appear to be outputting anything. Restarting after creating those directories didn't seem to make a difference either.

I do suspect that it might be a CDH 5.0.0 issue, but I thought I'd check here first regardless.















When using CM, it renames the oozie.log to oozie-cmf-abunch-of-stuff; so that’s that same log.  The catalina log files don’t do much once Oozie is started.  You probably have to restart Oozie once fixing the scratchDir error.  





Abe, isn’t there an issue with Hue getting the logs from YARN in beta2? 















Re-adding hue and csh lists















Ok awesome. It looks like you're hitting this bug in CDH5b2. This was fixed in master (https://issues.cloudera.org/browse/HUE-1960)... you should be able to patch your Hue by taking the attached patch from https://issues.cloudera.org/browse/HUE-1960 and applying it to your Hue installation. e.g:





cd /tmp




cd <hue root>



patch -p1 < /tmp/HUE-1960.0.patch







Then restart Hue.















Ok, that somewhat worked... I got the following results:


[root@cdhnode10 hue]# patch -p1 </tmp/HUE-1960.0.patch 
patching file apps/jobbrowser/src/jobbrowser/views.py
Hunk #1 FAILED at 240.
1 out of 1 hunk FAILED -- saving rejects to file apps/jobbrowser/src/jobbrowser/views.py.rej
patching file apps/jobbrowser/src/jobbrowser/yarn_models.py
Hunk #1 FAILED at 223.
1 out of 1 hunk FAILED -- saving rejects to file apps/jobbrowser/src/jobbrowser/yarn_models.py.rej
patching file desktop/core/src/desktop/lib/rest/http_client.py
Hunk #1 FAILED at 112.
Hunk #2 FAILED at 121.
2 out of 2 hunks FAILED -- saving rejects to file desktop/core/src/desktop/lib/rest/http_client.py.rej
patching file desktop/core/src/desktop/lib/rest/resource.py
Hunk #2 FAILED at 73.
1 out of 2 hunks FAILED -- saving rejects to file desktop/core/src/desktop/lib/rest/resource.py.rej


And just to prove I'm in the correct directory:


[root@cdhnode10 hue]# ls -l apps/jobbrowser/src/jobbrowser/views.py
-rwxr-xr-x 1 root root 20204 Oct 27 23:15 apps/jobbrowser/src/jobbrowser/views.py
[root@cdhnode10 hue]# 








CDH5b2 is patching for me. Are you using packages or parcels? If you're using packages, could you provide the list output (e.g. rpm -qa | grep hue). If you're using parcels, could you go to Cloudera Manager and provide the exact version with build number?






Also, could you provide the 4 files that failed so that I can compare with what I have on my system?















I'm using parcels. I'm not sure where to find it in CM, but from the /opt/cloudera directory, a cat of the cdh_version.properties file reveals:


[root@cdhnode10 cloudera]# cat ./parcels/CDH-5.0.0-0.cdh5b1.p0.57/lib/hue/cloudera/cdh_version.properties

# Autogenerated build properties
version=3.0.0-cdh5.0.0-beta-1
git.hash=a36a6b4270cef6a407658a6a8246b4bd28bd5106
cloudera.hash=a36a6b4270cef6a407658a6a8246b4bd28bd5106
cloudera.base-branch=cdh5-base-3.0.0
cloudera.build-branch=cdh5-3.0.0_beta1
cloudera.pkg.version=3.0.0+cdh5.0.0+266
cloudera.pkg.release=0.cdh5b1.p0.46
cloudera.pkg.name=hue
cloudera.cdh.release=cdh5.0.0-beta-1
cloudera.build.time=2013.10.28-02:14:16GMT


The files that failed were:


apps/jobbrowser/src/jobbrowser/views.py
apps/jobbrowser/src/jobbrowser/yarn_models.py
desktop/core/src/desktop/lib/rest/http_client.py
desktop/core/src/desktop/lib/rest/resource.py


Is this sufficient?








Sorry, you asked for the 4 files directly. I apologize, I will attach them when I have a chance. I have a feeling its because I'm using cdh5b1, not cdh5b2. I'll see if I can't upgrade to b2. Would it be easier just to installed cdh4.5?














i'm sorry, I thought that you had CDH 5b2 installed! This is not an issue in CDH 5b1... so disregard my comment. I'd focus on Robert's comments on the action.

















Done... I checked for any missing scratchDir entries, and created them. I restarted Oozie, then restarted HUE right after. I resubmitted the job, and still got "No available logs." with the spinning icon. I checked the oozie-cmf logs, and the only entries with ERROR in them are the ones stating that Pig returned with errors in the scripts, such as:

2014-02-18 20:47:46,404 INFO org.apache.oozie.command.wf.ActionEndXCommand: USER[ibonny] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000000-140218204137183-oozie-oozi-W] ACTION[0000000-140218204137183-oozie-oozi-W@pig] ERROR is considered as FAILED for SLA


(That's to be expected, there's probably errors in the script.)


Even when HUE comes back with "Progress: 100%", "Status:ERROR", it still says, "No available logs."








I don't see anything else erroring out other than the above.

















Can you click on the 'Status Error', then see if you can click on the log icon of the pig action (a 3-bar like icon and see if you have more logs?)


In the case you don't see anything (or even no jobs in Job Browser), the Oozie is still probably not setup correctly.


















When I click on the three bars beside the eye under Oozie, it takes me to the job browser and reports:








 Logs not available for attempt_1392478666138_0012_m_000000_0. Could be caused by the rentention policy






































Yes, I do get logs when accessing "job_attempt_logs/0" rather than "single_logs". Is this considered a bug in HUE, or something other than that?


































I am still getting "No Available Logs" when i run any pig-script from hue. I have cdh5.2.0 installed. 



Appreciate your help 






















You are probably hitting these ones. They are in 5.2.2 (~mid January) or 5.3 (released for Christmas).


A quick way to confirm this would be to look in the Browser js console, you'll see a 500 error from a GET request.






















































작성자:



시간:







































댓글 없음:















댓글 쓰기


















        

      









피드 구독하기:
댓글 (Atom)