This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this questionHudson is repeatedly failing after building few projects with following stack trace with error "No space left on device", though there is enough space on disk. There are no limits of quotas on any folder. Below is output of different system commands.
Here is key system information:
Hudson ver. 1.361
executable-war /opt/hudson/hudson.war
java.runtime.name OpenJDK Runtime Environment
java.runtime.version 1.6.0_18-b18
os.name Linux-Ubuntu 10.04
os.version 2.6.32-19-generic
There is 50% free space according to df
$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 147550696 67382688 72672840 49% /
none 1535580 304 1535276 1% /dev
none 1539732 504 1539228 1% /dev/shm
none 1539732 96 1539636 1% /var/run
none 1539732 0 1539732 0% /var/lock
none 1539732 0 1539732 0% /lib/init/rw
none 147550696 67382688 72672840 49% /var/lib/ureadahead/debugfs
next i checked open file handles by lsof, thats also within limit
lsof | wc -l
694
then i checked file handles by this command
cat /proc/sys/fs/file-nr
3392 0 306935
Detailed Error
[INFO] No tests to run.
[HUDSON] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] Building jar: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] [antrun:run {execution: default}]
[INFO] Executing tasks
[unzip] Expanding: /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources/CRMServices.Driver-soapui-project.zip into /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/src/it/resources
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/pom.xml to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/pom.xml
[HUDSON] Archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar to /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/modules/rogers.bt.deployment$eif.deployment.CRMServices/builds/2010-07-20_12-13-58/archive/rogers.bt.deployment/eif.deployment.CRMServices/1.2-bt/eif.deployment.CRMServices-1.2-bt.jar
[HUDSON] Re-archiving /opt/hudson/jobs/EIF_Branch_R1.2__Projects_PkgOnly/workspace/r1.2/projects/CRMServices/deployment/target/eif.deployment.CRMServices-1.2-bt.jar
[INFO] ------------------------------------------------------------------------
[ERROR] FATAL ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Failed to serialize hudson.model.Actionable#actions for class hudson.maven.MavenModuleSetBuild
No space left on device
[INFO] ------------------------------------------------------------------------
[INFO] Trace
java.langchannel stopped
ERROR: Failed to parse POMs
java.io.IOException: Remote call on Channel to Maven 开发者_运维知识库[/opt/bea/jdk160_05/bin/java, -cp, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-agent-1.363.jar:/opt/hudson/tools/Maven_2.2.1/boot/classworlds-1.1.jar, hudson.maven.agent.Main, /opt/hudson/tools/Maven_2.2.1, /opt/hudson/war/WEB-INF/lib/remoting-1.363.jar, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven-interceptor-1.363.jar, 55951, /opt/hudson/plugins/maven-plugin/WEB-INF/lib/maven2.1-interceptor-1.2.jar] failed
at hudson.remoting.Channel.call(Channel.java:564)
at hudson.maven.ProcessCache$MavenProcess.call(ProcessCache.java:156)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.doRun(MavenModuleSetBuild.java:483)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:416)
at hudson.model.Run.run(Run.java:1253)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:127)
Caused by: java.lang.Error: Unable to load resource hudson/maven/Messages.properties
at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:198)
at java.lang.ClassLoader.getResource(ClassLoader.java:977)
at java.lang.Class.getResource(Class.java:2074)
at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:83)
at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
at org.jvnet.localizer.ResourceBundleHolder.get(ResourceBundleHolder.java:102)
at org.jvnet.localizer.ResourceBundleHolder.format(ResourceBundleHolder.java:139)
at hudson.maven.Messages.MavenBuilder_AsyncFailed(Messages.java:233)
at hudson.maven.MavenBuilder.call(MavenBuilder.java:184)
at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:696)
at hudson.maven.MavenModuleSetBuild$Builder.call(MavenModuleSetBuild.java:640)
at hudson.remoting.UserRequest.perform(UserRequest.java:114)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:247)
at hudson.remoting.RemoteClassLoader.makeResource(RemoteClassLoader.java:267)
at hudson.remoting.RemoteClassLoader.findResource(RemoteClassLoader.java:194)
... 19 more
FATAL: : No space left on device
hudson.util.IOException2: : No space left on device
at hudson.XmlFile.write(XmlFile.java:168)
at hudson.model.Run.save(Run.java:1383)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
at hudson.model.Run.run(Run.java:1276)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:306)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:127)
Caused by: com.thoughtworks.xstream.io.StreamException: : No space left on device
at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:73)
at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:99)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:840)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:829)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:804)
at hudson.XmlFile.write(XmlFile.java:165)
... 7 more
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(FileOutputStream.java)
at java.io.FileOutputStream.write(FileOutputStream.java:260)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at java.io.BufferedWriter.flush(BufferedWriter.java:236)
at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:91)
at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:288)
at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:49)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:38)
at hudson.XmlFile.write(XmlFile.java:165)
at hudson.model.Run.save(Run.java:1384)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:595)
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
The "no space left on device" error isn't necessarily caused by running out of storage capacity, as it suggests, it can also be cause by running out of i-nodes on the filesystem. In other words, a given filesystem can only contain so many files. Running df will suggest everything's fine.
See this article for more info.
You either need to delete some files that you don't need any more, or put Hudson on a different filesystem.
It's not uncommon in situations where you know you'll have a lot of small files to build a filesystem with an explicitly larger i-node table.
It is due to overflow of memory.
run this command to check the available memory size
df
If you find 100% or near to 100% for any disk, please delete data from that directory, but be carefully, and then try again.
In most of cases, the /tmp directory will be 100%, so reboot your machine
sudo reboot
it will erase all files from tmp directory.
Also do ipcs -u and ipcs -p to see if the "out of space" is the SHM memory full and what apps is using it. If segments allocated are equal to sysctl kernel.shmmni, you are with a "full" shm. You can also compare with ipcs -l output, as this show the system limits.
Latest version of java might abuse the shm usage until it is none free. Either close the program that is abusing the shm or increase the kernel.shmmni and kernel.shmmax sysctl to increase the available shm space
![Interactive visualization of a graph in python [closed]](https://www.devze.com/res/2023/04-10/09/92d32fe8c0d22fb96bd6f6e8b7d1f457.gif)
加载中,请稍侯......
精彩评论