Alex,

Sorry for delay with replying.

Could you attach the server VCS logs covering the hanging and take a thread dump from the server during the hanging?

Also, please note the version of the Git plugin that you use (it is recommended to use the latest)

Attached the info.

I am running git plugin version 1.0.0.pub-39-d65289d918c522208b976a6845cc1b05afc9e764

Also, thread dumping from within teamcity hangs forever.

Alex,

I am afraid we will need thread dump from the server process (the one you attached is from the agent).

Yegor,

Sorry about that, I forgot that the checkout was made on the server.

Have now attached tomcat.jstack.txt

This looks like duplicate for TW-9611. We are experiencing the issue on our internal server too and will try to address the issue soon. Please watch the linked issue

I upgraded to version 1.0.0.pub-49-69f9053b81af8e545d2985a02a750680620ade46

I thought this issue was fixed there since TW-9611 was marked as fixed. And I thought that the "checking for changes" now had a timeout of 30 seconds but that is not the case. We still have hanging builds.

Do we have to set the system properties described here ? –> http://www.jetbrains.net/confluence/display/TW/Git

Also, this sshs:// connection in question is to localhost, so I am certain that there must also be an underlying issue too. There should not be any "timeouts" at all.

Alex,

After TW-9611 there is a timeout. The defaults are stated on the plugin page (30 minutes by default for fetch)

It turns out that there are still some communication issues. Let's reopen the issue to address it (however, probably this is related to underlying library: jgit, not TeamCity Git integration)

I understand that this is probably related to jgit, by looking at the stacktrace.

One thing that I have noticed is that when builds are triggered either by a schedule or by someone pressing the "run" button, the build has a very high tendency to hang.

However, when builds are triggered via a push/pull ( ie. a new commit), then it works fine.

Also, once a VCS root has started to show this behaviour, it never starts working again. (new changes are never picked up )

Hi,

We have encountered the same problems after upgrading TeamCity from 4.5 to 5.0.1 (Our Teamcity server runs on a Linux server)
We have not been able to do any builds at all with Teamcity 5.

I tested to do a new/clean install of Teamcity 4.5.6 + the GIT-plugin on a Windows machine - and it work without problems.
I tested to do a new/clean install of Teamcity 5.0.1 on a Windows machine - and it did fail.

So I think that something has happened when the plugin was integrated into Teamcity.

Question: Is it possible to in some way install/use the plugin into Teamcity 5.0.1 ?

Do you use MSSQL database?

No, our "production" Teamcity server is running on a Linux OS and is using a a PostgreSQL db.

and the additional tests I did on a computer running Windows were using the "internal" Teamcity db.

.... and the git repository is located on another server and is accessd using ssh

What version of GIT plugin works fine?

Not at office right now, so I cannot check the version installed. But was downloaded and installed today from http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt152&buildId=lastSuccessful

we are having similar problem. Any suggestions on how to get the GIT to work with 5.1? Our setup is: Teamicty: Windows 2003 platform, GIT: Linux. Connection Windows->Linux over SSH. Connection times out.

We're also seeing the same with Git using build 10715 (Win 2003 server) using ssh. This appears to occur most with projects that have multiple branch configurations. What we've noticed is that adding a branch config by cloning the master config seems to prevent the master build from getting any updates, i.e. from a code perspective the builds are stale. ATM the workaround for us is to nuke the checkout caches on the server. It would seem to me that the solution is to have the agent check the source out rather than the server, so that it has a complete copy of the Git index.

We are experiencing the same issue, we have to nuke the checkout caches and restart, I say it happens every 10 builds. We are using latest with Git

Guys,

1. Please ensure you use the latest git plugin available, the hanging should be fixed there, but the connection will be terminated with a timeout in the case the condition reproduces.
2. The issue only reproduces with ssh protocol, so if possible, try using git or http.

Just wanted to chime in. We are having the same problem. When i created the project it successfully checked it out and build it. However now it never leaves the checking for changes phase. I am checking the project out to a hardcoded checkout directory and it never even creates the .git folder there

this is even after having updated the git plugin to the latest version

When checking the logs i find the following information that indicates a deadlock

[2010-04-27 12:18:12,949] WARN - Server.util.NamedThreadFactory - Cached pool 18  at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.lockChangesCollecting(VcsChangesLoader.java:144)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.lockChangesLoading(VcsManagerImpl.java:680)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:191)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:18)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Cached pool 6  at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.lockChangesCollecting(VcsChangesLoader.java:144)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.lockChangesLoading(VcsManagerImpl.java:680)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:191)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:18)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Cached pool 15  at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.lockChangesCollecting(VcsChangesLoader.java:144)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.lockChangesLoading(VcsManagerImpl.java:680)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:191)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:18)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java:1)
at jetbrains.buildServer.vcs.impl.VcsChangesCollectorImpl$3.call(VcsChangesCollectorImpl.java)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Cached pool 17

Why do you think this is a deadlock? Do you have full thread dump?

Are there any updates on this ? We are having the same problem. Restarting our server every time it hangs isn't an option as we have many people using it. This problem is forcing us to look for an alternative CI solution if this bug can't resolved.

+1. We are facing the exact same issue too. Deleting the work directory in buildAgent solves the problem though. It's quite an irritant!

Seems that it is really bug of jgit+jsch, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=312773, https://bugs.eclipse.org/bugs/show_bug.cgi?id=312383 and https://bugs.eclipse.org/bugs/show_bug.cgi?id=312863 for details

Thanks for taking the time to investigate this. it seems the two most critical issues have been resolved in the jgit repo already so i checked out the latest jpgit development snapshot and compiled it

then took the sources from http://teamcity.jetbrains.com/viewLog.html?guest=1&tab=artifacts&buildTypeId=bt152&buildId=lastSuccessful

and updated the jgit and jsch dependencies in the project. I attached the result as jetbrains.git.zip. Maybe it's useful for someone. I just deployed it on the server and so far it seems to work alright. Too early to tell if it fixed the issue tho

fyi this has not occurred in the two days since i installed the plugin with the updated jgit version

You might also be interested in this bug in jgit: https://bugs.eclipse.org/bugs/show_bug.cgi?id=313082

TeamCity 5.1.2 contains fixes for this issue. If there will be hangs while git fetch, please send us ThreadDumps

With the 5.1 release, oddly, I didn't have any issues with Git fetches. Since the update to 5.1.2, after the FIRST fetch, it has started hanging on 'Checking for changes'.

Here's a thread dump:

"12:50:23 Loading VCS changes for git@missarepo1:usd259.git#master {id=9}; Changes loader 3 {id=9}" group="main" prio=5 id=43 java.util.concurrent.locks.ReentrantLock$NonfairSync@8b071a
java.lang.Thread.State: WAITING  at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(Unknown Source)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(Unknown Source)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Unknown Source)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Unknown Source)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(Unknown Source)  at java.util.concurrent.locks.ReentrantLock.lock(Unknown Source)
at jetbrains.buildServer.buildTriggers.vcs.VcsChangesLoader.lockChangesCollecting(VcsChangesLoader.java:77)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.lockChangesLoading(VcsManagerImpl.java:743)
at jetbrains.buildServer.vcs.impl.VcsManagerImpl.loadChanges(VcsManagerImpl.java:594)
at jetbrains.buildServer.serverSide.impl.auth.SecuredVcsManager.loadChanges(SecuredVcsManager.java:80)
at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$1.run(VcsModificationChecker.java:10)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)  at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)  at java.lang.Thread.run(Unknown Source)

"Thread-3" group="main" prio=5 daemon id=13
java.lang.Thread.State: TIMED_WAITING  at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:103)

"12:52:01 http-8081-1 /admin/editVcsRoot.html?action=editVcsRoot " group="main" prio=5 daemon id=17 java.lang.Object@17f6b6d
java.lang.Thread.State: BLOCKED
at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.testConnection(GitVcsSupport.java:1082)
at jetbrains.buildServer.controllers.admin.projects.EditVcsRootsController.doPost(EditVcsRootsController.java:107)
at jetbrains.buildServer.controllers.BaseFormXmlController$1.handleRequest(BaseFormXmlController.java:54)
at jetbrains.buildServer.controllers.AjaxRequestProcessor.processRequest(AjaxRequestProcessor.java:45)
at jetbrains.buildServer.controllers.BaseFormXmlController.doHandle(BaseFormXmlController.java:52)
at jetbrains.buildServer.controllers.BaseController.handleRequestInternal(BaseController.java:73)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:48)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:875)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:807)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:571)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:511)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at jetbrains.buildServer.rootDispatcher.TeamCityDispatcherServlet.service(TeamCityDispatcherServlet.java:36)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at jetbrains.buildServer.web.SetThreadNameFilter.runChainWithModifiedThreadName(SetThreadNameFilter.java:8)
at jetbrains.buildServer.web.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:26)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at jetbrains.buildServer.web.ResponseFragmentFilter.doFilter(ResponseFragmentFilter.java:5)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)  at java.lang.Thread.run(Unknown Source)

David, could you provide a full thread dump please?

Attached thread dumps to the ticket from yesterday (when I originally posted) and a dump for a new pull I kicked off today. Attached as threadDump.zip.

A possible reason for such behavior is that we added synchronization in every operation with jgit including "Test connection". With such synchronization only one operation is allowed in particular moment. So if you make "Test connection" while collecting changes, "Test connection" will wait for collecting changes to finish, which could take long time. Could you please try to install this build of git-plugin without synchronization in "Test connection" and check if it helps?

I thought that the build of Jetbrains.Git had fixed it. It seems to work fine the FIRST fetch after the server is up and going; however, subsequent fetches all hang at 'Checking for changes'.

I pushed a build before I went home last night and came in this morning to a timeout error. I've attached the timeout error and threaddump.

David, I succeed to reproduce it and reported a bug for jgit (https://bugs.eclipse.org/bugs/show_bug.cgi?id=317288). The only workaround I see at the moment is to set TeamCity internal properties teamcity.git.fetch.timeout and teamcity.git.clone.timeout to 0, in this case jgit code with a bug is not used. When reported bug will be fixed, timeouts should work as expected.

But be careful: if jgit will hang for some reason you will have to kill whole process, because no timeouts are used. You probably should make fetch in separate process (set teamcity.git.fetch.separate.process to true).

Great, thanks for submitting the bug. I added the timeouts and separate.process flag, but it threw out an error:

[2010-06-21 13:33:02,130] WARN [loader 4 {id=6}] - jetbrains.buildServer.VCS - Error while loading changes for root git@reposerver:project.git#branchName {id=6}, cause: 'git fetch' command failed. stderr: Exception in thread "main" java.lang.NullPointerException: The factory must not be null
 at org.eclipse.jgit.transport.SshTransport.setSshSessionFactory(SshTransport.java:99)
 at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.openTransport(GitVcsSupport.java:1286)
 at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.openTransport(GitVcsSupport.java:1267)
 at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.fetch(Fetcher.java:64)
 at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.main(Fetcher.java:42)


[2010-06-21 13:33:02,130] WARN [loader 2 {id=4}] - Triggers.vcs.git.GitVcsSupport - 'git fetch' command failed. stderr: Exception in thread "main" java.lang.NullPointerException: The factory must not be null
 at org.eclipse.jgit.transport.SshTransport.setSshSessionFactory(SshTransport.java:99)
 at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.openTransport(GitVcsSupport.java:1286)
 at jetbrains.buildServer.buildTriggers.vcs.git.GitVcsSupport.openTransport(GitVcsSupport.java:1267)
 at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.fetch(Fetcher.java:64)
 at jetbrains.buildServer.buildTriggers.vcs.git.Fetcher.main(Fetcher.java:42)

I took the lines back out of the internal.properties, restarted, and they picked back up. The few repos that it seems to be affecting the most don't change except weekly–and I can force a full clone on those as necessary for the time being until the jgit update works its way through.

Thanks!

Are you sure you use last attached build? Same exception was in issue TW-12515 and this build solved the problem.

I just redownloaded/reapplied the build you provided–same thing.

I have the exact same problem as David, should I download the patch and change the intenal team city properties?
I am using MS SQL 2008 server for storage.

@Mikael-

I redownloaded (again) the jetbrains.git.zip provided by Dmitry, restarted the agent... and it seemed to pick it up that time and install it. It should then show up in Server Configuraiton > External Plugins as version 13456.

Since then (so far), the internal.properties settings provided seem to work just fine... and on the past few pushes, the repos are updating accordingly. As Dmitry pointed out, that should work until jgit is updated.

David, Dmitry I can confirm that solution worked for me as well.
Thank you all for helping out!

Mikael, David I attached the build 13477 with fix of the jgit issue. This fix is not yet accepted by jgit team, but I hope it will. You can try to use it with default values of properties teamcity.git.fetch.separate.process, teamcity.git.fetch.timeout and teamcity.git.clone.timeout.

Excellent. That build seems to work. Used it all day with a few dozen pushes up to various repos and TC never flinched a bit. Thanks a lot!