Visual Studio crashes due to TypeScript running out of memory

[Vilx2]

TypeScript Version: 1.8.34.0

Since I updated my Visual Studio 2015 to Update 3 this morning, it has started to crash very often. Basically at least once an hour. It's quite random, and unfortunately I cannot share any code (it happens when working on our commercial product). But I can share a stack trace that I get out of Event Viewer:

Application: devenv.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at Newtonsoft.Json.JsonTextReader.ReadData(Boolean, Int32)
   at Newtonsoft.Json.JsonTextReader.ReadStringIntoBuffer(Char)
   at Newtonsoft.Json.JsonTextReader.ParseString(Char)
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonTextReader.ReadInternal()
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.Linq.JContainer.ReadContentFrom(Newtonsoft.Json.JsonReader)
   at Newtonsoft.Json.Linq.JContainer.ReadTokenFrom(Newtonsoft.Json.JsonReader)
   at Newtonsoft.Json.Linq.JObject.Load(Newtonsoft.Json.JsonReader)
   at Newtonsoft.Json.Linq.JToken.ReadFrom(Newtonsoft.Json.JsonReader)
   at Newtonsoft.Json.Linq.JToken.Parse(System.String)
   at Microsoft.CodeAnalysis.Editor.TypeScript.ScriptServices.JsonHelpers.TryGetResult[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.String, System.__Canon ByRef)
   at Microsoft.CodeAnalysis.Editor.TypeScript.ScriptServices.JsonHelpers.ParseEmitOutput(System.String)
   at Microsoft.CodeAnalysis.Editor.TypeScript.ScriptServices.Proxies.LanguageServiceProxy.GetEmitOutput(Microsoft.CodeAnalysis.Editor.TypeScript.Features.ProjectDataCache, Microsoft.CodeAnalysis.Editor.TypeScript.ScriptContexts.ScriptContext, Microsoft.CodeAnalysis.Document, Boolean, System.Threading.CancellationToken)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.CompileOnSave.CompileOnSaveHandler+<>c__DisplayClass18_0.<TryEmitAsync>b__0(Microsoft.CodeAnalysis.Editor.TypeScript.ScriptServices.Proxies.LanguageServiceProxy, Microsoft.CodeAnalysis.Editor.TypeScript.Features.ProjectDataCache, Microsoft.CodeAnalysis.Editor.TypeScript.ScriptContexts.ScriptContext)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.TaskHandler.ComputeValue[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.String, Microsoft.CodeAnalysis.Editor.TypeScript.ScriptContexts.ScriptContext, Microsoft.CodeAnalysis.Editor.TypeScript.Features.ProjectDataCache, Microsoft.CodeAnalysis.Editor.TypeScript.LanguageServiceOperation`1<System.__Canon>)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.TaskHandler+<>c__DisplayClass11_0`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].<PerformOperation>b__0()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.TaskHandler+<PerformSemanticFeatureOperationAsync>d__15`1[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.CompileOnSave.CompileOnSaveHandler+<TryEmitAsync>d__18.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.CompileOnSave.CompileOnSaveHandler+<TryEmitAndSaveOtherDocumentsAsync>d__17.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.CompileOnSave.CompileOnSaveHandler+<ProcessFiles>d__14.MoveNext()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(System.Threading.Tasks.Task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.CompileOnSave.CompileOnSaveHandler+<HandleFileSave>d__13.MoveNext()
   at Microsoft.CodeAnalysis.Editor.TypeScript.Features.CompileOnSave.CompileOnSaveHandler+<HandleFileSave>d__13.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0(System.Object)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
   at System.Windows.Threading.DispatcherOperation.InvokeImpl()
   at System.Windows.Threading.DispatcherOperation.InvokeInSecurityContext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at MS.Internal.CulturePreservingExecutionContext.Run(MS.Internal.CulturePreservingExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Windows.Threading.DispatcherOperation.Invoke()
   at System.Windows.Threading.Dispatcher.ProcessQueue()
   at System.Windows.Threading.Dispatcher.WndProcHook(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
   at MS.Win32.HwndWrapper.WndProc(IntPtr, Int32, IntPtr, IntPtr, Boolean ByRef)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(System.Object)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(System.Delegate, System.Object, Int32)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(System.Object, System.Delegate, System.Object, Int32, System.Delegate)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(System.Windows.Threading.DispatcherPriority, System.TimeSpan, System.Delegate, System.Object, Int32)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr, Int32, IntPtr, IntPtr)

From this it is visible that it crashes when doing a Compile-On-Save and that it runs out of address space. I have plenty of physical memory free (several GB), but since Visual Studio is a 32-bit process, it isn't hard to exhaust the address space.

The project is fairly large, but this hasn't happened before, and we haven't added any significant amount of new code lately.

[mhegazy]

Thanks. we would love to look at the project if possible. we would be happy to sing any NDA to get access to your code.

The stack trace does not tell really where the memory is gone before this call. other than compile on save, do you notice increase in the VS process private working set?

[mhegazy]

//cc @vladima

[Vilx2]

No, I didn't pay attention to overall memory usage of VS (though it's a pretty big memory hog by default anyway).

Another hint that just came to my mind was that what I was doing this morning was some refactoring. While not a big thing in and of itself, it did introduce quite a bit of compilation errors for a while. Perhaps that is related? I'll check out if I can force it to crash this way.

I'll ask my boss about the NDA tomorrow (it's late evening here right now).

[mhegazy]

If you can share a heap dump we can also see what is using up your memroy

[Vilx2]

Gladly! But how do I make one? When VS crashes, it just shows the regular "Windows is checking for a solution..." and then "Restarting Visual Studio...". There isn't even a JIT debugger prompt for some reason. :/

I checked it out some more. It seems that the errors don't have much to do with it. The project compiles fine right now, but the problem is still there. I enter a new blank line, save the file, and off it goes. VS hogs around one and a half CPU cores, and the memory usage keeps steadily climbing on all counters there are (including private working set), until a few minutes later it crashes.

If I manage to hit "Save" again before that happens, the memory usage drops (somewhat) and the process stops. There is a message "The project contained errors. Output generation skipped." When I save again, the process starts again. Etc. Since I save often, the actual crashes happened more rarely to me yesterday.

I tried deleting the .ntvs_analysis.dat file, but that didn't help. Is there any other file where the Compile-On-Save feature saves it's partially compiled result? Perhaps that is outdated.

Anyway, I'll ask about that NDA soon.

[mhegazy]

Can you watt until memory/CPU usage spikes, make a full heap dump using 'procexp' or 'procdump' and share it?

[Vilx2]

I just found how I can do it with Visual Studio itself (attach another VS as a debugger, wait to crash, then Debug->Save Dump). Will that do? I could navigate the managed heap from there.

[Vilx2]

Also, is there some private way how I could send you a link to the dump file? I don't mind sharing it with you, but I wouldn't want to make it publicly available.

[mhegazy]

feel free to email it to me mhegazy at microsoft dot com

[Vilx2]

Sent it.

[Vilx2]

For the benefit of others coming across this issue, adding info from an email by @mhegazy.

The problem is caused by the combination of "compile on save" and "combine javascript output into file". The output from the operation is very large and causes large object heap fragmentation. The workaround is to avoid "compile on save" and use tsc --watch instead, or just compile manually. The fix for the issue is already on the roadmap, but it's a large change and won't be available soon.