Skip to content

/ codeql-cli-binaries

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

database analyze not honoring the memory limits. #11

Open
mvanotti opened this issue Jan 24, 2020 · 7 comments
Open

database analyze not honoring the memory limits. #11

mvanotti opened this issue Jan 24, 2020 · 7 comments
Labels
CLI bug

Comments

@mvanotti
Copy link

@mvanotti mvanotti commented Jan 24, 2020

I am running the full cpp-lgmt query suite against a big codebase, and specified the memory limit to be 30GB, and 30 jobs, and I think it might be exceeding that limit.

According to HTOP, it is now using 70G of ram (still running and growing), and the number of threads is around 80.

This is the command I invoked:

mvanotti@markov:~/codeql-home$ ./codeql/codeql database analyze fuchsia-ql ql/cpp/ql/src/codeql-suites/cpp-lgtm-full.qls --format=sarif-latest --output=cpp-lgtm.sarif -j 30 -M 30240 --no-rerun
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Jan 24, 2020

Some of the program flags:

-Xmx15120M --off-heap-ram=15120 --verbosity=progress  --threads=30
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Jan 24, 2020

By looking at the output of pstree, I can see that there are 30 "pool" threads:

java(124002)-+-{AsyncAppender-W}(124029)
             |-{AsynchronousPro}(124146)
             |-{AsynchronousPro}(124148)
             |-{AsynchronousPro}(124149)
             |-{AsynchronousPro}(124150)
             |-{AsynchronousPro}(124151)
             |-{AsynchronousPro}(124152)
             |-{AsynchronousPro}(124153)
             |-{AsynchronousPro}(124154)
             |-{AsynchronousPro}(124155)
             |-{AsynchronousPro}(124158)
             |-{AsynchronousPro}(124159)
             |-{AsynchronousPro}(124161)
             |-{AsynchronousPro}(124162)
             |-{AsynchronousPro}(124189)
             |-{AsynchronousPro}(124201)
             |-{AsynchronousPro}(124202)
             |-{AsynchronousPro}(124203)
             |-{AsynchronousPro}(124209)
             |-{AsynchronousPro}(124212)
             |-{AsynchronousPro}(124214)
             |-{AsynchronousPro}(124215)
             |-{AsynchronousPro}(124216)
             |-{C1 CompilerThre}(124018)
             |-{C2 CompilerThre}(124017)
             |-{Common-Cleaner}(124024)
             |-{Finalizer}(124015)
             |-{G1 Conc#0}(124010)
             |-{G1 Conc#1}(124108)
             |-{G1 Conc#2}(124109)
             |-{G1 Conc#3}(124110)
             |-{G1 Conc#4}(124111)
             |-{G1 Conc#5}(124112)
             |-{G1 Main Marker}(124009)
             |-{G1 Refine#0}(124011)
             |-{G1 Refine#10}(124222)
             |-{G1 Refine#11}(124223)
             |-{G1 Refine#12}(124224)
             |-{G1 Refine#13}(124225)
             |-{G1 Refine#1}(124164)
             |-{G1 Refine#2}(124165)
             |-{G1 Refine#3}(124166)
             |-{G1 Refine#4}(124167)
             |-{G1 Refine#5}(124168)
             |-{G1 Refine#6}(124169)
             |-{G1 Refine#7}(124183)
             |-{G1 Refine#8}(124184)
             |-{G1 Refine#9}(124221)
             |-{G1 Young RemSet}(124012)
             |-{GC Thread#0}(124008)
             |-{GC Thread#10}(124039)
             |-{GC Thread#11}(124040)
             |-{GC Thread#12}(124041)
             |-{GC Thread#13}(124042)
             |-{GC Thread#14}(124043)
             |-{GC Thread#15}(124044)
             |-{GC Thread#16}(124045)
             |-{GC Thread#17}(124046)
             |-{GC Thread#18}(124047)
             |-{GC Thread#19}(124048)
             |-{GC Thread#1}(124030)
             |-{GC Thread#20}(124049)
             |-{GC Thread#21}(124050)
             |-{GC Thread#22}(124051)
             |-{GC Thread#23}(124052)
             |-{GC Thread#24}(124053)
             |-{GC Thread#2}(124031)
             |-{GC Thread#3}(124032)
             |-{GC Thread#4}(124033)
             |-{GC Thread#5}(124034)
             |-{GC Thread#6}(124035)
             |-{GC Thread#7}(124036)
             |-{GC Thread#8}(124037)
             |-{GC Thread#9}(124038)
             |-{Reference Handl}(124014)
             |-{Service Thread}(124022)
             |-{Signal Dispatch}(124016)
             |-{Sweeper thread}(124019)
             |-{TimeoutThread}(124181)
             |-{TimeoutThread}(124182)
             |-{TimeoutThread}(124185)
             |-{TimeoutThread}(124187)
             |-{TimeoutThread}(124188)
             |-{TimeoutThread}(124191)
             |-{TimeoutThread}(124192)
             |-{TimeoutThread}(124193)
             |-{TimeoutThread}(124194)
             |-{TimeoutThread}(124196)
             |-{TimeoutThread}(124197)
             |-{TimeoutThread}(124199)
             |-{TimeoutThread}(124200)
             |-{TimeoutThread}(124220)
             |-{TimeoutThread}(124227)
             |-{TimeoutThread}(124231)
             |-{TimeoutThread}(124233)
             |-{TimeoutThread}(124234)
             |-{TimeoutThread}(124235)
             |-{TimeoutThread}(124236)
             |-{TimeoutThread}(124237)
             |-{TimeoutThread}(124238)
             |-{VM Periodic Tas}(124023)
             |-{VM Thread}(124013)
             |-{java}(124003)
             |-{pool-1-thread-1}(124078)
             |-{pool-1-thread-1}(124087)
             |-{pool-1-thread-1}(124088)
             |-{pool-1-thread-1}(124089)
             |-{pool-1-thread-1}(124090)
             |-{pool-1-thread-1}(124091)
             |-{pool-1-thread-1}(124092)
             |-{pool-1-thread-1}(124093)
             |-{pool-1-thread-1}(124094)
             |-{pool-1-thread-1}(124095)
             |-{pool-1-thread-1}(124096)
             |-{pool-1-thread-2}(124079)
             |-{pool-1-thread-2}(124097)
             |-{pool-1-thread-2}(124098)
             |-{pool-1-thread-2}(124099)
             |-{pool-1-thread-2}(124100)
             |-{pool-1-thread-2}(124101)
             |-{pool-1-thread-2}(124102)
             |-{pool-1-thread-2}(124103)
             |-{pool-1-thread-2}(124104)
             |-{pool-1-thread-2}(124105)
             |-{pool-1-thread-2}(124106)
             |-{pool-1-thread-3}(124080)
             |-{pool-1-thread-3}(124107)
             |-{pool-1-thread-4}(124081)
             |-{pool-1-thread-5}(124082)
             |-{pool-1-thread-6}(124083)
             |-{pool-1-thread-7}(124084)
             |-{pool-1-thread-8}(124085)
             `-{pool-1-thread-9}(124086)
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Jan 24, 2020

(Previously, I ran this same command with 100GB of memory limit, but it wasn't honored, and it ended up exceeding the machine's total memory).
@p0 p0 added the bug label Jan 24, 2020
@mvanotti
Copy link
Author

@mvanotti mvanotti commented Jan 24, 2020

I left it running overnight, and memory didn't grow beyond 82GB, but it looked like it wasn't making any progress: there were still 6 queries running, but only one core was at 100%. After 18 hours it hadn't finished so I stopped and started it again.
@p0
Copy link
Collaborator

@p0 p0 commented Mar 19, 2020

For legacy reasons, the CodeQL evaluator treats the off-heap RAM limit as advisory -- it will attempt to stay within it, but if necessary will try to use more. The alternative would be to abort evaluation with an out-of-memory error, but introducing this behaviour would break customers who rely on the analysis concluding while using a little bit more memory than requested.

We're considering whether we can gate this behaviour behind a flag, or perhaps make the limit hard-by-default and provide a flag to treat it as soft again.
@hmakholm
Copy link
Contributor

@hmakholm hmakholm commented Mar 19, 2020

@Manouchehri:

Does database create work the same?

The expensive parts of database creation are not affected by what @p0 wrote. Unfortunately we don't have much in the way of knobs to turn to adjust the memory usage in this phase, although limiting the number of threads should indeed relieve some of the pressure.

Another approach you could try would be to increase the heap limit of the JVM that runs codeql. You can pass a -J-Xmx1234M option to codeql database create, where 1234 is a number of megabytes. (Without this, the JVM defaults to fixed fraction of the physical RAM according to the OS).
@Manouchehri
Copy link

@Manouchehri Manouchehri commented Mar 19, 2020

@hmakholm I deleted my comment earlier after figuring out it was unrelated myself, but thank you for providing a solution anyway! =D
@adityasharad adityasharad added the CLI label Jul 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
CLI bug
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
@mvanotti @adityasharad @Manouchehri @hmakholm @p0
You can’t perform that action at this time.