Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create database seems to be stuck after --command is complete #24

Closed
alexey-tereshenkov-oxb opened this issue Apr 8, 2020 · 3 comments
Closed

Comments

@alexey-tereshenkov-oxb
Copy link

@alexey-tereshenkov-oxb alexey-tereshenkov-oxb commented Apr 8, 2020

@hmakholm
This issue is a bit in between the actual cli tool and the C++ extractor - not sure how to debug.

The steps to reproduce:

git clone https://gitlab.com/libeigen/eigen
cd eigen
mkdir build
cd build
cmake ..
codeql database create db --language cpp --command="make buildtests -j8" -j8 -vvvv --logdir=./qllogs --no-cleanup

The make buildtests is reaching 100% and there is no activity, however, there are no logging messages in the terminal as if the extractor doesn't do anything. Inspecting the log:

> tail qllogs/database-create-20200408.103644.678.log 
[2020-04-08 11:32:21] [build] [100%] Linking CXX executable matrix_square_root_5
[2020-04-08 11:32:21] [build] [100%] Built target matrix_square_root_5
[2020-04-08 11:32:23] [build] [100%] Linking CXX executable matrix_square_root_4
[2020-04-08 11:32:23] [build] [100%] Built target matrix_square_root_4
[2020-04-08 11:32:26] [build] [100%] Linking CXX executable gmres_2
[2020-04-08 11:32:26] [build] [100%] Built target gmres_2
[2020-04-08 11:32:48] [build] [100%] Linking CXX executable polynomialsolver_2
[2020-04-08 11:32:48] [build] [100%] Built target polynomialsolver_2
[2020-04-08 11:33:09] [build] [100%] Linking CXX executable sparse_extra_3
[2020-04-08 11:33:09] [build] [100%] Built target sparse_extra_3

The db structure:

> tree db -L 3
db
├── codeql-database.yml
├── log
│   └── build-tracer.log
├── src
│   ├── home
│   │   └── user.name
│   └── usr
│       ├── include
│       └── lib
├── trap
│   └── cpp
│       ├── compilations
│       ├── compiler_mimic_cache
│       ├── source
│       ├── tarballs
│       └── tmp
└── working
    └── tracing
        ├── compiler-tracing2022088708754768740.spec
        └── compiler-tracing2022088708754768740.spec.environment

16 directories, 4 files

If I run the make buildtests -j8 outside of the codeql call, it completes and the process terminates with exit code 0. So the command itself is valid and runnable.

I've waited for a few days thinking that maybe the extractor process has some work to do, but no files are being modified in the db directory over a few days since the make buildtests -j8 reached 100%. Maybe the work is going on, but there is an issue printing logging messages? But it would be very bizarre that it would need more than 2 days to complete creating a database - the project itself is not very big.

I've tried running this both on Linux and MacOS.

Linux:

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.4 LTS
Release:        18.04
Codename:       bionic

MacOS:

> sw_vers -productVersion 
10.15.2
> system_profiler SPSoftwareDataType
      System Version: macOS 10.15.2 (19C57)
      Kernel Version: Darwin 19.2.0

If there is any other piece of information you'd need to help me troubleshoot (compilers present and used and so forth), please let me know.

codeql version
CodeQL command-line toolchain.
Version: 2.1.0.

I've asked @jbj on lgtm.com about header-only projects support and I wonder if this can have anything to do with that?

@jbj
Copy link

@jbj jbj commented Apr 8, 2020

Ping @igfoo

(our response to this may be delayed because of Easter)

@igfoo
Copy link

@igfoo igfoo commented May 18, 2020

Thanks for the report. While the build did reach the end, it is a multi-threaded build, and one of the early extractions is still running. We've had a look into it, and it's caused by a performance issue in our C++ frontend. Unfortunately, it doesn't look like the fix is easy, and as it seems to rarely bite (this is the only project we've seen it in) I'm afraid fixing it will be low priority.

@alexey-tereshenkov-oxb
Copy link
Author

@alexey-tereshenkov-oxb alexey-tereshenkov-oxb commented May 18, 2020

@igfoo ah that's good to know, thanks for letting me know. I completely understand the priority - thanks again for looking into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.